Turing's Reaction

Dialogue as a model for interactivity in multimedia

Presented by Geoffrey Rockwell at MITH, March, 2002.

Abstract

It is common to describe certain computer-based artifacts as interactive. We think we know what this means, but like many terms it vanishes before the definition. In this talk I will try to first argue that it is important to ask about interactivity and I will then defend a definition of interactivity in multimedia with special attention to the discourse around interactivity in computer games.

Introduction

In "Computing Machinery and Intelligence" [1] Alan Turing proposed a game that could help resolve the question "Can machines think?" In the game, which Turing called the "Imitation Game" and which we today call the "Turing Test", an interrogator enters into a conversation through a terminal (or teleprinter) with a computer and a person. By asking questions of his or her two interlocutors the interrogator tries to guess which interlocutor is the computer and which is the person. If the computer fools the interrogators as often as not that they are the human then we can say of that computer that it is artifically intelligent. The interrogator is thus both questioner and judge - both participant and observer.

The game allows us to substitute for the philosophical question, "Can a machine think?" a pragmatically answerable question, namely, "Can a machine fool interrogators into thinking it is human?" This second question can be empirically tested, and in fact is, every year, in a limited version of the test, by competitors for the Loebner Prize.[2]

Some have complained that the Turing Test with its emphasis on written interaction may not be a test of intelligence or, at least, it should not be the only test of intelligence. I want to focus on the legacy of the Turing Test as a paradigm for human computer interaction. Setting aside the question of machine intelligence, the Turing Test gave computing a model for what a truly interactive computer should be able to do, a model that sets written dialogue up as the paradigm of intelligent reaction. This paradigm influenced the direction human computer interface development took until the mid 1980s. The command line type of interface, which we all used to love before it was replaced by the desktop, was a dialogical interface where you typed commands for your not-so-intelligent interlocutor and waited for their response. It was only a matter of time before the people in AI and Natural Language Processing were going to make the computer more articulate and we would be able to dialogue with our servants as if with humans. Dialogue even survived into the current interface paradigm, the desktop, in the form of agents and "dialog boxes"[3] though now our interlocutors appear as paper clips and animated bicycles.

This paper, however, is not about the dialogue model of human computer interaction, or about all that forgotten research on dialog management that, after the development at Xerox of the Graphical User Interface, looks so dated. No, this paper is about interactivity and response, and dialogue is the paradigm for interactivity - dialogue as way of understanding what we think the ideal in interactivity could be. In this paper I am therefore going to do the following:

1. Try to convince you that interactivity is an interesting problem,
2. Review the pockets of literature where we might look for a discourse on interactivity,
3. Ressurect dialogue, but this time the written dialogue and what we know about how that works rhetorically,
4. And then leverage that to propose a theory about interactivity.

Why ask about interactivity?

We think we know what interactivity is, don't we? So why ask about it? People are interactive when they respond appropriately to each other, when they "act" on each other. But now the word has been liberally applied to all sorts of digital artifacts to the point where it is not clear if there is anything that is not interactive.[4] The first reason, therefore, for asking about interactivity is because it has become such an overused and trendy word. It is the obvious truths that are overlooked and yet yield the richest insight when questioned.

Second, interactivity is important to the critical study of digital media because it is what is new in "new media." When confronted with multimedia works we have ample theoretical resources to deal with the graphic design, the interface, narrative, text, video and audio. Each of the media in multimedia have been theorized, and there are plenty of new media theories that are enhanced old media theories. For example, Lev Manovich in The Language of New Media [5] discovers that new media are actually cinema and he builds his otherwise excellent book from this vantage having, in the beginning, dismissed interactivity for interface. But what if there was something irreducible in digital multimedia, something that cannot be theorized by extending our current rhetorical theories? What if that something new was the interactivity of these works? The possibility of the new alone should warrant an exploration of this residue.

Third, we have a particular problem in digital media studies which is the study of computer games. Computer games, not the refined hypertext fictions that are usually theorized, are what users are buying and playing, and therefore they are what our theories need to account for. The challenge of computer games in particular is how to deal with them as games, as works that users interact with in play. Because they are games, they cannot easily be reduced to other rhetorical genres. The problem, if you will, is that while there has been some theorizing about play, we have not tackled games in general as a rhetorical genre and are therefore unprepared to deal with digital media that present themselves as games to be played not read.

Another way of putting this is that, if you treat a computer game as nothing more than the sum of its graphical, textual, and other media components you miss the point, which is that it is a game and one that is played. It is in the playing that the user has fun and it is in the playing that the rhetorical experience of the work lies, not in the combination of media. Only if we address that aspect of a game that reacts to us can we explain their popularity and their rhetorical effect.

Theories of Interactivity

I do not have room here to survey the literature around play and interactivity in the normal way, summarizing selected positions. In addition I should note that one of the interesting features of the literature around games and interactivity is that there is no one literature, there are literatures, and they are closed in disciplinary silos not talking with each other. The idea of computer games as a serious research project is so new, that the isolated pockets of interested thinkers haven't yet started to meet. For this reason I am going to survey the types of fields where we can find discussion of interactivity and computer games, and they are:

1. After the Columbine High School massacre in Littleton and the Tabor, Alberta copy-cat killings there has been a explosion of interest in computer games and violence.[6] The academic literature coming from psychologists and sociologists on the subject of the pejorative effects of computer games predates the horrific events in Littleton (there is almost always someone who warns us about new media) but these events have brought the literature press attention.[7] We can call this a sociological or psychological literature interested in the social and psychological effects of games, not their rhetorical effects.

2. There is a literature around educational games coming from educational theorists and the instructional technology community. By contrast, while the first silo is critical of the effects of games, most of the educational literature is aimed at exploiting the motivating character of games for educational ends.[8] Instructional technology research tends to breathlessly report on the possibilities for educational games rather than carefully assess the outcomes of expensive projects.

3. There is a popular literature about the history of computer games, personalities in the industry, and companies like Atari. An example of this sort is Zap!: the Rise and Fall of Atari[9] though a recent work, Supercade[10] , wins the prize for the best illustrations. Associated with this literature is the business press about the computer game business. Business sources seem perpetually surprised to report the money being made on games which sell to a comparatively narrow bandwidth of the population. (My teenage son being their chief consumer.) This literature generally doesn't deal with interactivity, though some of the works like Supercade document the history of games and provide screen shots for those wishing to review the evolution of interactive games.[11] And, inevitably stories about classic games will include descriptions of how the game played, stories which, with the loss of workable copies may be our only record of those games.[12]

4. There are a number of computer game related WWW sites and popular magazines aimed at the consumers of these games. A student of mine, Gord Roberts, has assembled a textbase called GamesBase of 570 samples from magazines (mostly Nintendo Power) and other sources that can be found at http://tactweb.humanities.mcmaster.ca . This textbase is a searchable sociolinguistic resource for those interested in studying the discourse around computer games in the popular gaming literature. While it is not about interactivity, it is a place to start a study about how interactivity is discussed in the gaming community.

5. Perhaps the most interesting source, if you believe we should listen to practitioners, is the literature written by game designers and the computer game press about how to design a computer game. This ranges from the more reflective works like Chris Crawford's The Art of Computer Game Design to "how-to-in-24-hours" books of dubious value except to dummies.[13] Because such works are aimed at game developers (or wannabe developers like me) they almost always have a chapter on game-play or interaction. What is important about this literature is that describes interactivity from the perspective of the game designer and thus provides a window into the discourse of those that create the interactive.

6. There is a philosophical literature around play and sport of which the best known example is perhaps Huizinga's Homo Ludens, though there is also a sub-field of philosophy emerging around the philosophy of sport that deals with play in sport and the nature of the game.[14]

7. Finally there is, of course, the literature around hyeprtext, hypermedia, and new media of which Lev Manovich's The Language of New Media is one of the most recent, and I think best interventions. This literature generally deals with issues of interactivity either in the limited sense of the hypertext link (surely there is more to interactivity than linking), or deals with interactivity as a feature of the interface.[15] Other works of note are Aarseth's, Cybertext; Perspectives on Ergotic Literature, and Janet Murray's, Hamlet on the Holodeck.

No doubt there are other sites of discourse around the interactive. In particular I think of the literature on interface design coming out of computing, but this is a literature I want to bracket for reasons that will be clearer later. This list will do for now as a survey of where we can look..

Dialogue and the Definition of Interactivity

So what is interactivity and how can dialogue help us understand it? Setting aside the influence of the Turing Test, it may not be clear why I chose to define interactivity by looking at dialogue. One reason is that the definitions of others tend to suggest dialogue. For example, Chris Crawford defines interaction as "a cyclic process in which two actors alternately listen, think, and speak."[16] Sounds like dialogue to me. This definition picks up on a feature of Huizinga's and later Gadamer's definitions of play - that, "In this faculty of repetition lies one of the most essential qualities of play."[17] There is in play and interactivity a cycle of actions back and forth like the interaction of interlocutors in a dialogue. But what really suggested this line of enquiry to me was a section of Stewart Brand's The Media Lab on interactivity, titled "The Conversational Desktop".[18] Brand starts that section off by observing, "I had heard bits and pieces of the Media Lab party line about interactivity for so long that I finally asked Andy Lippman for the full story and taped his answer. ... The conversation about conversation that Lippman and I fell into is here presented verbatim." (p. 45) He continues to note that, "the point is that very awkwardness, the perpetual mutual trespass of real conversation." (p. 46) Not only is what follows this introduction an early definition of interactivity (and by that I mean a definition before the hype around the term), it is one that is reported to have emerged from conversations at the Media Lab (an important site for new media development), but the definition takes the form of a transcribed dialogue, which the rest of the book is not. Brand presents the dialogue as an example of that which is talked about in the conversation, a move not uncommon in philosophical dialogues.

But how does Lippman react to the question about interactivity? In the dialogue he defines it as, "Mutual and simultaneous activity on the part of both participants usually working toward some goal, but not necessarily." (p. 46) He provides corollaries which are:

1. Interruptibility - that an interactive system allows participants to interrupt each other, not just alternate complete turns

2. Graceful Degradation - that an interactive system will always encounter something it can't respond to fully, in which case it needs to gracefully "degrade"

3. Limited Look-Ahead - that an interactive system is capable of looking ahead in order to anticipate the other

4. Impression of an Infinite Database - that an interactive system gives the impression of being able to handle an infinite variety of interactions with an infinite number of responses

5. No Default - that an interactive system does not push a default trajectory for the interaction

Dialogue and Eavesdropping

I am going to digress now to talk about dialogue, another one of these slippery terms.

First, it should be noted that our paradigm for what dialogue can be has come down through a genre of philosophical literature - the philosophical dialogue and in particular the Socratic dialogues of Plato. Plato's Socratic dialogues present themselves as records of exemplary conversations that include the hero of dialogue, Socrates. When we look closely at the Socratic dialogues we discover that they do not work quite as we think they do. By and large they are scripted conversations that are meant to be overheard by their intended audience not examples of how we should engage each other constructively. This pattern shows up in the conversations and is replicated by the rhetorical structure of written dialogues where the reader is invited to join the audience.

The common view of the Socratic method, that the Socratic teacher engages his interlocutor for their benefit, does not hold when we look at the dialogues and when we ask how they work rhetorically, both as conversations and as written works. Socrates' interlocutors rarely leave enlightened - rather, they leave the encounter frustrated and embarrassed in front of the intended audience of youths for whom the performance was arranged. Xenophon, in one of his Socratic dialogues (Memorabilia, IV. II.)[19] actually tells us how Socrates used stand-off staged conversations to humiliate and seduce educated youths. Socrates would stage a conversation with one of his circle within hearing of the youth - a conversation designed to intrigue and humble the youth. These conversations with the tame members of the club were scripted and not open. Only once the vain youth was exposed to the staged dialogue would Socrates approach the youth off-line (so to speak), further hammer him with questions and once he was deconstructed invite him to join the circle.

My point is that the paradigmatic dialogues that have come down to us are conversations generated by predetermined rules and they are designed for the benefit of the eavesdropper not the interlocutors. This is a pattern that is true not just of the written dialogue, but of the spoken dialogue - think of how we talk about dialogue - for a conversation to be dialogue we have to be able to stand back from it and overhear it. Dialogue is not for the interlocutors, but for the eavesdroppers, even when we participate in the script.

A second point to be made about dialogue is that it has traditionally been seen as a literature of leisure at the margin of serious philosophy. Starting with Cicero and more thoroughly theorized later, the dialogue has been considered the play of philosophy. Dialogue is what noble men engage in when they are not working, as an alternative to the fashionable entertainments of the poets, who, you will remember Plato was going to banish from the ideal state. An Italian theorist and writer of dialogues, Sperone Speroni (1500-1588), compared the gardens of dialogue where we wander after work, rest, and are entertained (but are not nourished), to the productive fields of Aristitotelean discourse which feed us and the state.[20]

Interactivity as Dialogue

So, what does this understanding of dialogue offer us as a way of reacting to interactivity? The first point, which will seem counterintuitive, is that interactivity is not between the user and the computer. Instead I propose that interactivity is something we eavesdrop on. Interactivity is between the characters on the screen that we watch. We can control one or more of those characters, but the interactivity is between agents of the same order - namely the components/characters on the screen and, just as in dialogue we listen in from outside, even when we are participants, so in interactive multimedia we stand outside from our avatars controlling them and watching how they interact with the world.

If we apply these observations about dialogue to interactivity we come to the startling conclusion that interactivity is designed to be watched, that the point of interactivity is not necessarily the mutual reaction of the human and the computer, but is instead the scripted interaction presented on the screen between objects and characters within the game.

That is not to say that the player is a passive witness. Rather, I am suggesting a double involvement of the player. On the one hand they control a character or some other more primitive presence on the screen. That control is a matter of HCI (human computer interaction) or ergonomics (as far as the physical controls are concerned) and Interface (as far as the representation of the player in the virtual world is concerned.) Interactivity, then is that which happens between your avatar and the virtual world populated by virtual objects and characters.

Advantages and Disadvantages to this Theory

This way of speaking about interactivity as separate from ergonomics, HCI, and interface design has the virtual virtue that it is closer to the language of immersion around games. I don't say that I interacted with a computer where I saw my avatar interact with a blood-dripping deamon from outer space. I say I killed the deamon (and it sprayed blood all over me.) The interaction is that which takes place between the programmed entities within the game. I forget myself in the flow of the game and act as if I were that character on screen. I forget the controls and the interface and, for a moment, play my role.

This perspective on interactivity has other advantages. It reflects the way game design happens in the world of consoles and personal computers. The game designers rarely control the physical interface, they can only control the objects that appear and how they interact with each other. Further, it allows us to focus on the essential contribution of game designers (as opposed to the graphic designers that work on games) - the development of the game engine. These game engines are like the rules that cover board games; they define what the objects and characters are and how they can interact in what sort of world. The player controls one or more characters and the others are controlled by an AI. The developers of the game engines program the interactivity between the objects they define which allows this definition of interactivity to focus on the programming of interactivity as the concrete work of the game designer.

This framework also gives us a handle on how to critique a computer game, especially the highly interactive ones that are hard to discuss using hypertext theory or narrative theory.

1. We can ask about the world in which the interaction takes place. What is the setting or the stage of the dialogue.

2. We can ask about the other interlocutors or characters out character meets in the game. What type of people inhabit the game world and how do they react to out player?

3. We can ask about the hero - the player's character that you control.

a. Specifically we can ask about the possibilities presented us for our character. Who can you be? What are the options given for defining your character? (Note how often the first phase of the play of computer games involves choosing your character from a palette and adorning her.)

b. How is our avatar represented to us? What are the ways the interface tells you about yourself from control panels to third-person perspectives on your role.

c. What can your character do in the world? What actions is it programmed to be able to do?

4. Most importantly we can ask about the rules that govern the game. These rules in a computer game are algorithms implemented in code in the game engine. They are what define the interactivity between characters and world. These rules for behavior are the rules for play that make the game a game and not just a collection of media.

Finally, to return to advantages of this theory, it makes room for the humanities to engage an area that can be dominated by technical and engineering issues. This is why I chose to bracket the field of HCI and Interface design when surveying the literature. The humanities asks how things work rhetorically, and this theory is designed itself to fit interactivity not in a tradition of human computer interface, but in a rhetorical tradition that includes dialogue as a pattern for structuring representation. If humanities computing is going to contribute to the discussion we have to make clear what we bring to the table and how computer games are not just advanced computing interfaces, but also communicative works, artistic works, and rhetorical works worthy of treatment in a tradition of playful artifacts.

There are, however, some disadvantages to this approach to interactivity. First, there is the danger of an infinite regress, or the appearance of a regress. If I put off interactivity as that which happens not between human and computer, but as that which happens between objects in the game world, that would appear to still leave unanswered the question of what interactivity is. We could find that the interactivity between characters is in fact also put off and witnessed - leading to an infinite regress.

Second, the bracketing of HCI and Interface theory from interactivity, while it works for some types of games, does not work for those games where the physical interface is a custom part of the game. Examples would be the Tamagotchi, Furby, and other digital pets. Each of these, in so far as they are games, has a close integration between play and physical design which makes it hard to discuss interactivity without discussing the physical design and ergonomics of these robotic toys. I would further say that I think such games are going to become more popular and therefore more important to treat, especially since they appeal to girls in a way that console games have not.

End

Notes

1. Turing, Alan, "Computing Machinery and Intelligence". Originally published in 1950, Mind, 59, p. 433-460. For an online version see: http://www.loebner.net/Prizef/TuringArticle.html, accessed March 14, 2002.

2. For more on the Loebner Prize see: http://www.loebner.net/Prizef/loebner-prize.html, accessed May 17, 2002.

4. See the online version of Chapter 1 of Chris Crawford's Underestanding Interactivity where he has images of ads which feature the word prominently. http://www.erasmatazz.com/Book/Chapter%201.html, accessed March 14, 2002.

6. See http://www.salon.com/news/special/littleton/ , accessed March 14, 2002. One of the essays collected is relevant to this paper entitled, "Doom, Quake and mass murder" can be found at: http://www.salon.com/tech/feature/1999/04/23/gamers/index.html , accessed March 14, 2002.

8. An example of this would be, Rieber, Lloyd, "Seriously Considering Play", Educational Technology Research & Development 44.2 (1996): 43-58. A preprint of this is available at http://it.coe.uga.edu/~lrieber/play.html, accessed March, 2002.

13. Crawford, Chris, The Art of Computer Game Design. This out of print book is available in PDF form at Crawford's site http://www.erasmatazz.com/ . At this site you will also find a Library of other materials by Crawford on game design and interactivity.

16. Crawford, Understanding Interactivity, http://www.erasmatazz.com/Book/Chapter%201.html , accessed March 14, 2002.

Created by GMR, 2002