- 1 Gee 2003; 2004; 2013; 2014; 2015.
1In this paper I will be concerned with “designed experiences”. Designed experiences are experiences – in the real world or via media – that are designed to elicit specific effects or affects. In my work on digital media and learning1 I have investigated how teachers, in their classrooms, or game designers, in their games, design experiences that are meant, in both cases, to lead to learning. However, designed experiences can be intended to elicit other things than learning. They can also seek to elicit things such as social change, attitude or behavioral changes, emotions, or other effects of (or on) the body, the mind, or the soul.
2When parents raise a child, they usually do not just turn the child loose to have experiences in the world. Rather than such a “sink or swim” approach, they often re-arrange the world to design (create and constrain) experiences for the child. In this way the child can learn safely and develop along positive ways.
- 2 Barsalou 1999; Bergen 2012; Gee 2004; Clark 1997; Glenberg 1997.
3All humans, young and old, learn from experiences in the world. According to contemporary research on the human mind, this is, in fact, the major source of human learning2. However, just as with children, adults often do not learn well from raw experience. They do not always know what to pay attention to and can become overwhelmed by the myriad of details in any experience. Even with adults, in many cases, mentors, leaders, guides, teachers, media designers, and artists design experiences for them. Eventually people (hopefully) also learn to design their own experiences in order to deliberately facilitate their own learning, growth, and change.
4Humans do not always strongly distinguish between experiences they have had in the real world and experiences they have had in media3. They get similar effects from both. Sometimes they do not even remember the source of an experience that helped build their mental storehouse. This is why media like books, films, art, and games are important for human development.
- 4 Bergen 2012; Gee 2004; 2015; Glenberg and Gallese 2012.
5Humans do not just learn from experience. They use their prior experiences to think, plan, and get ready for action in the ‘here and now’ when they need to act to accomplish a goal. They do this by building simulations in their minds based on what has happened in the past4. Humans use their past experiences to play out in their minds different possible courses of actions and outcomes when they have to act. These simulations can be factual or fictional, since humans can easily rearrange parts of our past experiences into imaginative combinations.
6Art is also a form of designed experience. Art uses design to help or guide people to have certain sorts of aesthetic experiences. But, before we get to art and aesthetic experience, let’s think a moment about language. Language has form (structure, grammar) and meaning (content). Normally people use the form (structure, grammar) of a piece of language to get meaning (and affect) and pay little overt attention to the form itself or how it works to create meaning. In a sense, they “see right through” the form to rapidly get to meaning (“content”). This is why people can often remember what something meant but not how it was actually said. It is if we look through a window onto the world but never really see the window or how it shapes or frames what we see.
7Art – like language – uses form (design features; relations among parts and wholes; its “grammar”) to create meaning. But art often guides the viewer to pay attention to how form gives rise both to meaning and feeling5. Art helps the viewer see the window as a frame. In turn, art helps the viewer to use this insight about the relationships between form and meaning to appreciate and value something in a new or illuminating way. Art, in this sense, is like science: it is a form of discovery.
8The design of a work of art involves not just the design of an object (e.g., a painting) or performance (e.g., dance). It involves also the design of an experience a person can have with the object or performance. The design of an art object or artistic performance has affordances for how one can experience it, though this experience can never be totally determined. Artists often try to design with an eye to helping people use these affordances in certain ways (though not always only one way).
9I am well aware this is all rather vague. But it is the best I can do now so that we can get on with the discussion without further philosophical musings.
- 6 Gee 2014; Schegloff 2006.
10For human beings, oral face-to-face conversational interaction is the primordial form of communication6. In a conversation one person designs language for a given recipient in order to elicit a particular sort of response from that recipient. This is called “recipient design”. The recipient then takes a turn at talk and responds, with due regard for the recipient design of the first speaker’s turn at talk. The responder uses his or her own recipient design in order to try to elicit certain sorts of responses from the first speaker when he or she takes back the floor. It is not for nothing that people have talked about the “art of conversation”.
11Plato in the Phaedrus7 famously pointed out that people cannot have conversations with books and paintings because when you ask them something or speak to them, they will not respond. They will not take their turn at talk.
12Certain forms of media today, however, will respond8. In a video game, when a player does something – and the player most certainly must design his or her action with respect for the game and its rules if the player wants to succeed – the game will answer back and let the player know how the game has “received” his or her “turn”. This is a new form of “conversation” (reciprocally-designed communicative turn-taking).
13We humans can also – in this extended sense – have conversations with the world9. In fact, this form of “conversation” is, in a sense, even more primordial than oral communication, since it antedates language in the course of evolution. Often, when we act in and on the world, we humans design what we do to elicit an effect we want to get from the world (and this better involve designing with due respect for the world and the context we are in, since the world can bite). The world responds to our action with an effect (outcome, response) that we perceive (appreciate) as good, bad, or neutral for our goals. Then, we act again with due respect (we hope) for what the world has “said”, and continue to pursue our goals.
14When we get a response from another person in oral conversation, from a video game in a game, or from the world when we act on it (probe it), we must appreciate that response in a certain way, we must assess it as good, bad, or neutral for our goals. Knowing how to make such judgments can be called an “appreciative system”10. We usually acquire such appreciative systems by being socialized into different (and sometimes competing) groups that have developed a certain sort of “taste” or “good judgement” within specific domains, whether these domains be birdwatching, playing real-time strategy games, gardening, or viewing modernist art. To the extent that art creates new ways to appreciate and value things, it helps people form new appreciative systems, to acquire new forms of “taste”.
15So, we are using here an extended sense of “conversation” (interactive turn taking) that allows us to say that we humans can have conversations with other people (now via social media as well as face to face), video games, and the world. Let’s call all these “designed interactive experiences”.
16In an oral conversation, each person is a designer for the other, seeking to elicit certain responses from the other. In a video game, the game is designed to elicit certain sorts of responses from players and players seek, via their game play, to get the game to respond back in certain sorts of ways.
17In interacting with the world, a person is a designer of actions, hoping to shape the world’s responses (get certain sorts of effects or outcomes). The world, of course, is not a designed thing (it is an evolved system), but, crucially, humans (even scientists) treat the world as if it were a “designed thing” when they want to get meaning from it and accomplish their goals11. Anything that we humans cannot see as designed (falling into meaningful parts and wholes) is random for us and not a suitable conversational partner. We see the world’s responses to our actions as meaningful and not random (or we not likely to survive long).
18So, back to Plato’s complaint. Books, films, sculpture, and paintings cannot respond. We cannot have a conversation with them. They, are in that sense, not interactive designed experiences. They are mute.
19And, yet, as anyone who has appreciated a good book, film, sculpture, or painting knows, there is a sense in which we can and do have conversations with things like books, films, and paintings. But those conversations are ones in which, in a sense, we have to supply both sides. We have to “question”, “probe”, “speak to” the work of art and then, with due consideration for its design, respond for it.
20So, it is better, perhaps, to distinguish not between interactive designed experiences and non-interactive designed experiences, but between “interacted conversations” and “enacted conversations”. We can, I believe, judge the quality of an oral conversation, a work of art, a game, or a set of interactions with the world (as in gardening) in terms of how good (meaningful, insightful, illuminating, life enhancing) a conversation design made possible in each case. Of course, this not the only way to judge quality.
21Now I will discuss how video games might illuminate certain aspects of art and experience – not all, just some – as I have delineated them above. Indeed, any new medium or artistic form has the potential to offer new sources of light and insight about important aesthetic issues, and, in the process, transform how we think about and do art.
22A video game is a designed virtual world wherein there are certain sorts of problems to solve. Video games are a form of constrained and well-designed problem solving12. Players enter this virtual world with the desire to solve these problems (the ones given to them by the game’s designers) and often also with the desire to form some of their own personal goals and accomplish them in their own ways. Players often, but not always, interact with a game’s virtual world via an avatar, a sort of surrogate body for the player through which he or she can manipulate the virtual world. In other sorts of games, players manipulate the world directly without the intermediary of an avatar.
- 13 Gee 2004; Glenberg 1997; Glenberg and Gallese, 2012; Tomasello 2014.
23Above I pointed out that humans learn primarily from experience and that they use stored versions of their past experiences to plan, think, and act in new situations in the future. According to contemporary research13, it turns out that experiences are best for human learning when the person has a goal-based action to take in the experience, an action whose outcome the person affectively cares about. When a person has such a motivating action to take in an experience, the person stores, remembers, and can use the experience more deeply and more adeptly than in an experience where there is no such affectively motivating action. Passive experience is much less good for learning than is active experience.
24Let’s call such active experiences “motivated action experiences”. Games are just such motivated action experiences and, thus, they are promising platforms for human learning. Indeed, this is why people like me and many others have been interested in games as a promising technology for learning, problem solving, and education in schools and society.
25Games can be about any set of problems (the set of problems constitute the game’s “content”). There are games about warfare (e.g., Call of Duty), stealth (e.g., Metal Gear Solid), leading lives in communities (e.g., the Sims), building civilizations (e.g., Civilization), designing objects and environments (Minecraft), colonizing planets (e.g., Spore), solving algebraic equations (DragonBox), collaborating (e.g., Two Brothers), surviving war (e.g., This War of Mine), revitalizing the environment (e.g., Flower), cleaning house as a four-inch robot (e.g., Chibi-Robo), immigration dilemmas (e.g., Papers Please), families and gender issues (e.g., Gone Home), and many, many more things.
26Even large-scale commercial games can be “art” in the sense that they offer us strong and emotionally charged images, music, actions, and stories (e.g., Castelvannia: Symphony of the Night; Shadow of the Colossus; Okima; The Legend of Zelda: The Windwaker; Bloodborne). However, here I am interested in the specific ways video games can create aesthetic effects of the sort I have referred to above as “appreciation”.
27In a single short paper it is impossible to get at issues about games and art by a survey of different games. So here I will discuss but one game: Thomas Was Alone (Twa). Twa is a game that uses very simple 2D images and sparse environments with limited possibilities for action and limited amounts of dialogue. It is about as minimal as a game can get, but for that very reason it exposes quite clearly the basic structures and functions of game worlds and how they can create artistic effects and aesthetic experiences.
28We will see that games are multi-modal forms of art. This means they use and combine different modalities – images, actions, sounds, music, and words – to achieve their effects. In this sense, even minimal games like Twa have a very complex “grammar” (or set of integrated “grammars”).
29Of course, not all games achieve – indeed, no other game achieves – the same effects in the same way as Twa. Each game is special on its own. However, my point here will be that other games work with a similar sort of palette of design tools and within a similar sort of design space. At the same time, it is important to remember that discussions of games as art are relatively new and my discussion here is only be meant to be suggestive and in no way definitive.
30Below is an image from Twa. Note that this image is static, but players never really experience game images as merely static. The placement of the characters in this image was determined by the player’s actions. Different choices about what to do – about a sequence of actions – would have resulted in a different image. Furthermore, the very next screen following this one will be created by the player’s actions based on his or her assessment of the problem to be solved at this point in the game.
31Each sequence of images a player causes, by his or her choices and actions, is rendered meaningful by those choices and actions within the overall context of the player’s strategies and goals. Each sequence is also rendered meaningful, as well, by the player’s interactions with the game’s story (if it has one) and the game’s overall design (the designer’s goals, the game’s characters and environments, the game mechanics, and the words and emotions expressed in the game).
32The small colored rectangles in Twa are like words in a language. They can be combined together just as words can be combined into phrases. For example, players can stack the rectangles on top of each other in certain orders (e.g., red on orange on yellow on blue). Order matters in a minimal way. Each shape has a characteristic movement of its own and each can be move independently, but when they are combined, the bottom one determines the movement of the whole stack. The bottom one functions like a verb in language. In the game, the player must get all the shapes to the end of each level.
33So we have a clear, albeit simple, grammar (design) here. Just like words, the shapes and their actions also take on more specific, nuanced, and extended meanings in the different contexts the game and the player’s actions place them in within the game. For example, in the case of language, the word “coffee” takes on two different contextually-specific meanings in “The coffee spilled, get a mop” and “The coffee spilled, get a broom”. In Twa, each rectangle takes on more specific meanings as the game’s story and game play unfold across time.
34Twa has a story. The story is narrated by a narrator whose narration is both heard and printed on the screens. Within the story each shape has a name and something of a backstory as an artificially intelligent agent inside a computer whose programming has gone awry. The shapes are trying to escape the system. In an interesting twist, each shape has certain unique abilities and limitations that fit with the character’s personality trait and role in the story.
35For example, Thomas, the red rectangle, has an up-beat attitude and can do an average jump. John, the yellow rectangle, is arrogant and eager to show off and can jump quite high. Claire, the blue square, who starts off feeling bad about herself but comes eventually to see herself as a super-hero, cannot jump well or move fast, but she can float and move in water and thereby save others by giving them rides across water.
36The game’s story allows us to assign meanings far beyond “short jumping yellow rectangle”. They allow us to assign emotional and narrative meanings to the rectangles as they act alone and together. The story – and our cultural knowledge about escape stories and about computers – give us cultural models or frames within which we can give richer interpretations to what is happening.
37In Twa the game’s oral narration is also printed on the screen. Since Twaprints the words of the narration on the screen, we do not need the voice of the narrator to recover the words. Indeed, we have often read the words before we have finished hearing them. This way of combining oral (heard) and written (seen) words, serves to subtract, in a sense, the words from the oral narration (renders them redundant). This means, in turn, that the oral narration mainly functions to carry the intonation (pitch and tone of the voice) of the narrator’s voice, the musical and affective part of speech. This is the only non-redundant part of what we hear from the narrator. The narration in Twa is in a British accent that is amazingly good at indicating the emotions and attitudes of the characters (rectangles though they be), emotions like fear, self-loathing, loneliness, liking and love, caring, arrogance, humility, and trust.
38Players attribute emotions and attitudes as deeper meanings for each character by considering the different contexts the characters are in. Consider the image from Twa above. Given the words on the screen, the positions of the characters, and the situation the characters are in at this point in the story and in our game play, we can attribute to John (the tall yellow rectangle) here a contextually specific meaning like: John wants to help, though not necessarily for altruistic reasons, but because he likes to show off and look good to others. This is an inference we make about John based on the context we are in, our overall game play so far, and the game’s story and design features (like the use of intonation for emotion).
39Let’s turn now to the ways meanings and inferences are built up in the context of the order, sequence, and flow of screens in the game. Let’s assume, for the sake of argument, that a screen prior to the image above had John, the yellow triangle, down on the same level as Thomas (the red rectangle) and Chris (the orange square). Assume further the player has jumped John – a very good jumper – up to where we see him in the image, that is, up above Thomas and Chris. From this vantage point, a player can clearly see that he or she could move John to the right, down the little alley, and then again further to the right and away from Thomas and Chris. John would, thus escape the level and that is a key goal of the game (and a key goal of John’s).
40But the player can also readily see that moving John to the right will not get Thomas and Chris up the ledge so they can move on in the game as well. They cannot alone or together jump high enough to get up the ledge. John must come back down and allow Chris to jump on top of Thomas and then to have Chris jump from Thomas to John and, finally, to jump up to the ledge. Thomas can then jump on John and then up to the ledge. And only then can John jump back up by himself. This creates a sequence in the player’s mind, a sequence that he or she can then create.
41In the image I have printed above we see that John is higher than Thomas and Chris and that he can easily go on without them. We see, too, that if the game (and its story) is to continue, he must go back down, place himself again on the same level as Thomas and Chris, help them, and then move on last (not first) himself. All these meanings derived from sequence reinforce the sorts of meanings we have drawn from the story and contexts of play. John thinks more highly of himself than he does of the others. Forced to go back and help, he has to rationalize this as not a weakness, but as a strength. This strength is not only that his help is essential to mitigate Thomas’s and Chris’s weaknesses. It is also that John will look good in the act and others will see how special he is.
42So far we have been discussing how the design of Twa works to create meaning. But we argued earlier that when a player plays a game, the player is also having a conversation with the game, indeed a more overt and reciprocal one than readers often can have with written texts. The player and the game respond to each other in turn. In Twa players are acting on and in the game world and they are watching how the game reacts back in terms of narration, what the characters do and “feel”, and whether the player’s actions are successful towards solving the game’s goals (which the player has accepted) and the player’s goals (which the player has set).
43So what is the conversation with Twa about? What sort of experience is it? What does it allow us to appreciate in new and illuminating ways? This question would, of course, be answered differently by different players. However, an indication of what I and many other players got out of our conversation with TWA is captured very nicely in a review of the game printed (in part) below: http://kotaku.com/5929049/thomas-was-alone-the-kotaku-review.
44Amazingly, I felt more connected to Thomas Was Alone’s colored, polygonal blocks than I have to most human characters in recent games. Part of the game’s genius is how attached you feel to Thomas, John and the other tinted, four-sided personas you encounter in Mike Bithell’s indie platformer.
45For me, Twa offers an illuminating insight into how we human beings perceive, attribute, and react to personality and emotions. It offers insight into how we humans use personality and emotions to attribute intentions, responsibility, purpose, and meaning to people and their actions. It offers insight into how story (as told and as enacted in this case) both creates emotions in us and for characters, but also how emotions transform our sense of and reception to a story.
46All of sudden we see the deeply human affective aspect of life stripped of its human face, but, nonetheless, become all the more human in the act. In the act, too, the role of personality and emotions in life, and maybe in our own lives, is rendered new and strange. We have new fodder for thought and emotion. I myself wonder why – if it is so easy to feel for and with these rectangles and to be exhilarated when they finally get free – we humans have such trouble feeling for and with other real humans just because they are “other” and do not seem like us or look like us.
47For me, as I reflect on my Twa experience, the way we humans attribute emotions and make meaning from such attributions can seem both fake and dangerous and, at the same time, a source of caring and empathy. The system seems fraught, fragile, resilient, and uplifting all at the same time. Like many video games, Twa causes me to see myself, others, and the real world in a new way. Since gamers actually act in and on a virtual world, they can offer imaginatively transfer certain perspective they have gained on the game world to the real one.
48Thomas Was Alone is but one game. Nonetheless, I want to suggest that the materials it uses are some of the design tools from which a variety of different aesthetic effects can be achieved by other games. However, I must admit that since video games are a new platform for art, we do not know much yet about the variety of these effects.
49The design materials Twa uses are 1) game designer goals melded with player goals, 2) problem solving as content, 3) designer and player co-created sequences of choice and action, 4) enacted stories that give and get meaning from choice and action, and 5) multi-modal combinations of elements like words, actions, images, sound and music, and, in Twa’s case, a deft separation of words as meaning and words as affective contours of pitch and tone.
50These are some of the materials with which games can create aesthetic experiences. As with music, painting, performance, film, and writing, the purpose of a game may not always be to create art. It may be, rather, to entertain, to teach, to motivate, or to convince. But, along with these, it may, indeed, be to create new forms of insight and appreciation.