Location-Based Mixed and Augmented Reality Storytelling

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

11

Location-Based Mixed and Augmented Reality Storytelling Ronald Azuma

CONTENTS 11.1 Motivation 11.2 Reinforcing 11.3 Reskinning 11.4 Remembering 11.5 Conclusion References

11.1

259 261 265 269 272 274

MOTIVATION

One of the ultimate uses of mixed reality (MR) and augmented reality (AR) will be to enable new forms of storytelling that enable virtual content to be connected in meaningful ways to particular locations, whether those are places, people or objects. By AR, I refer to experiences that superimpose or composite virtual content in 3D space directly over the real world, in real time (Azuma, 1997). However, this chapter also includes a broader range of MR experiences that blend real and virtual in some manner but may not require precise alignment between the two (Milgram and Kishino, 1994). Initially, AR applications focused on professional usages that aided perception of a task that needed to be done in a complex 3D environment, such as medical surgeries or the maintenance and assembly of equipment. This focus was logical, because in the early days of AR, the equipment required was so specialized and expensive that only professional applications seemed economically viable. Later, access to MR and AR technologies became democratized through marker and image-based tracking via

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

cameras attached to desktop and laptop computers, smartphones and tablets. This enabled almost everyone to run certain forms of MR and AR on devices that they already bought for other purposes besides AR. Today, we see a variety of MR and AR applications that target the mass market for advertising, entertainment, and educational purposes. In the future, these experiences will advance to the point of establishing new forms of media that rely upon the combination of real and virtual to tell stories in new and compelling ways. In traditional media, such as books, radio, movies, TV and video games, the content is entirely virtual and disconnected from the location where the content is experienced. MR and AR storytelling experiences will offer new ways to tell stories, in different ways than traditional media, with new advantages and disadvantages compared against established media. A wide variety of platforms and systems run the AR and MR storytelling experiences that I cover in this chapter. There is no single platform or system that dominates, partially because AR and MR technologies are still evolving and also because the creators build or adapt custom systems to fit their particular experiences. A system that runs inside the controlled environment of a museum can be very different from mobile systems that bike riders carry with them as they ride through a city. This lack of standard platforms increases the challenge of telling stories with AR and MR technologies, requiring storytellers to also become familiar with the capabilities and limitations of the underlying technologies. Storytelling is fundamentally important, and any advancements in media technology that enable people to tell stories in new and potentially more compelling ways can have profound impact. While almost everyone enjoys good stories as a form of entertainment, the importance of storytelling runs much deeper than that. Telling a story is an important method of education and instruction. Stories can contain lessons, codified bits of wisdom that are passed on in a memorable and enjoyable form. Technological developments that make the story clearer and more memorable can aid retention and understanding. I firmly believe that, in the long run, one of the ultimate uses of MR and AR technologies will be as a new form of location-based media that enables new storytelling experiences. The goal of this chapter is to discuss location-based MR and AR storytelling. I provide a hypothesis of why this might be a powerful new form of media, and specify three approaches for achieving this potential. The bulk of this chapter provides an overview of various experiences or concepts that provide a glimmer of the potential inherent here. While I attempt to discuss a representative sample of approaches in this area, this is not a comprehensive survey, so it does not cover all previous work in this field. To conclude the chapter, I discuss both a fundamental challenge limiting this new type of media, along with what an ultimate payoff might be.

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

MR and AR storytelling is a particular subset of a much broader area of locationbased experiences that include ARGs (alternate reality games), puzzle hunts, crossmedia and trans-media experiences, pervasive games, and performance art. For a broader discussion of these other types of experiences, please see Benford and Giannachi (2011), Harrigan and Wardrip-Fruin (2007), and Montola, et al. (2009). This chapter focuses on MR and AR systems that explicitly attempt to tell a story to the participants, rather than ones where the focus is on playing a game, solving a puzzle, or providing entertainment or an artistic experience. This chapter also focuses on location-based storytelling experiences that generally occur outside a participant’s home and, in many cases, operate only at specific sites. Therefore, I do not cover the case of augmenting a real book with 3D virtual content to supplement the story that exists in the traditional book. A commercial example of an AR book is Anomaly (Brittenham and Haberlin, 2012). The reason this chapter does not cover AR books is my hypothesis about what will make Augmented Reality storytelling compelling: the combination of real and virtual must be meaningful and powerful, where the core of the experience requires both the real and virtual components. If the experience is based on reality by itself, with little contributed by the augmentations, then there is no point in using AR. Conversely, if the core of the experience comes solely from virtual content, then the augmentation part is only a novelty and it will not be a viable new form of media. Many AR experiences fall into the latter case. In the case of books, DVD cases and movie posters, what is compelling about reality is not the book, DVD or poster. It is the virtual content represented or embodied by those objects. The compelling content resides in the ideas in the books and the movies themselves, not the physical objects. Therefore, an experience that augments a book or movie poster with virtual augmentations derives all its power from purely virtual content. Reality then becomes a backdrop that forms the context of the experience, and perhaps part of the user interface, but reality is not a core part of the content. I hypothesize that there are at least three approaches for AR storytelling where both real and virtual form critical parts of the experience: • • •

Reinforcing Reskinning Remembering

In the next three sections, I discuss these three in more detail and describe examples and concepts of each approach.

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

11.2. REINFORCING In reinforcing, the AR storytelling strategy is to select a real environment, whether that is an object, person, or location, that is inherently compelling by itself, without augmentation. Then, the AR augmentations attempt to complement the power that is inherent in reality itself to form a new type of experience that is more compelling than either the virtual content or reality by themselves. Let me provide a conceptual example. Let’s assume the goal of an experience is to educate a participant about the Battle of Gettysburg. A student wishing to learn about that battle could watch a 1993 movie, called Gettysburg, which had star actors, superb cinematography, a compelling soundtrack, thousands of Civil War reenactors, and parts of it were even filmed on the site of the battle. However, Gettysburg is also a real location. If you are so inclined, you can travel there and see the battlefield yourself, in person. And if you do this, you will see large grassy fields, stone fences, and many monuments. You will not see any reenactments of the battle or other virtual recreations that take you back to that fateful time in 1863. Yet, if you know why that spot is important in American history, then simply being there, on the actual spot where the event happened, is a powerful experience. I remember standing at the spot of Pickett’s Charge, on the Union side, and feeling overcome by emotion. (For the reader unfamiliar with the American Civil War, Pickett’s Charge was the culmination of the Battle of Gettysburg. The Confederates lost both Gettysburg and Vicksburg, on July 3 and 4, 1863. These two events are generally considered to be the turning point of that war. Because the Union won the American Civil War, the United States became one unified, indivisible country rather than separated into two or more countries.) An AR storytelling experience that was located on the battlefield itself might be able to draw from the best of both real and virtual to provide a new type of compelling experience. The virtual content embodied by the film Gettysburg, emplaced in the actual location where the battle took place, could be more powerful than experiencing either independently. My favorite example of this strategy is 110 Stories, which was designed by Brian August (August, 110Stories – What’s your Story?). This experience runs on a mobile phone. If you are near or in Manhattan, the application uses the compass and tilt sensors in the phone to render an outline of where the Twin Towers in the World Trade Center should be, against the New York City skyline (Figure 11.1). To me, there are two design decisions that make 110 Stories particularly poignant. The first was the choice of how to render the buildings. Even on mobile phones, we have graphics hardware that could render a detailed, perhaps nearly photorealistic representation of the Twin Towers. But the application does not do that. Instead, it renders only the outline of the buildings, as if sketched against the skyline with a grease pencil. While

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

this reduces the realism, I believe this makes the experience more effective, because it matches the message that this experience tries to send: That the towers are no longer there. And they are supposed to be there. The second aspect is that after this application augments Manhattan with the outline of the towers, it invites the participant to not only capture and upload that augmented image, but to also submit a brief story. Why did the participant choose to take this picture? What does it mean to the photographer? If you go to the 110 Stories website, you will see many stories that participants have chosen to record and share. One that I have never forgotten tells how the author grew up in Manhattan, and his brother told him that as long as he could see the Twin Towers, he would never be lost because they told him which direction was south. They would always guide him home. And the story concluded by stating that his brother was now forever at home, with the implication being that his brother was in the Twin Towers on September 11, and that he missed him.

Figure 11.1 110 Stories.

In the Voices of Oakland experience, participants experience stories about people who are buried in Oakland Cemetery, the oldest cemetery in Atlanta (Dow et al., 2005). The core experience is a singular linear story that takes participants along a designated path through the cemetery, but with options to explore additional content if the participant chooses. The creators describe the historic locations as imbued with an aura that can make such experiences compelling. Because this is a historic cemetery,

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

the creators could not modify anything about the environment, nor add markers or other elements to aid tracking. Furthermore, they took care to develop an experience that was appropriately serious in tone and respectful of the cemetery, the relatives of the occupants, and all other stakeholders. The virtual augmentations consisted of audio played when the participants were at the correct spots. Professional actors provided the narration of the character voices. Due to limitations in tracking, a Wizard of Oz approach was taken, where another person selected the content to be played based upon the participant’s context and actions on the control device. At Columbia University, Steven Feiner’s group developed a technique called Situated Documentaries, in which MR techniques were used to enable a broad range of virtual content to be seen in the context of the real locations where events actually happened. In Columbia’s example, they built a location-based experience where a participant would walk around the Columbia campus to see audio, videos, images, web pages, 3D augmentations and other media describing past events (Höllerer et al., 1999). Their experiences told stories about the history of the Columbia University campus, describing a student revolt, how students used tunnels to occupy buildings that were guarded by police, and how one campus building was the former location of the Bloomingdale Asylum. Commercial versions of the Situated Documentaries technique now exist. The Streetmuseum Londinium app from the Museum of London provides a historical view of London in Roman times (Museum of London, Londinium App). As the participant walks to designated sites in the city, he or she can experience audio, video, and imagery based on archaeological finds from that time period (Meyers, 2011). The Streetmuseum application enables images or videos to appear to be augmented over real backgrounds in the participant’s surrounding environment. A similar effort was started by PhillyHistory.org, a repository of historical images of Philadelphia. In April 2011, they released a mobile application that uses Layar as the platform for augmenting a user’s view of the city with historical photos aligned with the real background (PhillyHistory.org, 2011). Hayes (2011) provides further information about these and other commercial projects. Dow Day is an AR experience in which participants role play journalists in the year 1967, investigating student protests at the University of Wisconsin-Madison against the Dow Chemical company for manufacturing napalm used in the Vietnam War (Squire et. al., 2007). While set up as a game that requires participants to learn how journalists perform their job, it also serves to engage them in learning about this specific historical era and events through personal stories and testimonies, where these virtual materials were tied to the real locations where the events happened. Dow Day uses the augmented reality and interactive storytelling (ARIS) platform.

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

As a final example, I will discuss The Westwood Experience, a location-based MR experience that I worked on (Azuma, The Westwood Experience by Nokia Research Center Hollywood, Wither et al., 2010). We conducted this experience in December 2009 and January 2010 in a part of Westwood, CA that is south of the UCLA campus. It was an experiment testing a variety of MR effects to enhance a location-based, linear story that the participants experienced as they walked the streets of Westwood. In brief, the participants assembled in a theater where they met an actor portraying the honorary Mayor of Westwood. They were given mobile phones and earphones and left the theater on their own, guided by clues to specific points in the town. First, they experienced effects that visualized the town in the year 1949, attempting to turn the clock back to that time period. Then they heard a story of a striking young woman that the protagonist met, loved, and lost. They experienced this story at the locations where these events were supposed to have happened: at the café where they met, the building where they spent the night together, a jewelry store where he bought a ring, and the last spot where he saw her as she disappeared in a taxi. The payoff, and the reason why this experience was located in Westwood, comes at the end, when our protagonist informs us that now he wants us to meet the woman he just told us about in his story. The power comes when the participants realize how this is going to happen. The woman he talked about was a real person, and the way they will meet her is not by seeing a video or image of her on their mobile phones, nor even a 3D virtual model augmented in space. They will meet her, the real person, by visiting Westwood Village Memorial Park Cemetery. The narration at this point sets the expectation of how the power of this place will affect their thoughts and behaviors: She became what she said she would, a movie star… In the end, Norma came back to Westwood too. She’s between engagements now, ‘resting’ as actors sometime say. She’s not alone, but among many others, some who lived their lives as publicly as she did, many of their names once as familiar as hers. I come to see her every so often, as many others do. I’m taking you to her now. I know you’ll be mindful of the customs proper to the place we find her, a place of real people and real endings. In contrast to Voices of Oakland, which the participants knew would be in a cemetery, this experience surprised most of the participants by ending in a cemetery, particularly because this one is small and hidden away behind numerous high-rise buildings. They were guided to a specific crypt, where they discover that the woman in the story was Norma Jeane Baker, better known to the world by her other name: Marilyn Monroe. We used a Situated Documentaries technique at this spot, showing newsreel footage of her funeral. In certain sequences from the newsreel, the participants can see

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

a clear, meaningful and one-to-one correlation between the crypts they see in the surrounding environment and the footage that plays on the mobile device. Combined with the somber music that was composed specifically for this spot, the effect was a powerful and poignant coda to this experience. The emotions that participants reported feeling at this spot were different than if they simply read a tourist guidebook and then visited the cemetery. By experiencing a story about her prior to visiting her crypt, they were left to contemplate her life before becoming a movie star, and to wonder if the story they just experienced might have been real. Many AR storytelling experiences that rely on the reinforcing strategy use the technique of connecting the story to the past. Being able to increase the range of stories that can be told through new AR and MR effects will require advancements in our ability to track and augment historic outdoor environments. The Archeoguide project (Vlahakis et al., 2002) was an early effort to develop a platform for augmenting archaeological sites. Reinforcing as a strategy has strengths and weaknesses. On the positive side, the experience does not rely solely upon the virtual content by itself. Since reality itself is compelling on its own, the real world does some of the work of providing the meaningful experience. It may be easier to design and build virtual content that complements reality rather than virtual content that must shoulder the entire burden of being compelling by itself. In 110 Stories and The Westwood Experience, I believe there are examples demonstrating that this strategy can succeed. On the negative side, the experience is tied to a specific location. A person wishing to participate in 110 Stories must travel to Manhattan. A specific experience does not scale; it cannot be experienced at any arbitrary location, but rather only the one it was designed for. However, many different experiences might be built for different locations around the world. Furthermore, the story must be tied to the characteristics of the real location. One cannot tell any arbitrary story and expect reinforcing to work; instead, the story must complement the reality that exists at the chosen site. In The Westwood Experience, we walked around Westwood with the writer, and he wrote the story to incorporate real elements in the town, such as a jewelry store. And we were very aware that our experience ended in a real cemetery. A story that was disrespectful of that reality or that provided experiences inappropriate to that location would at best fail to harness the power of that real place, and at worst would be offensive. Reinforcing requires the story to appropriately complement reality.

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

11.3. RESKINNING In reskinning, the strategy is to remake reality to suit the purposes of the story you wish to tell. Reality is either something that the creators specifically set up and then augment, or the experience is designed to recharacterize whatever real surroundings exist. Unlike reinforcing, there may not be anything particularly special or evocative about the real location, which means experiences based on reskinning can potentially scale to operate in most arbitrary real locations, or reward the participant for finding locations that work well for the experience. However, most of the power from the experience must now come from the virtual content and how it adapts and exploits the real world to fit that virtual content. Rainbows End is a Hugo-award winning science fiction book written by Vinge (2006) that provides one ultimate concept of reskinning. In this book, nearly perfect AR is ubiquitously available to people who can operate the latest wearable computing systems, which use displays embedded in contact lenses and tracking provided by a vast infrastructure of smart motes that permeate almost all inhabited locations. Within this world, there are Belief Circles, which are persistent virtual worlds that are linked to and overlaid upon real locations. Each Belief Circle has a particular theme, such as a fantasy world set in medieval times. When a user chooses to subscribe to a Belief Circle, he or she sees the surrounding world changed to fit the theme. For example, in a medieval Belief Circle, nearby real buildings might appear to be castles and huts, and people on bicycles might instead appear to be knights on horseback. A Belief Circle has a large group of people who subscribe to it and create custom content, and when others view that content, the creators can receive micropayments. We can view a Belief Circle as a persistent, co-located virtual world that links directly, one to one, to our real world and uses the principle of reskinning to change reality to fit the needs of the virtual content and experience. Unlike the world of Rainbows End, we do not currently have ubiquitous tracking and sensing, so there are few examples of reskinning, and those often rely upon real environments that were specially created to support the needs of the story. Two examples are AR Façade and Half Real. Façade is an interactive story experience where the participant plays the role of a dinner guest visiting a couple whose marriage is just about to break apart (Mateas and Stern, 2007). Façade supports free text entry so that the participant can type in anything to converse with the two virtual characters while walking around freely in the virtual environment. The experience is not a linear story. Depending on what the participant does or says, various story beats are triggered and experienced. For example, walking up to or commenting on a particular object or picture will trigger

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

certain narrative sequences. Façade by itself is a virtual environment that runs on a PC and monitor. Researchers built an AR version, called AR Façade, in which they built a real set that replicated the apartment that is the setting of this experience (Figure 11.2), and participants wore a wearable AR system and walked around the real set to see virtual representations of the couple (Dow et al., 2008). The goal was to provide the participants a greater sense of actually occupying a real apartment and interacting more naturally with the apartment and its virtual inhabitants. For example, instead of typing in what they would say, participants now simply said what they wanted, directly. Rather than relying upon voice recognition, human operators working behind the scenes then typed in what the participants said into the system. The evaluation did not directly attempt to measure whether AR Façade was more engaging or compelling than Façade by itself. However, there was evidence that AR Façade did affect some participants emotionally. Some chose to quit early rather than participate in an experience that was an uncomfortable social situation that they were expected to take an active role in. Others became highly engaged, showing visible signs of surprise and emotional connection, such as running to follow one of the virtual characters when she leaves.

Figure 11.2 The real set of the apartment in AR Façade.

Half Real is a theatrical murder mystery play that uses spatial AR to merge real actors and a physical set with virtual content and to engage the audience with interactive situations where the audience members vote on how an investigation proceeds (Marner et al., 2012). Actors were actively tracked so that virtual labels could be attached to them. Each audience member had a ZigZag handheld controller to vote

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

when prompted. The real set is painted white so that projective textures can change the appearance of the set during the performance. The creators had to work out numerous system issues to provide the reliability, robustness and transportability required of a professional stage production. Half Real completed a tour in South Australia and subsequently played for a 3-week, sold out run in Melbourne. Future possibilities include using the augmentations to change the appearances of the actors themselves, rather than just the set and the backgrounds.

Figure 11.3 The Intel Scavenger Bot demonstration at CES 2014, reskinning a real environment with a virtual grid pattern.

Applying the reskinning technique outside of controlled real environments, such as those of AR Façade and Half Real, requires AR systems that can detect and understand the real world. Kinect Fusion exploits the depth-sensing capabilities of the Kinect to build a volumetric representation of a real environment, enabling a system to track off of that space and augment it more realistically, with correct occlusions (Newcombe et al., 2011). Such a system represents a step along the directions needed for AR and MR systems to more commonly enable the reskinning technique. For example, the Scavenger Bot demonstration that Intel showed at the Consumer Electronics Show (CES) 2014 showed a system that could scan a previously unknown tabletop environment and then change its appearance by applying a different skin upon

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

the environment (Figure 11.3). This system can also handle dynamic changes in the environment. While we are now seeing systems that can detect and model the real world, these models generally lack semantic understanding. True reskinning will require systems that can detect and recognize the semantic characteristics of the environment and objects. The University of Central Florida provided an example of reskinning the interior of a museum to better engage visitors with the exhibits. In the MR Sea Creatures experience in the Orlando Science Center, visitors saw the museum interior transformed to be underneath the sea, and skeletons of ancient sea creatures on display then came to life (Hughes et al., 2005). Visitors navigate a virtual rover vehicle to collect specimens around the museum. At the end, they see an animation of one dinosaur grabbing a pterodactyl out of the air and holding it in its mouth, which then transforms back to the real world where the visitors see the real fossil of that dinosaur with the pterodactyl in its mouth. The Aphasia House project is an exciting new application of MR storytelling which enables patients suffering from traumatic brain injury to tell their own personal stories to therapists, not for the purpose of entertainment, but as a critical part of guiding a doctor in determining how to treat a patient (Stapleton, 2014). People suffering from aphasia are impaired in their ability to communicate due to severe brain injuries. They may lose the ability to speak, read, or write. Preliminary results from this project indicate that immersive storytelling in an MR environment may enable some patients to reconnect with their abilities to tell stories, and a doctor involved in this project testifies that this breakthrough would not have been possible in a purely Virtual Environment or without the augmentations provided in the MR environment. What appeared to be critical was building a real environment (a kitchen) that could be augmented in a variety of ways to elucidate familiar previous experiences from the patient: making coffee, eating a bagel, touching countertops, and doing that in a multimodal way so that he felt, heard, and smelled familiar sensations. This is an example of applying reskinning to evoke stories out of a patient in the pursuit of a serious goal: helping patients recover their own abilities to communicate. Reskinning relies most on the power of the experience coming from the virtual content, rather than the real environment, so a key strategy may be to exploit virtual content that participants are already familiar with. When this content is created by professional storytellers and audiences who have already read the books, seen the movies, or otherwise experienced the virtual content, then a new experience that leverages that same content is not starting from scratch. It has an advantage in that the audience already finds the virtual content compelling. One example of this is the Wizarding World of Harry Potter at the Universal Studios Orlando theme park in Florida. While this is not explicitly an example of AR or MR storytelling, it is an example of this

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

leveraging strategy. Since most visitors are already familiar with the Harry Potter books or films, when they walk through that area of the theme park and experience the attractions and shops there, they draw from their memories and previous knowledge of this fantasy world. Such leveraging is the basis of many cross-media or trans-media approaches, and it can be quite successful. The Wizarding World of Harry Potter was sufficiently popular that Universal expanded it in the summer of 2014. Alice’s Adventures in New Media was an early AR narrative experiment that leveraged the world of Alice in Wonderland, written by Lewis Carroll (Moreno et al., 2001). In this system, a participant sat at a table and saw three other characters from the book. The participant could interact with the characters by performing various actions such as serving and sipping tea, which affected the narrative snippets.

Figure 11.4 A Leviathan demonstration in the Intel booth at CES 2014.

At CES 2014, Intel ran a series of AR demonstrations based upon the steampunk fantasy world of Leviathan, written by Westerfeld (2009). These AR demonstrations were intended to inspire visitors about the potential for AR storytelling that used this leveraging strategy (Azuma, Leviathan at CES 2014). I was part of a large team of people who created and ran these demonstrations. The world of Leviathan is set in an

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

alternate Earth, during World War I, where mankind discovered genetic engineering very early. Therefore, in some countries a biological revolution supplanted the industrial revolution, and people chose to fabricate new types of living things to suit their purposes. For example, the Leviathan itself is an enormous flying airship in the form of a whale, replacing dirigibles. In our demonstrations, we brought virtual representations of the Leviathan and other creatures inspired by the book into the real environment, both during the Intel CEO’s keynote presentation and in the Intel booth on the CES show floor (Figure 11.4). While these demonstrations did not tell stories by themselves, they served as an inspiration of how this leveraging strategy could result in compelling new storytelling media when applied through the reskinning strategy of AR and MR storytelling.

11.4

REMEMBERING

In remembering, the AR storytelling strategy is to draw upon memories and retell those stories, generally at the particular place where those memories and stories happened. The belief is that combining the memories and stories with the actual real location can result in a new experience that is more powerful than the real location by itself, or the virtual content by itself. For example, I could revisit the site of my wedding ceremony and see the gazebo where that occurred. While I have photos and videos of that event, communicating my personal story of that day and what that meant to me might be done in a more powerful manner as an AR or MR experience, merging that virtual content with the actual location where my wedding occurred. The strategy of remembering is similar to reinforcing, but there are some differences. The locations used in the reinforcing approach have particular meanings and power that most people agree upon and know. For example, the site of the Battle of Gettysburg draws its power from a specific event. While interpretations might vary, the meaning is shared and agreed to by almost all participants, and that constrains the experiences based on reinforcing to conform to that meaning. Remembering, in contrast, is generally more personal and individual. With this approach, the potential stories and memories can vary greatly, even at the same location. For example, Sproul Plaza on the campus of the University of California, Berkeley could be home to a wide variety of memories and stories. One person might remember participating in the Free Speech Movement at that spot, while another knows it as the place where he first met his future spouse, and yet another has memories of Pet Hugs sessions where students could hug therapy dogs to reduce their stress.

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

Figure 11.5 Three Angry Men, seeing two other jurors from one point of view.

Even when divorced from a particular location, memories and viewpoints by themselves can make compelling experiences. Three Angry Men (MacIntyre et al., 2003) and its successor, Four Angry Men, are experimental AR narrative demonstrations that enable participants to access and experience the memories and thoughts of characters in a narrative, from their particular points of view. Inspired by the drama Twelve Angry Men, written by Reginald Rose, these experiences place the participant in the viewpoint of a jury member deliberating on a case. When seated at a particular chair in a table, the participant sees the other jurors from the perspective of the juror who is sitting in the chair he or she is occupying (Figure 11.5). The participant not only hears what the other jurors say and what his or her persona is saying, but he or she also hears the inner thoughts of the character sitting in that chair. The participant is free at any time to switch seats. When he or she does so, the deliberation continues but the participant now hears and sees things from a different juror’s perspective and hears that juror’s inner thoughts. For example, one juror with liberal leanings sees another African-American juror as a potential ally but the third juror as prejudiced. When the participant moves to the seat of the prejudiced juror, the entire experience changes. While the initial juror heard the prejudiced juror as loud and unreasonable, the prejudiced juror hears himself as reasonable, if a bit frustrated. Even the appearances of the jurors change depending on the viewpoint. To the prejudiced juror, the African-American juror’s appearance and behaviors transform to conform to his biases. Three Angry Men provides an example of how AR storytelling could communicate, at a first-person level, how stories and memories change based on personal perspectives and biases, which has been called the Rashomon effect after Akira Kurosawa’s film. REXplorer was a system that encouraged participants to explore and learn about the historic town of Regensburg, Germany and its well preserved medieval city center

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

through MR storytelling techniques (Ballagas et al., 2008). Although REXplorer is primarily a game that asks participants to go on quests to specific locations within the city center, it motivates these quests through virtual characters, ghosts who used to inhabit the town, who have requests to make of the participants and stories to tell them. The participants learn about these characters and their stories and perform tasks such as carrying a love letter to another character inhabiting a different location in town. By performing these tasks, participants indirectly explore the historical city center, with the goal of learning history in a more entertaining and enjoyable manner. Some participants found that using stories in this manner injected life into a historical tour that otherwise might have been dry and boring. Rider Spoke is a location-based experience in which bike riders were encouraged to record personal stories and memories associated with particular locations, at the spot where those occurred (Benford and Giannachi, 2011). The virtual content consisted of the audio recordings. Riders could add recordings only in spots that did not already have content associated with that location, ensuring that each location had unique content. The system provoked the participants to leave significant and evocative memories. For example, one instruction asked a participant to find a spot that his or her father would like and to talk about that. In this experience, participants were not just passive consumers of content but active generators, contributing their own personal stories, as if mapping diary entries to specific spots in the city. Rider Spoke was conducted in 10 cities across the world. You Get Me was a 2008 experience in which participants selected one of eight young people to hear his or her stories and, perhaps, make a connection to that person (Blast Theory, 2008). The eight young people had communication and tracking equipment and walked around a park. Each person has a key question that he or she wanted help in answering. Participants went to computer terminals at the Royal Opera House, about 5 miles away from the park, to select one of the young people and then explored the park virtually. As the participant moved in the virtual representation of the park, they heard stories relating to that person, the personal geography of how that park maps to the chosen young person. For example, one person arranged stories around a swimming pool in which she nearly drowned. The stories give clues for answering that person’s question. The participant can then track down the person in the park and attempt to answer the question. If the person thinks the answer is insufficient, he or she can reject it and force the participant to explore more of the personal geography and stories. But if the person finds the answer intriguing, he or she can invite the participant for a private chat or phone call. And in the conclusion, the person takes a photo of himself or herself with the park in the background and sends it to the participant. You Get Me is a compelling experience, connecting the participant in an intimate manner, via MR storytelling techniques, to one of eight real people with real concerns and real

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

stories, overlaid on a real park that each person uniquely maps as his or her own personal geography. Since we do not have ubiquitous tracking systems with the desired accuracies needed to build indoor/outdoor AR and MR experiences, our ability to implement storytelling experiences based on the remembering approach is constrained, but the potential is there to enable individuals to create and make available their own personal stories for others to see in the context where they occurred. Not all stories have to be written by professional storytellers, aimed at a mass market audience. Some stories might be of interest only to your family or a close circle of friends, or only to specialized audiences. But that does not make them any less important. Sometimes, the stories that are the most important to us or to others are the ones that represent ourselves, who we are, what our goals and aspirations are, and where we came from.

11.5

CONCLUSION

AR storytelling is still in an early, exploratory phase. While there have been many initial experiments, as this chapter has discussed, I feel that as a form of media, it is still very early in its development. It reminds me of an early phase of the development of motion pictures, where some of the first movies featured footage of moving trains, showing what the technology could do. Advancing the technology of moving pictures from those early days into the art form of cinema that we know today required progress on many fronts, not just in technology, but also in art, design, and business models. In AR and MR storytelling, we do not yet have the equivalents of the early pioneers in cinema, such as Buster Keaton, Sergei Eisenstein, and D.W. Griffith. These future pioneers will need to overcome some of the core challenges of this new form of media while simultaneously unlocking its potential. One of the most important challenges in AR and MR storytelling is motivating people to make the necessary effort to participate in these location-based media. These experiences generally require participants to leave their homes and travel to particular locations or venues. That requires effort and costs resources, in terms of time, money, etc. In comparison, one can watch a film or a see a TV show almost anywhere. It takes little effort to turn on the TV and watch a show on the DVR, see a movie, or play a video game in one’s house. Why would someone get off the couch and instead participate in these new location-based media? The answer is that AR and MR storytelling experiences must become compelling enough to convince participants that this effort is worthwhile. Despite our TV sets,

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

game consoles and comfortable couches, people still leave home to go to a movie theater, see a sporting event in a stadium, go to theme parks, visit a museum, travel to distant sites on vacation, etc. Those experiences are attractive enough that people willingly spend the extra effort to participate in those. Initially, AR and MR storytelling might leverage such situations, augmenting those experiences that are already proven to draw people out of their homes. As the medium develops, I look forward to such experiences being sufficient by themselves to attract participants. What would be a payoff that would make people eager to participate in locationbased experiences? The Walt Disney Company provided an example in a 2013 Alternate Reality Game called The Optimist (Andersen, 2013). This provided a series of experiences that Disney fans could participate in, culminating in an elaborate puzzle hunt that took place at the 2013 D23 Expo and in Disneyland. The people who knew about these events and who chose to attend at the specific locations and dates were rewarded with access to locations that the general public normally cannot enter. These locations included the Club 33 private club, Walt Disney’s apartment above the fire station on Main Street, and the Lilly Belle caboose car on one of the railroad trains. For Disney fans, visiting these locations provided highly desirable and special experiences, ones they would remember and forever cherish. While compelling, this is a specific approach that requires special locations and does not generalize or scale to most situations. A more general approach toward achieving compelling experiences will be to realize the potential inherent in the medium to see the world around you through the eyes, viewpoint and mindset of another person. To me, an ultimate expression of the potential of AR and MR storytelling is if it can cause you to view the world in a different way, and if this impact is powerful enough that it actually changes your own belief system, how you view the world and make decisions. I can give an example of the desired impact through something that happened to me through real life experience: • •

A friend of mine, who worked on several projects with me, had a stroke. He now requires a powered wheelchair to travel anywhere.

I now view the world differently than I did prior to this incident, because I have traveled with him to many events. Before, I would not think twice about curbs or stairs or other things that are insurmountable obstacles to my friend. Now, I am sensitive to the locations of ramps, elevators, and other items that provide wheelchair access. AR and MR storytelling experiences have the potential to change how we view the world, to make us see the world from a different perspective, such as that of my friend, and to in turn change our belief systems and values. This different perspective can be

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

cultural, political, historical, social, or any other dimension. But if an experience can change me, in a way similar to what I just described, that is proof that experience is compelling. We know that traditional media, such as film, plays, and books, have this power and there are examples in each where people have found those stories memorable, compelling and life altering. When we have equivalent examples in AR and MR storytelling, that exploit the new potentials in this form of media, then we will know that it has matured sufficiently to stand equally with established media. I look forward to this day.

REFERENCES Andersen, M. “The Optimist” draws fans into fictionalized Disney history. Wired, July 23, 2013. http://www.wired.com/2013/07/disney-the-optimist-arg/ (accessed May 5, 2014). August, B. 110Stories – What’s your story? http://110stories.com (accessed May 5, 2014). Azuma, R. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6 (4), 1997, 355-385. Azuma, R. Leviathan at CES 2014. http://ronaldazuma.com/Leviathan_at_CES2014.html (accessed May 2, 2014). Azuma, R. The Westwood Experience by Nokia Research Center Hollywood. http://ronaldazuma.com/westwood.html (accessed May 2, 2014). Ballagas, R., A. Kuntze, and S. Walz. Gaming tourism: Lessons from evaluating REXplorer, a pervasive game for tourists. Pervasive Computing 2008, Sydney, New South Wales, Australia, May 19-22, 2008, pp. 244-261. Benford, S. and G. Giannachi. Performing Mixed Reality. Cambridge, MA: MIT Press, 2011. Blast Theory. You Get Me. http://www.blasttheory.co.uk/projects/you-get-me/ (accessed May 12, 2014). Brittenham, S. and B. Haberlin. Anomaly. Anomaly Publishing, 2012.

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

Dow, S. J. Lee, C. Oezbek, B. MacIntyre, J. D. Bolter, and M. Gandy. Exploring spatial narratives and mixed reality experiences in Oakland cemetery. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, Valencia, Spain, June 15-17, 2005, pp. 51-60. Dow, S., B. MacIntyre, and M. Mateas. Styles of play in immersive and interactive story: Case studies from a gallery installation of AR Façade. In Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, Yokohama, Japan, December 3-5, 2008, pp. 373-380. Harrigan, P. and N. Wardrip-Fruin, eds.. Second Person: Role-Playing and Story in Games and Playable Media. Cambridge, MA: MIT Press, 2007. Hayes, G. Transmedia Futures: Situated documentary via augmented reality. 2011. http://www.personalizemedia.com/transmedia-futures-situated-documentary-viaaugmented-reality/ (accessed May 5, 2014). Höllerer, T., S. Feiner, and J. Pavlik. Situated documentaries: Embedding multimedia presentations in the real world. In Proceedings of the 3rd IEEE International Symposium on Wearable Computers 1999, San Francisco, CA, October 1819,1999, pp. 79-86. Hughes, C., C. Stapleton, D. Hughes, and E. Smith. Mixed reality in education, entertainment and training. IEEE Computer Graphics and Applications, 25(6), 2005, 24-30. MacIntyre, B., J. D. Bolter, J. Vaughn, et al. Three angry men: An augmented-reality experiment in point-of-view drama. In Proceedings of the 1st International Conference on Technologies for Interactive Digital Storytelling and Entertainment, Darmstadt, Germany, March 24-26, 2003, pp. 230-236. Mateas, M. and A. Stern. Writing Façade: A case study in procedural authorship. In Second Person: Role-Playing and Story in Games and Playable Media, P. Harrigan and N. Wardrip-Fruin (eds.). Cambridge, MA: MIT Press, 2007, pp. 183-207. Marner, M., S. Haren, M. Gardiner, and B. Thomas. Exploring interactivity and augmented reality in theater: A case study of Half Real. In IEEE International Symposium on Mixed and Augmented Reality 2012, Arts, Media and Humanities Proceedings, Atlanta, GA, November 5-8, 2012, pp. 81-86. Meyers, K. Revealing Londinium Under London: New AR App. Cultural Heritage Informatics Initiative. http://chi.anthropology.msu.edu/2011/07/revealinglondinium-under-london-new-ar-app/ (accessed May 5, 2014).

Chapter 11 in 2nd Edition of Fundamentals of Wearable Computers and Augmented Reality, Woodrow Barfield, (editor), CRC Press, August 2015, pp. 259-276.

Milgram, P. and F. Kishino. A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12), 1994, 1321-1329. Montola, M., J. Stenros, and A. Waern. Pervasive Games: Theory and Design. Burlington, MA: Morgan Kaufmann Publishers, 2009. Moreno, E., B. MacIntyre, and J. D. Bolter. Alice’s adventures in new media: An exploration of interactive narratives in augmented reality. In CAST01, Bonn, Germany, September 21-22, 2001, pp. 149-152. Museum of London. Londinium App. http://www.museumoflondon.org.uk/Resources/app/StreetmuseumLondinium/home.html (accessed May 5, 2014). Newcombe, R., S. Izadi, O. Hilliges, et al. KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2011, Basel, Switzerland, October 26-29, 2011, pp. 127-136. PhillyHistory.org, Implementing Mobile Augmented Reality Technology for Viewing Historic Images. An Azavea and City of Philadelphia Department of Records White Paper. 2011. http://www.azavea.com/research/companyresearch/augmented-reality/ (accessed May 5, 2014). Squire, K., M.F.Jan, J. Matthews, et al. Wherever you go, there you are: Place-based augmented reality games for learning. In The Educational Design and Use of Simulation Computer Games. Sense Publishing. 2007, pp. 265-296. Rotterdam, The Netherlands Stapleton, C. Developing stories that heal – A collaboration between Simiosys and the Aphasia house. http://simiosys.com/blog/?p=459 (accessed June 16, 2014). Vlahakis, V., N. Ioannidis, J. Karigiannis, et al. Archeoguide: An augmented reality guide for archaeological sites. IEEE Computer Graphics and Applications, 22(5), 2002, 52-60. Vinge, V. Rainbows End. New York: Tor Doherty Associates, 2006. Westerfeld, S. Leviathan. New York: Simon Pulse, 2009. Wither, J., R. Allen, V. Samanta, et al. The Westwood Experience: Connecting story to locations via mixed reality. In IEEE International Symposium on Mixed and Augmented Reality 2010, Arts, Media and Humanities Proceedings, Seoul, Korea, October 13-16, 2010, pp. 39-46.