Augmented reality using personal projection and retroreflection

Pers Ubiquit Comput DOI 10.1007/s00779-011-0374-4

ORIGINAL ARTICLE

Augmented reality using personal projection and retroreflection David M. Krum • Evan A. Suma • Mark Bolas

Received: 22 December 2010 / Accepted: 24 February 2011 ! Springer-Verlag London Limited 2011

Abstract The support of realistic and flexible training simulations for military, law enforcement, emergency response, and other domains has been an important motivator for the development of augmented reality technology. An important vision for achieving this goal has been the creation of a versatile ‘‘stage’’ for physical, emotional, and cognitive training that combines virtual characters and environments with real world elements, such as furniture and props. This paper presents REFLCT, a mixed reality projection framework that couples a near-axis personal projector design with tracking and novel retroreflective props and surfaces. REFLCT provides multiple users with personalized, perspective-correct imagery that is uniquely composited for each user directly into and onto a surrounding environment, without any optics positioned in front of the user’s eyes or face. These characteristics facilitate team training experiences which allow users to easily interact with their teammates while wearing their standard issue gear. REFLCT can present virtual humans who can make deictic gestures and establish eye contact without the geometric ambiguity of a typical projection

D. M. Krum (&) ! E. A. Suma ! M. Bolas Institute for Creative Technologies, University of Southern California, 12015 Waterfront Drive, Playa Vista, CA 90094-2536, USA e-mail: [email protected] E. A. Suma e-mail: [email protected] M. Bolas e-mail: [email protected] M. Bolas USC School of Cinematic Arts, University of Southern California, 900 West 34th Street, Los Angeles, CA 90089-2211, USA

display. It can also display perspective-correct scenes that require a realistic approach for detecting and communicating potential threats between multiple users in disparate locations. In addition to training applications, this display system appears to be well matched with other user interface and application domains, such as asymmetric collaborative workspaces and personal information guides. Keywords Augmented reality ! Head-mounted projection ! Training ! Retroreflective screens ! Pico-projector 1 Introduction In order to create flexible yet cost-effective training, military, law enforcement, emergency response, and other domains have employed simulation systems which combine computer-generated imagery with the physical ‘‘tools of the trade.’’ For example, flight simulators use physical flight controls in combination with computer graphics for cockpit windows. Firearms simulators for law enforcement can use real weapons, or modified training weapons, with imagery projected on walls. Augmented reality and, more broadly speaking, mixed reality [13] can carry this approach further, with training scenarios guided by programmable computer-generated content, but centered in tangible settings that allow users to take physical action. Mixed reality environments can be presented using projection screens that are embedded into staged flats or scenes [16]. However, projection screens cannot provide a perspective-correct view of the virtual elements without tracking the viewer. Furthermore, only a single perspective-correct view can be provided, accommodating only one viewer. This lack of universal perspective creates

123

Pers Ubiquit Comput

situations where a virtual human might establish eye contact or point at a particular person in a group, but no one can be truly certain if they are the intended recipient. Trainees will wonder, ‘‘Is it me, or the person next to me?’’ Additionally, incorrect perspective might lead to negative training in terms of detecting and communicating potential threats that, based on their position in the environment, should be visible only to certain squad members, but not others. Traditional virtual and mixed reality approaches are stubbornly problematic for squad-level training. While previous researchers have attempted to perform training with multiple users in immersive projection systems such as a CAVE, e.g., [9], only one head-tracked user receives the correct perspective, leaving the remaining trainees with a distorted view of the scene. Head-mounted displays are also incomplete solutions, since display screens in front of the trainees’ faces block their ability to see and communicate with each other. Optical or camera-based seethrough displays do allow squad members to see each other, but are limited in that they restrict the real-world field-of-view and require users to look through glass screens to view their squadmates. There has been much recent work with augmented reality mobile devices; however, these displays suffer from similar limitations when introducing multiple users. As no single virtual or augmented reality system has overcome all these limitations, providing an effective solution for immersive squad-level training is an important challenge that has yet to be solved completely. To address these unique challenges of immersive squad-level training, we have developed a near-axis retroreflective projector framework called REFLCT (Retroreflective Environments For Learner-Centered Training) [3]. This framework consists of a one or more pico-projectors, one or more retroreflective surfaces, and some form of tracking which is used to register the projected imagery on the retroreflective surface. REFLCT builds on useful characteristics of currently available systems with an end goal of unobtrusively delivering mixed reality training experiences. In its most basic form, REFLCT: • • • • • •

Places no glass or optics in front of a user’s face Needs only a single projector per user. Provides each user with a unique and perspectivecorrect image. Situates imagery within a physical themed and propbased environment. Can be low power, lightweight, and wireless. Works in normal room brightness.

With a basic REFLCT system, each user wears a tracked head-mounted projector (see Fig. 1). Each user can only

123

Fig. 1 A REFLCT head-mounted projector unit incorporates a projector, tracking markers, and a USB video camera onto a construction helmet

see the image generated by their own projector, due to the use of retroreflective screens. The retroreflective screens have the property of reflecting the light from each projector back to where it came from and is thus visible only to the wearer of the projector. By varying the configuration of the projector(s), retroreflective surface(s), and tracking, we can create instances of REFLCT that support a variety of training, collaboration, and personal information applications. To demonstrate some possible REFLCT variations, we have, for example, modified projectors for 3D stereoscopic imagery, used human face-shaped retroreflective surfaces, and plan to use camera-based fiducial tracking for smartphone-based systems. In this paper, we will first focus on the military training application and then later discuss additional variations and applications of REFLCT.

2 Related work There have been three main approaches to creating individualized and perspective-correct imagery for multiple users: projector arrays, head-mounted displays (HMDs), and head-mounted projective displays (HMPDs). Projector arrays coupled with asymmetrically diffusing screens [2, 10] can create individualized perspective-correct views, but are expensive in terms of hardware and calibration effort. They require more projectors than users, with projectors positioned everywhere that a user might be. In most cases, they are configured to offer only horizontal image isolation. Eye-tracked autostereoscopic systems may be used to reduce the number of projectors required, but commercial systems are limited in size and viewing angle [19].

Pers Ubiquit Comput

Head-mounted displays (HMDs) can provide perspective-correct mixed reality imagery for multiple users, using either a video or optical overlay upon a real view of the world. Video overlays mix synthetic imagery with a live camera view of the world using standard opaque head-mounted displays. Video overlays exhibit some artifacts, such as video frame lag (typically 1/30th s) and the downsampling of the world to video resolution. Optical overlays use translucent displays, allowing the real world to be seen through the display. Unfortunately, this often causes the virtual imagery to be translucent. Optical overlays can also make tracker lag and noise more apparent as the virtual imagery is compared to the real world. With either type of overlay, HMDs add bulky optical elements in front of the user’s eyes. These elements make it difficult for trainees to see each others’ eyes and facial expressions. They can also interfere with sighting down a weapon. Head-mounted projective displays (HMPDs) can also be used for individualized virtual and augmented reality imagery. The previous generation of HMPDs differs from REFLCT in several ways. Chief among these is the use of projectors that project onto an optical combiner, a semitransparent mirror surface, in front of a user’s eyes to create a projection path aligned with a user’s optical path [4–7, 18]. The partially reflective surface in front of the eyes can interfere with eye contact and head movements such as sighting down a rifle. Our approach with REFLCT can be compared to the wearable projection system described by Karitsuka and Sato [8]. However, their display is mounted to the user’s back, which means the image is projected over the shoulder to surfaces directly in front of the user’s body, preventing virtual imagery from being viewed with side-to-side head turns. By employing more compact components and a more optimal optical configuration, we are able to mount the projector directly on the head, thus maintaining a small, fixed distance between the projector and the user’s eyes. We have also made a number of new modifications to projector and retroreflective screen configurations. Project Tuatara [12] is another related system, consisting of a laser-based projector mounted to a gyroscopically tracked gun-shaped game controller. However, this system is designed for game play and does not use retroreflective screens, preventing multiple users from training together due to overlapping projector images. Interactive Dirt is another related projection system that is designed for soldiers and employs shoulder-mounted projectors [11]. Unlike REFLCT, Interactive Dirt does not employ retroreflective screens which limits use with multiple projectors and relies on shoulder-mounted projection, since it does not employ tracking to stabilize the projector.

3 Apparatus and method 3.1 Apparatus: a prototype test environment REFLCT is a head-mounted projector framework that takes advantage of the imperfect performance characteristics of retroreflective materials. While the majority of energy is reflected exactly on axis, there is a fair amount of light bleed at slightly off-axis angles. We leverage this ‘‘defect’’ to enable the user’s eyes to be slightly displaced from the light path of the projector. Additionally, the retroreflective properties of the material are still strong enough so that two users standing next to each other will not be able to view each other’s projected image. Thus, by mounting a picoprojector near a user’s eyes, this reflected energy will be seen by that user, and nobody else. This approach is not feasible with older generation, lower brightness projectors, which need to be on the same optical axis as the user’s eyes, facilitated by an optical combiner in front of the user’s eyes. However, newer pico-projectors offer enough brightness such that this is not an issue. In the current REFLCT prototype, each user wears a construction helmet fitted with a REFLCT projection unit (see Fig. 1). Each unit is fashioned out of a high-density fiberboard ring. The ring supports a number of active LED markers for motion capture and a DLP-based picoprojector. We use DLP Pico-Projector Development Kit Version 1 projectors from Texas Instruments, which use a Texas Instruments DLP engine and an optical design by Young Optics. The pico-projector is vertically mounted and projects down upon a small mirror oriented at 45", which reflects the light forward. This places the optical axis of the projection closer to the user’s eyes. A PhaseSpace Impulse motion capture system determines the position and orientation of the REFLCT projection unit and distributes this information, via VRPN [20], to a PC. The PC then renders the proper perspective view using the Panda3D graphics library. The projector is connected by a DVI cable to the PC. Due to the DVI cable, this prototype is somewhat limited in terms of user movement, but we will describe a mobile configuration later, in Sect. 5. Retroreflective screen material is placed wherever virtual elements are to be displayed (see Fig. 2). A number of props, such as simulated cinderblock walls, sandbags, and camouflage netting, creates a military-themed stage and also blends the screens into the environment. Other props would be used for alternative training settings. Retroreflective coatings can also be added to props, in the form of retroreflective cloth, retroreflective tape, or even a coating of fine retroreflective glass beads, allowing an image to be applied to a wall or even a sculpted human form. We have applied this retroreflective coating to existing surfaces such as wood, creating unique retroreflective props that maintain

123

Pers Ubiquit Comput

Fig. 2 An over the shoulder view of an animated virtual character, projected on a retroreflective surface by a REFLCT helmet

their original appearance but can be enhanced by augmented reality projection. In addition to the projector, a USB video camera is mounted to the helmet to capture everything, both real and virtual, that the user can see. During demonstrations, this can be used to share each user’s viewpoint with an audience. Since the retroreflective screen materials reflect imagery only to the wearer of the projector, the audience cannot see the projected imagery directly. These cameras could also record training sessions and help with after action reviews of training performance. We expect to use helmet-mounted cameras for vision-based tracking in the near future. The software development environment is based on the Python scripting language and the Panda3D graphics library. The virtual human character is modeled, rigged, and animated in Autodesk Maya and exported for Panda3D. The character’s voice and visemes (mouth movements) are derived from a Python library that interfaces with the Microsoft Speech API. Control and synchronization between multiple PCs were performed using a Python-based Open Sound Control networking library. This particular software toolset, centered on Python language development, was chosen to facilitate programming and content generation by a wide range of individuals, from non-programmers to advanced computer scientists. Our laboratory has members and associates which include cinema school students, artists, mechanical engineers, and computer scientists, so it is important to provide a development environment accessible to the ideas of many potential developers. 3.2 Method: a prototype user experience To showcase the capabilities of a basic REFLCT system, we have created a demonstration that allows two participants to share an interaction with a virtual human. In this demonstration, two participants enter our mixed reality stage and don helmets containing REFLCT projection

123

units. A virtual US Army sergeant appears in front of them (see Fig. 2), on a propped and staged set, who introduces the display technology. To illustrate how the display facilitates virtual human interactions, he asks them to indicate to whom he is pointing, as he points to each participant in turn. He then makes eye contact with each participant, again asking each participant to respond in turn. The sergeant then mentions that he needs one of the participants to perform a task. He looks back and forth between the two of them, as if to decide which one to chose. He finally points at one of the participants and says ‘‘How about you…I need you to check out the pit over there,’’ while gesturing towards a pile of concrete and sand bags surrounding a pit. This pit has a retroreflective screen across the bottom. When the participant walks to the pit, he notices an image of artillery shells at the bottom of the pit, which can be used in improvised explosive devices. To demonstrate how perspective can be used in training, the sergeant then directs the participant to turn around and look at a screen representing a hallway entrance. The participant is not able to see anything besides an empty hallway due to his vantage point. However, the second participant can see straight down the hallway, observing a man standing there. The first participant can then sidestep towards the second participant to also reveal the man. To reinforce the perspective capabilities of the display, an additional screen, located high up on a wall, simulates a second story window. A virtual character in the window is visible from some angles and hidden from others.

4 Results REFLCT provides users with a personal, perspectivecorrect view of virtual elements that can be used to present social interactions with virtual humans and demonstrate the importance of movement and establishing new sight lines in urban terrain. In our demonstrations, all of the participants are able to identify the intended recipient of eye contact and pointing gestures, regardless of whether they are the recipient or a bystander. Additionally, qualitative anecdotal responses from experts in military training have been very positive. They comment that REFLCT is liberating in comparison to other head-mounted displays, since it allows the user to look out into an environment without glass or other elements in front of the face. Furthermore, they note the potential for mounting REFCLT to the soldiers’ existing gear, which is something that is not possible with the displays typically used in immersive training. REFLCT’s virtual imagery offers realistic depth cues as images are projected directly onto props, and the user’s eyes can focus at the appropriate distance, instead of a

Pers Ubiquit Comput Fig. 3 The retroreflected illumination level near the optical axis of the pico-projector (left) is much less than the retroreflected illumination level at 16 inches from the optical axis (right). This screen was placed behind a concrete wall prop

Fig. 4 A test of retroreflective sign material and a projector/mobile phone during daylight. The projection is still very discernable to the user

fixed distance determined by the HMD optics. Additionally, real objects can come between the user and the virtual image. This allows natural occlusion of the virtual image, instead of computed occlusion, where all potential occluders must be tracked and masked out during virtual image generation. REFLCT’s technique is beneficial for proper handling of occluding objects such as a user’s hands and weapon, which are rapidly moving and will often occlude images of hostile virtual opponents. The typical distance between users provides more than enough illumination fall-off to enable individual views. A 16 inch translation between axes of projection is enough to ensure that two users, each with a personal REFLCT display and standing shoulder to shoulder, will experience unique images (see Fig. 3). Outdoor tests also indicated that a system built with off-the-shelf pico-projectors could be used on an overcast day (see Fig. 4). We have found that projector focus, or more specifically, projector depth of field is a design constraint that becomes less of an issue with brighter projectors or laser-

based projectors. A brighter projector can use a smaller lens which is in focus over a large range of distances. A laser-based projector effectively has no lens, since the beam scans back and forth to paint an image, so focal distance is not an issue. In our tests, we have seen laserbased projectors exhibit the same type of retroreflection and off-axis extinction behaviors as standard projectors. REFLCT’s current pico-projectors have a field of view of about 30" horizontal. When users turn their heads, there is a noticeable limit, or window, of visibility for virtual objects. As pico-projectors become commoditized, shorter throw, wider field of view projectors should become available. The field of view can also be addressed by using a curved mirror, a different lens configuration, or a small array of blended projectors. Due to tracker noise, we are continuing to experiment with both the PhaseSpace Impulse active optical marker tracking and camera-based fiducial tracking. Since the projector is head mounted, and the projection is at a distance, there is ‘‘lever arm’’ that magnifies any noise or lag in tracker orientation. Virtual images can bob slightly out of sequence with head motion. By minimizing the end-toend lag of tracking, image generation, and image projection, we can reduce the lag-induced bob. Tuning of the motion capture system, such as careful distribution of markers over the helmet, can also improve tracking noise, reducing the noise component of the bob. Camera-based fiducial tracking (or other ‘‘inside-out’’ forms of tracking) should provide additional stability and precision since the ‘‘lever arm’’ is accommodated in the tracking system geometry.

5 Discussion Building on our experience with the example training application and the prototype testing platform, we have

123

Pers Ubiquit Comput

continued to experiment with variations on the REFLCT framework. In this section, we discuss some alternative REFLECT configurations and applications that we have conceptualized, prototyped, or built. Particularly significant configurations include our construction of a 3D REFLCT unit, utilizing the first 3D stereoscopic pico-projector of which we are aware, and our experiments with various retroreflective screen forms. 5.1 Alternative projector configurations REFLCT configurations involving multiple projectors are being tried with the goals of stereoscopic imagery and wide field of view imagery (see Fig. 5). With regard to stereo, intraocular distance (the measurement between a single user’s eyes) is too small to allow effective isolation of left and right 3D stereoscopic views given the imperfect offaxis performance of retroreflective beads and cloth. An additional form of extinction, such as polarization or time sequential optics is required to limit bleed for stereoscopy. Anaglyph stereo, such as red/blue color coding, can create the extinction necessary, but imprecision in projector placement has a strong effect on relative brightness of the left and right eye images, limiting effectiveness. When projectors become small enough to consider head-mounted projector arrays, it may be possible to create a system that works by bonding higher performance retroreflective materials with anisotropic diffusers. Initial experiments indicated that this more closely focuses the returning energy between the eyes and allows for head roll. Following the approach of time-multiplexed stereo, we have built and demonstrated a 3D stereoscopic version of REFLCT, using a modified pico-projector. The modifications use the sequential RGB color cycle of the DLP projector to create a time-multiplexed stereoscopic image. The pico-projector presents color using three LEDs (red, green, and blue) which light up in sequence and illuminate a set of dichromatic mirrors and then a DLP micro-mirror array,

Fig. 5 Experimental projector configurations for a wider field of view (left) or for 3D stereoscopic imagery (right)

123

creating red, green, and blue images. The green image has twice the exposure duration of the red and blue images, likely due to brightness and perceptual requirements. We attached a pair of shutter glasses which are triggered by signals to the LEDs and rewired the LEDs to always be on. This removes color from the projected imagery, but allows separate left and right eye channels. One eye shutter is closed for green signal, while the other shutter is closed for both red and blue signals. We use Panda3D to render stereo imagery with a magenta and green anaglyph encoding. By wearing this modified projector and looking through the shutter glasses, users can perceive a grayscale stereoscopic image. While this version of REFLCT places shutter glasses in front of the user’s eyes, this trade off may be acceptable for some applications. To our knowledge, this is the first 3D stereoscopic pico-projector system to be built and demonstrated. To create a product that supports practical use in training systems, we have developed and demonstrated a version of REFLCT that runs on a smartphone with a batterypowered pico-projector. The imagery is rendered by the Unity game engine, using tracking data received over the smartphone’s WiFi connection. This compact module can clip onto a soldier’s helmet using a standard night vision goggle mount (see Fig. 6). By performing all rendering on the smartphone, we are able to completely remove the encumbrance introduced by cables, allowing for true freedom of movement. After training, the modules can be detached, recharged, and reprogrammed for the next set of trainees. Future versions could also employ the

Fig. 6 A prototype REFLCT module, powered by a smartphone, is compact enough to attach to a night vision goggle mount for military training

Pers Ubiquit Comput Fig. 7 Retroreflective material can be applied to a mannequin head prop and illuminated with an virtual character face

smartphone’s sensors: camera, gyroscopes, compass, and accelerometers, to aid in tracking. Additionally, smartphones are now being produced with integrated picoprojectors, which would further reduce the footprint of the device. 5.2 Alternative screen forms The success of standard retroreflective surfaces with REFLCT has encouraged us to explore additional ways to integrate retroreflective materials into themed sets designed for mixed reality training. Toward this end, a number of innovative retroreflective approaches are being developed, including partially transparent and spatially curved surfaces. Complex-shaped retroreflective props can be created by coating a figure with retroreflective cloth, or by depositing glass beads onto the surface of the prop. This enables a very realistic spatial presentation of imagery as in Shader Lamps [17]. Virtual humans could be presented thusly, using sculpted human forms and applying a projection of animated facial expressions (see Fig. 7). In order to present virtual images in the middle of a room, we are mixing clear sealants with glass beads and bonding this solution to glass surfaces. We are also experimenting with a series of different density laser-cut patterns in retroreflective cloth (see Fig. 8). When no image is projected, such surfaces essentially fade away to invisibility since users can see props and live actors through them. While the perforated screens allow partial transparency of the virtual imagery, they cannot fully occlude objects that may pass behind the virtual image. To create virtual images that can appear anywhere in the room, we could install a number of these partially

transparent screens to quickly deploy or retract from the ceiling. However, we are also experimenting with capturing and carrying retroreflective materials in a laminar fluid flow, such as water or air as in Olwal et al. [15]. Our trials have shown the high density of retroreflective glass beads prevents easy transport in water. Furthermore, the indices of refraction for glass and water are too similar, dispelling the retroreflective effect when mixed together. On the other hand, a controlled cascade of released glass beads can create a retroreflective cloud or stream that could be made to appear anywhere within a physical environment (see Fig. 9). There is also a need in mixed reality applications for extremely robust projection surfaces. For example, some military training systems involve paint pellets rounds fired

Fig. 8 An example of a perforated retroreflective test surface allowing ‘‘compositing’’ of virtual imagery and live actors

123

Pers Ubiquit Comput

Fig. 9 A cascade of retroreflective beads illuminated by a projector. Such a cascade could be used to create virtual elements anywhere within a space

from standard firearms. We have found that commercially available road sign material is retroreflective and has a steel backing. We sent sign material for testing under repeated firings at close range of these paint pellet rounds (see Fig. 10) at the Infantry Immersion Trainer (IIT) [14]. The material appears to clean well and perform again after a simple wipe down, although the material does present a chromatic rainbow effect with off-axis reflection. 5.3 Alternative interfaces and applications Retroreflective surfaces open up new user interface capabilities and possible applications since they allow the preservation of perspective, deictic gesturing, and eye contact across a group of concurrent users. These and other properties make them applicable to a wide range of training. For example, since retroreflective material can be made robust to paint rounds from firearms, REFLCT can be used in squad-level simulators featuring shoot/no shoot decisions and live paint rounds. We also believe that REFLCT can be used in a number of additional training Fig. 10 Hardened retroreflective screens can withstand paint pellet simulated munitions (left) and then be cleaned (right). Note arrows pointing to the corner of the ‘‘L’’

123

scenarios that teach ‘‘soft skills,’’ such as working with language translators or cross-cultural negotiation. Individuals working with translators must often ‘‘read’’ the body language of both the translated party and the translator for reactions and undertones that are often lost in translation. These may be subtle gestures and eye contacts easily obscured by standard projected displays. Similar body language observational skills are also required in negotiation training. Retroreflective screens can be thought of as extremely high-gain projection screens that drastically increase the efficiency of pico-projectors for individual users. This enables the projectors to be used in products with smaller form factors, with less battery usage and longer life, and over longer distances, because the high gain of the retroreflective material concentrates energy return to the user. This could yield wearable computer configurations with pico-projectors mounted in eyeglasses, headphones, or Bluetooth headsets, and with display surfaces taking the form of retroreflective clothing like sleeves, shoes, wristbands (see Fig. 11), or even retroreflective handkerchiefs that are unfolded only when needed. Furthermore, the spatially targeted nature of the information presentation is well matched to other applications that require user-specific data. For example, cell phones with embedded projectors could be used as ‘‘cheek-based displays.’’ At an airport, a user would hold a cell phone projector to his/her cheek and look at a shared retroreflective surface to see personalized real-time directions, guidance arrows, and flight information, in the user’s preferred language (see Fig. 12). Face-to-face collaboration is another application for which REFLCT could be used. Devices, such as the Stanford Duo [1], enable a number of unique multi-user interactions, but are inherently limited by the need to multiplex imagery, typically in time, color, or polarization. Retroreflective materials, coupled with head-mounted

Pers Ubiquit Comput

Fig. 11 Low-power projectors mounted in glasses or headsets could illuminate retroreflective clothing for mobile/wearable computing

Fig. 13 A product design critique done over a video conference could be made easier with annotations projected on a retroreflective pad by a REFLCT unit

6 Conclusion

Fig. 12 A ‘‘cheek-based’’ cell phone display could present personalized information, like flight status, on a public retroreflective surface

projection, naturally segment imagery between each user and thus can be implemented to provide multi-person collaborative work surfaces. Note that because only one projector is needed per user, such interfaces are not limited to a single workbench-like surface. Retroreflective cloth could be inexpensively applied to multiple desktop surfaces as well as walls. Remote collaboration could also be supported between a remote user with a camera and projector-equipped cell phone and second user, perhaps equipped with similar cell phone, or a more traditional phone or computer. The remote user could show an object, such as a prototype model for a new product, over video for critique by the second user. The remote user could place the prototype on a retroreflective pad, allowing the second user to make annotations that become visible to the remote user through projection. When the remote user moves the cell phone, visual tracking of the object and retroreflective pad would help maintain a correct perspective projection of the annotations, keeping them correctly aligned and localized to the object (see Fig. 13).

While head-mounted projectors have existed for some time, the latest generation of bright and compact pico-projectors has made practical a new set of mixed reality displays, represented by REFLCT. With its capabilities for virtual human interaction and personalized perspective views, REFLCT reinvigorates team-based mixed reality training. REFLCT is an adaptable display framework, allowing for a number of variations and improvements, including 3D stereoscopy, complex retroreflective surfaces, camerabased tracking, and new form factors. We expect that REFLCT’s capabilities will grow and change over time. We hope to use REFLCT to explore a number of new domains beyond mixed reality training, such as collaboration, visualization, and mobile information access. Acknowledgments The authors wish to thank John Hart for guidance with this project, as well as Thai Phan, Brad Newman, and David Nelson for numerous contributions. This work was funded by the US Army Research, Development, and Engineering Command (RDECOM) via an Institute for Creative Technologies Seedling grant. The content of the information does not necessarily reflect the position or the policy of the US Government, and no official endorsement should be inferred.

References 1. Agrawala M, Beers AC, McDowall I, Fro¨hlich B, Bolas M, Hanrahan P (1997) The two-user responsive workbench: support for collaboration through individual views of a shared space. ACM SIGGRAPH 327–332. doi:10.1145/258734.258875 2. Baker H, Li Z. (2009) Camera and projector arrays for immersive 3D video. In: IMMERSCOM’09: International conference on immersive telecommunications, pp 1–6 3. Bolas M, Krum DM (2010) Augmented reality applications and user interfaces using head-coupled near-axis personal projectors with novel retroreflective props and surfaces. Pervasive 2010 Ubiprojection Workshop

123

Pers Ubiquit Comput 4. Fergason JL (1997) Optical system for a head mounted display using a retro-reflector and method of displaying an image, US patent 5621572 5. Hua H, Gao C, Brown L, Biocca F, Rolland JP (2002) Design of an ultra-light head-mounted projective display (HMPD) and its applications in augmented collaborative environments. In Proceedings of SPIE 4660:492–497 6. Hua H, Gao C (2005) A polarized head-mounted projective display. IEEE International symposium on mixed and augmented reality (ISMAR), pp 32–35 7. Inami M, Kawakami N, Sekiguchi D, Yanagida Y, Maeda T, Tachi S (2000) Visuo-haptic display using head-mounted projector, IEEE virtual reality, pp 233–240 8. Karitsuka T, Sato K (2003) A wearable mixed reality with an onboard projector, IEEE International symposium on mixed and augmented reality (ISMAR), pp 321–322 9. Koepnick S, Hoang R, Sgambati M, Coming D, Suma E, Sherman W (2010) RIST: radiological immersive survey training for two simultaneous users, computers & graphics (in press) 10. Matusik W, Pfister H (2004) 3D TV: a scalable system for realtime acquisition, transmission, and autostereoscopic display of dynamic scenes, ACM SIGGRAPH, pp 814–824 11. McFarlane DC, Wilder SM (2009) Interactive dirt: increasing mobile work performance with a wearable projector-camera system. ACM International conference on ubiquitous computing (Ubicomp’09), pp 205–214 12. Microvision, Inc. (2009) Microvision’s PicoP display engine at heart of realistic game demo at Intel extreme masters tournament,

123

13.

14.

15.

16.

17.

18.

19.

20.

Microvision, Inc. Press release, http://phx.corporate-ir.net/ phoenix.zhtml?c=114723&p=irol-newsArticle&ID=1364520& highlight= Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. Institute of electronics, information and communication engineers (IEICE). Trans inf Syst E77-D(12):1321–1329 Muller P, Schmorrow D, Buscemi T (2008) The infantry immersion trainer: today’s holodeck, marine corps gazette, pp 14–18 Olwal A, DiVerdi S, Candussi N, Rakkolainen I, Hollerer T (2006) An immaterial, dual-sided display system with 3D interaction, IEEE virtual reality, pp 279–280 Pair J, Neumann U, Piepol D, Swartout B (2003) FlatWorld: combining Hollywood set de-sign techniques with VR. IEEE Comput Gr Appl 23(1):12–15 Raskar R, Welch G, Low K, Bandyopadhyay D (2001) Shader lamps: animating real objects with image-based illumination. Eurographics rendering workshop Rolland JP, Biocca F, Hamza-Lup F, Ha Y, Martins R (2005) Development of head-mounted projection displays for distributed, collaborative, augmented reality applications. Presence: Teleoper. Virtual Environ 14(5):528–549 Stolle H, Olaya J-C, Buschbeck S, Sahm H, Schwerdtner A (2008) Technical solutions for a full-resolution auto-stereoscopic 2D/3D display technology. In: Proceedings of SPIE, p 6803 Taylor II RM, Hudson TC, Seeger A, Weber H, Juliano J, Helser AT (2001) VRPN: a device-independent, network-transparent VR peripheral system. ACM VRST, pp 55–61