4 Ideas That Will Change Augmented Reality

I’m often asked where Augmented Reality is heading and what AR will look like in the next 5-10 years. Let’s take a peek ahead. (This article is part of a larger project and book  that maps AR’s past and future as a new medium.)

If this is your first time visiting Augmented Stories, please allow me to introduce myself: I’m a designer, PhD researcher, and consultant specializing in AR since 2005. In addition to working hands on with AR to conceptualize, prototype and design compelling experiences, I consult in commercial industry, advising and leading clients as their ‘AR Sherpa’ to chart new territory in this emerging field. I travel internationally to give public talks on AR (including two TEDx talks), and I’m also an active member in the AR research and academic community.

AR is advancing rapidly as a new medium, moving beyond just a technology. One of the things I’ve dedicated myself to in this field is exploring what AR does best and the opportunities that exist in AR as a medium for storytelling and creating engaging experiences.

Each medium has unique characteristics to be harnessed, capabilities that extend beyond other mediums. We’re presently in this exciting phase of AR, of identifying, as well as being able to technically influence, what these capacities and criteria are and can become. Just like film, television, radio and photography when first novel, AR presents brand new terrain ripe for creative exploration. With this comes the possibility of forming new conventions and developing a stylistic language of AR.

Part of the task is to determine the distinct characteristics that define AR, with a focus on what differentiates AR from previous forms. (Debates on medium specificity criticize how essentialist traits can be limiting, in turn, prescribing what the aesthetics of these media should look like; however, I’m not advocating that we lock ourselves into these properties, rather, that our exploration begins here and we continue to evolve, bend, extend, and query those qualities in new ways.)

I often liken AR to cinema when it was first new, and I believe film is the closest relative we can align AR to currently, particularly in investigating AR as a new medium and identifying stylistic motifs.

The Vimeo user Kogonada recently posted a wonderful series of videos surveying filmmaking tropes. A montage of stylistic motifs from the oeuvre of various directors are cleverly curated, including one-point perspective from Stanley Kubrick, Quentin Tarantino’s framing from below, shots from above by Wes Anderson, Breaking Bad’s object POV aesthetic, and the sounds of Darren Aronofsky.

So what does this have to do with AR?

These directors introduced and perfected their own motifs within the medium of film. And, so, I believe we will see the same emerge in AR. AR will be mastered by creative geniuses, introducing us to new stylistic modes within the medium. New tropes (visual, aural, and sensorial) specific to AR will emerge that would not have been possible in other media formats.

The larger project and book I am currently working on is inspired by the “100 Ideas that Changed…” book series which looks at a range of media forms including film, graphic design, photography, and architecture, and the design ideas and critical concepts that shaped these mediums as we know them today.  Looking ahead and writing from the future to the past, what are the ‘100 Ideas That Changed AR’? Here are 4 to tease things out a little and get things started:

  1. HAPTICS & TOUCH

Image: AR Haptics demo utilizing a Phantom Stylus & HMD from the Magic Vision Lab, University of South Australia.

To date, AR has focused primarily on the visual, leaving the other senses behind. Incorporating a sense of touch and tactile feedback through haptics in AR can help to greatly heighten the sense of the ‘real’ when interacting with virtual content to enrich the overall experience. Imagine being able to feel the scales of a fish in a virtual model as though that fish were really there before your eyes. (I detail my first hand experience of haptics, the possibilities for integration with tablets and the opportunities for storytelling here.) Haptics in AR enables the user to feel a multitude of textures and surfaces that are unique to the context and can also be altered to the individual or scenario. The other example I give is in relation to my AR Pop-up Book, “Who’s Afraid of Bugs?” in which you can hold virtual creepy crawlies in your hand; imagine being able to feel that virtual spider crawling over your hand and the intensity and greater sense of reality this would add to heighten the experience.

Recent work in this arena includes REVEL, from Disney Researchers Ivan Poupyrev and Olivier Bau, which applies reverse electrovibration to create the illusion of changing textures as a user sweeps their fingers across a surface. REVEL can provide tactile feedback both on a touchscreen, or, quite wonderfully, on the physical object itself (without the use of a virtual glove or stylus). As the video above details, this haptic technique can even be applied to projections. Now, imagine, what would it be like to feel scenes in a projected film? What new tropes and stylistic motifs might emerge in such a genre? Will there be an AR director of our times who will become known for their unique sensibilities and style of touch in haptics? (This is a project I’m presently exploring, so do get in touch if you’re keen to collaborate & support this creative investigation).

  1. BREAKING THE SCREEN/FRAME

AR presents the opportunity to interact with a whole new dimensional space that traces a history of illusion from trompe l’oeil to the magic lantern to stereoscopic 3D films. Early cinemagoers were said to have been astonished by the first films, witnessing trains that seemingly charged off the screen and directly at the audience. That couldn’t be a better way to describe some of the visual special effects AR can make possible today; however, we’re now able to step inside that scene and look at that train from a multitude of perspectives. As seen with the SONY video below, AR presents a unique opportunity to engage with 3D interactive photo-realistic sculptural photography, scenes that we are able to enter, explore, and even alter.

I begin to address the potential aesthetic sensibilities that breaking the screen/frame in AR presents here. This has tremendous potential in altering the way we tell stories, with the narrative of the augmented environment spilling into our physical surroundings. Stylistic motifs will emerge that explore storytelling beyond the limits of the single frame, expanding in multiple directions, puncturing conventional space and no longer confined within a distinct screen. The AR participant and audience will be at the center of this, the space navigating around the user; the user’s perception, context and active engagement will dictate and define the illusion.

  1. POINT OF VIEW (POV)

One of the things I believe is distinct to AR is its capacity to be used as a powerful medium to translate perspectives, to see the world through another person’s eyes. (If we think back to the Kogonada video montages on Vimeo and the Breaking Bad sequence of object POVs, this too is possible, to witness an experience from the perspective of an object; see 3b below.) Human, or non, the possibilities exist for new stylistic motifs to emerge based on this principle.

POV in AR can range from narrative experiences and shifting between character viewpoints, to teaching tools and manuals, with a diversity of experiences from entertainment to utility, fiction to non-fiction.

Steve Mann, too often unrecognized in AR, contributed critical work to the field in the early 90’s with his wearable computer devices and concepts of Mediated Reality. Mann described Mediated Reality and his WearCam eyewear as enabling a finely customized visual interpretation of reality unique to the needs of each individual wearer. These highly personalized experiences could allow for a different interpretation of the same visual reality.

Image: Mediated Reality pioneer and University of Toronto professor Steve Mann.

When I encountered Mann’s work in the 90’s, I often thought: perhaps in the future you will want to filter your reality through someone else’s perspective. Imagine experiencing the world via the reality filter of a friend in a social network, a celebrity, someone you admire or idolize, or perhaps even a complete stranger. By enabling someone to see the world through another’s eyes, POV in AR can also be applied as a tool to build empathy, (see my previous article on this topic here), which could serve various benefits from conflict-resolution to design thinking (empathy is a trait of a good designer).

3b. SEEING WITH YOUR HANDS & OTHER LIMBS

Branching from #3, this is another form of POV that can assist in navigating and conveying information about a specific environment or location. Concepts like the EyeRing by MIT researchers and Google’s patent for ‘Seeing With Your Hands’ present other means of interacting with AR and computer vision aside from the human eye.

EyeRing can aid the visually impaired to have a form of sight. A camera, worn on the finger as a ring, identifies and recognizes images, colors, and text, transmitting data to a smartphone where an application transforms the collected information into a digital voice. In addition to assistive technology for the blind, the researchers suggest a ‘tourist helper’ application where the name of a landmark is heard once pointed at with the device.

Image: Google’s patent for “seeing with your hands”.

Google’s “seeing with your hands” patent describes a glove for gathering and conveying information which enables “a user to ‘see’ an inaccessible environment” by wearing a device on one’s hands (or “other areas of the body” including a foot, hip or back) equipped with “detectors”. The device may record a series of images and then display those back to the user. Although Google’s Project Glass (which I discuss here in a previous article) is not specifically referenced in the patent, a wearable display that is remote from the glove is mentioned. One can imagine the device being paired with Glass as a possible display. The patent repeatedly refers to predetermined motions, which could serve to combine gesture with AR. One of the motions detailed is a zoom function, which could enable highly magnified views of the user’s environment, that might previously have been deemed “inaccessible”. Another possible motif here emerges: of making the invisible visible, transforming the micro into the macro.

  1. THE MÉLIÈS EFFECT

It is no secret that filmmaker & magician Georges Méliès is one of my creative heroes. I often refer to Méliès in my talks, exhibits and articles, and named him the Patron Saint of AR in a special guest post for The Creator’s Project (A partnership with Intel and Vice) on what would have been Melies’ 150th birthday.

At a time when cinema was about documenting actualities, magician and pioneer filmmaker Georges Méliès extended the medium in incredibly novel ways to conjure impossible realities and define bold new conventions for film.

Méliès imagined fantastical worlds in his films, where the marvelous reigned over the mundane, inanimate objects became animate and forms forever shifted, disappeared and reappeared—nothing was fixed or impossible. Through the medium of film, Méliès created enchanted realities.

Méliès presented stories to embellish his latest effect; the narrative was driven by his toolkit, by the tricks he had at hand. Méliès’ stories evolved from his investigation and exploration of the novel technology of cinema and the characteristics of the medium. Méliès was a true explorer: He embraced accidents and glitches in the newfound medium of film; he had the capability to look at mistakes and see the creative potential in them, not discard them as bad or wrong. In fact, this is how the stop-trick, or double-exposure came to be: Méliès’ camera jammed while filming the streets of Paris; upon playing back the film, he observed an omnibus transforming into a hearse. Rather than discounting this as a technical failure, or ‘glitch’, he utilized it as a technique in his films.

A prolific maker, creating over 500 films, Méliès contributed critical technical advances to the medium (fades, dissolves and animations), yet, what is of equal great importance is how Méliès did not stop at these technical achievements alone; he dedicated himself to finding ways to use his technique to present compelling content. Perhaps the best way to summarize the ‘Méliès Effect’ in AR, as I’m labeling it, is in the words of John Andrew Berton Jr.: “Even though Méliès’ work was closely involved with the state of the art, he did not let that aspect of his work rule the overall piece. He used his technique to augment his artistic sense, not to create it” [1].

I interpret this as Méliès maintaining an artistry in the medium, being enthused by the possibilities of the technology and allowing the medium to guide his explorations and work, but not overrule it. Méliès serves as an inspiration to my creative practice and work in AR in that the technology inspired stories and the direction of the work, but he also gave himself the freedom of experimentation to move beyond the constraints and invent new ways of applying the technology. As such, Méliès introduced new formal styles, conventions, and techniques specific to the medium of film. This is the task at hand for AR pioneers and adventurers: to evolve novel styles and establish new conventions towards a language and aesthetics of AR.

This list is part of an ongoing work. What’s on your list of the ideas that will change AR? Let’s continue the conversation on Twitter, the boardroom, or at your next event. Get in touch; a wonderful world of AR awaits and it’s ours for the making.

[1]    Berton Jr., John Andrew. “Film Theory for the Digital World: Connecting the Masters to the New Digital Cinema.” Leonardo: Journal of the International Society for the Arts, Sciences and Technology. Digital Image/Digital Cinema: A special edition with the Special Interest Group in Graphics (SIGGRAPH) Association of Computing Machinery, New York: Pergamon Press, 1990, p.7.

UPDATE: Purchase “Augmented Human: How Technology is Shaping the New Reality” from:
Amazon (USA)
Amazon (Canada)
Indigo (Canada)
Barnes and Noble (USA)
Book Depository (Worldwide)

“Hallucinatory AR” or ‘the chance encounter of an umbrella and a sewing machine upon an operating table’

“Hallucinatory Augmented Reality (AR)”, 2007, was an experiment which investigated the possibility of non-markers to generate AR imagery.

These projects evolved out of accidents, incidents in earlier experiments in which  the AR
software was mistaking non-marker imagery for AR markers and attempted to generate AR imagery. This confusion, by the software, resulted in unexpected and random flickering AR imagery. I decided to explore the creative and artistic possibilities of this  effect further and conduct experiments with non-traditional marker-based tracking. The process entailed a study of what types of non-marker images might generate such ‘hallucinations’ and a search for imagery which would evoke or call upon multiple AR imagery/videos from a single image/non-marker.

halluc-AR-originalimgLeft: This was the final image used (non-marker) to evoke “Hallucinatory AR”.  Source: Unknown (I am searching through my original notes and files from 2007 to locate a source. If this is your image, kindly contact me.)

Upon multiple image searches, one image emerged which proved to be quite extraordinary. This cathedral stained glass window was able to evoke four different AR videos, the only instance, from among many other images, in which multiple AR imagery appeared. Upon close examination of the image, focusing in and out with the web camera, a face began to emerge in the black and white pattern (See video documentation). A fantastical image of a man was encountered. Interestingly, it was when the image was blurred into this face using the web camera that the AR hallucinatory imagery worked best, rapidly multiplying and appearing more prominently. Although numerous attempts were made with similar images, no other such instances occurred; this image appeared to be unique.

halluc-ar-all

The challenge now rested in the choice of what types of imagery to curate into this hallucinatory viewing: what imagery would be best suited to this phantasmagoric and dream-like form?

My criteria for imagery/videos were like-form and shape, in an attempt to create a collage-like set of visuals. As the sequence or duration of the imagery in Hallucinatory AR could not be predetermined, the goal was to identify imagery that possessed similarities, through which the possibility for visual synchronicities existed.

Themes of intrusions and chance encounters are at play in Hallucinatory AR, inspired in part by Surrealist artist Max Ernst. In “What is the Mechanism of Collage?” (1936), Ernst writes:

One rainy day in 1919, finding myself on a village on the Rhine, I was struck by the obsession which held under my gaze the pages of an illustrated catalogue showing objects designed for anthropologic, microscopic, psychologic, mineralogic, and paleontologic demonstration. There I found brought together elements of figuration so remote that the sheer absurdity of that collection provoked a sudden intensification of the visionary faculties in me and brought forth an llusive succession of contradictory images, double, triple, and multiple images, piling up on each other with the persistence and rapidity which are particular to love memories and visions of half-sleep. (427)

Of particular interest to my work in exploring and experimenting with Hallucinatory AR was Ernst’s description of an “illusive succession of contradictory images” that were “brought forth” (as though independent of the artist), rapidly multiplying and “piling up” in a state of “half-sleep”. Similarities can be drawn to the process of the seemingly disparate AR images jarringly coming in and out of view,  layered atop one another.

One wonders if these visual accidents are what the future of AR might hold: of unwelcome glitches in software systems as Bruce Sterling describes; or perhaps we might come to delight in the visual poetry of these augmented hallucinations that are “As lovely as the chance encounter of an umbrella and a sewing machine upon an operating table.” *

*Comte de Lautreamont’s often quoted allegory, famous for inspiring both Max Ernst and Andrew Breton, qtd. in Williams, p.197.

Contact Helen Papagiannis for further information and to view other demos.

Max, Ernst. “What is the Mechanism of Collage?” Theories of Modern Art. Ed. Herschel B.
Chipp. Berkeley: University of California Press, 1974: 427.

Williams, Robert. “Art Theory: An Historical Introduction.” Malden, MA: Blackwell Publishing, 2004: 197.