Designing the Future of Augmented Reality: slides from PechaKucha

PechaKucha3

Sometimes you just need to relinquish control.

20 slides, 20 seconds each, and no control of the slide clicker. GO!

I had a fantastic experience at PechaKucha Toronto in December. My presentation, “Designing the Future of Augmented Reality” is now posted here.

More on the wildly fun International PechaKucha format here.

I have some exciting announcements on upcoming talks and my book to share very soon! Hope to see you at one of these events!

You can also find me on Twitter: I’m @ARstories.

*Update: Feb 24/15: Flattered to be featured as the Presentation of the Day on the PechaKucha website!

Screen Shot 2015-02-24 at 12.35.46 PM

How to Leave Your Laptop (at Starbucks) While You Pee: Invoked Computing

Experienced this dilemma? Mark Wilson (@ctrlzee), Senior Writer at Co.Design, tweeted yesterday, “If someone designs a solution to the leave your laptop with a stranger while you pee at starbucks problem, I promise to write about it.” Augmented Reality (AR) and Invoked Computing may just have the solution.

@ctrlzee tweet

A research group at the University of Tokyo has developed a concept for AR called Invoked Computing, which can turn everyday objects into communication devices. By making a gesture invoke the device you wish to use, you can activate any ordinary object to suit your communication needs. The computer figures out what you want to do and will grant the selected object the properties of the tool you wish to utilize. A proof of concept (see video) has been created for a pizza box which functions as a laptop computer, and a banana which serves as a telephone.

Invoked Computing presents a scenario where new functions are now layered atop ordinary objects, which do not normally possess those traits. Invoked Computing is the beginning of a new era of responsive environments that are on demand, context-dependent, and needs driven. Wired writer Bruce Sterling comments on how Invoked Computing affords the possibilities for sustainability and no material footprint because you can invoke and access everything.

In my recent talk at Augmented World Expo (AWE) 2014 in Silicon Valley, following Robert Scoble‘s keynote on “The Age Of Context”, I discussed how, as both a practitioner and a PhD researcher, I’ve watched AR evolve over the past 9 years. I suggested adding two new words to the AR lexicon: overlay and entryway to describe the two distinct waves in AR I’ve observed.

Overlay is exactly as it sounds, and defines the first wave of AR as we’ve grown to known it: an overlay of digital content atop the real-world in real-time. We are now entering the second wave of AR, entryway, where the definition of AR is expanding to include things like wearables, big data, artificial intelligence, machine-learning, and social media. This second wave represents a more immersive and interactive experience that is rooted in contextual design. Invoked Computing is a prime example as it combines the overlay properties we’ve seen in the first wave of AR with an on-demand experience that is personalized to the end-user.

So, go ahead and pee; that laptop will just shift back into a pizza box when you no longer need it.

Invoked Computing is one of The 40 Ideas That Will Change Reality (the title of my upcoming book).

Let’s continue the conversation. Find me on Twitter, I’m @ARstories.

Google Glass & Augmented Reality Eyewear: “Oh, the Places You’ll Go!” Defining a New Era in Visual Culture

Augmented Reality eyewear and Google’s Glass will take us to new heights, quite literally: the first sequence in the “How It Feels [through Glass]” video was shot via Glass in a hot air balloon.

nadar-glass-montage2

It was 155 years ago that the first aerial photograph was taken on a balloon flight in 1858 over Paris, France by artist Nadar (born Gaspar Félix Tournachon)(1820-1910). A pioneer in the newly emerging medium of photography, Nadar also attempted underground photography using artificial light to produce pictures of the catacombs and sewers of Paris. Nadar’s technical experiments and innovation took us to places via his camera that were previously inaccessible to photography, inspiring new ways of seeing and capturing our world.

AR eyewear and Glass offer this same opportunity at a time when AR is emerging as a new medium, which will give way to novel conventions, stylistic modes, and genres. Referencing Dr. Seuss’s book in the title of this article, AR also promises to transport us to wondrous, magical places we’ve yet to see.

This article is a follow up to a post I wrote a year ago posing the questions: Will Google’s Project Glass change the way we see & capture the world in Augmented Reality (AR) and what kind of new visual space will emerge?

As both a practitioner and PhD researcher specializing in AR for nearly a decade, my interests are in how AR will come to change the way we see, experience, and interact with our world, with a focus on the emergence of a new media language of AR and storytelling.

I’ve previously identified Point-of-View (PoV) as one of “The 4 Ideas That Will Change AR”, noting the possibilities for new stylistic motifs to emerge based on this principle. I’d like to revisit the significance of PoV in AR at this time, particularly with the release of Google Glass Explorer Edition. PoV, more specifically, “Point-of-Eye”, is a characteristic of AR eyewear that is beginning to impact and influence contemporary visual culture in the age of AR.

Google Glass

Image: Google Glass

AR eyewear like “Glass” (2013) and Steve Mann’s “Digital Eye Glass” (EyeTap) (1981) are worn in front of the human eye, serving as a camera to both record the viewer’s environment and superimpose computer-generated imagery atop the present environment. With the position of the camera, such devices present a direct ‘Point-of-Eye’ (PoE), as Mann calls it, providing the ability to see through someone else’s eyes.

AR eyewear like Glass remediates the traditional camera, aligning our eye once again with the viewfinder, enabling hands-free PoE photography and videography. Eye am the camera.

Contemporary mass-market digital photography has us forever looking at a screen as we document an event, rather than seeing or engaging with the actual event. As comedian Louis C.K. so facetiously points out, we are continually holding up a screen to our faces, blocking our vision of the actual event with our digital devices. “Everyone’s watching a shitty movie of something that’s happening 10 feet away” he says, while the ‘resolution on the actual thing is unbelievable’.

Glass presents an opportunity where your experience in that moment is documented as is without having to stop and grab your camera. Glass captures what you are seeing as you see it through PoE, very close to how you are seeing it. Google Co-Founder Sergey Brin states, “I think this can bring on a new style of photography that allows you to be more intimate with the world you are capturing, and doesn’t take you away from it.”

Google Glass video

Image: Recording video with Google Glass, “Record What You See. Hands Free.”

I agree with Brin; Glass will bring on new stylistic modes and conventions through PoE, which also appears to be influencing other mediums outside of AR.

Take for instance the viral Instagram series “Follow Me” by Murad Osmann featuring photographs of his hand being led by his girlfriend to some of the world’s most iconic landmarks.

FOLLOWME

Photographs by Murad Osmann, “Follow Me” series, 2013.

(Similar in style to the above video recording visual from Google Glass of a ballerina taking and leading the viewer’s hand.)

The article “How Will Google Glass Change Filmmaking?”, identifies two other examples in contemporary music videos: the viral first-person music video for Biting Elbows and the award-winning music video for Cinnamon Chasers’ song “Luv Deluxe”.

In “The Cinema as a Model for the Genealogy of Media” (2002), Andre Gaudreault and Phillipe Marion state, “The history of early cinema leads us, successively, from the appearance of a technological process, the apparatus, to the emergence of an initial culture, that of ‘animated pictures’, and finally to the constitution of an established media institution” (14). AR is currently in a transition period from a technological process to the emergence of an initial AR culture, one of ‘superimposed pictures’, with PoE as a characteristic of the AR apparatus that will impact stylistic modes, both inside and outside the medium, contributing to a larger Visual Culture.

Gaudreault and Marion identify key players in this process as: the inventors responsible for the medium’s appearance, camera operators for its emergence, and the first film directors for its constitution. ‘Camera operators’ around the world are beginning to contribute to AR’s emergence as a medium, and through this process, towards an articulation of a media language of AR. Mann, described as the father of wearable computing, has been a ‘camera operator’ since the 90’s. In 2013, Google Glass’s early adopter program selected 8000 ‘camera operators’ to explore these possibilities, with Kickstarter proposals since from directors for PoE film projects including both documentaries and dramas. What new stories will the AR apparatus enable? Like cinema before it, what novel genres, conventions, and tropes will emerge in this new medium towards its constitution?

Let’s continue the conversation on Twitter: I’m @ARstories.

4 Ideas That Will Change Augmented Reality

I’m often asked where Augmented Reality is heading and what AR will look like in the next 5-10 years. Let’s take a peek ahead. (This article is part of a larger project and book I’m working on that maps AR’s past and future as a new medium.)

If this is your first time visiting Augmented Stories, please allow me to introduce myself: I’m a designer, PhD researcher, and consultant specializing in AR since 2005. In addition to working hands on with AR to conceptualize, prototype and design compelling experiences, I consult in commercial industry, advising and leading clients as their ‘AR Sherpa’ to chart new territory in this emerging field. I travel internationally to give public talks on AR (including two TEDx talks), and I’m also an active member in the AR research and academic community.

AR is advancing rapidly as a new medium, moving beyond just a technology. One of the things I’ve dedicated myself to in this field is exploring what AR does best and the opportunities that exist in AR as a medium for storytelling and creating engaging experiences.

Each medium has unique characteristics to be harnessed, capabilities that extend beyond other mediums. We’re presently in this exciting phase of AR, of identifying, as well as being able to technically influence, what these capacities and criteria are and can become. Just like film, television, radio and photography when first novel, AR presents brand new terrain ripe for creative exploration. With this comes the possibility of forming new conventions and developing a stylistic language of AR.

Part of the task is to determine the distinct characteristics that define AR, with a focus on what differentiates AR from previous forms. (Debates on medium specificity criticize how essentialist traits can be limiting, in turn, prescribing what the aesthetics of these media should look like; however, I’m not advocating that we lock ourselves into these properties, rather, that our exploration begins here and we continue to evolve, bend, extend and query those qualities in new ways.)

I often liken AR to cinema when it was first new, and I believe film is the closest relative we can align AR to currently, particularly in investigating AR as a new medium and identifying stylistic motifs.

The Vimeo user Kogonada recently posted a wonderful series of videos surveying filmmaking tropes. A montage of stylistic motifs from the oeuvre of various directors are cleverly curated, including one-point perspective from Stanley Kubrick, Quentin Tarantino’s framing from below, shots from above by Wes Anderson, Breaking Bad’s object POV aesthetic, and the sounds of Darren Aronofsky.

So what does this have to do with AR?

These directors introduced and perfected their own motifs within the medium of film. And, so, I believe we will see the same emerge in AR. AR will be mastered by creative geniuses, introducing us to new stylistic modes within the medium. New tropes (visual, aural, and sensorial) specific to AR will emerge that would not have been possible in other media formats.

The larger project and book I am currently working on is inspired by the “100 Ideas that Changed…” book series which looks at a range of media forms including film, graphic design, photography, and architecture, and the design ideas and critical concepts that shaped these mediums as we know them today.  Looking ahead and writing from the future to the past, what are the ‘100 Ideas That Changed AR’? Here are 4 to tease things out a little and get things started:

  1. HAPTICS & TOUCH

Image: AR Haptics demo utilizing a Phantom Stylus & HMD from the Magic Vision Lab, University of South Australia.

To date, AR has focused primarily on the visual, leaving the other senses behind. Incorporating a sense of touch and tactile feedback through haptics in AR can help to greatly heighten the sense of the ‘real’ when interacting with virtual content to enrich the overall experience. Imagine being able to feel the scales of a fish in a virtual model as though that fish were really there before your eyes. (I detail my first hand experience of haptics, the possibilities for integration with tablets and the opportunities for storytelling here.) Haptics in AR enables the user to feel a multitude of textures and surfaces that are unique to the context and can also be altered to the individual or scenario. The other example I give is in relation to my AR Pop-up Book, “Who’s Afraid of Bugs?” in which you can hold virtual creepy crawlies in your hand; imagine being able to feel that virtual spider crawling over your hand and the intensity and greater sense of reality this would add to heighten the experience.

Recent work in this arena includes REVEL, from Disney Researchers Ivan Poupyrev and Olivier Bau, which applies reverse electrovibration to create the illusion of changing textures as a user sweeps their fingers across a surface. REVEL can provide tactile feedback both on a touchscreen, or, quite wonderfully, on the physical object itself (without the use of a virtual glove or stylus). As the video above details, this haptic technique can even be applied to projections. Now, imagine, what would it be like to feel scenes in a projected film? What new tropes and stylistic motifs might emerge in such a genre? Will there be an AR director of our times who will become known for their unique sensibilities and style of touch in haptics? (This is a project I’m presently exploring, so do get in touch if you’re keen to collaborate & support this creative investigation).

  1. BREAKING THE SCREEN/FRAME

AR presents the opportunity to interact with a whole new dimensional space that traces a history of illusion from trompe l’oeil to the magic lantern to stereoscopic 3D films. Early cinemagoers were said to have been astonished by the first films, witnessing trains that seemingly charged off the screen and directly at the audience. That couldn’t be a better way to describe some of the visual special effects AR can make possible today; however, we’re now able to step inside that scene and look at that train from a multitude of perspectives. As seen with the SONY video below, AR presents a unique opportunity to engage with 3D interactive photo-realistic sculptural photography, scenes that we are able to enter, explore, and even alter.

I begin to address the potential aesthetic sensibilities that breaking the screen/frame in AR presents here. This has tremendous potential in altering the way we tell stories, with the narrative of the augmented environment spilling into our physical surroundings. Stylistic motifs will emerge that explore storytelling beyond the limits of the single frame, expanding in multiple directions, puncturing conventional space and no longer confined within a distinct screen. The AR participant and audience will be at the center of this, the space navigating around the user; the user’s perception, context and active engagement will dictate and define the illusion.

  1. POINT OF VIEW (POV)

One of the things I believe is distinct to AR is its capacity to be used as a powerful medium to translate perspectives, to see the world through another person’s eyes. (If we think back to the Kogonada video montages on Vimeo and the Breaking Bad sequence of object POVs, this too is possible, to witness an experience from the perspective of an object; see 3b below.) Human, or non, the possibilities exist for new stylistic motifs to emerge based on this principle.

POV in AR can range from narrative experiences and shifting between character viewpoints, to teaching tools and manuals, with a diversity of experiences from entertainment to utility, fiction to non-fiction.

Steve Mann, too often unrecognized in AR, contributed critical work to the field in the early 90’s with his wearable computer devices and concepts of Mediated Reality. Mann described Mediated Reality and his WearCam eyewear as enabling a finely customized visual interpretation of reality unique to the needs of each individual wearer. These highly personalized experiences could allow for a different interpretation of the same visual reality.

Image: Mediated Reality pioneer and University of Toronto professor Steve Mann.

When I encountered Mann’s work in the 90’s, I often thought: perhaps in the future you will want to filter your reality through someone else’s perspective. Imagine experiencing the world via the reality filter of a friend in a social network, a celebrity, someone you admire or idolize, or perhaps even a complete stranger. By enabling someone to see the world through another’s eyes, POV in AR can also be applied as a tool to build empathy, (see my previous article on this topic here), which could serve various benefits from conflict-resolution to design thinking (empathy is a trait of a good designer).

3b. SEEING WITH YOUR HANDS & OTHER LIMBS

Branching from #3, this is another form of POV that can assist in navigating and conveying information about a specific environment or location. Concepts like the EyeRing by MIT researchers and Google’s patent for ‘Seeing With Your Hands’ present other means of interacting with AR and computer vision aside from the human eye.

EyeRing can aid the visually impaired to have a form of sight. A camera, worn on the finger as a ring, identifies and recognizes images, colors, and text, transmitting data to a smartphone where an application transforms the collected information into a digital voice. In addition to assistive technology for the blind, the researchers suggest a ‘tourist helper’ application where the name of a landmark is heard once pointed at with the device.

Image: Google’s patent for “seeing with your hands”.

Google’s “seeing with your hands” patent describes a glove for gathering and conveying information which enables “a user to ‘see’ an inaccessible environment” by wearing a device on one’s hands (or “other areas of the body” including a foot, hip or back) equipped with “detectors”. The device may record a series of images and then display those back to the user. Although Google’s Project Glass (which I discuss here in a previous article) is not specifically referenced in the patent, a wearable display that is remote from the glove is mentioned. One can imagine the device being paired with Glass as a possible display. The patent repeatedly refers to predetermined motions, which could serve to combine gesture with AR. One of the motions detailed is a zoom function, which could enable highly magnified views of the user’s environment, that might previously have been deemed “inaccessible”. Another possible motif here emerges: of making the invisible visible, transforming the micro into the macro.

  1. THE MÉLIÈS EFFECT

It is no secret that filmmaker & magician Georges Méliès is one of my creative heroes. I often refer to Méliès in my talks, exhibits and articles, and named him the Patron Saint of AR in a special guest post for The Creator’s Project (A partnership with Intel and Vice) on what would have been Melies’ 150th birthday.

At a time when cinema was about documenting actualities, magician and pioneer filmmaker Georges Méliès extended the medium in incredibly novel ways to conjure impossible realities and define bold new conventions for film.

Méliès imagined fantastical worlds in his films, where the marvelous reigned over the mundane, inanimate objects became animate and forms forever shifted, disappeared and reappeared—nothing was fixed or impossible. Through the medium of film, Méliès created enchanted realities.

Image: A still from Georges Méliès’ 1902 classic film “A Trip to the Moon” (Une Voyage dans la Lune).

Méliès presented stories to embellish his latest effect; the narrative was driven by his toolkit, by the tricks he had at hand. Méliès’ stories evolved from his investigation and exploration of the novel technology of cinema and the characteristics of the medium. Méliès was a true explorer: He embraced accidents and glitches in the newfound medium of film; he had the capability to look at mistakes and see the creative potential in them, not discard them as bad or wrong. In fact, this is how the stop-trick, or double-exposure came to be: Méliès’ camera jammed while filming the streets of Paris; upon playing back the film, he observed an omnibus transforming into a hearse. Rather than discounting this as a technical failure, or ‘glitch’, he utilized it as a technique in his films.

A prolific maker, creating over 500 films, Méliès contributed critical technical advances to the medium (fades, dissolves and animations), yet, what is of equal great importance is how Méliès did not stop at these technical achievements alone; he dedicated himself to finding ways to use his technique to present compelling content. Perhaps the best way to summarize the ‘Méliès Effect’ in AR, as I’m labeling it, is in the words of John Andrew Berton Jr.: “Even though Méliès’ work was closely involved with the state of the art, he did not let that aspect of his work rule the overall piece. He used his technique to augment his artistic sense, not to create it” [1].

I interpret this as Méliès maintaining an artistry in the medium, being enthused by the possibilities of the technology and allowing the medium to guide his explorations and work, but not overrule it. Méliès serves as an inspiration to my creative practice and work in AR in that the technology inspired stories and the direction of the work, but he also gave himself the freedom of experimentation to move beyond the constraints and invent new ways of applying the technology. As such, Méliès introduced new formal styles, conventions, and techniques specific to the medium of film. This is the task at hand for AR pioneers and adventurers: to evolve novel styles and establish new conventions towards a language and aesthetics of AR.

This list is part of an ongoing work. What’s on your list of the ideas that will change AR? Let’s continue the conversation on Twitter, the boardroom, or at your next event. Get in touch; a wonderful world of AR awaits and it’s ours for the making.

[1]    Berton Jr., John Andrew. “Film Theory for the Digital World: Connecting the Masters to the New Digital Cinema.” Leonardo: Journal of the International Society for the Arts, Sciences and Technology. Digital Image/Digital Cinema: A special edition with the Special Interest Group in Graphics (SIGGRAPH) Association of Computing Machinery, New York: Pergamon Press, 1990, p.7.

Google’s Project Glass & Defining a New Aesthetics in Augmented Reality

Will Google’s Project Glass change the way we see & capture the world in Augmented Reality (AR)? What kind of new visual space will emerge?

Google recently shared first-person perspective photographs and video captured directly from the Project Glass wearable AR eyewear prototype featuring video of a Glass team member doing a backflip on a trampoline (above) and photographs that  would be quite difficult for you to take while holding a camera in your hands. In a presentation at the Google+ Photographer’s Conference, Tech Lead Max Braun stated, “I think this can bring on a new style of photography that allows you to be more intimate with the world you are capturing.” Braun also pointed out how Glass is a connected device and how the moments you capture can be immediately shared. While going through some of the photographs team members had taken using Glass, he noted, “Some of the shots make you really feel like you’re there.” He referred to Glass as “an evolution of cell phone photography. It’s the next step of the camera that’s always with you.”

Having worked with AR as both a PhD researcher and designer for the past 7 years, my interests are in how AR, as a new medium, will come to change the way we see, experience, and interact with our world. Although Project Glass is still in an early prototyping phase, and the photography and video that have been shared are not AR, they do offer intriguing possibilities for how such AR eyewear can alter how we come to capture and share images and video of our surroundings and what new aesthetic styles of imagery this may generate.

We are at a moment in AR’s emergence as a new medium when we can look both to the future and to the past: still seeing the previous forms that are shaping AR while paving new paths, contributing to novel styles and conventions. It is imperative for artists, designers and storytellers to work collaboratively with computer engineers in industry and academia to steer AR forward and contribute to a new aesthetics and language of AR.

Image: David Hockney, “Merced River, Yosemite Valley, Sept. 1982”, Photographic Collage.

The photographs (and even video) from Project Glass, for me, recall artist David Hockney‘s photographic collage work, known as his “joiners”, in which multiple individually shot photographs were layered to compose a larger image. In conversation with writer and friend Lawrence Weschler, Hockney states of his photocollages, “I realized that this sort of picture came closer to how we actually see, which is to say, not all-at-once but rather in discrete, separate glimpses which we then build up into our continuous experience of the world” (Weschler 11). Hockney continues, “There are a hundred separate looks across time from which I synthesize my living impression of you” (11). The act of seeing is a process of synthesis akin to Hockney’s combination of photographs, each square documenting a separate look to compose a totality of the cumulative experience of seeing “across time” and forming, as Hockney states, a “living impression”, shaping and growing continually. Hockney’s joiners were greatly influenced by the Cubist artists’ sense of multiple angles and movement.

Image: David Hockney, “The Scrabble Game, 1983”, Photographic Collage.

Hockney states an “ordinary photograph” is missing “lived time”, referring to photography as “looking at the world through the point of view of a paralyzed Cyclops – for a split second”, and hence not conveying a true “experience of living in the world” (9). He notes to Weschler, “If instead, I caught all of you in one frozen look, the experience would be dead – it would be like looking at an ordinary photograph” (11).

Is there something extraordinary, then, about the photographs captured with Project Glass? Yes, I believe there is, and we’re only beginning to see what might be possible. So what’s so special about these images then; how do they differ from “ordinary photographs”? It’s interesting to think of the Project Glass photos as being captured by a “Cyclops”, to refer to Hockney, because, well, that’s basically what they are: a single lens attached to your head that sees and captures the world from a first-person perspective. Yet, to continue with Hockney’s conceptualization, I believe these images get closer to conveying “a true experience of living in the world”, one where your experience in that moment is documented as is without having to stop and grab your camera. Braun comments on “how effortless and natural it is to do so”, with Google Co-Founder Sergey Brin adding, “I think this can bring on a new style of photography that allows you to be more intimate with the world you are capturing, and doesn’t take you away from it.” Project Glass captures what you are in fact seeing in that moment, and very close to how you are seeing it. The experience is still very much alive, not dead, as Hockney argued of ordinary photographs, and ‘living’ also in the sense that these experiences can immediately be shared with others over a network.

Hockney’s collages were an inspiration to some of my early creative experiments in AR, in a series I refer to as the AR Joiners, 2008-2009. The AR Joiners extended Hockney’s concepts to use 2D video clips in AR in a tactile composite form, of individual paper markers overlapping to create one larger AR collaged scene. Each of the short video clips that compose the AR Joiners was recorded over a series of separate moments, as opposed to one take that was cut into multiple fragments running on the same timeline. This was a conscious design choice: the AR Joiners were about the body moving in time (in both capturing the video footage and to later have the viewer reassemble it as an AR experience, piecing together the separate video clips across time with paper markers, akin to Hockney’s photocollage process), in distinct moments and views, which accumulate to combine a total memory of the space or experience across time. (The AR Joiners are discussed in the ISMAR 2009 paper “Augmented Reality (AR) Joiners, A Novel Expanded Cinematic Form ” published by IEEE).

Image: “Rome Colosseum”, Screen Capture of AR Joiner, Helen Papagiannis, 2009. Each square shows 2D planar video playing atop AR markers to create a composite of a total scene across time.

When I was working on the AR Joiners from 2008-2009, Microsoft’s Photosynth had also recently launched. Photosynth, for me, recalled the aesthetic of Hockney’s Joiners. At the time, Photosynth was a web-based photo visualization tool (now available as a panorama app on smart phones) that could generate a three-dimensional (3D) representation from a collection of two-dimensional (2D) photos of a place or object. Software analyzed the photos for similarities and then constructed a 3D layered display of the photos through which viewers could navigate and delve further into the scene. “Synths”, as they were referred to, created a totality of the cumulative experience of seeing “across time”, comparable to Hockney’s Joiners (1970-1986).

Image: Photosynth of the Colosseum in Rome by Photosynth user Protec.

So what does all of this have to do with Google’s Project Glass?

I believe Hockney’s Joiners, the AR Joiners, and Photosynth each contribute to an aesthetic that Project Glass, in documenting our lived experiences of the world, has the potential to extend into new visual conventions. I’d like to propose that each of the above projects applies a (neo)baroque aesthetic, one which I think is very important for AR and we will see more of as AR continues to evolve into a new medium beyond just a technology.

In the article, “Architectures of the Senses: Neo-baroque Entertainment Spectacles”, 2003, Angela Ndalianis writes,

“The baroque’s difference from classical systems lies in the refusal to respect the limits of the frame. Instead, it intends to invade spaces in every direction, to perforate it, to become as one with all its possibilities” (Ndalianis 360).

This description of the baroque aligns quite nicely with Hockney’s Joiners, AR Joiners and Photosynth, which each demonstrate ways of moving beyond the limits of the single frame, expanding in multiple directions, puncturing conventional space.

Image: “The Glorification of Urban VIII”, Painting, Pietro da Corona, Rome, 1633-1639.

Ndalianis identifies Pietro da Corona’s ceiling painting “The Glorification of Urban VIII” (Rome, 1633-1639) in the Palazzo Barberini as baroque, where “the narrative from one panel literally spills into the narrative of another” and “the impression is such that, in order to spill into the next visual and narrative space the figures and objects perceptually appear to enter into our own space within the Palazzo Barberini” (361). In contrast she notes, “A strictly classically aligned composition would, instead, have enclosed and kept discrete the separate narrative borders.” (361). AR enters into “our own space”, with the narrative of the augmented environment spilling into our physical surroundings.

Ndalianis discusses how the spectator is central in (neo)baroque space and vision. She writes,

“With borders continually being rewritten, (neo)baroque vision provides models of perception that suggest worlds of infinity that lose the center that is traditionally associated with classically ordered space. Rather the center is to be found in the position of the spectator, with the representational center changing depending on the spectator’s focus. Given that (neo)baroque spectacle provides polycentric and multiple shifting centers, the spectator, in a sense, remains the only element in the image/viewer scenario that remains centered and stable. It is the audience’s perception and active engagement with the image that orders the illusion” (358).

With AR, the position of the spectator is the center, with the possibility of changing the AR experience depending on the spectator’s focus, position, and context. Just as Ndalianis writes, it is the spectator’s “perception and active engagement” with the AR “that orders the illusion.” In a previous article, I described AR as primarily a lean back model; however, AR has great potential to become an interactive lean forward model, one in which “active engagement” will make the spectator’s context, interests, and motivations even more central to ordering and defining the illusion.

Although the photographs Google shared are 2D and not interactive in this early stage, Google’s Project Glass has the potential to impact a “lean forward” model in AR and contribute to a (neo)baroque style in AR where the spectator, through customized eyewear, will really be at the center in a new ordered visual space. The possibilities for capturing first-person perspective photography and video directly via the eyewear may come to help define a new set of aesthetics and stylistic tendencies, perhaps one closer to Hockney’s vision of “lived time” and a “record of human looking” in his joiners. I am intrigued to see how the current 2D images and video will expand beyond the frame once Project Glass is AR enabled, to quote Ndalianis again, extending “space in every direction, to perforate it, to become as one with all its possibilities.”

Image: Haptics Demo from the Magic Vision Lab, University of South Australia.

But, dear readers, I cannot leave you here on just a visual note. We shall not limit AR to strictly a visual experience, it must be fully sensorial as we push ahead as an industry and community of researchers to grow the medium. Ndalianis writes, “When discussing the neo-baroque we also need to consider an architecture and regime that engages the sensorium” (367). She refers to “haptic, gustatory, olfactory, to the auditory and the visual”, all of which AR as a new medium needs to experiment with and extend into, beyond computer vision and tracking. Adrian Cheok’s keynote at ISMAR 2011 in Basel, Switzerland addressed the need for AR to engage the other senses. With projects exploring taste and smell in AR, like Meta Cookie from the University of Tokyo, and work being done in AR haptics, such as at the Magic Vision Lab, University of South Australia, AR will continue to expand in new ways, beyond visual frames and into the full human sensorium, to truly become “one with all its possibilities”.

Let’s continue the conversation on Twitter (I’m @ARstories), or in the comments section below.

References:

Ndalianis, Angela. “Architectures of Senses: Neo-baroque Entertainment Spectacles.” Rethinking Media Change: The Aesthetics of Transition. Cambridge, Mass.: MIT Press, 2003.

Weschler, Lawrence. Cameraworks, New York: Knopf, 1984.

*Update: May 30, 2012: Thanks very much to Bruce Sterling for picking up my article and posting it on Wired.com today!

*Update Feb 21, 2013: New video from Google showing how Glass will work

The Future of Augmented Reality is in our Hands with Haptics & Touch Screens

There are two questions that I’m often asked: ‘What’s in store for the future of AR?’ and ‘What would you like to see in the future of e-books and tablets?’

My answer to both is haptics and tactile feedback.

In August 2011 I had the pleasure of visiting the Magic Vision Lab at the University of South Australia and experiencing their AR haptics demo. Wearing a head-mounted display (HMD) and using a Phantom stylus, I could feel the scales of a virtual fish which appeared before me. I was able to touch virtual objects and receive tactile feedback, as though these were real, physical objects I was interacting with. This completely threw off my sensibilities of the real, having difficulty distinguishing between what was real and what was virtual. This experience signified an important shift for me in the medium of AR: in the past, the only tactile component of AR was that which physically existed in our environment.

Image: Haptics Demo from the Magic Vision Lab, University of South Australia.

I was fascinated by the sense of touch and tactile feedback paired with AR. However, I was left desiring a more direct interaction in this experience, without the HMD or stylus.

Enter Senseg’s touch technology for tablets, which premiered at the Consumer Electronics Show (CES) last week. If we can merge this with AR, I truly think it can be a game changer and help push the medium forward in new important ways that are currently absent. To date, AR has been primarily a vision-based medium; we haven’t really got to augmenting touch and we can’t ignore these other very ‘real’ senses for much longer.

In the short video above, Dave Rice, VP of Senseg, discusses the technology as adding tactile effects to touch screen displays including smart phones, tablet computers, touch pads and gaming devices. He discusses the possibilities for gaming applications (I personally think this would be incredible to apply to storytelling as well) and describes a treasure hunt game in which a treasure chest is hidden and can only be found by feeling around on the screen. Dave says, “There were no visual cues there and that’s pretty exciting because now we can move to the world of feel to complement what you’re seeing, or to work independently from it and really create a new world to explore.”

For me this perfectly describes the future of AR and its potential. I think about this last quote and how it applies to my recent AR Popup Book “Who’s Afraid of Bugs?”, the first AR book designed for iPad 2. For me the next step in this book is to be able to touch and feel the texture of the virtual spider that magically appears. Imagine petting the spider and feeling each tiny hair.

(Also with today’s announcement from Apple on iBooks textbooks for iPad, “a new kind of textbook that’s dynamic, current, engrossing, and truly interactive”, imagine how haptics and tactile feedback could change the future of education in e-books, as well as AR. Talk about ‘bringing the curriculum alive’.)

A Brief Rant on the Future of Interaction Design”, is a very relevant and excellent article in which Bret Victor asks us to aim for a “dynamic medium that we can see, feel, and manipulate”. Bret’s article immediately resonated with me when I read it in November and I shared it with the AR community via Twitter as something important we needed to be aware of and really work towards.

Image: Bret Victor

Bret emphasized the use of our hands to feel and manipulate objects. He writes, “The sense of touch is essential to everything that humans have called ‘work’ for millions of years.”

“Now, take out your favorite Magical And Revolutionary Technology Device. Use it for a bit. What did you feel? Did it feel glassy? Did it have no connection whatsoever with the task you were performing?” Bret calls this technology, “Pictures Under Glass”, noting that it sacrifices “all the tactile richness of working with our hands”.

Bret links to research that’s been around for decades in haptics, tangible-user interface (TUI) and even Touchable Holography. He comments on how this research has always been marginalized, but that “maybe you can help”.

AND WE CAN. As Bret so wonderfully states, the most important thing about the Future is that it is a choice. As an AR industry and community, it is our choice as to how this medium evolves. “People choose which visions to pursue, people choose which research gets funded, people choose how they will spend their careers.”

Let’s do this. Kindly get in touch if you’re keen to collaborate!

Let’s also continue the conversation in the comments area below and on Twitter; find me, I’m @ARstories.

*Hat tip to Stephen Ancliffe for sharing the Senseg video with me.

UPDATE (March 6, 2012): The Next Web and The Guardian published articles on haptics/touch-feedback possibly being the secret feature of the iPad 3. Could AR’s tactile future be here very, very soon? Let’s hope so. My original post was published on January 19, 2012.

UPDATE (March 7, 2012): Sadly, haptics didn’t make it into Apple’s big “New iPad” announcement today. Let’s hope for touch-feedback capabilities in the next iPad. Now, that would truly be a magical device.

TEDxDubai talk: Augmented Reality & The Power of Imagination

I’m very pleased to share video of my recent TEDxDubai talk! Naturally, I discussed Augmented Reality 🙂 but I also tied in my Tilt Shift photography to this year’s TEDx theme of “The Beauty of Small Things”.

Fighting a fever and a flu bug, I mustered all my might and was so happy to make it on that stage! Let’s say, it really was an out of body experience! 🙂

My talk featured my children’s AR pop-up book “Who’s Afraid of Bugs”. The book was created using AR software by Junaio. (You can also read more here in a recent interview I did.)

I spoke to the importance and power of imagination and make-believe, and how they pertain to AR at this critical junction in the medium’s evolution. When we make-believe and when we imagine, we are in two places simultaneously; make-believe is about projecting or layering our imagination on top of a current situation or circumstance. In many ways, this is what AR is too: layering imagined worlds on top of our existing reality.

As a designer/artist/PhD researcher I am both a practitioner and a researcher, a maker and a believer. As a practitioner, I do, create, design; as a researcher I dream, aspire, hope. I am a make-believer working with a technology that is about make-believe, about imagining possibilities atop actualities. Now, more than ever, we need more creative adventurers and make-believers to help AR continue to evolve and become a wondrous new medium, unlike anything we’ve ever seen before.

Make-believe is an extension of wonderment for me. In my previous TEDx talk, in 2010 at YorkU (embedded below) on wonderment and the creative process, I spoke to wonder guiding my work in AR, and the importance of developing wonderment in technology alongside compelling content/storytelling as the medium advances.

I am grateful to the superhuman TEDxDubai curators Giorgio Ungania and Natascia Radice, all of the wonderful volunteers, the mega talented fellow speakers and each of the incredibly engaging delegates for a truly stellar event. Thank you, thank you, thank you.