AR: Beyond Conventional Time and Space
There’s a wonderful spirit of invention in the Augmented Reality (AR) community right now — the incredible experimentation and work being done with Apple’s ARkit is tremendously exciting. It’s truly thrilling to see AR accelerate both creatively and technically. Keep the demos coming!
Working with AR for 12 years now, it’s been amazing to experience AR’s evolution (read all about it and where things are headed in Augmented Human). It’s also been fun digging back into the archives recently to revisit some of my early AR prototypes from 2005 and later (especially the pre-iPhone projects and thinking about ARkit applications today).
Zach Lieberman’s ARkit experiments and process have been particularly inspiring to watch. Lieberman’s AR camera app test brought to mind artist David Hockney’s stunning and innovative photocollages — referred to as ‘joiners’ — from the 1980’s. Hockney’s joiners were a strong influence in my early AR prototypes. I was dazzled by Hockney’s approach to representing time and space, amplifying the abstraction and dynamism of Cubism, and building on the work of artists like Pablo Picasso and Georges Braque.
David Hockney, The Skater, 1984, photographic collage.
Hockney said of his photocollages, “I realized that this sort of picture came closer to how we actually see, which is to say, not all-at-once but rather in discrete, separate glimpses which we then build up into our continuous experience of the world”. Hockney presented a new way of seeing via the camera, one that mirrored the way we see in reality: through multiple glimpses that we piece together. AR, too, has the potential to rethink and present a new way of seeing and interaction with our world.
My AR Joiners series (2008-2009) paid homage to Hockney’s work. The AR Joiners widened Hockney’s concepts to use 2D video clips in AR with individual paper AR markers overlapping to create one larger AR collaged scene. The short video clips that composed the AR Joiners were each recorded over a series of separate moments (as opposed to one long video take that was cut into multiple fragments running on the same timeline). This was a conscious design choice: the AR Joiners were about the body moving in time, akin to Hockney’s photocollage process, with distinct moments and views, accumulating in a total memory of the space and experience across time. (Read more about the AR Joiners in a paper I presented at ISMAR 2009, Augmented Reality (AR) Joiners, A Novel Expanded Cinematic Form published by IEEE.)
Hockney’s joiners, the AR Joiners, and the experiments we’re seeing today with ARkit create new visual conventions beyond traditional time and space, all working toward building a novel language of AR. Another contemporary example is floatO, a photography iOS app that uses ARkit by artist Dan Monaghan.
In Architectures of the Senses: Neo-baroque Entertainment Spectacles (2003), Angela Ndalianis writes,
“The baroque’s difference from classical systems lies in the refusal to respect the limits of the frame. Instead, it intends to invade spaces in every direction, to perforate it, to become as one with all its possibilities.”
Ndalianis’s description of the baroque aligns quite nicely with Hockney’s joiners, the AR Joiners, and even Lieberman’s and Monaghan’s ARkit explorations; each of these works demonstrates ways of moving beyond the limits of the single photographic frame, expanding time in multiple directions, and puncturing conventional space.
But perforating the boundaries of reality doesn’t stop here: to truly grow the possibilities in AR, we will need to move past strictly vision-based experiments and engage the entire human sensorium with auditory, haptic, gustatory, olfactory, and visual experiences (in Augmented Human, there is a chapter dedicated to each of the senses and the opportunities with AR).
This is truly just the beginning of the dynamic, shape-shifting, and wonder-inducing new reality that is to come. I can’t wait to see, hear, touch, smell, and taste what’s next.
Let’s continue the conversation on Twitter: I’m @ARstories.
PURCHASE “AUGMENTED HUMAN” BOOK: ENGLISH: AMAZON (US), AMAZON (CA), AMAZON (UK), ITUNES, GOOGLE PLAY
KOREAN TRANSLATION: ACORN PRESS, CHINESE TRADITIONAL SCRIPT TRANSLATION: GOTOP, ALSO AVAILABLE IN INDIA: SHROFF PUBLISHERS