AR Storytelling: The Body and Memory Making
When designing AR experiences and AR stories, we too often forget something very important: the human body.
Augmented content is not two-dimensional or flat; it unfolds in our physical space, in our personal surroundings. We’re walking around it and crouching on the floor, exploring it from different angles and heights. AR stories are about our body in relation to the virtual constructions inhabiting our space as much as they are about the content presented.
Earlier this month, The New York Times debuted their first AR enabled article within their iOS app, a preview piece for the Winter Olympics in Pyeongchang, South Korea. Using an iPhone or iPad, readers can meet Olympic athletes in AR—figure skater Nathan Chen, big air snowboarder Anna Gasser, short track speed skater J.R. Celski, and hockey goalie Alex Rigsby—as if they were paused mid-performance.
John Branch, the author of the NYT AR article, observes how, despite all of the camera angles, watching sports on television creates an experience where you are “passively cocooned on the couch as a mere spectator to miniaturized athletes squeezed through a two-dimensional plane.” The NYT has readers moving and actively engaging with the content in AR, walking around the room where you’re reading the story. Far from being “cocooned on the couch”, you’re crawling on the floor to look speed skater J.R. Celski in the eye. And the 3-D visualizations are not miniaturized like on your TV; they’re sized true to life, and to scale. For example, you’re looking up at figure skater Nathan Chen as he appears 20 inches off the ground in your room, the height he would be mid quadruple jump.
The NYT gets this aspect of AR storytelling right: it’s not just about the athletes’ bodies and their form, it’s also about the way you’re maneuvering your body in conversation with the story and your space as you experience the content.
But there’s something else happening here that we also need to take into account when designing AR experiences.
I was recently chatting with Theresa Poulson about AR storytelling and my new book Augmented Human (Theresa is developing an incubator for creators to advance emerging forms of non-fiction storytelling at Video Lab West). We were discussing the NYT AR experience and Theresa said something I found very intriguing and something that I believe is often overlooked in AR. She mentioned how she walks past the place in her office daily where she encountered the 3D Olympic athlete and she remembers it as though it really happened there in that room, which it in fact did.
As I shared in my AR keynote at FutureX Live 2017 in Atlanta, we’re no longer just designing stories, we’re now designing memories with AR.
In 2016, the first experiences for the Microsoft HoloLens developer edition were introduced, including a game called Fragments, a crime drama that plays out in your physical environment and has you searching for clues in your space to solve the mystery. Kudo Tsunoda, CVP Next Gen Experiences, Windows and Devices Group, Microsoft, said, “Trust me, the first time one of our Fragments characters comes in to your home, sits down on your sofa, and strikes up a conversation with you it is an unforgettable experience.” It really is, and I especially remember the virtual rats.
“Fragments blurs the line between the digital world and the real world more than any other experience we built,” said Tsunoda. “When your living room has been used as the set for a story, it generates memories for you of what digitally happened in your space like it was real. It is an experience that bridges the uncanny valley of your mind and delivers a new form of storytelling like never before.”
There’s a higher level of emotional engagement with experiences like Fragments because the story is unique to your space, the position of your body, and your gaze. There is a direct contextual relationship with content responding to you and your environment. The way you experience Fragments in your home will be different from the way I experience it in my home. Spatial mapping and custom artificial intelligence allow a room’s layout to influence the placement of virtual content in the game, such as a piece of evidence hidden behind your furniture.
The emotions from the game stay with you long after you take the headset off, transforming into memories that are virtually scribed onto your environment, like an augmented palimpsest. And so, the importance of being cognizant of this fact and conscientious as we continue to design and develop AR stories and experiences. The audience is inviting you into their physical and mental homes. It will leave a virtual footprint. With both public and private spaces becoming stages for AR stories, let’s remember: always be generous and kind to your user. It will leave a lasting impression.
Let’s continue the conversation. I’m @ARstories on Twitter.
PURCHASE “AUGMENTED HUMAN” BOOK: ENGLISH: AMAZON (US), AMAZON (CA), AMAZON (UK), ITUNES, GOOGLE PLAY
KOREAN TRANSLATION: ACORN PRESS, CHINESE TRADITIONAL SCRIPT TRANSLATION: GOTOP, ALSO AVAILABLE IN INDIA: SHROFF PUBLISHERS