A project by Karolina Sobecka with software development by Jim George
“Sniff is an interactive projection in a storefront window. As the viewer walks by the projection, her movements and gestures are tracked by a computer vision system. A CG dog dynamically responds to these gestures and changes his behavior based on the state of engagement with the viewer…
…Sniff is created with Unity3d Game Engine which renders the dog in real time and allows to dynamically change his behavior based on the video tracking data. The sidewalk is illuminated with infrared lights. An infrared-sensitive camera is used to monitor the sidewalk in front of the display windows. We track the position of each viewer and implement simple gesture recognition, so that fast, big actions are interpreted as threatening and slow actions directed at the dog (for example hand extended in his direction) are interpreted as friendly. The dog keeps track of the attitude of the viewer and forms a relationship with them over time based on the history of interaction.”
“There are many worlds and many realities in our universe. When one reality, or one world-view is superimposed on another, it is inevitable that social, economic and cultural problems arise. Hierarchies of worlds are constructs of a bygone era. Ecologies of worlds should guide us in considering our future… We can begin by designing environments that can respond to physical, environmental, or social needs. Not only the needs of human beings, but also of the organisms and elements with whom we share the Biosphere.” – fo.am
Rezzing occurs in the space in-between worlds. Rezzing happens in the moment we switch from one reality to another: where the structure of synthetic worlds is unveiled. We see these spaces appear gradually – textures, alpha channels and audio appear in layers. Forms start as simple grey patterns that morph and evolve via emergent detail. These patterns resolve as final forms that adhere to in-game physics and flop into “place”.
When I begin rezzing – and am between avatars – my body disappears. Then, the simple basic shape beneath is exposed with pitch black skin and bizarre proportions. Finally, my body parts materialise.
I stand naked: staring ahead as my clothes begin to appear, one piece at a time. As the textures of my skin are downloaded, my blurry body is redrawn in photorealistic detail. In Second Life, Linden Labs has added a feature where rezzing avatars are surrounded by a cloud whilst forming. This cloud presumably covers the moments of nakedness while an avatar’s clothes are appearing and bare pixel genitalia are exposed.
There is no geophysical equivalent to the act of rezzing. The closest phenomenon is the act of awakening from – or falling into – dreams. When an object or avatar is rezzed in a synthetic environment, its data representation is downloaded from the database into the local client. On screen, a visual “something” is created from synthetic “nothing” – an ontological novelty out of the pure void. This act reveals a flaw in the materiality and persistence of these worlds or a type of virtual ontology similar to Deleuze’s Spinozan plenty without void.
After encountering the whooshing sound that indicates teleporting, I am dropped into an incomplete world. Often during this phase, my avatar manifests in a falling animation. First, all is sky and water which faithfully glistens with the sun (according to environment settings). Then, distant objects appear. In complex areas this can take minutes as particle scripts initialise and begin to swirl and glow before the details of architecture appear.
During the rezzing process, as a user’s body begins to form they step into a swirl of affect. This affect may induce feelings of identification with the avatar or a revulsion from it. This emotional polarisation may produce a sense of pleasure in seeing or a sense of disjunctive discomfort. The activity of the database creates its own unreproducable order dependant on the speed of the bytes transferred. Hair or pants/skirt may take minutes to download, with the avatar blinking into space with a bald head or exposed thong in the meantime. At this juncture, the avatar hover-stands in an unfolding environment and waits for the expected transactions of the “normal” synthetic world to begin.
How do we come to understand the resonances, affects and effects of rezzing into synthetic environments? With augmented reality making headline news, can we think of other ways of entering other realities which are not limited to visual modes? What about pain? Sound? Smell? Can Mixed Reality Performances be used to develop and explore these methods of realityshifting? If we can think of ways of finding spaces between realities, then can we think of the space between realities as similar to the space between genders and sexualities? Could entering a space between realities free us of certain rules, be a strategy of liberation and transformation? Part 2 will explore these questions.
“Augmented Reality”. It doesn’t quite roll off the tongue in a manner that could be described as euphonious. The term sounds lopsided and clunky. Definitely not two words that I find compelling or evocative. Those two words are the literary equivalent of a blunt instrument: slow, heavy, and strong. In fact, the term feels like a badly written movie that goes straight to DVD (and that you’ll eventually find it in the bottom bin at a Wal-Mart sale for $2.99). You can’t even make a workable abbreviation out of it; if you say “AR” in the wrong crowd, they will think you are referring to Accounts Receivable or Arkansas.
While we shouldn’t judge a book by its cover (or even by the movie “based on the book”), we likewise shouldn’t judge a technology based on its name. In a similar vein, we shouldn’t be quick to discount augmented reality based on early examples/demonstrations that appear gimmicky. It is easy to miss the full earth-shaking, mind-rattling, jaw-dropping paradigm-shifting potential of future AR as both the technology and the industry matures. We are at the dawn of something new: it is almost impossible to understand the full scope and impact of what is coming. In many respects, it’s as if we have discovered a new country full of promise and hope. This “AR country” offers enormous potentiality for change, as well as many associated risks.
And just what is this augmented reality stuff anyway?
Augmented Reality in its most basic form is the blend of the real and the virtual. Beyond this, there is some contention as to what AR is or isn’t. There’s also the issue of whether any given example could fall under the categories of Mixed Reality, Virtuality, or something else entirely. We could construct various models and/or other litmus tests to determine if something should be referred to as AR, or we could easily adopt any of the more common definitions.
For now, let’s just keep it simple and a little broad. AR is the blend of the real and the virtual which can be experienced through a number of modes or modalities. It usually requires a digital video camera, a monitor, and either a printed marker or a pre-defined image which is tracked (which effectively replaces the marker). This definition is particularly suited to the past and the present state of AR technology.
In the near future, AR will incorporate geolocative, spatial, contextual, interactive, semantic, mobile, massively multi-user, and pervasive technologies. In the long-term AR will evolve into a platform that is extraordinarily dynamic and immersive. The popular/primary interface will include a pair of wearable displays with transparent lenses similar to a head’s up display. The form of these wearable displays will be nearly identical to a contemporary pair of Ray Bans or Oakleys:
This interface will be linked (hopefully wirelessly) to a mobile internet device that is likely to be clipped to a belt or sewn into clothes.
So what does all this mean? Why am I constantly going on about the blue sky potential of mobile augmented reality? With all combined AR elements, we will effectively be able to create an experience that is like a rudimentary Star Trek Holodeck. Interactive virtual objects, information, and life sized avatars will blend with the world around us:
…and will appear like semi-transparent holograms or digital ghosts. We will own virtual pets. Data visualizations will exist for everything from directional floating arrows to information tags anchored to every object (including us). 3D movies will be completely redefined. MMORPGs will be played in public parks. Doctors will see patients overlaid with X-Ray and MRI information. Education will come alive in the classroom….
There are thousands of potential applications and mobile AR experiences that will change nearly every aspect of our lives. A media revolution will occur; we will be thrust into a new information age where we are no longer chained to bulky PCs, heavy laptops and/or power hungry monitors.
This above vision is one that I am pursuing through my company, Neogence Enterprises. Although Augmented Reality has been in existence previously – the list of true early pioneers, innovators, and academics is long – Neogence wants to be at the forefront of taking AR to a new functional level. It may be a few years before our full vision is realized. There are plenty of technical hurdles still to overcome; in the meantime Neogence will aggressively push ahead one step at a time, building up piece by piece. If all goes well, we will be launching the first commercial version of a global mobile augmented reality network on October 10th, 2010 at 1010am Eastern. We plan on releasing bits and pieces along the way with some closed beta testing in the Spring. We want to build this emergent technology correctly and create something that is infinitely extensible and expandable. We intend to focus on the end-user experience and empower you (the user) to create wonderfully original applications and content.
Join us on our journey and help us build the future. In the next week or so, Neogence will open mirascape.com. We will allow for closed beta registration in the Spring. I have some special plans for the first 100,000 unique sign ups when we launch. The future awaits…
I am becoming something else. In this moment, this being-in-transition, I am willfully stepping into the unknown. I am between realities. I can only imagine what I want to become, and then choose to become that new thing, but it is radically ungraspable, inconceivable. I can never know the reality of what I am choosing to become, desiring to become. My decision to transform can never be the right one, because it is always based on an illusion, a fantasy, a false conception with only a few points of data, not the rich details of an embodied life. As the transformation unfolds, those unknown events begin to occur, like seeing my breasts in the mirror for the first time after shaving my chest closely, feeling the movement in my orgasm change into something new or just walking down the street for a moment as a girl, unnoticed and not needing any special attention. My decision to become something else is always a decision to become mythopoetic, because the reality of the new state is always unknown, imaginary, a construct, a fantasy. Yet I don’t seek to decry this radical state of uncertainty but to embrace it. The very moments of everyday perception are also simply intersections of a real materiality with my symbolic and imaginary processing engines making sense of them, down to the way that I understand what pleasure is and what pain is and when the two become too close so as to be confused. And a choice to not transform is of course still a choice to transform into a different state, as our bodies are all in permanent transition, aging, training, consuming, producing, perceiving, creating new folds in our craniums.
Through this process, I am also becoming an artist. Yet this is simply another fantasy which I use to structure my desires and find direction. Artist, porn star, student, professor, father, mother, husband, wife, lover, child, priest, these are all simply performances of being, yet their being a performance makes them no less real, nor more real, just another fold in the swirling interplay, the kaleidoscope of realities that is our being.
While one can draw one’s fantasy, or write it out in words, 3D virtual worlds bring us one step closer to seeing in front of our eyes the fantasy films which play behind our eyes, yet there are many more steps to bring us closer to dreams. In my dreams I smell, I feel my body in action, I have visceral emotions, yet software such as Second Life is far from emulating such unreal realities. Still, we can make steps closer to dreams, with motion capture, head mounted displays, tactile interfaces, wish pressure interfaces. I wish for another reality, the electricity on my skin changes, transferring the new desired location to the system, and the pressure interface responds, as my chair morphs from a car seat to a comfy recliner in my skybox…loading world…arriving.