Augmentology"...is a concise manual of reality for our digital age."

Mark Hancock,_Augmentology: Interfaccia Tra Due Mondi_

[Sponsored by The Ars Virtua Foundation/CADRE Laboratory for New Media]

This TED talk by Peter Molyneux:

…demos Milo, a hotly anticipated video game for Microsoft’s Kinect controller. Perceptive and impressionable like a real 11-year-old, the virtual boy watches, listens and learns — recognizing and responding to you.

The demonstration begins with an explanation of how Milo is constructed. A combination of the following three elements allow Milo to exist:

  1. A Kinect Camera
  2. Artificial Intelligence developed by Microsoft
  3. Emotional Artificial Intelligence built by Lionhead Studios.

Milo moves through a synthetic environment predicated on User-directed biofeedback/body gestures: no mechanical controllers are necessary. Unfortunately, Milo’s introductory learning curve [which is integral to the "game" leveling system] involves inherent gender bias: if you’re a girl, your initial game variable is a Butterfly whereas if your a boy, you’ll be presented with a Snail.

The demonstration goes on to illustrate how Milo’s face is comprehensively AI driven. His facial movements include blush response, nostril “flare” size [indicating stress], “body matching” [causing neuro-linguistically driven facial alterations] and responses to verbal cues. Peter then describes how Milo’s personality development is predicated on a Cause-and-Effect dynamic. This causality is showcased via 3 examples:

  1. The User can choose to direct Milo to squash a snail: if the User does it will effect “…how Milo develops”.  The specifics of the verbal stimulus employed including how the User vocalises [specific phrases and intonations] all contribute to a database that informs and effects future interactions.
  2. The User teaches Milo to skim stones over the surface of a river [skewed gender stereotyping is again evident here].
  3. The User choosing to clean Milo’s room: Milo’s recognition of the User’s beneficial intervention and verbal engagement promotes sustained developmental interaction based on [what Peter terms] “deep psychology”.

This “deep psychology” [or what is described in synthaptic terms as "augmentology"] encourages a User’s empathy loadings. This in turn allows such games to shift towards complex experientially-defined engagement. These games surpass the hollow reinforcement of contemporary Social Games such as Farmville: instead, the User “levels up” by knitting fictionalised engagement with personality/identity construction and personalised growth variables. The element of cloud-directed learning [coaching synthetic humans whose social and chronological development depends on "crowdsourced" input] creates enormous opportunities for instruction and feedback via these types of  “Reality Gaming” systems [highlighted here by Seth Priebatsch]:

“There are many worlds and many realities in our universe. When one reality, or one world-view is superimposed on another, it is inevitable that social, economic and cultural problems arise. Hierarchies of worlds are constructs of a bygone era. Ecologies of worlds should guide us in considering our future… We can begin by designing environments that can respond to physical, environmental, or social needs. Not only the needs of human beings, but also of the organisms and elements with whom we share the Biosphere.” – fo.am

Rezzing occurs in the space in-between worlds. Rezzing happens in the moment we switch from one reality to another: where the structure of synthetic worlds is unveiled. We see these spaces appear gradually – textures, alpha channels and audio appear in layers. Forms start as simple grey patterns that morph and evolve via emergent detail. These patterns resolve as final forms that adhere to in-game physics and flop into “place”.

Rezzing 1

When I begin rezzing – and am between avatars – my body disappears. Then, the simple basic shape beneath is exposed with pitch black skin and bizarre proportions. Finally, my body parts materialise.

I stand naked: staring ahead as my clothes begin to appear, one piece at a time. As the textures of my skin are downloaded, my blurry body is redrawn in photorealistic detail. In Second Life, Linden Labs has added a feature where rezzing avatars are surrounded by a cloud whilst forming. This cloud presumably covers the moments of nakedness while an avatar’s clothes are appearing and bare pixel genitalia are exposed.

There is no geophysical equivalent to the act of rezzing. The closest phenomenon is the act of awakening from – or falling into – dreams. When an object or avatar is rezzed in a synthetic environment, its data representation is downloaded from the database into the local client. On screen, a visual “something” is created from synthetic “nothing” – an ontological novelty out of the pure void. This act reveals a flaw in the materiality and persistence of these worlds or a type of virtual ontology similar to Deleuze’s Spinozan plenty without void.

After encountering the whooshing sound that indicates teleporting, I am dropped into an incomplete world. Often during this phase, my avatar manifests in a falling animation. First, all is sky and water which faithfully glistens with the sun (according to environment settings). Then, distant objects appear. In complex areas this can take minutes as particle scripts initialise and begin to swirl and glow before the details of architecture appear.

555 KUBIK | facade projection |

During the rezzing process, as a user’s body begins to form they step into a swirl of affect. This affect may induce feelings of identification with the avatar or a revulsion from it. This emotional polarisation may produce a sense of pleasure in seeing or a sense of disjunctive discomfort. The activity of the database creates its own unreproducable order dependant on the speed of the bytes transferred. Hair or pants/skirt may take minutes to download, with the avatar blinking into space with a bald head or exposed thong in the meantime. At this juncture, the avatar hover-stands in an unfolding environment and waits for the expected transactions of the “normal” synthetic world to begin.

How do we come to understand the resonances, affects and effects of rezzing into synthetic environments? With augmented reality making headline news, can we think of other ways of entering other realities which are not limited to visual modes? What about pain? Sound? Smell? Can Mixed Reality Performances be used to develop and explore these methods of realityshifting? If we can think of ways of finding spaces between realities, then can we think of the space between realities as similar to the space between genders and sexualities? Could entering a space between realities free us of certain rules, be a strategy of liberation and transformation? Part 2 will explore these questions.

“Augmented Reality”.  It doesn’t quite roll off the tongue in a manner that could be described as euphonious. The term sounds lopsided and clunky. Definitely not two words that I find compelling or evocative. Those two words are the literary equivalent of a blunt instrument: slow, heavy, and strong. In fact, the term feels like a badly written movie that goes straight to DVD (and that you’ll eventually find it in the bottom bin at a Wal-Mart sale for $2.99). You can’t even make a workable abbreviation out of it; if you say “AR” in the wrong crowd, they will think you are referring to Accounts Receivable or Arkansas.

While we shouldn’t judge a book by its cover (or even by the movie “based on the book”), we likewise shouldn’t judge a technology based on its name. In a similar vein, we shouldn’t be quick to discount augmented reality based on early examples/demonstrations that appear gimmicky. It is easy to miss the full earth-shaking, mind-rattling, jaw-dropping paradigm-shifting potential of future AR as both the technology and the industry matures. We are at the dawn of something new: it is almost impossible to understand the full scope and impact of what is coming. In many respects, it’s as if we have discovered a new country full of promise and hope. This “AR country” offers enormous potentiality for change, as well as many associated risks.

And just what is this augmented reality stuff anyway?

Augmented Reality in its most basic form is the blend of the real and the virtual. Beyond this, there is some contention as to what AR is or isn’t. There’s also the issue of whether any given example could fall under the categories of Mixed Reality, Virtuality, or something else entirely. We could construct various models and/or other litmus tests to determine if something should be referred to as AR, or we could easily adopt any of the more common definitions.

For now, let’s just keep it simple and a little broad. AR is the blend of the real and the virtual which can be experienced through a number of modes or modalities. It usually requires a digital video camera, a monitor, and either a printed marker or a pre-defined image which is tracked (which effectively replaces the marker). This definition is particularly suited to the past and the present state of AR technology.

In the near future, AR will incorporate geolocative, spatial, contextual, interactive, semantic, mobile, massively multi-user, and pervasive technologies. In the long-term AR will evolve into a platform that is extraordinarily dynamic and immersive. The popular/primary interface will include a pair of wearable displays with transparent lenses similar to a head’s up display. The form of these wearable displays will be nearly identical to a contemporary pair of Ray Bans or Oakleys:

This interface will be linked (hopefully wirelessly) to a mobile internet device that is likely to be clipped to a belt or sewn into clothes.

So what does all this mean? Why am I constantly going on about the blue sky potential of mobile augmented reality? With all combined AR elements, we will effectively be able to create an experience that is like a rudimentary Star Trek Holodeck. Interactive virtual objects, information, and life sized avatars will blend with the world around us:

…and will appear like semi-transparent holograms or digital ghosts. We will own virtual pets. Data visualizations will exist for everything from directional floating arrows to information tags anchored to every object (including us). 3D movies will be completely redefined. MMORPGs will be played in public parks. Doctors will see patients overlaid with X-Ray and MRI information. Education will come alive in the classroom….

There are thousands of potential applications and mobile AR experiences that will change nearly every aspect of our lives. A media revolution will occur; we will be thrust into a new information age where we are no longer chained to bulky PCs, heavy laptops and/or power hungry monitors.

This above vision is one that I am pursuing through my company, Neogence Enterprises. Although Augmented Reality has been in existence previously – the list of true early pioneers, innovators, and academics is long – Neogence wants to be at the forefront of taking AR to a new functional level. It may be a few years before our full vision is realized. There are plenty of technical hurdles still to overcome; in the meantime Neogence will aggressively push ahead one step at a time, building up piece by piece. If all goes well, we will be launching the first commercial version of a global mobile augmented reality network on October 10th, 2010 at 1010am Eastern. We plan on releasing bits and pieces along the way with some closed beta testing in the Spring. We want to build this emergent technology correctly and create something that is infinitely extensible and expandable. We intend to focus on the end-user experience and empower you (the user) to create wonderfully original applications and content.

Join us on our journey and help us build the future. In the next week or so, Neogence will open mirascape.com. We will allow for closed beta registration in the Spring. I have some special plans for the first 100,000 unique sign ups when we launch. The future awaits…

Becoming Dragon

I am becoming something else. In this moment, this being-in-transition, I am willfully stepping into the unknown. I am between realities. I can only imagine what I want to become, and then choose to become that new thing, but it is radically ungraspable, inconceivable. I can never know the reality of what I am choosing to become, desiring to become. My decision to transform can never be the right one, because it is always based on an illusion, a fantasy, a false conception with only a few points of data, not the rich details of an embodied life. As the transformation unfolds, those unknown events begin to occur, like seeing my breasts in the mirror for the first time after shaving my chest closely, feeling the movement in my orgasm change into something new or just walking down the street for a moment as a girl, unnoticed and not needing any special attention. My decision to become something else is always a decision to become mythopoetic, because the reality of the new state is always unknown, imaginary, a construct, a fantasy. Yet I don’t seek to decry this radical state of uncertainty but to embrace it. The very moments of everyday perception are also simply intersections of a real materiality with my symbolic and imaginary processing engines making sense of them, down to the way that I understand what pleasure is and what pain is and when the two become too close so as to be confused. And a choice to not transform is of course still a choice to transform into a different state, as our bodies are all in permanent transition, aging, training, consuming, producing, perceiving, creating new folds in our craniums.

Becoming Dragon Day 6

Through this process, I am also becoming an artist. Yet this is simply another fantasy which I use to structure my desires and find direction. Artist, porn star, student, professor, father, mother, husband, wife, lover, child, priest, these are all simply performances of being, yet their being a performance makes them no less real, nor more real, just another fold in the swirling interplay, the kaleidoscope of realities that is our being.

A mixed reality performance using an online 3D virtual world simply highlights the fantasy nature of our everyday interactions, of the physical world, by referring back to the physical, stirring up our memories and conceptions of embodiment. A mixed reality performance is a misnomer, as every step of our waking lives is a mix of realities, our self-perceptions, muscle memory, proprioceptions, others’ perceptions of us, our perceptions of their perceptions of us as they look at us, or don’t, our understanding that we are walking, taking on a step, on a sidewalk, by a building. A mixed reality performance simply highlights this fact, or this fiction, and allows one to see and begin to question the mythopoetic structure of reality. While performance sought to get closer to the real, to escape mediation, I embrace the pleasure of bits projected on my skin and the flickering of digital lights in my eyes, of the simulacrum of my own fantasy which creates that same fantasy. The mixture of real and imaginary is more real for being so.

While one can draw one’s fantasy, or write it out in words, 3D virtual worlds bring us one step closer to seeing in front of our eyes the fantasy films which play behind our eyes, yet there are many more steps to bring us closer to dreams. In my dreams I smell, I feel my body in action, I have visceral emotions, yet software such as Second Life is far from emulating such unreal realities. Still, we can make steps closer to dreams, with motion capture, head mounted displays, tactile interfaces, wish pressure interfaces. I wish for another reality, the electricity on my skin changes, transferring the new desired location to the system, and the pressure interface responds, as my chair morphs from a car seat to a comfy recliner in my skybox…loading world…arriving.

I am transreal. Look at me. When you do, a million iridescent scales across my dragon hide flick, move and align to create a multiplicity of perceptions, transversal illusions and realities cutting through each other, intersecting, dancing. Look at me. You see a shimmering of my fantasies and yours, a convergence of your minute sensory events, your imaginary constructs and my desires. Look at me. The mythopoetic elements of your reality and mine come into contact, unwind and become a recombinant event of male and female and something else, something more, for just an instant. Perhaps after that initial instant, one of your myths takes over your perception and you decide that you understand, but before that, I instill confusion and doubt. I can see it on your face.

I am becoming mythopoetic, a shapeshifting creature of legend, a dragon. Standing here, on the border, the sunlight through the clouds defeating the fence, I am transreal, between realities, moving through layers of the symbolic, the imaginary and the real, simultaneously quivering, swapping out and swapping back in, too fast to find the border between them. I am existing between my fantasies and desires, which are driving the changing form of my body, and the moment of perception in which you see me and call me maam, sir, dude, miss, or avoid choosing a category. Speaking, being with different people throughout the day, my body and name changes, my realness or unrealness oscillates. You see me standing here, but really, you see my avatar, my body, which is under construction. We bring our illusions together. You see soft skin. I see the pills and the bloody razor that made it soft, making me feel happier, more feminine. You see scales, I see textured prims and their glow values.

A dark moment in the street at night, your illusions of masculinity swirl up against the confusion I install in you, and you attack. My reality becomes a blur, a flurry of motion, and a sharp chemical emotional reaction, as I strike back with pressurized chemical weapons. Yet even in that moment, I am transreal, between my reality and yours, only finding a hard fissure between the two.
.
In bed with my lover, we are transreal, deep in our illusions of each other, feeling our very real emotions for each other, between bodies, looking into her eyes, slipping out of myself and my concerns and out into the bright nebula of pleasure…