Essay by David Cox & Archimedia
August 15th 2013
Albrecht Dürer Mechanical creation of a perspective image by Albrecht Dürer
This is not a technological determinist essay, and its author is not a technological determinist. Interest in the technologies of seeing and perception can operate outside of a neoliberal economic model. Rather, the principles that have underwritten the best and most poignant work in the area of the technologies I’ll be describing have reflected a social democratic agenda, socialist even. The tools are neutral. Anyone can build a wearable computer, and indeed should. In my experience, wearable computers are the punk rock of computing and have been for thirty years. Generally speaking, you built them yourself, or you did not use them.
Wearing computers and by doing so seeking to mediate the information and ideas between oneself and one’s environment do not suggest as many critics of the technology claim an individualist philosophy, nor do they imply a spirit of isolationism. On the contrary, wearable computers are designed to work with those who are likewise connected. Not through commercial cloud systems like Google and its shallow mechanisms of data gathering, but through individual servers, and the customized ways these can be adapted to fit the needs of the groups that use them. There could be wearable computers for Snowden, Assange, and Manning that would enable their heroic efforts to be the better effected. These might rely upon a connection to a server set up by say, Richard Stallman, to use a metaphorical example.
Google Glass does not own the concept of a popular wearable computer or a popular AR, even if it ‘adapted’ the best aspects of the display technology from the best and the brightest. You cannot truly customize a Google Glass device. It is simply a more efficient way to use what are already ‘locked in’ social media services that facilitate the ever more efficient means of reporting on your own life and that of everyone around you. A true wearable computer works against this model, to empower the individual, to facilitate what Prof. Steve Mann calls souseveillance, or seeing from below. You wear the computer to gain control over what is seen, recorded, stored and used in your life, not to make the act of doing so benefit the commercial sector, and its ever more cozy relationship with the State Department and the National Security Agency..
When Albrecht Durer demonstrated via his 17th Century etchings how to draw in perspective, he did so by showing mechanisms that forced the student to view the makeup of the world as a series of single viewpoints. You had to look through a hole to see but a part of a mandolin, draw that part, then move on to the next part. This method encouraged the student to understand the world as a totality of multiple viewpoints, a principle which to this day underpins contemporary AR technology. Only this year has the technology made itself available to enable wearable to ‘paint’ a room with infrared beams to enable a 3D ‘camera’ to ‘see’ objects in front of it such that those objects can be used to track the placement of virtual objects over the top of them. From Durer to the Space Glasses (with Steve Mann as the Durer of our time), the discipline of seeing the world as a multiplicity of points con-joins the renaissance with that our new aesthetic era.
This notion of teaching oneself to see the world as if it were in fact a totality of singular points is on the one hand the hallmark of universality that we still live under. Man at the center of the universe. With the singular vision, all views converge on the eye of the beholder.
|“The Dam Busters”. An improvised wooden viewfinder sight device used to determine the distance from the dam before the bombardier must drop the bomb. Each of the vertical pegs must line up with the ends of the dam to provide the correct position.
In the 1950s, the propaganda British War film “The Dam Busters” depicted the bombing raid on the German Ruhr dam complex. The special bombs used needed to be dropped from a low altitude, and at an exact distance. In the film, the main character gets the idea to build a wooden “V” shaped device that enables the bombardier to line up two wooden pegs at the ends of the device with the ends of the dam he is aiming at. Again, the singular eye augmented with the wooden pegs that stand-in for real things, in this case, the physical extremities of a distant, but rapidly approaching bomb target.
In the 1977 homage to the Dam Busters, Star Wars, George Lucas reproduces the scene, this time using a small ‘targeting computer’ that lets Luke Skywalker know when it is time to fire his torpedo. Mediated vision, military seeing and the calculation of time, space and motion as addenda to human sight have been hallmarks of cinema from the beginning. Of course as the film shows, its best to turn the computer off, and use intuition. A good wearable computer is a combination of both – technology and intuition.
Augmented and Mediated Reality
The alien’s helmet in “Earth vs The Flying Saucers”. The metal turns out to be fully transparent. Reality can be augmented.
In “Earth Versus the Flying Saucers” scientists discover that invading aliens wear helmets that let them see the world around them. From the outside the helmets appear solid. From the inside, they are completely transparent, and can translate language. They are effectively real-time language translation devices; wearable computers, with built-in head mounted displays combined
Heads-up. 1982 Douglas Trumbull film “Brainstorm”. A lovely parable about hackers overcoming militarists for control of a new technology of great promise for the future of the imagination. Chrisopher Walken dons the early version of the Brainstorm Headset
The boardroom demo scene from “Brainstorm”. The zoned-out suits who have been investing in the Brainstorm technology finally get to wear the devices and have high-definition multi-sense recorded thoughts and experiences played back. The military are quick to see the Psych-War potential.
“Sword of Damocles” system, Ivan Sutherland, 1965
“Sword of Damocles” system, Ivan Sutherland, 1965
“The Ultimate Display,” Sutherland, I.E., Proceedings of IFIPS Congress 1965, New York, May 1965, Vol. 2, pp. 506-508.
In the 1982 film Brainstorm directed by Douglas Trumbull, a high tech Silicon Valley team have developed a headset that can record experiences directly onto tape. As they slowly perfect the technology, the headset shrinks to the size of a tiara. The recordings become more accurate. Someone decides to loop an orgasm and gets addicted to it. Soon seeing its potential for psychological warfare and counter-intelligence, the US military backers of the project take it over and push out the liberal humanist tech engineers, who fight back with hacking skills. It’s a great story for our time. The actual trip-like Brainstorm scenes alone are worth the price of entry, but the relevance for our story is the notion of an advanced form of augmented consciousness, made possible with technologies that are creatively re-purposed to cross boundaries. At the end of the film the team overcome the odds to successfully record the moment of death itself, enabling a person who is alive, to die, without dying.
The development of the head mounted display in the context of computers probably begins with early experimental systems like “Sword of Damocles” developed by Ivan Sutherland in 1965. This system allowed the user to view 3D generated computer graphic, and to see those graphics as if they were in the same environment as the user themselves. Closer to Virtual Reality than Augmented Reality, the significance of the Sword of Damocles system lies in its use of separate monitors, one for each eye, and the system of motion tracking. As the user turns his/her head, the content appears to move in relation to the user’s head accordingly. Sutherland is also known for his ‘sketchpad’ program which was among the first computer graphics programs to enable the production of 3D graphics.
Today’s smart devices (phones, tablets etc.) with touchscreens enable augmented reality or AR. They come complete with a camera and with sensors like accelerometers, GPS, Wi-Fi cards, and cellphone receivers. These working together enable the user to have information about locations appear to float in place relative actual position geographically. Nearby waypoints, points of interest can appear relative to your position simply by holding up your phone or tablet.
The Terminator’s AR view, “The Terminator”, 1981
AR was probably first brought to most people’s attention via the “Terminator” and “Robocop” movies of the 1980s and 1990s. The robot vision that superimposes text and graphics over a view of the real world from then on became synonymous with the view of the Cyborg.
Most portable game devices today come with a camera built-in. Devices like the 3DS and the Sony PS Vita provide owners with AR content in the form of cards that which when scanned present 3D or 2D content to the user. This content appears to float between the device and the card that is read by it. The content can be animated, and can even appear to interact with the world around it. For example virtual pets can appear to dance, respond to the hand of the user, and so on.
|The view through the sunglasses in “They Live” – A political form of Augmented Reality revealing the true ideological basis of capitalism and its advertising.
Real Time Data and AR
Data that is provided in real time by services can be viewed using AR real-time, such as flight paths and flight numbers of commercial aircraft in the airspace around you. These often show the heading, bearing, takeoff time and estimated landing time of planes in the air. There are planets and star-system based AR apps. These let you view the heavens, not just as they appear now, but millions of years ago, and millions of years hence. In enabling the presentation of multiple time frames simultaneously, such technologies as AR have been dubbed as ‘Atemporal’ by writers like Bruce Sterling. Sterling’s regular attendance and keynotes at the AWE conferences ensures a science fiction dimension to an event which though dominated by trade and business, is still art and technology innovation oriented to feel like ‘the early days’ of something.
Wearable computers and Augmented Reality offer ways to further examine the complex inter-relationship between real and virtual space. These technologies to a large extent invert the paradigm of representing space on a flat screen, instead enabling information appear to permeate and interact with physical space itself. The area of wearable computers ties in with this broad theme, as these customized small portable and ‘always-on’ computers enable the user to fully engage with the space around them and to annotate that space with information pertinent to it, and while doing so, communicate with other “Cyborgs” (as wearable computing aficionados refer to themselves) via wireless local area network cards.
There are inter-related themes which emerge when considering the possibilities of augmented reality – these have to do with the role played by media where digital content can be dynamic, reflecting changing aspects of the environment, or one’s relationship to it.
Given that dynamic information pertinent to a specific geo-spatial location can be made available to users of wearable computers, the role of this information resembles that of conventional fixed signage (ads, traffic signs, billboards etc)
SIGNAGE - symbols for navigation, information and instruction while online and moving through urban environments
MIGRATION - the constant moving from one place to another by the user of a wearable computer means that the act of moving itself can be the basis for certain types of digital media which are dynamic – an example is the windmill which slows down and speeds up depending on the rate of data traffic in Karrie Karahalios’ Diorama project. Although not a wearable computing project, this application was an example of basic augmented reality.
SPACE - that geospatial area occupied by the user as s/he undertakes daily tasks while using the wearable computer. Terrain traversed, land covered, urban space negotiated. There are several types of space which augmented reality gives rise to, that of i) the information, that of ii) the environment, and a third type of space which is the iii) conceptual intersection of the physical, the informational and how they work together in the mind of the user.
DYNAMIC MEDIA - media which update refresh and present themselves in the context of constantly shifting environmental conditions e.g. a directional sign which constantly points to the way point desired.
My 1998 film “Otherzone” enabled me to visually speculate on what augmented reality and wearable computers might look like several years from now. The media lab researchers had shown me what could be done with the (then) contemporary technology of shoebox sized wearables and oversized head mounted displays. It was not difficult to imagine in a few years the wearable computer disappearing into the structure of the display itself. The effect would be that of a pair of ‘intelligent sunglasses connected permanently to the ‘net – hence ‘net-spex’. This has come true with technologies such as Google Glass and Space Glasses.
Otherzone can be viewed here:
Otherzone’s “Netspex”, like today’s self-contained wearable computers, enabled the wearer to see and hear full multimedia as they move through urban space. In the case of the film’s villain, the Nam Melogue (played by performance artist Stelarc) has a monocular version of netspex which enables him to instantly view any camera in the city. The heroine’s netspex let her view videomail messages, and show her waypoints in the surrounding environment.
My film “Otherzone” 1998 – The Nam Melogue’s AR monocle let him access any camera in the city – total surveillance.
|Otherzone (1998) Kareen manipulates||3D icons|
Netspex as used in my film “Otherzone’ (1998) for navigation and locating waypoints in the environment, customized to the user’s interest and needs
I was first introduced to the area of wearable computers when I visited the MIT Media Lab in Boston in 1995. This visit was by invitation of the late Mr William J Mitchell (then Dean of Architecture and Urban Planning at MIT) who had visited Melbourne and RMIT that year promoting his book ‘City of Bits’.
I met Steven Mann and Thad Starner on this visit at the Software Agents Group section of the Media Lab. Both Mr Mann and Mr Starner had developed different approaches to the idea of wearing the computer on the body.
Wearable computers in the 1990s were small computers that draw power from rechargeable lithium-ion batteries. Wearables enable the delivery of real time information usually via head mounted displays (HUDs) which the wearer uses for data delivery instead of a monitor. Input to the computer is provided by a range of means, either via sensors on board the device itself, such as infrared scanners, cameras, or other sensors, user input such as voice commands, keyboard input or hand or body gesture input
Wearable computer users are not immersed in a virtual reality, rather information augments the world around them, and information appears superimposed over the real world.
Wearable computers and Augmented Reality offer ways to further examine the complex inter-relationship between real and virtual space. These technologies to a large extent invert the paradigm of representing space on a flat screen, instead enabling information appear to permeate and interact with physical space itself.
Head Mounted Displays (HDMs)
Head mounted displays (or HMDs) enable the user of a computer to see the information on that computer as an overlay superimposed over the view of the world around them. With tracking software it is possible to have the data ‘placed’ over areas of the room, or the broader environment. There are various ways that data can be placed – GPS data can be used, or in some cases, infrared scanning can be done using a device like the ‘leap motion’ system that scans the room with very fine resolution ‘painting’ process which then relays to the worn computer where objects should appear via the HMD. The computer user can manipulate objects using gesture control as if they were physical things, as is done in my film “Otherzone” where Kareen Hedding is shown moving objects like AR keys and locks and piggy banks around using her ‘Netspex’. In the film she is a proto Snowden, downloading information that will expose the truth behind her powerful employer, the Machines All Nations Corporation, the last tech company left on earth.
The Leap Motion system works in a similar way to the motion tracking used by Microsoft’s ‘Kinect’ games controller that turns body motion into a means to manipulate games characters.
A pioneer in the development of head mounted displays and wearable computers are Professor Steve Mann of the University of Toronto. Since the late 1970s Prof Mann has been developing wearable computers and head mounted displays. His earliest one comprised a crash helmet with a camcorder viewfinder taped to it, and radio antennae mounted to the top. It ‘talked’ to the computer via radio waves. Later models involved having the image reflect off a mirror into a single eye, and this innovation culminated in the patented notion of ‘Eyetap’ in which the eye itself acts as both the camera and part of the display projector system. Mann now tours with his collection of head mounted displays or HMDs to conferences where his contribution to the evolution of the technology is undisputed. Mann is also a great champion of the principle that AR and wearable computers should be used to advance personal liberty and to facilitate what he calls souseveillance – ‘seeing from below’ as opposed to ‘seeing from above’ or ‘surveillance’.
Mann has joined the META AR team who has developed a competitor to Google Glass via kickstarter using Epson HMD and Leap Motion 3D tracking.
Early Head Mounted Display from late 1970s worn by its inventor, Steve Mann
Dynamism and Flow
There are inter-related themes which emerge when considering the possibilities of augmented reality – these have to do with the role played by media where digital content can be dynamic, reflecting changing aspects of the environment, or one’s relationship to it. Given that dynamic information pertinent to a specific geo-spatial location can be made available to users of wearable computers, the role of this information resembles that of conventional fixed signage (ads, traffic signs, billboards etc) Signage and symbols for navigation, information and instruction while online and moving through urban environments
MIGRATION – the constant moving from one place to another by the user of a wearable computer means that the act of moving itself can be the basis for certain types of digital media which are dynamic.
Many contemporary films and videogames show scientists and specialists manipulating data in 3D as holographic fields of information. Floating, glowing holograms of the solar system, say as in PROMETHEUS. The terrains of Pandora in AVATAR, displayed for military and scientist alike. The head-up display is a product of the military fighter cockpit, and is installed in many recent model cars and passenger aircraft such as the Boeing 787. It combines the real world with information about that world simultaneously.
AR and Privacy
Augmented reality apps abound for the tsunami of Chinese-made touchscreen devices; sensor-studded, Wi-Fi enabled, the modern data user is attuned to her environment much like a pilot, or a sci-fi movie or game character. After thirty years of AR in pop culture, from ROBOCOP to TERMINATOR to HALO to the windows-within-windows of every GUI you ever used people are attuned today more than ever to superimpose data over their field of view.
Google Glass – The Mainstreaming of AR
Bruce Sterling dons Google Glass: “This is very ‘80s” he is reported to have said.
In 2014 the Google Glass AR wearable system will go on sale. Glass is a head mounted wearable computer with a built-in see-through display and 720 X 420 video camera & microphone. The tech lead on the project was Thad Starner whom I worked with at the MIT Media Lab in 1998 when I was also working with Karrie Karahalios on the DIOMARA project.
Glass has has wi-fi, GPS, but the system is not strictly Augmented Reality, Google Glass is better described as “Annotated Reality” in that what you see are pop-up notes about what is going on around you rather than objects and items that supplant or appear to stand in for real objects. Also the Google system is closed – it can only work with Google’s own proprietary search engine and other tools related to its artificial intelligence-like data gathering system. Glass bears a striking similarity to Steve Mann’s “eyetap” system from 1999, particularly the concept of the display being reflected in a 45% prism to enable the eye to see directly through it.
In his keynote address at the Augmented World Expo this year, Mann discussed the similarities onstage:
In the talk Mann makes a point which begs the question of meaningful AR – ‘if glass cannot help you see better, what good is it?” – I interpret this as a direct challenge to Google Glass, which far from helping individuals ‘see’ at all, on the contrary, obscures their view with advertising driven annotations that serve to simply bring obtrusive social media content physically closer to a user who now does not have to stop all other activity to perform them with a smart device with a touchscreen. True AR seeks to augment not just vision, but the entirety of experience itself. Reality itself. True AR is a form of electronically mediated Situationism.
With partner Molly Hankwitz (together we form Archimedia) I built a wearable computer using a raspberry Pi computer and my old Virtua-IO glasses from the late 1990s that works fine for most basic computing needs but it has been field tested for Situationist-style drifting. Notes are taken using it while walking through urban areas.
Archimedia’s Raspberry Pi based wearable computer ,the “Debord”. Total cost, about $200. Designed to enable the user in her drifts around the city. When Bruce Sterling saw it, he took a picture. CC Archimedia 2013
There are more DIY wearable computing solutions on offer, and at the low-end the humble Raspberry Pi represents a keen entry-level $35 very portable computer for those wanting to build a low cost computer to carry around. As for head mounted displays, these are bound to become cheaper as time goes on, but the META system, now named SPACEGLASSES which is based on an EPSON head mounted display is the sort of rig that most people can afford, costing several hundred dollars. Not so Google Glass which sells for up to $1,500 a set. No doubt in several years, as with Walkmans, MP3 players, and other consumer electronics devices that started out closed, expensive and a high-end fashion item, prices will fall and the innovation can really start to take place.
Controversy has always surrounded wearable computers. In general terms these revolve around the conflict between on the one hand the empiricist argument that the devices are part of uniforms, and extensions of institutional power and authority. The user is not supposed to know how they work, only that they work. The alternative view is that the devices are customizable, individualized, made to suit the specific needs of the user, who then connects with others of like mind and technical sensibility to form collectives. In the mid -1990s these conflicts played themselves out in most hubs of power, wherever large contracts were to be argued over. As in the film “Brainstorm” the Pentagon and the government had its own notion of wearable computing should be for – a device of uniform specification, issued to a mass, whose use of it would render the content that of an aggregate. This is the model used for Google Glass, which is a more efficient way to suck up personal videos, emails etc by a population who neither knows nor cares how its data is made use of. The other model is a customizable system that can be made to adapt to the needs of a user base who decides for itself not only how the devices are used, but how the data generated by the devices is to be used.
Steve Mann has dedicated his career to the principle of personal technology as ‘an architecture of one’ Mann believes that wearable computing and augmented reality should be a force for good in the world, a force for social justice and a force to benefit society. Not for profit alone. Not for surveillance. Not for military and control. Not for policing. Not to better further the narrow interests of property, ownership, and privatization.
It is thus incumbent upon us all to develop the new technologies with a mind to the great power of the innovations they imply and above all else, the important social relations they are sure to bring forth.
Picture I took of Steve Mann at the MIT Media Lab when I interviewed him in 1996 Copyright David Cox
Myself with Steve Mann when I interviewed him at the AWE2013 Conference, Santa Clara Copyright David Cox
SPACEGLASSES offers true AR in that the graphics are true 3D, and 3D graphics creation is part of the long term idea of the project. The use of 3D tracking (similar to the Kinect system) means that hand gestures can be used for manipulating objects that appear in the field of view as in films like “Iron Man” and “Otherzone”.
Meta AR (AKA SpaceGlasses) – a true AR developer kit that enables hands-on customization and unlike Google Glass, true AR, not simply annotation of information over your view of the world. Plus it has Steve Mann onboard, which means that it has some theoretical chops behind it is probably not only driven by advertising and profit but a genuine interest in the potential social and cultural potential of the medium.
Archimedia is a research collective based in San Francisco. Its members include David Cox and Molly Hankwitz.
Steve Mann personal website:
David Cox Research Page
META AR renamed “SPACE GLASSES” PROMO
Game Draw on Meta
AR and ART
Augmented (hyper)Reality: Domestic Robocop
AR in Cars
FROM WAGNER TO VR
Augmented Reality Through Wearable Computing
Steve Mann Keynote, AWE2013
Bruce Sterling Keynote, AWE2013:
Will Wright Keynote, AWE2013:
AR & CINEMA
Steve Mann Interview
Pranav Mistry: The thrilling potential of SixthSense technology