Last week I had a showing at Equilibrium, the Human Mashup. The show’s goal was to ask "How can art and technology and the synergy of the two address humanity’s next evolutionary step?". Being a staunch singularitarian this was right up, or is it down, my proverbial alley. I built an application that allows a user to take a self portrait photograph and then draw themselves on a wall simply by using and moving their hand. The premise was to use the human as the sole interface to draw themselves, exploring the convergence of man and technology. A one day only installation, I didn’t have to high of hopes for it’s exposure. On the flip side, this being my first showing ever I was like a greedy 8 year old on Christmas morning, and extremely anxious to see how people would interact and respond to the installation. The methodology of controlling an interface without the normal peripherals such as a mouse, keyboard, touch screen, kiosk, or otherwise was foreign to all that attended. What transpired blew me away. The attendance, response and exposure far exceeded my hopes. I barely had to explain at all to users how to run the application. People just stepped right up to the installation and began to interact with it. They were actually learning on their own while performing the drawing. People were actually converging with the technology. Furthermore, those that were done performing their drawing became evangelists, telling those who followed them their idea of how best to use the application. At that moment it became cemented in my mind that I had succeeded in my experiment. People had invested themselves in the piece to the point of bringing out their own inner artist. Several times I stepped in to ‘coach’ people on how to get the most out of the motion detection while drawing their portrait. The response I got was unexpected. People took personal offense that I was pushing my artistic ideas on their own drawing. Again, success. I was also sent to the floor several times by attendees who said they had come to the event specifically to see my work which they had been following online. There were even several people who mentioned that they had hooked my work up to projectors at parties they were holding. Wha?! Seriously, amazed.
I’d like to thank several people for the night. Steve Benoit for all his help with testing the application, videography and film editing of the event. The result is the shweet movie posted above. Rachel Kjack and Adrienne Fritze from SAO and Working Artists Online respectively for their organization and inviting me to submit my work. All that attended, I met some inspirational folk. And finally, my wife Lisa for her unending support and diggin’ me even when I have late night dreaded ‘code eyes’.
In continuing exploration into motion tracking using flash and a webcam I’ve created an application that enables you to track multiple objects based on color. The holy grail of the application is of course to track multiple objects without specifying color. However, this is a good step forward and actually opens up many other avenues for use. Color, after all, is ubiquitous.
The truly useful part of the code is getColorBoundsRect. If you’re unfamiliar with this function, it determines a rectangular region that either fully encloses all the pixels of a specified color within a BitmapData object, or fully encloses all pixels that do not include the color. The function returns a rectangle around the color area. You can then reference this rectangles x and y respectively.
var areaColor:Rectangle = SomeBitmapData.getColorBoundsRect( 0xFFFFFFFF, _colorArray[i].color, true );
The function is rather finicky about the depth of where the object is. Since it’s only searching for a certain RGB value it tends to lose it if the lighting or the object gets too small in the camera’s depth of field. To overcome this you could search for similar colors. Soulwire has written a sweet Color utility class that can aid you in this endeavor. However, searching for a range of colors will also come with it’s challenges, as the more range of color you search on the tougher it will be to pinpoint the exact object you want to track.
You have to admire a job that allows you to make a bumble-bee/owl (a bumblowl?) fly around a mural merely by waving your finger… all while drinking beer.
This is an elaboration on my actionscript webcam motion detection experiments, built as an installation for the Inverge party hosted by North. The idea was to take motion detection and combine it with art in the form of digital drawing and 2d space in the form of a wall mural. The concept is to allow human interaction to affect physical space, freeing oneself from all traditional user interface. The application works by constantly monitoring a webcam, taking snapshots of each frame every millisecond. The pixels in the current frame are compared to pixels in the previous. The difference in brightness within each section determines whether or not motion had occurred in this area of the screen. If motion is detected the coordinates of a rectangular grid around said motion is recorded and the owl is ordered to go to that point. The line drawings begin and follow the owl recording a ‘flight path’. A mask is drawn to prevent the owl and its drawing trail from going in front of some of the trees giving the mural a sense of dimension.
Thanks to the great Eric Natzke for the drawing inspiration and open source, and NORTH for being forward thinking enough to encourage digital experimentation.
Expounding on my AS3 webcam motion detection experiments I’ve designed and built a prototype for an installation that allows a person to draw on a wall using their limbs.
The concept is to allow human interaction to affect physical space, freeing oneself from all traditional user interface. The application works by constantly monitoring a webcam, taking snapshots of each frame. Those frames are then constantly drawn to a bitmap data object. Then each pixel in each frame is monitored by comparing the pixels in the current frame to the pixels in the previous one. The difference in brightness within each section determines whether or not motion had occurred in this area of the screen. If motion is detected the coordinates of a rectangular grid around said motion is recorded and the drawing begins. To slow down the drawing I draw at the x and y coordinates of the white ball which is tweened with easing rather than drawing directly where there are points of motion.
Thanks to steve for filming and editing this video, asluv for the laser drawing class and NORTH for being forward thinking enough to encourage digital experimentation.
You can try the experiment small scale and online here: AS3 webcam drawing
The concept is to create a new type of user interface rebuking the typical peripherals of the keyboard and mouse. Instead, a user would merely stand in front of a wall and control the interface with their movements. I’ve seen other attempts at doing this. See Microsoft Surface, iphone, this thing et. al. All cool for sure, but all done using touch screens. Touch screens are so 2006. Even Tom Cruise used gloves. Pfft. The idea is to free oneself from all physical interface. To use human gestures.
I haven’t tried it properly installed and projected on a wall yet.
Actionscript 3 application that performs motion tracking via webcam.
Excuse the choppy film but the movie was created by capturing the screen at 10fps.
The application works by constantly monitoring a webcam, taking snapshots of each frame. Those frames are then constantly drawn to a bitmap data object. I can then compare each pixel in each frame by comparing the pixels in the current frame to the pixels in the previous one. The difference in brightness within each segment (from one calculation to the next) determines whether or not motion had occurred in this area of the screen. If motion is detected the coordinates of a rectangular grid around said motion is recorded. To track whether or not the “ball” has been effected I compare the current x and y coordinates of the rectangle to the coordinates of the ball. The direction of the ball is determined by again comparing the points of the motion area to the “ball” point. ie: if the x coordinate of the motion area of the last frame is less than the x coordinate of the ball I tell the ball to go to the right. Just apply that logic to all possible angles and you get the physics of the ball.
A simple fun game, but you could easily make the mental leap to imagine how this logic could be applied to a “minority report” type interface or used as a large installation in an art museum or public space.
