Xbox 360 Live Vision Camera Technology

[Spartan_X]

An Update on Microsoft's Xbox 360 Live Vision Camera Technology

At the 2006 Gamefest, Microsoft's Game Technology Conference that took place in Seattle August 14 and 15, a number of middleware developers - as well as Microsoft itself - showed off various game-related technological advances for both Windows and the Xbox 360 platform. The most infamous new development, of course, was Microsoft's announcement of user-created Xbox 360 game content, via the GarageGames-developed XNA Game Studio Express.

XNA Game Studio Express will "democratize game development" by allowing users of all caliber and experience to create original game content using an intuitive drag-and-drop interface, along with managed C# code. Games can be created and shared for free on the Windows platform, though content sharing and downloading on the Xbox 360 console - according to Microsoft's online FAQ - may only occur if all users involved are part of Microsoft's "creator's club," which requires a $99 annual membership fee.

The news is groundbreaking, for certain, but has arguably overshadowed other Xbox 360 advancements closer to home; namely, the Live Vision camera accessory, which is set for release next month.

The Xbox Live Vision camera

In the following pages, Gamasutra explores the Live Vision's features, its upcoming software, and a number of middleware tools, including depth perception capability from GestureTek that could threaten Nintendo's Wii controller's unique charm.

Xbox Live Options

The Live Vision camera was officially announced last May, as an accessory for Xbox Live users to interact with each other via video conferencing in games such as Uno on Live Arcade, as well as facemapping via technology from DigiMask, which we will go over later in this article.

At the Expo portion of Gamefest, Xbox producer Jeff Stone showed us some of the default functions of Live Vision, the software for which is already installed and running on any Xbox 360s that underwent the spring software update.

Video Chat via Xbox Live

As first demonstrated at E3 2006, Live Vision enables a simple video chat function for users of Xbox Live. As seen in the above photograph, each user can enable a number of video filters for identity protection and style. The demonstration we saw showed no signs of lag, even with filters applied. The conferencing took place between two 360s that were literally sitting next to each other; a quick demonstration of a connecting stream to Stone's office, however, laid to rest any doubs of its functionality.

As expected, users can also email photos to each other, applying additional filter effects where desired, including a pseudo "night vision," contrast manipulation, stretch and distortion effects, and more. Photo emails can be sent to any Live user, though opening a message from someone outside of a user's friends list will provide a prompt to accept or decline a photograph, to prevent any unwanted content.

In addition to video conferencing and email photos, Live Vision can also be used to create a new profile thumbnail picture, for use in a user's gamertag. Like both the video conferencing and photo options, gamertag photos have a variety of filter options.

Unlike the other two options, however, gamertag profile pictures are viewable only by individuals on a user's friends list. For the rest, the user's default gamertag picture will be displayed.

Explanations for this discrepency in access varied from the obvious - prevention of copyrighted and inappropriate content - to the capitalistic, with one Gamefest panel in particular reminding developers that user pics "are a revenue stream, rather than a promotional opportunity."

TotemBall

A still from TotemBall

Also on display was TotemBall, the first gesture-based game to be shown for Live Vision. As previously reported on Gamasutra, TotemBall was developed by UK-based Strange Flavour and Freeverse, both of whom are mainly known for their Macintosh titles, including previous camera and gesture-based titles such as ToySight.

Xbox Producer Jeff Stone demonstrates TotemBall

Other titles confirmed to utilize Live Vision include the Live Arcade title Uno, which has already shipped with support included (which, obviously, will only work once Live Vision is attached), and two games that use automatic facemapping technology: World Series of Poker from Activision, and UbiSoft's Rainbow Six Vegas.

Next, we'll examine this facemapping technology, which was developed by Digimask, and take it for a test run on Xbox 360 hardware.

DigiMask's Face Renderer

Digimask's face rendering technology has been used in a number of applications, including cell phone greetings, a promotional campaign for the Hulk feature film, television programs, and, yes, video games.

In fact, Digimask's middleware technology has been utilized in a number of PlayStation 2 titles, using Sony's EyeToy accessory, under the name of "Eyetoy Cameo." Published titles include EyeToy: Play 2 (Sony, 2004), Karaoke Party Revolution (Konami), games in the Tony Hawk series, and various sports franchises, released by both Sony and Electronic Arts.

At the Gamefest Expo, we tested out this technology - which will be utilized in at least two upcoming titles - running on Xbox 360 hardware. The following photographs show internal demonstration software, and are not indicative of what will be seen in-game.

Digimask's technology renders a user's face using a front and profile shot. The software records a short film clip, and users are able to manipulate it frame-by-frame to choose the perfect shot to use for the renderer.

In the demo, we were then given the option to choose a hairstyle, along with eyewear (and choice of frame colors). Hair color is automatically detected. For the sake of the demo - and because this author's hair is difficult to render properly - we went with the simple addition of a baseball cap.

The model looks fairly accurate, and at a much higher resolution than we've seen using the EyeToy. The model looks around absently, blinks its eyes, and can be manipulated into showing emotion, as seen below.

Coming up next, we'll explore new technology from GestureTek, which is making strong headway into accurately detecting both motion and depth through the Live Vision camera in ways previously unseen in commercial products.

GestureTek

Sunnyvale, California-based GestureTek is the patent creator and holder for their Video Gesture Control technology which, following a licensed use to Sony's EyeToy in 2005, was announced as being licensed for Xbox 360 Live Vision development last May.

The patent, U.S. #5-534917, covers all "video-based interactive gesture control of computer processes." In other words, any interactive technology with a user being displayed on-screen has licensed GestureTek's technology.

GestureGame.exe

A simple demonstration program, with the file name of GestureGame.exe, shipped with Live Vision's development kit in late July (though it has been available internally at Microsoft since February). The demonstration, with source included, offers a simple level of interactivity, allowing the user to "touch" and interact with floating objects on the screen. The user is not displayed; rather, only motion is detected. The white splotch on the above screenshot is actually a moving hand.

GestureTek also displayed face-tracking technology, as seen above. The program very quickly recognizes a user's face, detecting facial structure and features, and is able to track it.

As seen in the photograph, a yellow box surrounds this user's face. He is able to move around rapidly, and the box stays with him, moving fluidly in real time. When another user steps on screen, the box still maintains its link with the original user. When the original user leaves, the tracker - looking for the closest match to the original face - moves on to the newcomer. If the original user comes back, the box returns. It's very smart, but unfortunately is not included in the Live Vision development kit, and must be licensed directly from GestureTek.

Building on its face tracking technology, GestureTek has also developed a solution for tracking individual, non-facial objects on-screen, such as handheld peripherals. This is done in a remarkably simple manner. The tracker looks at an object's color and shape, and is able capture and track it in a number of ways.

Objects can be moved around rapidly, such as in the face tracker technology. They can be rotated and moved with relative ease. They can also, by way of tracking an objects size, be detected spatially: that is, theoretically, a peripheral's distance from the screen can be tracked in real-time, which is more than mildly reminiscent of Nintendo's Wii controller.

The first demo shown looks a bit like the GestureTek-developed Wizard's Cavern. A peripheral - in this case, a foam "magic wand" with a bright blue bulb at the end- is used to trace lines and patterns on-screen. The program recognizes the accuracy of the pattern, and the "sparkles" that emanate on screen sense speed, so that a slow movement can cause a thicker line.

The most impressive of GestureTek's technology was its early work at depth perception. As seen here, an object - in this case, a foam sword with brightly colored features - is tracked in real time, spatially. The tracking boxes shrink as the object is moved away from the screen, and grow as it comes closer.

It might be tempting to perceive this technology as a threat to Nintendo's Wii controller, but it should be noted that GestureTek itself isn't banking on this, considering its limitations.

"It's just a demo," shrugged GestureTek CTO Francis MacDougall when pressed, admitting that he doesn't see brightly-colored peripherals being popular among publishers.

The Xbox 360 Live Vision camera ships September 19 in the United States, along with TotemBall, the first and thus far only game announced with gestural controls. The accessory is compatible with Microsoft Game Studios' already-shipped Uno as a video conference mechanism, and according to Jeff Stone, "seven or so casual games" will be shipping this fall with video chat options. Confirmed titles using DigiMask's face mapping technology include Activision's World Series of Poker and UbiSoft's Rainbow Six Vegas. Development kits are available from Microsoft as of July.

News source: Gamasutra

[Andrew Lyle Global Moderator]

sorry i didn't read a lot of it, just a quick in between, but i remember when Perfect Dark wanted to add this technology into it's multiplayer game, it didn't work, and they end up scrapping the project. i like this camera, but i have no use for it

[mattmatik]

That's pretty cool, especially the Texas Hold 'Em/Vegas use of the camera. I'll probably end up getting one, but I really would like for video conferencing to include more than just 2 users.

[dblaze]

That's pretty cool, especially the Texas Hold 'Em/Vegas use of the camera. I'll probably end up getting one, but I really would like for video conferencing to include more than just 2 users.

My thoughts exactly

[PL_ Veteran]

Nice.

So, we could start seeing ports of Wii games :shifty:

[The Teej]

Nice.

So, we could start seeing ports of Wii games :shifty:

Yeah but it would require users to pay for an additional peripheral, something Devs would have to think about :p

[icecaveman]

TotemBall sure has amazing graphics. Why is that guy with a soucage in his mouth?

[PL_ Veteran]

TotemBall sure has amazing graphics. Why is that guy with a soucage in his mouth?

It's a tongue :p

[magik]

TotemBall sure has amazing graphics. Why is that guy with a soucage in his mouth?

It does look like a sausage... certainly the weirdest looking tongue I have ever seen.

[dnast]

That face-mapping technology is pretty impressive if you ignore the "sausage", lol.

The only way I might get the camera is if they get the "head gestures" right. I'd love to be able to quickly glance (slightly, of course) in either direction and be able to look 90 degrees to the left or right in a first-person game.

[TheNay]

Looks cool, funny how the PS3 will have the same thing. Not sure about Wii tho.

Multi-webcam chat would be really neat, I know many who wanted it for WLM8 and it never made it.

Wonder if MS will support it for the 360.

[NightmarE D]

That digimax facial stuff is amazing looking. Pretty cool that we can now do this ourselves in the comfort of our homes. I know we've been able to do it for years now on the PC for games like Quake, but that was pretty easy to take a picture of yourself and slap it over the face of another skin and they never looked that great.

And don't any of you sony fanboys come in here saying the PS2 or PS3 is being ripped off by Microsoft either. Getting tired of seeing that being said. This kind of stuff has been on PC's for years now but you people always ignore that.