Kinect hacking may lead to Minority Report style UI control

Minority Report style interface control could be in(on) the cards for Microsoft's future operating systems. The full capabilities of Kinect are only just being discovered, excitingly, hackers have taken to the device like ducks to water and new capability demos are already hitting the scene.

Blogger Pradeep Viswav seems to have as specific interest in just what Kinect is capable of, and has compiled quite a few videos that demonstrate Kinect not only as a PC control device, but also expanding on the 3D view encountered recently. Virtual "touch" could be the next wave in UI interaction. This demo shows Windows 7 being controlled by Kinect.

Emily Gobeille and Theo Watson have been experimenting with Kinect motion capture. They have released a demo utilizing depth in combination with the libfreenect ofxKinect drivers. The demo enables a virtual bird to be controlled by the movements of your arm, even going as so far as monitoring joints.

These types of hacks are very new and at the moment are just taking baby steps towards being truly useful. It's only a matter of time before hackers develop an implementation that could be integrated into a user environment. The earlier demo showing 3D room manipulation could also be enhanced with more cameras to create a full, manipulatable view of a location.

With Microsoft's recent purchase of Canesta, the company obviously has further plans for motion control than just gaming. This may be an integral part of a new operating system, focusing on a new control method.

Report a problem with article
Previous Story

Windows 8 will be too little, too late

Next Story

Neowin Giveaway: 3x Denon AH-D510R - Winners Announced

39 Comments

Commenting is disabled on this article.

This is pretty cool, and Kinect not having been out that long already.
I bet this will boost Microsofts sales even more if people work out ways to use this for other things.

Loving the idea of the interface interaction in the first video

Don't understand this fascination with Minority Report UI.

Yeah, it was a cool sight when that movie came out a long time ago. But the guy was working mostly with video, had to wear special gloves/equipment and it was cumbersome even for that task.

Plus we're not all Tom Cruise. It won't look cool when you're doing it.

Does anyone even think about what it would actually be like to use a UI like this for anything non-trivial?

MS should embrace this fast and make some money from it. Knowing MS they will fight it for years until its boring, then it will be released as a selling point for Windows 9.

Orange Battery said,
MS should embrace this fast and make some money from it. Knowing MS they will fight it for years until its boring, then it will be released as a selling point for Windows 9.

Nope, I reckon MS are allowing this to happen as they did with alleged leaks for Windows 7 to get the hype and bugs worked out of it in order to push it into Windows 8, I can see this being released with new Windows 8 laptops with Kinect sensors built in or an additional periphery for those who aren't willing to shell out

I never understood why people liked the Minority Report type interface so much. It would get really tiring fast holding your arms out constantly. It's cool and all, but not the future of interface..

Really advanced stuff. The ability to rotate the camera in a 3d space if fantastic.

I wonder if you had a 2nd kinect on the opposite side of the room if you can make true 3d depth sensory environments.

ccoltmanm said,
Really advanced stuff. The ability to rotate the camera in a 3d space if fantastic.

I wonder if you had a 2nd kinect on the opposite side of the room if you can make true 3d depth sensory environments.


you can (but I guess 3 kinect's would be better), it's just a question of software solution

I want it built in to my TV so I can skip channels, turn up the volume, pull up the menu etc by waving my hand etc.

swipe to the left to go back a channel, swipe to the right go up a channel, swipe up for volume up, down for volume down etc.

joemailey said,
I want it built in to my TV so I can skip channels, turn up the volume, pull up the menu etc by waving my hand etc.

swipe to the left to go back a channel, swipe to the right go up a channel, swipe up for volume up, down for volume down etc.

I am working on a media player that will have these features. The next step will be to integrate this into Windows Media Center. MS Should do this directly.

Just the beginning but there will need to be a Kinect 2.0 or whatever before it becomes great. Baby steps are good but better specs in the future will probably make it useful.

They already have the tech for this. They released the 1.0 level version cameras purely to hit the low consumer price point. But higher resolution cameras, etc. already work with Kinect for even more amazing things.

superconductive said,
is it just me or does there seem to be a quite noticeable lag between the camera sensor input and the resultant actions in those videos?

Not that I can see.

superconductive said,
is it just me or does there seem to be a quite noticeable lag between the camera sensor input and the resultant actions in those videos?

Yes there is a noticeable lag but that's mostly due to the programs not being fully optimized.

Minimoose said,
The bird one had a lot of lag, but the other ones we're reasonably fast given the time frame they were built in.

Yeah, only in the bird one, and it's normal. Kinect is intended for light rooms

Xilo said,
Nothing will ever replace the mouse for desktop and laptop use.

Of course not. The idea here is not to replace, but to complement.

If I could play, pause, move forward, backwards, control the volume, etc in a movie I'm watching just with gestures and without having to find a damned remote, I'd think that's a pretty good starting point.

_dandy_ said,

Of course not. The idea here is not to replace, but to complement.

If I could play, pause, move forward, backwards, control the volume, etc in a movie I'm watching just with gestures and without having to find a damned remote, I'd think that's a pretty good starting point.

Not for us maybe, the people who have grown up with it, but I can see my two boys, who can hop between OS/ quite easily, this would be no problem for them. Although I really am impressed with the capabilities, I see so many movie interfaces becoming a reality from these hacks, in there we have the obligatory Minority Report, but there is also the VDU used in Avatar as well as Jarvis from Iron Man. Limitless potential

I used to do some development for PS3 Eye which I thought was an amazing device the program for. I would happily drop my PS3 Eye for Kinect. Depth perception is a HUGE yes for me.

I have a feeling Microsoft knew that the kinect would be hacked and used for PC integration.
If they did then it's smart on their part. Having other people you don't have to pay to develop for them

Wow, this is so interesting. Who could have predicted that the Kinect not only would be a sales success, but also entice this viral social exploration? Microsoft really packed some good tech into this little thing, but I'm especially impressed at the last video about object recognition; that is really freaky how fast it recognizes them. I've seen my Kinect recognize me when I stand on the gaming area, but for some reason that video makes me see very soon the day when computers will talk and interact with us like people.

Charles Keledjian said,
Wow, this is so interesting. Who could have predicted that the Kinect not only would be a sales success, but also entice this viral social exploration? Microsoft really packed some good tech into this little thing, but I'm especially impressed at the last video about object recognition; that is really freaky how fast it recognizes them. I've seen my Kinect recognize me when I stand on the gaming area, but for some reason that video makes me see very soon the day when computers will talk and interact with us like people.

Those were some of the first things demo'd by Microsoft...a woman drew something on a sheet of paper and the kid on screen recognized it. She then "handed" him the sheet of paper and he "took it".
Jeez this sounds so wrong, all the virtual interaction...do I really say she "handed it and he took it"?

This is pretty sweet and I totally agree, this needs to be released by M$ to the public, heck they should sell a little program that has all the drivers built in, that way they can make a buck or too.

Gaurav Agrawal said,
MS should release Kinect SDK for non-gaming purposes....

Lol, Microsoft is undoubtedly developing this type of interface for a future OS, however many years away it may be. Until then, I don't see them releasing a Kinect SDK for programmers.

St4rOcean said,

Lol, Microsoft is undoubtedly developing this type of interface for a future OS, however many years away it may be. Until then, I don't see them releasing a Kinect SDK for programmers.

They don't need to release it, as you can see people are already making stuff. By the time Microsoft makes something useful there will already be a bunch of alternative or even better programs that do the same thing.

De.Bug said,
They don't need to release it, as you can see people are already making stuff. By the time Microsoft makes something useful there will already be a bunch of alternative or even better programs that do the same thing.

The point of MS using it is that the code will be easier to use with .NET (but the hax will stay out there for other platforms to use it)