Kinect optical camouflage hack makes you invisible

 

Remember Predator, that classic 80's movie starring Arnold Schwarzenegger and Carl Weathers? The duo led a team of commandos who were fighting a semi-invisible enemy in the jungle. This semi-visible enemy was heralded back then as a cutting edge special effect and won the movie an Academy Awards nomination for Best Visual Effect in 1988. Fast forward 20+ years and now you can duplicate this effect with some programming knowledge and a Microsoft Kinect. 

The Predator-like invisibility hack was developed by Takayuki Fukatsu and showcased in a recent Popsci article. Fukatsu developed the hack using OpenFrameworks, an open source C++ toolkit designed for "creative coding" projects such as this. Unfortunately for folks looking to replicate this effect, Fukatsu provides no details on how he was able to achieve this effect, but we can always speculate. The Kinect is adept at distinguishing between a body and the background and Fukatsu presumably leverages this ability in this hack. The hack most likely takes an image of the background and superimposes it on your body in real time. For those that enjoy YouTube videos of exceedingly long lengths, Fukatsu shows off this effect in a seven minute long video which is embedded below. If you do not have seven minutes to spare, you can skip ahead to 1:30 in the video and watch the next minute or so to get the idea behind the hack. 

While this invisibility hack is more fun than practical, it showcases some of the creative and exciting things people are doing with the Microsoft Kinect. More than merely a gaming device, folks have used the Kinect as a NES controller, a means to wield an on-screen lightsaber using a stick, a hands-free controller for Windows 7, a web browser, and even a robot! With 2.5 million Kinect devices in the wild, it will be interesting to see what new hacks and innovations are coming up next.


Report a problem with article
Previous Story

Microsoft to use AMD Fusion II in next gen Xbox

Next Story

Google sandboxes Flash Player in latest Chrome builds

51 Comments

Commenting is disabled on this article.

Does any one notice that the cheesy music is coming from they keyboard on the left. You see can see the keys move..

Shiranui said,
Crap.

Get someone else to walk behind him while he is moving and I'll be impressed.

why do YOU! need to spout "crap? the same thing would happen to another person if they were
in the same room as him, so back in your hole Troll.

Shiranui said,

Get someone else to walk behind him while he is moving and I'll be impressed.

That should be possible given what I've seen of the hardware. It's able to tell people apart.
After watching the video, I think he was just going after that Predator effect--even adding a wave effect to it?

Actually I take that back. Being able to tell that there are two people on screen doesn't help at all.

Someone already pointed out that he is using a photo as a backdrop. This is only using the depth information then. The keys moving* may just be a weird anomaly with the video. (* posted below)

Navan said,
wait, so the guy is invisible on the screen? big deal. =/
Yeah I have been invisible for many years and yet people didn't say that was impressive.

-T- said,
At around 2:22 you see it glitch and it gives a little reveal on how this may have been done

Hmm... looks like he is probably using two Kinects simultaneously.

-T- said,
At around 2:22 you see it glitch and it gives a little reveal on how this may have been done
I saw that, It's not even the background there. So technically its not camouflaging him with the background its just taking a picture and putting it on top of him.

Wir3Tap said,
I saw that, It's not even the background there. So technically its not camouflaging him with the background its just taking a picture and putting it on top of him.

That's also why he can walk through walls at 2:40.

-T- said,
At around 2:22 you see it glitch and it gives a little reveal on how this may have been done

i could have guessed without watching the video... the background is entirely static, and the actual video was made by totally relying on the depth sensor

basically he took a picture to make it look like his living room and pass the video off as something legit, he couldve used any picture and it wouldve worked the same way, but people's reactions would have been different...

iKenndac said,
Can we stop calling cool image effects "hacks" please?

Thanks.

The kinect was not supposed to be used for this type of activity...thus the 'hack'

bdsams said,

The kinect was not supposed to be used for this type of activity...thus the 'hack'

So is everything that Kinect is used for on PCs going to be called a hack?

bdsams said,

The kinect was not supposed to be used for this type of activity...thus the 'hack'

Re-purposing does not equal hacking

A hacker generally means either someone who breaks into computers and computer networks OR who heavily modifies the software or hardware of one's own computer system/devices. Neither of which is happening here.

It's a camera... being a camera, and a depth sensor.... being a depth sensor. Kinect is working perfectly fine, doing it's intended job of capturing data. And it's sending it's data down the USB cable as per normal operation None of Kinect's operations or software has been modified in any way - PC hobbyists are simply using the data it's sending down, no hacking of any sort

Edited by ~Johnny, Dec 2 2010, 5:07pm :

iKenndac said,
Can we stop calling cool image effects "hacks" please?

Thanks.

I agree... That's why Microsoft freaked out at first, because a hack (tends to) means hardware modifications, which is what Microsoft didn't want. They said nothing about hooking it up to a computer, just you aren't allowed to screw with the internals.

Though I don't know what else to call this... After all, they aren't even really repurposing the Kinect, other than using it on a PC than an Xbox, but I would think you could do everything they are doing on the PC on the Xbox.

Mr aldo said

Though I don't know what else to call this... After all, they aren't even really repurposing the Kinect, other than using it on a PC than an Xbox, but I would think you could do everything they are doing on the PC on the Xbox.

You can. Infact nearly all of the stuff in the demos (including moving around the 3D camera image) has already been done in games and software on the 360 (excluding this so far... as it doesn't really have much ingame application ).

Something like "Kinect Hobbyiest developer creates invisible camouflage effect" would be better than just including hack all the time @__@

iKenndac said,
Can we stop calling cool image effects "hacks" please?

Thanks.



Hacking (English verb to hack, singular noun a hack) refers to the re-configuring or re-programming of a system to function in ways not facilitated by the owner, administrator, or designer

Was the original configuration for Kinect made to be able to be used as a webcam making the user it is pointed at invisible?

No? Then shut the **** up

iKenndac said,
Can we stop calling cool image effects "hacks" please?

Thanks.

Stop the QQ please... if this was an iHack ... you would be wow thats cool.

bdsams said,

The kinect was not supposed to be used for this type of activity...thus the 'hack'

The first thing to talk about is, Kinect was not actually hacked. Hacking would mean that someone got to our algorithms that sit inside of the Xbox and was able to actually use them, which hasn't happened. Or, it means that you put a device between the sensor and the Xbox for means of cheating, which also has not happened. That's what we call hacking, and that's what we have put a ton of work and effort to make sure doesn't actually occur. What has happened is someone wrote an open-source driver for PCs that essentially opens the USB connection, which we didn't protect, by design, and reads the inputs from the sensor. The sensor, again, as I talked earlier, has eyes and ears, and that's a whole bunch of noise that someone needs to take and turn into signal.

- Microsoft's Alex Kipman speaking formally on NPR's Science Friday

aftas said,

Was the original configuration for Kinect made to be able to be used as a webcam making the user it is pointed at invisible?

No? Then shut the **** up

Seriously? That's your argument? For real? It's very flawed (: With enough smarts, you could do this any normal webcam. Would that be a hack? I mean that webcam wasn't made for camouflage people either? No, of course it's not a hack! The webcam is MADE for making video data, and that's what it does. It does not care what you do with the video data, it's just giving it to you to do whatever you want to it, minding it's own business XD The same principal applies here.

Likewise with the KINECT. The KINECT is not doing any of this camouflage trickery. The KINECT is doing it's job, totally normally. The KINECT is not hacked. The KINECT is sending data as it normally does. It's webcam and it's depth sensor are getting data, and sending it down the USB cable, as it is designed to do. We are using that webcam/sensor data to do our own thing, independent of any of Kinect's internal workings, and the Kinects operation is in no way effected. They're collecting data Kinect puts out, UNHACKED, and then forwarding that to another program which plays around with it and does this. It's a smart program maybe, but there is no hackery?

~Johnny said,

Seriously? That's your argument? For real? It's very flawed (: With enough smarts, you could do this any normal webcam. Would that be a hack? I mean that webcam wasn't made for camouflage people either? No, of course it's not a hack! The webcam is MADE for making video data, and that's what it does. It does not care what you do with the video data, it's just giving it to you to do whatever you want to it, minding it's own business XD The same principal applies here.

Likewise with the KINECT. The KINECT is not doing any of this camouflage trickery. The KINECT is doing it's job, totally normally. The KINECT is not hacked. The KINECT is sending data as it normally does. It's webcam and it's depth sensor are getting data, and sending it down the USB cable, as it is designed to do. We are using that webcam/sensor data to do our own thing, independent of any of Kinect's internal workings, and the Kinects operation is in no way effected. They're collecting data Kinect puts out, UNHACKED, and then forwarding that to another program which plays around with it and does this. It's a smart program maybe, but there is no hackery?


Please tell me another "webcam" that has a depth sensor incorporated?

aftas said,

Please tell me another "webcam" that has a depth sensor incorporated?

Yes I am obviously aware on that, if you read what I said I did point you could do it with "enough smarts", y'know, to make up for the lack of depth sensor. There was infact a tech demo of some one using an ordinary old PlayStation eye toy - just a webcam essentially - to achieve a similar effect to what is shown above, and you can still find it on youtube.

The point being, this is not a hack. The Kinect hardware was made to capture and send depth sensor data, a video feed, and audio data. That is the nature and purpose of it's technology - it is built to do that. There is no other purpose to it. Everything else that happens as a result of it is from external software, in the case of the 360 - it's the "Kinect software" stored on the 360 - but this is external and separate from the Kinect hardware. The Kinect hardware is essential "dumb". It doesn't perform any particular complex manipulation of this data - its just captures it's data and sends it on.

The PC hobbyist community is therefore using the Kinect hardware as it was designed to used - simply to collect data and pass it on to an external program to crunch and manipulate the data, in the case of the 360 being the "Kinect Software", and on the PC, it's all these hobbyist programs.

Ergo, using the Kinect hardware like this is NOT A HACK! @__@

Edited by ~Johnny, Dec 3 2010, 12:46am :

aftas said,

Please tell me another "webcam" that has a depth sensor incorporated?

Just to add to the above - there is therefore no "reconfiguring" and no "reprogramming" of the Kinect hardware, and ergo the above video does not facilitate your provided definition of what "Hacking" is.

So to sum it all up, this is not a hack, and it's not hacking. (:

faraaz said,

The first thing to talk about is, Kinect was not actually hacked. Hacking would mean that someone got to our algorithms that sit inside of the Xbox and was able to actually use them, which hasn't happened. Or, it means that you put a device between the sensor and the Xbox for means of cheating, which also has not happened. That's what we call hacking, and that's what we have put a ton of work and effort to make sure doesn't actually occur. What has happened is someone wrote an open-source driver for PCs that essentially opens the USB connection, which we didn't protect, by design, and reads the inputs from the sensor. The sensor, again, as I talked earlier, has eyes and ears, and that's a whole bunch of noise that someone needs to take and turn into signal.

- Microsoft's Alex Kipman speaking formally on NPR's Science Friday


Another person representing something also large said that he had been to al 57 states. Just because you are in a big position doesnt mean you know what you are talking about.


Yes I am obviously aware on that, if you read what I said I did point you could do it with "enough smarts", y'know, to make up for the lack of depth sensor.

That isnt a depth sensor. That is emulating a depth sensor which is not the same.


Just to add to the above - there is therefore no "reconfiguring" and no "reprogramming" of the Kinect hardware, and ergo the above video does not facilitate your provided definition of what "Hacking" is.

Has Microsoft released a application which allows you to use Kinect as a webcam and make a (somewhat) 3 dementional image? Can any "webcam" by itself do this?

Kinect was NOT made to show this. Applications were made to understand the data Kinect recieves and manipulate it to give different results.

Therefore it is a hack.

aftas said,

That isnt a depth sensor. That is emulating a depth sensor which is not the same.
Has Microsoft released a application which allows you to use Kinect as a webcam and make a (somewhat) 3 dementional image? Can any "webcam" by itself do this?

Kinect was NOT made to show this. Applications were made to understand the data Kinect recieves and manipulate it to give different results.

Therefore it is a hack.

Ugh. I didn't say it was a depth sensor. It's not trying to emulate a depth sensor either, it's using a different, far more complex software method to achieve a similar result.

Microsoft HAS software which you let you use Kinect as a webcam (Video Kinect), and make & move around the 3D image ("Kinect Tuner", found in Demos & Trials), on the 360. Not that it matters because you seem to be avoiding massive points of the argument entirely. (And egro, with your logic Kinect was therefore made to show this)

Though don't think you get the point. Kinect, the hardware, was made too:
1 - Capture 3 dimensional data from it's depth sensor
2 - Capture live video
3 - Capture live audio
5 - Send all that data down the USB cable to something else to deal with.

Every other function of the Kinect does not come from the Kinect hardware, it comes from the Kinect software. The Kinect software was made to do everything else. The Kinect hardware was made just to collect data, which the Kinect software could use. PC users are not using, and have no interaction with the Kinect software at all - they're using the Kinect hardware to do it's actual job. @____@

~Johnny said,

Ugh. I didn't say it was a depth sensor. It's not trying to emulate a depth sensor either, it's using a different, far more complex software method to achieve a similar result.


You commented that a webcam and a depth sensor could do the same as Kinect. Thats why I asked you what device can also do this.


Microsoft HAS software which you let you use Kinect as a webcam (Video Kinect), and make & move around the 3D image ("Kinect Tuner", found in Demos & Trials), on the 360. Not that it matters because you seem to be avoiding massive points of the argument entirely. (And egro, with your logic Kinect was therefore made to show this)

Microsoft has the software to interpret what Kinect recieves and gives. We dont. We hacked it (using software) so it (the software) can show us what Kinect recieves and gives.

Microsoft has a method to use the Kinect on PC? Please show me.


Though don't think you get the point. Kinect, the hardware, was made too:
1 - Capture 3 dimensional data from it's depth sensor
2 - Capture live video
3 - Capture live audio
5 - Send all that data down the USB cable to something else to deal with.

Ill fill in 4:
4 - Prepare the Xbox 360 to recieve the data

Yet we changed 4 to make it that the PC recieves that data. That, and it seems to hurt you for some odd reason, is what has been hacked.


Every other function of the Kinect does not come from the Kinect hardware, it comes from the Kinect software. The Kinect software was made to do everything else.

Which is for the Xbox 360. Not for the PC. Therefore it is hacked. Hacked = modified. Would you perfer that I say modified?


The Kinect hardware was made just to collect data, which the Kinect software could use. PC users are not using, and have no interaction with the Kinect software at all - they're using the Kinect hardware to do it's actual job. @____@

Because the Kinect software does not exist. We hacked up the software so the Kinect can send data to the PC's software.

Again, I dont know why everyone is so hurt. The Kinect has been hacked to be used on the PC. End of story.


BTW, Im done with the argument. If you want to incorrect state that the Kinect has not been hacked, you are free to do so and be wrong about it.
Santa Clause also exists.

aftas said,

You commented that a webcam and a depth sensor could do the same as Kinect. Thats why I asked you what device can also do this.

If you'd like to quote the exact place where I said that...

aftas said,

Microsoft has the software to interpret what Kinect recieves and gives. We dont. We hacked it (using software) so it (the software) can show us what Kinect recieves and gives.

We have not hacked it. The data is not encrypted in any way, shape or form. We just take the raw data that the Kinect sensors are displaying. Just like you would do with any other form of sensor / webcam. Kinect uses standard data feeds. Displaying the raw data from a webcam is not hacking. Displaying raw data from a temperature sensor is not hacking. Display raw data from a pressure sensor is not hacking. Why is displaying data from a depth sensor and a webcam considered hacking then? These devices are not DESIGNED to display data. They are DESIGNED to collect data, for something else to display.

aftas said,
Ill fill in 4:
4 - Prepare the Xbox 360 to recieve the data

It doesn't care about what's on the receiving end of it's USB cable. There is no Xbox specific lock down in the Kinect that only allows it to communicate with Kinect. As soon as Kinect gets told to, it just collects raw data and sends it on - it does not prepare the other end. The other end could be totally ignoring it or not even exist -the Kinect hardware doesn't really care - that is not it's job. Xbox 360 handles that. Plugging a USB cable into another device does not count as hacking. Kinect would happily send it's data to ANY USB socket if you wanted it to, just by sending it a command to do it's job. It does not care! The raw data stream will just get sent there - that's no hacking, that's is what it is designed to do. The device in the other end has to prepare itself to receive the data (which of course, since it's the thing that sent the command to turn on Kinect, it probably has)

aftas said,

Which is for the Xbox 360. Not for the PC. Therefore it is hacked. Hacked = modified. Would you perfer that I say modified?

And this is the point - NOTHING has been modified. There was no original codebase for us to modify. The Kinect hardware has had no modification to it.

aftas said,

Because the Kinect software does not exist. We hacked up the software so the Kinect can send data to the PC's software.

The Kinect software exists on the 360 - if we had modified that, taken it all and ported it to the PC, that may have been hacking. Writing something brand new from the ground up which does not modify anything, or circumvent any form of security or encryption is not hacking.

*facepalm* You're probably not understanding something here... but whatever. A lot of us can see it's not hacked, especially after everything I've pointed out. I should write an essay on this xD

Edited by ~Johnny, Dec 3 2010, 2:03pm :

~Johnny said,

If you'd like to quote the exact place where I said that...


Sure.
21 Hours Ago

Yes I am obviously aware on that, if you read what I said I did point you could do it with "enough smarts", y'know, to make up for the lack of depth sensor. There was infact a tech demo of some one using an ordinary old PlayStation eye toy - just a webcam essentially - to achieve a similar effect to what is shown above, and you can still find it on youtube.


We have not hacked it. The data is not encrypted in any way, shape or form.

Read the definition of hack again. Obviously you are not correctly informed on what it means.


We just take the raw data that the Kinect sensors are displaying.

And interpret it with what officialy software?


It doesn't care about what's on the receiving end of it's USB cable.

In other words, you connect it to a PC or any other device, and it works? Recieving, sending and showing the data correctly and out of the box with NO additional work or somewhere?
Might want to tell Microsoft.


And this is the point - NOTHING has been modified. There was no original codebase for us to modify. The Kinect hardware has had no modification to it.

The modification is that Kinect's data is now recieveable (sp?) by another device other than the Xbox 360. That is a hack as it is not supported by the original maker.


The Kinect software exists on the 360 - if we had modified that, taken it all and ported it to the PC, that may have been hacking.

Hacking or cracking; Debatable but yes, that itself would ALSO be a hack.


Writing something brand new from the ground up which does not modify anything, or circumvent any form of security or encryption is not hacking.

You are modifing the enviroment the device was originally used for. Your inability to understand surprises me..


*facepalm* You're probably not understanding something here... but whatever. A lot of us can see it's not hacked, especially after everything I've pointed out. I should write an essay on this xD

You seem not to understand the bases of hacking. You are obviously looking at too big of a picture and dont seem to understand that small hacks (such as this) are also hacks. Most of us see that it is hacked. Write your essay but you are simply wasting time.


I replied because I am just jawdropping at your atittude that thinks you are correct when you have no idea.
Read a bit: http://en.wikipedia.org/wiki/Hacking

Brent1700 said,
meh, I can do this with my invisibility cloak. NOT impressed.

Right...but don't you need to roll a natural 20?

sviola said,

Right...but don't you need to roll a natural 20?

And if you botch, that's just not good. Especially if you're running from someone/something...

Brent1700 said,
meh, I can do this with my invisibility cloak. NOT impressed.

I can go invisible just by thinking it, but it doesnt work if anyone including me is watching - so people have their doubts, but it does work!

duddit2 said,

I can go invisible just by thinking it, but it doesnt work if anyone including me is watching - so people have their doubts, but it does work!


I can go invisible without thinking about it. Happens all the time, but most often when I turn out the lights.