Hi everyone,
Thought some of you might be interested in my page on how to use your Kinect with Scratch: http://stephen-howell.tumblr.com
The Kinect is a great add-on for the XBox 360. Many people had their doubts, and although core gamers will decry the Kinect, it can be really good fun to play around with. Literally, you are the controller. There has been some interest in hacking the Kinect to work with computers other than the XBox, and there are drivers available now to use with Windows, MacOSX and Linux.
I became interested in coding Processing to work with the Kinect, but soon became disheartened when it appeared that bone data (or joint positions) were not readily available, instead only 3D depth data (cool but not enough) was readable.
Then I found the OpenNI drivers (see links below) and the OSC server from Sensebloom. They were able to send joint data encoded as OSC commands to Processing, and I wrote up my own little Processing receiver to test it. It worked so well, I delved into their C++ code to see if I could send the same data to Scratch.
When I couldn’t see a way for Scratch to interpret the OSC code, I read about the remote sensing over the network that Scratch allows. This was perfect; I could send Scratch commands from a C program which was reading joint data from the Kinect.
So after making a few Scratch proof-of-concept games to test it, I decided to release it all for you guys to test out.
You will need:
A standalone Kinect (one bought without a console) OR a Kinect that came in a console bundle and the separately available USB plug to connect it to your PC
A Windows PC (other versions can be made by coders familiar with coding on OSX or Linux)
The OpenNI.org drivers installed
The OpenNI2Scratch program
Scratch
Some Kinect aware Scratch programs. Here are a few: Skeleton, Ball Bounce and Space Invaders
(Links to all of these are available on the tumblr page mentioned above)
Once you’ve set that up, you have to connect the Kinect to your PC and wait for the drivers to install. Then run the OpenNI2Scratch program. Run Scratch, execute the Scratch program you want to control. Calibrate the Kinect - this involves standing a fair bit back from it and standing in a ‘calibration pose’ for a few seconds - see picture on site. Once the program has detected you, it will send the joint positions to Scratch. After that, see the variables on the sample Scratch programs to see how to use the joint data.
Enjoy.
Stephen Howell
Offline
<say[ awesome ]for( ever )secs>
Last edited by jcml (2011-03-27 13:49:01)
Offline
If only I had a Kinect...
Offline
markyparky56 wrote:
If only I had a Kinect...
Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.
Offline
SHowell wrote:
markyparky56 wrote:
If only I had a Kinect...
Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.
True, I saw a thing on the internet similar to this, but it used a wii remote and infrared lights to track the person, I see the kinect is a far more efficient way to do this.
Btw, how would you make the Kinect track two people and give it to scratch?
Offline
markyparky56 wrote:
SHowell wrote:
markyparky56 wrote:
If only I had a Kinect...
Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.True, I saw a thing on the internet similar to this, but it used a wii remote and infrared lights to track the person, I see the kinect is a far more efficient way to do this.
Btw, how would you make the Kinect track two people and give it to scratch?
Well, currently, I only send the current (single) user, and ignore the others. I could send multiple skeletons, but there are 2 options: Send all the joint data for a given skeleton and then broadcast 'update skeleton 1' or 'update skeleton 2' etc. OR duplicate the head_x etc. variables to head1_x.
I could do it either way if there is interest in it. What do you think, would people be interested in 2 user scratch-kinect programs?
Regards,
Offline
SHowell wrote:
markyparky56 wrote:
SHowell wrote:
Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.True, I saw a thing on the internet similar to this, but it used a wii remote and infrared lights to track the person, I see the kinect is a far more efficient way to do this.
Btw, how would you make the Kinect track two people and give it to scratch?Well, currently, I only send the current (single) user, and ignore the others. I could send multiple skeletons, but there are 2 options: Send all the joint data for a given skeleton and then broadcast 'update skeleton 1' or 'update skeleton 2' etc. OR duplicate the head_x etc. variables to head1_x.
I could do it either way if there is interest in it. What do you think, would people be interested in 2 user scratch-kinect programs?
Regards,
It would show them how to have multiple players/users and allow them to expand from there if they wanted.
Offline
markyparky56 wrote:
If only I had a Kinect...
Ditto.
(First Advanced Topics post!)
Offline
Stephen,
Thanks a ton for your posting and examples. I was able to load your programs on both Windows XP (32-bit) and on a Windows 7 (64-bit) machine.
It works like a charm.
I added some of the steps I took (copied from the OpenNI forum and edited):
Installation steps
-------------------------------
1) Install OpenNI:
http://www.openni.org/downloadfiles/2-openni-binaries
I tried both "stable" and "unstable" versions, not sure which is better.
OPENNI-Win32-1.0.0.23.exe
2) Install the Kinect sensor plugin:
https://github.com/avin2/SensorKinect
Run the precompiled SensorKinect-Win32-5.0.0.exe file from the
Bin directory
3) Install NITE:
http://downloads.primesense.com/NITE-Win32-1.3.0.17.exe
Use the key: 0KOIk2JeIBYClPWVnMoRKn5cdY4=
(note: the "=" at the end matters. Also, put it in a font that lets you see the difference between uppercase i, lowercase L and the number one; if you are typing it in, and not cut-and-pasting
4) Copy the XML files from
"c:\Program Files\Prime Sense\Sensor\SampleXMLs\NITE\Data"
into
"c:\Program Files\Prime Sense\NITE\Data"
(overwrite files)
5) Copy the XML files from
"c:\Program Files\Prime Sense\Sensor\SampleXMLs\OPENNI\Data"
into
"c:\Program Files\OpenNI\Data"
(overwrite files)
Then I loaded your openni2scratch program.
A few notes:
It was a little tricky getting the Kinect drivers to load on Windows XP. Easier on Windows 7.
For those wondering, no compiling or programming was necessary to get the Scratch environment to work. I am running Norton Security Suite and did not have to make any changes to the settings.
I ended up with three Devices in the Device Manager:
Other devices
Xbox NUI Audio - with an error icon
PrimeSensor
Kinect Camera
Kinect Motor
The Kinect green light will blink slowly once the drivers are loaded properly. It never goes solid green.
I didn't succeed in getting the Kinect audio drivers loaded, I'll leave that for another day.
I find calibration faster if I make fists, instead of open hands.
It runs slow on an old Windows XP (and I mean OLD). However, on a quadcore Xeon system with Windows 7, it's pretty responsive.
I would definitely be in favor of a two-person version.
My kids (2, 4, 6, 8) already love it - and with Scratch, the 6 and 8 year olds can program things themselves. A far cry from the msbasic that I started with.
-Rich
Offline
if i am correct the kinect also has a voice recinization feture ,would it be possible to allow scratch to "hear" commands too?
that would be awesome
,jcml
Offline
I could be completely wrong, but I think that the mic audio hardware is separate to the 3d camera, and as such we do not (yet) have drivers. Voice commands involves huge programming challenges though, so could be a while before we have a simple solution for it.
Offline
This is really cool! Now, like the missing iPhone app, kinect games can be made in scratch!
Yet, I just need one.
Offline
ssss wrote:
could you do something similar with the ps2's "eyetoy"? I have it working as a cheap webcam with good quality... at the moment
Afraid not, the Kinect has 2 cameras that allow a 3D model (with depth) to be calculated, and the OpenNI drivers allow us access to the bone data.
Offline
Offline
In terms of the Kinect plug:
As far as I know all Kinect's have a specialized plug (which I will call "fat" USB).
The Kinect's motorized tilt mechanism needs more power than USB provides, and to avoid needing a separate power cord, Microsoft made a new "fat" USB plug that has a USB signal and also more power. I think it would have been easier if they just had a separate power cord (like many external USB hard drives), but I guess they thought it was worth having one less wire/plug. Wikipedia described the Kinect in more detail.
So, you always have a "fat" USB plug, which will not fit into a normal USB slot. The end of the cable that attaches to the Kinect appears permanently attached, so you cannot change cords.
If you buy the new Xbox with Kinect, you just plug the "fat" USB plug into your new xbox, which has a special "fat" USB plug location.
If you buy a separate Kinect, it comes with an adaptor that you plug the "fat" USB plug into. That adaptor is a "Y" adaptor, and has a regular USB plug and a power cord as outputs.
So, if you just want to use a Kinect just on your PC, you can buy the Kinect separately and you'll get the "Y" adaptor. (retail around $150)
If you buy a new xbox with Kinect, you will need to separately buy the "Y" adaptor. At this point Microsoft sells it, but I haven't seen it anywhere else ($35 I think). I presume cheaper ones will be coming out.
I couldn't wait, so even though I had a new xbox with Kinect, I just went out and bought a separate Kinect for my PC. Now I have two, and at least I save the hassle of switching it back and forth between my PC and my xbox. I might order the "Y" adaptor - then I could use two PCs, two Kinects, and MESH to have a multiuser Scratch game.
-Rich
Offline
SHowell wrote:
ihaveamac wrote:
I assume the Kinect has a usb plug?
Standalone Kinects (bought without an XBox) have, there is an addon cable available for the bundled ones.
What if you had two, reasonably good quality, web-cams? could you emulate it that way? If I think about it, it would mean basicly making your own kinect, what kind of output do you get for the Kinect? do you get something that needs to be de-coded or just points of the body? (and I geuss a whole lot more)
(For those residents of the UK, the Kinect is £127!)
http://www.amazon.co.uk/Kinect-Sensor-Project-Natal-Adventures/dp/B0036DDW2G/ref=sr_1_1?ie=UTF8&qid=1294765435&sr=8-1
Offline