This is a read-only archive of the old Scratch 1.x Forums.
Try searching the current Scratch discussion forums.

#1 2011-01-08 13:19:24

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Scratch and the Kinect

Hi everyone,

Thought some of you might be interested in my page on how to use your Kinect with Scratch: http://stephen-howell.tumblr.com

The Kinect is a great add-on for the XBox 360. Many people had their doubts, and although core gamers will decry the Kinect, it can be really good fun to play around with. Literally, you are the controller. There has been some interest in hacking the Kinect to work with computers other than the XBox, and there are drivers available now to use with Windows, MacOSX and Linux.

I became interested in coding Processing to work with the Kinect, but soon became disheartened when it appeared that bone data (or joint positions) were not readily available, instead only 3D depth data (cool but not enough) was readable. 

Then I found the OpenNI drivers (see links below) and the OSC server from Sensebloom. They were able to send joint data encoded as OSC commands to Processing, and I wrote up my own little Processing receiver to test it. It worked so well, I delved into their C++ code to see if I could send the same data to Scratch.

When I couldn’t see a way for Scratch to interpret the OSC code, I read about the remote sensing over the network that Scratch allows. This was perfect; I could send Scratch commands from a C program which was reading joint data from the Kinect.

So after making a few Scratch proof-of-concept games to test it, I decided to release it all for you guys to test out.

You will need:

A standalone Kinect (one bought without a console) OR a Kinect that came in a console bundle and the separately available USB plug to connect it to your PC

A Windows PC (other versions can be made by coders familiar with coding on OSX or Linux)

The OpenNI.org drivers installed

The OpenNI2Scratch program

Scratch

Some Kinect aware Scratch programs. Here are a few: Skeleton, Ball Bounce and Space Invaders

(Links to all of these are available on the tumblr page mentioned above)

Once you’ve set that up, you have to connect the Kinect to your PC and wait for the drivers to install. Then run the OpenNI2Scratch program. Run Scratch, execute the Scratch program you want to control. Calibrate the Kinect - this involves standing a fair bit back from it and standing in a ‘calibration pose’ for a few seconds - see picture on site. Once the program has detected you, it will send the joint positions to Scratch. After that, see the variables on the sample Scratch programs to see how to use the joint data.

Enjoy.

Stephen Howell

Offline

 

#2 2011-01-08 17:33:07

jcml
Scratcher
Registered: 2010-10-08
Posts: 12

Re: Scratch and the Kinect

<say[ awesome ]for( ever )secs>

Last edited by jcml (2011-03-27 13:49:01)


I need help for one huge a mesh game i need to no how to make a world-wide scratch program if you have any idea's just coment on one of my projects

Offline

 

#3 2011-01-08 20:30:17

Paddle2See
Scratch Team
Registered: 2007-10-27
Posts: 1000+

Re: Scratch and the Kinect

Wow...that is really impressive!


http://i39.tinypic.com/2nav6o7.gif

Offline

 

#4 2011-01-08 21:07:14

fullmoon
Retired Community Moderator
Registered: 2007-06-04
Posts: 1000+

Re: Scratch and the Kinect

I CANNOT WAIT TO TRY THIS.


http://i302.photobucket.com/albums/nn100/fullmoon32/wow.jpg

Offline

 

#5 2011-01-08 22:47:51

hdarken
Scratcher
Registered: 2008-06-26
Posts: 1000+

Re: Scratch and the Kinect

This sounds amazing! I can't wait to hear more about it.


http://i.imgur.com/VskBk.png
http://i.imgur.com/tgxVZ.jpg

Offline

 

#6 2011-01-09 09:49:04

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

If only I had a Kinect...


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#7 2011-01-09 10:29:15

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

markyparky56 wrote:

If only I had a Kinect...

Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.

Offline

 

#8 2011-01-09 10:45:52

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

markyparky56 wrote:

If only I had a Kinect...

Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.

True, I saw a thing on the internet similar to this, but it used a wii remote and infrared lights to track the person, I see the kinect is a far more efficient way to do this.

Btw, how would you make the Kinect track two people and give it to scratch?


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#9 2011-01-09 10:53:30

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

markyparky56 wrote:

SHowell wrote:

markyparky56 wrote:

If only I had a Kinect...

Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.

True, I saw a thing on the internet similar to this, but it used a wii remote and infrared lights to track the person, I see the kinect is a far more efficient way to do this.

Btw, how would you make the Kinect track two people and give it to scratch?

Well, currently, I only send the current (single) user, and ignore the others. I could send multiple skeletons, but there are 2 options: Send all the joint data for a given skeleton and then broadcast 'update skeleton 1' or 'update skeleton 2' etc. OR duplicate the head_x etc. variables to head1_x.

I could do it either way if there is interest in it. What do you think, would people be interested in 2 user scratch-kinect programs?

Regards,

Offline

 

#10 2011-01-09 11:05:11

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

markyparky56 wrote:

SHowell wrote:


Can't help you there I'm afraid, Santa Claus brought one to my 7 year old boy.
However you don't need the XBox 360 for Scratch & Kinect to work, so it might be more affordable to get a Kinect on its own without the XBox bundle.

True, I saw a thing on the internet similar to this, but it used a wii remote and infrared lights to track the person, I see the kinect is a far more efficient way to do this.

Btw, how would you make the Kinect track two people and give it to scratch?

Well, currently, I only send the current (single) user, and ignore the others. I could send multiple skeletons, but there are 2 options: Send all the joint data for a given skeleton and then broadcast 'update skeleton 1' or 'update skeleton 2' etc. OR duplicate the head_x etc. variables to head1_x.

I could do it either way if there is interest in it. What do you think, would people be interested in 2 user scratch-kinect programs?

Regards,

It would show them how to have multiple players/users and allow them to expand from there if they wanted.


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#11 2011-01-09 16:28:15

MiffinTheMuffin
Scratcher
Registered: 2010-12-11
Posts: 1000+

Re: Scratch and the Kinect

markyparky56 wrote:

If only I had a Kinect...

Ditto.

(First Advanced Topics post!)


http://i.imgur.com/FDvbV.png

Offline

 

#12 2011-01-09 22:49:23

richardatscratch
New Scratcher
Registered: 2011-01-09
Posts: 5

Re: Scratch and the Kinect

Stephen,

Thanks a ton for your posting and examples. I was able to load your programs on both Windows XP (32-bit) and on a Windows 7 (64-bit) machine.

It works like a charm.

I added some of the steps I took (copied from the OpenNI forum and edited):

Installation steps
-------------------------------
1)      Install OpenNI:
http://www.openni.org/downloadfiles/2-openni-binaries

I tried both "stable" and "unstable" versions, not sure which is better.

OPENNI-Win32-1.0.0.23.exe

2)      Install the Kinect sensor plugin:
https://github.com/avin2/SensorKinect
             Run the precompiled SensorKinect-Win32-5.0.0.exe file from the
Bin directory

3)      Install NITE:
http://downloads.primesense.com/NITE-Win32-1.3.0.17.exe
Use the key: 0KOIk2JeIBYClPWVnMoRKn5cdY4=
(note: the "=" at the end matters. Also, put it in a font that lets you see the difference between uppercase i, lowercase L and the number one; if you are typing it in, and not cut-and-pasting

4)    Copy the XML files from
"c:\Program Files\Prime Sense\Sensor\SampleXMLs\NITE\Data"
into
"c:\Program Files\Prime Sense\NITE\Data"
(overwrite files)

5)    Copy the XML files from

"c:\Program Files\Prime Sense\Sensor\SampleXMLs\OPENNI\Data"
into
"c:\Program Files\OpenNI\Data"
(overwrite files)

Then I loaded your openni2scratch program.

A few notes:

It was a little tricky getting the Kinect drivers to load on Windows XP. Easier on Windows 7.

For those wondering, no compiling or programming was necessary to get the Scratch environment to work. I am running Norton Security Suite and did not have to make any changes to the settings.

I ended up with three Devices in the Device Manager:
Other devices
  Xbox NUI Audio - with an error icon
PrimeSensor
  Kinect Camera
  Kinect Motor

The Kinect green light will blink slowly once the drivers are loaded properly. It never goes solid green.
I didn't succeed in getting the Kinect audio drivers loaded, I'll leave that for another day.
I find calibration faster if I make fists, instead of open hands.
It runs slow on an old Windows XP (and I mean OLD). However, on a quadcore Xeon system with Windows 7, it's pretty responsive.

I would definitely be in favor of a two-person version.

My kids (2, 4, 6, 8) already love it - and with Scratch, the 6 and 8 year olds can program things themselves. A far cry from the msbasic that I started with.

-Rich

Offline

 

#13 2011-01-10 04:22:45

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

Brilliant, thank you so much Richard, I will create a much longer installation tutorial now that I see the interest generated by this. I have 4 kids too, and they can't get enough of the 'dancing cat'.

Offline

 

#14 2011-01-10 12:48:56

jcml
Scratcher
Registered: 2010-10-08
Posts: 12

Re: Scratch and the Kinect

if i am correct the kinect also has a voice recinization feture ,would it be possible to allow scratch to "hear" commands too?

that would be awesome
,jcml


I need help for one huge a mesh game i need to no how to make a world-wide scratch program if you have any idea's just coment on one of my projects

Offline

 

#15 2011-01-10 12:56:34

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

I could be completely wrong, but I think that the mic audio hardware is separate to the 3d camera, and as such we do not (yet) have drivers. Voice commands involves huge programming challenges though, so could be a while before we have a simple solution for it.

Offline

 

#16 2011-01-10 15:14:53

Sparta2
Scratcher
Registered: 2010-12-31
Posts: 11

Re: Scratch and the Kinect

yikes  thats amazing


http://ui22.gamespot.com/1429/spartan_2.jpg http://www.davidbarrkirtley.com/images/300phalanx.jpg
lol  this is not Sparta people... or is it?

Offline

 

#17 2011-01-10 17:55:37

ihaveamac
Scratcher
Registered: 2007-09-22
Posts: 1000+

Re: Scratch and the Kinect

This is really cool! Now, like the missing iPhone app, kinect games can be made in scratch!

Yet, I just need one.  sad


~ihaveamac - visit ihaveamac.net

Offline

 

#18 2011-01-10 18:08:43

ssss
Scratcher
Registered: 2007-07-29
Posts: 1000+

Re: Scratch and the Kinect

could you do something similar with the ps2's "eyetoy"?  I have it working as a cheap webcam with good quality... at the moment


Hey.  It's me SSSS, back from the dead!  smile

Offline

 

#19 2011-01-10 18:28:06

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

ssss wrote:

could you do something similar with the ps2's "eyetoy"?  I have it working as a cheap webcam with good quality... at the moment

Afraid not, the Kinect has 2 cameras that allow a 3D model (with depth) to be calculated, and the OpenNI drivers allow us access to the bone data.

Offline

 

#20 2011-01-10 18:41:46

ihaveamac
Scratcher
Registered: 2007-09-22
Posts: 1000+

Re: Scratch and the Kinect

I assume the Kinect has a usb plug?


~ihaveamac - visit ihaveamac.net

Offline

 

#21 2011-01-10 18:54:22

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

ihaveamac wrote:

I assume the Kinect has a usb plug?

Standalone Kinects (bought without an XBox) have, there is an addon cable available for the bundled ones.

Offline

 

#22 2011-01-10 20:37:10

bobthejoe68
Scratcher
Registered: 2009-05-25
Posts: 69

Re: Scratch and the Kinect

Awesome!


http://i46.tinypic.com/p9g86.png

Offline

 

#23 2011-01-10 21:25:24

what-the
Scratcher
Registered: 2009-10-04
Posts: 1000+

Re: Scratch and the Kinect

Cool but I don't have a Kinect.


http://imageshack.us/m/64/9034/ddfss.pngMy site
Find someone post count. Click posts under username. Find number of pages. Times that by 40 for min and 60 for max and you have a rough estimate of post count.

Offline

 

#24 2011-01-10 23:32:56

richardatscratch
New Scratcher
Registered: 2011-01-09
Posts: 5

Re: Scratch and the Kinect

In terms of the Kinect plug:

As far as I know all Kinect's have a specialized plug (which I will call "fat" USB).

The Kinect's motorized tilt mechanism needs more power than USB provides, and to avoid needing a separate power cord, Microsoft made a new "fat" USB plug that has a USB signal and also more power. I think it would have been easier if they just had a separate power cord (like many external USB hard drives), but I guess they thought it was worth having one less wire/plug. Wikipedia described the Kinect in more detail.

So, you always have a "fat" USB plug, which will not fit into a normal USB slot. The end of the cable that attaches to the Kinect appears permanently attached, so you cannot change cords.

If you buy the new Xbox with Kinect, you just plug the "fat" USB plug into your new xbox, which has a special "fat" USB plug location.

If you buy a separate Kinect, it comes with an adaptor that you plug the "fat" USB plug into. That adaptor is a "Y" adaptor, and has a regular USB plug and a power cord as outputs.

So, if you just want to use a Kinect just on your PC, you can buy the Kinect separately and you'll get the "Y" adaptor. (retail around $150)

If you buy a new xbox with Kinect, you will need to separately buy the "Y" adaptor. At this point Microsoft sells it, but I haven't seen it anywhere else ($35 I think). I presume cheaper ones will be coming out.

I couldn't wait, so even though I had a new xbox with Kinect, I just went out and bought a separate Kinect for my PC. Now I have two, and at least I save the hassle of switching it back and forth between my PC and my xbox. I might order the "Y" adaptor - then I could use two PCs, two Kinects, and MESH to have a multiuser Scratch game.

-Rich

Offline

 

#25 2011-01-11 12:08:41

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

ihaveamac wrote:

I assume the Kinect has a usb plug?

Standalone Kinects (bought without an XBox) have, there is an addon cable available for the bundled ones.

What if you had two, reasonably good quality, web-cams? could you emulate it that way? If I think about it, it would mean basicly making your own kinect, what kind of output do you get for the Kinect? do you get something that needs to be de-coded or just points of the body? (and I geuss a whole lot more)

(For those residents of the UK, the Kinect is £127!)
http://www.amazon.co.uk/Kinect-Sensor-Project-Natal-Adventures/dp/B0036DDW2G/ref=sr_1_1?ie=UTF8&qid=1294765435&sr=8-1


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

Board footer