This is a read-only archive of the old Scratch 1.x Forums.
Try searching the current Scratch discussion forums.

#26 2011-01-11 12:36:39

richardatscratch
New Scratcher
Registered: 2011-01-09
Posts: 5

Re: Scratch and the Kinect

The Kinect has an infrared transmitter and camera - that is what is used to "see" you.

The regular camera is just used in the Xbox to take pictures or video conference. I don't think it contributes to the "sensing" of where you are. I don't think we have access to that yet in scratch.

Others have described in more detail than I have tried to follow what the Kinect output is. Try a quick Google search.

The net result at the Scratch level, is that by using Stephen Howell's program, you get variables in Scratch that give you the x and y coordinates of each body part/joint.

Then you can do things like use the x position of the right hand to pick a keyboard note to play; as you move your hand, it's like you're playing a piano (at the hand level, not the finger level).

I did a quick slide-show where if you crossed your hands (rhand x < lhand y) it switched the background.

I don't know if there are any libraries yet that sense higher level actions (jumped, threw ball, turned around, kicked with the left foot). In theory, those could all be built in scratch itself, and broadcast when they detect a certain type of motion.

I also haven't seen a z position. Stephen, is that available? That could be useful for detecting walking towards or away from the camera. Otherwise, maybe you could use the distance from the head to the hips as a proxy for how close/far the subject is.

-Rich

Offline

 

#27 2011-01-11 13:22:46

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

richardatscratch wrote:

The Kinect has an infrared transmitter and camera - that is what is used to "see" you.

Yes, it would be a lot simpler to buy a Kinect than try to reinvent it!

richardatscratch wrote:

The regular camera is just used in the Xbox to take pictures or video conference. I don't think it contributes to the "sensing" of where you are. I don't think we have access to that yet in scratch.

Well, we have access to the pixel data (what the camera sees) but to send it to Kinect would involve sending each pixel individually, Scratch has no network support yet for blobs of data (that I can see).

richardatscratch wrote:

Then you can do things like use the x position of the right hand to pick a keyboard note to play; as you move your hand, it's like you're playing a piano (at the hand level, not the finger level).

I did a quick slide-show where if you crossed your hands (rhand x < lhand y) it switched the background.

Cool, I hope you can share some Kinect-enabled projects in the future.

richardatscratch wrote:

I don't know if there are any libraries yet that sense higher level actions (jumped, threw ball, turned around, kicked with the left foot). In theory, those could all be built in scratch itself, and broadcast when they detect a certain type of motion.

There is gesture support in the OpenNI drivers I'm using, but I don't know yet a)  how to specify what a given gesture is and b) whether the open drivers actually supply gesture recognition, it may be a 'future functionality' feature.

richardatscratch wrote:

I also haven't seen a z position. Stephen, is that available? That could be useful for detecting walking towards or away from the camera. Otherwise, maybe you could use the distance from the head to the hips as a proxy for how close/far the subject is.
-Rich

A version with z position coming up just for you Rich, I didn't think anyone would want it so I left it out originally. I spent most of today with version 2 (mirror mode support) and version 3 (multiuser support) but multiuser is not tested yet. Version 4 will include z axis support, will try and have that released tonight. I should be correcting exams, but this is much more fun  smile

Stephen

Offline

 

#28 2011-01-11 13:30:32

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

markyparky56 wrote:

SHowell wrote:

ihaveamac wrote:

I assume the Kinect has a usb plug?

Standalone Kinects (bought without an XBox) have, there is an addon cable available for the bundled ones.

What if you had two, reasonably good quality, web-cams? could you emulate it that way? If I think about it, it would mean basicly making your own kinect, what kind of output do you get for the Kinect? do you get something that needs to be de-coded or just points of the body? (and I geuss a whole lot more)

(For those residents of the UK, the Kinect is £127!)
http://www.amazon.co.uk/Kinect-Sensor-Project-Natal-Adventures/dp/B0036DDW2G/ref=sr_1_1?ie=UTF8&qid=1294765435&sr=8-1

No, there are IR transmitters and very complex algorithms at work, easier to buy a Kinect (if you can afford one) I'm afraid.

Offline

 

#29 2011-01-11 14:48:40

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

markyparky56 wrote:

SHowell wrote:


Standalone Kinects (bought without an XBox) have, there is an addon cable available for the bundled ones.

What if you had two, reasonably good quality, web-cams? could you emulate it that way? If I think about it, it would mean basicly making your own kinect, what kind of output do you get for the Kinect? do you get something that needs to be de-coded or just points of the body? (and I geuss a whole lot more)

(For those residents of the UK, the Kinect is £127!)
http://www.amazon.co.uk/Kinect-Sensor-Project-Natal-Adventures/dp/B0036DDW2G/ref=sr_1_1?ie=UTF8&qid=1294765435&sr=8-1

No, there are IR transmitters and very complex algorithms at work, easier to buy a Kinect (if you can afford one) I'm afraid.

Yeah, see what you mean. Maybe for furture investment.


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#30 2011-01-11 20:51:10

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

OK, new version of the Kinect/Scratch program for everyone to try.

This has a simple GUI which allows you to launch the Kinect and Connect to Scratch by clicking buttons. It gives some feedback in a log window. Currently does not tell you if you are posing correctly, working on that.

New features include: Mirror mode (x-axis reflected), Multi-user begun (flaky, works for 2 players mostly, needs more testing and my 2 year old refused to stand in the calibration pose long enough to ensure it worked consistently), and z-axis (for richardatscratch). Z axis is a value that decreases (negatively) the farther you get from the Kinect sensor.

Screenshot: http://27.media.tumblr.com/tumblr_levyu1ROgU1qg7yt4o1_500.png

New version available on my blog at http://stephen-howell.tumblr.com

Hope you all enjoy it (at least, those of you with Kinects!)

Stephen

Offline

 

#31 2011-01-12 10:27:45

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

Forgot to mention:

User 1: head_x, head_y, head_z
User 2: head2_x, head2_y, head2_z

Regards,

Stephen

Offline

 

#32 2011-01-12 11:49:05

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

Z axis is a value that decreases (negatively) the farther you get from the Kinect sensor.

As in there is a set 0, or 0 is where you start off at?


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#33 2011-01-12 12:35:50

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

markyparky56 wrote:

SHowell wrote:

Z axis is a value that decreases (negatively) the farther you get from the Kinect sensor.

As in there is a set 0, or 0 is where you start off at?

0 is the closest it can recognise, about 1 meter from the sensor.

Offline

 

#34 2011-01-12 12:40:46

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

markyparky56 wrote:

SHowell wrote:

Z axis is a value that decreases (negatively) the farther you get from the Kinect sensor.

As in there is a set 0, or 0 is where you start off at?

0 is the closest it can recognise, about 1 meter from the sensor.

Ok. And the furthest?

Last edited by markyparky56 (2011-01-12 12:41:13)


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#35 2011-01-12 12:50:09

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

Well, I got to about -200 before I hit the wall.

Offline

 

#36 2011-01-12 12:56:51

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

Well, I got to about -200 before I hit the wall.

I geuss its not possible to find the maximum then.


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#37 2011-01-12 13:43:44

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

Well, it is if you have a big room, the sensor will eventually be unable to see you.
I'll test it later, currently have a bigger issue, multi-user is not working properly!

Offline

 

#38 2011-01-12 13:48:08

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

Well, it is if you have a big room, the sensor will eventually be unable to see you.
I'll test it later, currently have a bigger issue, multi-user is not working properly!

Whats wrong? Is it getting them mixed up?


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#39 2011-01-12 14:44:47

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

markyparky56 wrote:

Whats wrong? Is it getting them mixed up?

No, it's just that when users leave and come back, the OpenNI drivers treat it as a new user. My program only expects users 1 and 2, but it could be 3 and 9 or whatever.
Fixing it now, but returning it to single user for now.

Offline

 

#40 2011-01-12 15:01:10

richardatscratch
New Scratcher
Registered: 2011-01-09
Posts: 5

Re: Scratch and the Kinect

In terms of the z-axis, I got to -256 before it lost me (and I hadn't hit the wall yet). When I walk right up to the Kinect, I get 56 (so the range is -256 to 56, give or take). It lost me when I was that close.

For me x ranged from 400 to -400
y ranged from -250 to 250

Maybe I could have gotten a little more, but the kinect was having trouble tracking me reliably.

If you have OpenNI loaded, you can use:

OpenNI\Samples\Bin\Release\NiUserTracker.exe

to see what Kinect "sees". It seems that the easiest way to get "calibrated and tracked" is to start close to the Kinect and walk away slowly, and then stand in the calibration position. It usually takes only a few seconds for me.

I don't think we have motor control yet. I usually prop the Kinect up on a small pad or pencil to it is angle correctly. It's a bit difficult to angle it for both me and my 2 year old daughter. One could hook it up to the xbox, adjust he tilt there and then put it back on the PC.

-Rich

-Rich

Offline

 

#41 2011-01-13 11:08:21

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

I don't know this, but I know its possible (Just using certain cables I geuss). You could connect your PC/laptop to your TV and use it like you're monitor?


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

#42 2011-01-13 18:16:35

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

New version guys and girls, try this: http://stephen-howell.tumblr.com/post/2734703739/new-version-of-openni2scratch

There are some brief instructions there as well for setting up some scripts in Scratch to use this.

Offline

 

#43 2011-01-13 23:11:27

richardatscratch
New Scratcher
Registered: 2011-01-09
Posts: 5

Re: Scratch and the Kinect

The newest version works fine.

I uploaded my first project. It uses Stephen Howell's Kinect-Skeleton project and adds throwing a ball. It's pretty basic and the math is a bit rough. But it does let you throw a ball in an arbitrary direction.

It will only work if you download it and you have a Kinect. You need to be running Stephen's program for the Kinect interface.

Take a look.

If there seems to be interest, I also have a "balloon popping game" and a basic "pick up sprites and move them around". All pretty rough, but testing out ideas to use with the Kinect.

-Rich

Offline

 

#44 2011-01-13 23:34:37

terminator355
Scratcher
Registered: 2008-07-31
Posts: 100+

Re: Scratch and the Kinect

Hi, I am really interested in this project, but I only have a Mac! Can anyone post a tut on how to get this up and running on Mac?


No new stuff for now guys  wink  join me for some Black Ops on xbox: termhn or for some rock band on PS3: rocket232

Offline

 

#45 2011-01-14 03:03:24

johnnydean1
Scratcher
Registered: 2010-02-12
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

richardatscratch wrote:

The regular camera is just used in the Xbox to take pictures or video conference. I don't think it contributes to the "sensing" of where you are. I don't think we have access to that yet in scratch.

Well, we have access to the pixel data (what the camera sees) but to send it to Kinect would involve sending each pixel individually, Scratch has no network support yet for blobs of data (that I can see).
Stephen

You could always put the data into a list and send it that way.


You can now reach me on Twitter @johnnydean1_

Offline

 

#46 2011-01-14 05:12:37

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

johnnydean1 wrote:

SHowell wrote:

richardatscratch wrote:

The regular camera is just used in the Xbox to take pictures or video conference. I don't think it contributes to the "sensing" of where you are. I don't think we have access to that yet in scratch.

Well, we have access to the pixel data (what the camera sees) but to send it to Kinect would involve sending each pixel individually, Scratch has no network support yet for blobs of data (that I can see).
Stephen

You could always put the data into a list and send it that way.

The remote sensor network communications document I found only mentions sending broadcast and sensor-updates for named global variables, I see no way to send a list. Can you link to some further documentation?

Offline

 

#47 2011-01-14 05:20:55

SHowell
Scratcher
Registered: 2011-01-08
Posts: 27

Re: Scratch and the Kinect

terminator355 wrote:

Hi, I am really interested in this project, but I only have a Mac! Can anyone post a tut on how to get this up and running on Mac?

Sadly, I don't have a Mac, and the code uses .net so it isn't simple to port. I suggest it's about a days work for a good mac C developer to port if there were OpenNI drivers available. Any old pcs that you can use?

Offline

 

#48 2011-01-14 11:18:36

terminator355
Scratcher
Registered: 2008-07-31
Posts: 100+

Re: Scratch and the Kinect

SHowell wrote:

terminator355 wrote:

Hi, I am really interested in this project, but I only have a Mac! Can anyone post a tut on how to get this up and running on Mac?

Sadly, I don't have a Mac, and the code uses .net so it isn't simple to port. I suggest it's about a days work for a good mac C developer to port if there were OpenNI drivers available. Any old pcs that you can use?

Actually, I've got an old PC, and could run windows 2000 on my Mac through boot camp, but that's really old, and I'm not sure it would work that well. Or would it work okay?


No new stuff for now guys  wink  join me for some Black Ops on xbox: termhn or for some rock band on PS3: rocket232

Offline

 

#49 2011-01-14 12:11:49

johnnydean1
Scratcher
Registered: 2010-02-12
Posts: 1000+

Re: Scratch and the Kinect

SHowell wrote:

johnnydean1 wrote:

SHowell wrote:


Well, we have access to the pixel data (what the camera sees) but to send it to Kinect would involve sending each pixel individually, Scratch has no network support yet for blobs of data (that I can see).
Stephen

You could always put the data into a list and send it that way.

The remote sensor network communications document I found only mentions sending broadcast and sensor-updates for named global variables, I see no way to send a list. Can you link to some further documentation?

There is no documentation I am afraid - this forum is probably the expert. What I mean is if you have a list
#( 'R' 'R' 'G' 'B' )
and you turn it into a single string.
"R,R,G,B"
you could send that by the sensor-update to set a (|list| sensor value) block.
You could then use Scratch by saying:
replace item 1 of SCRATCHList with (letter 1 of (|list| sensor value) )
replace item 2 of SCRATCHList with (letter 3 of (|list| sensor value) )
replace item 3 of SCRATCHList with (letter 5 of (|list| sensor value) )
replace item 4 of SCRATCHList with (letter 7 of (|list| sensor value) )

so your ScratchList would have all the list transfered.
I hope this helps.


NOTE:
To send lists to Scratch/Incorperate them into strings, would require modifying Scratch's Source Code Found Here


You can now reach me on Twitter @johnnydean1_

Offline

 

#50 2011-01-15 15:53:43

markyparky56
Scratcher
Registered: 2008-03-20
Posts: 1000+

Re: Scratch and the Kinect

johnnydean1 wrote:

SHowell wrote:

johnnydean1 wrote:

You could always put the data into a list and send it that way.

The remote sensor network communications document I found only mentions sending broadcast and sensor-updates for named global variables, I see no way to send a list. Can you link to some further documentation?

There is no documentation I am afraid - this forum is probably the expert. What I mean is if you have a list
#( 'R' 'R' 'G' 'B' )
and you turn it into a single string.
"R,R,G,B"
you could send that by the sensor-update to set a (|list| sensor value) block.
You could then use Scratch by saying:
replace item 1 of SCRATCHList with (letter 1 of (|list| sensor value) )
replace item 2 of SCRATCHList with (letter 3 of (|list| sensor value) )
replace item 3 of SCRATCHList with (letter 5 of (|list| sensor value) )
replace item 4 of SCRATCHList with (letter 7 of (|list| sensor value) )

so your ScratchList would have all the list transfered.
I hope this helps.


NOTE:
To send lists to Scratch/Incorperate them into strings, would require modifying Scratch's Source Code Found Here

Two things.

1. R[RGB? Why is there two Rs?

2. RGB as far as I know can be upto 3 digits long, e.g. 255-255-255 is pure white and 0-0-0 is pure black.

Last edited by markyparky56 (2011-01-15 15:53:55)


http://j.mp/jgVnTq
Check out my game engine development site: NewDawn I'm a Level 171 Scratcher.I am http://bit.ly/nkvLNT

Offline

 

Board footer