Scratch

bobbybee · 2011-12-11 18:58:54

Barely known, but easy to get information on, is the Scratch File Format. The spec is outdated, but it exists. But what this topic is about is the sprite file format instead. If we had this, we could write programs that would be able to export data to scratch. Imagine the possibilities.

To start, I made a one dot sprite with no scripts, and exported it to a file. From there, I opened the file up. Here is a dump of the file's first 32 bytes in my hex editor.

ObjS.Stch....!}..c....c...c.....
4F 62 6A 53 01 53 74 63 68 01 00 00 00 21 7D 05
15 63 00 00 02 01 63 00 00 03 63 00 00 04 05 00

If you've done any work with scratch files, you recognize at once the first 10 bytes--a scratch object. This gives me the impression that the scratch sprite files are really just sprite objects separated from the rest of the file. The pattern that can be recognized is that the character c keeps coming up. Lower-case c's ascii code is 99, or 63 in hex. The number 99 is significant in base 10, and gives the impression that it means something. A little decompiling of scratch reveals this piece of code.

(fragment, cut for ease of reading)

Code:

fixedFormatClasses
    "Answer an array of records for fixed-format classes."

    ^ #(
        "id    class                    read selector        write selector"
        (35    ColorForm                getForm:id:            putForm:id:)
    "99 reserved for object references"
    "100-255 reserved for user-defined classes"
    )

The interesting part is the note--99 must mean that we are looking at a file full of object references. Weird.

This document is a work in progress, and it needs your help to finish it. If you know something about the Scratch file format's that could possibly help development of this kindly respond.

DISCLAIMER: I am not on the Scratch Team and cannot guarantee that any information is accurate.

-bobbybee

nXIII · 2011-12-11 20:45:37

I'm pretty sure that a .sprite file is just the Sprite serialized with ObjStream. A .sb file is the stage serialized with ObjStream; this contains all the sprites in the "sprites"--and, indirectly, "submorphs"--fields.

The object references are there to preserve identity and store potentially circular structures--for example, a sprite contains a reference to its parent (the stage), and its parent contains a reference to that sprite. If the stage was serialized each time it appeared in an object's field, attempting to serialize it would result in an infinite loop (serialize stage -> serialize child sprite -> serialize parent stage -> ...). With object references, each object is serialized only the first time it is encountered (including an object ID), and if it occurs again, a reference is inserted in its place. When the objects are deserialized, these object references are resolved using a lookup table, so they store references to the same object.

MathWizz · 2011-12-11 21:19:33

http://download.scratch.mit.edu/source- … Format.txt big_smile

bobbybee · 2011-12-12 06:32:00

nXIII, that would explain what those object references are for. MathWizz, I already knew about that part, but this is the individual sprite files, rather then the entire project itself.

Thanks to both of you.
-bobbybee

blob8108 · 2011-12-12 07:59:55

Just in case you missed it, there's the following in the document MathWizz linked:

Code:

5.5 Object References

An object reference allows a field in one object to contain a pointer to another object. It has the following format:

<99: 1 byte constant><object table index: 3 byte big-endian integer>

The value 99 is a reserved classID value used to indicate an object reference. The first object table index is 1, unlike C or Java arrays where the first entry is at index 0.

Example: An object reference to the second entry in the object table is encoded as four bytes: 99, 0, 0, 0, 2.

I think nXIII may be right -- sprites are serialised in a very similar fashion to projects, so that document should have most of what you need.

Last edited by blob8108 (2011-12-12 08:03:00)

bobbybee · 2011-12-12 15:47:15

I'll take a look at that. Thanks everyone.

blob8108 · 2011-12-14 14:30:05

I was working on reverse-engineering some of this stuff, and I made a little wiki to collect my findings. Hope it helps! smile

hsshah · 2011-12-14 15:00:38

Have you tried parsing the Project Summary directly?
I got to know about it as a reply to my post yesterday. Shift-Click on File --> Write Project Summary

blob8108 · 2011-12-14 15:06:06

hsshah wrote:
Have you tried parsing the Project Summary directly?
I got to know about it as a reply to my post yesterday. Shift-Click on File --> Write Project Summary

Thanks! That's interesting... smile That would definitely help for reading project files, but not for writing Scratch files... There's no way of importing the summary hmm

Last edited by blob8108 (2011-12-14 15:06:28)

bobbybee · 2011-12-14 15:56:11

With some squeak hacking, you should be able to read it in.

blob8108 · 2011-12-14 16:15:11

bobbybee wrote:
With some squeak hacking, you should be able to read it in.

I don't think that does the right thing, though. Writing Scratch files seems better, somehow. Also, that way sounds far too easy. tongue

Did you try to join that wiki? If so, I'll hit accept... smile

bobbybee · 2011-12-14 16:22:06

Yeah, I did try to join the wiki. (It clearly could use a little help and I always have expressed an interest in reverse-engineering, so it should be fine)

-bobbybee

blob8108 · 2011-12-14 16:44:37

bobbybee wrote:
Yeah, I did try to join the wiki. (It clearly could use a little help and I always have expressed an interest in reverse-engineering, so it should be fine)

Oh, I'd love the help! Just wanted to make sure 'twas you. smile And reverse-engineering is fun...

bobbybee · 2011-12-14 16:51:20

I know right. Just wondering, what section of the file format should I research. I am working on an app to write sprite files, if it matters.

blob8108 · 2011-12-15 05:02:22

bobbybee wrote:
I know right. Just wondering, what section of the file format should I research. I am working on an app to write sprite files, if it matters.

Sprite files miss out the info table, so they're slightly easier to deal with. I've figured out the formats of most of the fixed-format objects, but not always what they do (like the field called "Form"). There's a lot missing under user-class objects, like the properties stored of each one. It should be easy to find in the Squeak browser, if you look at the storeFieldsOn method under "object i/o" on each object.

nXIII · 2011-12-15 09:26:06

blob8108 wrote:
Sprite files miss out the info table, so they're slightly easier to deal with. I've figured out the formats of most of the fixed-format objects, but not always what they do (like the field called "Form"). There's a lot missing under user-class objects, like the properties stored of each one. It should be easy to find in the Squeak browser, if you look at the storeFieldsOn method under "object i/o" on each object.

I summarized how to decode Forms here.
For user classes, look at the #initFieldsFrom: message in each class's instance methods; they show the order in which their fields are encoded.

blob8108 · 2011-12-15 15:42:01

nXIII wrote:
I summarized how to decode Forms here.

Thanks! I think I'm beginning to understand it. For what I'm trying to do, I think can just parse it as an object with 5/6 fields and ignore its contents; I'm not going to try and decode it just yet tongue

Is it okay, though, if I copy your description to the little wiki I made? Happy to remove it if not. smile

nXIII wrote:
For user classes, look at the #initFieldsFrom: message in each class's instance methods; they show the order in which their fields are encoded.

That's vaguely what I've been doing. I wish there was some automated way to export those lists of fields for each class from Squeak in one go... hmm

Last edited by blob8108 (2011-12-15 15:42:40)

bobbybee · 2011-12-15 15:44:08

So, what should I work on?

blob8108 · 2011-12-15 15:47:55

bobbybee wrote:
So, what should I work on?

Descriptions of the properties of each (or even some) of the user-class objects would be great. smile As nXIII pointed out:

nXIII wrote:
For user classes, look at the #initFieldsFrom: message in each class's instance methods; they show the order in which their fields are encoded.

(It should be under "object i/o" on each object.)

Feel free to edit any of the wiki pages to correct/add/fix/question something — that's what it's for (:

bobbybee · 2011-12-15 15:49:37

On it.

nXIII · 2011-12-15 15:52:50

blob8108 wrote:
nXIII wrote:
For user classes, look at the #initFieldsFrom: message in each class's instance methods; they show the order in which their fields are encoded.
That's vaguely what I've been doing. I wish there was some automated way to export those lists of fields for each class from Squeak in one go...

You can browse implementors of #storeFieldsOn: or #initFieldsFrom:version:; that will give you a list (new workspace -> type "storeFieldsOn:" -> cmd+m or cmd+b)

bobbybee · 2011-12-15 15:54:50

Nice tip, nXIII. That should save me a ton of time.

blob8108 · 2011-12-15 15:58:22

nXIII wrote:
blob8108 wrote:
nXIII wrote:
For user classes, look at the #initFieldsFrom: message in each class's instance methods; they show the order in which their fields are encoded.
That's vaguely what I've been doing. I wish there was some automated way to export those lists of fields for each class from Squeak in one go...
You can browse implementors of #storeFieldsOn: or #initFieldsFrom:version:; that will give you a list (new workspace -> type "storeFieldsOn:" -> cmd+m or cmd+b)

That is so useful. Thank you! smile

bobbybee · 2011-12-15 15:59:01

Also, I've add a description on the wiki about SoundMedia. Are morphs marked with a star morphs that are important and need a description if they don't have one?

blob8108 · 2011-12-15 16:07:38

bobbybee wrote:
Also, I've add a description on the wiki about SoundMedia. Are morphs marked with a star morphs that are important and need a description if they don't have one?

The asterisk is from the original document — I think it just means they're in current use.

Thanks for your help! smile

Last edited by blob8108 (2011-12-15 16:07:57)

Scratch

archived forums

#1 2011-12-11 18:58:54

The .sprite File Format Specifications -- .Sprite Reverse Engineered

Code:

#2 2011-12-11 20:45:37

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#3 2011-12-11 21:19:33

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#4 2011-12-12 06:32:00

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#5 2011-12-12 07:59:55

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

Code:

#6 2011-12-12 15:47:15

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#7 2011-12-14 14:30:05

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#8 2011-12-14 15:00:38

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#9 2011-12-14 15:06:06

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

hsshah wrote:

#10 2011-12-14 15:56:11

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#11 2011-12-14 16:15:11

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

bobbybee wrote:

#12 2011-12-14 16:22:06

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#13 2011-12-14 16:44:37

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

bobbybee wrote:

#14 2011-12-14 16:51:20

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#15 2011-12-15 05:02:22

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

bobbybee wrote:

#16 2011-12-15 09:26:06

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

blob8108 wrote:

#17 2011-12-15 15:42:01

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

nXIII wrote:

nXIII wrote:

#18 2011-12-15 15:44:08

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#19 2011-12-15 15:47:55

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

bobbybee wrote:

nXIII wrote:

#20 2011-12-15 15:49:37

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#21 2011-12-15 15:52:50

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

blob8108 wrote:

nXIII wrote:

#22 2011-12-15 15:54:50

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#23 2011-12-15 15:58:22

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

nXIII wrote:

blob8108 wrote:

nXIII wrote:

#24 2011-12-15 15:59:01

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

#25 2011-12-15 16:07:38

Re: The .sprite File Format Specifications -- .Sprite Reverse Engineered

bobbybee wrote:

Board footer