Scratch

DarthPickley · 2012-08-23 18:51:24

What I want to do is to have a script that can go through all my 960 projects, and for each project, find its number of views, love its, favorites, downloads or remixes, and put it into a list of some sort. Then I could use the list to figure out which of my projects has the most favorites or something like that.

I know sort of about the scratch API, but not enough to write a script myself.

Is this possible? Can someone else try making one?

Thanks.

MathWizz · 2012-08-23 20:37:45

PHP?

RAWR WHY CAPS FILTER?? WHY? *inserts lowercase bbcode >:)*

Last edited by MathWizz (2012-08-23 20:38:46)

SJRCS_011 · 2012-08-24 06:46:08

Yes, it is definitely possible, or at least part of it is. I would offer to do it for you, but I wouldn't be able to this weekend, and because school Is starting, it would take a while. Do you still want me to try to make it for you?

ProgrammingFreak · 2012-08-24 06:55:41

Hmm, challenging. It is possible with php, and perhaps javascript.

(I'm going to unofficially play around with this idea.)

Hardmath123 · 2012-08-24 07:22:46

Or Python (I'm learning Python, so it seems like a solution to everything).

scimonster · 2012-08-24 07:31:03

ProgrammingFreak wrote:
Hmm, challenging. It is possible with php, and perhaps javascript.

(I'm going to unofficially play around with this idea.)

JS blocks cross-domain ajax. That's actually the only thing IE is good for, that security risk.

Get a list of all your project IDs (using the API for that), then go through them all, getting the info (using that API), and then sort it however you want.

ProgrammingFreak · 2012-08-24 07:42:40

scimonster wrote:
ProgrammingFreak wrote:
Hmm, challenging. It is possible with php, and perhaps javascript.

(I'm going to unofficially play around with this idea.)
JS blocks cross-domain ajax. That's actually the only thing IE is good for, that security risk.

Dangit. tongue

TheSuccessor · 2012-08-24 08:06:43

To enable cross-domain ajax in Chrome, just launch it with --disable-web-security at the command line or through a shortcut.

ProgrammingFreak · 2012-08-24 08:20:52

Hey, Sci. Did you ever make a documentation for the API? I don't know all the elements to it, so yeah. If not, is there a way I could learn them elsewhere? I know like the userauth and stuff, but not really anything with projects.

MathWizz · 2012-08-24 09:48:48

ProgrammingFreak wrote:
Hey, Sci. Did you ever make a documentation for the API? I don't know all the elements to it, so yeah. If not, is there a way I could learn them elsewhere? I know like the userauth and stuff, but not really anything with projects.

Ctrl + F. Works every time. xD

ProgrammingFreak · 2012-08-24 10:35:13

MathWizz wrote:
ProgrammingFreak wrote:
Hey, Sci. Did you ever make a documentation for the API? I don't know all the elements to it, so yeah. If not, is there a way I could learn them elsewhere? I know like the userauth and stuff, but not really anything with projects.
Ctrl + F. Works every time. xD

xD

roijac · 2012-08-24 10:50:05

Hardmath123 wrote:
Or Python (I'm learning Python, so it seems like a solution to everything).

urllib & beautysoup?

blob8108 · 2012-08-24 12:58:01

Hardmath123 wrote:
Or Python (I'm learning Python, so it seems like a solution to everything).

...because it *is* the solution to everything! big_smile

BeautifulSoup is a rather lovely HTML parsing library for Python (:

roijac wrote:
urllib & beautysoup?

EDIT: Didn't see that... tongue

Last edited by blob8108 (2012-08-24 12:59:21)

DarthPickley · 2012-09-01 17:52:02

Okay guys, thanks for the response, it helped me get much farther.

I started writing something in java (because it is a language I actually know somewhat) and made a script that parsed the results of scratch.mit.edu/api/getprojectsbyusername/DarthPickley into a line-by-line list of project IDs.

I also have found the API source code because of

MathWizz wrote:
Ctrl + F. Works every time. xD

which caused me go on a search of scimonster's past comments regarding the API.

Right now, I want to know if anyone with java experience could tell me the code I could use to prompt something like

Code:

Scanner pageReader = new Scanner( getWebPage( "http://scratch.mit.edu/api/getprojectinfobyid/" + projectIDs[i] ) );
String projectInfo = pageReader.nextLine();
// Parse projectInfo by ":" delimiters
// collect selected information into lists

or something, I don't even know if you are able to do that with java.

But my java code right now is basically a file reader. Thanks.

DarthPickley · 2012-09-01 19:49:03

Okay, I've actually gotten internet working, wasn't that hard actually.

EDIT: now finished with code!

my current java code is:

Code:

 // Scratch project Statistics
// includes loves, faves, downloads, remixes, but not views.
import java.io.*;
import java.util.*;
import java.net.*;

public class DataCollecter {
    static ArrayList<Integer> projectIDs;
    int[] name, favs, loves, views;


    public static void main(String args[])
    {
        projectIDs = read("ProjectID.txt");
    }
    public static ArrayList<Integer> read(String fileName){
        // returns the file contents as an ArrayList
        ArrayList<Integer> arr = new ArrayList<Integer>();
        try {
            Scanner read = new Scanner(new File(fileName));
            while (read.hasNextLine()) {
                int project = read.nextInt();
                arr.add(project);    // add to arr
                Object[] infoList = parseInfo(getData(project)+":").toArray();
                /*     infoList
                 *  0    author_id
                 *  1    name
                 *  2    description
                 *  3    created
                 *  4    tags
                 *  5    country
                 *  6    loveit
                 *  7    num_favoriters
                 *  8    remixer
                 *  9    remixes
                 *  10    numberOfSprites
                 *  11    totalScripts
                 *  12    numberofcomments
                 *  13    numberofdownloads
                 */
                System.out.println( project +"\t\t"+
                        infoList[6] +" loves \t"+
                        infoList[7] +" faves \t"+
                        infoList[9] +" remix \t"+
                        infoList[13] +" dwnlds \t : "+
                        infoList[1]);
            }
        } catch(Exception e){
            e.printStackTrace();
            //    System.out.println("there was error! no find file!");
        }
        return arr;
    }

    public static String getData(int ID)
    {
        String data = "";
        try
        {
            URL url = new URL("http://scratch.mit.edu/api/getprojectinfobyid/" + ID);
            URLConnection connection = url.openConnection();
            connection.setDoInput(true);
            InputStream inStream = connection.getInputStream();
            BufferedReader input = new BufferedReader(new InputStreamReader(inStream));

            String line = "";
            while ((line = input.readLine()) != null)
                data += line;
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
        return data;
    }

    public static ArrayList<String> parseInfo(String line) {
        ArrayList<String> parsed = new ArrayList<String>();

        int ind=0;
        while (ind < line.length()-1) {
            String item = line.substring(ind,line.indexOf(":",ind));
            if (item.equals(""))
                parsed.add("0");
            else
                parsed.add(item);
            ind = line.indexOf(":",ind)+1;
        }
        return parsed;
    }
}

This code resulted in a nice list that took a few minutes to create (with my almost 1000 projects)
and it was obvious which of them have been front-paged, by the loves, faves, and downloads in the double-digits.

Still, I might yet work on adding a script that automatically gets a list of the project IDs based on the person's Username.

Still, there are no views. sad

Last edited by DarthPickley (2012-09-01 21:44:47)

DarthPickley · 2012-09-02 13:43:02

I now know my project with most favorites, and downloads,
and also my project with the most love-its

Code:

948566         139 loves     116 faves     9 remix     656 dwnlds      : Beaker%20Sim%203
...
1678269        146 loves     87 faves     10 remix     297 dwnlds      : Flash%20Player%20test%3A%203d%20computer%20desk

I can also make it print only the projects that have a number of love-its above some value, for instance 20 (usually projects of mine with over 20 love-its have been significant, projects that get on the front page or that have been large projects I was working on.)
this gives me

Code:

 Projects with 20+ loves :
2737317        28 loves      11 faves      1 remix      58 dwnlds       : BitPoly
1714932        112 loves     81 faves      2 remix      339 dwnlds      : Wheels%20demo%201.5
1678269        146 loves     87 faves      10 remix     297 dwnlds      : Flash%20Player%20test%3A%203d%20computer%20desk
1429041        44 loves      23 faves      1 remix      84 dwnlds       : Spring
1073314        34 loves      26 faves      2 remix      100 dwnlds      : MARS%202.6%20FINAL
948566         139 loves     116 faves     9 remix      656 dwnlds      : Beaker%20Sim%203
860484         32 loves      24 faves      0 remix      71 dwnlds       : MARS%3F%3F%3F
859113         30 loves      24 faves      1 remix      91 dwnlds       : MAZE%20beta%201.0
858007         30 loves      4 faves       0 remix      55 dwnlds       : scrolling%20maze%20demo
827702         27 loves      24 faves      1 remix      100 dwnlds      : Rainbow%20Galaxy%202.0
690156         40 loves      38 faves      6 remix      82 dwnlds       : ChipRoll%27d
653500         124 loves     85 faves      15 remix     386 dwnlds      : Rainbow%20Galaxy
555174         47 loves      17 faves      6 remix      137 dwnlds      : AI%20Snake%20vs.%20Worm%20Competition%21
326350         52 loves      38 faves      2 remix      156 dwnlds      : Wave%20Harmonics%20v1

After I get it to do this for any username, I have another idea for where to go with this...

maybe creating a graph of 2-5 levels of my friends, to see how closely linked the community is.

DarthPickley · 2012-09-02 17:42:02

Finished with the program! enter your own username in order to get a report of your projects!
https://www.dropbox.com/s/28sv23avu1qxd … ecter.java

LS97 · 2012-09-04 11:44:31

Seems I came a couple days too late: the Scratch Team developed such tool at stats.scratch.mit.edu. It's not that exact URL though, but there was a post about "personalised stats" in the Announcements forum...

playzooki · 2012-09-14 14:48:57

How would i download the file? I cant find a way. D:

Magnie · 2012-09-14 17:13:45

LS97 wrote:
Seems I came a couple days too late: the Scratch Team developed such tool at stats.scratch.mit.edu. It's not that exact URL though, but there was a post about "personalised stats" in the Announcements forum...

I think they actually took that down due to bugs or Scratch 2.0 problems.

Hardmath123 · 2012-09-14 21:31:25

DarthPickley wrote:
maybe creating a graph of 2-5 levels of my friends, to see how closely linked the community is.

Hey, I was thinking about that: http://scratch.mit.edu/forums/viewtopic.php?id=105712

zubblewu · 2012-09-14 23:51:35

How do you actually use it? Please forgive my faulty memory tongue

LS97 · 2012-09-15 09:49:57

Magnie wrote:
LS97 wrote:
Seems I came a couple days too late: the Scratch Team developed such tool at stats.scratch.mit.edu. It's not that exact URL though, but there was a post about "personalised stats" in the Announcements forum...
I think they actually took that down due to bugs or Scratch 2.0 problems.

Oh, they did? That's a pity, I loved checking my stats every once in a while!

Magnie · 2012-09-15 10:46:14

LS97 wrote:
Magnie wrote:
LS97 wrote:
Seems I came a couple days too late: the Scratch Team developed such tool at stats.scratch.mit.edu. It's not that exact URL though, but there was a post about "personalised stats" in the Announcements forum...
I think they actually took that down due to bugs or Scratch 2.0 problems.
Oh, they did? That's a pity, I loved checking my stats every once in a while!

After looking into it more, I guess it still exists: http://stats.scratch.mit.edu/community

Scratch

archived forums

#1 2012-08-23 18:51:24

Scripts for getting project data

#2 2012-08-23 20:37:45

Re: Scripts for getting project data

#3 2012-08-24 06:46:08

Re: Scripts for getting project data

#4 2012-08-24 06:55:41

Re: Scripts for getting project data

#5 2012-08-24 07:22:46

Re: Scripts for getting project data

#6 2012-08-24 07:31:03

Re: Scripts for getting project data

ProgrammingFreak wrote:

#7 2012-08-24 07:42:40

Re: Scripts for getting project data

scimonster wrote:

ProgrammingFreak wrote:

#8 2012-08-24 08:06:43

Re: Scripts for getting project data

#9 2012-08-24 08:20:52

Re: Scripts for getting project data

#10 2012-08-24 09:48:48

Re: Scripts for getting project data

ProgrammingFreak wrote:

#11 2012-08-24 10:35:13

Re: Scripts for getting project data

MathWizz wrote:

ProgrammingFreak wrote:

#12 2012-08-24 10:50:05

Re: Scripts for getting project data

Hardmath123 wrote:

#13 2012-08-24 12:58:01

Re: Scripts for getting project data

Hardmath123 wrote:

roijac wrote:

#14 2012-09-01 17:52:02

Re: Scripts for getting project data

MathWizz wrote:

Code:

#15 2012-09-01 19:49:03

Re: Scripts for getting project data

Code:

#16 2012-09-02 13:43:02

Re: Scripts for getting project data

Code:

Code:

#17 2012-09-02 17:42:02

Re: Scripts for getting project data

#18 2012-09-04 11:44:31

Re: Scripts for getting project data

#19 2012-09-14 14:48:57

Re: Scripts for getting project data

#20 2012-09-14 17:13:45

Re: Scripts for getting project data

LS97 wrote:

#21 2012-09-14 21:31:25

Re: Scripts for getting project data

DarthPickley wrote:

#22 2012-09-14 23:51:35

Re: Scripts for getting project data

#23 2012-09-15 09:49:57

Re: Scripts for getting project data

Magnie wrote:

LS97 wrote:

#24 2012-09-15 10:46:14

Re: Scripts for getting project data

LS97 wrote:

Magnie wrote:

LS97 wrote:

Board footer