Loading a dict which from json should be the same as reading the file directly

[jamesbradbury]

Given this example why is the top code block not the same as the bottom?

(
~data = File.readAllString(
    thisProcess.nowExecutingPath.dirname +/+ "anal.json"
).parseYAML;

fork{
    ~input = FluidDataSet.new(s, \input);
    s.sync;
    ~input.load(~data)
}
)

(
fork{
    ~input = FluidDataSet.new(s, \input);
    s.sync;
    ~input.read(thisProcess.nowExecutingPath.dirname +/+ "anal.json"); 
}
)

I think that reading a JSON file (made by other FluCoMa objects) from disk into a dictionary and then loading that dictionary should be equivalent to calling .read directly on the object.

[spluta]

This is an SC issue. SC is parsing the json file differently than what FluidDataSet wants.

Sam

[jamesbradbury]

Is it a bug or a feature do you think?

[spluta]

I don’t know, but it doesn’t like my “correctly” parsed one either. I don’t see the difference between d and ~jsonB:

(
~json = File.readAllString("/Users/spluta/Library/Application Support/SuperCollider/Extensions/LiveModularInstrument/modules/NN_Synths/07_SamplePlayer0/model0/datasetTree.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add('cols' -> ~json["cols"]);
~jsonB.add('data' -> Dictionary());
~json["ids"].do{|item, i|
	[item,i].postln;
	~jsonB['data'].add(item.asSymbol -> ~json["data"][i])
};
)

d = Dictionary.with(
        *[\cols -> 2,\data -> Dictionary.newFrom(
            100.collect{|i| [i, [ 1.0.linrand,1.0.linrand]]}.flatten)]);

d['cols']
d['data']
~jsonB['cols']
~jsonB['data']

(
fork{
    ~input = FluidDataSet.new(s, \input);
	
    s.sync;
    ~input.load(~jsonB); 
}
)

[jamesbradbury]

That was my intuition. That it should work but doesn’t

[tremblap]

intuition won’t help here. The best bet is to import d, save it, and try to load it back via your own file parsing (I don’t know why you would do that but I’m sure there is a use case).

I see a difference between d and ~jsonB in this code but I might be wrong… double-quote in the ‘data’ to “data” - if I find a minute I will try to do what I said above.

[tremblap]

ok I couln’t resist:

d = Dictionary.with(
        *[\cols -> 2,\data -> Dictionary.newFrom(
            100.collect{|i| [i, [ 1.0.linrand,1.0.linrand]]}.flatten)]);

d['cols']
d['data']


(
fork{
    ~input = FluidDataSet.new(s, \input);
	
    s.sync;
    ~input.load(d); 
}
)

~input.write("/tmp/in.json")

(
~json = File.readAllString("/tmp/in.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add('cols' -> ~json["cols"]);
~jsonB.add('data' -> Dictionary());
~json["ids"].do{|item, i|
	[item,i].postln;
	~jsonB['data'].add(item.asSymbol -> ~json['data'][i])
};
)


~jsonB['cols']
~jsonB['data']

this way i make a valid dataset from a valid dict. I save it as json.

when I try to load it the parseYAML way, I get all strange things.

the line:
~jsonB.add(‘cols’ -> ~json[“cols”]);

works but there is not ~json[“ids”] upon load so you get nothing in your Data dict and the whole thing goes titsup

so this is what you need:

(
~json = File.readAllString("/tmp/in.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add('cols' -> ~json["cols"]);
~jsonB.add('data' -> ~json["data"]);
};
)

and bob’s you’re uncle… but no!

I get an invalid format.

Let me reboot.

[spluta]

I’m doing the quotes on purpose to try to get the exact same looking data. But it ain’t working and I am fine just reading the file.

[tremblap]

It is strange. I get with my last code a valid format yet I’m told it isn’t. I don’t see a native way to save the dict so I can compare it, there must be a comma, or something stupid, added with parseYAML. In Max we had to mess up so much for stuff to behave and in SC it works… actually I should just read the load code wait a second

[spluta]

the items in the arrays are strings, not floats:

~jsonB[‘data’][“0000.wav”][0]-1
vs
~jsonB[‘data’][“0000.wav”][0].interpret-1

[tremblap]

ok I’ll need to investigate a bit more with a proper brain on. This is surreal because in this code d and ~jsonB are exactly the same…

Maybe @groma can have a look as it is his code under the hood but I really cannot see the difference between the 2 dicts (they are super small here so I can see them all in the post window!)

d = Dictionary.with(
        *[\cols -> 2,\data -> Dictionary.newFrom(
            4.collect{|i| [i, [ 1.0.linrand,1.0.linrand]]}.flatten)]);

d['cols']
d['data']


(
fork{
    ~input = FluidDataSet.new(s, \input);
	
    s.sync;
    ~input.load(d); 
}
)

~input.write("/tmp/in.json")

(
~json = File.readAllString("/tmp/in.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add(\cols -> ~json["cols"]);
~jsonB.add(\data -> ~json["data"]);

)


(
fork{
    ~input2 = FluidDataSet.new(s, \input2);
	
    s.sync;
    ~input2.load(~jsonB); 
}
)

[jamesbradbury]

Ultimately, if you load a JSON file written by a FluidDatasSet then you should be able to either load it directly or put it into a dict (optionally edit it) and then make the dataset load that.

From what I can tell by testing in Max, you don’t need the id’s key of the dictionary. That is the old school format IIRC.

[spluta]

This works:

~jsonB = Dictionary.with(*[\cols -> ~json[“cols”].interpret, \data -> Dictionary.newFrom(~json[“data”].keys.collect{|key| [key.asSymbol, ~json[“data”][key].collect{|item| item.interpret}]}.asArray.flatten)]);

The issue is that all the data, including the number in cols, are strings.

[tremblap]

found it. they look the same but they are not the same. check this:

~jsonB[“data”][0.asString]
vs
d[“data”][0.asString]
vs
d[‘data’][0]

but still I don’t get it

PS this was written before I read the clever answer of sam, hence the still pending question, but the casting of labels is still funny to read so I’ll leave it here - it also shows I’m slow

[tremblap]

I was there with you, I just cast the wrong bit! Thanks for solving this! So @jamesbradbury there is no bug, just a casting issue of loading a string as an array of float. You can cast it as @spluta does here.

I still don’t know why, but maybe you might want to save all your preset in one gigantic meta dict that you parse and load in various objects. so don’t forget to cast your string’d arrays of floats.

Another thing that I discovered in my post just before Sam’s solution is that the labels seem to be quite permissive (my generating code was not casting them as string) so @groma must have put some stringnification under the hood

[spluta]

if you just want to save a data set and you don’t care about readability, you can use:

myObject.writeArchive
and
Object.readArchive

But those files look funny.

[tremblap]

I think @jamesbradbury is about to import stuff from Python, so he does care about cross-platform-readability at least…

[tedmoore]

I’ve been hassling with string vs. float thing also. it’s tricky because for a JSONFileReader to be truly agnostic, it won’t know if you want that info as a string or as a float. For universality, my vote would be to keep things as strings and let the user cast to float when they need to.

But a, perhaps, more important point:

Maybe this is obvious, but for the names of things (such as dictionary keys), it seems like it would be best to just go without the “symbols”, since strings are a more universal type, so more cross-platform friendly (I think currently the FluidFolderLoader stores keys as symbols).

That way whether one loads a “Loader” index from file (keys will be strings!) or makes it in the moment, they keys will all be strings!

[jamesbradbury]

I’m a bit behind here as the discussion went deeper into the SC side of things but I disagree that every thing should be a string which is then cast by the user. The JSON loader should be specced such that you can read the data back in and it will parse to the JSON Object that it was in memory. Correct me if I’ve lost track of the thread here.

[tedmoore]

If it’s loading into a Flucoma object, then yes, it should load it as it was saved.

There was also talk of a generic json loader, for which I wouldn’t want it assuming my “0” is a float, it might be an ID. Does JSON protocol have a clever way of specifying type? Kind of like how OSC specifies?