Loading a dict which from json should be the same as reading the file directly

Given this example why is the top code block not the same as the bottom?

(
~data = File.readAllString(
    thisProcess.nowExecutingPath.dirname +/+ "anal.json"
).parseYAML;

fork{
    ~input = FluidDataSet.new(s, \input);
    s.sync;
    ~input.load(~data)
}
)

(
fork{
    ~input = FluidDataSet.new(s, \input);
    s.sync;
    ~input.read(thisProcess.nowExecutingPath.dirname +/+ "anal.json"); 
}
)

I think that reading a JSON file (made by other FluCoMa objects) from disk into a dictionary and then loading that dictionary should be equivalent to calling .read directly on the object.

This is an SC issue. SC is parsing the json file differently than what FluidDataSet wants.

Sam

1 Like

Is it a bug or a feature do you think?

I donā€™t know, but it doesnā€™t like my ā€œcorrectlyā€ parsed one either. I donā€™t see the difference between d and ~jsonB:

(
~json = File.readAllString("/Users/spluta/Library/Application Support/SuperCollider/Extensions/LiveModularInstrument/modules/NN_Synths/07_SamplePlayer0/model0/datasetTree.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add('cols' -> ~json["cols"]);
~jsonB.add('data' -> Dictionary());
~json["ids"].do{|item, i|
	[item,i].postln;
	~jsonB['data'].add(item.asSymbol -> ~json["data"][i])
};
)

d = Dictionary.with(
        *[\cols -> 2,\data -> Dictionary.newFrom(
            100.collect{|i| [i, [ 1.0.linrand,1.0.linrand]]}.flatten)]);

d['cols']
d['data']
~jsonB['cols']
~jsonB['data']

(
fork{
    ~input = FluidDataSet.new(s, \input);
	
    s.sync;
    ~input.load(~jsonB); 
}
)

That was my intuition. That it should work but doesnā€™t

intuition wonā€™t help here. The best bet is to import d, save it, and try to load it back via your own file parsing (I donā€™t know why you would do that but Iā€™m sure there is a use case).

I see a difference between d and ~jsonB in this code but I might be wrongā€¦ double-quote in the ā€˜dataā€™ to ā€œdataā€ - if I find a minute I will try to do what I said above.

ok I coulnā€™t resist:

d = Dictionary.with(
        *[\cols -> 2,\data -> Dictionary.newFrom(
            100.collect{|i| [i, [ 1.0.linrand,1.0.linrand]]}.flatten)]);

d['cols']
d['data']


(
fork{
    ~input = FluidDataSet.new(s, \input);
	
    s.sync;
    ~input.load(d); 
}
)

~input.write("/tmp/in.json")

(
~json = File.readAllString("/tmp/in.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add('cols' -> ~json["cols"]);
~jsonB.add('data' -> Dictionary());
~json["ids"].do{|item, i|
	[item,i].postln;
	~jsonB['data'].add(item.asSymbol -> ~json['data'][i])
};
)


~jsonB['cols']
~jsonB['data']

this way i make a valid dataset from a valid dict. I save it as json.

when I try to load it the parseYAML way, I get all strange things.

the line:
~jsonB.add(ā€˜colsā€™ -> ~json[ā€œcolsā€]);

works but there is not ~json[ā€œidsā€] upon load so you get nothing in your Data dict and the whole thing goes titsup

so this is what you need:

(
~json = File.readAllString("/tmp/in.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add('cols' -> ~json["cols"]);
~jsonB.add('data' -> ~json["data"]);
};
)

and bobā€™s youā€™re uncleā€¦ but no!

I get an invalid format.

Let me reboot.

Iā€™m doing the quotes on purpose to try to get the exact same looking data. But it ainā€™t working and I am fine just reading the file.

It is strange. I get with my last code a valid format yet Iā€™m told it isnā€™t. I donā€™t see a native way to save the dict so I can compare it, there must be a comma, or something stupid, added with parseYAML. In Max we had to mess up so much for stuff to behave and in SC it worksā€¦ actually I should just read the load code wait a second

the items in the arrays are strings, not floats:

~jsonB[ā€˜dataā€™][ā€œ0000.wavā€][0]-1
vs
~jsonB[ā€˜dataā€™][ā€œ0000.wavā€][0].interpret-1

ok Iā€™ll need to investigate a bit more with a proper brain on. This is surreal because in this code d and ~jsonB are exactly the sameā€¦

Maybe @groma can have a look as it is his code under the hood but I really cannot see the difference between the 2 dicts (they are super small here so I can see them all in the post window!)

d = Dictionary.with(
        *[\cols -> 2,\data -> Dictionary.newFrom(
            4.collect{|i| [i, [ 1.0.linrand,1.0.linrand]]}.flatten)]);

d['cols']
d['data']


(
fork{
    ~input = FluidDataSet.new(s, \input);
	
    s.sync;
    ~input.load(d); 
}
)

~input.write("/tmp/in.json")

(
~json = File.readAllString("/tmp/in.json").parseYAML;

~jsonB = Dictionary();
~jsonB.add(\cols -> ~json["cols"]);
~jsonB.add(\data -> ~json["data"]);

)


(
fork{
    ~input2 = FluidDataSet.new(s, \input2);
	
    s.sync;
    ~input2.load(~jsonB); 
}
)

Ultimately, if you load a JSON file written by a FluidDatasSet then you should be able to either load it directly or put it into a dict (optionally edit it) and then make the dataset load that.

From what I can tell by testing in Max, you donā€™t need the idā€™s key of the dictionary. That is the old school format IIRC.

This works:

~jsonB = Dictionary.with(*[\cols -> ~json[ā€œcolsā€].interpret, \data -> Dictionary.newFrom(~json[ā€œdataā€].keys.collect{|key| [key.asSymbol, ~json[ā€œdataā€][key].collect{|item| item.interpret}]}.asArray.flatten)]);

The issue is that all the data, including the number in cols, are strings.

1 Like

found it. they look the same but they are not the same. check this:

~jsonB[ā€œdataā€][0.asString]
vs
d[ā€œdataā€][0.asString]
vs
d[ā€˜dataā€™][0]

but still I donā€™t get it

PS this was written before I read the clever answer of sam, hence the still pending question, but the casting of labels is still funny to read so Iā€™ll leave it here - it also shows Iā€™m slow :slight_smile:

I was there with you, I just cast the wrong bit! Thanks for solving this! So @jamesbradbury there is no bug, just a casting issue of loading a string as an array of float. You can cast it as @spluta does here.

I still donā€™t know why, but maybe you might want to save all your preset in one gigantic meta dict that you parse and load in various objects. so donā€™t forget to cast your stringā€™d arrays of floats.

Another thing that I discovered in my post just before Samā€™s solution is that the labels seem to be quite permissive (my generating code was not casting them as string) so @groma must have put some stringnification under the hood :slight_smile:

if you just want to save a data set and you donā€™t care about readability, you can use:

myObject.writeArchive
and
Object.readArchive

But those files look funny.

1 Like

I think @jamesbradbury is about to import stuff from Python, so he does care about cross-platform-readability at leastā€¦

Iā€™ve been hassling with string vs. float thing also. itā€™s tricky because for a JSONFileReader to be truly agnostic, it wonā€™t know if you want that info as a string or as a float. For universality, my vote would be to keep things as strings and let the user cast to float when they need to.

But a, perhaps, more important point:

Maybe this is obvious, but for the names of things (such as dictionary keys), it seems like it would be best to just go without the ā€œsymbolsā€, since strings are a more universal type, so more cross-platform friendly (I think currently the FluidFolderLoader stores keys as symbols).

That way whether one loads a ā€œLoaderā€ index from file (keys will be strings!) or makes it in the moment, they keys will all be strings!

Iā€™m a bit behind here as the discussion went deeper into the SC side of things but I disagree that every thing should be a string which is then cast by the user. The JSON loader should be specced such that you can read the data back in and it will parse to the JSON Object that it was in memory. Correct me if Iā€™ve lost track of the thread here.

If itā€™s loading into a Flucoma object, then yes, it should load it as it was saved.

There was also talk of a generic json loader, for which I wouldnā€™t want it assuming my ā€œ0ā€ is a float, it might be an ID. Does JSON protocol have a clever way of specifying type? Kind of like how OSC specifies?