Giving fluid.dataset~ an index as well as a label

New Ideas Pre-Release Toolbox2 New Ideas
1

So I’ve been trying to build test corpora with the -comparison example and on top of other issues I’ve run in to I’m now having a thing where I can’t use my polybuffer~-based workflow since all the labels in the dataset are the path/filename (as generated by the abstraction). Don’t get me wrong, that’s super handy to have, but at the same time, it was also super handy to have indices that I could use to feed a play~ object being fed by a polybuffer~. I can generate a parallel collor something to keep track of indices or hack at the-comparison` abstractions so that it stores indices-as-labels instead of paths-as-labels, but I can definitely see the usefulness of having the latter sometimes.

Even better would be to allow for arbitrary data to be stored in a dataset so any column can be a number, symbol, etc…

The main reason I’m after indices as not paths is that I want a fast(/easy) playback engine, so loading a concatenated version of audio files, or worse, loading each file as I need it, gets really clunky.

2

As we’ve discussed before fluid.dataset~ doesn’t have an inherent ordering, so having a numerical index associated with each point would be a pretty deep change to the data structure. We’re looking at some ways of making a ‘default’ fallback of integer ids easier to work with, seeing as these will suffice in many situations.

As far as using those abstractions goes, can you not just hack them to use different labels in the first place?

I think it’s singularly unlikely that dataset will ever become omnivorous, because the performance and complexity implications are pretty heavy.

1 Like
3

Indeed, it’s mainly this.

Those particular abstractions are fairly complex in terms of what they pass around, so it kind of crosses the threshold of “modding the thing” vs “building a completely different thing”.

I can see that, it just starts getting clunky when you have a dataset for the number crunching, then a dict for your metadata/fits, then a coll for the data you also need, but don’t want to fit to anything etc…