Adding a symbolic layer to fluid.datasets~

[rodrigo.constanzo]

I could have sworn I posted about this before, but did a fairly thorough search, and other than a mention in passing in other threads, there’s not a solid thread/request for it.

I’ve presently been squinting at my screen for like 20min, jumping between help files, frantically printing fluid.dataset~s to the Max window, all to try and figure out descriptors/stats are presently in my flattened buffer.

I have this analysis chain here:

which is producing a 98d space of 7 spectral descriptors, along with 7 statistics and one derivative.

I want to be able to produce a coll that has labels for what’s in each point in the vector, so I can test some SVM/PCA shenanigans on it, but for the life is it harder than it seems. I feel like I’m re-reading the same sentence over and over in a book and getting nowhere with it. I can fluid.buf2list the @stats buffer and kind of make sense from that, but once it’s flattened it turns into a multidimensional “indices math” problem. (I’ve not even started doing the 20MFCC or 40melband equiv of this proces…).

Given that buffers are 1d, there sadly isn’t a way to attach symbolic information to each step of the process at source, so it would be great to have something like fluid.labelset~ but that wasn’t so narrow in it’s usability (e.g. a “black box”(i.e. it would be great to create a “label set” that contained information about what was in a “data set” (both in terms of rows as well as columns (my main point of confusion here))).

So it’s kind of a two part request here. One is having some kind of native way to do this (like adding “labels” to “data”), and the other one is being able to programmatically generate this information in the first place. As in, having some kind of trace and/or legible pattern through the descriptors->stats->flatten pipeline such that you know what is where without having to pull out a piece of paper and draw a diagram.

This would also be incredibly handy if it ever becomes possible to pick/choose what descriptors/statistics you want in the first place as you would presumably need some way to specify what data you want in the first place, or at minimum, an easier way to select what data gets carried on in the process (that hopefully/mercifully doesn’t involve counting unlabelled indices…(presently lower than buffer~s-as-containers in my list of least favorite things)).

[weefuzzy]

If I follow, you’re suggesting

1. column labels in datasets
2. something for buf* processors that would provide channel / index labels

But moreover (and implicitly) some idea that these would propagate between analyses (?) e.g.

• you take a bunch of (possibly multichannel) spectral shape descriptors and some MFCCs, smoosh them together and then run stats on stuff,
• by the end of which the labelling fairy has kept track of what’s happened and is able to provide a lookup that knows that (say) channel 0, frame 0 of something is the mean of MFCC 0
• and, moreover, that when then flattening a 2D analysis buffer, the same labelling fairy is able to track this and know knows that frame 42 (say) is the median of MFCC-whatever’s 1st derivative (or something)

Which, computationally, is all simple enough but interface-wise is far from it, in terms of not adding overly to the cost / complexity of analysis chains, or unintentionally restricting flexibility. As usual, the sort of thing that would be interesting to workshop through wrapper abstractions, and see where it leads.

[rodrigo.constanzo]

@weefuzzy

column labels in datasets

That would be great if they were intrinsic to datasets. They can be ignored for “long flat generic all numbers” matching, but if you’re wanting to know and/or thin things out, it would be unimaginably game changing.

@weefuzzy

something for buf* processors that would provide channel / index labels

For the descriptor-based stuff, either the ability to choose what you want so it’s less of an issue in the first place (@output centroid → @output mean, so you have one number and that’s that) or perhaps a dump output/message that lets you know what’s what. This becomes more useful/significant when you get to multichannel audio and/or the stats/flattening stages.

@weefuzzy

by the end of which the labelling fairy has kept track of what’s happened and is able to provide a lookup that knows that (say) channel 0, frame 0 of something is the mean of MFCC 0

Don’t threaten me with a good time!

I guess what I’m saying is at the moment this stuff is being kept track of anyways. It’s just via channels and indices, which are unpleasant for (most) humans to parse. It’s not like each of these processes (descriptors->stats->select->flatten) are putting numbers in random orders (which would still work in an ML context). The order/position/orientation are all known. I guess I’m craving a way to know what that is that doesn’t require counting indices.

@weefuzzy

Which, computationally, is all simple enough but interface-wise is far from it, in terms of not adding overly to the cost / complexity of analysis chains, or unintentionally restricting flexibility.

A lot of the interface problems emerge from the fact that buffers (and datasets) can only contain numbers, so the interface “problems” have to work around that (initial) interface decision. Otherwise it would be (potentially) part of the native data structure.

That being said, this can be something that is offered as a flag or as part of an initial dataset creation (and perhaps ignored for buf-based processes, unless you use a dump message as I suggest above, which can just not do for realtime use).