Ways to test the validity/usefulness/salience of your data

[rodrigo.constanzo]

@tedmoore

One way to get around this is to do some careful book-keeping just once. You could put into a dictionary what all the channels of a buffer (or dimensions of a dataset) are so that later you can just refer to that.

Definitely. I started working out a thing like this ages ago to have a meta/macro dict to keep track of columns, as well as fit data and other metadata. And even suggested having some native way for the objects to output something similar, but it’s all very brittle and fragile if you want to change any single aspect of stuff (say, less MFCCs, different fft settings, larger analysis window, etc…).

@tedmoore

One thing you could do is fitTransform into a new dataset (which now contains your order PCs as dimensions), figure out how many PCs you want to keep based on a scree, then just fluid.datasetquery~ to select the top however many dimensions (PCs) you want. I’m sure this would be faster.

It’s been a bit since I do this in practice, but doesn’t the final step of the PCA/SVM-ish thing give you a list of indices (e.g. 4 6 18 21 22 30 48 61), then in order to build a fluid.dataset~ from that (to feed UMAP) you need to go and fetch those individual columns. I guess there’s perhaps some zl-land munging that can happen there, and since it doesn’t really matter what these columns represent anymore, they could be moved without labels or whatever.

It just seems like a useful process that would benefit from some kind of native-ish solution like telling fluid.umap~:
weights pca.output 90(%), fittransform source.dataset target.dataset

@tedmoore

What I have been doing is looking at the scree plot, finding how many PCs I want to keep based on that, then re-running then FluidPCA and asking for how many I want to keep. It’s a bit of a double take, but eh.

Hmm. I could be remembering wrong, but doesn’t PCA return the same results no matter how many components you ask for? Like the first 5 components of a PCA5 are the same as a PCA500. And that the dump output of PCA contains the whole matrix either way.

@tedmoore

I don’t know of an algorithm that gets at this… and I don’t think that the fluid-verse has a way to attack this. What it sounds like to me is to be able to “pin” a few data points in particular locations in a space (a 2D space for example) and then ask a dimension reduction algorithm such as UMAP to start it’s iterative shuffling of data points to try to optimise it’s cost function (but not letting it move the pinned points!). My guess is that something like this exists, but I don’t know what which algorithm.

Yeah that’s interesting. Again, doable by poking and vibing it, it just gets tricky if you’re trying to orient towards a position that satisfies more than one criteria. Or using that as a jumping off point.