Intelligent Feature Selection (with SVM or PCA)

[rodrigo.constanzo]

@weefuzzy

I just wanted to underline that all the ghastly entrails are there

I’ll have a look with a clear head tomorrow.

@weefuzzy

Unfortunately it scales quite badly with training set size, because the Cunning Trick involves taking the distance (or similar) between every single pair of training points.

Hmm, how poorly does it scale up? The use case here is being applied to descriptors (rows) in the dataset no? So like, 300ish tops?

Or does the SVM also get run across the whole dataset to do this cunning stuff?

@weefuzzy

What PCA is great for is telling whether or not (and, roughly where) your features are already correlated, and giving you a rotation of the space it was trained on that makes the most it can from those correlations in a linear setting.

I should (probably) revisit/retest this stuff, but in my initial experiments a while back, when optimizing for speed (obviously!) I found that having a lower amount of “natural” dimensions was faster (and not significantly less accurate) than having a higher dimensional space, which is then run through some other processes on the way (i.e. PCA).

Granted, that may change if I end up implementing some kind of LPT structure, but given that I don’t really care about pitch (or more importantly, it’s not a very useful descriptor when being run on 256 samples), it may be useful to massage stuff down to equal numbers of descriptors per “thing” that I’m interested in (e.g. loudness, timbre, morphology).

All of that is to say is that this general exploration is to kind of whittle down which (natural) descriptors are the most significant, without transforming them first. There’s also other knock-on/useful effects of this (e.g. having human readable/perceptual dimensions left at the end), but that’s kind of gravy for my purposes.