Removing outliers while creating classifiers

[rodrigo.constanzo]

Mucho helpful response!

@weefuzzy

So, thing with ‘outliers’ is that (barring an actual objective definition) the problem they create is generally an overfitting one: i.e as atypical points in the training data (that you wouldn’t expect to see in test or deployment), they exert too great an influence on the model training, making it less generally useful.

Yup yup. I think for the general classification stuff I’ve been doing it’s worked well enough as the sounds are, generally-speaking, distinct enough that the small amount of junk in there doesn’t really mess things up. Obviously cleaner and more accurate stuff would be better there, but it was mainly when working on interpolation (between classes) more recently, I felt like something like this was necessary since I’m basically “interpolating” by navigating nearest neighbors, so junk around the edges of the zones has a bigger impact.

@weefuzzy

…certain magic numbers can do the job.

Exciting! Now to find out what those magic numbers are…

@weefuzzy

there is no such general method that doesn’t involve at least some thinking about what ‘outlier’ means in a given case. Rank order statisical things like median and IQR aren’t a magic bullet.

I did think about this after making this post a bit, as there could be cases where the data is all nice and tidy with nothing out of line (very unlikely obviously), so the “outlier” becomes a bit more conceptual. I imagine this is comfortably in the "it depends"™ territory, but is throwing out 5-10% of the entries of every class based on a more generic metric (distance from mean/median or something) “bad”? (for context, I’m often giving around 50-75 examples per class, though this can be as low as 10-15 with quicker trainings)

I’m certain there will be nuance in refining things passed that point, but at the moment I’m doing nothing, and surely doing something is better than doing nothing…

@weefuzzy

there will be a certain amount of dumping and iterating whatever you do, and that’s probably unavoidable until a communal ‘we’ zero in on some approaches to this that make sense for musical workflows, and we can abstract stuff away…

I imagined this would be the case. Boy howdie is it faffy to do something to a dataset based on the information in a labelset! I held out some small hope that it would be possible to hijack robustscale’s output to prune things rather than rescaling things and call it a day.

@weefuzzy

That said, some of your colleagues in PRISM have started rolling some evaluation stuff for themselves – maybe they can be persuaded to share?

Cool, have dropped Sam (Salem) a line to see what’s up. I am intrigued.