On feature scaling/data sparseness (in a specific spectral context)

Usage Questions Pre-Release Toolbox2 Usage
21

The documentation isn’t anywhere near complete yet, as you’re discovering. FWIW, I don’t think this feature is working as designed at the moment (I have passed this on as a result of you raising), so if I were you I wouldn’t invest more time trying to understand what it’s doing on this release.

However, in lieu of finished documentation, here are the answers for when this feature is working:

if the early stopping is my responsibility of that of the object

The object

when the validation data is used and what for

During each iteration of training, to see if the updated weights also show improved loss against unseen data

how I access any values related to validation and use them

They are some randomly selected fraction (which you specify) of the dataset you passed to fit

Fortunately it’s the same pattern for all the non-audio objects currently. The rightmost is where the action is for all TB2 objects, and will emit the selector name for whatever message just finished. The middle is the progress outlet (not applicable here). The left is a bang out (not applicable here)

Soon the redundant outlets will be magiced away.

22

Thanks - that’s all useful - I’m likely validating by hand for now, so I will set it to zero.

If you can be sure to be clear on when outputs disappear that will be helpful as obviously patches will break, although it should generate a max error, so that is always easier to trace.

23

Outlets won’t disappear without warning, and certainly not in the immediate horizon.

24

Hi yes the validation is broken, will be fixed soon.

25

Thanks - what is the criterion (or criteria) for early stopping, and will it be controllable?

26

I believe the criterion is that the training loop will bail if the model loss w/r/t the reserved validation data fails to improve for five consecutive iterations. The number of iterations isn’t exposed as a parameter.

27

Thanks - I guess I was wondering if there was a threshold on “fails to improve”, rather than iterations, but perhaps this is simply a true or false scenario in terms of the exact number of correct classifications.

28

It’s straightforwardly boolean: is the model loss this time smaller than last time? The loss is something like the mean square difference between the model prediction and the training example.

Gerard will know much more than I about how common this approach is, and what others may be available elsewhere.