# FR: adding a 'fit stop' message to regressors

**URL:** <https://discourse.flucoma.org/t/fr-adding-a-fit-stop-message-to-regressors/2484>\
**Category:** New Ideas\
**Created:** [December 20, 2024, 7:11pm UTC](https://discourse.flucoma.org/t/fr-adding-a-fit-stop-message-to-regressors/2484 "2024-12-20T19:11:28Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [January 28, 2025, 11:14pm UTC](https://discourse.flucoma.org/t/fr-adding-a-fit-stop-message-to-regressors/2484/8 "2025-01-28T23:14:28Z")

</div>

> [@weefuzzy](#):
>
> So: the early stopping criterion is against the error on held out _validation_ data rather than the training data (if `@validation` \> 0), so there isn’t actually a simple way to implement early stopping outside the object: one would have to manually make and test against a validation set.
> 
> That’s not ideal, we should consider adding an option to report the validation loss as well, which is lowish hanging fruit. More complicated would be to have something `fit-like` that actually persists the whole relevant state between calls – tracking the validation loss, but also using the same partitions of the data.

So as I mentioned in [this other thread](https://discourse.flucoma.org/t/spectralshape-rolloff-5-vs-rolloff-95/), been working on a project with [Jordie Shier](https://jordieshier.com) where we’ve been training up a load of NNs to predict longer time series from shorter ones. Basically a much more fleshed out version of some of the stuff I’ve been doing in SP-Tools where you can take a 256 sample window and predict the descriptors for a 4410 sample window.

Long story short, in order for this to work really well, Jordie ended up computing these in Python rather than in Max, and when plotting the loss how well different approach did, it was clear that overfitting was a big issue with the Max approach.

Here’s a spreadsheet showing what’s up:

 ![Screenshot 2025-01-28 at 11.05.24 PM](https://discourse.flucoma.org/uploads/default/original/2X/c/c8499251989706768deed177ca6f7225f6551d05.png)

So the rows are different audio files and example subsets, and the columns are different engines training up NNs. B and C are `fluid.mlpregressor~` and D is Jordie’s Python-based implementation of the same code (matches, but not 100% numerically). The biggest difference is in the column G where it’s night-and-day difference.

So all of that is to say that being able to parse the _validation_ loss while iterating through a training set would be game changing on the viability of training up a chonky NN in Max (without making a single large run and pinwheeling Max).

I guess having an option to keep the partitioning across `iter` calls would be great too, but that just ends up doing some cross-validation which isn’t ideal, but also not terrible. Particular as compared to the danger of overfitting.

I don’t know enough about how the guts of the algorithm are running, but presumably it can just dump a message prepended with `validationloss` or maybe just `loss`, in addition to the `fit` message it sends out.

edit: Also re-read [this useful thread](https://discourse.flucoma.org/t/mlpregressor-training-versus-validation-loss/) where some of this is unpacked, but in the context of SC where I guess getting the validation loss in an ongoing manner is possible.

---

_[View the full topic](https://discourse.flucoma.org/t/fr-adding-a-fit-stop-message-to-regressors/2484)._