# Onset-based regression (JIT-MFCC example)

**URL:** <https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464>\
**Category:** Pre-Release Toolbox2 New Ideas\
**Created:** [April 26, 2020, 2:51pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464 "2020-04-26T14:51:09Z")\
**Posts on this page:** 10\
**Page:** 3

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [May 3, 2020, 9:41am UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/41 "2020-05-03T09:41:32Z")

</div>

and again, XKCD has the answer with Friday’s graph:

> **[Turtle Sandwich Standard Model](https://xkcd.com/2301/)**
>
> It's possible the bread and shell can be split into a top and bottom flavor, and some models additionally suggest Strange Bread and Charm Shells.

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [May 4, 2020, 5:45pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/42 "2020-05-04T17:45:50Z")

</div>

> [@tremblap](#):
>
> Now if you want to see your data, you have a problem that we all have: visualising 96D in 2D. see below for some examples on how imperfect that is.

Yeah that’s a bummer that. I guess with data reduction you (potentially) lose some of how differentiated things are.

> [@tremblap](#):
>
> [This thread](https://discourse.flucoma.org/t/a-blackbox-free-new-tool/326) is about the app I was talking about. [This](https://discourse.flucoma.org/t/edit-no-more-confidential-sharing-experimental-results-on-descriptor-space-dimension-reduction-experiment/161) and [this](https://discourse.flucoma.org/t/a-dimensionality-reduction-visualisation/391) threads are about other visualisation possibilities.

Sadly this appears fucked. I downloaded it (AudioStellar) and set it up, and it just sits on 0% initializing or gives me an “error path not found” whenever I try to load samples.

@jamesbradbury’s example isn’t online anymore either.

I created a mini set of samples so there are 20 of each of center and edge hits, all 512 samples long.  
[tiny samples.zip](https://discourse.flucoma.org/uploads/short-url/2EDeTDxcHb9Ha1IBQxIRYci2Z9v.zip) (44.1 KB)

> [@tremblap](#):
>
> Again, as we discussed, if you try more segregated values (further in the space, very different) as your training data, or another approach is to train the ambiguous space. I’ve never tried either but you can now.

Hmmm. At the moment I’m getting pretty solid differentiation when I train sounds that are actually different (e.g. center and rim), but for most of the sounds that are actually different, there’s not really a middle ground or ambiguous space as there’s a different surface involved. I did try training hitting the rim with nearer the tip of the stick vs nearer the shoulder, and that worked without any mistakes. I didn’t test to see where that difference started though.

The center to edge, I guess, is the hardest one to tell, in terms of regular snare hits, since they sound the most similar. Perhaps this is not the case, but my thinking is that if I can get those working smoothly, the rest will be a piece of cake.

---

<div class="post-metadata">

**Author:** ![jamesbradbury](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/jamesbradbury/32/1252_2.png) [@jamesbradbury](https://discourse.flucoma.org/u/jamesbradbury)\
**Post date:** [May 4, 2020, 6:06pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/43 "2020-05-04T18:06:35Z")

</div>

> [@rodrigo.constanzo](#):
>
> @jamesbradbury’s example isn’t online anymore either.

I plan to rectify this soon as its going to be part of a potential journal submission. Stay tuned…

---

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [May 4, 2020, 6:26pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/44 "2020-05-04T18:26:02Z")

</div>

> [@rodrigo.constanzo](#):
>
> Yeah that’s a bummer that. I guess with data reduction you (potentially) lose some of how differentiated things are.

I recommend reading [this paper](https://pure.hud.ac.uk/en/publications/adaptive-mapping-of-sound-collections-for-data-driven-musical-int) we published under the indisputable authority of @groma … or just look at the graphs, it’ll tell you what you want to know: indeed you always loose and the various reductions will give you various pros and cons, various affordances, hence proposing in that paper to compare them and see which one fits your needs. As @weefuzzy said, 'It depends’™

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [May 12, 2020, 10:55am UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/45 "2020-05-12T10:55:18Z")

</div>

I was reminded today of the [oldschool NMF-as-KNN approach](https://discourse.flucoma.org/t/pre-processing-for-training-for-real-time-nmf/145) I was experimenting with early in the process, and came across [this post](https://discourse.flucoma.org/t/pre-processing-for-training-for-real-time-nmf/145/19) which shows how effective `@filterupdate 1` was at refining the filters and selections.

> [@Pre-processing for (Training for real-time NMF)](https://discourse.flucoma.org/t/pre-processing-for-training-for-real-time-nmf/145/19):
>
> First, a little bump (for @tremblap) about extracting an IR from an audio file, as it would be good to revisit the filtering aspect of this process. And more excitingly, the new(ly fixed) @filterupdate 1 is having some nice results on my dicts. Just the raw @ranks: And some @filterupdate 1 with @iterations 1000: I still haven’t tested the matching itself (for reasons expressed elsewhere), but this is a promising step. I also want to test out what @groma suggested in …

Pre `@filterupdate 1`:  
 ![c33695207779208bcf68c9d3b43094a81ca042c7_2_397x500](https://discourse.flucoma.org/uploads/default/original/1X/6fb809a42866ee7e088154cecd9d136b4f3776b0.jpeg)

Post `@filterupdate 1`:  
 ![16237701609a34a8fc6c56ff514caee296d60960_2_397x500](https://discourse.flucoma.org/uploads/default/original/1X/e372d9b568e1fde2f39342d7298ba2f8d8851131.png)

I’m (fairly) certain this is a naive question, which is probably not possible given the algorithms at play, but is there a way to train some points, and then run arbitrary audio through to have it refine the selections based on the input?

Obviously one can add more points to each classification, but perhaps something like this could help catch or fix the edge cases. (Would be amazing if there was a 2d view where the arbitrary audio that it was fed is mapped, and you could go through and say “this point is A, this point is B” and have the network update accordingly.

---

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [May 12, 2020, 1:18pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/46 "2020-05-12T13:18:12Z")

</div>

> [@rodrigo.constanzo](#):
>
> I’m (fairly) certain this is a naive question, which is probably not possible given the algorithms at play, but is there a way to train some points, and then run arbitrary audio through to have it refine the selections based on the input?

Yes you can with machine learning. Soon. You train, add points, train with problem point and/or tweaking parameters. @weefuzzy will confirm but this is what is happening under the hood anyway in nmf, reducing the error by iteratively training… @groma and him keep telling me that nmf is a sort of unsupervised yet seedable classifier… now they will correct me here if I say stupid(er than usual) things

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [May 12, 2020, 1:42pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/47 "2020-05-12T13:42:02Z")

</div>

Great. I’ll wait on the next release to push this further, or rather, refine this further.

It just seemed like an obvious way to refine the selections without “start over and hope for the best”.

---

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [May 12, 2020, 1:59pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/48 "2020-05-12T13:59:08Z")

</div>

> [@rodrigo.constanzo](#):
>
> I’ll wait on the next release

maybe the one after. We are devising packages. stay tuned!

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [June 27, 2020, 12:38pm UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/49 "2020-06-27T12:38:49Z")

</div>

I decided to revisit this, as I’ve learned a bunch from the melbands range and smoothing/reduction in [this thread](https://discourse.flucoma.org/t/spectral-compensation/299/100), as well as some problematic fft settings in [this thread](https://discourse.flucoma.org/t/fluid-bufmelbands-not-outputting-bands-at-certain-fft-settings/525).

After some quick initial testing (took me a bit to figure out the new syntax/messaging) I still get better results from the MFCCs vs the melbands. I did, however, now get good results from the MFCCs when using a smaller analysis window of 256 (using @weefuzzy’s suggestion of `@fftsettings 256 64 512`). I think I get _slightly_ better matching using a larger analysis window of 512 but the tradeoff in latency isn’t worth it.

So at this point it’s looking like I’m going to be extracting every bit of juice of those 256 samples (“normal” descriptor stuff/stats, 40 melbands, 12mfccs+stats), each for a separate functions later on.

> [@tremblap](#):
>
> _[Rod question about “updating” the training points]_ - Yes you can with machine learning. Soon. You train, add points, train with problem point and/or tweaking parameters … Maybe the one after. We are devising packages. stay tuned!

Also wanted to give a bump to this section. I don’t remember any specific mention of the semi-supervised tweaking/updating in the last release(s), but the objects have been refactored, so it’s possible it could have happened and I didn’t notice.

---

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [June 29, 2020, 9:17am UTC](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464/50 "2020-06-29T09:17:26Z")

</div>

> [@rodrigo.constanzo](#):
>
> the objects have been refactored, so it’s possible it could have happened and I didn’t notice.

not yet. this is a new series of objects that will replace the quick and efficient KNN with neural nets… slower but cleverer. Stay tuned!

[Previous page](https://discourse.flucoma.org/t/onset-based-regression-jit-mfcc-example/464.md?page=2)