Chaining onset detectors (novelty -> ampslice)

Code Sharing
1

In seeing that loads of people seem to like fluid.bufnoveltyslice~ (@tremblap in APT and @jamesbradbury in his ReaComa stuff), it got me wanting to figure something out where it would be more temporally accurate once it found where the general slices were.

I like the idea of slicing via novelty, but having such slop in terms of when the onset was detected kind of kills it for me.

So what this patch does is use fluid.bufnoveltyslice~ (straight from the help file) to find the windows where novelty happened. I then iterate over those indices using something more (temporally) accurate (fluid.bufampslice~).

The results with this simple test file are very promising!


----------begin_max5_patcher----------
3302.3oc6cssjhiiD8YpuBED6C6NKSE5huNOU69Ar+.yrQEFr.7TFaBaSWU2
SLy29pa1XajAYZYn5Mndn.7Mk4QoRk5jRx+wSyluL+CZ4bvu.9UvrY+wSylI
ND+.yT+d17cQerJMpTbYyWkuaGMqZ9B44pneTINd01jRP417CowfkT.8inUU
oeETskBJi1Qqu9jXwUmu72+YWX8AyNrKIKkVIJAzwClenp9nP0Q2GUsZaR1l
WKnqpjxMgfeFt.33Q3ef783efgOCA+W987mO8D+eK9N0unrnzu9MlhEsZKnr
JpnBrOOIqBrNuPnkozrMUaA4qE+58jr372AkIeqQ0SSxnqxOjIddjSwCm.6f
G9j.APD.e1k8gKTfN9WIdriVVFsgdBdD3BgOCBI7+iXkE+CW4u7DefQH1GZp
2cPZ0SrV8Dc45chqnB2Un0XLWOU2j74T808T4cLe9UAAYz2Yx8IHv2RAwzzn
U5rsIgiPGwCqitDGtVQBEppqmAp3B6qlaJxOrWiVh8rqVJMUcQSoVNj8LTi9
gbsikpWnz0jT+vg71k2PC0RZpNkiXmJOOegsIAJZF5AurxsLJai0MSq.KAI5
TS7H7pdF0zILrk6TWh4pYB2iuE0zeFfzomP6Xq5FfFosps0OQejzBcQLnUEc
0phNWzhEIaU5Yf6FtJtP40YBpQYtXSXZr15UG6zJs17U1JkGMvMuijJPJPmi
H61.0Unf7PObuGcUxhBTiJ5ao9IUsMkUhD+6Q+jhPeWWvBoG72PK.Rmbmnvd
VxYjLv.UKUDhbiCwaO32lulxDKPdVISb+s45h0aD1uWVWU8iF5Xjp19wwbYJ
UPkFNa95jT5WnEkI4YsJ7Yyi1uu0gm05V3vxumKdPAKZNTRl7PnlCUP+RR88
6zbznBlRUwznCEBYb9Gd0tu3Ol7XZQ1gDwSRdPVEjRjDUEYLqpx87Hp42LuF
q9zGgIjuuzVPFtgibzMAvi.EqtdSZ9p2nwsFnDC91SyRx1WPKYCuKpRI7MmN
ltN5PZ0qqyypDiZiKk79Izb90JQT6I4JgP9+WEIQoMZvlhj37LtPzopfe35h
iobRmWtsUFwUjEsWyMyr9X3x.mj0Ps5P4xnBdM0xTZqFfLK777ztmp49Roqq
TmdeRVVOTrJe+vmrHYy1ybuKyYmb24d1hyT95gL4YekYTT8ZYzW5h1UQoopF
rce7eDkkrKphVkHqBvvlSRyhXJ51xUE4ooczW4Y9hlyDyLxWQeOItZqnfZaL
vt7j80FQyapkiS1PKq5drpnMkcORY0WkfdqCcXopQ7qUzc6SYZQ2Kf07Horp
ba96kpKr1PqM.bjKm1Mpa68qywOqO+ttBQsOplduGza+vdAGxSHFEJb94IZM
nBDskmvy53eVsyek+kqDJ5zgP+3XRZFowI8EDLHbfFDNvmGNj..xyQxxkYng
Jh0iQpN83RMQA.uA.mvqvT4BXiry.bfrOySFT1ECJZ5QkCeKA.AvAfDz0z7g
Xh8BV1Who1K0CV8zAsN8PzdJ8s+BrNI6X2k8gIzfvD4Z8xnLcbjCBz2LXZcZ
dzsCX1PY8gWUvIPee4.Pi2U3vwHCH9PnXeDPLz86habCqkGVulVbd6Fh88Fi
BBZ4N1GMF6l51WVEhD5zPcNOb2QvqsciJeBP4fUp+uAdW5bI0oYocLNBgOKl
9QqQHXEHRJOCgQgWgQBzDeKDoWkA.oqBANNr.d1jFHXOg.wOudnoL+Pwp5ZF
kKDPWYiEeaEqYUc7s+ZSnd.T2J5hX4nNgFUaMVAi26H.afjw6e..MrY0nkBj
g3S3DJDdmf85kAxDJCbuIFADdSnPviv1Hj.cFyUzzIZPSEs6N.Edx0sKIVjS
akek.OQ7zvPYGdxt53+5V5CvTmSZ.+It5NzT2lC6WPcvZZvlyo5H9UIsNuxC
8KY4gJoa9175MJ5G1jluLJsGmA5Xm3oiBmcXOcc5gj3mYwowCeMMYE8u.uHQ
QP49j2nkfWXc1wNdoLHNvKrNYETKWBP3.vKqiJqX+bOaLktG+Ub96Y7nSYOr
z72UmGig8uD4gxyp1VPK2lmFCP7eudcqCvdHr96EBmZ9afZhYp6D1.qIkaif
3WxEI9E6JGIhqJW3ljZwlvJuCrdCsCs2X0.Tev584X8VDBI7Ai2OX79Ai21j
w6Gz79fl2Gz7dKo4UDqkZTFstJcwZc6YA1qenWOXA9GDVfGvr5VxELB5+fL3
GjA+CNYveJX67yAwuO367Aem2E9Nu2D8OVNWePu5jRuJSHqx0MshwMLPtOhS
aZEarzxA22dIh88wOGBGzhfNrS8+u.4j1jOxgiwqCX3am4QNwwQLS4wgCDTm
YAzYOpX2E8VMg4ZYhE4YGlXkigTo2gdOXhUKSrtNx1CNBXhHimyA69fM1Grw
9fM1IiM1ukBJyKFZXaN1mxQY133cFHViwif9iOMDwhCmLXAJI9v0+SHtvIte
UJMpXP56QS.aYjizvZLrnXuWHr2Z1X+mfglQ+A1eF8iHpkiWf9EU5kVXoSOb
TDkEmuCfG.S7mNLQRvpGxbSlaVNMJx.dvAyog2T4cQsN+bQipUj8IS87o6YH
ZlgSPpdD3guyQtV+bmoGtG3kimEZz2oYiS8HWb+rZ1vAlAWHU91GX7BjXgyX
VHU2gbJqRMnXLtCAOd1O4eJ3IPlhKzmwk.xOAbGxAL109cJoZIEHyUA9N0mz
4xhEhLAYwR4+XLYwRpzeJShEx09IwRE2uevOFIwxzblbJS28xI.hfj6aFxIH
.t4GSUluLLwWXU1LlLgvH9+gSYN.IllcooTHLdswLkoBg6o2Love3IAO7NKZ
g254mu.LLwK.1aBaKooJ4LRwcOU100mSmTXRMh+TJDllXQmoLo0PSctMkVEN
lJDSoQgosOlRGrAioMJdfbM2gc8ABjoI4Lrn47Zkfoi4vQuDubyp7TYRvXwj
HhVs4evmc6H5QqVwhBq00CeNfeUAjPLxi+MeXX.zQb6ctS9nND22qIY7DzQ6
u3CfpBsMBrnEpWvJ3SSt.OpJllkztWHc0Zc.mdiA37vSGk0ASbwgbgk3iccD
eC663htUh9IaiJmW3GApunsEw5jzzFEe1IOW1CVMZk4aJhhSNto11ByPMVHd
r+7Wn8a8JZ0shquU+f.FhuP62zeq02INDhBEVjDkEI+arCgzbmQYaTIQz+Xl
vjUdE46yKZxn5yjvN22gp7F.ncbPGa8dVSgSaoNdKh9CWaDVyh5eGlYLDK9F
CmX5msrj++jYHyfaJ0EIa1PK.d.E+9p0cXRlXSZVNwRzMWRHWwVwbS635pNQ
Kg5+0xdVa1.fRN5jyxFH96ZOrlKIcmzPaS5NtbMcnzt4qiOzMHnqPexNA07T
5lnUesiZizrPAIVZRJQFBmI0MN5+k1h+oSMBv+NOMd9Y1lPau8SAOVgTqhax
xYWbZxp2ZKyCNk3aSXz7h7CYwxoefG6gZw8mQOc6gnD6rcLJWqljf6xtXrXC
9EnxU5opHzNKFWoJFH9ua3kmCTsSuRSpbmjI+lzyEjE94KhUBBf.gPc6Z0Xn
clHb77ozZhvERtISDtge+BPAwT9j6hFCRyWIBMuD72EXw+3W39yK4a69oo7W
+.QURG7zMIYLsp2txeTF6BphV8FHo73CMI6Yv+IuhxtR1sGkxmwIhEq9psQY
YzzR.qqUP4Al7EK10+k2YxQurcVDSAZbE5eq6VQt.UTKs75ZRjise2Hrp5.y
e5I0JltP5Id2ZbQ8tCPsYqpljmDxD7NRP7JRP9FRP9BRP99QP95QPK.4oAfb
riCbGG4ZmTRiurqsfP66BeP.gD37LfOLT9KMBeDl+Ri.S3e3gBYefggrOLFV
P1reMrzeuLDro.Utz1jQV9WnoUe8BaUFslEzfWVutpjIYL0or8NWgtEMoU5X
D4IemaXxNw7E1jJD2X2j9HAw9j2nfxS4PqyB5XHR7PG418jpKSKIbmr.LTxP
ZsNDt5RhXPA0ln3qtfP2rRxwjRB0IMB8RZw0qjlTsYAUzDyCjMpzbMnfbORQ
40WPPSzHhEZZ4ZZA8cicXCwNqTPWxsjUZXYBzgsPA4aRAAsg2OSZvZCMxDaA
GKUNWxTv2F.GzbfyxtXMpJq6E0aJVPbkLIHo4l327q6hr1CJ6li5qunMJBBm
NlK8fIWW46eHwGNAxsb.9utE1NjNynFagJNF0MiM54TTRWpkHxyVkD9RMHfm
qAgurlFoViEpAsG77MIRLGCr+kgt2aU7wK0dqdudqbuSW0dCuh85uZ8DYnnW
pUkZu9Tp1KCK+wSiIMpWWJTGa5SUIcYfL1zMaMsqtGJMo5U4QmZTqHVcH.Pu
fM5zdpKkm0IjZvTcdko47JSw40kdS8o1b3zZNPJMqybl9putsNLqVrcZKMv5
5RopbbVVRmLCrtbe5Oe5+APtLoEC
-----------end_max5_patcher-----------

At the moment it’s using some fairly generic settings for fluid.ampslice~, but these can conceivably fail to return a value within the window of fluid.bufnoveltyslice~. I thought about using @tremblap’s code which iterates over a threshold value with setTarget 1, so it will find the right setting to return a single slice within the novelty window, but wanted to post this proof of concept first.

There can also be types of material which would return a slice in terms of novelty, but can fail to have a useful amp-based threshold crossing (e.g. something with a soft/smooth attack), but for a lot of material something like this could work nicely.

I could see something like this being wrapped up in an abstraction as being very useful.

1 Like
2

You could also recursively slice with increasingly smaller hop sizes until it completely disintegrates!

1 Like
3

Actually could go proper recursive and have big outer level novelty stuff, then have those chunks segmented by smaller window novelty stuff, then use ampslice for the “actual” slices inside that.

1 Like
4

I’ve revisiting this ancient idea here (it’s been on my list for a while), particularly in the context of manually hunting for a zero-crossing post-amplitude-based onset detection (thread) with the idea to build a 3 layer offline onset detector.

  1. fluid.bufonsetslice~ (it seems like @metric 9 is most generic/useful here?)
  2. fluid.bufampslice~ based on the returned results of step 1 (if no onset is detected default to results from step 1)
  3. a faux @lookback crawling through n amount of samples before the onsets from step 2 to find a zero crossing

With that in mind, a few questions.

///////////////////////////////////////////////////////

I guess the idea of using something other than fluid.bufampslice~ would be to catch onsets that aren’t amplitude based, so fluid.bufonsetslice~ seems like the idea choice. Is @metric 9 (RComplexDev) a good all-around choice?

It’s defined in the reference in a kind of reverse-chronological cascade of info:

6 = ‘6’
PhaseDev takes the past 2 frames, projects to the current, as anticipated if it was a steady state, then compute the sum of the differences, on which it thresholds (like Onsets phase)

8 = ‘8’
ComplexDev same as PhaseDev, but in the complex domain - the anticipated amp is considered steady, and the phase is projected, then a complex subtraction is done with the actual present frame. The sum of magnitudes is used to threshold (like Onsets complex)

9 = ‘9’
RComplexDev same as above, but rectified (like Onsets rcomplex)

I had a poke/test in the “parameter combinations” tab in the fluid.bufonsetslice~ helpfile and @metric 9 seemed to give the results that were most useful across the listed examples. That being said I don’t really understand the (technical) distinction between metrics 6/8/9.

///////////////////////////////////////////////////////

The second question is about the cascading of time. So if I understand the way onsets are reported back from fluid.bufonsetslice~ given the temporal wiggle room of fft-based onset detection, the returned slices are quantized to the start of an fft window and the “real” onset occured at any point within that fft window.

So if I have a slice at sample 1000 and I have @fftsettings 1024 -1 -1 then the “real” onset would be anywhere between samples 1000 and 2024.

Is that correct? Or more specifically, does the hop not matter here? Like would it instead be anywhere inside the [window - hop] such that the “real” onset would be anywhere between samples 1000 and 1512 instead?

The reason this bit matters as it would define the @numframes that fluid.bufampslice~ will be checking for.

///////////////////////////////////////////////////////

The third question is more a thinking-out-loud type thing. Once I have a returned slice that has either gone from fluid.bufonsetslice~ → fluid.bufampslice~ or just from fluid.bufonsetslice~ (if fluid.bufampslice~ fails to return a value), and I then want to look for a zero crossing ala @lookback in fluid.bufampgate~ (as discussed in this thread), would I search for a zero crossing before and after the detected onset?

Or rather, I’m currently thinking something like this, where the process would fork at the last step.

If [fluid.bufonsetslice~ → fluid.bufampslice~] returns a value, then search for a zero-crossing only before the detected onset as fluid.bufampslice~ is quite temporally accurate and I just want to add a kind of @lookback functionality to the detected onset position. In this case it would be a fairly small amount of samples (~16 samples).

If [fluid.bufonsetslice~ → fluid.bufampslice] then search for a zero-crossing only after the detected onset from fluid.bufonsetslice~ since it will, by definition, happen after that point. In this case it would also be a fairly large amount of samples (~256 samples) and then picking the latest position if multiple zero crossings are detected since the temporal slop here would be intrinsically higher.

5

Disappointed there isn’t a discourse badge for oldest bump as this is one of my time-travelliest ones!

Screenshot 2023-07-11 at 1.39.48 PM

1 Like
6

Is this solved? quick answers:

q1: I love 9 as it does - it does a prediction of the future (like 6) in the complex domain (like 8) (works well for pitch material more than noise) then it sums the differences with the actual future but considers downward distance as important as upwards. I presume the output of onsetfeatures would show you that, so if unclear, let me know and I can try to make a didactic example.

q2: for window - check the ‘where is my data’ example in the other thread. You are right in theory, but you forget the windowing. if your click is at sample 1001 for instance, it’ll be mostly silenced by the windowing in that frame…

q3: @a.harker had a good patch to hone in on an attack in a frame - that might be more efficient?

q4: (aka new badge for old bounce) feel free to propose one if you find one on https://meta.discourse.org/ :slight_smile:

7

Not really (more below).

Algorithm 9 does appear to be the most effective one across the board. Oddly enough on realtime audio I can never get it to work as well on varied material as I can with ampslice (even for non-drum sounds) but for offline stuff, it likely grabs more stuff than ampslice might on its own.

Having a more descriptive definition like this in the reference would be useful as algorithms 6-9 make little sense in terms of reading what it is doing.

I did implement all the bits I was working on here:

I forget the specifics of it, but I do a bunch of conditional stuff where I check either before and/or after the detected onset to refine the timing. Theoretically it’s a good idea and was working well for audio in which both onsets where found, but I started noticing that fluid.bufampslice~ would never find an onset within the prescribed window… (I had conditional stuff where if ampslice didn’t find anything, it would default to the bufonsetslice timing).

All of this has come to a standstill because of this:
https://discourse.flucoma.org/t/fluid-bufampslice-not-working-correctly-when-using-numframes/

At the moment, it’s actually possible to use fluild.bufampslice~ as I mention above as it doesn’t work at all if using a different @startframe and/or @numframes, and actually never picks up the first onset in a buffer regardless (if the onset is as near the start of the audio file).

Unless I do some really wonky workaround where I set “fake” @startframes thousands of samples before I’m actually interested in and fluid.bufcompose~ fake audio to the start of a buffer in order to catch the onsets there, I can’t actually do any of this.

As I mention in the other thread, I suspect the envelope followers are starting from negative infinity and take a really long time to catch up, meaning that it’s not useful for the start of buffers, and not useful for any small/medium size @numframes because the envelopes never get into useful range.

It could be something other than that, but it definitely appears to be in bug territory here.

8

it is on my radar. amongst many other things, obviously, so slow horizon.

1 Like
9

I have an external called something like “finesseslices” that is for my internal use - there are no current plans to make it public or do the work of documenting and supporting it. I don’t expect that would fit @rodrigo.constanzo’s aim to have currently supported minimal dependencies.

10

Can you (roughly) describe what/how you finessed it?

So far the look ahead/behind/contextual thing is working ok (on the few times it works with fluid.bufampslice~ at the moment). But would love to improve it if possible.

11

I can describe exactly how I finessed it in the form of the attached code - look at the slices() method. There are some very general comments, but the code is detailed. Any other answer will have to wait a while, as I have quite a bit on and I’ve not looked at this in quite some time.

finesseslices.txt (13.0 KB)

1 Like
12

For info it’s a cpp file, but I can’t attach them, so I changed the extension.

1 Like