SP-Tools - Machine Learning tools for low latency real-time performance

Hmm, how would this then go with the onebang stuff as well (like at a route bang level above the gate 2 (for this specific example)?

If I understand this correctly, this would just force the high-priority bangs from above to kindly respect what stuff downstream wants?

For the most part the @blocking 1 happens for literally only the first hit, just to size/resize buffers and such (uness I’m running offline analysis), so I guess this would just sanitize that initial hit, with the onebang thing preventing re-bang-ing while another process is happening.

For this bit from above, when I’m getting the errors/crashing, I’m not manually sending bangs as well. I’m doing unrelated UI stuff like tabbing to another app, or switching to another tab in the help file etc… As in, not interacting with the relevant code at all.

onebang emits on the thread it was called from afaik.

Yes. defer (or qlim) will force messages on to the main thread if they’re not already there. [delay 0] does the same for bangs with respect to the high priority thread.

Something is, or you wouldn’t get the crashes you do. It doesn’t even need to be bangs (though this would explain it best).

`

I’ve implemented both of @weefuzzy’s suggestions here. So far no yellow warnings (or crashes), but I no longer have/use the old massive overview patch.

edit: I accidentally left it running for a couple hours while I went out and ran errands, and not a single yellow error the whole time! Also tested it with the big chonky overview patch and no issues there either.

@johannes and @MartinMartin , can you try replacing these files in your 0.6 install and see if you still get crashes on the overview patch? (The M4L devices won’t change until I manually change their guts) If this works I’ll tidy up the changes for the next release and propagate it out to the M4L devices as well.

Archive.zip (59.7 KB)

I replaced the files and was testing sp.corpusplayer~ helpppatcher for
about 10 min and had no crashes. so yeah, that seems to solve it – at least for that particular helppatcher! thanks Rodrigo.

1 Like

Ok, downloaded the archive.zip and replaced the files - only 4 maxhelp files no abstractions (?)

I still got these 4 yellow Errors while using the concatsynth: (also just letting it play in the background)

fluid.bufloudness~: Process() called on empty queue
fluid.bufmelbands~: Process() called on empty queue
fluid.bufmfcc~: Process() called on empty queue
fluid.bufloudness~: Process() called on empty queue

I didn’t got errors using concatcreate.

Also the m4l concat match was working without fuss.

Max crashed while using concatcreate and concatsynth simultaneously.
Here is the crash report:
https://goonlinetools.com/snapshot/code/#no8d95sywl8spf9z1lk7

I was working for about 2hrs. So in my experience no crashes when using the two processes not in parallel.

Potentially bit of a silly question, but did you quit/reopen max after replacing the files? (the helpfiles would be unchanged either way) Otherwise it keeps on with the ones loaded into memory.

The M4L devices are, as of now, still unchanged too.

yes, I replaced them first and opened max afterwards.

But yeah, still exploring your abstractions and through them the flucoma objects. If I can be more precise the following days I will come back here…

Hmm.

From (my limited understanding of) the crash report, it does seem to point to the same threading funny business. I would have thought the changes made it safe across the board.

Here’s the v0.7 update:

v0.7 - SP-Tools v0.7 - Ramps, Data Processing, Novelty, and Timestretching

  • BREAKING CHANGES - all objects that had a separate control inlet, now take those messages in the left-most inlet
  • added new “ramp” objects for structural and gestural changes (sp.ramp, sp.ramp~)
  • added new “data” objects for transforming, looping, and delaying descriptors (sp.databending, sp.datadelay, sp.datagranular, sp.datalooper~, sp.datatranspose)
  • added novelty-based segmentation for determining changes in material type (sp.novelty~)
  • added timestretching functionality to sp.corpusplayer~ and the Corpus Match M4L device

/////////////////////////////////////////////

Thanks to @weefuzzy there’s some nice bugfixes and workarounds to make things safer/nicer.

5 Likes

Rod you are on fire :fire: This looks like a fantastic update.

1 Like

Really pleased with the data stuff. Not really seen descriptors tweaked in that way before.

So much cool stuff going on here, thanks for sharing.

To what extent do you think some of your tools/approaches work for a non-SP hardware setup?

For example I’m currently messing around with a KeyTam but using piezos and other more simple/commodity sensors, and I would love to play around with your patches!

A bunch of the core objects (sp.onsetframe~, sp.descriptors~, etc…) all let you specify what kind of mics/setup you are using, so you can use the @input attribute to say what kind of mic you’re using, and if you’re using more than one input:

The different input modes are as follows:
0 = SP sensor
1 = drum trigger (contact mic)
2 = air microphone
3 = SP sensor + air microphone
4 = drum trigger + air microphone

This handles routing and stuff, but also applies some EQ/filtering that I’ve found useful for each of the types of mics (mainly for the Sensory Percussion sensor and piezo discs).

For most of my purposes I use @input 3 as it gives me the “best of both worlds” with the super fast onset detection from the Sensory Percussion hardware, but better descriptors from a DPA 4099. That being said, I have found that I get the best results from classification with just the Sensory Percussion sensor on its own.

So in short, it should all work, but your mileage will vary with some of the fussier processes (mainly classification really). With something like the KeyTam (ooh, what are your thoughts on it so far?), you can get a really wide range of sounds just from the open tuning, so that would really confuse the classifier anyways. It would work wonders for the vanilla descriptors and corpus matching stuff though.

On my list of things I’m eyeballing is eventually trying to come up with a DIY version of the Sensory Percussion hardware. Or rather, something inspired by that approach (hall sensor-based), so it will be easier to make some, and mount them on existing drums permanently (like in a KeyTam). My hardware/engineer chops are such that this is a really slow investigation from me as even though I’ve taken my sensor apart and taking pics of everything, I can’t easily draw a schematic or know how to make one from scratch. If this overlaps with your knowledge/interest, I can ping you an email with the info/pics I have and we can chat from there further.

1 Like

indeed sp.databending and sp.datadelay is quite a unique cross-modal thinking, I love it.

one pointer: sp.dataloop is cool and could be expended to do the mutations and swaps I was doing in sandbox#3, I remember Diemo loving it

congrats again for this inspiring package!

1 Like

one last comment - I’m looking at the codebase now and the amount of work and care and tweak and customization is just incredible. :star_struck: :heart_eyes:

1 Like

Thanks! The delay one sounds really convincing, and obviously opens up loads of possibilities.

What is it that you do with it?

in that case I was simply doing swaps of chunks - if I requested 10% it would take 2 blocks of 10% and swap them. very blunt, very fun on gestural material. 2% many times as it was playing was very powerful to make mutations not too chaotic.

1 Like

p.s.

I’ve made an SP-Tools discord server (https://discord.gg/Vy2zuKQkqN) which I will properly announce on the next (v0.8) release, but wanted to put this out there already if anyone is keen on joining in already.

It’s a pretty big update this one (bringing the abstraction count up to 61!). There’s one core thing I’m working on which still needs tweaking, and beyond that it’s just polishing up the help/examples. Here’s a teaser with the changelog at the moment:

Changelog

v0.8 - Sines, Synthesis/Modelling, and Documentation

  • added synthesis and physical modelling objects (sp.karplus~, sp.resonators~, sp.resonaroscreate~, sp.shaker~, sp.sinusoidplayer~, sp.sinusoids~, sp.waveguidemesh~, sp.lpg~, sp.lpgcore~)
  • added Max for Live devices for some of the new processes (Resonators, Sinusoids, Waveguide Mesh)
  • added new descriptor type (sines) and corresponding objects (sp.sineframe, sp.sines~, sp.sinesrt~) (you should reanalyze all your corpora)
  • added utilities for filtering and creating clusters of triggers and gates (sp.probability~, sp.triggercloud~, sp.triggerframe~)
  • added proper Max documentation via reference files, autocomplete, etc…
  • added absolute start and length parameters to sp.corpusplayer~ (via startabsolute and lengthabsolute messages)
  • added Scramble transformation to sp.databending
  • added sample rate adjustments to sp.concatcreate~ and sp.corpuscreate~ so they now properly analyze files at any sample rate (previously they assumed all files were @ 44.1k/48k)
2 Likes

Here’s version v0.8!

This one’s been a long time coming and is the biggest update so far, adding a whopping 16 new abstractions and 3 new Max for Live devices (full changelog below).

There’s also a Discord server now for updates, sharing patches/corpora, new ideas, etc…:

Changelog

v0.8 - Sines, Synthesis/Modelling, and Documentation

  • added synthesis and physical modelling objects (sp.karplus~, sp.resonators~, sp.resonaroscreate~, sp.shaker~, sp.sinusoidplayer~, sp.sinusoids~, sp.waveguidemesh~, sp.lpg~, sp.lpgcore~)
  • added Max for Live devices for some of the new processes (Resonators, Sinusoids, Waveguide Mesh)
  • added new descriptor type (sines) and corresponding objects (sp.sineframe, sp.sines~, sp.sinesrt~) (you should reanalyze all your corpora)
  • added utilities for filtering and creating clusters of triggers and gates (sp.probability~, sp.triggerbounce~, sp.triggercloud~, sp.triggerframe~)
  • added absolute start and length parameters to sp.corpusplayer~ (via startabsolute and lengthabsolute messages)
  • added scramble transformation to sp.databending
  • added slew parameter to sp.speed
  • added ability to loop floats and ints (as well as conventional descriptors) to sp.datalooper~
  • added proper Max documentation via reference files, autocomplete, etc…
  • added sample rate adaptation to all realtime and offline analyses. previously things were optimized and assumged for 44.1k/48k but now everything works at every sample rate (up to 192k)
  • added some puredata abstractions and help files to the package (in the puredata folder)
2 Likes

Ok, here’s the next (absolutely massive) update of SP-Tools (v0.9):

The changelog is a bit long for this one (still pasted below), but the broad strokes are a complete rework of the scope of the project such that it no longer assumes “drums” as the default audio input, so it is more broadly usable by all sorts of input. (It is still super optimized for drums and low-latency stuff, but if there’s a selectable option (e.g. for Sensory Percussion hardware), it is no longer the default setting). There’s also been a massive amount of work done to improve the ease of use and overall discoverability. The overview patch reflects this as you can browse by objects/categories, use cases, or just musical examples directly. Finally there are a bunch of new patches for audio processing and signal decomposition.

Changelog

v0.9 - Scope, Ease of Use, and Audio Processing

  • BREAKING CHANGE - objects with “train” in the name (sp.classtrain, sp.clustertrain, sp.setuptrain~) have been renamed to be more consistent with other objects (sp.classcreate, sp.clustercreate, sp.setupcreate~)
  • BREAKING CHANGE - sensitivity parameter in all onset-based objects has been fine-tuned to work across a wider range of instruments and materials so your previous settings may likely need adjusting
  • BREAKING CHANGE - threshold parameter renamed to floor in all onset-based objects (sp.onset~, sp.onsetframe, sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc…)
  • BREAKING CHANGE - threshold parameter renamed to sensitivity in sp.novelty~ keeping it inline with other onset detection objects macro controls (sp.onset~, sp.spectralonset~)
  • BREAKING CHANGE - mode parameter renamed to timestretchmode in sample playback objects keeping to remove clash with the new mode attribute
  • BREAKING CHANGE - input mode order and modes changed on all onset-based object (sp.onsetframe, sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc…)
  • BREAKING CHANGE - clusters trained with sp.clustertrain now count from 1 instead of 0 and are returned as ints instead of a symbols (so a 1 instead of a “1”)
  • BREAKING CHANGE - sp.classifierdisplay @drum attribute reworked to @defaultdisplay
  • BREAKING CHANGE - fade in/out and curve parameters renamed in sp.corpusplayer~ to be consistent with the new sp.sampler~ (attack=in, hold=out(and inverted), attackcurve=incurve, holdcurve=outcurve)
  • overview patch (SP-Tools Overview.maxpat) completely revamped with better discoverability and package navigation
  • added snippets for common use cases to SP-Tools package (accessible via the snippets side panel)
  • restructured Max Package folders to make it easier to navigate and add additional corpora
  • restructured github repo to separate versions (Max, pd), Max for Live devices, and additional corpora
  • added class-combined versions of all descriptor analysis objects making it easier to get classification combined with descriptor analysis (sp.classdescriptors~, sp.classmfccs~, etc…)
  • added sp.controllermatch to browse corpora using a single parameter
  • added sp.controllerptich to create pitch-quantized melodic phrases from a single parameter
  • added zero-latency convolution (sp.convolver~) which allows for reverb, amp sim, mic correction, and any other convolution-based audio processes
  • added convolution-based corpus matching via sp.corpusconvolver~ which combines corpus matching and realtime convolution
  • added combined abstractions for common processes (sp.corpusanalysis~, sp.corpussampler~)
  • substantially improved core concatenation algorithm
  • added radius and neighbor selection in concat matching
  • added pitch and loudness compensation to concat playback
  • added transpose and pretranspose to concat playback
  • added sp.corpusclustermatch for creating clusters from corpora, allowing for grouping of similar sounds within a corpus (e.g. kick, snare, hat, etc…)
  • added objects for shaping and processing descriptor lists (sp.descriptorcurves, sp.descriptorreplace)
  • added an envelope follower with some quirky features (sp.envelope~)
  • added sp.gridscale for scaling and shifting XY-based controller inputs (combines well with sp.gridmatch and the new sp.boppad)
  • added pitch and time quantization with sp.quantizepitch and sp.quantizetime~
  • added data and clock sequencers with sp.sequenceclock~ and sp.sequencedata~
  • added Schmitt filter for descriptor filtering for when you need hysterisis in the filtering (sp.schmitt)
  • added various audio processes for splitting and decomposing signals (sp.scatter~, sp.scramble~, sp.shatter~, sp.shattercreate~, sp.sift~, sp.smear)
  • added spectral-difference onset detection (sp.spectralonset~)
  • added new kitchensink descriptor type, as a combination of descriptor/melband/MFCC buffers which comes in onset, real-time, and class-based variants (sp.kitchensink~, sp.kitchensinkrt~, sp.classkitchensink~)
  • added neural network classifier type to sp.classtrain, sp.clustertrain, and sp.classmatch for greatly improved speed and accuracy
  • added support for the BopPad controller (sp.boppad)
  • added input modes to all the physical modeling objects (sp.karplus~, sp.resonators~, sp.shaker~, sp.waveguidemesh~) allowing for realtime actuation of the models, in addition to onset/trigger-based actuation
  • added ability to reverse and loop samples, and jump directly to sample position in sp.corpusplayer~
  • added @mode attribute to sp.onsetframe~ to switch between amplitude and spectral onset detection (also impacts all onset-based descriptors objects (sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc…))
  • added ability to seed clusters in sp.clustertrain
  • added utility for subdividing a spaces of arbitrary size and steps (sp.slicecurve)
  • added utility for time-alining trigger/gates with descriptor output (sp.triggeralign~)
  • added utility for increasing the length of triggers and gates (sp.triggerlength~)
  • added AHR-style envelope with sp.triggershape~
  • improved time-alignment for trigger/gate outputs of descriptors objects (sp.descriptors~, sp.melbands~, sp.mfcc~, sp.sines~, etc…)
  • reworked sp.classifierdisplay to automatically load class names from all class matching objects
  • reworked sp.descriptordisplay to automatically display the correct type of descriptor
  • added dump output message to all class-based objects (sp.classmatch, sp.classdescriptors, etc…)
  • added mic correction (via a new input mode attribute @input 5) to all onset-based descriptor analysis objects for better descriptor results when using only an SP sensor
  • added @buffer attribute to sp.corpusplayer~ to simplify non corpus-based sample playback
  • added @history attributes to all realtime descriptor objects (sp.descriptorsrt~, sp.melbandsrt~, sp.mfccrt~, sp.sinesrt~) to smooth descriptors
  • added @input modes to sp.onset~ allowing for optional filters optimized for different input types
  • added onthreshold and offthreshold to sp.novelty~ given you finer grain control of thresholding
  • improved (and fixed) melband transposition in sp.datatranspose
  • added optional arguments to all corpus matching abstractions to load corpus and setups automatically (e.g. sp.corpusmatch corpus_china setup_snare will automatically load the China Cymbal corpus and load/enable the snare setup)
  • added ability to load default files to all .json loading objects (e.g. sp.resonators~ resonators_bell)
  • added header to every .json file to differentiate them (you should reanalyze all your corpora, classes, and setups)
  • when loading a setup into any corpus matching abstraction (sp.corpusmatch, sp.corpusconvolver~, sp.corpussampler~) it will automatically enable that setup
  • improved default starting values in sp.controllers so it gives useful results while it autocalibrates
  • added ability to trigger sp.triggerbounce~ and sp.triggercloud~ with bangs as well as trigger/gates
  • added ability to trigger synthesis objects (sp.lpg~, sp.karplus~, sp.resonators~, sp.waveguidemesh~) with bangs as well as triggers/gates
  • added 8 more abstractions to the puredata version
  • changed FFT settings for medium and all sizes for corpus and setup analysis to have better pitch accuracy in the larger time scales. You should re-analyze all your own corpora and setups.
  • you can now load mp3 files into sp.corpuscreate
  • fixed M4L device loading bug(s)
  • abstractions that only function as part of an existing abstraction (sp.lpgcore~, sp.folderloop, etc…) have had their sp. prefix replaced with widget and have had their help/reference pages removed
1 Like