SP-Tools - Machine Learning tools for low latency real-time performance

For the last few months I’ve been working pretty hard on consolidating a bunch of the stuff I’ve been doing over the last few years into a package that basically piggy-backs on FluCoMa and adds a ton of drum-specific examples, and importantly, tweaks and fine tuning.

At the moment I’d consider it alpha as I’m still going to add some more functionality, I may mess around with the structure some, etc… but so far it’s working pretty well I think.

Here’s a teaser video I made:

And a video overview where I walk you through the overview patch:

It’s all up on GitHub:

11 Likes

Great stuff Rod! No idea you were bringing all that together – really valuable.

1 Like

Thanks!

It’s been bits and pieces here and there, but after cracking a decent classification recipe and getting somewhere useful with other core ideas, I figured it would be good to lump it all together. At the moment I’m trying to refine the standardization stuff from this thread, as well as still plugging away at the time-travel/regression idea, so I’ll add those bits as I develop them.

2 Likes

Yeah this is really great, well done.

Excited to see how the community who uses the same hardware latches on to it!

1 Like

Awesome!

2 Likes

Big v0.2 update adding “controllers” (meta-parameters) and “setups” (scaling and neural network prediction).

I managed to include the stuff from the biasing thread and got the regressor working finally (though can probably be improved).

4 Likes

thanks for sharing this stuff rodrigo!
i’ve been keeping an eye on your SP experiments for a while and the new demos are fantastic

this toolset seems really responsive and quite flexible

i’m downloaded the suite now and have a basic question: does this require config or training like the official SP software? do you have any tips on initial setup?

1 Like

Hey!

So it really depends on what you want to do. If you wanna do the “SP Software Thing” then you need to use sp.classtrain (or sp.clustertrain) to give it the hits you want to train and stuff. But if you want to do corpus-based sample playback, or loads of the other things in there, then you can just use it as is.

What I’d recommend is creating a class/zone training (ala the SP software) with sp.classtrain and saving that, and then you can also create a setup which is new in v.02 with sp.setuptrain~ and then you have all your bases covered.

One thing I did somewhat avoid doing was trying to build “a sampler”, since that gets quite specific and esoteric for everyone’s needs. There’s some playback stuff in there, mainly for the corpus-based stuff, but the idea is what once sp.classmatch tells you what zone/class you hit, you then play back the sample (or whatever) on your own.

For a v0.3 (or maybe later) I do want to build a somewhat generic sampler ala the SP software/Simpler, but I don’t want to go too far in that direction as that’s essentially its own project (“a sampler”).

Let me know if taht makes sense and/or if you have other questions. It might be worthwhile making a “getting started” video guide too.

1 Like

this is very helpful! i also rewatched the overview clips & plan to try some things this weekend as i have time

1 Like

Let me know how you get on with it and if there’s anything that doesn’t make sense. It’s tricky with something like this as I’ve been pretty engrossed in it for years, so it can be easy to overlook things that aren’t immediately obvious to others.

Thanks for sharing this. I love to see how your ideas evolved over the years since CCCombine and earlier experiments. Looking forward to road-test it.

1 Like

New v0.3 update:

Added the ability to filter a corpus/dataset (using fluid.datasetquery~). I still find the syntax for that object really terse, but wrapping it up in layers of abstraction has made it much easier to use (e.g. filter loudness > -35 or time_centroid == 50, etc…).

For now it’s a one-query-only thing due to how it’s abstracted, but I may figure out a way to make it still readable to do multiple queries in the next update.

Also do all the scaling post-filtering, which is quite nice as otherwise the descriptor space gets pretty fucked.

Also completely revamped the playback stuff with a more “full featured” playback sampler (start/length, attack/hold + curves, speed, etc…) and added in some “realtime” descriptor analysis (JIT stuff based on the fftsettings). I’m wanting to implement some mosaicking options in future updates, so that’s setting the groundwork for that eventually.

wow this is very very inspiring @rodrigo.constanzo - I am quite impressed by how far you can push things. Congrats and thanks for sharing!

1 Like

Whoops, just realized I forgot to post the v0.4 update:

And now here’s v0.5 behind it:

Changelog

v0.5 - SP-Tools v0.5 Video Overview

  • added Max for Live devices for some of the main/flagship functionality (Concat Match, Controllers, Corpus Match, Descriptors, Speed)
  • added sp.gridmatch abstraction for generic controller-based navigation of corpora
  • added support for the Erae Touch controller (sp.eraetouch)
  • improved path stability when loading example corpora

My Erae Touch arrived at a good time, so built some support for that, as well as generic grid-based navigation (e.g. Sensel Morph, Gamepads, etc…)

Really loving the M4L bits too. I honestly wish it was that easy to map/use stuff in Max natively. Really solid QoL improvements.

2 Likes

v0.6 update.

Mostly bringing a lot of the functionality into M4L (16 devices!).

I also started in on the documentation (using @jamesbradbury’s spiffy documax), though none of that is live in the current update. It’s gonna take a while to roll all those files as there are presently 38 abstractions in Max-land.

1 Like

Oh nice. If you need help with documax let me know. It would be great to flesh out the tool so that I can save at least one other person from XML.

1 Like

wow thanks! More examples is what I needed!
Had a workshop a year ago but still with a beta release, so looking at some working patches is gold <3

2 Likes

I’m watching this - fantastic work @rodrigo.constanzo !

1 Like

hi rodrigo,
not sure why but max 8 (latest version) crashes quite often when. I run your sp-tools on my m1 mbp (latest version of monterey). so far I tried your sp-tools v06 overview and demo patches. from what I can see, It happens after 1-2 minutes of playing the patch (eg. sp.corpusplayer~) mostly when i check the detail and double-click on the sub patches, abstractions, poly objects.
are there max setting that could cause it?
let me know if there is anything that I could try or check.

still a great work, thanks a lot!
best, johannes

The overview patch is a bit fickle due to some funky threading stuff/bug going on in the realtime analysis stuff. I’ve stripped the overview patch back for the next version, but I still get those threading error messages every now and again (though no crashes).

Are you getting crashes when using the vanilla help files, or only from the overview patch?

(I would also get crashes when “doing stuff” like clicking on windows or tabbing over, due to the threading issue)