FluCoMa and Modular Synths

I know at least @tremblap (and perhaps @spluta and @leafcutterjohn? ) is using some FluCoMa stuff with modular synth stuff, so I figured it’d be good have a thread that was specific around that topic with code/examples/etc…

The first bit of code I want to share is an Analysis.amxd I made (which is now part of confetti, which I’ve linked to already on the forum):

(the scope~s aren’t part of the device they are there to show what’s going on)

I mainly made them to work in Max-land, so along the bottom is a bunch of diff outputs, all signal rate, with the light grey ones being s&h values (driven by the onset detection).

I’ll probably tweak the underlying code a little bit, but the core idea has been suuuper useful for my experiments.

This is my testing patch:

It uses the super useful AudioMix package for all the boring plumbing/mixer-y shit, but with this I have three analysis buses, which I can send out to my ES-8.

I’m also curious about the automatic scaling thing that @tremblap built/showed me a while back, since that would come in handy for certain things I’m building at the moment. (do you have a tidy/shareable version of that @tremblap?)

I also want to revisit the CV->NMF stuff I was playing with around the first plenary:

I’ve not played with it in a while, but I remember it working really well for ‘offline’ stuff, but I never managed to get a real-time version working due to buffer sizes, and downsampling funny-business.

Either way, curious what else others have been up to with this kind of thing.

1 Like

Interesting post! Let me try to make my answer not too long… first, a few other people who have openly been interested in this in the plenaries are @tedmoore and @rdevine1

Secondly, I have to say that I’ve done a few things along those lines in since 2009 but they usually in bits that are used ad-hoc in a piece or a project, then shelved until I can have access to more potent technology or that my brain catches up with what is available. They are usually in the same idea, which came around when I was exploring descriptors in a way to control descriptor browsing. When working with CataRT at the time, and modifying it for sandbox#3 the fascinating Diemo Schwarz told me of a project that was going on in the few years before, where they used a robot to iterated through complex physical modelling patches to describe them and then navigate them in descriptor spaces… that put me on fire, and I did a first few attempts with modular then, not super successful. That frustration led to the FluCoMa project :slight_smile: Since then, some public version of such iterative software have emerged, like this

In parallel, I did one that worked better for me in 2014, which I have reused since then a bit… it was based on @a.harker 's descriptor and entrymatcher. I was interested to control the pitch of a very chaotic synth, to resynthesize an audio stream. In the end, I found it more effective to do a single control via CV, and sample 1000 points iteratively. Then I would use a stream of 3 descriptors (pitch, centroid and amplitude) from my target(control) and match each to a different pitch through entrymatcher controlling the chaotic synth, then a simple mapping to an LPF and a VCA for the other 2, and the result was quite potent enough for me to have fun with a training time that was manageable… you can hear a local bird recording (a willow warbler) I’ve resynthesized here: https://soundcloud.com/pierre-alexandre-tremblay/oiseau-solo1 the patch is very ugly creative-coding-in-composition-time-messy but I can share it if someone wants to have a peak…

Now for the immediate future, I am interested in more than one dimension at a time, with machine-learning, both from a live stream of descriptors (bass) and from segmented targets, but this is work in progress and far from finished. I had a first poke at this idea with the fantastic Wekinator last year, then with a basic neural network my son has coded a few weeks back (proud dad moment here) for which I’ve done a quick demo with a virtual chaotic synth… I’m happy to go in more details on any of my plans for both ideas (real-time control and segmented targets) if that could interest anyone.

Ah yes, forgot about @tedmoore and @rdevine1. I imagine @rdevine1 in particular will be doing loads with that when the next concert (eventually!) happens.

You’ve shown me the patch a couple of times but I didn’t remember if it was based on discrete points (it is I guess) or if you mapped a continuous space. I remember the ‘scanning’ part of the patch being quite clever and efficient too. So are you using distance calculations to do some kind of interpolation between the sampled values or is it all quantized? (from the sounds of the soundcloud demo there’s some steppiness)

I was wanting to build/test something simpler where I would map something like centroid to a filter (or filter-y thing) response, so I was going to do some kind of one-to-one mapping just taking measurements along a point and just plotting it on a function object to interpolate between the rest.

Obviously this wouldn’t work in a one-to-many or many-to-many, not to mention getting all ML-y, but it seemed like a reasonable place to start.

I wanna see the patch!

1 Like

My version you saw is based on discrete points, and I take the nearest. The version I tried with Wekinator was testing various interpolation but for non-linear synths, that was not conclusive then. Then trying the MLP stuff with Édouard gave me something fun to embrace the non-linearity, but I have not tried the new plans, which are with 2 many to many mappings in an assisted learning fashion…

The MLP stuff you can see here https://huddersfield.box.com/s/rir81s7tsssv8iqq5e03808jbzyrf0tc and you can implement with Wekinator a similar idea: you train OSC messages from a series of presets off you go!

Be careful what you wish for…

1 Like

Yeah I remember the non-linear stuff really complicating things.

Did you ever try just plotting the points on separate function objects (one per dimension) to see if that gave you anything useful?

Would still be good to see your code, for the sake of the thread/discussion, but also to see about the jumping around the register and sampling cleverly (though for linear measurements, it doesn’t matter too much either way).

This kind of model (training sweet spots and then navigating a 2d space) doesn’t interest me too much, but this could be interesting with some of the self-organizing map stuff and/or doing some training via “resynthesis” models (i.e. feeding it sounds, and using those as training points).

I did something similar comparing the few nearest neighbours, but the thing was that they were at opposite places of the space…

You need to use your imagination a bit… if you can map to 2D you can map to anything. MLP is actually good as a classifier, you just pick whichever score you get on the data you get out… for instance look at the other classifiers we have done so far in the example folder, you can make, with many inputs, the bassdrumness, snareness, and highhatness values, train with 10 of each from a bunch of inputs, and you will get values out… I’ll do more example once we have something along those lines in our toolset, but the ideas are in the Kadenze class of Wekinator if you don’t want to wait :slight_smile:

I get the idea, it’s more of the interest and how I like to approach things creatively. And actually, to put it in the form of anecdote, when I was younger, my mom asked me what I liked to have as a snack. I told her I liked apple sauce, as I was into it at the time. And she bought a ton of it. Needless to say I got tired of it and didn’t like apple sauce anymore. So that’s all to say that I don’t like making decisions about what I like, because then I don’t like it anymore.

So this paradigm of “this is a specific sound I like, give me more like it” doesn’t interest or inspire me. Which is also why after watching all the ML Wekinator I didn’t install or test any of that stuff out because it’s all (pretty much) based on, map this spot to this, and this spot to this, and now train…

Yeah I can see that being the case.

I just put together a simple thing (manually) where I took 20 measurements off an LPG (LxD) while pumping some Disting pink noise through it (can you believe I don’t have normal noise in my skiff?!) and then ran the centroid analysis off the scratchy turntable stuff and have created a simple resynthesis of it! Granted the sounds here are fairly straightforward and linear-ish, but it works as a first initial test.

Here’s the response (visual scale is 0.-1. / 0.-127.):
Screenshot 2020-04-26 at 6.25.35 pm

So linear for a big chunk until the end.

And here’s a comparison (the “real” snare audio is highpassed 110Hz whereas the noise is unfiltered (other than the LPG)), which sounds pretty convincing! It “feels” right too, not to laggy or unrealistic, which is quite weird given it’s a synthetic sound. It sounds really badass if I hard pan them too since I get a faux-stereo effect going.

The crossfader is also being sent to a VCA with the same signal-rate control I use when chopping up the direct snare audio.

that is the main difference between our use. Yours is very linear. Imagine mine is something like 440Hz is at 0.1 and 445Hz is at 0.5, but in between you might have anything like 220 and 880 around 0.2 and 0.3 so interpolation is much harder. it is not random, so there might be a way, like the spirals in this fun learning toolbox but very hard to find out with the right combination of descriptors…

anyways, I’ll show you more when I get to something more convincing with my non-linear patches…

1 Like

ok, I completely forgot this since I got excited on other similar threads… here we go, very dirty and undocumented, as I like my private patches. I had to spend a few minutes to remember how it works, which is not so bad for a patch from 17 Nov 2014 :slight_smile:

The audio return is on the [adc~ 11] on the left, and came from the synth from the expert sleeper via adat. The single control out is done in the middle of the patch [dac~ 11] going again to the expertsleeper via adat. Just beside this you have a switch which is off at load time (make sure you doubleclick that loadmess if you copy/paste the code)

In mode 0 (analysis) you can play first manually the float object above the switch between 0 and 1000, where 0 is outputing DC -1 and 1000 is outputing DC +1. If that modulates your patch, then you press the bang at the top and the clever 3 pass FFT re-setting is happening, scanning 1000 points for pitch with @a.harker descriptors and entrymatcher.

Once the sample is done, flick to mode 1 (playback) and you can play the keyboard or the funny sequence on the bottom right.

You can see the interpolation questions I had at the bottom. They generated my desire to be able to do data processing in Max… which generated FluCoMa grant app. Here we go :slight_smile:

1 Like


Don’t know how much you tidied before posting, but it’s pretty readable for a PA patch. Obviously some confusing bits in the middle, but it looks like I may be able to get it to work (will report back).

None! This is as is, not a single cord was moved… even the creative notes to self left there at the bottom right!

Alright! I was able to get it kind of working with a Fourses so that’s something!

I initially thought it didn’t work as I was getting nothing, but after playing with the transpose on the keyboard/sequence I found some spots.

It found only a few areas of pitch along the sweep, plus it’s hard to get that module to spit out something that resembles semi-pitchy material.

I’ll test it on a more complex patch when I have something suitable up and running.

do I detect 2 almost real compliments in 2 posts in a row? Are you ok? :rofl:

yeah, if you are more patient, you could mod the patch to enter 10k points of which 100 at full resolution, that should give you more precision… but i’m never that patient :wink: replacing entrymatcher with knn in there would be good, so we could check clustering too (in relation to our other thread)

Hehe, I like to keep all my sass in one thread at the time… :stuck_out_tongue_winking_eye:

Yeah the pre-training time doesn’t bother me too much. That super crude stuff I did with a hardware patch bay a few years ago took around 20minutes to analyze a single patch. Faster is better obviously though.

then take more time by changing the multiplier of FFT size in the middle of the patch, before you change the number of points. that will allow unstable pitch to settle.

My next steps for this are along the lines of bringing statistics of time domain windows like in LPT and other patches like fluid corpus map, to allow even more chaotic material to be musaik’d) (wow that is an ugly past participle)

1 Like

Not using FluCoMa stuff at the moment, but in working on the (epic) Kaizo Snare blog I’m recording and added other bits of related media to the project.

This is expanding on some of the stuff I did in Kaizo Snare where I take a bunch of secondary controller streams from the fader (overall activity level, distance between direction changes, time between direction changes, “velocity” from that (distance/time), etc…) and use these to control the whole patch. So 8 streams from the ES-8 controlling everything from the single fader.

Would be interesting to gestural decomposition either in an NMF-y way like that CV stuff I did at the start (which I never got working in a real-time manner) or something like that Heretic thing from the other thread.

Even some “simple” novelty stuff to send triggers when new material is detected.

The funny thing is I set off to try to get as much from a single fader as I could, and that shit fills up real quick!

1 Like

What’s the synthesis? Concatenative or one big fat synth?

Don’t know how big it is, but it’s pretty fat:

So this is modular synth stuff, with the computer/patch-side of things just doing the sensor parsing and sending it out to the modular.

Is the fourses module the same as the gen~ patch?