Using an MLP regressor to control synth params (Re-usable GUI Class)

Here’s what I did to create a re-usable interface for an MLP regressor to use with any synth.
Basically you can take this base class and extend it to use with your synth.

A couple of notes:

  • The MLPGUI Class expects a makeSynth method to exist in your subclass. Maybe some OOP nerds can help me make this look nicer and error-proof.
  • You will need to override the n parameter of the base class in order to declare the number of parameters to be controlled by the MLP Regressor (see next post for an example) otherwise you will have an error.
  • Your synth should accept an output buffer from the MLP regressor (here called y_buf) as an input argument. Note that there is currently an open PR to allow FluidKrToBuf.kr to be initialized before the buffer is created (for example, if you are compiling a SynthDef on the server). In my example, I am setting the buffer size of FluidKrToBuf.kr to the n parameter of the class which is equal to the total number of parameters.
    Buf to kr enhancements by tedmoore · Pull Request #42 · flucoma/flucoma-sc · GitHub

So far it works pretty nicely for me. Here is the main class:

MLPGui {
  var s,n,name,
  <>w,<>synth,dsin,dsout,nn,prediction_buf,x_buf,y_buf;

  *new { |s,n|
    ^super.newCopyArgs(s ? Server.default,n).init
  }

  init {
    fork {
      dsin = FluidDataSet(s); s.sync;
      dsout = FluidDataSet(s); s.sync;
      nn = FluidMLPRegressor(
        server: s,
        hidden: [7],
        activation: FluidMLPRegressor.sigmoid,
        outputActivation: FluidMLPRegressor.sigmoid,
        learnRate: 0.1,
        batchSize: 1,
        validation: 0); s.sync;
      prediction_buf = Buffer.alloc(s,n); s.sync;
      x_buf = Buffer.alloc(s, 2); s.sync;
      y_buf = Buffer.alloc(s, n); s.sync;
    };
    Task { this.makeGUI(n) }.play(AppClock);
  }

  makeGUI {
    var counter,predicting,multisliderview;
    counter = 0;
    predicting = false;

    w = Window(name, Rect(0,0,1000,400));
    w.userCanClose = false;
    multisliderview = MultiSliderView(w,Rect(0,0,400,400))
    .size_(n)
    .elasticMode_(true)
    .action_({ |msv|
      msv.value.postln;
      y_buf.setn(0, msv.value);
    });

    Slider2D(w,Rect(400,0,400,400))
    .action_({ |xy|
      [xy.x, xy.y].postln;
      x_buf.setn(0, [xy.x, xy.y]);

      if(predicting, {
        nn.predictPoint(x_buf, y_buf, action: {
          y_buf.getn(0,n, action: { |prediction_values|
            {multisliderview.value_(prediction_values)}.defer;
          });
        });
      });
    });

    Button(w,Rect(800,0,200,20))
    .states_([["Make Synth"],["Stop Synth"]])
    .action_({ |b|
      b.value.postln;
      if (b.value == 0, { if (synth.isPlaying == false, { synth.run(false); "stop my synth".postln }) });
      if (b.value == 1, { if (synth.isPlaying == false, { this.makeSynth; "make my synth".postln }) });
    });

    Button(w,Rect(800,20,200,20))
    .states_([["Add Point"]])
    .action_({ |b|
      var id = "example-%".format(counter);
      counter = counter+1;
      dsin.addPoint(id,x_buf);
      dsout.addPoint(id,y_buf);
    });

    Button(w,Rect(800,40,200,20))
    .states_([["Train"]])
    .action_({ |b|
      nn.fit(dsin,dsout, action:{|error|
        "loss: %".format(error).postln;
      });
    });

    Button(w,Rect(800,60,200,20))
    .states_([["Not Predicting",Color.yellow,Color.black], ["Predicting",Color.green,Color.black]])
    .action_({ |b|
      predicting = b.value.asBoolean;
    });

    Button(w,Rect(800,80,200,20))
    .states_([["Print Input Dataset"]])
    .action_({ dsin.print });

    Button(w,Rect(800,100,200,20))
    .states_([["Print Output Dataset"]])
    .action_({ dsout.print });

    Button(w,Rect(800,120,200,20))
    .states_([["Randomize Parameters"]])
    .action_({ multisliderview.valueAction_(Array.fill(n, { 1.0.rand }));
    });
  } // end makeGUI

}
3 Likes

and here is an example using @tremblap 's famous Fart Synth by extending the base class:


  *new { |s,n,name|
    ^super.newCopyArgs(s ? Server.default,10,"chaosSynth").init
  }

  makeSynth {
    fork {
      synth = { |y_buf|
        var val=FluidBufToKr.kr(y_buf,n), osc1, osc2, feed1, feed2, base1=69, base2=69, base3=130;
        #feed2,feed1 = LocalIn.ar(2);
        osc1 = MoogFF.ar(SinOsc.ar((((feed1 * val[0]) +  val[1]) * base1).midicps,mul: (val[2] * 50).dbamp).atan,(base3 - (val[3] * (FluidLoudness.kr(feed2, 1, 0, hopSize: 64)[0].clip(-120,0) + 120))).lag(128/44100).midicps, val[4] * 3.5);
        osc2 = MoogFF.ar(SinOsc.ar((((feed2 * val[5]) +  val[6]) * base2).midicps,mul: (val[7] * 50).dbamp).atan,(base3 - (val[8] * (FluidLoudness.kr(feed1, 1, 0, hopSize: 64)[0].clip(-120,0) + 120))).lag(128/44100).midicps, val[9] * 3.5);
        Out.ar(0,LeakDC.ar([osc1,osc2],mul: 0.1));
        LocalOut.ar([osc1,osc2]);
      }.play; s.sync;
      synth.set(\y_buf, y_buf);
    }
  }

}
1 Like

Thank you to @tedmoore for his help with this. If anybody has any comments, suggestions or improvements please let me know and I would be really eager to hear them.

In the coming days I will try to do something similar with a KDTree classifier to provide re-usable abstractions which bundle synchronous code, common helper methods and some kind of basic GUI.

2 Likes

One useful improvement that I can think of off the bat would be a way to save the training sets. I will eventually get around to this but if someone feels like helping with that it would be very much appreciated.

2 Likes

At the workshop did @tedmoore or @tremblap show you the ability to save the internals of any of the data processing objects with the write method? You can dump them to JSON and then load them back in which is handy.

Now that you mention it I think they did say something about it.

In this case would it make sense to save both FluidDataSets? I see that FluidMLPRegressor also has a read method.

I will henceforth call it that way, with due credits given, obviously :slight_smile:

1 Like

We did indeed :slight_smile: The advantage of saving the training data is that you can retrain, but if you are curating various subset of the fart synth™ control space you don’t really need it… and this is where saving the MLP training is great: you can save them as various preset and recall subspaces, for instance have a training for long sounds, a training for noisy iterative ones, etc

If you haven’t seen it, check the video explanation of @spluta as this is what he does with a much crazier fm synth!

You can find it here!

1 Like

Hey PA: I notice that there’s a weird click whenever I start the fart synth™. I think this must have to do with the LocalIn.ar feedback loop. Do you get this click as well?

I didn’t notice - how do to initialise the 10 faders seem to change the starting sound more than anything. Did you try with something that is somewhat static ?

So I tried two things just now. Before I had been initializing the synth with an empty y_buf. I tried writing an array of zeroes into the buffer. This didn’t solve the click.

Then I replace Out.ar with OffsetOut.ar and this seemed to solve my issue.

1 Like

that is good to know. I’ll have to ask @tedmoore about that UGen and its implications in a feedback network as I don’t see how it would change things…

Hmm, actually I take that back. I’m getting the click again sometimes. I’ll take a look at it with Ted when we get the chance.

Hmm. That UGen is just for sample accurate starting of a synth. It does say that if it is used with In.ar the input will read normally from the bus, but I’m not sure how it would apply to the LocalIn.ar.

If it’s useful, the thing I would do in this scenario is to just add a Line.kr before the output that ramps from 0 to 1 over 30 milliseconds so that each time the synth is started it has a short envelope.

Three things to try:

  1. set the buffer to rands between 0 and 1 instead of all 0s

  2. use .zap from from the SafetyNet quark on the output of the MLP. This will get rid of any NaNs that you may be getting right at the start of the synth. Add ‘val=val.zap;’ above osc1

  3. clip your MLP outs between 0 and 1.

Sam

2 Likes