Fluid.mlpregressor~ not returning prediction inside collapsed M4L device

pmeyer · April 8, 2024, 4:04pm

Hello,

I have a Max for Live device that is built around Ted’s “controlling a synth using a neural network tutorial” (https://www.youtube.com/watch?v=XfNZzQPdPG0) with fluid.mlpregressor~. The patch is working as expected when the patch is opened in the Max editor, but when I collapse it down to the Live device chain, it the predictpoint message does not trigger any output. fit appears to work correctly, because the object responds with a fit value and also returns a dictionary when dumped.

Expected Behavior
After training using the fit message, fluid.mlpregressor~ responds to predictpoint messages with 'predictpoint BUFFERNAME`, in both the expanded and collapsed Max for Live Device state

Actual Behavior
fluid.mlpregressor~responds to predictpoint messages in the expanded state, but does not in the collapsed state.

I’m running FluCoMa 1.0.7 inside Max 8.6.2 with Live 12.0.1.

The M4L device I’m using for testing is here: https://drive.google.com/file/d/1AEccb7jpWpfUVHQqS_MfUN010WRL9Eek/view?usp=drive_link.

Max Support Information:

{
	"version" : "Version 8.6.2 (d076223e34e) (arm64 mac)",
	"platform" : "mac",
	"arch" : "arm64",
	"osversion" : "Mac OS X Version 13.5.2 (Build 22G91) arm64",
	"samplerate" : 44100,
	"iovs" : 512,
	"sigvs" : 256,
	"scheduler_in_audio_interrupt" : "on",
	"audio_drivername" : "Core Audio",
	"audio_driver_subname" : "",
	"license" : "permanent full",
	"machine_id" : "33b2d95acdfa379d0292a15379e68b8a",
	"addons" : 	{
		"rnbo_v1" : "full license"
	}
,
	"eventinterval" : 2,
	"schedinterval" : 1.0,
	"overdrive" : "on",
	"pollthrottle" : 40,
	"queuethrottle" : 100,
	"sysqelemthrottle" : 1000,
	"refreshrate" : 30.0,
	"schedslop" : 25.0,
	"eventprobing" : 1,
	"mixerparallel" : "off",
	"mixercrossfade" : 0,
	"mixerlatency" : 30.0,
	"mixerramptime" : 10.0,
	"videoengine" : "avf",
	"gfxengine" : "glcore",
	"packages" : 	{
		"AHarker Externals" : "1.0.0",
		"AudioMix" : "1.0.3",
		"BEAP" : "1.0.4",
		"cv" : "",
		"Dots" : "",
		"ease" : "1.2.3",
		"FluidCorpusManipulation" : "1.0.7",
		"gen-filters" : "1.0.0",
		"gl3" : "0.3.3",
		"go" : "1.0.0",
		"hap" : "1.0.6",
		"jit.mo" : "1.1.6",
		"Jitter Tools" : "1.0.10",
		"JSUI Analog Knob" : "0.0.0",
		"Learn Max" : "0.1.0",
		"link" : "1.5.5",
		"Max for Live" : "1.0.9",
		"max-mxj" : "8.2.0",
		"maxdevtools" : "",
		"maxforlive-elements" : "1.0.10",
		"Meristream" : "",
		"Mira" : "1.2.1",
		"Node for Max" : "2.1.3",
		"pdm" : "0.0.0",
		"philip-meyer-max-tutorials" : "",
		"Rhythm and Time Toolkit" : "1.0.0",
		"RNBO" : "1.2.6",
		"seq-dev" : "",
		"Syphon" : "1.0.9",
		"th.scala" : "",
		"VIDDLL" : "1.2.8",
		"Vizzie" : "2.2.2"
	}

}

tremblap · April 8, 2024, 6:25pm

now, the world leaders of M4L are here: @jamesbradbury and @rodrigo.constanzo

I never use it so I wouldn’t know where to start

pmeyer · April 8, 2024, 7:15pm

thanks @tremblap! Hey James and Rodrigo -

Hmm so I tested some more and now it is working… I don’t exactly know what changed. I will keep testing and let you know what I find.

One thing that I know that was happening with the object (and also potentially causing Max / Live to crash), was this:

If I send fluid.mlpregressor~ a blank dictionary, it posts the error fluid.mlpregressor~: Invalid JSON format to the Max console (which I would expect), but also sets all of the object parameters to 0 (which isn’t the object’s default state):

.

I would expect the object to just return the invalid JSON error but leave the params and its existing model in tact.

At one point the m4l device was trying to load the model stored as a dict in a Live parameter to fluid.mlpregressor~, and in cases where the contents of the Live parameter were empty, this invalid JSON issue happened.

jamesbradbury · April 9, 2024, 9:13am

Hey Phillip, can you grant access to the device?

rodrigo.constanzo · April 9, 2024, 11:38am

I have also experienced funny business with dependencies not being declared (by Max) properly. I think I had to had the fluid.libmanipulation (I think that’s the name, I’m away from my computer at the moment) manually to all my devices that need it.

So worth double-checking that too JIC.

tremblap · April 9, 2024, 2:13pm

now I need to test this as I think I agree with your expectations (if it fails loading, keep the previous state)

let me come back to you later if I confirm and if I can see if easy to fix…

pmeyer · April 9, 2024, 3:59pm

Sorry about that, permissions updated!

tremblap · April 12, 2024, 10:51am

interesting - the object keeps its states inside (you’ll see that if you dump it) but freaks out on the rest… I need to check in the json loading code if it not bailing at the right moment.

tremblap · April 12, 2024, 11:06am

wow, you caught something big - for those who like C++ jokes about notes to self:

github.com

flucoma/flucoma-core/blob/ab9c6501e8de8f118d313260ae02ebc5ba5ee2d2/include/data/FluidJSON.hpp#L406-L412


      
          //TODO
          bool check_json(const nlohmann::json &j, const MLP &) {
            return fluid::check_json(j,
              {"layers"},
              {JSONTypes::ARRAY}
            );
          }

good news, it is an easy fix. we just need to actually do the check

I’ll push a PR with a fix and it’ll be fixed in the next public release.

pmeyer · April 12, 2024, 3:36pm

haha, very nice. thank you pierre.

pmeyer · September 11, 2024, 6:06pm

Hi @tremblap! I want to check in on this. I am working on this device with a collaborator and am running into fairly consistent crashing issues upon initialization. Do you have a timeline for the next release, or perhaps it could be possible for us to work with a pre-release version? Thanks so much! Philip

tremblap · September 11, 2024, 8:11pm

not done yet, in a chaos moving countries… but when are you planning to release?

also, do you have problem bundling the device? Some people struggle on this other thread to find how to make it work consistently… even insiders from Cycling @MattS6464 @rodrigo.constanzo @jamesbradbury @weefuzzy @DominicThibault so sharing know-how there would be great

pmeyer · September 11, 2024, 9:39pm

I was running into troubles with freezing the max for live devices and reported it in the Cycling 74 beta group. James proposed a solution of including the initialization file that links the fluid.libmanipulation to the Max object name aliases. I’ll test that when I have some time, but it isn’t my priority at the moment because I’m not distributing this device publicly. Instead it is for an artist who is building an interactive sculpture. As a workaround we simply installed FluCoMa on the artist’s machine.

The exhibition is in early October, but we need the device to become stable as soon as possible in order to begin to train the model on gestures. Until very recently, the object seemed to be stable as long as I did not try to initialize it with @ arguments in the Max object box, but now it seems to be causing Max and Live to crash quite frequently. As a workaround, I am working on a version that hosts the fluid. elements in a separate Max patch, outside of Live, and then am communicating between the Max for Live device and the fluid objects in the Max patch using OSC. It seems to be generally more stable, but I am still getting intermittent crashing.

If you are able to prioritize this and build a pre-release version for us to try soon, I would really appreciate it. If not, I completely understand!

weefuzzy · September 12, 2024, 9:49pm

If this an issue with crashes when the object initialises, seems like it’s probably something different to the JSON checking, because that’s unconnected to initialisation.

Quick Qs:

Max version / OS? Still 8.6.2 on Mac? What version of Live?
Crashes only in Live, or in Max too?
Do you have any crash reports?
Do you have a reliably crashy patch?

FWIW, this absolutely does work – just paste the contents of FluidCorpusMaipulation/init/fluid.libmanipulation-init.txt into the openactions for your M4L project and it should resolve just fine. (James and I are currently working on trying to make this happen by magic, but no dice yet).

pmeyer · September 12, 2024, 10:22pm

Thanks @weefuzzy!

Max version is 8.6.4. Support information is below. Live version is 11.3.30. I’ve also tested in Live 12.0.25 and the same issue occurs. MacOS 13.6.9

It rarely if ever crashes in standalone Max, but crashes often in Max for Live. I know have a version that houses the Flucoma elements in a separate Max patch that talks over OSC to the device, which seems to be more stable.

Crashes most often occurs when the device is loaded in Live, or when the device is edited and saved. Sometimes it crashes Live only, sometimes it crashes Max only, and sometimes both. For me it does not crash every single time it’s loaded in Live, but probably more than half the time. For my collaborator (also 8.6.4 and Live 11) it crashes basically every time they try to load the device on their machine.

Discourse won’t allow me to upload attachments, but here are the device and two crash reports: https://www.dropbox.com/scl/fo/xjsmv03zulz2llhpdn495/ALUof70r8C4bY0MiO-jok4M?rlkey=xboi6gtapvj1zs0fepiuajegn&st=s7i3xulg&dl=0

Steps to reproduce the crash:

Open Ableton Live
Drag the Textile Controller.amxd into a MIDI track
If no crash, delete the device and repeate the previous step

Most of the time I don’t get a crash the first time I drag the device in, but after I delete it and try again, it almost always crashes.

Max Support Info:

{
	"version" : "Version 8.6.4 (0112d5ff36b) (arm64 mac)",
	"platform" : "mac",
	"arch" : "arm64",
	"osversion" : "Mac OS X Version 13.6.9 (Build 22G830) arm64",
	"samplerate" : 88200,
	"iovs" : 512,
	"sigvs" : 64,
	"scheduler_in_audio_interrupt" : "on",
	"audio_drivername" : "Core Audio",
	"audio_driver_subname" : "",
	"license" : "permanent full",
	"machine_id" : "33b2d95acdfa379d0292a15379e68b8a",
	"addons" : 	{
		"rnbo_v1" : "full license"
	}
,
	"eventinterval" : 2,
	"schedinterval" : 1.0,
	"overdrive" : "on",
	"pollthrottle" : 40,
	"queuethrottle" : 100,
	"sysqelemthrottle" : 1000,
	"refreshrate" : 30.0,
	"schedslop" : 25.0,
	"eventprobing" : 1,
	"mixerparallel" : "off",
	"mixercrossfade" : 0,
	"mixerlatency" : 30.0,
	"mixerramptime" : 10.0,
	"videoengine" : "avf",
	"gfxengine" : "glcore",
	"packages" : 	{
		"AHarker Externals" : "1.0.0",
		"AudioMix" : "1.0.3",
		"BEAP" : "1.0.4",
		"bl" : "1.0.0",
		"CNMAT Externals" : "1.0.5",
		"cv" : "",
		"Dots" : "",
		"ease" : "1.2.3",
		"FluidCorpusManipulation" : "1.0.7",
		"gen-filters" : "1.0.0",
		"gen-rack" : "",
		"gl3" : "0.3.3",
		"go" : "1.0.0",
		"hap" : "1.0.6",
		"jit.mo" : "1.1.6",
		"Jitter Tools" : "1.0.10",
		"JSUI Analog Knob" : "0.0.0",
		"Learn Max" : "0.1.0",
		"link" : "1.5.5",
		"Max for Live" : "1.0.9",
		"max-mxj" : "8.2.0",
		"maxdevtools" : "",
		"maxforlive-elements" : "1.0.12",
		"Meristream" : "",
		"Mira" : "1.2.2",
		"ml.star" : "1.3.0",
		"Node for Max" : "2.1.3",
		"odot" : "1.3.6",
		"pdm" : "0.0.0",
		"philip-meyer-max-tutorials" : "",
		"Rhythm and Time Toolkit" : "1.0.0",
		"RNBO" : "1.2.6",
		"seq-dev" : "",
		"spat5" : "5.3.4",
		"Syphon" : "1.0.9",
		"th.scala" : "",
		"VIDDLL" : "1.2.8",
		"Vizzie" : "2.2.2"
	}

}

weefuzzy · September 13, 2024, 3:25pm

Ok, think I fixed it. Can you try out the new nightly build?

Be sure to persuade M4L to use the updated version if you’ve been working with a frozen device!

Problem was that the Max object wrapper was basically ignoring when JSON parsing failed and then overwriting the object state with total junk.

pmeyer · September 13, 2024, 4:19pm

Hi @weefuzzy thanks for your help with this!

I tried this new version and experienced the same issue. You can find the crash report at the same link as above and my Max info is below.

I am pretty sure I am using the latest version… I am working with the unfrozen device and replaced the FluCoMa package folder in Max 8 > Packages with the nightly build you provided. I also went through the Project folders for the AMXD to make sure that there weren’t any rogue old versions lying around, because Max for Live likes to copy dependencies when you freeze/unfreeze.

{
	"version" : "Version 8.6.4 (0112d5ff36b) (arm64 mac)",
	"platform" : "mac",
	"arch" : "arm64",
	"osversion" : "Mac OS X Version 13.6.9 (Build 22G830) arm64",
	"samplerate" : 88200,
	"iovs" : 512,
	"sigvs" : 64,
	"scheduler_in_audio_interrupt" : "on",
	"audio_drivername" : "Core Audio",
	"audio_driver_subname" : "",
	"license" : "permanent full",
	"machine_id" : "33b2d95acdfa379d0292a15379e68b8a",
	"addons" : 	{
		"rnbo_v1" : "full license"
	}
,
	"eventinterval" : 2,
	"schedinterval" : 1.0,
	"overdrive" : "on",
	"pollthrottle" : 40,
	"queuethrottle" : 100,
	"sysqelemthrottle" : 1000,
	"refreshrate" : 30.0,
	"schedslop" : 25.0,
	"eventprobing" : 1,
	"mixerparallel" : "off",
	"mixercrossfade" : 0,
	"mixerlatency" : 30.0,
	"mixerramptime" : 10.0,
	"videoengine" : "avf",
	"gfxengine" : "glcore",
	"packages" : 	{
		"AHarker Externals" : "1.0.0",
		"AudioMix" : "1.0.3",
		"BEAP" : "1.0.4",
		"bl" : "1.0.0",
		"CNMAT Externals" : "1.0.5",
		"cv" : "",
		"Dots" : "",
		"ease" : "1.2.3",
		"FluidCorpusManipulation" : "1.0.7+sha.4aa49e0.core.sha.aa4f5601",
		"gen-filters" : "1.0.0",
		"gen-rack" : "",
		"gl3" : "0.3.3",
		"go" : "1.0.0",
		"hap" : "1.0.6",
		"jit.mo" : "1.1.6",
		"Jitter Tools" : "1.0.10",
		"JSUI Analog Knob" : "0.0.0",
		"Learn Max" : "0.1.0",
		"link" : "1.5.5",
		"Max for Live" : "1.0.9",
		"max-mxj" : "8.2.0",
		"maxdevtools" : "",
		"maxforlive-elements" : "1.0.12",
		"Meristream" : "",
		"Mira" : "1.2.2",
		"ml.star" : "1.3.0",
		"Node for Max" : "2.1.3",
		"odot" : "1.3.6",
		"pdm" : "0.0.0",
		"philip-meyer-max-tutorials" : "",
		"Rhythm and Time Toolkit" : "1.0.0",
		"RNBO" : "1.2.6",
		"seq-dev" : "",
		"spat5" : "5.3.4",
		"Syphon" : "1.0.9",
		"th.scala" : "",
		"VIDDLL" : "1.2.8",
		"Vizzie" : "2.2.2"
	}

}

weefuzzy · September 13, 2024, 4:34pm

Hmm, downer. Can’t break it here now! Can you freeze and upload your current device, and I’ll prod it again here.

Meanwhile, a quick way to double check that you have the right version in the device is by sending the version message to the mlp in Live and checking that the string in the Live max console matches the one in Max’s console

pmeyer · September 13, 2024, 5:24pm

Oh, nice tip, thanks! Confirming the version is correct. Frozen device is in the dropbox folder!

pmeyer · September 13, 2024, 6:37pm

Ok… I think I have figured out the steps to reproduce. Below is a simple patch that is triggering the crash for me. It seems that sending a load message before the object is initialized causes it to crash. This happens in my device because Live parameters are being used to store the device state with the Live set, so that when the user re-opens a previously saved set, they can pick up where they left off w/r/t training data and model.

<pre><code>
----------begin_max5_patcher----------
466.3ocsTEraaCCC8r7WgfN6EDqr3ztekhh.YKtTUHKYHImlfhtu8IQY2kr5
fYf1cHJgjOJxG4K50BBqwdB7L5OnOPIjWKHDzUxAYzlv5DmZ0BOBiYfWrMOy
JygBvo.5VqNBqBOo7R3npElhaF5TFMDvbqFcpjXJwq4aat.ncHLgbyn2rqv4
dH2hrFg4.qjxTlvzWzGGA2KBsOoLG16f1PF+1sqVWR42kN2sC+85UqoOlR3s
hhzQ4mi28NnGLRp1JjKkz74Ic0sH8Bn3tZ7r5+AEkpXsliZ7ORsp4o11aQsz
cqrFg6bZcd0m+Mo2xSm0WP5wL7hifberehIrWDBNUyPHKyIuycBC5Z.r0WWN
5QCGDsmuxUuvI5f.31CFQiFtQvNQe+6gwn3v+KZE7S8fRtpSGUaGbf2ac+hl
Bpz.uSz5rKU5UUe+7aH9MEeKaWbO9erMb9rJPL+3aDl+9sF79R9udl3sCt1o
NX7cB5e1uRvGTFQR4bAlpLlYm5KsN7kTmzL7SWopETI9GpSdPF0ZGAmeDLVh
nL5YqKYdWIZpLYyZzzEeUdBOtpYBWbQFhawAG1VrS0emkS0JAmYPgRnhD4hk
DknlnX22Kx7.UxEuU7a.MlUekA
-----------end_max5_patcher-----------
</code></pre>