Kmeans real-time classifier on arbitrary input (SuperCollider)

[tedmoore]

This code runs a KMeans (every 10 seconds) on the most recent 30 seconds of audio (with whatever analysis vector you want), which is then used to classify the current audio into one of ‘k’ classes (you choose how many). It then lights up a color showing which class is currently being heard.

The idea is that it could be used to generate some audio-reactive media (lights, video, …) with audio that it has never seen before (show up to the gig and someone is sitting next to you improvising). I’ve found that for this kind of thing, the classification approach can be more engaging than a regression approach because it allows for fast changes between multimedia parameters that track with a listener’s perception of aural changes.

I’ve done basically this with a neural net in the past (also with a no-input-mixer), but of course that needs to be supervised and pre-trained, where as this could (potentially) do something interesting right out of the box with whatever is going on.

The audio file I’ve been testing this on is a no-input-mixer recording, which is a nice use case since the no-input-mixer has a somewhat narrow number of timbres can be quite easily differentiated (distant clusters, hopefully). Other sources also work, but perhaps with less audio-visual clarity.

Of course each time the KMeans recomputes, the classes/clusters might be in a different order, but maybe that’s just a nice feature in that the multimedia parameter mappings are not fixed for all of the performance.

Next might be taking the centroids and doing something clever with them, like organizing the multimedia parameters a certain way or doing a different audio-analysis–to–visual-parameter mapping based on the centroid.

highlight all and Cmd-Return should run it
You’ll need to supply your own audio file.
it’ll be just black for about 10 seconds first before it has enough data to classify––then you’ll see the colors.

I have a few questions for @tremblap @weefuzzy @groma below the code

// if you highlight all and cmd-return, it should run...
(
s.waitForBoot({
	Task({
		var updatePeriod = 10 /*in seconds*/; // how often to recompute the k means

		// how long into the past should we keep track of for clustering into k means
		var totalHistory = 30 /*in seconds*/;

		var bus = Bus.audio(s); // for sending audio to the analysis synth
		var synth; // plays a buffer
		var analysis_synth; // does analysis

		var trigRate = 30; // how many times to update every second

		// audio input - put your sound file here (or route different incoming audio to the analysis synth)
		var audioBuf = Buffer.read(s,"/Users/ted/Documents/_CREATING/_PROJECT FILES/Machine Learning/Training Data/Audio/source audio/ml examples/elec/stereos/no_input_mixer 01.wav");
		//var audioBuf = Buffer.read(s,"/Users/Ted/Music/ml test files/saxophone 01.wav");

		// keep track of how many frames have been processed so we know when we have enough data to run the k means
		var frameCounter = 0;

		var nb_of_dims = 13; // just the 13 mfccs for now...
		var dataset = FluidDataSet(s,\raw_data,nb_of_dims);
		var normed_dataset = FluidDataSet(s,\normed_dataset,nb_of_dims);
		var scaler = FluidNormalize(s);
		//var ranges = ControlSpec(inf,-inf).dup(nb_of_dims);
		var assimilate_point; // a function that normalizes the incoming points with historical extremes

		var kmeans = FluidKMeans(s);
		var k_ = 4; // what the k will be for k means
		var get_cluster; // a function that will take in a buffer (vector) update the view with  it's color

		var nHistories = trigRate * totalHistory; // total number of histories to keep track of

		var updateBuf = Buffer.alloc(s,nb_of_dims); // a buffer to use for sending new points to the FluidDataSet
		var normedBuf = Buffer.alloc(s,nb_of_dims);
		var clustered = false; // is true once at least one clustering has happened (at one point one can predict on new points)
		var buffer_filled = false; // is true once we have saved 'nHistories' vectors to the FluidDataSet
		var uvs; // userviews that will flash showing current cluster prediction
		var win; // window where the colors flash

		// the colors that flash (4 for now because 'k_' = 4, would need more with a larger k)
		var colors = [Color.cyan,Color.yellow,Color.green,Color.red];

		s.sync;

		// play sound
		synth = {
			arg rate = 0;
			var sig = PlayBuf.ar(audioBuf.numChannels,audioBuf,rate,loop:1);
			//var sig = SoundIn.ar(0);
			Out.ar(bus,Mix(sig));
			sig;
		}.play;

		s.sync;

		// define this function that will take in a buffer (vector) update the view with  it's color
		get_cluster = {
			arg unnormed_buf;

			scaler.normalizePoint(unnormed_buf,normedBuf,{

				kmeans.predictPoint(normedBuf,{
					arg cluster;

					/*				cluster.postln;
					"".postln;*/

					uvs.do({
						arg uv, i;
						if(cluster == i,{
							defer{
								//uv.visible_(true)

								// using alpha here because it might be nice to set the alpha to the
								// amplitude or something so it flickers with the sound...
								uv.background_(uv.background.alpha_(1));
							};
						},{
							defer{
								//uv.visible_(false);

								// alpha = 0 means don't show me this.
								uv.background_(uv.background.alpha_(0));
							};
						})
					});
				});
			});
		};

		// synth that does the analysis and sends the vector back to the language
		analysis_synth = {
			arg in_bus;
			var input = In.ar(in_bus);

			// put in whatever kind of analysis you want
			var vector = FluidMFCC.kr(input); // 13 by
			vector = vector[0..12];
			//KMeansRT.kr(kmeansrtBuf,vector,k_,1,0,1).poll;

			SendReply.kr(Impulse.kr(trigRate),"/analysis_vector",vector);
		}.play(synth,addAction:\addAfter,args:[\in_bus,bus]);

		// the vectors are sent from the server to here:
		OSCFunc({
			arg msg;
			var vec = msg[3..15]; // take only the 13 mfccs for now

			// load the vector to the server
			updateBuf.loadCollection(vec,0,{
				//arg buf;

				//"buffer to update: %".format(updateBuf).postln;
				if(buffer_filled,{ // if we have already put 'nHistories' points in the FluidDataSet, then...

					// update it
					dataset.updatePoint(frameCounter.asString,updateBuf,{

						// only if at least one prediction has happened, make a prediction!
						if(clustered,{
							get_cluster.(updateBuf);
						});
					});
				},{

					// if not, then, add it
					dataset.addPoint(frameCounter.asString,updateBuf,{

						// only if at least one prediction has happened, make a prediction!
						if(clustered,{
							get_cluster.(updateBuf);
						});
					});
				});

				// modulo the frameCounter
				frameCounter = (frameCounter + 1) % nHistories;

				// if we've adding 'nHistories' points, set the flag
				if(frameCounter == (nHistories - 1),{buffer_filled = true;});
			});
		},"/analysis_vector");

		// kind of cheeky, but the synth doesnt play the sound until everything is in place...
		synth.set(\rate,1);

		// for all of time, fit the k means to the dataset, but wait 'updatePeriod' in between (default is 10 sec)
		Task({
			inf.do({
				arg i;

				updatePeriod.wait;

				scaler.fit(dataset.asString,{
					scaler.normalize(dataset.asString,normed_dataset.asString,{

						kmeans.fit(normed_dataset,k_,500,action:{
							//"clustering done".postln;
							clustered = true;
						});
					});
				});
			});
		},AppClock).play;

		// window for showing the current cluster prediction
		Window.closeAll;
		win = Window("",Rect(100,100,600,800)).front;
		win.background_(Color.black);
		uvs = k_.collect({
			arg i;
			var width = win.bounds.width / k_;
			var uv = UserView(win,Rect(width * i,0,width,win.bounds.height));
			uv.background_(colors[i].alpha_(0));
			//uv.visible_(false);
		});
	},AppClock).play;
});
)

// **************** Cmd. crashes server? *********************

I find that when I Cmd-Period this (just to change something and re-run it) it crashes the server sometimes… I can try to isolate it more, just wondering if that’s a known thing the the FluidDataSet tools.

Also, when this runs I get this rolling in the post window:

I don’t know what to make of it. The code seems to be running great though.

Thank you!

[weefuzzy]

@tedmoore

I can try to isolate it more, just wondering if that’s a known thing the the FluidDataSet tools.

All and any crash reports welcome. If you open console.app and go to User Reports, you should see a delightful litany of crash reports, clearly labelled by application.

I’ve seen that sort of error message in the SC console before when trying to do things like plot changed buffers before their information has been updated language-side…

[tremblap]

This is sooooo cool!

A few ideas to try, and a few comments:

Comment#1: I have so much to learn in SC in there I’ll translate it to Max to understand each line. Thanks so much for sharing

Comment#2: the tmp file error is a SC one, as it seems that 30 buffer to file to language writing per second is not making it happy at line 112 - we have seen that in the past but I wonder if you have used the same approach instead of setn then sync in the past ? I presume we could do a similar test with vanilla objects only and see if we get errors… and if so we can send a report to the (amazing) SC dev team.

idea#1: trying with MFCC1-12 (dismissing the first coefficient MFCC0 as it is mostly loudness)

idea#2: trying the JiT approach of doing a non-real-time buffered audio that is then averaged for the full updatePeriod with a circular buffer of the totalHistory.

I’m sure @groma and @weefuzzy will have other ideas to share here.

[tremblap]

@tedmoore

loadCollection

so here we go, a Vanilla example that creates the same type of errors:

b = Buffer.alloc(s,13);

(
Routine{
	inf.do({
		b.loadCollection(Array.rand(13,-1.0,1.0),0,{"done".postln;});
		60.reciprocal.wait;
})}.play;
)

but fear not, here is the same task that is faster (because we do not move much)

(
Routine{
	inf.do({
		b.setn(0,Array.rand(13,-1.0,1.0));
		s.sync;
		"done".postln;
		60.reciprocal.wait;
})}.play;
)

I was able to crank it to 500 updates per seconds without errors, but I don’t know if it updates them all… so I did that instead.

(
a = 0;
Routine{
	inf.do({
		b.setn(0,a.dup(13););
		s.sync;
		// a.postln;
		a = a +1;
		500.reciprocal.wait;
})}.play;
)

and I definitely do not reach 500 a second I must saturate the sync mechanism… so I removed it and it is quite fun - it is crazy fast! but I wonder what are the dependencies there that are missing… I know any buffer action is sequential so if you send it the setn message then one of ours it will have to wait to do it - @weefuzzy will have a better understanding here of the dependencies, but I’ll try your code now with and without the sync

[tedmoore]

Thanks for your thoughts!

@tremblap

idea#1: trying with MFCC1-12 (dismissing the first coefficient MFCC0 as it is mostly loudness)

Yes, I considered this and understand that it’s how this is usually done–is there value in leaving in MFCC0 (as it is mostly loudness)? For example if the classification one is trying to do would benefit from loudness information? Or would it be better to use a loudness measurement for that? Perhaps this is a question for @weefuzzy?

@tremblap

idea#2: trying the JiT approach of doing a non-real-time buffered audio that is then averaged for the full updatePeriod with a circular buffer of the totalHistory .

Ah yes–great suggestion! Thank you.

[tremblap]

@tedmoore

is there value in leaving in MFCC0 (as it is mostly loudness)?

I tend to use a proper loudness descriptor with some sort of ear-emulation algo behind, like ours which uses the ITU curves. Having spectrum independent of loudness give you the possibility to weigh it independently… for instance, if you are looking for classes of sound, loudness should not be considered (an ‘s’ is an ‘s’, and a ‘f’ is an ‘f’)

[tremblap]

Hey @tedmoore did you explore a bit more along those lines? I’m about to port this to Max with some of the ideas I suggested above, but maybe you have done further experimentations that can enlighten me even more ?

[tedmoore]

I haven’t done more with this, but I did put it on my covid to do list a few days ago! I’ll play with it over the next few days and see if anything more comes about.

[tedmoore]

Greetings,

I revisited this to take a look at PA’s main suggestion, which was to do the analysis JIT style by filling a buffer and then analyzing it all at once. It ended up not working out. I ran into a couple bottlenecks / design considerations.

1. The first is that it’s actually quite elegant to have the analysis vector created on the server in one place, and then used both to populate the dataset, but also to do the classification. If one wants to create the analysis vectors for the dataset from the buffer, it would require hard coding in both places what parameters are in the vector. Also, if one wanted to use multiple Fluid analyses for their database vector (FluidMFCC & FluidSpectralShape for example), one would have to “analyze” the buffer twice, once with FluidBufMFCC and again with FluidBufSpectralShape, and then concatenate those vectors. It seems simpler to keep the analysis vector creation on the server.
2. Is there a way to initialize the number of entries in a FluidDataSet? For example, I want a new FluidDataSet on server “s”, named \teds_data, with dimensionality 13, and 600 entries: FluidDataSet(s,\teds_data,13,600). The reason I ask is that I find myself doing this thing:

if(i_have_the_num_entries_i_need,{
dataset.updatePoint(i.asString,vector);
},{
dataset.addPoint(i.asString,vector);
})

This way, I’m overwriting data points (filling it up like a circle buffer), but I can only do the overwriting if it’s the proper size (proper number of entries). If I could initialize the size, then I could just use updatePoint.

I realize I could do this manually at the outset right after creation:

n_points_i_want.do({
arg i;
dataset.addPoint(i.asString,0.dup(nb_of_dims));
});

Just wondering if it would be an easy feature to add (also might be useful to know for memory allocation?).

3. The next issue I encountered was that once I did the FluidBufMFCC and got the features buffer back (which is very quick), it then took quite a long time to get that data into a FluidDataSet so that I could then use FluidKMeans. The process of taking the frame out of the FluidBufMFCC features buffer and getting it into it’s own buffer so that I could use that to update it in the dataset, is too slow to be useful (to use in real-time applications). So I think for this reason and for #1 above, the best approach is the initial one–to add (or overwrite) vectors in the database as the come off the server with SendReply–and to also use that SendReply as the source of the vectors for the classification.
4. Lastly, my FluidKMeans seems to be crashing the server on .fit, but not every time. I haven’t isolated it more, but I’m posting the code here and the crash report.

// if you highlight all and cmd-return, it should run...
(
s.waitForBoot({
	Task({
		var updatePeriod = 5 /*in seconds*/; // how often to recompute the k means
		
		// how long into the past should we keep track of for clustering into k means
		var totalHistory = 10 /*in seconds*/;
		var circBuf = Buffer.alloc(s,totalHistory * s.sampleRate);
		
		var bus = Bus.audio(s); // for sending audio to the analysis synth
		var synth; // plays a buffer
		var analysis_synth; // does analysis
		
		var trigRate = 30; // how many times to update every second
		
		// audio input - put your sound file here (or route different incoming audio to the analysis synth)
		var audioBuf = Buffer.read(s,"/Users/ted/Documents/_CREATING/_PROJECT FILES/Machine Learning/Training Data/Audio/source audio/ml examples/elec/stereos/no_input_mixer 01.wav");
		//var audioBuf = Buffer.read(s,"/Users/Ted/Music/ml test files/saxophone 01.wav");
		
		// keep track of how many frames have been processed so we know when we have enough data to run the k means
		var frameCounter = 0;
		
		var nb_of_dims = 12; // just the 12 mfccs for now...
		var dataset = FluidDataSet(s,\raw_data,nb_of_dims);
		var normed_dataset = FluidDataSet(s,\normed_dataset,nb_of_dims);
		var scaler = FluidNormalize(s);
		//var ranges = ControlSpec(inf,-inf).dup(nb_of_dims);
		var assimilate_point; // a function that normalizes the incoming points with historical extremes
		
		var kmeans = FluidKMeans(s);
		var k_ = 4; // what the k will be for k means
		var get_cluster; // a function that will take in a buffer (vector) update the view with  it's color
		
		var nHistories = trigRate * totalHistory; // total number of histories to keep track of
		
		var updateBuf = Buffer.alloc(s,nb_of_dims); // a buffer to use for sending new points to the FluidDataSet
		var updateBuf2 = Buffer.alloc(s,nb_of_dims);
		var normedBuf = Buffer.alloc(s,nb_of_dims);
		var features = Buffer(s);
		var clustered = false; // is true once at least one clustering has happened (at one point one can predict on new points)
		var buffer_filled = false; // is true once we have saved 'nHistories' vectors to the FluidDataSet
		var uvs; // userviews that will flash showing current cluster prediction
		var win; // window where the colors flash
		
		// the colors that flash (4 for now because 'k_' = 4, would need more with a larger k)
		var colors = [Color.cyan,Color.yellow,Color.green,Color.red];
		
		s.sync;
		
		// play sound
		synth = {
			arg rate = 0;
			var sig = PlayBuf.ar(audioBuf.numChannels,audioBuf,rate,loop:1);
			//var sig = SoundIn.ar(0);
			Out.ar(bus,Mix(sig));
			sig * -20.dbamp;
		}.play;
		
		s.sync;
		
		// define this function that will take in a buffer (vector) update the view with  it's color
		get_cluster = {
			arg unnormed_buf;
			
			scaler.normalizePoint(unnormed_buf,normedBuf,{
				
				kmeans.predictPoint(normedBuf,{
					arg cluster;
					
					/*				cluster.postln;
					"".postln;*/
					
					uvs.do({
						arg uv, i;
						if(cluster == i,{
							defer{
								//uv.visible_(true)
								
								// using alpha here because it might be nice to set the alpha to the
								// amplitude or something so it flickers with the sound...
								uv.background_(uv.background.alpha_(1));
							};
						},{
							defer{
								//uv.visible_(false);
								
								// alpha = 0 means don't show me this.
								uv.background_(uv.background.alpha_(0));
							};
						})
					});
				});
			});
		};
		
		// synth that does the analysis and sends the vector back to the language
		analysis_synth = {
			arg in_bus;
			var input = In.ar(in_bus);
			
			// ------------------ original approach ---------------------
			// --------- (still need this to send real time vectors to lang) -----------------
			
			// put in whatever kind of analysis you want
			var vector = FluidMFCC.kr(input); // 13 by default
			vector = vector[1..12];
			//KMeansRT.kr(kmeansrtBuf,vector,k_,1,0,1).poll;
			
			SendReply.kr(Impulse.kr(trigRate),"/analysis_vector",vector);
			
			// ---------- jit circle buf approach as suggested by pa -------------
			
			RecordBuf.ar(input,circBuf);
			Silence.ar;
		}.play(synth,addAction:\addAfter,args:[\in_bus,bus]);
		
		// the vectors are sent from the server to here:
		OSCFunc({
			arg msg;
			var vec = msg[3..14]; // take only the 12 mfccs for now
			//vec.postln;
			
			// ----------------------- original approach --------------------------
			/*			// load the vector to the server
			updateBuf.loadCollection(vec,0,{
			//arg buf;
			
			//"buffer to update: %".format(updateBuf).postln;
			if(buffer_filled,{ // if we have already put 'nHistories' points in the FluidDataSet, then...
			
			// update it
			dataset.updatePoint(frameCounter.asString,updateBuf,{
			
			// only if at least one prediction has happened, make a prediction!
			if(clustered,{
			get_cluster.(updateBuf);
			});
			});
			},{
			
			// if not, then, add it
			dataset.addPoint(frameCounter.asString,updateBuf,{
			
			// only if at least one prediction has happened, make a prediction!
			if(clustered,{
			get_cluster.(updateBuf);
			});
			});
			});
			
			// modulo the frameCounter
			frameCounter = (frameCounter + 1) % nHistories;
			
			// if we've adding 'nHistories' points, set the flag
			if(frameCounter == (nHistories - 1),{buffer_filled = true;});
			});*/
			
			// updateBuf2.loadCollection(vec,0,{
			// 	if(clustered,{
			// 		get_cluster.(updateBuf2);
			// 	});
			// });
			
			if(clustered,{
				Routine({
					updateBuf2.setn(0,vec);
					s.sync;
					get_cluster.(updateBuf2);
				}).play;
			});
		},"/analysis_vector");
		
		// kind of cheeky, but the synth doesnt play the sound until everything is in place...
		synth.set(\rate,1);
		
		// for all of time, fit the k means to the dataset, but wait 'updatePeriod' in between (default is 10 sec)
		Task({
			
			// wait for the whole circle buffer to fill up!
			totalHistory.wait;
			
			inf.do({
				arg i;
				i.postln;
				// ------------- original approach
				/*				scaler.fit(dataset.asString,{
				scaler.normalize(dataset.asString,normed_dataset.asString,{
				
				kmeans.fit(normed_dataset,k_,500,action:{
				//"clustering done".postln;
				clustered = true;
				});
				});
				});*/
				
				// get the mfcc features
				FluidBufMFCC.process(s,circBuf,features:features,action:{
					features.postln;
					
					// load the features into the dataset;
					features.numFrames.do({
						arg frame_i;
						"frame i: %".format(frame_i).postln;
						
						//features.copyData(updateBuf,(frame_i * features.numChannels) + 1,0,nb_of_dims);
						
						features.loadToFloatArray((frame_i * features.numChannels) + 1,nb_of_dims,{
							arg float_array;
							/*							updateBuf.loadCollection(float_array,0,{
							if(i == 0,{ // if this is the first time, add a point, otherwise, update
							dataset.addPoint(frame_i.asString,updateBuf);
							},{
							dataset.updatePoint(frame_i.asString,updateBuf);
							});
							});*/
							updateBuf.setn(0,float_array);
							s.sync;
							if(i == 0,{ // if this is the first time, add a point, otherwise, update
								dataset.addPoint(frame_i.asString,updateBuf);
							},{
								dataset.updatePoint(frame_i.asString,updateBuf);
							});
						});
					});
					
					// normalize the dataset
					scaler.fit(dataset.asString,{
						scaler.normalize(dataset.asString,normed_dataset.asString,{
							kmeans.fit(normed_dataset,k_,500,action:{
								"clustering done".postln;
								clustered = true;
							});
						});
					});
				});
				
				updatePeriod.wait;
			});
		},AppClock).play;
		
		// window for showing the current cluster prediction
		Window.closeAll;
		win = Window("",Rect(Window.screenBounds.width + 100,100,600,800)).front;
		win.background_(Color.black);
		uvs = k_.collect({
			arg i;
			var width = win.bounds.width / k_;
			var uv = UserView(win,Rect(width * i,0,width,win.bounds.height));
			uv.background_(colors[i].alpha_(0));
			//uv.visible_(false);
		});
	},AppClock).play;
});
)

Process: coreaudiod [236]
Path: /usr/sbin/coreaudiod
Identifier: coreaudiod
Version: 5.0 (5.0)
Code Type: X86-64 (Native)
Parent Process: launchd [1]
Responsible: coreaudiod [236]
User ID: 202

PlugIn Path: /Library/Audio/Plug-Ins/HAL/BlackHole.driver/Contents/MacOS/BlackHole
PlugIn Identifier: audio.existential.BlackHole
PlugIn Version: 0.2.6 (1)

Date/Time: 2020-04-05 15:04:52.800 -0400
OS Version: Mac OS X 10.15.4 (19E266)
Report Version: 12
Bridge OS Version: 3.0 (14Y908)
Anonymous UUID: 725D907C-009E-FB2F-BA7A-3FC2D83702A3

Sleep/Wake UUID: 338BDEC6-7350-46C9-978F-08FF761DEBBE

Time Awake Since Boot: 130000 seconds
Time Since Wake: 52000 seconds

System Integrity Protection: enabled

Crashed Thread: 7 audio IO: BlackHole_UID

Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00000001056fc000
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [236]

VM Regions Near 0x1056fc000:
MALLOC_LARGE 000000010567c000-00000001056fc000 [ 512K] rw-/rwx SM=PRV
–> VM_ALLOCATE 00000001056fc000-00000001056fd000 [ 4K] r–/rwx SM=PRV
VM_ALLOCATE 00000001056fd000-00000001057fd000 [ 1024K] rw-/rwx SM=PRV

Application Specific Information:
dyld3 mode

Thread 0:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff6a2cedfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff6a2cf170 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff302460b5 __CFRunLoopServiceMachPort + 247
3 com.apple.CoreFoundation 0x00007fff30244b82 __CFRunLoopRun + 1319
4 com.apple.CoreFoundation 0x00007fff30243ffe CFRunLoopRunSpecific + 462
5 coreaudiod 0x0000000101c14c29 main + 5308
6 libdyld.dylib 0x00007fff6a18dcc9 start + 1

Thread 1:: AMCP Logging Spool
0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.caulk 0x00007fff63d67b16 caulk::mach::semaphore::wait() + 16
2 com.apple.audio.caulk 0x00007fff63d679b2 caulk::semaphore::timed_wait(double) + 106
3 com.apple.audio.caulk 0x00007fff63d677c4 caulk::concurrent::details::worker_thread::run() + 30
4 com.apple.audio.caulk 0x00007fff63d671e4 void* caulk::thread_proxy<std::__1::tuple<caulk::attributes, void (caulk::concurrent::details::worker_thread::)(), std::__1::tuplecaulk::concurrent::details::worker_thread* > >(void) + 45
5 libsystem_pthread.dylib 0x00007fff6a392109 _pthread_start + 148
6 libsystem_pthread.dylib 0x00007fff6a38db8b thread_start + 15

Thread 2:
0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.caulk 0x00007fff63d67b16 caulk::mach::semaphore::wait() + 16
2 com.apple.audio.caulk 0x00007fff63d679b2 caulk::semaphore::timed_wait(double) + 106
3 com.apple.audio.caulk 0x00007fff63d677c4 caulk::concurrent::details::worker_thread::run() + 30
4 com.apple.audio.caulk 0x00007fff63d671e4 void* caulk::thread_proxy<std::__1::tuple<caulk::attributes, void (caulk::concurrent::details::worker_thread::)(), std::__1::tuplecaulk::concurrent::details::worker_thread* > >(void) + 45
5 libsystem_pthread.dylib 0x00007fff6a392109 _pthread_start + 148
6 libsystem_pthread.dylib 0x00007fff6a38db8b thread_start + 15

Thread 3:: HAL Async Logger
0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.CoreAudio 0x00007fff2f77908a ca::mach::semaphore::wait() + 16
2 com.apple.audio.CoreAudio 0x00007fff2fa688ce ca::concurrent::details::worker_thread::run(ca::attributes) + 390
3 com.apple.audio.CoreAudio 0x00007fff2fa689fd void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void (ca::concurrent::details::worker_thread::)(ca::attributes), ca::concurrent::details::worker_thread, ca::attributes> >(void*) + 188
4 libsystem_pthread.dylib 0x00007fff6a392109 _pthread_start + 148
5 libsystem_pthread.dylib 0x00007fff6a38db8b thread_start + 15

Thread 4:: Audio HAL Overload Reporting Spool
0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.CoreAudio 0x00007fff2f77908a ca::mach::semaphore::wait() + 16
2 com.apple.audio.CoreAudio 0x00007fff2fa688ce ca::concurrent::details::worker_thread::run(ca::attributes) + 390
3 com.apple.audio.CoreAudio 0x00007fff2fa689fd void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void (ca::concurrent::details::worker_thread::)(ca::attributes), ca::concurrent::details::worker_thread, ca::attributes> >(void*) + 188
4 libsystem_pthread.dylib 0x00007fff6a392109 _pthread_start + 148
5 libsystem_pthread.dylib 0x00007fff6a38db8b thread_start + 15

Thread 5:
0 libsystem_kernel.dylib 0x00007fff6a2d70fe __select + 10
1 com.zerodebug.audiomux 0x0000000101f8bffd LocalClient::Work() + 1581
2 com.zerodebug.audiomux 0x0000000101f95dae client_thread_proc(void*) + 14
3 libsystem_pthread.dylib 0x00007fff6a392109 _pthread_start + 148
4 libsystem_pthread.dylib 0x00007fff6a38db8b thread_start + 15

Thread 6:
0 libsystem_pthread.dylib 0x00007fff6a38db68 start_wqthread + 0

Thread 7 Crashed:: audio IO: BlackHole_UID
0 libsystem_platform.dylib 0x00007fff6a383dc9 _platform_bzero$VARIANT$Haswell + 41
1 audio.existential.BlackHole 0x0000000101ed2f47 BlackHole_DoIOOperation + 759
2 com.apple.audio.CoreAudio 0x00007fff2fb1182f HALS_PlugInEngine::_ReadFromStream_Read(HALS_IOEngine2_IOContextInfo&, HALS_IOEngine2_StreamInfo&, AudioServerPlugInIOCycleInfo const&, unsigned int, void*) + 91
3 com.apple.audio.CoreAudio 0x00007fff2fa1dbbf HALS_IOEngine2::_ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 869
4 com.apple.audio.CoreAudio 0x00007fff2fb11740 HALS_PlugInEngine::_ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 54
5 com.apple.audio.CoreAudio 0x00007fff2fa21599 invocation function for block in HALS_IOEngine2::ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 678
6 com.apple.audio.CoreAudio 0x00007fff2fa6d6a4 HALB_CommandGate::ExecuteCommand(void () block_pointer) const + 98
7 com.apple.audio.CoreAudio 0x00007fff2fa1b316 HALS_IOEngine2::ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 138
8 com.apple.audio.CoreAudio 0x00007fff2f9f51b6 HALS_IOContext::PerformIO_ReadFromStream(void*, unsigned int, HALS_IOStreamInfo&, unsigned int) + 86
9 com.apple.audio.CoreAudio 0x00007fff2f9ef268 std::__1::function<int (void*, unsigned int, HALS_IOStreamInfo&, unsigned int)>::operator()(void*, unsigned int, HALS_IOStreamInfo&, unsigned int) const + 42
10 com.apple.audio.CoreAudio 0x00007fff2f9f55ab std::__1::function<int (unsigned int, HALS_IOStreamInfo&, unsigned int)>::operator()(unsigned int, HALS_IOStreamInfo&, unsigned int) const + 37
11 com.apple.audio.CoreAudio 0x00007fff2f9f3d2c HALS_IOContext::PerformIO(AudioTimeStamp const&, unsigned int, int, unsigned int) + 968
12 com.apple.audio.CoreAudio 0x00007fff2f9f29b0 HALS_IOContext::IOThreadEntry(void*) + 9536
13 com.apple.audio.CoreAudio 0x00007fff2f9f03bf invocation function for block in HALS_IOContext::HALS_IOContext(HALS_System*, HALS_Client*, HALS_System::PowerState) + 13
14 com.apple.audio.CoreAudio 0x00007fff2f95ec22 HALB_IOThread::Entry(void*) + 72
15 libsystem_pthread.dylib 0x00007fff6a392109 _pthread_start + 148
16 libsystem_pthread.dylib 0x00007fff6a38db8b thread_start + 15

Thread 7 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x0000700001804540 rcx: 0x0000000000050900 rdx: 0x000000010567c000
rdi: 0x00000001056fc000 rsi: 0x0000000000000000 rbp: 0x0000700001804410 rsp: 0x0000700001804410
r8: 0x0000000072656164 r9: 0x00000000000035ec r10: 0x00000001058fd000 r11: 0x0000000000287600
r12: 0x00007fbbbbb1eb30 r13: 0x00007fbbbbb13200 r14: 0x00007a3c59b29cd8 r15: 0x00007fbbbbb13cd0
rip: 0x00007fff6a383dc9 rfl: 0x0000000000010206 cr2: 0x00000001056fc000

Logical CPU: 2
Error Code: 0x00000007 (invalid protections for user data write)
Trap Number: 14

[tremblap]

I’m back from a few days of unsuccessful resting, so my brains are slow, but I’ll check this in the next few hours and come back to you with thoughts.

[tremblap]

Hello again! Caffeine has kicked in, so I am now able to reply to some questions, but I will have to discuss most with @weefuzzy and @groma as we are working on such improvments on the interface and manipulation of data, and in particular in SC.

Your point 1 I’m not sure I get. The analysis vector can be kept on the server, you can keep it in a buffer there, no? I might not get your proposal if that is not what you meant. We are also toying with copying vectors between datasets, and therefore you could then make a dataset of descriptor vectors, and play with them as you want.

Point 2, not a the moment, I’ll forward to the team.

Point 3, @groma will have a comment since we were talking about that very workflow and I think he had a solution.

Point 4 is more complicated to trace, but @weefuzzy is it possible it is the threading exception you found?
@tedmoore I get ‘Silence.ar Class not defined’ but I reckon it is Silent.ar(). I don’t manage to crash your code in either version (the one you have and the tip of the dev compile) but there is a slight difference of behaviour.

I get, after a certain time, in the post window:

0
Synth(‘temp__18’ : 1017)
Buffer(17, 862, 13, 86.1328125, nil)
frame i: 0
frame i: 1
and frame goes up forever

The only difference I get between current tip and the version you have is that I managed to crash the server when I stop the process in the old version (yours) once. I did not manage to crash it in my version…

I am aware that this answer is opening more questions… but this is super useful, thanks for posting, and more soon!

[groma]

Hello, regarding issue 3, I think @tremblap is referring to this example, which simplifies the flattening of interleaved buffer obtained via FluidBufStats
~dest = Buffer.alloc(s, ~statsBuf.numChannels * ~statsBuf.numFrames);
~statsBuf.getToFloatArray(0, action:{|a| ~dest.setn(0, a.unlace(~statsBuf.numChannels).flatten)});

I think here the problem is different, although I think clearly you are making an extra trip with the SendReply, instead you could store vector in a buffer in the server when it comes out of FluidMFCC. I have not tried, but maybe an array of RecordBuf can be used to write each channel of the output into a position in the buffer, or something like that, then in the language load that buffer in the dataset.
I agree it is not optimal and we need a more direct way to communicate to FluidDataset server side.

[weefuzzy]

A note on the crash. It looks like a memory stamp. I’ve only been able to get it to happen once so far, and not when I had the debugger attached

Unfortunately this was against our dev branch, so my initial hope that this might have been something I’d already fixed looks to be folorn. It won’t be the threading thing @tremblap mentioned, but it could still be another threading thing as my crash was when freeing the synths, and this part of the wrapper is currently the Most Problematic.

[tedmoore]

@tremblap

The analysis vector can be kept on the server, you can keep it in a buffer there, no? I might not get your proposal if that is not what you meant. We are also toying with copying vectors between datasets, and therefore you could then make a dataset of descriptor vectors, and play with them as you want.

Ah yes, I think what I was imagining was a circle buffer of audio, which would then be analyzed with a (or multiple) FluidBuf- analyses from the language. But now I see what would be better is to do all the analysis in real-time on the server (no FluidBuf- analyses at all), and pack that into a vector there, then write it to a circle buffer of descriptors––not a circle buffer of audio. However I think the problem of loading into the FluidDataSet remains. This relates to @groma’s reply below, so I’ll detail it there.

[tedmoore]

@groma

making an extra trip with the SendReply

Yeah for sure. I realize that what would be better (as I mentioned in my reply to @tremblap) would be to do all the analysis on the same synth def and then write the analysis vector some kind of circle buffer rather than store it in an audio circle buffer to be analyzed later. That way my analysis is done and already on the server in a known place.

Is there (or can there be) a way to convert a circle buffer of analysis vectors (nChannels in the buffer = the dimensionality of the data, nFrames = number of data points) directly into a FluidDataSet or just overwrite an existing FluidDataSet with whatever is in that buffer? Or better yet something like

FluidDataSet.kr(analysis_vector)

right on the server that will just fill it up (and circle around, continuously filling it up like a circle buffer)? That way whenever I want to run a FluidKMeans or whatever, the data is already there!

Otherwise, my circle buffer of analysis vectors would still need to be sliced up into individual frames and added to a FluidDataSet with .addPoint(copied_buffer) right?

I think maybe the best solution right now is to be constantly writing my analysis vector to a 1 frame buffer (with nChannels = dimensionality of the vector) on the server, and then in the language, in a loop, running FluidDataSet.updatePoint(buffer) on the same buffer over and over (since the data in it will be different every time). Does that sound like a reasonable approach to this circle buffer of analysis data idea?

I’ll try to implement this soon and see how it runs.

Thanks all!

[tremblap]

@tedmoore

just overwrite an existing FluidDataSet with whatever is in that buffer?

not as elegantly as you propose yet, but you can overwrite items, yes, as you propose

@tedmoore

Does that sound like a reasonable approach to this circle buffer of analysis data idea?

sounds like an expensive one I think, but since you cannot copy vectors out of datasets yet (we are working on solutions on this as we are also users and have what seems to be the same stress points) then you can try this way and see what happens.

If we were able to run stats on vectors, then that would solve this even better I think - you could write your dataset of descriptors, then make stats to another dataset and use the latter to do your classification…

[tedmoore]

@groma

but maybe an array of RecordBuf can be used to write each channel of the output into a position in the buffer

it turns out RecordBuf won’t work because then the buffer would be 1 frame x nDimensions channels, but to use FluidDataSet.addPoint it has to be 1 channel x nDimensions frames. However this will work in a synth for a buffer of the latter shape:

vector.do({
	arg val, i;
	BufWr.kr(val,my_buffer,i);
});

Perhaps FluidDataSet.addPoint could be modified to take a buffer of either shape (since the “size” of them are the same)?

[tedmoore]

I completed an edit of this with the design suggestions. Basically avoiding sending vectors to the language and back to the server. The result and performance are about the same. It would still be great to add the features desired above, but feel free to check out this if it’s useful.

There is one question that came out of this. I was sometimes getting this error: “Error: Already processing”. I think it had to do with creating a FluidDataSet, maybe one whose name was already used on the server? Is there a way to create a FluidDataSet and avoid this?

// if you highlight all and cmd-return, it should run...
(
s.options.device_("Scarlett 6i6 USB");
s.waitForBoot({
	Task({
		var updatePeriod = 5 /*in seconds*/; // how often to recompute the k means
		var totalHistory = 10 /*in seconds*/; // how long into the past should we keep track of for clustering into k means
		var bus = Bus.audio(s); // for sending audio to the analysis synth
		var synth; // plays a buffer
		var analysis_synth; // does analysis
		var trigRate = 30; // how many times to update every second
		var audioBuf = Buffer.read(s,"/Users/ted/Documents/_CREATING/_PROJECT FILES/Machine Learning/Training Data/Audio/source audio/ml examples/elec/stereos/no_input_mixer 01.wav");

		var nb_of_dims = 12; // just the 12 mfccs for now...
		var dataset;// = FluidDataSet(s,\raw_data,nb_of_dims);
		var normed_dataset;// = FluidDataSet(s,\normed_dataset,nb_of_dims);
		var scaler = FluidNormalize(s);

		var kmeans = FluidKMeans(s);
		var k_ = 4; // what the k will be for k means
		var get_cluster; // a function that will take in a buffer (vector) update the view with  it's color

		var nHistories = trigRate * totalHistory; // total number of histories to keep track of

		var updateBuf = Buffer.alloc(s,nb_of_dims); // a buffer to use for sending new points to the FluidDataSet
		var normedBuf = Buffer.alloc(s,nb_of_dims); // a buffer to put a normalized datapoint in
		var clustered = false; // is true once at least one clustering has happened (at one point one can predict on new points)
		var buffer_filled = false;
		var uvs; // userviews that will flash showing current cluster prediction
		var win; // window where the colors flash
		var serverBuf = Buffer.alloc(s,nb_of_dims);
		var dummyBuf = Buffer.alloc(s,nb_of_dims);
		var update_counter = 0;

		// the colors that flash (4 for now because 'k_' = 4, would need more with a larger k)
		var colors = [Color.cyan,Color.yellow,Color.green,Color.red];

		s.sync;

		dataset = FluidDataSet(s,\raw_data,nb_of_dims);
		normed_dataset = FluidDataSet(s,\normed_dataset,nb_of_dims);

		s.sync;

		/*
		because the datasets are "named" on the server, does it make sense that i should probably clear them each time,
		just to be sure that there isn't any residual data lingering in these "names"
		*/
		dataset.clear;
		normed_dataset.clear;

		dummyBuf.setn(0,0.dup(nb_of_dims));

		s.sync;

		// initialize the size of the dataset
		/*		nHistories.do({
		arg i;
		i.postln;
		dataset.addPoint(i.asString,dummyBuf);
		s.sync;
		/*

		i decided against this dataset size initialization strategy, since it took a little bit of time to
		complete and as i was iterating on this code, i decided it was too long...

		*/
		});*/

		s.sync;

		// play sound
		synth = {
			arg rate = 1;
			var sig = PlayBuf.ar(audioBuf.numChannels,audioBuf,rate,loop:1);
			//var sig = SoundIn.ar(0);
			Out.ar(bus,Mix(sig));
			sig;// * -20.dbamp;
		}.play;

		s.sync;

		// define this function that will take in a buffer (vector) update the view with  it's color
		get_cluster = {
			arg unnormed_buf; // take in the raw vector that is on the server

			//normalize it into the normedBuf
			scaler.normalizePoint(unnormed_buf,normedBuf,{

				// use that to predict
				kmeans.predictPoint(normedBuf,{
					arg cluster;

					// update views.
					uvs.do({
						arg uv, i;
						if(cluster == i,{
							defer{
								// using alpha here because it might be nice to set the alpha to the
								// amplitude or something so it flickers with the sound...
								uv.background_(uv.background.alpha_(1));
							};
						},{
							defer{
								// alpha = 0 means don't show me this.
								uv.background_(uv.background.alpha_(0));
							};
						})
					});
				});
			});
		};

		// synth that does the analysis
		analysis_synth = {
			arg in_bus;
			var input = In.ar(in_bus);
			var vector = FluidMFCC.kr(input); // 13 by default
			vector = vector[1..12];

			// a "clock" to send to the language... explained below at OSCFunc...
			SendReply.kr(Impulse.kr(trigRate),"/clock");
			//vector.poll;

			// i wish...
			//RecordBuf.kr(vector,serverBuf);

			vector.do({
				arg val, i;
				BufWr.kr(val,serverBuf,i);
				/*

				has to work this way because FluidDataSet.updatePoint needs to have a 1 channel buf with
				nFrames = num dimensions

				the RecordBuf.kr above would be simpler, however then the buffer would need to be a 1 frame
				buf with nChannels = num dimensions

				maybe ".updatePoint" could be modified to take a buffer that is either *1 channel x nDimensions frames* OR
				*nDimensions channels x 1 frame* ?

				*/
			});

			Silent.ar;
		}.play(synth,addAction:\addAfter,args:[\in_bus,bus]);

		OSCFunc({
			var label = (update_counter % nHistories).asString;

			if(update_counter % trigRate == 0,{
				"currently updating data point: %".format(label).postln;
			});

			if(buffer_filled,{
				dataset.updatePoint(label,serverBuf);

				if(clustered,{
					get_cluster.(serverBuf);
				});
			},{
				dataset.addPoint(label,serverBuf);
				if(update_counter == (nHistories - 1),{buffer_filled = true;});
			});

			update_counter = update_counter + 1;
			/*

			i originally had this update functionality (i.e., the "updatePoint" and the "get_cluster") happening just in a loop in a routine with
			"trigRate.reciprocal.wait" in the loop, but it turn sout that "updatePoint" takes some time, so
			each iteration of the loop was much longer than trigRate.reciprocal.

			so now i'm using an Impulse.kr on the server to "clock" my "addPoint" and "get_cluster" through an OSCFunc

			this is better though because then i'm not sending the vector back to the language, loading it back to the server in
			*another* buffer, to finally add to the dataset

			(for what it's worth, this way of clocking something in the language with an OSCFunc is something i find myself
			doing often... it works quite well, however does seem a little weird. does anyone know of a better way to be
			thinking about this?)

			*/
		},"/clock");

		Routine({

			// wait for the whole buffer to fill up!
			totalHistory.wait;
			/*

			infinitely loop this: do the things to get the kmeans...

			*/
			inf.do({
				arg i;
				"clustering for the % th time".format(i).postln;

				scaler.fit(dataset.asString,{
					"fitting to the scaler complete".postln;
					scaler.normalize(dataset.asString,normed_dataset.asString,{
						"normalizing the dataset complete".postln;
						kmeans.fit(normed_dataset,k_,action:{
							"fitting kmeans complete".postln;
							clustered = true; // this way after this has happened at least once, we'll know we can do "predict" above
						});
					});
				});

				updatePeriod.wait; // only update this ever so often...
			});
		}).play;

		// window for showing the current cluster prediction
		Window.closeAll;
		win = Window("",Rect(Window.screenBounds.width + 100,100,600,800)).front;
		win.background_(Color.black);
		uvs = k_.collect({
			arg i;
			var width = win.bounds.width / k_;
			var uv = UserView(win,Rect(width * i,0,width,win.bounds.height));
			uv.background_(colors[i].alpha_(0));
			//uv.visible_(false);
		});
	},AppClock).play;
});
)

[tremblap]

Hello!

Thanks for this. We’re in a tight deadline for a paper now, but I’m sure help will be on its way soon on this too, stay tuned!

[weefuzzy]

@tedmoore

I was sometimes getting this error: “Error: Already processing”

Hmm. In my imagination, you shouldn’t ever see this message in relation to FluidDataSet. It’s to do with making repeated calls to a threaded non-realtime object it’s finished its work. Creating language-side instances with the same name gives you a representation of the same object on the server, so I guess its possible that requests end up overlapping this way, except that all the methods for FluidDataSet are meant to be synchronous (w.r.t the server command FIFO), because the object itself is not thread-safe. I will investigate.

Re speed. Without ignoring the need to come up with a server-side workflow, I think the back and forth approach you’re currently stuck with isn’t helped by the amount of syncing that goes on. I’m now wandering about the wisdom of baking a sync into the the back-end. If you fancy a dangerours experiment, try commenting out the sync on line 29 in FluidManipulationClient:

                    result = FluidMessageResponse.collectArgs(parser,msg.drop(3));
                    this.server.sync;
                    c.test = true;
                    c.signal;

In theory this should be safe , but if I’m wrong about these functions executing synchrnously w/r/t the server command thread, strange things might happen.

Finally, about intialization: the challenge is that datasets are a mapping between keys and data, so an ‘empty’ point still needs a unique label associated with it.