# Kmeans real-time classifier on arbitrary input (SuperCollider)

This code runs a KMeans (every 10 seconds) on the most recent 30 seconds of audio (with whatever analysis vector you want), which is then used to classify the current audio into one of ‘k’ classes (you choose how many). It then lights up a color showing which class is currently being heard.

The idea is that it could be used to generate some audio-reactive media (lights, video, …) with audio that it has never seen before (show up to the gig and someone is sitting next to you improvising). I’ve found that for this kind of thing, the classification approach can be more engaging than a regression approach because it allows for fast changes between multimedia parameters that track with a listener’s perception of aural changes.

I’ve done basically this with a neural net in the past (also with a no-input-mixer), but of course that needs to be supervised and pre-trained, where as this could (potentially) do something interesting right out of the box with whatever is going on.

The audio file I’ve been testing this on is a no-input-mixer recording, which is a nice use case since the no-input-mixer has a somewhat narrow number of timbres can be quite easily differentiated (distant clusters, hopefully). Other sources also work, but perhaps with less audio-visual clarity.

Of course each time the KMeans recomputes, the classes/clusters might be in a different order, but maybe that’s just a nice feature in that the multimedia parameter mappings are not fixed for all of the performance.

Next might be taking the centroids and doing something clever with them, like organizing the multimedia parameters a certain way or doing a different audio-analysis–to–visual-parameter mapping based on the centroid.

highlight all and Cmd-Return should run it
You’ll need to supply your own audio file.
it’ll be just black for about 10 seconds first before it has enough data to classify––then you’ll see the colors.

I have a few questions for @tremblap @weefuzzy @groma below the code

// if you highlight all and cmd-return, it should run...
(
s.waitForBoot({
var updatePeriod = 10 /*in seconds*/; // how often to recompute the k means

// how long into the past should we keep track of for clustering into k means
var totalHistory = 30 /*in seconds*/;

var bus = Bus.audio(s); // for sending audio to the analysis synth
var synth; // plays a buffer
var analysis_synth; // does analysis

var trigRate = 30; // how many times to update every second

// audio input - put your sound file here (or route different incoming audio to the analysis synth)
var audioBuf = Buffer.read(s,"/Users/ted/Documents/_CREATING/_PROJECT FILES/Machine Learning/Training Data/Audio/source audio/ml examples/elec/stereos/no_input_mixer 01.wav");
//var audioBuf = Buffer.read(s,"/Users/Ted/Music/ml test files/saxophone 01.wav");

// keep track of how many frames have been processed so we know when we have enough data to run the k means
var frameCounter = 0;

var nb_of_dims = 13; // just the 13 mfccs for now...
var dataset = FluidDataSet(s,\raw_data,nb_of_dims);
var normed_dataset = FluidDataSet(s,\normed_dataset,nb_of_dims);
var scaler = FluidNormalize(s);
//var ranges = ControlSpec(inf,-inf).dup(nb_of_dims);
var assimilate_point; // a function that normalizes the incoming points with historical extremes

var kmeans = FluidKMeans(s);
var k_ = 4; // what the k will be for k means
var get_cluster; // a function that will take in a buffer (vector) update the view with  it's color

var nHistories = trigRate * totalHistory; // total number of histories to keep track of

var updateBuf = Buffer.alloc(s,nb_of_dims); // a buffer to use for sending new points to the FluidDataSet
var normedBuf = Buffer.alloc(s,nb_of_dims);
var clustered = false; // is true once at least one clustering has happened (at one point one can predict on new points)
var buffer_filled = false; // is true once we have saved 'nHistories' vectors to the FluidDataSet
var uvs; // userviews that will flash showing current cluster prediction
var win; // window where the colors flash

// the colors that flash (4 for now because 'k_' = 4, would need more with a larger k)
var colors = [Color.cyan,Color.yellow,Color.green,Color.red];

s.sync;

// play sound
synth = {
arg rate = 0;
var sig = PlayBuf.ar(audioBuf.numChannels,audioBuf,rate,loop:1);
//var sig = SoundIn.ar(0);
Out.ar(bus,Mix(sig));
sig;
}.play;

s.sync;

// define this function that will take in a buffer (vector) update the view with  it's color
get_cluster = {
arg unnormed_buf;

scaler.normalizePoint(unnormed_buf,normedBuf,{

kmeans.predictPoint(normedBuf,{
arg cluster;

/*				cluster.postln;
"".postln;*/

uvs.do({
arg uv, i;
if(cluster == i,{
defer{
//uv.visible_(true)

// using alpha here because it might be nice to set the alpha to the
// amplitude or something so it flickers with the sound...
uv.background_(uv.background.alpha_(1));
};
},{
defer{
//uv.visible_(false);

// alpha = 0 means don't show me this.
uv.background_(uv.background.alpha_(0));
};
})
});
});
});
};

// synth that does the analysis and sends the vector back to the language
analysis_synth = {
arg in_bus;
var input = In.ar(in_bus);

// put in whatever kind of analysis you want
var vector = FluidMFCC.kr(input); // 13 by
vector = vector[0..12];
//KMeansRT.kr(kmeansrtBuf,vector,k_,1,0,1).poll;

// the vectors are sent from the server to here:
OSCFunc({
arg msg;
var vec = msg[3..15]; // take only the 13 mfccs for now

// load the vector to the server
//arg buf;

//"buffer to update: %".format(updateBuf).postln;
if(buffer_filled,{ // if we have already put 'nHistories' points in the FluidDataSet, then...

// update it
dataset.updatePoint(frameCounter.asString,updateBuf,{

// only if at least one prediction has happened, make a prediction!
if(clustered,{
get_cluster.(updateBuf);
});
});
},{

// if not, then, add it

// only if at least one prediction has happened, make a prediction!
if(clustered,{
get_cluster.(updateBuf);
});
});
});

// modulo the frameCounter
frameCounter = (frameCounter + 1) % nHistories;

// if we've adding 'nHistories' points, set the flag
if(frameCounter == (nHistories - 1),{buffer_filled = true;});
});
},"/analysis_vector");

// kind of cheeky, but the synth doesnt play the sound until everything is in place...
synth.set(\rate,1);

// for all of time, fit the k means to the dataset, but wait 'updatePeriod' in between (default is 10 sec)
inf.do({
arg i;

updatePeriod.wait;

scaler.fit(dataset.asString,{
scaler.normalize(dataset.asString,normed_dataset.asString,{

kmeans.fit(normed_dataset,k_,500,action:{
//"clustering done".postln;
clustered = true;
});
});
});
});
},AppClock).play;

// window for showing the current cluster prediction
Window.closeAll;
win = Window("",Rect(100,100,600,800)).front;
win.background_(Color.black);
uvs = k_.collect({
arg i;
var width = win.bounds.width / k_;
var uv = UserView(win,Rect(width * i,0,width,win.bounds.height));
uv.background_(colors[i].alpha_(0));
//uv.visible_(false);
});
},AppClock).play;
});
)

// **************** Cmd. crashes server? *********************


I find that when I Cmd-Period this (just to change something and re-run it) it crashes the server sometimes… I can try to isolate it more, just wondering if that’s a known thing the the FluidDataSet tools.

Also, when this runs I get this rolling in the post window:

I don’t know what to make of it. The code seems to be running great though.

Thank you!

All and any crash reports welcome. If you open console.app and go to User Reports, you should see a delightful litany of crash reports, clearly labelled by application.

I’ve seen that sort of error message in the SC console before when trying to do things like plot changed buffers before their information has been updated language-side…

1 Like

This is sooooo cool!

A few ideas to try, and a few comments:

Comment#1: I have so much to learn in SC in there I’ll translate it to Max to understand each line. Thanks so much for sharing

Comment#2: the tmp file error is a SC one, as it seems that 30 buffer to file to language writing per second is not making it happy at line 112 - we have seen that in the past but I wonder if you have used the same approach instead of setn then sync in the past ? I presume we could do a similar test with vanilla objects only and see if we get errors… and if so we can send a report to the (amazing) SC dev team.

idea#1: trying with MFCC1-12 (dismissing the first coefficient MFCC0 as it is mostly loudness)

idea#2: trying the JiT approach of doing a non-real-time buffered audio that is then averaged for the full updatePeriod with a circular buffer of the totalHistory.

I’m sure @groma and @weefuzzy will have other ideas to share here.

1 Like

so here we go, a Vanilla example that creates the same type of errors:

b = Buffer.alloc(s,13);

(
Routine{
inf.do({
60.reciprocal.wait;
})}.play;
)


but fear not, here is the same task that is faster (because we do not move much)

(
Routine{
inf.do({
b.setn(0,Array.rand(13,-1.0,1.0));
s.sync;
"done".postln;
60.reciprocal.wait;
})}.play;
)


I was able to crank it to 500 updates per seconds without errors, but I don’t know if it updates them all… so I did that instead.

(
a = 0;
Routine{
inf.do({
b.setn(0,a.dup(13););
s.sync;
// a.postln;
a = a +1;
500.reciprocal.wait;
})}.play;
)


and I definitely do not reach 500 a second I must saturate the sync mechanism… so I removed it and it is quite fun - it is crazy fast! but I wonder what are the dependencies there that are missing… I know any buffer action is sequential so if you send it the setn message then one of ours it will have to wait to do it - @weefuzzy will have a better understanding here of the dependencies, but I’ll try your code now with and without the sync

1 Like

Yes, I considered this and understand that it’s how this is usually done–is there value in leaving in MFCC0 (as it is mostly loudness)? For example if the classification one is trying to do would benefit from loudness information? Or would it be better to use a loudness measurement for that? Perhaps this is a question for @weefuzzy?

Ah yes–great suggestion! Thank you.

1 Like

I tend to use a proper loudness descriptor with some sort of ear-emulation algo behind, like ours which uses the ITU curves. Having spectrum independent of loudness give you the possibility to weigh it independently… for instance, if you are looking for classes of sound, loudness should not be considered (an ‘s’ is an ‘s’, and a ‘f’ is an ‘f’)

Hey @tedmoore did you explore a bit more along those lines? I’m about to port this to Max with some of the ideas I suggested above, but maybe you have done further experimentations that can enlighten me even more ?

I haven’t done more with this, but I did put it on my covid to do list a few days ago! I’ll play with it over the next few days and see if anything more comes about.

1 Like

Greetings,

I revisited this to take a look at PA’s main suggestion, which was to do the analysis JIT style by filling a buffer and then analyzing it all at once. It ended up not working out. I ran into a couple bottlenecks / design considerations.

1. The first is that it’s actually quite elegant to have the analysis vector created on the server in one place, and then used both to populate the dataset, but also to do the classification. If one wants to create the analysis vectors for the dataset from the buffer, it would require hard coding in both places what parameters are in the vector. Also, if one wanted to use multiple Fluid analyses for their database vector (FluidMFCC & FluidSpectralShape for example), one would have to “analyze” the buffer twice, once with FluidBufMFCC and again with FluidBufSpectralShape, and then concatenate those vectors. It seems simpler to keep the analysis vector creation on the server.
2. Is there a way to initialize the number of entries in a FluidDataSet? For example, I want a new FluidDataSet on server “s”, named \teds_data, with dimensionality 13, and 600 entries: FluidDataSet(s,\teds_data,13,600). The reason I ask is that I find myself doing this thing:
if(i_have_the_num_entries_i_need,{
dataset.updatePoint(i.asString,vector);
},{
})


This way, I’m overwriting data points (filling it up like a circle buffer), but I can only do the overwriting if it’s the proper size (proper number of entries). If I could initialize the size, then I could just use updatePoint.

I realize I could do this manually at the outset right after creation:

n_points_i_want.do({
arg i;
});


Just wondering if it would be an easy feature to add (also might be useful to know for memory allocation?).

1. The next issue I encountered was that once I did the FluidBufMFCC and got the features buffer back (which is very quick), it then took quite a long time to get that data into a FluidDataSet so that I could then use FluidKMeans. The process of taking the frame out of the FluidBufMFCC features buffer and getting it into it’s own buffer so that I could use that to update it in the dataset, is too slow to be useful (to use in real-time applications). So I think for this reason and for #1 above, the best approach is the initial one–to add (or overwrite) vectors in the database as the come off the server with SendReply–and to also use that SendReply as the source of the vectors for the classification.
2. Lastly, my FluidKMeans seems to be crashing the server on .fit, but not every time. I haven’t isolated it more, but I’m posting the code here and the crash report.
// if you highlight all and cmd-return, it should run...
(
s.waitForBoot({
var updatePeriod = 5 /*in seconds*/; // how often to recompute the k means

// how long into the past should we keep track of for clustering into k means
var totalHistory = 10 /*in seconds*/;
var circBuf = Buffer.alloc(s,totalHistory * s.sampleRate);

var bus = Bus.audio(s); // for sending audio to the analysis synth
var synth; // plays a buffer
var analysis_synth; // does analysis

var trigRate = 30; // how many times to update every second

// audio input - put your sound file here (or route different incoming audio to the analysis synth)
var audioBuf = Buffer.read(s,"/Users/ted/Documents/_CREATING/_PROJECT FILES/Machine Learning/Training Data/Audio/source audio/ml examples/elec/stereos/no_input_mixer 01.wav");
//var audioBuf = Buffer.read(s,"/Users/Ted/Music/ml test files/saxophone 01.wav");

// keep track of how many frames have been processed so we know when we have enough data to run the k means
var frameCounter = 0;

var nb_of_dims = 12; // just the 12 mfccs for now...
var dataset = FluidDataSet(s,\raw_data,nb_of_dims);
var normed_dataset = FluidDataSet(s,\normed_dataset,nb_of_dims);
var scaler = FluidNormalize(s);
//var ranges = ControlSpec(inf,-inf).dup(nb_of_dims);
var assimilate_point; // a function that normalizes the incoming points with historical extremes

var kmeans = FluidKMeans(s);
var k_ = 4; // what the k will be for k means
var get_cluster; // a function that will take in a buffer (vector) update the view with  it's color

var nHistories = trigRate * totalHistory; // total number of histories to keep track of

var updateBuf = Buffer.alloc(s,nb_of_dims); // a buffer to use for sending new points to the FluidDataSet
var updateBuf2 = Buffer.alloc(s,nb_of_dims);
var normedBuf = Buffer.alloc(s,nb_of_dims);
var features = Buffer(s);
var clustered = false; // is true once at least one clustering has happened (at one point one can predict on new points)
var buffer_filled = false; // is true once we have saved 'nHistories' vectors to the FluidDataSet
var uvs; // userviews that will flash showing current cluster prediction
var win; // window where the colors flash

// the colors that flash (4 for now because 'k_' = 4, would need more with a larger k)
var colors = [Color.cyan,Color.yellow,Color.green,Color.red];

s.sync;

// play sound
synth = {
arg rate = 0;
var sig = PlayBuf.ar(audioBuf.numChannels,audioBuf,rate,loop:1);
//var sig = SoundIn.ar(0);
Out.ar(bus,Mix(sig));
sig * -20.dbamp;
}.play;

s.sync;

// define this function that will take in a buffer (vector) update the view with  it's color
get_cluster = {
arg unnormed_buf;

scaler.normalizePoint(unnormed_buf,normedBuf,{

kmeans.predictPoint(normedBuf,{
arg cluster;

/*				cluster.postln;
"".postln;*/

uvs.do({
arg uv, i;
if(cluster == i,{
defer{
//uv.visible_(true)

// using alpha here because it might be nice to set the alpha to the
// amplitude or something so it flickers with the sound...
uv.background_(uv.background.alpha_(1));
};
},{
defer{
//uv.visible_(false);

// alpha = 0 means don't show me this.
uv.background_(uv.background.alpha_(0));
};
})
});
});
});
};

// synth that does the analysis and sends the vector back to the language
analysis_synth = {
arg in_bus;
var input = In.ar(in_bus);

// ------------------ original approach ---------------------
// --------- (still need this to send real time vectors to lang) -----------------

// put in whatever kind of analysis you want
var vector = FluidMFCC.kr(input); // 13 by default
vector = vector[1..12];
//KMeansRT.kr(kmeansrtBuf,vector,k_,1,0,1).poll;

// ---------- jit circle buf approach as suggested by pa -------------

RecordBuf.ar(input,circBuf);
Silence.ar;

// the vectors are sent from the server to here:
OSCFunc({
arg msg;
var vec = msg[3..14]; // take only the 12 mfccs for now
//vec.postln;

// ----------------------- original approach --------------------------
/*			// load the vector to the server
//arg buf;

//"buffer to update: %".format(updateBuf).postln;
if(buffer_filled,{ // if we have already put 'nHistories' points in the FluidDataSet, then...

// update it
dataset.updatePoint(frameCounter.asString,updateBuf,{

// only if at least one prediction has happened, make a prediction!
if(clustered,{
get_cluster.(updateBuf);
});
});
},{

// if not, then, add it

// only if at least one prediction has happened, make a prediction!
if(clustered,{
get_cluster.(updateBuf);
});
});
});

// modulo the frameCounter
frameCounter = (frameCounter + 1) % nHistories;

// if we've adding 'nHistories' points, set the flag
if(frameCounter == (nHistories - 1),{buffer_filled = true;});
});*/

// 	if(clustered,{
// 		get_cluster.(updateBuf2);
// 	});
// });

if(clustered,{
Routine({
updateBuf2.setn(0,vec);
s.sync;
get_cluster.(updateBuf2);
}).play;
});
},"/analysis_vector");

// kind of cheeky, but the synth doesnt play the sound until everything is in place...
synth.set(\rate,1);

// for all of time, fit the k means to the dataset, but wait 'updatePeriod' in between (default is 10 sec)

// wait for the whole circle buffer to fill up!
totalHistory.wait;

inf.do({
arg i;
i.postln;
// ------------- original approach
/*				scaler.fit(dataset.asString,{
scaler.normalize(dataset.asString,normed_dataset.asString,{

kmeans.fit(normed_dataset,k_,500,action:{
//"clustering done".postln;
clustered = true;
});
});
});*/

// get the mfcc features
FluidBufMFCC.process(s,circBuf,features:features,action:{
features.postln;

// load the features into the dataset;
features.numFrames.do({
arg frame_i;
"frame i: %".format(frame_i).postln;

//features.copyData(updateBuf,(frame_i * features.numChannels) + 1,0,nb_of_dims);

arg float_array;
if(i == 0,{ // if this is the first time, add a point, otherwise, update
},{
dataset.updatePoint(frame_i.asString,updateBuf);
});
});*/
updateBuf.setn(0,float_array);
s.sync;
if(i == 0,{ // if this is the first time, add a point, otherwise, update
},{
dataset.updatePoint(frame_i.asString,updateBuf);
});
});
});

// normalize the dataset
scaler.fit(dataset.asString,{
scaler.normalize(dataset.asString,normed_dataset.asString,{
kmeans.fit(normed_dataset,k_,500,action:{
"clustering done".postln;
clustered = true;
});
});
});
});

updatePeriod.wait;
});
},AppClock).play;

// window for showing the current cluster prediction
Window.closeAll;
win = Window("",Rect(Window.screenBounds.width + 100,100,600,800)).front;
win.background_(Color.black);
uvs = k_.collect({
arg i;
var width = win.bounds.width / k_;
var uv = UserView(win,Rect(width * i,0,width,win.bounds.height));
uv.background_(colors[i].alpha_(0));
//uv.visible_(false);
});
},AppClock).play;
});
)


Process: coreaudiod [236]
Path: /usr/sbin/coreaudiod
Identifier: coreaudiod
Version: 5.0 (5.0)
Code Type: X86-64 (Native)
Parent Process: launchd [1]
Responsible: coreaudiod [236]
User ID: 202

PlugIn Path: /Library/Audio/Plug-Ins/HAL/BlackHole.driver/Contents/MacOS/BlackHole
PlugIn Identifier: audio.existential.BlackHole
PlugIn Version: 0.2.6 (1)

Date/Time: 2020-04-05 15:04:52.800 -0400
OS Version: Mac OS X 10.15.4 (19E266)
Report Version: 12
Bridge OS Version: 3.0 (14Y908)
Anonymous UUID: 725D907C-009E-FB2F-BA7A-3FC2D83702A3

Sleep/Wake UUID: 338BDEC6-7350-46C9-978F-08FF761DEBBE

Time Awake Since Boot: 130000 seconds
Time Since Wake: 52000 seconds

System Integrity Protection: enabled

Crashed Thread: 7 audio IO: BlackHole_UID

Exception Codes: KERN_PROTECTION_FAILURE at 0x00000001056fc000
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [236]

VM Regions Near 0x1056fc000:
MALLOC_LARGE 000000010567c000-00000001056fc000 [ 512K] rw-/rwx SM=PRV
–> VM_ALLOCATE 00000001056fc000-00000001056fd000 [ 4K] r–/rwx SM=PRV
VM_ALLOCATE 00000001056fd000-00000001057fd000 [ 1024K] rw-/rwx SM=PRV

Application Specific Information:
dyld3 mode

0 libsystem_kernel.dylib 0x00007fff6a2cedfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff6a2cf170 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff302460b5 __CFRunLoopServiceMachPort + 247
3 com.apple.CoreFoundation 0x00007fff30244b82 __CFRunLoopRun + 1319
4 com.apple.CoreFoundation 0x00007fff30243ffe CFRunLoopRunSpecific + 462
5 coreaudiod 0x0000000101c14c29 main + 5308
6 libdyld.dylib 0x00007fff6a18dcc9 start + 1

0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.caulk 0x00007fff63d67b16 caulk::mach::semaphore::wait() + 16
2 com.apple.audio.caulk 0x00007fff63d679b2 caulk::semaphore::timed_wait(double) + 106
3 com.apple.audio.caulk 0x00007fff63d677c4 caulk::concurrent::details::worker_thread::run() + 30

0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.caulk 0x00007fff63d67b16 caulk::mach::semaphore::wait() + 16
2 com.apple.audio.caulk 0x00007fff63d679b2 caulk::semaphore::timed_wait(double) + 106
3 com.apple.audio.caulk 0x00007fff63d677c4 caulk::concurrent::details::worker_thread::run() + 30

0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.CoreAudio 0x00007fff2f77908a ca::mach::semaphore::wait() + 16
2 com.apple.audio.CoreAudio 0x00007fff2fa688ce ca::concurrent::details::worker_thread::run(ca::attributes) + 390

0 libsystem_kernel.dylib 0x00007fff6a2cee36 semaphore_wait_trap + 10
1 com.apple.audio.CoreAudio 0x00007fff2f77908a ca::mach::semaphore::wait() + 16
2 com.apple.audio.CoreAudio 0x00007fff2fa688ce ca::concurrent::details::worker_thread::run(ca::attributes) + 390

0 libsystem_kernel.dylib 0x00007fff6a2d70fe __select + 10
1 com.zerodebug.audiomux 0x0000000101f8bffd LocalClient::Work() + 1581
2 com.zerodebug.audiomux 0x0000000101f95dae client_thread_proc(void*) + 14

Thread 7 Crashed:: audio IO: BlackHole_UID
0 libsystem_platform.dylib 0x00007fff6a383dc9 _platform_bzero$VARIANT$Haswell + 41
1 audio.existential.BlackHole 0x0000000101ed2f47 BlackHole_DoIOOperation + 759
2 com.apple.audio.CoreAudio 0x00007fff2fb1182f HALS_PlugInEngine::_ReadFromStream_Read(HALS_IOEngine2_IOContextInfo&, HALS_IOEngine2_StreamInfo&, AudioServerPlugInIOCycleInfo const&, unsigned int, void*) + 91
3 com.apple.audio.CoreAudio 0x00007fff2fa1dbbf HALS_IOEngine2::_ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 869
4 com.apple.audio.CoreAudio 0x00007fff2fb11740 HALS_PlugInEngine::_ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 54
5 com.apple.audio.CoreAudio 0x00007fff2fa21599 invocation function for block in HALS_IOEngine2::ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 678
6 com.apple.audio.CoreAudio 0x00007fff2fa6d6a4 HALB_CommandGate::ExecuteCommand(void () block_pointer) const + 98
7 com.apple.audio.CoreAudio 0x00007fff2fa1b316 HALS_IOEngine2::ReadFromStream(HALS_IOContext*, unsigned int, unsigned int, HALS_IOEngineInfo const&, unsigned int, void*) + 138
8 com.apple.audio.CoreAudio 0x00007fff2f9f51b6 HALS_IOContext::PerformIO_ReadFromStream(void*, unsigned int, HALS_IOStreamInfo&, unsigned int) + 86
9 com.apple.audio.CoreAudio 0x00007fff2f9ef268 std::__1::function<int (void*, unsigned int, HALS_IOStreamInfo&, unsigned int)>::operator()(void*, unsigned int, HALS_IOStreamInfo&, unsigned int) const + 42
10 com.apple.audio.CoreAudio 0x00007fff2f9f55ab std::__1::function<int (unsigned int, HALS_IOStreamInfo&, unsigned int)>::operator()(unsigned int, HALS_IOStreamInfo&, unsigned int) const + 37
11 com.apple.audio.CoreAudio 0x00007fff2f9f3d2c HALS_IOContext::PerformIO(AudioTimeStamp const&, unsigned int, int, unsigned int) + 968
12 com.apple.audio.CoreAudio 0x00007fff2f9f29b0 HALS_IOContext::IOThreadEntry(void*) + 9536
13 com.apple.audio.CoreAudio 0x00007fff2f9f03bf invocation function for block in HALS_IOContext::HALS_IOContext(HALS_System*, HALS_Client*, HALS_System::PowerState) + 13
14 com.apple.audio.CoreAudio 0x00007fff2f95ec22 HALB_IOThread::Entry(void*) + 72

rax: 0x0000000000000000 rbx: 0x0000700001804540 rcx: 0x0000000000050900 rdx: 0x000000010567c000
rdi: 0x00000001056fc000 rsi: 0x0000000000000000 rbp: 0x0000700001804410 rsp: 0x0000700001804410
r8: 0x0000000072656164 r9: 0x00000000000035ec r10: 0x00000001058fd000 r11: 0x0000000000287600
r12: 0x00007fbbbbb1eb30 r13: 0x00007fbbbbb13200 r14: 0x00007a3c59b29cd8 r15: 0x00007fbbbbb13cd0
rip: 0x00007fff6a383dc9 rfl: 0x0000000000010206 cr2: 0x00000001056fc000

Logical CPU: 2
Error Code: 0x00000007 (invalid protections for user data write)
Trap Number: 14

I’m back from a few days of unsuccessful resting, so my brains are slow, but I’ll check this in the next few hours and come back to you with thoughts.

Hello again! Caffeine has kicked in, so I am now able to reply to some questions, but I will have to discuss most with @weefuzzy and @groma as we are working on such improvments on the interface and manipulation of data, and in particular in SC.

Your point 1 I’m not sure I get. The analysis vector can be kept on the server, you can keep it in a buffer there, no? I might not get your proposal if that is not what you meant. We are also toying with copying vectors between datasets, and therefore you could then make a dataset of descriptor vectors, and play with them as you want.

Point 2, not a the moment, I’ll forward to the team.

Point 3, @groma will have a comment since we were talking about that very workflow and I think he had a solution.

Point 4 is more complicated to trace, but @weefuzzy is it possible it is the threading exception you found?
@tedmoore I get ‘Silence.ar Class not defined’ but I reckon it is Silent.ar(). I don’t manage to crash your code in either version (the one you have and the tip of the dev compile) but there is a slight difference of behaviour.

I get, after a certain time, in the post window:

0
Synth(‘temp__18’ : 1017)
Buffer(17, 862, 13, 86.1328125, nil)
frame i: 0
frame i: 1
and frame goes up forever

The only difference I get between current tip and the version you have is that I managed to crash the server when I stop the process in the old version (yours) once. I did not manage to crash it in my version…

I am aware that this answer is opening more questions… but this is super useful, thanks for posting, and more soon!

Hello, regarding issue 3, I think @tremblap is referring to this example, which simplifies the flattening of interleaved buffer obtained via FluidBufStats
~dest = Buffer.alloc(s, ~statsBuf.numChannels * ~statsBuf.numFrames);
~statsBuf.getToFloatArray(0, action:{|a| ~dest.setn(0, a.unlace(~statsBuf.numChannels).flatten)});

I think here the problem is different, although I think clearly you are making an extra trip with the SendReply, instead you could store vector in a buffer in the server when it comes out of FluidMFCC. I have not tried, but maybe an array of RecordBuf can be used to write each channel of the output into a position in the buffer, or something like that, then in the language load that buffer in the dataset.
I agree it is not optimal and we need a more direct way to communicate to FluidDataset server side.

1 Like

A note on the crash. It looks like a memory stamp. I’ve only been able to get it to happen once so far, and not when I had the debugger attached

Unfortunately this was against our dev branch, so my initial hope that this might have been something I’d already fixed looks to be folorn. It won’t be the threading thing @tremblap mentioned, but it could still be another threading thing as my crash was when freeing the synths, and this part of the wrapper is currently the Most Problematic.

Ah yes, I think what I was imagining was a circle buffer of audio, which would then be analyzed with a (or multiple) FluidBuf- analyses from the language. But now I see what would be better is to do all the analysis in real-time on the server (no FluidBuf- analyses at all), and pack that into a vector there, then write it to a circle buffer of descriptors––not a circle buffer of audio. However I think the problem of loading into the FluidDataSet remains. This relates to @groma’s reply below, so I’ll detail it there.

Yeah for sure. I realize that what would be better (as I mentioned in my reply to @tremblap) would be to do all the analysis on the same synth def and then write the analysis vector some kind of circle buffer rather than store it in an audio circle buffer to be analyzed later. That way my analysis is done and already on the server in a known place.

Is there (or can there be) a way to convert a circle buffer of analysis vectors (nChannels in the buffer = the dimensionality of the data, nFrames = number of data points) directly into a FluidDataSet or just overwrite an existing FluidDataSet with whatever is in that buffer? Or better yet something like

FluidDataSet.kr(analysis_vector)


right on the server that will just fill it up (and circle around, continuously filling it up like a circle buffer)? That way whenever I want to run a FluidKMeans or whatever, the data is already there!

Otherwise, my circle buffer of analysis vectors would still need to be sliced up into individual frames and added to a FluidDataSet with .addPoint(copied_buffer) right?

I think maybe the best solution right now is to be constantly writing my analysis vector to a 1 frame buffer (with nChannels = dimensionality of the vector) on the server, and then in the language, in a loop, running FluidDataSet.updatePoint(buffer) on the same buffer over and over (since the data in it will be different every time). Does that sound like a reasonable approach to this circle buffer of analysis data idea?

I’ll try to implement this soon and see how it runs.

Thanks all!

not as elegantly as you propose yet, but you can overwrite items, yes, as you propose

sounds like an expensive one I think, but since you cannot copy vectors out of datasets yet (we are working on solutions on this as we are also users and have what seems to be the same stress points) then you can try this way and see what happens.

If we were able to run stats on vectors, then that would solve this even better I think - you could write your dataset of descriptors, then make stats to another dataset and use the latter to do your classification…

it turns out RecordBuf won’t work because then the buffer would be 1 frame x nDimensions channels, but to use FluidDataSet.addPoint it has to be 1 channel x nDimensions frames. However this will work in a synth for a buffer of the latter shape:

vector.do({
arg val, i;
BufWr.kr(val,my_buffer,i);
});


Perhaps FluidDataSet.addPoint could be modified to take a buffer of either shape (since the “size” of them are the same)?

I completed an edit of this with the design suggestions. Basically avoiding sending vectors to the language and back to the server. The result and performance are about the same. It would still be great to add the features desired above, but feel free to check out this if it’s useful.

There is one question that came out of this. I was sometimes getting this error: “Error: Already processing”. I think it had to do with creating a FluidDataSet, maybe one whose name was already used on the server? Is there a way to create a FluidDataSet and avoid this?

// if you highlight all and cmd-return, it should run...
(
s.options.device_("Scarlett 6i6 USB");
s.waitForBoot({
var updatePeriod = 5 /*in seconds*/; // how often to recompute the k means
var totalHistory = 10 /*in seconds*/; // how long into the past should we keep track of for clustering into k means
var bus = Bus.audio(s); // for sending audio to the analysis synth
var synth; // plays a buffer
var analysis_synth; // does analysis
var trigRate = 30; // how many times to update every second
var audioBuf = Buffer.read(s,"/Users/ted/Documents/_CREATING/_PROJECT FILES/Machine Learning/Training Data/Audio/source audio/ml examples/elec/stereos/no_input_mixer 01.wav");

var nb_of_dims = 12; // just the 12 mfccs for now...
var dataset;// = FluidDataSet(s,\raw_data,nb_of_dims);
var normed_dataset;// = FluidDataSet(s,\normed_dataset,nb_of_dims);
var scaler = FluidNormalize(s);

var kmeans = FluidKMeans(s);
var k_ = 4; // what the k will be for k means
var get_cluster; // a function that will take in a buffer (vector) update the view with  it's color

var nHistories = trigRate * totalHistory; // total number of histories to keep track of

var updateBuf = Buffer.alloc(s,nb_of_dims); // a buffer to use for sending new points to the FluidDataSet
var normedBuf = Buffer.alloc(s,nb_of_dims); // a buffer to put a normalized datapoint in
var clustered = false; // is true once at least one clustering has happened (at one point one can predict on new points)
var buffer_filled = false;
var uvs; // userviews that will flash showing current cluster prediction
var win; // window where the colors flash
var serverBuf = Buffer.alloc(s,nb_of_dims);
var dummyBuf = Buffer.alloc(s,nb_of_dims);
var update_counter = 0;

// the colors that flash (4 for now because 'k_' = 4, would need more with a larger k)
var colors = [Color.cyan,Color.yellow,Color.green,Color.red];

s.sync;

dataset = FluidDataSet(s,\raw_data,nb_of_dims);
normed_dataset = FluidDataSet(s,\normed_dataset,nb_of_dims);

s.sync;

/*
because the datasets are "named" on the server, does it make sense that i should probably clear them each time,
just to be sure that there isn't any residual data lingering in these "names"
*/
dataset.clear;
normed_dataset.clear;

dummyBuf.setn(0,0.dup(nb_of_dims));

s.sync;

// initialize the size of the dataset
/*		nHistories.do({
arg i;
i.postln;
s.sync;
/*

i decided against this dataset size initialization strategy, since it took a little bit of time to
complete and as i was iterating on this code, i decided it was too long...

*/
});*/

s.sync;

// play sound
synth = {
arg rate = 1;
var sig = PlayBuf.ar(audioBuf.numChannels,audioBuf,rate,loop:1);
//var sig = SoundIn.ar(0);
Out.ar(bus,Mix(sig));
sig;// * -20.dbamp;
}.play;

s.sync;

// define this function that will take in a buffer (vector) update the view with  it's color
get_cluster = {
arg unnormed_buf; // take in the raw vector that is on the server

//normalize it into the normedBuf
scaler.normalizePoint(unnormed_buf,normedBuf,{

// use that to predict
kmeans.predictPoint(normedBuf,{
arg cluster;

// update views.
uvs.do({
arg uv, i;
if(cluster == i,{
defer{
// using alpha here because it might be nice to set the alpha to the
// amplitude or something so it flickers with the sound...
uv.background_(uv.background.alpha_(1));
};
},{
defer{
// alpha = 0 means don't show me this.
uv.background_(uv.background.alpha_(0));
};
})
});
});
});
};

// synth that does the analysis
analysis_synth = {
arg in_bus;
var input = In.ar(in_bus);
var vector = FluidMFCC.kr(input); // 13 by default
vector = vector[1..12];

// a "clock" to send to the language... explained below at OSCFunc...
//vector.poll;

// i wish...
//RecordBuf.kr(vector,serverBuf);

vector.do({
arg val, i;
BufWr.kr(val,serverBuf,i);
/*

has to work this way because FluidDataSet.updatePoint needs to have a 1 channel buf with
nFrames = num dimensions

the RecordBuf.kr above would be simpler, however then the buffer would need to be a 1 frame
buf with nChannels = num dimensions

maybe ".updatePoint" could be modified to take a buffer that is either *1 channel x nDimensions frames* OR
*nDimensions channels x 1 frame* ?

*/
});

Silent.ar;

OSCFunc({
var label = (update_counter % nHistories).asString;

if(update_counter % trigRate == 0,{
"currently updating data point: %".format(label).postln;
});

if(buffer_filled,{
dataset.updatePoint(label,serverBuf);

if(clustered,{
get_cluster.(serverBuf);
});
},{
if(update_counter == (nHistories - 1),{buffer_filled = true;});
});

update_counter = update_counter + 1;
/*

i originally had this update functionality (i.e., the "updatePoint" and the "get_cluster") happening just in a loop in a routine with
"trigRate.reciprocal.wait" in the loop, but it turn sout that "updatePoint" takes some time, so
each iteration of the loop was much longer than trigRate.reciprocal.

so now i'm using an Impulse.kr on the server to "clock" my "addPoint" and "get_cluster" through an OSCFunc

this is better though because then i'm not sending the vector back to the language, loading it back to the server in
*another* buffer, to finally add to the dataset

(for what it's worth, this way of clocking something in the language with an OSCFunc is something i find myself
doing often... it works quite well, however does seem a little weird. does anyone know of a better way to be

*/
},"/clock");

Routine({

// wait for the whole buffer to fill up!
totalHistory.wait;
/*

infinitely loop this: do the things to get the kmeans...

*/
inf.do({
arg i;
"clustering for the % th time".format(i).postln;

scaler.fit(dataset.asString,{
"fitting to the scaler complete".postln;
scaler.normalize(dataset.asString,normed_dataset.asString,{
"normalizing the dataset complete".postln;
kmeans.fit(normed_dataset,k_,action:{
"fitting kmeans complete".postln;
clustered = true; // this way after this has happened at least once, we'll know we can do "predict" above
});
});
});

updatePeriod.wait; // only update this ever so often...
});
}).play;

// window for showing the current cluster prediction
Window.closeAll;
win = Window("",Rect(Window.screenBounds.width + 100,100,600,800)).front;
win.background_(Color.black);
uvs = k_.collect({
arg i;
var width = win.bounds.width / k_;
var uv = UserView(win,Rect(width * i,0,width,win.bounds.height));
uv.background_(colors[i].alpha_(0));
//uv.visible_(false);
});
},AppClock).play;
});
)


Hello!

Thanks for this. We’re in a tight deadline for a paper now, but I’m sure help will be on its way soon on this too, stay tuned!

1 Like

Hmm. In my imagination, you shouldn’t ever see this message in relation to FluidDataSet. It’s to do with making repeated calls to a threaded non-realtime object it’s finished its work. Creating language-side instances with the same name gives you a representation of the same object on the server, so I guess its possible that requests end up overlapping this way, except that all the methods for FluidDataSet are meant to be synchronous (w.r.t the server command FIFO), because the object itself is not thread-safe. I will investigate.

Re speed. Without ignoring the need to come up with a server-side workflow, I think the back and forth approach you’re currently stuck with isn’t helped by the amount of syncing that goes on. I’m now wandering about the wisdom of baking a sync into the the back-end. If you fancy a dangerours experiment, try commenting out the sync on line 29 in FluidManipulationClient:

                    result = FluidMessageResponse.collectArgs(parser,msg.drop(3));
this.server.sync;
c.test = true;
c.signal;


In theory this should be safe , but if I’m wrong about these functions executing synchrnously w/r/t the server command thread, strange things might happen.

Finally, about intialization: the challenge is that datasets are a mapping between keys and data, so an ‘empty’ point still needs a unique label associated with it.

1 Like