Large Corpus Analysis in SuperCollider

Usage Questions
1

Hi all -

I’d like to work with a body of files totaling about 200GB, but ultimately analyzing in a similar way to that discussed in the 2D Corpus Explorer demo.
My understanding is that each buffer is loaded to the server in this approach, which seems like it would be prohibitive with the size of the database I’m working with.
I guess what I’m wondering is whether there is an “offline” process whereby SuperCollider could parse and analyze a large set of files, create a dataset - and then use the dataset as a “score” to output all of the “points” as discrete wave files. Or maybe there’s a better way for dealing with files on this scale?
Are there any demos or tutorials for how to build proper scaffolding for larger projects in FluCoMa/SC?

Thanks as always!

2

I guess one simpler question would be:

When plotting with MFCC and umap, I’m visualizing a lot of somewhat abstract analysis information that I don’t necessarily understand - other than what I’m able to glean intuitively from working with a plotter. Arguably, if I were to dump the results into single buffers, I could use the X/Y coordinates as file names - but I wonder if there would be a way to title the files with concrete information about the MFCC make-up in each. I’ve pasted the code below, but commented the area that I imagine this taking place in..


~folder = FluidFilesPath();
~loader = FluidLoadFolder(~folder).play(s,{"done loading folder".postln;});

// sum to mono
(
if(~loader.buffer.numChannels > 1){
~src = Buffer(s);
~loader.buffer.numChannels.do{
arg chan_i;
FluidBufCompose.processBlocking(s,
~loader.buffer,
startChan:chan_i,
numChans:1,
gain:~loader.buffer.numChannels.reciprocal,
destination:~src,
destGain:1,
action:{"copied channel: %".format(chan_i).postln}
);
};
}{
"loader buffer is already mono".postln;
~src = ~loader.buffer;
};
)

// slice the buffer in non real-time
(
~indices = Buffer(s);
FluidBufOnsetSlice.processBlocking(s,~src,metric:9,threshold:0.05,indices:~indices,action:{
"found % slice points".format(~indices.numFrames).postln;
"average duration per slice: %".format(~src.duration / (~indices.numFrames+1)).postln;
});
)

// analysis
(
~analyses = FluidDataSet(s);
~indices.loadToFloatArray(action:{
arg fa;
var mfccs = Buffer(s);
var stats = Buffer(s);
var flat = Buffer(s);

fa.doAdjacentPairs{
	arg start, end, i;
	var num = end - start;

	FluidBufMFCC.processBlocking(s,~src,start,num,features:mfccs,numCoeffs:13,startCoeff:1);
	FluidBufStats.processBlocking(s,mfccs,stats:stats,select:[\mean]);
	FluidBufFlatten.processBlocking(s,stats,destination:flat);

	~analyses.addPoint(i,flat);

	"analyzing slice % / %".format(i+1,fa.size-1).postln;

	if((i%100) == 99){s.sync;}
};

s.sync;

});
)

(
~umapped = FluidDataSet(s);
FluidUMAP(s,numNeighbours:15,minDist:0.9).fitTransform(~analyses,~umapped,action:{"umap done".postln});
)

~umapped.print;

// normalize
(
~normed = FluidDataSet(s);
FluidNormalize(s).fitTransform(~umapped,~normed);
)

///i would think this would be the place to write audio files, though i'm not exactly sure how it would work.

~normed.dump{|dict|

FluidBufCompose.processBlocking(s, mainBuf,
startFrame,
numFrames,
destination:saveBuffer,
destStartFrame:startFrame,
action:{
saveBuffer.write(
"/Users/a/Desktop/untitled folder/"++i++".aif");   });
};

// fit a kdtree
~tree = FluidKDTree(s).fit(~normed);

// a function to play back and individual slice point
(
~play_slice = {
arg index;
{
var startsamp = Index.kr(~indices,index);
var stopsamp = Index.kr(~indices,index+1);
var phs = Phasor.ar(0,BufRateScale.ir(~src),startsamp,stopsamp);
var sig = BufRd.ar(1,~src,phs);
var dursecs = (stopsamp - startsamp) / BufSampleRate.ir(~src);
var env;

	dursecs = min(dursecs,1);

	env = EnvGen.kr(Env([0,1,1,0],[0.03,dursecs-0.06,0.03]),doneAction:2);
	sig.dup * env;
}.play;

};
)

// plot
(
~normed.dump({
arg dict;
var point = Buffer.alloc(s,2);
var previous = nil;
dict.postln;
defer{
FluidPlotter(dict:dict,mouseMoveAction:{
arg view, x, y;
[x,y].postln;
point.setn(0,[x,y]);
~tree.kNearest(point,1,{
arg nearest;
if(nearest != previous){
nearest.postln;
view.highlight_(nearest);
~play_slice.(nearest.asInteger);
previous = nearest;
}
});
});
}
});
)
3

Not a SC person so take my view with a grain of salt. For anything where I have a lot of files and it becomes tedious to work with due to analysis times, scale of data, etc. I start to think carefully about keeping as much on disk as possible and making lightweight mappings between data and audio files. I generally have a manifest.json or something to that effect which keeps everything linked, and means I can load things as I need to into RAM.

So not really a direct solution to your problem, but perhaps you can think about the manner in which you store data and retrieve it, and try to keep things as lightweight as possible in RAM. SSDs are fast enough now that even calling it off your hard drive when you need it is fine.

I may have also totally misread your question, in which case I apologise for the rant.

1 Like
4

Thanks James- I think keeping the files on the hard drive is definitely the way to go - but one challenge, at the moment, is figuring out how to organize/name them so they reflect something of their analyses.

I’m also still wondering if there’s a convenient way to segment the files by their MFCC coherency, rather than amplitudes/onsets..

5

Pretty difficult. If you wanted to get really in the weeds you could shove some metadata into a json / wav file via a special key or as a RIFF chunk. This is what REAPER does for some things.

6

Hello

So this project is quite dependant on 2 things for me to help you:

  1. are those sounds all small bits? will you reduce each slice as one like single-shot drums?
  2. how do you plan on playing them back?

The analysis part is going to be the same: load files maybe 8 at a time, keep track of where they are (file and index in file) and make entries. delete the files for now rince and repeat. I have dirty code that does that.

Then the playback - you can do what commercial samplers do, and load in ram just the start of each sound for random access - although if all of the 200g of sounds are on an ssd you might as well pay from the disk directly.

let me know more and I can help a little more.

7

What I’d really like to do is probably simpler than I’m making it out to be.
Essentially, I would like to use FluCoMa as a way of organizing a bunch of files on my computer. I don’t necessarily need to use the plotter to read from the SSD in real-time. The goal would be to have longer files - not just quick transients - that were perhaps sorted into folders and named accordingly, grouped with like-sounding recordings.

8

It sounds like you’re looking to use a clustering algorithm to sort a bunch of sound files / slices into different similar clusters? And then write it all back to disk so individual directories have all “similar” sounding files / slices?

This is very possible. There is no way to “know ahead of time” what those different clusters sound like, they’d just be “cluster 0”, “cluster 1”, etc. That’s just an aspect of doing unsupervised learning. After the sorting, you can listen to the clusters and give it a descriptive name.

FluidNoveltySlice does this. Set the feature to “MFCC”. You’ll likely want to find a small subset of your sounds to test in on first to dial it in and see if its results are matching you’re perception / assumptions.

9

Yes - I think this is what I’d like to be able to do - I’m not exactly sure how to have the clusters “named”, though, as a byproduct of the FluidNoveltySlice output?

I’ll see if I can sketch up a version of this, to make it more legible.

10

OK - so, here’s a mock-up - I realize there are two questions here.

The first question has to do with slicing. The goal is to take larger samples of recordings, instead of the more granular approach. Rather than finding onsets, I was thinking slices could be deduced by how long a given sound stays within a certain range of spectral coefficients. This might be the more complicated part of the whole thing. I was also curious about how people deal with a large corpus when ideal thresholds are probably different across different recordings. I imagine these are aspects of machine listening that are more specialized - so, as an average musician, I’m a little bit at a loss for where to look for these answers.

The second issue just has to do with the naming conventions of the code itself.
Let’s say we had everything sliced up as desired and we’re parsing through the indices, outputting the slices. The issue is that I just don’t understand how to get data from the MFCC analysis and apply it to the filename/directory. I feel like this might be easier to solve.

(
~analyses = FluidDataSet(s);
~durations = Dictionary();
~indices.loadToFloatArray(action:{
arg fa;
var mfccs = Buffer(s);
var stats = Buffer(s);
var flat = Buffer(s);

fa.doAdjacentPairs{
	arg start, end, i;
	var num = end - start;
	var saveBuffer = Buffer(s);

	FluidBufMFCC.processBlocking(s,~src,start,num,features:mfccs,numCoeffs:13,startCoeff:1);
	FluidBufStats.processBlocking(s,mfccs,stats:stats,select:[\mean]);
	FluidBufFlatten.processBlocking(s,stats,destination:flat);
	

	~analyses.addPoint(i,flat);
	~durations.put(i, num);
	
	FluidBufCompose.processBlocking(s, src,
		start,
		num,
		destination:saveBuffer,
		destStartFrame:start,
		action:{
			saveBuffer.write(
		 		"/Users/a/Desktop/untitled folder/"++i++".aif");   });    ///how do i use the MFCC data as a naming convention here? 

	"analyzing slice % / %".format(i+1,fa.size-1).postln;

	if((i%100) == 99){s.sync;}
};

s.sync;

~analyses.print;

});
)
11

NoveltySlice won’t give you any insight on how one might label clusters. It too is an unsupervised process.

What you describe is essentially now NoveltySlice works. It looks for big changes in whatever analysis it uses, so it could tell you when a sound MFCC has suddenly changed by a lot.

Another thing you could do is train a classifier (MLPClassifier or KNNClassifier) to classify slices or single MFCC analyses into different categories. You’d have to create a dataset representing these different categories ahead of time to do the training, but then you could sort the rest of the data automatically. Also, this would let you semantically name the categories ahead of time (because you’re making the supervised learning dataset) so you could have the directories named in a meaningful way.

There’s no magic solution here, but generally an “adaptive” threshold could be used. For example, if you know about how many “changes” you expect NoveltySlice to find, you could write some code that could keep adjusting the threshold up or down (within a range) until you got close (enough) to that target number of slices. You can google a pseudo code algorithm to see how this is done.

MFCCs are notoriously not “human-readable”. It’s a collection of 13 numbers. So you could make the file name those 13 numbers, but looking at a file named that would be quite meaningless (also a very long filename) (also, you might have collisions where two analyses could be the same, unlikely, but shrug). I think you’ll need to find another process (some suggested above) to determine how you’d label the slices.

12

indeed, this is where you could have fun. Let’s say you want your slices to be an average of 5 sec, you could use iterative slicing like in the demo I wrote for Max (in the example folder) - I vaguely remember doing it in SC too but cannot find it now… (edit: it is in the Examples/segmenting folder of FluCoMa-SC :smiley: ) - the idea would be that you aim for (dur_in_sec / 5) slice and let the slicer loop do its job.

Then, how to encode / formalise them is again a very interesting. I also refer people to @balintlaczko’s thread - and thesis after - on what made sense for him to compare between sounds.

There are other ways to ‘supervise’ a classification, for instance by finding examples of each, but as I read you, it seems you want to ‘discover’ those classes as you go. Maybe the idea is to cluster your slices by different algorithms, and enjoy the various bundles each description gives you? I hope what i write makes sense to you.