Hi everyone,
Below is a patch that lets you explore a corpus by analysing it with audio features and mapping those features onto a two-dimensional visual map. You can scrub through the space using your mouse 
Itâs a little bit rough but I hope the patch makes sense. Start by dropping in a folder of sounds and it should automatically do something. You can then configure the parameters of UMAP and the features to experiment further.
2d_sample_browsing.maxpat (108.3 KB)
Very cool!
Some super early suggestions:
numneighbours, mindist) defaults show up as I had no clue what the values were to then adjust from@loop attribute as thereâs no way to stop playbackIt would be great to combine this with the (semi)oldschool UI for selecting what parameters/stats/etc⊠rather than looking at a single feature.
This is more of a broader question. The use case here is âdragging a folder of individual samplesâ but then the internal workflow concatenates everything and then everything after that is managed via offsets. I know @tremblap quite likes this workflow, and it does make a ton of sense if you are starting from a single long file and then segmenting as part of the process, but it seems like this would create a load of friction further downstream if you want to adjust parameters, remove individual samples, tweak overlaps, etcâŠ
More specifically, what is gained by automatically concatenating everything into a single monolithic file? (to offset the procedural confusion it causes)
Iâll try and be brief to avoid derailing the thread (we can always move is this erupts). In short, I think itâs swings and roundabouts. The obvious alternative in Max would be to use a polybuffer~. One still then needs to keep housekeeping data (buffer numbers now, rather than offsets; or as well as offsets if youâre also segmenting the individual files).
As Owen said, the takeaway is swings and roundabouts. Personally, I prefer to chuck stuff in a polybuffer~ and go from there having had my audio sliced already and the audio files themselves carrying the meaningful information about what the start/end of a segment is. This patch emerged from a workshop with absolute beginners where many of them were not familiar with buffers~ let alone polybuffers~ and I opted to show them one way that was relatively anything in segments out. I should also add the context that this isnât meant to be the âcanonical FluCoMa corpus exploration patchâ. Itâs one instance of what one might do with the tools, and a possible arrangement of objects toward a singular method of decomposition.
fascist
Again, a consequence of teaching it is to have sounds repeatedly play so that you can compare sounds and talk over them while they play. Chuck an attrui on if you wanna stop it easily.
Thatâs my woopsies and should be fixed.
Iâd be curious to see how the patch fares with some one-shot drum samples 
Yeah, thatâs pretty tangential, but I personally find the offsets really confusing and unintuitive, particularly if you start pruning out samples as part of the process.
I just wasnât sure if there was some memory optimization that meant a single chonky buffer was better than individual polybuffers or something like that.
The other bits are definitely simple/easy fixes, was just pointing out things on first play with the patch.
Also the numbers get really big which makes it hard to parse from a cursory glance IMO. Weâre on the same page here 
Yeah nah, nothing fancy like that. I could whip up a polybuffer~ example of how I would do it (with @blocking 0) if that would be helpful to show the other side too?
This has a pretty stripped back version of polyphonic playback:
Donât know how easy the grafting would be for the analysis since it expects offsets of a single buffer though.
Iâd peg it as a 4/10 on faff scale.
Alright here we go:
2d_sample_browsing-rodstyle.maxpat (99.3 KB)
polyphonic playback with mc for Rod 
as well as being able to drop in a folder of audio files to fill up a polybuffer~ for the corpus.
Nice!
Still no default params though 
And the speedlim-less-ness of the playback still leaves me with some machine gun shivers.
2d_sample_browsing-rodstyle.maxpat (100.1 KB)
This is all good banter, my friends, but letâs not forget this is a beginnerâs patch. And that you are friends (I donât think everyone knows how you two know each other 
)
The feedback is good as it will help us hone in on what the rest of this series of beginning patches will include (the 101s) until we get to design the intermediates (201s) and advanced (301s).
Looking forward to seeing the rest of the series!
p
Looks nice James.
Given that visualization is the focus here, I would definitely make the LCD larger (double it in size for instance). Some of the colors donât show up very well - especially the yellow. Might be nice to use a perceptually uniform colormap (which you could grab from somewhere like matplotlib).
Finally, I agree with Rod that this patch would greatly benefit from letting the user work with UMAP or letting them pick plain olâ descriptors to plot. For instance, X=Loudness, Y=Zerocrossings. Or, more interesting: X=loudness, Y=1-D UMAP MFCCs. The reason being: you can learn a lot about which descriptors are interesting through visualization and browsing. However, I concede that this is more complicated from a UI point of view. Always a delicate balance.
that is in the follow-up series, as one way of dealing with high dimension count. Stay tuned!
@rodrigo.constanzo
@jamesbradbury
A disadvantage of polybuffer~ is the constant crashing in Max for Live⊠It is just unusable.
YESSSS I now have an official reason to not like it 
It has advantages (dynamic allocation and batch handling) but I had an irrational hatred for it - now you vindicated me
seriously though, a lot of our examples use a single long buffer which works well in other contexts. As for the crashs, @weefuzzy is a M4L powerabuser so he might have a trick or two to shareâŠ
First of all, thanks to the Flucoma team for these wonderful tools!
I am working on a piano timbre analysis patch in MAX and I am seeking some advice. I have been experimenting with the âvisual corpus explorationâ patch and followed the video tutorial series on âcreating a 2D corpus explorerâ. Starting from a corpus of piano recordings, I have created a 3D space representation of the timbre based on MFCC and UMAP dimensionality reduction.
Now I would like to analyze an incoming live audio stream and perform the same kind of analysis (Novelty or onset segmentation, MFCC computation, Statistical analysis and UMAP mapping) so that I can map the current timbre of the live piano on the 3D space created by the mapping learned on the corpus.
Initially, I was planning to use the signal in/out tools instead of the buffer in/out ones. But I realized that the fluid.bufstats~ and the fluid.stats~ do not perform the same kind of analysis. So, I am wondering if the correct approach is to somehow record the live incoming audio into a buffer, generate the slices in real-time and then perform the analysis with the buffer tools. Is this the appropriate approach?
Anyway, I am not sure whether someone has already worked on this kind of approach (learning a mapping with the corpus explorer and using it for real-time audio). Any advice on how to tackle this will more than welcome. Thanks in advance!
Hello and welcome!
What you are interested in is part of the large family of âaudio queryâ and there are various ways of doing it. It is very exciting and I do it all the time as I get very interesting results.
This is definitely one way to do it. I like to call it the Just-In-Time approach and there is an example on drums in the example folder of the package.
A simpler approach is to use the first 2 channels of fluid.bufstats (mean and standard deviation) as to match the fluid.stats output (which has both). We have some patches that do this too, and a tutorial coming online hopefully in the next month or two.
One important thing: MFCC are good for timbral diversity, but if you use them on a narrow timbral space (piano is quite consistent) then trying a higher number of coefficients (13 becomes 20 for instance) could help too.
Finally, there is a more involved way to do it, which is explained in the article on @tutschkuâs piece written by @jacob.hart here: Learn FluCoMa
===
So that is a lot of info What I recommend is to check the JIT example (JIT-NMF-Classifier) in the folder, and start a new thread if you have questions on it. Iâll reply quickly I promise
I hope you have fun!
Thank you very much for these hints!
I will work on them.
hi all, a flucoma newbie is here I tried to do something similar in SC, playing all the neighbours of a clicked sample sequentially, but i hear a very degraded sound quality, what did i do wrong? thanks for the hints!
best,
Alisa
s.options.numBuffers = 1024 * 15;
s.reboot;
(
// CONFIG
// ~folder = "/Users/alisakobzar/Desktop/string_orchestra/Strings/Violin/".standardizePath; //1636 files
// ~files = PathName(~folder).pathMatch("**/*").select({ |f| f.extension == "wav" });
(
~getWavFilesRecursive = { |dir|
var dirPath = PathName.new(dir);
var subdirs = dirPath.entries.select(_.isFolder);
var files = dirPath.entries.select { |f| f.isFile and: { f.extension == "wav" } };
subdirs.do { |sub| files = files ++ ~getWavFilesRecursive.(sub.fullPath) };
files
};
~folder = "/Users/alisakobzar/Desktop/string_orchestra/Strings/Violin/".standardizePath;
~files = ~getWavFilesRecursive.(~folder);
"Found % files.".format(~files.size).postln;
~files.select({ |f| f.extension == "wav" });
);
~superbuf = Buffer.alloc(s, 1, 1); // Will grow later
~offsets = List.new;
~durations = List.new;
~corpus = FluidDataSet(s).clear;
~reduced = FluidDataSet(s);
~pca = FluidPCA(s);
~ids = List.new;
)
// Merge and analyze
(
~totalFrames = 0;
~tempBufs = List.new;
)
// Step 1: Load and record lengths
({ ~files.do { |file, i|
var frames;
var buf = Buffer.read(s, file.fullPath);
~tempBufs.add(buf);
0.1.wait;
frames = buf.numFrames;
~offsets.add(~totalFrames);
~durations.add(frames);
~ids.add(i.asString);
~totalFrames = ~totalFrames + frames;
("done"++ i.asString).postln;
}}.fork;)
// Step 2: Allocate merged buffer
(~superbuf.free;
~superbuf = Buffer.alloc(s, ~totalFrames);)
// Step 3: Compose all into merged buffer
(~tempBufs.do { |b, i|
FluidBufCompose.processBlocking(s, b, destination: ~superbuf, destStartFrame: ~offsets[i]);
};)
// Step 4: Extract MFCC from each segment
({(~files.size).do ({ |i|
var loudnesses = Buffer(s);
var maxloud = Buffer.alloc(s,1);
var mfcc = Buffer.alloc(s, 13);
var entry = Buffer.alloc(s, 15);
var start = ~offsets[i];
var dur = ~durations[i].min(2048);
var centroids = Buffer(s);
var meancent = Buffer.alloc(s, 1);
FluidBufMFCC.processBlocking(s,
source: ~superbuf,
startFrame: start,
numFrames: dur,
features: mfcc,
numCoeffs: 13
);
//
FluidBufCompose.processBlocking(s, mfcc, destination: entry, destStartFrame: 0);
FluidBufLoudness.processBlocking(s,~superbuf,start,dur,features: loudnesses,select: [\loudness]);
FluidBufStats.processBlocking(s, loudnesses, stats: maxloud, select: [\high]);
FluidBufCompose.processBlocking(s, maxloud, destination: entry, destStartFrame: 1);
FluidBufSpectralShape.processBlocking(s,~superbuf,start,dur,features: centroids, select: [\centroid],unit: 1, power: 1);
FluidBufStats.processBlocking(s, centroids, stats: meancent, select: [\mean]);
FluidBufCompose.processBlocking(s, meancent, destination: entry, destStartFrame: 0);
s.sync;
~corpus.addPoint(~ids[i], entry);
("â added sample %".format(i)).postln;
});}.fork;)
// // Step 5: PCA
// ~pca.fitTransform(~corpus, ~reduced)/*.whiten*/;
// s.sync;
// ~reduced = ~corpus;
~corpus.print
~reduced.print
// ~corpus.write("/Users/alisakobzar/Desktop/string_orchestra/violin_corpus_dataset.json")
// ~superbuf.write("/Users/alisakobzar/Desktop/string_orchestra/violin_corpus_superbuf.aiff")
//umap
FluidUMAP(s,2,numNeighbours:10,minDist:0.1).fitTransform(~corpus,~reduced);
FluidNormalize(s).fitTransform(~reduced,~reduced);// normalize so it's easier to plot
~tree = FluidKDTree(s).fit(~reduced); // use a kdtree to find the point nearest to the mouse position
(
SynthDef(\sample, {
arg id, start, end;
var sig = BufRd.ar(1, ~superbuf, Phasor.ar(1,BufRateScale.ir(~superbuf), start, end));
Out.ar(0, sig * EnvGen.kr(Env([1,1,0], [((end-start)*SampleDur.ir)-0.01, 0.01]),doneAction: 2));
}).add;
)
(
~reduced.dump{arg dict;
{
var query = Buffer.alloc(s,2);
var oldval = -1;
~fp = FluidPlotter(dict: dict, xmin: -0.1, xmax: 1.1, ymin: -0.1, ymax: 1.1, mouseMoveAction: {
arg view, x, y;
// [x,y].postln;
query.setn(0, [x,y]);
/*~reduced*/ ~tree.kNearest(query, 10, {|rep|
var id, idx = 0, start, end;
// rep.asInteger.postln;
id = rep.asInteger;
idx = id;
start = ~offsets[idx];
//
end = start + ~durations[idx];
if (rep != oldval) {
idx.postln;
~files[idx].postln;
~fp.highlight = idx;
// ~playslice.(rep.asInteger);
// ~playslice = Synth(\sample, [\id, idx, \start, start, \end, end]);
~pattern = Pbind(\instrument, \sample,
// \id, Pseq(idx, 1),
\start, Pseq(start, 1),
\end, Pseq(end, 1),
\dur, Pseq(~durations[idx]/44100 + 0.1, 1),
).play;
oldval = rep;
}
});
}).pointSizeScale = 0.5;
}.defer;
}
)```