04-real-time-concat.scd (6.7 KB)
hey, I had a play with this patch and your house sounds: Stream House House by Mark Hanslip | Listen online for free on SoundCloud
1 Like
this sounds great. a really nice concat synthesis result!
t
Hey @tedmoore . What is line 77 supposed to do?
if((slice_index % 100) == 99){s.sync};
1 Like
Hello
This line enables a batch-processing approach. In effect, the patch is blocking the server in this case, but if you sent thousands of jobs in one go, we noticed that some get lost. so this is syncing the server (a time-consuming process) every 100 jobs.
I hope this helps?
2 Likes
Hi!
Iām messing around with this patch these days, trying to adapt some ideas to my live coding environment. (@tedmoore I finally got some time to do that, as I told you back in Trento!)
I managed to came out with something which actually works, and now Iām working on optimization. In this sense, got a few questions thoā¦ Iāll leave them here in separate comments for clarity.
1
Iām trying to optimize the buffer allocation. Specifically, in ~analyze_to_dataset, as far as I understood var features_buf = Buffer(s); var stats_buf = Buffer(s); var flat_buf = Buffer(s); are ātemporaryā buffers which serves to store the analysis, which is then added as datapoints into var dataset = FluidDataSet(s); . Therefore, it makes sense to free them after the analysis have been done.
However, if I try to do that:
~analyze_to_dataset = {
arg audio_buffer, slices_buffer, action;
var features_buf = Buffer(s);
var stats_buf = Buffer(s);
var flat_buf = Buffer(s);
//...
slices_buffer.loadToFloatArray(action:{
arg slices_array;
fork{
slices_array.doAdjacentPairs{
//... all the processes
};
s.sync;
action.value(dataset);
};
});
features_buf.free; stats_buf.free; flat_buf.free;
}
ā¦when I actually try to evaluate the analysis I got this:
ERROR: FluidBufMFCC: Invalid features buffer
That feels a little bit weird to me: shouldnāt all the analysis being done at that point in the function? What am I missing?
I agree that it seems like your strategy should work. Can you post the whole code or point me to which file youāre modifying so I can take a look?
T
Sure! Here it is:
chunk_analysis.scd (1.7 KB)
Just put a folder with some random audio files on line 5.
On my side, everything works, unless I uncomment line 71, then I got the aforedmentioned error.
Because youāre forking a thread and doing the processing in that fork, the rest of the function continuesā¦right on down to the place where you free the buffers (you get there before all the processing is done).
I moved the buffer freeing to be inside the fork and happen once weāre sure all the analysis is complete. Check it out:
s.boot
// GLOBAL
(
// ~source_folder = "/Users/ardan/Desktop/NUOVO_LC/chunks/"; // <-- folder with some audio files
~source_folder = "/Users/ted/Desktop/favs-mono/";
~n_mfccs = 13;
~load_files = FluidLoadFolder(~source_folder);
~load_files.play(s, { ~full_buf = ~load_files.buffer});
~indeces_list= List[];
)
(
~load_files.index.keysDo{ |key|
var bound = ~load_files.index[key][\bounds][1].postln;
~indeces_list.add(bound);
}
)
(
~indeces_list.addFirst(0).sort.asCollection;
~indeces_buf = Buffer.alloc(s, ~indeces_list.size);
~indeces_buf.loadCollection(~indeces_list)
)
(
~analyze_to_dataset = {
arg audio_buffer, indeces_buffer, n_mfccs, action;
var temp_mfcc_buf = Buffer(s);
var temp_mfcc_stats_buf = Buffer(s);
var temp_mfcc_flat_buf = Buffer(s);
var dataset = FluidDataSet(s);
indeces_buffer.loadToFloatArray(action:{
arg slices_array;
fork{
slices_array.doAdjacentPairs{
arg start_frame, end_frame, slice_index;
var num_frames = end_frame - start_frame;
FluidBufMFCC.processBlocking(s,
audio_buffer,
start_frame,
num_frames,
features: temp_mfcc_buf,
startCoeff: 1,
numCoeffs: n_mfccs);
FluidBufStats.processBlocking(s,
temp_mfcc_buf,
stats: temp_mfcc_stats_buf,
select: [\mean]);
FluidBufFlatten.processBlocking(s,
temp_mfcc_stats_buf,
destination: temp_mfcc_flat_buf);
dataset.addPoint("slice-%".format(slice_index), temp_mfcc_flat_buf);
if((slice_index % 50) == 49){s.sync};
};
s.sync;
temp_mfcc_buf.free; temp_mfcc_stats_buf.free; temp_mfcc_flat_buf.free;
action.value(dataset);
};
};
);
};
)
(
~analyze_to_dataset.(~full_buf, ~indeces_buf, ~n_mfccs, {
arg dataset;
~source_dataset = dataset;
~source_dataset.print;
});
)
(
~source_dataset.dump{
arg ds;
ds["data"].keysValuesDo{
arg id, val;
"%\t%".format(id,val).postln;
};
};
)
1 Like
ā¦Hereās the other question I have at the moment (probably more to followā¦ 
)
2
This is maybe a strictly SC-related question, still a good gym to learn.
I readapted the code in Section 8. Now, letās say I have two Synths that sends OSC messages via SendReply.kr:
SynthDef.new(\concat, {
| onset_threshold = 0.1, delay = 0, dur_coeff = 1, pan = 0 |
var input, onset, trig;
input = DelayN.ar(PlayBuf.ar(1, ~target_buf, BufRateScale.ir(~target_buf), loop:1), delay);
onset = FluidOnsetSlice.ar(input, metric: 0, threshold: onset_threshold);
trig = A2K.kr(Trig1.ar(onset, ControlDur.ir));
FluidKrToBuf.kr(FluidMFCC.kr(input, numCoeffs: ~n_mfccs, startCoeff: 1), ~mfccbuf);
SendReply.kr(trig, "/mfcc_neighbor", [dur_coeff, pan]);
}).add;
SynthDef.new(\single_knear, {
| bufn, num_neigh, pan, trig=1 |
SendReply.kr(trig, "/knear_mfcc_neighbor", [bufn, num_neigh]);
}).add;
OSCdef(\mfcc_neighbor,{
arg msg;
var dur_coeff = msg[3];
var pan = msg[4];
~scaler.transformPoint(~mfccbuf, ~scaledbuf);
~kdtree.kNearest(~scaledbuf,1,{
arg nearest;
var int = nearest.asString.split($-)[1].asInteger;
Synth.new(\chunk_player, [\index, int, \dur_coeff, dur_coeff, \pan, pan]);
});
},"/mfcc_neighbor");
OSCdef(\knear_mfcc_neighbor,{
arg msg;
var bufn = msg[3];
var num_neigh = msg[4];
~scaler.transformPoint(~fix_analysis_buf, ~scaledbuf);
~kdtree.kNearest(~scaledbuf, num_neigh,{
arg nearest_n;
var nearest = nearest_n.choose;
var int = nearest.asString.split($-)[1].asInteger;
nearest_n.postln;
nearest.postln;
Synth.new(\chunk_player, [\index, int, \dur_coeff, 1, \pan, 0]);
});
},"/knear_mfcc_neighbor");
The idea here is that I want to have two way of play chunks: one via concatenative synthesis (similar to the original code), and the other one which simply chooses one out of n k-nearest chunk with respect to a fix analysis that I stored in a separate buffer.
Hereās my full example code for reference:
chunk_analysis.scd (4.5 KB)
(for sake of simplicity, Iām using the same example audio file both for playback and both for calculating the fix mean)
How threads are managed within SC is still extremely hazy to me, so I wanted to experiment a bit.
Iām using the same scaler and the same ~scaledbuf to retrieve the index of the chunk being played by both pipelines - lines 147 and 160. In my head, that āshouldā optimize the code.
Indeed, if I play both Synths x and y (lines 197-198) that seems to work fine - x keeps playing continuously, while y needs to be evaluated individually.
In my code, Iām also printing the k-nearest array from which I choose from along with the chosen slice, and indeed, I have something like this:
-> Synth('single_knear' : 1009)
[ slice-27, slice-29, slice-31, slice-32 ]
slice-31
-> Synth('single_knear' : 1011)
[ slice-27, slice-29, slice-31, slice-32 ]
slice-27
-> Synth('single_knear' : 1013)
[ slice-27, slice-29, slice-31, slice-32 ]
slice-32
However, if I keep evaluating y at a fast pace, i occasionally run into the following error:
ERROR: Message 'choose' not understood.
Perhaps you misspelled 'cos', or meant to call 'choose' on another receiver?
RECEIVER:
Symbol 'slice-71'
In the error, there is a slice (71 in this case) which do not belongs to the array. But Iām not sure at all where (and why) the error happens!
I think if num_neigh == 1, then .kNearest returns a symbol. If num_neigh > 1, then .kNearest returns an array of symbols.
Test that and see if it is correct. Then you can make a check: if num_neigh is > 1, use .choose, otherwise, thereās nothing to choose from, just use the returned symbol.
I didnāt test it, let me know what you find.
Mmm, currently Iām always passing num_neigh=4, thatās why it feels weirdā¦ Also, the fact that the error seems related to a slice number (e.g. 71) that is not present in the symbol array (e.g. [27,29,31,32]) makes me assume that the error is actually coming from the other OSCDefā¦
Is it possible that some conflict happens when the two .kNearest are evaluated too closely?
Thatās a good hypothesisāand a question for @weefuzzy .
You might try keeping all of this on the server using FluidKDTreeās .kr method. Thereās an example in the helpfile. Youāll need to get the slice start positions in a FluidDataSet rather than a buffer.
I tried this:
OSCdef(\knear_mfcc_neighbor,{
arg msg;
var bufn = msg[3];
var num_neigh = msg[4];
~scaler.transformPoint(~fix_analysis_buf, ~scaledbuf);
~kdtree.kNearest(~scaledbuf, num_neigh,{
arg nearest_n;
var nearest, int;
if (num_neigh <= 1, {
nearest = nearest_n
},{
nearest = nearest_n.choose
});
int = nearest.asString.split($-)[1].asInteger;
Synth.new(\chunk_player, [\index, int, \dur_coeff, 1, \pan, 0]);
});
},"/knear_mfcc_neighbor");
Now, if I keep evaluating y = Synth.new(\single_knear, [\num_neigh, 1]);, indeed I get no error, it seems. Otherwise, if I go back to y = Synth.new(\single_knear, [\num_neigh, 2]);, the error sometimes appears again.
ā¦And yep, the good practice would probably be to keep everything well separated, I was just wondering why that is happening!
Iām doing some tests and becoming more convinced this is the issue.
The issue goes away if I use two FluidKDTrees:
s.boot
// GLOBAL
(
// ~source_folder = "/Users/ardan/Desktop/NUOVO_LC/chunks/"; // <-- folder with some audio files
~source_folder = "/Users/ted/Desktop/favs-mono/";
~n_mfccs = 13;
~load_files = FluidLoadFolder(~source_folder);
~load_files.play(s, { ~full_buf = ~load_files.buffer});
~indeces_list= List[];
)
(
~load_files.index.keysDo{ |key|
var bound = ~load_files.index[key][\bounds][1].postln;
~indeces_list.add(bound);
}
)
(
~indeces_list.addFirst(0).sort.asCollection;
~indeces_buf = Buffer.alloc(s, ~indeces_list.size);
~indeces_buf.loadCollection(~indeces_list)
)
(
~analyze_to_dataset = {
arg audio_buffer, indeces_buffer, n_mfccs, action;
var temp_mfcc_buf = Buffer(s);
var temp_mfcc_stats_buf = Buffer(s);
var temp_mfcc_flat_buf = Buffer(s);
var dataset = FluidDataSet(s);
indeces_buffer.loadToFloatArray(action:{
arg slices_array;
fork{
slices_array.doAdjacentPairs{
arg start_frame, end_frame, slice_index;
var num_frames = end_frame - start_frame;
FluidBufMFCC.processBlocking(s,
audio_buffer,
start_frame,
num_frames,
numChans: 1,
features: temp_mfcc_buf,
startCoeff: 1,
numCoeffs: n_mfccs);
FluidBufStats.processBlocking(s,
temp_mfcc_buf,
stats: temp_mfcc_stats_buf,
select: [\mean]);
FluidBufFlatten.processBlocking(s,
temp_mfcc_stats_buf,
destination: temp_mfcc_flat_buf);
dataset.addPoint("slice-%".format(slice_index), temp_mfcc_flat_buf);
temp_mfcc_flat_buf.postln;
if((slice_index % 50) == 49){s.sync};
};
s.sync;
action.value(dataset);
};
};
);
//temp_mfcc_buf.free; temp_mfcc_stats_buf.free; temp_mfcc_flat_buf.free;
};
)
(
~analyze_to_dataset.(~full_buf, ~indeces_buf, ~n_mfccs, {
arg dataset;
~source_dataset = dataset;
//~source_dataset.print;
});
)
(
~kdtree1 = FluidKDTree(s);
~kdtree2 = FluidKDTree(s);
~scaled_dataset = FluidDataSet(s);
~scaler = FluidStandardize(s);
~scaler.fitTransform(~source_dataset, ~scaled_dataset);
~kdtree1.fit(~scaled_dataset, {"kdtree1 fit".postln;});
~kdtree2.fit(~scaled_dataset, {"kdtree2 fit".postln;});
~mfccbuf = Buffer.alloc(s, ~n_mfccs);
~scaledbuf = Buffer.alloc(s, ~n_mfccs);
)
(
~target_path = FluidFilesPath("Nicol-LoopE-M.wav");
~target_buf = Buffer.read(s,~target_path);
)
// SOME DEFS:
(
SynthDef.new(\chunk_player, {
| index, dur_coeff = 1, pan = 0 |
var start, duration, dur_scaled, env, sig, out;
start = Index.kr(~indeces_buf, index);
duration = (Index.kr(~indeces_buf, index + 1) - start) / SampleRate.ir(~full_buf);
dur_scaled = max(0.07, duration * dur_coeff);
env = EnvGen.kr(Env([0, 1, 1, 0], [0.02, dur_scaled - 0.07, 0.05]), doneAction: 2);
sig = PlayBuf.ar(1, ~full_buf, BufRateScale.ir(~full_buf), trigger: 0, startPos: start);
out = Out.ar(0, Pan2.ar(sig, pan));
}).add;
SynthDef.new(\concat, {
| onset_threshold = 0.1, delay = 0, dur_coeff = 1, pan = 0 |
var input, onset, trig;
input = DelayN.ar(PlayBuf.ar(1, ~target_buf, BufRateScale.ir(~target_buf), loop:1), delay);
onset = FluidOnsetSlice.ar(input, metric: 0, threshold: onset_threshold);
trig = A2K.kr(Trig1.ar(onset, ControlDur.ir));
FluidKrToBuf.kr(FluidMFCC.kr(input, numCoeffs: ~n_mfccs, startCoeff: 1), ~mfccbuf);
SendReply.kr(trig, "/mfcc_neighbor", [dur_coeff, pan]);
}).add;
SynthDef.new(\single_knear, {
| bufn, num_neigh, pan, trig=1 |
SendReply.kr(trig, "/knear_mfcc_neighbor", [bufn, num_neigh]);
}).add;
OSCdef(\mfcc_neighbor,{
arg msg;
var dur_coeff = msg[3];
var pan = msg[4];
// "mfcc_neighbor".postln;
/* ~n_mfccs.postln;
~mfccbuf.postln;
~scaledbuf.postln;
~scaler.postln;
~scaler.dump{arg scl; scl.keysValuesDo{arg k, v; k.post; " ".post; v.postln;} };*/
~scaler.transformPoint(~mfccbuf, ~scaledbuf);
~kdtree1.kNearest(~scaledbuf,3,{
arg nearest;
var int = nearest.asString.split($-)[1].asInteger;
Synth.new(\chunk_player, [\index, int, \dur_coeff, dur_coeff, \pan, pan]);
});
},"/mfcc_neighbor");
OSCdef(\knear_mfcc_neighbor,{
arg msg;
var bufn = msg[3];
var num_neigh = msg[4];
// "knear_mfcc_neighbor".postln;
~scaler.transformPoint(~fix_analysis_buf, ~scaledbuf);
~kdtree2.kNearest(~scaledbuf, num_neigh,{
arg nearest_n;
var nearest, int;
if (num_neigh <= 1, {
nearest = nearest_n
},{
if(nearest_n.size != num_neigh) {
"*****************************************************".postln;
"num_neigh: %".format(num_neigh).postln;
"nearest_n: %".format(nearest_n).postln;
"*****************************************************".postln;
};
nearest = nearest_n.choose;
});
int = nearest.asString.split($-)[1].asInteger;
Synth.new(\chunk_player, [\index, int, \dur_coeff, 1, \pan, 0]);
});
},"/knear_mfcc_neighbor");
)
(
~fix_analysis_buf = Buffer(s, ~n_mfccs);
~analyze_sample = {
var temp_mfcc = Buffer(s);
var temp_stat = Buffer(s);
FluidBufMFCC.processBlocking(s, ~target_buf, features: temp_mfcc, windowSize: 2048, action:{"done".postln;});
FluidBufStats.processBlocking(s, temp_mfcc, stats: temp_stat, select: [\mean]);
FluidBufFlatten.processBlocking(s, temp_stat, destination: ~fix_analysis_buf);
};
~analyze_sample.()
)
x = Synth.new(\concat);
y = Synth.new(\single_knear, [\num_neigh, 4]);
x.free;
y.free;
1 Like
Gotcha, Iāll use two FluidKDTrees then.
ā¦Still, if @weefuzzy could explain whatās really happening there, it would be useful to know!
Oh, btw, happy new year everybody! 
1 Like
Donāt know whatās going on yet, although Iām not at my brightest today. I canāt yet reproduce a problem with a stripped back synthetic example, but that could be because I donāt yet grasp whatās going wrong.
s.reboot;
//==============================================================
//Make some quite literally random data and throw it in a tree:
p = Pgauss(0,10).asStream;
//batch of buffers
(
~bufs = 100.collect({
Buffer.sendCollection(s,p.nextN(4));
});
)
//put in ds
~ds = FluidDataSet(s);
// fill ds
(
fork{
100.do{|i|
i.postln;
~ds.addPoint(i.asSymbol,~bufs[i], {
"added %".format([i]).postln;
});
};
}
)
//make a tree
(
~tree = FluidKDTree(s);
~tree.fit(~ds);
)
//=========Hit the tree quite hard===========================
~query_buffers = 2.collect{Buffer.alloc(s,4);};
//synthdef factory
(
~make_synth = {|resp, buf|
SynthDef.new(resp.asSymbol, {
|num_neigh|
var a = NamedControl.kr(\a,[0,0,0,0]);
BufWr.kr(a[0], buf,0);
BufWr.kr(a[1], buf,1);
BufWr.kr(a[2], buf,2);
BufWr.kr(a[3], buf,3);
SendReply.kr(DC.kr(1), "/" ++ resp, [buf.bufnum, num_neigh]);
FreeSelf.kr(1);
}).add;
};
)
//thrashing machine
(
var a = Pgauss(0.002,0.01).asStream; //for random wait durations
var f = {|expected_k| {
arg msg;
var bufn = msg[3];
var num_neigh = msg[4];
if(expected_k != num_neigh.asInteger,{
"ARGGHNO".throw;
});
~tree.kNearest(bufn.asInteger, num_neigh,{
arg nearest_n;
var nearest;
nearest = nearest_n.asArray.choose;
nearest_n.asArray.postln;
nearest.postln;
});
}};
~make_synth.value("msgA", ~query_buffers[0]);
~make_synth.value("msgB", ~query_buffers[1]);
OSCdef(\msga,f.value(1),"/msgA");
OSCdef(\msgb,f.value(4),"/msgB");
Routine({
loop({
Synth.new(\msgA, [\num_neigh, 1,\a,p.nextN(4)]);
a.next.abs.wait;
})
}).play;
Routine({
loop({
Synth.new(\msgB, [\num_neigh, 4,\a,p.nextN(4)]);
a.next.abs.wait;
})
}).play;
)
1 Like
Ok, doing a bunch of OSC tracing around the failing code, I think the problem is more to do with how weāre dealing with OSC message dispatch language side for objects like KDTree (but all the rest of these data objects too),
(if you have two OSC callbacks competing on the same object, they can end up getting āeach otherāsā messages because weāve keyed the callbacks to the object instances; Iāll need to revisit what weāre doing, remember why, and then make it more robustā¦)
2 Likes
Right, shortest reproduction I can think of that shows the issue:
s.reboot
(
~ds = FluidDataSet(s);
~buf = Buffer.alloc(s,1);
~ds.addPoint(\1,~buf,{"Never called :-("});
~ds.addPoint(\2,~buf,{"I get called twice?".postln});
)
The action callback for the second call to addPoint wipes out the first, and thatās essentially whatās happening above: the callbacks for kNearest are fighting with each other.
Not sure what a robust fix would be yet.
3 Likes
Dear @ardan thanks for your questions - I think this thread would benefit all users (at the moment it is limited to the CCRMA workshop participants) - would you mind if I was making it public? The SC community would benefit a lot from it I think!
@markhanslip and @heretogo I reckon it is ok with you two too?
1 Like