I’m trying to concatenate large amounts of samples, but currently the fluid.concataudiofiles gives no clear indication of how long this can take and if I should abort or wait a little longer. So I’ve tried to work on a version that uses @blocking 0. It seems to work quite okay and prints the total progress.
But now I noticed that the progress seems to slow down quite a lot over time and was wondering what could be the reason for this or if anyone has some suggestions to improve this patch? For example if I concatenate 1500 samples (all are between 1-10 seconds long, total length about 10 minutes), it starts out quite fast (like 30-50 ms per file), but this gradually increases to 800-900 ms per file in the end. I thought this might have something to do with fluid.bufcompose~ automatically resizing the buffer if the length is exceeded, so I tried to set a total amount of time for the buffer~ (since i don’t really care if there is a lot of silence in the end), but that doesn’t seem to make a difference. Do you have any other suggestions or should I just wait patiently? (now with a clear indication of where it is in the process I don’t really mind, haha)
Without looking my gut says that each time a file is concatenated the destination is copied to and is longer. Each time a non-blocking process is triggered it copies the contents of the buffer into memory somewhere else, so this repetitive copying, combined with it taking up more and more space likely contributes.
Without looking either, but reading your about your buffer length, I have a quick test to propose to do, as we know our non-blocking is doing suboptimal memory management at the moment (@weefuzzy is on the case but it is a deep design thinking one).
You could try to bufcompose the section you are analysing to a temp, small buffer, and pass that one to the non-blocking job. If that is more stable, that confirms this is the case. If that is not more stable, it might be something worse and I can check for memory leak and such pleasurable things…
I’m not entirely sure I get what you mean here. So at the moment I’m not yet analyzing any of the buffer contents. It’s just only the concatenation, and I’ll wait till it’s finished before doing any further processing. So I’m loading a file in a [buffer~], then increment the (deststartframe $1) and then process the [fluid.bufcompose~ @blocking 0], the bufcompose will trigger the next path to be loaded to buffer etc, etc.
Do you maybe mean I should first move the content of the buffer~ to a different [fluid.bufcompose~] (like basically making a copy?).
Ah! So that is why it is going wrong. Non-blocking is not useful there for 2 reasons
multithreading needs you to copy the memory to a thread so there is copy overhead
you are only copying so you would copy to copy
But what is your problem is that everytime you add, you have to allocate more memory. But I had that problem and even made a solution Check the Example folder in the BufCompose section and you will see something that is doing it in 2 passes and it is very, very quick. One pass to get the dimensions of the final buffer, and one pass to read them all in the right spot. Doing this does not need the buffer to grow every time you copy (it does it in place) and then the speed is linear.
Alright! This was indeed what I needed. So conclusion: [fluid.bufcompose~] in combination with [deferlow] and presetting the buffersize is blazingly fast. Other big bottlenecks: don’t print the progress, this slows down the process drastically, and the fluid.waveform~ for displaying large files is also quite heavy.
If you use the regular [waveform~] linked to the destination buffer it actually shows a nice ‘work-in-progress’ gradually building the waveform while bufcompose~ is working. So this does the trick for me. Processing the same 1500+ files now took me 2 seconds (which before was definitely around 5 minutes…)
Patch for anyone interested (although very similar to @tremblap 's example). Did include an abort button and input is a list of paths similar to fluid.concataudiofiles:
For anyone interested, here is some code you can put in a [node.script] to recursively search folders for any audiofile that is a wav or aiff and output them all as a list. Make sure you $ npm i fast-glob. Combine this with the patch below to use an [opendialog] as input.
const max = require('max-api');
const fg = require('fast-glob');
max.addHandler('load', (p) => {
let glob = '**/*.+(wav|WAV|aif|AIF|aiff|AIFF|mp3|MP3)';
let files = fg.sync(p + glob);
max.post(`found ${files.length} files`);
max.outlet(files);
});
(now with a clear indication of where it is in the process I don’t really mind, haha)