Hi,
I’m currently working on optimising the feature calculation process for a dataset, and I’m experiencing some issues with the non-blocking + queue mode.
What I’m trying to do is trigger several different analyses in parallel on a folder of audio files. However, I keep encountering some recurring problems, usually related to fluid.bufflatten~:
-
Sometimes the object keeps banging even after the analysis of the series of samples has already finished, as if it were stuck.
-
When adding points to a fluid.dataset~ after the analysis pipeline, I sometimes get repeated results from the same sample analysis. It looks as if the same sample is being analysed at each cycle instead of the queue moving on to the next one.
Is this due to my misunderstanding of how to use non-blocking mode, or could there be something wrong?
Thanks,
Matteo
Archive.zip (40.8 KB)
hello!
Non-blocking queue for batch processing can be a nightmare… I actually never use it, and the simple reason is that, each object having its own thread, you have no way what is in which state when a job is finished.
Let me explain:
in a chain A→B, job A is fast. job B is slow. You send 9 buffers in A.
to exagerate, A processes 10 faster than B processes 1 job. So after jobA1, jobB1 starts, during which A2 to A9 are being processed and their out buffer updated… 8 other bangs are sent into B but they are cuing. When it is time for B2 to happen, what is the state of each buffer…
So I don’t use it. I do a single chain of non-blocking and use the flag at the bottom to incremement the top.
There is a way to make it work though: you change the name of all intermediary buffers. So that will work. but it will probably be faster to run 2 (or 4) of my chains in parallel, a similar level of buffer management for true parallel processing between those chains.
I hope this helps? If not, we can make a simple example:
A: bufcompose
B: bufNMF
That way we can work out the logic. We can also make a very scholar example with bufthreaddemo and decide how long each process takes 
this object’s existance is exactly for that pedagogical purpose.
1 Like
Ok, I think I understand what the problem is: it is a bit like a supply chain in which each process has its own pace.
In my specific case, switching from default mode to non-blocking mode improved the speed a lot. So, if I understand correctly, would the best solution be to leave all the objects in default mode, and set only the last object of each process to non-blocking?
Thanks
Matteo
I think that the best approach is to all put them in non-blocking, but to not use the queue and manage your iteration yourself with a loop. When the last task is done, start the top of the chain again.
then when that works, you can do the same with multiple parallel threads, splitting the jobs manually