@resynthupdate flag/modality for fluid.bufnmf~

[rodrigo.constanzo]

In the thread about CV Splitting, I got an idea for being able to feed fixed seeds to fluid.bufnmf~ but as an audio audio seed, not as a fixed filter/activation seed.

@rodrigo.constanzo

is there a way to feed fluid.bufnmf~ what one of your ranks should be, and have it calculate the rest of them? I’m primarily thinking of having it do some decomposition, then taking one (or more) of the ranks and throwing some DSP at it (filtering/distortion/whatever), and then requesting what the rest of the ranks would be, to make that sound up.

So the idea would be to extend the @filterupdate and @envupdate modes extended to regular audio.

An example:

• Take a bit of audio, run @rank 10 @resynthbuf decompositions on it in fluid.bufnmf~.
• Take one (or more) of the channels of decompositions and process it in some way (filtering, distortion, waveshaping, etc…).
• Feed it into fluid.bufnmf~ with a @resynthupdate flag (same available modes) requesting as many @ranks as desired.
• Have fluid.bufnmf~ “fill in the blanks” for as many @ranks are requested.

The same mode flags would apply, using @resynthupdate 1 to have the audio used as a seed that is modified, and @resynthupdate 2 being a fixed seed.

I guess this would also open the possibility of feeding it arbitrary audio as a @resynthupdate seed (like a completely separate bit of audio (that’s the appropriate length/channels/etc…)), and having it “fill in the blanks” around it, including phase cancellations and such, based around the desired null-summed output.

This could also have interesting ramifications for Computer Assisted Orchestration (as in @tremblap’s thread) where you could decompose a signal, feed the ranks into Orchis/Orchids/Orchidee, then take (some of) the audio from that back into fluid.bufnmf~ to use as fixed @resynthupdate seeds to better decompose the rest of the audio from.

Lastly, I don’t know if this is computationally possible at all…, but it’s conceptually interesting.

[weefuzzy]

I’m not sure I follow. You’re saying we would feed NMF (at least) two buffers of time-domain audio: one to use as a target (A), and the other to provide a seeding dictionary entry (B)?

So, internally, we’d do a rank-1 decomposition of (B), and use the result of that as part of the starting dictionary, whose other entries would be seeded with noise as normal?

My suspicion is that, under some circumstances, it might not converge: if (B) is like (A) but with some extra stuff (e.g. harmonics from waveshaping), there’s nothing in NMF to deal with taking stuff away from (B) to get to (A) – in fact, it’s designed specifically to avoid cancellation effects.

[rodrigo.constanzo]

What I mean is feeding fluid.bufnmf~ a single time-domain bit of audio (A) as a seed (for a simple use case), then request n-ranks from it after that, so it would fill the rest of the ranks with noise/whatever (B+n) it normally does.

It is then set to run on another piece of audio ©.

It would then build around (A) to recreate © (from r-ranks), knowing full well that (B+n) may be filled with some really horrible shit (even more so if (A) and © are completely unrelated audio files).

As I said, I’m not sure this is possible, was just thinking.