I’m following in the footsteps of Rob Clouth after he gave this enlightening talk at Sonar Barcelona last year… he has built his own MaxforLive patch which does this: https://youtu.be/O7IpRDuvZJQ?t=2430
In the LetItBee script, the source texture is transferred to recreate the target sound. The problem is that it’s slow and a little buggy with large files, so I’ve been looking for other solutions that will produce similar results.
I’m very new to FluCoMa… but it seems like the documentation suggests using the decomposed spectral and amplitude activations to act like a kind of realtime vocoder. I was wondering if there was any apparent way to copy what the LetItBee script is doing. I’d like to decompose a source, and re-compose a seperate target, transferring the timbre of the source to the target. I imagine that this will have to be an offline process, but I’m unsure about where to begin.
Hi @encanti and welcome! @groma led us on a paper around this topic for the DaFx conference, last year as it happens. This has yielded an object, fluid.nmfmorph that will be part of the second wave of the toolkit, entering public beta soon.
Meanwhile, however, the companion page for the paper has both python code and compiled Max externals (although these will be of an earlier vintage to what will eventually be released).
Thanks for sharing this. I’ve read the paper but didn’t know about Rob Clouth’s talk. The results he gets are immediately useful. I’m going to dig a bit further into this!
Hello. We actually implemented the algorithm in the LetitBee paper in an object which we called fluid.nmfcross, which will also be released with the beta. The one mentioned by @weefuzzy is a different algorithm. In NMFCross, one spectrogram is used as a dictionary of bases, and the target is reconstructed via NMF, with several additional constraints. In our experience it was a bit tricky to get good sounding results with this method, and it generally requires a high-rank NMF.
The algorithm we proposed (NMFMorph) is based on previous work by J. Burred, here you do two separate NMF decompositions for two sounds, then use the bases of one and the activations of the other. Using optimal transport we can also do a continuous interpolation between the original and the new NMF bases.
This gives you a clear interface in that you are using the activations (so temporal dynamics) of one sound and the bases (spectral components, usually capturing pitch and / or timbre) of the other, but we are not really trying to reconstruct the target sound.
@weefuzzy Thank you so much for the morphing paper recommendation with the companion page link! I’ve managed to download these Max prototypes, and having a lot of fun with NMF morphing already.
@groma I can’t wait to get my hands on that NMFCross beta! Getting out of python and into a Max workflow is going to be a game changer for me.
By the way, incase anybody is interested - I needed to mod the Let It Bee python script in two essential ways: Make it stereo, and make it accept a corpus instead of a single audio file. My friend A.Russo modded the python script to fit these specs. So if you’re interested, here is NMF Let It Bee script in stereo and now accepting folders as corpus:
An example bash script with the corpus function would look like:
python3 Musaicing.py --sourceCorpus corpusfoldername/ --target targetfile.wav --result sourcecorpustotarget.wav --sr 44100 --NIters 30
Also - and I can not recommend this enough… give a deep listen to Rob Clouth’s “Zero Point” album. It is one of the most nuanced, complex, beautifully crafted electronic music production masterpieces of the decade, and the bulk of the sound design was made with NMF synthesis. Definitely a huge inspiration and benchmark piece for NMF style concatenative synthesis as a high end sound design medium: Zero Point | Rob Clouth
Welcome to the forum! I don’t know your Max fluency so let me know if this is too much, but I find tht the bufnmf~ object also keeps on giving for me.
There are examples in the folder where it is used as dirty cross-synthesis in a sort of complex vocoder style approach, but there is also a classifier example where you could match elements/objects recognised in a source to triggers instead… yet another option is to do musaiking of a gated version of each voice of the output… and again you can just replace/process one of such voice and keep the original intact.
It is not perfect but I’ve had a lot of fun with it, and shared some of them in the folders, and some on the forum, as have done some others.
Feel free to share where you hit a learning bump, so we can help !
Is there any way I can get my hands on NMFcross beta? I’m producing a record for my masters research project with all sound design based around the LetItBee NMF algorithm, and it would be absolutely colossal to get the job done with Max instead of my hacky python script. I’ll be happy to report back with bugs / etc. any way I can be helpful in testing the object.
Hello, yes it needs a bit of review on our side but I reckon it could be in a shareable state in the next few weeks. Some testing will be useful indeed.
Hello! I’m going to be attending a FluCoMa workshop next week in Edinburgh checking back to see if there’s any new developments on NMF-based mosaicing added to the new betas.
checking back to see if there’s any new developments on NMF-based mosaicing added to the new betas.