Like so many “big tech company” “music AI” things, it sounds like fucking dog piss, but it’s kind of interesting as well:
From the demo video it sounds like pitch tracking and sending it into a (really good) acoustic model, which is, sadly, drowning in reverb.
I imagine there’s some concatenation-type things going on as well, as you can hear from one of the examples at the very end (low tongue drum thing to violin).
I actually really like it for some of the dumb errors it makes. You can really hear the voice in this one:
and also super impossible playing in this one:
Also make sure to play with the loudness slider - it gets interesting when its turned down to -10
It’s interesting to see how it responds to other types of sounds.
It is super pitch heavy in terms of where it leans/analyzes. It doesn’t really know what to do with data noise.
It seems to take a bit to render too, which, unless it’s doing it in the browser, there’s some heavy lifting going on in the analysis.
examples.zip (5.4 MB)
It is actually quite impressive - the artefacts are no problem for the kinds of sounds I’d feed them 
But I think @groma will tell us that training is costly on this
10 hours to get something half decent, and that would be on one of their TPUs I imagine.
ok, not for now for me (not really any use-case of making my synths sound like flutes
So apparently someone from IRCAM has made an external for Pd that lets you run ddsp tone transfer models.
I’ve just compiled it and tried it feeding it pitch and envelope data from sigmund~, but I can only manage to get a very bitcrushed-like sound no matter how I scale the envelope signal (which is not scaled at all in the example patch, only lowpassed, but still same bitcrushed sound). May be of interest to someone here though… There’s only links to pretrained violin and sax models, but it looks like you could train your own (in theory, at least).
Don’t know if this was the case before, but it looks like you can train your own models here too now:
As always, the examples shown are pretty boring, but could be interesting training up some models with weird/interesting sounds.
@jamesbradbury @renatrigiorese @bledsoeflute this online training brags about 2-3h to train on the free version of colab. Did you try it? I’m curious and might give it a go - they are a bit shy on the file you need to provide (monophonic…) but hey, with such a short training time, it is worth trying it with my naughty synths.
ok I spent too much time fighting a colab pad that has so many bugs I cannot even get to the training… has anyone been successful in running the ddsp trainer online?
