Tone Transfer - From Google/Magenta

Like so many “big tech company” “music AI” things, it sounds like fucking dog piss, but it’s kind of interesting as well:

From the demo video it sounds like pitch tracking and sending it into a (really good) acoustic model, which is, sadly, drowning in reverb.

I imagine there’s some concatenation-type things going on as well, as you can hear from one of the examples at the very end (low tongue drum thing to violin).

I actually really like it for some of the dumb errors it makes. You can really hear the voice in this one:

and also super impossible playing in this one:

1 Like

Also make sure to play with the loudness slider - it gets interesting when its turned down to -10

1 Like

It’s interesting to see how it responds to other types of sounds.

It is super pitch heavy in terms of where it leans/analyzes. It doesn’t really know what to do with data noise.

It seems to take a bit to render too, which, unless it’s doing it in the browser, there’s some heavy lifting going on in the analysis.

examples.zip (5.4 MB)

It is actually quite impressive - the artefacts are no problem for the kinds of sounds I’d feed them :slight_smile: But I think @groma will tell us that training is costly on this :slight_smile:

10 hours to get something half decent, and that would be on one of their TPUs I imagine.

ok, not for now for me (not really any use-case of making my synths sound like flutes :slight_smile:

So apparently someone from IRCAM has made an external for Pd that lets you run ddsp tone transfer models.

I’ve just compiled it and tried it feeding it pitch and envelope data from sigmund~, but I can only manage to get a very bitcrushed-like sound no matter how I scale the envelope signal (which is not scaled at all in the example patch, only lowpassed, but still same bitcrushed sound). May be of interest to someone here though… There’s only links to pretrained violin and sax models, but it looks like you could train your own (in theory, at least).

3 Likes