Training for real-time NMF (fluid.nmfmatch~)

So looking through Alpha-02, specifically the fluid.nmfmatch~ help file and couldn’t help but think that this would be super useful in a bunch of the things I do.

This may be a simple/stupid question (it’s been a bit since I mess with the FluCoMa files), but is it possible to train the dicts for this via “human sorting”? As in, feeding it a single sound (or representative sounds) and building a dict from that, then doing the same for each sound you want to have it match.

From what I can tell, all of the examples in the help file (including the pretrained piano) require fluid.bufnmf~ to separate the source material into components for you.

I guess the analog would be, instead of having `fluid.bufnmf~’ chug for hours on the piano audio file, if it would have been possible to play in individual notes (A0, A#0, etc…) to build the same dictionary.

1 Like

read the piano training method with the 1rank pretraining. This is what I do first then do the synthetic long nmf~ but you might have luck with a small synthetic buffer of ranks like I do for my pre-training… give it a whirl and let us know - it should work to some extent.

The source piano audio file isn’t included, but I take it that you are literally telling it to analyze the 3 seconds which correspond to each piano note?

I see, it took me a while to figure out that it’s writing each dict as an individual buffer channel. Looks like double-clicking on buffer~ pno-dicts piano-dicts.wav shows all 89 dicts, but buffer~ pno-dicts only shows one, which confused the shit out of me.

As a mini feature-request, a non-polyphonic version would be handy in the help file as piano/sustained sounds come with additional problems and complications that aren’t applicable in other cases.

you are right about the piano training, I’ve also pushed it on the shared audio files of the forum if you want to hear it.

Have you tried the bird finder patcher with drums instead? A combination of both should help you voice your desires - and then I am happy to help too but I don’t understand your last question (non-polyphonic)… do you mean to know which one it is likely to be more between the N ranks?

I’ll take a closer look at the other patch.

I mean more in terms of how fluid.nmfmatch~ responds when being fed sustained sounds (since no component contains the “chords” being played it jumps around wildly).

I’m mainly, and obviously, thinking that this would be great for the training of specific drum sounds and/or their nearest equivalents, and with that also thinking of latency, so being able to tell what was nearest as a single shot would be idea (without having to go into the scheduler while having a separate onset detection algorithm).

I think that between the 2 examples you have something to try there. Let me know where/when you get stuck!

I’ll go back and create some recordings of training-suitable drum hits and give it a go. (I did some simultaneous recordings using DPAs and the Sensory Percussion trigger last year, but they were performative/mixed hits)

1 Like

Ok, so I finally got around to making some recordings and then a patch to analyze with, and I’ve run into some problems I don’t understand.

The idea of the patch is to allow for training 10 different sounds, with the intention of then being to match against them. Unlike the piano example, I’ve built the patch to take individual recordings (ranging from 16s to 56s) to create all the individual dictionaries.

This part appears to be working fine (I think). Where it goes bad is where I try to create the preseed dictionary.

Why are there 89 ranks/buffers for 88 notes?
Why is it analyzing the whole (combined) audio file looking for the ranks?
What should the output of it look like? (without your source audio file it’s not possible to run the training patch)

Here are the audio files:
http://rodrigoconstanzo.com/party/nmf_audio.zip

After that, the real-time matching doesn’t work, but I think that’s because the preseed dictionary step is messed up.


----------begin_max5_patcher----------
5973.3oc68r1iaiaseN4WAgw8Bzd6DG9VT4Ka1toc2Ena281rWrnHoXfrMmw
ZirjqjblLsn4298PRIYIaIaM1xNSx5.jYzP8fm27vCO7v+8SexnIIePmMB8B
zaPO4I+6m9jmXaxzvSJ96mLZQvGlFEjYerQSSVrPGmO5J28x0eH219+6JcVd
XRb1Kda7uL+dTXFZxpahWbCJHNH59+UX7snXze.Q9ptue.5t4IQZTvpYgIna
BizvCW83vytHHe5bzpLyCOK7lazo.nft4lbTlNOGZM6qP+tIvk5TTTPtNd58
e0u28IBxQ2krJZFZhFcaRxrFuE5ljTTbR7yVFBcfdFZV5pEnrjUwyLeQ2KtL
UCcHp4mG89LTTxcnaR0+STpNKIZkgJX5yuOCEliVljkENAPp7D39KRduFkrJ
OJTmB2NFkOWCXxTy6DjdOZZpNv.Qnr7fa0eUIUNJLVOEfFKolfKZE.nL.+CL
u70s+HgyrLmjI+5yHD+xO2RCYD5kqS0SycLeeoZr3JjmhOlvoJJkJERJzBGK
GiuBQ3jwXz+n3CDuZQXbjN2JRPV2nAyJZEaZ7+7zmZ9wUGor025XS.GJ2.1n
EEvOZNH.LdzdIGzZevoIQIoNbFO1WIobIl6ywbBmeEzDF6QnXOkB66wTTaST
ONiSTJkGU3IAhQMRQCJrpDVtIINOK7eoszG0XbsliCVXadz2oiduNObZvNXK
JkvvV3BpgIP8D1ew2GqvzO2DLUedXOF8SmRr1JOukRbqhwzGOrMuglsAZNFF
k.9kGnN4y4Llvm4ArRhz7cQb5iKd3eMArFtJ0v7.KkFtHzQKWAMN2Z7KJI4c
nnv2oAq1l6GlYaJy01hUQ4gKAl9z4Aww5nLTxMn2Nh71QiOPdO+bw6kCMumy
rLXOLyp458njcWLjpgOGrvPyKFadz4hrKFbUNergPKIRCYGF95SokRq9PVT3
Lc5nswdwZzHEPQvahq0wAfOB1tq3dNHH+9kZG9MZzUv+qPFvyE.afNsjAYvV
bMrshlh26nKTyuvXLCKHdBOEwLtOymXG2Wn1GIDjwfuc3jvnv766lNRO.5Xr
9Nffskz6MQqBmMtzevOhdI7RoAwuCPVzKmmrzf6HAghd4cgw1+fi8knWBt7U
6uBABu0xSl88fQpfF.eRsN+k8LiWYY6z5UcA5pgP1huEElkulusCt.iLV5oD
fxizinLiUv3Vd.S7vb8hLbT5kFmzAudyZQLlfU6PeDDkrvtmbrR3A1Yk99TG
ZIbdwzoIQb65g06KcZAtTfLfhL3nw6A.E3m0nLOYTvxk0Z9I0dECE3WcV3TW
U0TXrqIbUSo52GV99zpVCRALNGPWXPSKI4CR9n0elDP0OdUXE6vxKJ.IKU2X
bKaYgkGKyo710ng9VUPL25LAl37ATtltAb0aiRl9N8r5f7njk53v35Ct131y
z2D.FnttgoW53VueoswVuYkA5uNMLHpBAtMMbVRrAHZvILMW1cuALr3LuTGY
rOQbvxVdYPfCHKcbSXxR4qxlDjZXTE1QK4TixSRhZdqp2KReSdwsWFB9szjJ
lmrr6alFd67c7tSRfatXWea6cxtdUr6tWCxD4WmE79lT67fnnBUyle9ODDGB
V+fQEcr.Jt5ltwRlmMMMIJpA95ty6a4Ny.Y7o56BmkO21Q0EFfGObYoPznJt
7rva0Y4MaClzZVyVxxu2Qzq0zpIE5vWmqWrzLQ5lOPiXRTWgstQrFsuSuddP
d9bp79oaOf1gWP62Sntr9BPqS8h4L0tk2PcNVRWdE0kmQOobrlBabGHKCLFl
EbqtUVVpFDRl13d0nj75susWSMP3VnTLpiPYswxUaNxz1zIZmzCxiB5A63nG
ERNT1WJzC5PHe7XibzvcslTCmCaez4E60v3GnWYu5qihFeWv66fJoD6jLcST
RPtYFHSBhus2xP.kx36G2iRjd9RqyeDkX+Dwts4POSxT2jjBCuhr3MCrtabu
zPOP.ITeE5tTv2utHkCiFnvJrQ7oetIy8r.vQMDAQHcQf3CurluUAkv3eoKb
QFByYe9JaYmU9NEtnCnvkiZUJa48EtrU0xWbbtRo7Zwp+milwrKfwND0jCtj
V6iXxoCsf25obaBrUGy0xBkl62NQMKYU5zRjsvQbTSHblYIgiqB.wapHalmq
WrzGJLv5ILn3mPff1WffdBABy.Ee5gBiIkO4BEpAPpXsFiMhFcnxrNfXFaGu
1FVpq+Ac7p0VJJCfV6v9jaA60QUwd3Ia8Df0pBSLibO0U0tUiXVP84DhxDWB
FLzEUZuBtPXLoTmXT7djxWjK39Xa.M3RpGWXuRQYBlMHGs7pz08Il365IruB
ycWAMQ1JXHOYDXVrHTcd35Q7vrrfIKSRqha4Xlei2aUdxsoAyB0tvwWEFh07
q0Aa0sJKvStcfnLCLXVml5ei1D2pXp+kU0i6xtYj5HsIDTM3HdJozyRNUXhR
4thKvT51gJJC9.1rBo4Wfhk9VdJL7.0wOTXLgu8GnJFVWGFaB.tthGw7gAWL
uJshQS8Tb117nFwdBzQ9wk5XzqChyPuVuHbRRzrliq0.V4dBByhiTelx2ckf
.Hv18yCQruh0uCI+GfTXSYeUkDuxCKTJ2UsRb1T1WJvNViz7OO2Utu1YQzuF
4LX5zMk8XFlskPvKgSis.dqLilw9DyId1WEXhbOOmTK02IIKj0MWdBT5dmaY
T+k4M7ac2Je0fe2hXV+GmX3E70TVa0b1Gn1oZJ9pZ+XSVTKLXmnZktsWgj+N
YtaPb.t4Yf3X8Y8.3jfEEXLHpecQWikKtfbt3qaMuqSAucXrEVyhVo0nsuZO
Vz7TlQ8up0qZ+U+rxOfs0Te3RDt4YcPRyV9eMes.5Dy+bHI6l.7eLZk9Yj9B
4sMhBwSxcCyUgD9llHmQjHM4t3iBKVC6hJaJryLV7M2GbbHAlBiL6L66wXTm
NNkPHryGR7soZ8QhEVt.y7OmGtFmwNeHveWGEkbW+wf5KY6ar4OgvGiU6cbZ
vGXOmkThhXGCP.bOwYAQWsXhN8aA+1OJFkQ8.aE2Df6FBK1Ppac+jhDoNGBM
XwAY2UxkRlegJhWg+r.CvmsWWgOCHdw2n7CLxjQHyt1k8KWGjmmFNYUtK7C0
y9oGTVZbaTxjfnhbvnht1RRb7z0ClNP4cJaL5GJB9YqYTVqYhsMYpOvzBkfc
YOESXCrJyWn73XJyjDGD2RGwHeZRRzUKVGXnlTA4Akfngw1vM2LOQ6NeCIX+
wTBmJjTvePJ7mFRBFu6byqD8gYisnHHXid0O80iraTD3MGcko+e8p34IqLN7
9OZkbwFtLT7OfHsPC47Q6hL0Gxi.nDLgGVHjR.2L4DK02kawcQbnm5zwzEUn
jzOhnsh1Db23cV3swAQ8B08wauJDTt+tkLXCFxuLJ3dSVy9wVwvxUWezznvk
y0l7vykmyUSdnSb+p1ux7+06AqcRgfdAa2mT3s0cXRWNWgwOrz0UTz5rf7fM
sqavwNh0bvD6lMSCPoMm8F8CASAY6jr4nu6Uu34+eYvLNedZxrzvaSd9quOd
5y+ozDaRT97+bzpuI4GBd9OYGBH64+McPjIQBMawtm+5Py7wP+4.XLklILhM
Gaqr9tmm6cgwNtlcYzLsT6AhRRV1bUqfQLhygQLt1jKmcDX+D.UBAdF.D2lO
uz7CdyHmtT6R1x2r0v4ke.SZGlT521lu+jfLPVXUrYHJ2F5f27ALhmEccy1M
6cvr4g2j2du+OWETlg7uA45lQa+Alljl5h7aa8wMIFmTcCfWCK1fBDsJsLuP
2l9XRM4tgAy6kkmpAHo89OcQfwcvcCj1GZ27.GSrlu+PWO8c.Os8UeY3cEgN
F8MlcaoF8SlDkVOC8p01.58dSqgoIxP68BWZM5x41MmIyk+Yep7WoKBIYBPI
ShmFjulBFpaeGCfGZJDCWi.QUONIPAiQ+bZPXL56imE99vYfg.z2El2FIx2e
noPR2dHA+Hf.08lLQqW1FwPt+8YmT5MFFTFKAOCDjh8rrneNw9a9MXha+Dy7
rIXBS4782ibYClbYClbYClbfIf2q9w+5e5zrUEnXqdpxNMCN6yfDpaYJLiDz
aGssGVucTWIg3ndQDHvjsLqhkPx84jh4mS3GUtkiOODkYAS+HhfncP.7X6gB
vJ1ru1zQUHNFwfyDF++7wtv0cmMyaDxhcSPZUhvWrYDadDpkL89oQ5OhTJbW
jI7vQlbVOjxOG1XOE4jf.leonKZiz+3LpRv1BFfir3I9LvnpYdncoPI85ijR
eSJ4BZCgJGaVUNNSfEph5XRe1oEzyR1vuQBkzTyY2C49PnBt8PoSIhg2EZi6
VBoQxx4lsXy0DBf93Y5Ozk3zmzD2lbByQWu9BDa+bIoybyVCeRfLiAl9kByd
mP5S4GeuPgYHUS98dZ3R3GBTbxjU5aps6w5VXg7H.zHmS43dqg0h.+NoZegr
j0.PlCfRKQgx6fVN1Fisrq.XYFXYi.XQ48q5Ec7q0VWKBs5ztFzhh.a6y2p5
v742RPmilflzFMr6hBjU33p9KjzJghHOjvbRGRcCk5jpaTV2COapCcEg5fou
CsLMYJLYkZardL5YDTXqqDuxuSdeePbAqkJGnhN1am4ef7zWNnLgCrsZZlm5
vvWChZMBpdvQwuirs323kIJlqRlvjrKQw+RT7uDE+CMv0ZP5Ylo.Q2QjEDGY
U9fnH1fpPn9JE0UyEMk.PSdWQkLehDarCdEx6n182mqPXlrXBLUDzB3OxPiQ
KxPubYvrYlhzLEFmz7+FDr1RlhsVtDVmkHKZiRj0tqUlaur4MrC8.3Ol1LrG
OehGWg4RyFDj3LQUuHMtMCRbVhD1NXPqh+0jvXDuqhjzdVZpq13+6U51yrYz
1f5wcoUHi3KTXroNidUQ1IbnR2hyCwy5zC5kEkVi4yewhEuHKqqEvpmU2l8R
CE3pzIzQ79LHdqR7wGuUGdywOBB25IgDQFfPR6Q9jPiFRwnM25bMIRzCv7ro
znZF9zrwmJZpOzlcrfvOJCWunukyB4orxdH6ajNM1COYAjVheHPwohVv6KGo
j0cZJ6LFTj0mkPw.E7SITP6KTPOkPAouPwIq52zWnftqBOyWzaHLr7EB1Xeo
psXJsqCrlx.G0R5eJoaUM7OQGXMKCh0QsE8O7oExSMmLQtXETdPYLwtFMaug
ZaoLCzQduNY8p7TzR4VEn7IZam5uU4BwWhsELk00ng9zTcvays8uXnO4.bG7
Teb8YE.UHWeVAPwb05yJ.2e03rBnqCK.zKcGBAD3Bc18w4yW+.thI2K0wuuV
a12w8oVsblYqHP56AN.azwFacw1dKUbXk7o87Fn8TA2Sr2MA.k4RLxKI+c+B
aL0UyAYzhCpAW4sTPuD23KwM9Rbiezk82DeqwYE6ytr+96iCq11S18AEfXGb
9eWPFHRVa4+M62B4+sa0vKNUXtj+2EDjVkHtj+2aRlb1Otj+2U1SH1cFoirb
I+uai1PnreSm+2Npf6H.4R9eeI+uuj+2Wx+6K4+8WZQntoiH0bBoknQIkGV1
NpntiQJl8PVWP8TLJlQdTTao5N2NkrCCaItHKQ785Sb4tjNmWRmyKgk6RX4t
jNmWRmyKoy4kz47R5bdIcNujNmWRmyuzSmygHIw9RIg4dDk7fexSjxGGIU5i
iDr8wQxFODId8W5oyI9EX4XlrsflI1UJbUDqnhb357kAmcU+C3dmz5e.w4nL
LB21mJAOdq+Acw0+44gYHKVhBL0m5LD3aMJuHUOPI2.HEZVn4v0EdcTlI6Ug
mIMYAJ.kEGjpsYDI52Yp56nIZ3S76G+132F+84HWEuNCkOWix.FAJKWuLCE3
ZAlXusazyPKCChSP5ODrXYjFcWXN.LwHPkRuXYtAFlnM.igEhxSP45nHzcyg
I.X6ZKLg.zv8TlZ7sdlEFHnmg9F.r0ouMlBW+mlcq9swL3p+V3hr4I4U2kWq
M2SIbsf94vkuMVV7GuF3KQyLufm4Smljkgdso3W+1XEzvq.D.PG2GvGZvdOy
GEfEr4umCfdq4IJQ1hf7tpgt9RacSVPbUaAbQVAPN85ctihlVT7nhCRwqWES
Cm6iR2YQs.etpdFcsxBFSyMxe3+KxAU3wo3CK97NxgmotZzzMaBkbPwp+.nM
lSkbyDwa6rdfS5Dupl18NtvXLoog3oyASD2biIJf1SLh8RZDR6B0PoToKA8Y
tyhaO+tnLhVoLk5ktCX7pgjqw6GMj5Pb7oVGRH7GykDBEFwQvcmPHmOEpNKG
MrST4nonJy25518ot1zXsLzpX.SNpO9ob09OSbJX5.ukIYXlGAWTRZnrcyya
WcfOfmHN1jL8lphyiYzdWc4ACyCv7y+6YMrwVWmqkSn.wwXJU3KGCZFfwBBS
54pRSLF8PTKHmf87B391xjLcalZYhQGoxgjwFKEdLtRVMLBVctP8cMLBiOvC
izVUotXKnnbkso0K3yfOlgQ.ezvIb395ezo2PaSiftep292xQtUwlP71dAWT
mspX1tDRTmAYDpeszg3D3IwnAz2zTMLEno511IVGXgMSz44vGWct71bOJAVp
n8L9pMEggSOnXZ+MKlerykZvCrJFRXGeULrvc51h1wifhXHAeJ8ZjwOukvvN
QS9oDKI9hGoUpwcUkF8OFu8ZczrGAEowgZON61Ly02qyVeOZI9CGquiDgZ6D
kQvImKuG6Za7K4CdDq+s6N229PMW3TGuYykgofCs8BkhajAycsBP0xB+sDC5
cW40idRLDcDtQdf2ENg4qWtyiruv8puHGceI6Qew4C.Ijy6MVcrcUene9MEc
Zl47GbO62GARe4.fip9v3LU23xE99v6Iu91SGMN42idhMDcjWOzlMkJ4iumT
8g34clD5snzZ7dQ3rkIgw4YkGftdlUYRRTiovD.jxpPiH7vsdmsUevGEN30G
pk2PHp0GiQdCwHGx9nmZh.7QOtAS02d5nE050fF3gXb29LrqbHLIP6AJIY6R
8gY2YPDk6Wfilk+0PqkH6iptXHDd48Z3rAXLFBqmczwhQ8YDF5YRbheJ7.gh
6SOSFBTTzGZ4PXEf2GbhLH942GbhMD8TezrHCxLW5SGssET2z.2X+4Y5jM1W
darm71d+308dway8gmMY6b6IoMl545sPho7r8Z6FJ65enJivJ21aMIESt8lv
nnpoNWOM8JC8wH2cq15e0mlM0mSHJyTosygVZuxLYZQ8Dv08NjxWhCS0FSMO
JWR83B6UJJSvrAIXiWittuvDeWOf8UXt6JnIRi4s2d7A1d57L+pmeUdxsoAy
BKxgZbiTI7pBIIS1ds4VAazBffFV9N0k7pXF+kUkGn4sy.zQZSZn0fp5ojRO
KYQgIJk6Jt.SoMiPQF7x1yf1luMEK8s7Dhzi5noJLlva9xlPrYeuqCiMhu5J
5LymRrzYZEihZiRRyOP8ssDnh7iK0wnWGDmgdsdQ3jjnY0xUk5vG2SPXVbh5
yT9tqDvjr2H9K8Qzrhs0gzYOkXZJeppjJUdXgR4tZKBvlxmRA1Q1MEtGom6J
2W5jHdVFMroS2T9gUFeKFuDlZK.WaFxLLm3YeMfov87bRcTemjnPVZ46HUHd
WVTHXQ6WLl61khQMXiTD4v0+XffkEAePVsukaGL5TMoyvZ1BCwIBUoW4UHM1
IyXCD1T2fFNDt1F.ZuTdPaErKS8qKBYrFvEjglOzXwaGJdwgaCol0fRs4suZ
GVC7TlQztp0q190dTNFWSsi9wEquyo5gzkkmUye..mY9CojkaYk9iQqzkq4c
6PUaVQIdRtyLdE.5aZhbB.vzj6hevP3Z3RToaxNQP32bevCG.wTXjDmIOOFi
5zcnljyc3AvuMUqO.HzR8L6VOlyiIyf+COv820lD+e2PW8MQ9araibgOFq14
3JfeSdNKKDEwZ+S.Tbwfh.1Dy8aA+4dvDWinH1x9EvPeBKjRpaMaP.vTmCEF
Hr21cjboj4WHN5U3qCP37Y6zEoS.B4lBYG0Skm9ed5+eTYXlR
-----------end_max5_patcher-----------

Here are some additional questions on the patch that don’t deal with things being “broken”.

Why is nmfmatch using different fft settings? (better latency?)

What would be good fft settings for non-pitched drum sounds? (would prefer better latency vs low freq resolution

Is it possible to remove outliers in the dictionary creating stage? (or is it necessary to be very careful with the training data via curation)

How can the matching aspect be optimized for real-time use? (not relying on an external onset detector?, fft settings, iterations and other ML settings which produce better matching results vs initial analysis time)

Have you updated the objects? For me there are 88 dicts in the final training used…

I’m on the alpha02 and I swapped everything out when updating it.

The matching has 88 dicts, but the training subpatch has 89s for all the bits.

Should it be 88? (so in my case 10)

Responding quickly after a long day, without looking at your patch yet

The idea (that failed in that example) was to create a non-note dict, all the noises and stuff… not worked so well so declared failure.

the idea was to create a single rank per note, then combine them, then re-run the process on the whole thing to allow the algo to tweak itself (segregate more between the ranks) - If your patch works well, you could run the nmfmatch on the seed dicts combined and that should work too…

the output to what? nmfmatch will give you activations (i.e. a ‘volume’ for each dictionary as a list)

I hope this helps?

2 Likes

it is not supposed to, but more overlap means more time precision?

Try as low as 256 fft 64 hop and see. It all depends on how different your source sounds are I reckon, but I have not tried. It is like any fft process: you will loose bass resolution, but if you are all in the top-mid then you should be fine.

What I did is to make note per note rank 1 dict, and assemble them as a huge multi-rank dict. I don’t understand outliers in that context. Did you check the ‘didactic’ example to get a better understanding of what it does? In the bufnmf you can see different trainings and their influence…

That you have to try. It is the whole point of having a creative coder on the project: try stuff and see how you get on :wink:

2 Likes

Ah cool, wasn’t expecting a reply so quickly! Some quick response reponses before going to bed as well.

Ah right, so I can remove the +1-ness of it.

I see. So it just updates the pno-dicts in place? And the audio and acts dicts are just for regular resynthesis stuff?

Sorry, I meant the fluid.bufnmf~ step where it reprocesses all the dicts. When I ran it I got 11 buffers filled with ones, but it could be because I ran it with a completely empty buffer/dict.

So the fft settings should be consistent between the preprocessing steps and real-time matching?

I’ll give that a spin and see. At the moment the sounds I’m using are all very similar. As in, I’m training sounds like “center of snare”, “edge of snare”, “rimshot” etc… There are some differences, but they all acoustic/open snare sounds. If I get this going I’ll then try it on some ‘full kit’ training data and see how it responds.

My training data is very different from yours in that I recorded like 40-50 hits on each section of the drum, and varying volumes, and then analyzed that. So it’s possible that in that there will be some examples that aren’t ideal (like trying to hit 40 rimshots in a row and missing one or two). I can go in and edit those out, but it was more a question of whether when training it, that it would be possible to give it a percentile argument or something.

I get that, it’s just some things are completely opaque in terms of the algorithm and help file (e.g. iterations). Now I can do a bunch of reading up on nmf, but I would be starting from scratch vs you guys that know a bit more of what’s happening.

Give me a day to reply - in NYC working with Wet Ink (a-ma-zing) but will try to check that tomorrow morning

1 Like

Ok played with this some more today and got it “working”. And by working I mean that I got some reasonable data out of each of the steps, and got the real-time matching to spit stuff out.

I also made a ‘batch’ version of the patch that allows pointing it to a folder of arbitrary wave files and it will do its thing on them. (patch below)

Here are the trained dicts:
drums-dict.wav.zip (5.3 KB)

I’m still unsure about a few things though.

With this part here. I’m not sure I follow on the steps here. So I processed each file to create individual dicts, then buffcompose~'d them into a single 10-channel file. THEN I ran the fluid.bufnmf~ file on the audio from all of them combined. This produces the audio/envelope buffers just fine, but the dicts appear to be unchanged by this process.

The final, and more critical part is that I can’t see a good way to get the matching algorithm to spit out the closest match in a timely manner. In looking at the matching data some notes/attacks are pretty clear, but others not so much. And if I copy the exact algorithm from your piano patch, waiting 100ms is pretty useless for onset-based processing.

As we’ve (@tremblap and @a.harker) have spoken about a couple of times, it would be great to reverse engineer the Sensory Percussion ML/onset algorithm to be able to train it on arbitrary hits and get it to work without their software (which spits out only MIDI).

I think I’ll make a separate thread for this, but I found their patent for “Systems and Methods for Capturing and Interpreting Audio”, which goes into some detail about what/how they are doing things.

It’s definitely ML stuff:


----------begin_max5_patcher----------
7273.3oc6c00jaaik8Y6eErTs6Ky1QFeCR+x5LIyj7vjcREO6N0VdlpKJIpt
YLEoVRJa2Spw+1W7AoD+.jDRhjsRFmTta0jRh.GbwEWbwAG7Ku7EKVk7ofrE
Nu14cNu3E+xKewKTWRdgWT72uXwN+OsNxOS81VrNY2tf37E2ouWdvmxUWGtz
426mu9Qm+RpeXbX7Ckui8xqJ966SCVmqeRX1RvcNHxRWJGh8ntbB.gIDw0f.
4svvk.m+dwWP7gcgwQA4pmOr3haShy25uNn50DuwjC4kuSPwUC2nJeIq94uh
BVT4imE9OTebDcInxki82ot7huOH5CA4gq8WHu4+7kuT9i6rDlhC9n3I1Bk1
6rRBG+XZx5frr+XRzlfzAAJH.sjvfPDfAnDlKmJtlpT6fPcATHifBpMnP7JK
.52W9S6CzO8EKtSzdGE3mFk7wEGeLpxonXqq7E0dA1EFE7gfzrvj3JMJuXg+
98Ut7Kp7QjP1Omn9hbu63kBi0WBb7RoAeHr7yiNdU+TAZkKfpCo51qOwHKN8
0jHP13Cgphh9hhFuhhjpYR1NmsuvDR0ZVd6S3O0Si+D8uXP4ucYdm.cgcvCQ
Iqeevlpk4EI6CD8A1mFjI5q3mWT3Od6MAa8ODkeeUyPHZow6WZka7lGMV+5z
P+ni0fGRC2jDKKD0ZJjWt7w8NGnxBR9ySUF06H1eugOrvVQfKcbyLQk7P1J+
TYK0pnfpMUKxSRhpeqietnfs4E2deXbbCTLOYe22LM7gG64ytJQbyc88cqtS
18Gh028dgQQ98Y9enNZm6GEUzYt9W+m7iC24mK7PnaBPfi2LH1WTQeLacZRT
Ts5q9Nevvc1HLxWG7wvM4OpdPUMFDu8v8kFQKN1JuI7gfr75WK2+gr5WIK+I
MnW4RGVUzI997fc6iD0h5ugZiJTsGaU2d0tduiRT2G3Z+n0GjOSm7DA55r4P
ptGRk2bTXbv5jCw405ya1IIzS0KkSD8RIPnmKGCJ7R5pGMAWwDuqQT5dDjVi
hPWTdc0fBE9Vl.jZeZXbty5Cooh2gizA6h9gBBCuDIp2Ltqrh6o7Wg.ia0mL
WUeg28TkYxioIGd3QE.jM.Bf0FCh3JV5AnttXtqGfIwDHbRPCvbgFqSCjfgX
3zzDggg506jcQbR15r9QegOrnLmvXmUG1t8XTEmcuIQOHkID1kqGdXJ5BQ3y
Ep4uYiylv04BeMN6DiZFVfTWIJw4LcGMwn1LFgwQtTHsvNCMEH1r0qyO1O5o
+QfyGeLIJn.lbNjIp+NIwQO4rMMYmyexII04mJM6tTTzEqbSQDg6SPdDHxk.
4dEA3R4SALNacW+3iAwNhfvBbhR72HQuBf8hGli5tzCKgFBBoiGcJ5ZhmMCM
Itn7oKbZI5cdccHcUAyV3zRLJ3D.LyloyiIezYme7S5A79OuXCFToAChhVxa
z+hwl.LBQaVg94CY4gaEShtL7UzbE6j+gLYDlNqiRDuXiX5QIOb4PIPG6f6R
Hlh7DXHhA4PWWQbmSgeJn6MCPlEDIvAGemsUyVw4CgD1Rkecn6RWHmIbhIhU
GRDAmIBWCni4XjAQxMCHVF0gFLaLYGCnEysHJLcPXfoHzUHZtbnYz6tgZMEp
pt7hX3QSPk1ctpyllUqo4q.8l5p7rMc0U9hnbDs0EdLZ6uvP8GgQZCccHMZC
fQF.viI.TKwtMLycV4rZn1a.QOwAlvkGBSg.NAA0iHCalQ2yoti5ntypVfZk
gWYSlLKupe672mKbJT9+CgThP55BobuFjB2ARg5EoBkF8090bAWaBDdNkYfe
fgLn3tvKJ4ZvKXGCe.6Eut0fmxNdDwHqTWWDk4ho5vimD3AwuMfmba5pUhMX
1RulSDGoy.zH6WBwrs21bAT68euibwQBiy72sOyAJB6ZnzshAcYSAQVXTgOa
iJ7up5yUjYdDDuj0bZSSRmN9MB9nSb+eaQ8UD+usXP6Ic+Ptn+nvbRLBHhhK
SeM5p5F1QnQbzLEaTnkidQwxUwgCntviq0MxaIcnZN5rMUfC5.Z1rVbjqg3f
QJ6hjv.mHPilKyERM05QFgXtC0YpYA9Do.piDcRNftHHPiOtIhBzAYA5lv.C
SZftHNPMxCXAAB5iDAXseOLqQpNLyi.K3RfM7IXPNEXIuB5gaA1wufA3Xvf7
LXPtFL.eCFlyACx6.K3dfM7O3b3fPO7PXPtHzOeD5mSB8yKgd4lPW7SvLGE5
fmBVwUAy7UnoCjldoac+g7V2b78.gk4Fmrf7luGSgAIBODK9OHxSLptKTNNG
BKBYjCIHF1Cx.x0NT3aGYZH9dGlue24sREDtYwsuXjpMxW8Q+lX7ccxtUgwA
N6D+QlyRmcYNuYu+F0xngb.p+0ppzU1nOmlE40jsJbOHm3JGe0UlUJ8hKQGp
cg1a6Bp61ERqJSOdx6jpdKZU5L3J+4uw8P7OmDF6PromCGrzsYSDAn4MJzi5
B.bwU6J2YmaOGZ2sPHxBqZBp9uagdRpHKcdy1jTw3CNO93q2s60YYV.85Y8W
rFyRL2J7Eco8.fCiuaiR7ymOfUUorFnz.D93O6Gl.WpCbFXPXpHUtMdak4su
4Pmpu+3MAepQ7tO6vJjCmSbENk3JZFvUc48RiFgyTQi.gPbwkrEx62iJnGHG
sXrsQqOOCY7.CDTnBdjuu9A5rjCoqKsCJMWbZCLhngyCiONEs2cx0lC7B8Yc
wkPv4VBAybITFKp0kPowxrWBImSIrr5LqkPTgokUkPxyVIDcNkPzyQIDeNkP
xLWBgmSqLZ39xUtY0zZsPl2gM2qy+x8944ogqNjqcgZJmdWzb+eHJYkeTwL5
ON0jNROvKaWAl1LwpYoxmc1jdPLkSQvuCsBPPJZIQsfOXOzRDGP87vLJBqun
X3zIXsyn8u1YGiSdVWT+xThH9UjL6hViaD9RFDxoTLGBzKdlGcBVaHJ9VYAY
WI5etxQsw5bNt85raOt3sjIWnQHTXnQ4.LTZ3cU1XzNPKjU7FoXcZK+SUsow
tFbdM+D+da3mrCMgL2kdbFgC7HbDSSeeMChGYaOr2sgsmLe5ZlvpXmkkvDhr
D2jwldSBo.52n6YfnRp7I5DY45bScW1j8DzofTWn9W6RcRPu6FkjITDaIRzm
S58RkOOvTXJg42Z7KIKHxYnvIPbxR2Nilf3tDeMqyaGPkcLDbNwIa4PIjtjx
HLuiNusxV5rAH3MFBo5eOHeAN1eiCZtUH4VfRzNQIRG83f1ziq7eyYOuTQgI
vovqnk8+LDUJGNA1V0VWgNWVmQDmNrKH9vflN5b+gV5QkCn45QnZm0HFVCP1
6tNLOXWQZ3V.fNeky2DHsdW9Q+OHqc2I+A.It9eXyCA0uJVb0eJbW1iI4F+T
jJ2u8mlpuqyeIbe8avJtwaeL4fjh50uKWVDSSxxbdad352W+ltha9s961Grw
vCzSbS0mQVfpcKHPdqGChhTWtIvML4jgd0axREybV.G2qW69lD7pHHX4G9G7
WK2ccYO578e6qe0+cVPZ1qRS1jF9Pxqd6Swqe0Olln3awq9iQG9ljev+U+nZ
t4Yu5mB7ijqdjS7tsupjBahJet+qrtidsJp+g7j8I6OTtD9UMRjeds84e8q+
e9CKlKdWsJHXuktQMwIOBcJnjGcwW3UkM7pBo2yHXjdC3o2Gdxs53WHV0WHV
0WHV0sIwpDcsy7eHnS9J7s+4+q+fMKhomZK.4h05GvXPNgdVcXxuNXMkErNe
.7DJBuiI9OwrpDg8Q5k44i3RACuAXkzF+0e1A5fr.rv5stflPuT5XX70C3vw
2.nyu6yViKFsg7nlnv+H1EkOLANxBeHVLj6MA+FeZcTvmcbcAVCqZWcZUaXJ
wQvMENNzvER6MGpCvgZCRBg.oYXATRmXnj48qigMjqH8msG8fHrh+PDLEPcK
1taXxDyOPF2V6x1KRvs.a1zXGBOKrYiSdtXI3rw.KN4bXOjmw2bR5F8LqgyJ
cNNqhNo+RNXdK4mCs233aJP+rK5vaFT+bnxWY8bdIan24zYj+bTB4mKFN2D1
7bbJ.+BEut7kWTMK4sN60RvsiL2GBv7qfhe.cj+7eeil+WekT2Hyp9Za2Cyb
1RnH1HOpmGqXwivXzTnAGzaCdkrM5P3lkqNrUD.w9jrZgS1qVk.cWBaHel.2
of.NzgWB24SnIV+9ileZlFpr+BcfVBaF0oCnU6Bb14hbd2XVXw619Ym27Xxd
Y9vcXDm27wvX0qQTlya1tM+zenEPXgyKEWNcdSpe76Ecwey1vHwcDeYpd+Wp
h1cr0f6tTtUJP.Dj3VvpQfmosL30poLyrQbyY2XTIbcIUDvKySsoyo0XW0lf
rPpFp7NpNKldEdu1aPmYBzJILFe5vLrM1Jc.ZCJjasmK3H.bs1ASF2scbrlH
rBqN...E85JIRlM334KXv0F3nJHYTTCmDKJq.lR4AY1.F2t.FSJeXuFLmN9I
ZkvfSPko3pq.Xshmt0TR5Z+RTOV5datN2xP6o.2Ug.NkkBnkkBYzjSVoPplL
VUJHtSYon4D66pAgMgEBpskh11v6B2rOQLfWQmDDkdTSvoKk6WE.CBvkwRyX
5PBMbyoppYqEOcRAXTyMcVW4hvcHDFqW4JrVHdQZkmr9eww5Ay8PG+qordQr
oODcpQWaLekLBsQyPSzEnmUrqV8aKVcqZ+EUKsCXArplaMCNsvqUtqQSM7hs
wc8ftGHZjTdhT3JY.LjB.HBARTaankTi2ZhpVDq8LLkfKw1lXu56yyQsPfsM
1DNdBgBL2xRAFN0kBabl.Y0SJ63VJX11hLkNVw1FWfZXqICKn1Ft3jZWX6P3
bzTWJroEobIC5wWLTO.FmPaJOyxPyF7dT4ZG2PXLYbbG2aJwCvnL1DVLQUUc
PtSmpWmQnkp4s1ysPdvV6MA8HZsuyTgEVOspoLTWj0dw4SYovV+mnIuTX0r6
ZnyDiaofXIVLolE1FsEgNXPjh9U5cxJaonSKDSnmxyMUm0IC2appX15ApLHq
ooT.GM+fxyTnB2VPgutiyc+j2rJWdJqNvwo5P.O+UGaClaJCfQAU1foT7TWJ
rILplxFy3VJrc3ALapKEVk2hoLjRnsgWimxoZ.gm4j+5tGeYRFYZcHgq2bwM
9K.sRJxp+WkI3Ah0CyzYBdJtXIeVFhKK+xkb7I2O2UZxaEcg6LOY3qeBJ05b
Vq1hsYb8nMtVzE.6RLkC4b9IECnwYSmwU8w7J9X3by7Lqp0YuckMjcMMBx7o
Em97fRuCKItWxYcOrc03nJi1ISFFoZXgdqpjhIL5N0I4n7g47QgQ6cNeLML2
lpua6SwXOzXgEtiLVTi8FcnoV03QUO0bj5TBDxYbJuP1egv9q3lskQsq3tnN
q3ljNqQp6M.9ZYjdmPcS6zCl1nGujv8.fxsYLq0oB2E1Kl4cI8h26e5H9U7V
Oxe2IqdjJb3sQuqNI1V0NtuUJJgqShz6f42o1YkU9ArxiWtojq9EJZ0K11q7
i6GyEqdn1WljSebwTbfXNhR7tyxKU8wJOAuSROtgiWRufVEwXc4GWi8taDXD
kXYeZHghykHx4YLYv+AFe5g2obDzKKkNWesxiu6rnvJCOZ5f.1UuUyj+BHr8
.T3QeHXjdWNRp5DoQya0VM.5NqtBbHvTRKRQjUqBiByep1M7i27j3G5MScIn
kEjK9RDU9JFvM61XvG2wMntg9GTaZrDO2iaVXXmCNTUXGVH2n9gxVzW6fGuw
Lzr8Kd21cxF3O67FwGRSdOvEw8upT9yv3P1FkUosEFtjwcEiK6x357RVXYUk
ueWX2JHuyQmhByxG0gl2Wvfo9FPVd3cICKuqc6pQdpMr+ZHn9.gmj3hx316P
ZKLIqEkyzwfbVXPJKLKiE8KgEljuhiDqe.YqnqiBHZwo5KmUgXVFUqhATphg
TohdUnBKTmhNTlhgUkhdTjhdUihdUhhdTgh9UfhdUehATdhgTcBaUbhNTahd
UZhtUYhtUXhtUWhya9vsTTBCpIwfJIQaUjXBjrW093.MDIII5HSc4pnhHbDj
UxedW1DHCgPzykT81dmS207u6L.R8bP0GXYD2qQ7y5hjxy3dK35wiBiG8d3c
RviaDEJt1L1uW3vx4aUu5qKjuL6fIA1zpOFzkNAcxbo2hcxNiT.M.TR0yZwC
MAlbt2h1bekubi2YwAvaA.oy0LDOIFWjeiZbU3d2zPgSioFz8lzT6vlvDar0
J2.mlbqQlBY9+pU4+IYeU3ZKyBbmxUT7VfiE2D.Az8pQhSlIpX66vN4zTCkc
XdqZBZ2+C0zR1xoRZtnu5AgSpni4Zyz1Muz7dg9cUSYHqliNI6tftxLwoxBG
S8JY53nMksA8mCV9AIGSgGgg3Dp5UtRBInRqmgOJ5zyD.8zOIfmKfnek3RvZ
4+6EclG4WXL0uXuZetC4IOj5uIrXuTALt83uqh0Rp3cZRf+zYKs52gIqsiMp
+oCq8ssg73VorVaBGAXdp1DgeQjFOcA.HoE7zJg5.4lXV9Ab8nDNW8JBxiqZ
dnrZeVoWb0m99vXYZfBN19f8DdjkehS4oEoxSqoBf8lhGaN5wZ7LrLpaO5dz
JzUxdSW8qLVnaZOxn.MbK0ZMFW+J821rXNVay+UAMHbpHPLE56gEMn5VDnv3
nc6f+50hu4Zebb4BnfIkUPCqfhN4GQA6Z9w4tBrPCn.nqq9UhuIDp8muZBuD
NI+y6ChcdqeblyaC1EtRd7XT0c4Dzo685Ey3u9Xsf05uyWkZKrX0rN8iIt7J
BvhUSNdGxge2EVQ+Z65V20R20k8itKzQeA7hdjyC3zXqeacKonWiXLHjWUqc
o2KBENWE8VS1n+B+k0dMN9cq38rzyW6WMf2StTJuUdGZ+JyezeUEGP6dpmuE
Qycc9YXMqZ+qDqk.mvdygkrdVe+9nCAeEz1RtoAgjm1Q5gTOVITGTTvYrRjl
7w3qpVbprSO5SAOy0hu4I+qqR.PhXB0tx4XLR2GGIOmZmuJw2kFDbk0BUqfT
0Lw5HjkA9MeUf+2fnnjOZeMn55H9N0JIR8..2AG6UDmGW6IE5BUiAPEsdzYo
hdX2pfzuSD01U0PoNM3TlaTQHDTUsAV069jVIR0AD9c0NZ1Nifrrn72UX6LB
ig8J5cwKlDlnsya75m8ab96hW57CEYMrO9MBzK9OlpRiH1i5xI.DlHCRUmMa
7fzIp553CsmwEGYVR092H5QhtUiA.eePzGBxCW6eITFr5gLTODoAB7VhfDDk
gDwsgD+Iw3Y3kYTn5AKz29iesbCaCXfxibm2dH9wjCx.SMSZJrIDhcQTZq9I
dy3QOm+iiBBWeXHU.WXJGPoLVgZqi7Zp15WHGgIcyW5SGfWiSkUmUojzO6fr
pZ6AZmGdDwqeSGr0rvBzcMul9metboMx+IIMt9b+tHT6gJwTFZ04.qEECUej
ygnYTSUQ7PUw6L+J4+j7mS3Q1O8oScvVGEt+w.IOZzrP7DEZ23m62zut7c2Q
tl8WkkDIFKP.NJ1qr3sgxIR47G8ECFTeInUzD6nSqAdeuOLVW+UK5i7JUdCQ
II6quNMBW8w4BW82KoiTGIjOIM7gPAtHoeRRY7RMUZq+uC9kD+Tw.1rv00Ee
M0kxODWvjx24PH0+FJeLh55C4OV51q4yQZcUbmZWOaefltRuqUP.6CE1eqSR
SKTGOCe5sIxnG0irVoZ17MktyON27CQxoor7z.wipmOa+khRpZaF.yhNjVxb
p13RczqRP4KTmVaKZiHYOFtsZkousizHFCAZoy2jFHL1b9QIW+B137sm5m0y
t+QqifHhRphJ1zOSUTDvYKJhtYs5oisstY9NUunrD5krsQLTy8X+KHgU4T8Y
qlVbOc0QlwgegvpegvpegvpWH2cZbDmYjWSZcQSSIUaHWxMM0K6SKrKNxxZO
bWsisrdvHnHh31GYYSxIyIblNWRMbTj08ozKBv69XHyVijtNJRwyTM928Yqp
qFaqMezhckcO38eXV25HvZhOvuMcbg0MLocavXSfaiZGOXyCtzmqzhURt0A.
l4oUyVdjl+b5D.NraDsxu4A5UOnADwr8v75JOnyqc.dcFGdWykhrqwCDaxTj
cN4JTj8mWszlaMcKaqOPsOumFWws11hFo6RF34U1sMnR3SMncVEM3rhZfaAw
ahYqBsxlR8riwOGr.9LS0ZXODL92DK7mkJt.sPSqGKcVviOy5rvPKdGrHcib
PCok34ak6busV3tbmUNqFzPw..BYikF2zsJBnLLtJItwxNBEZhvn0Qv0cl6H
zU9mac9co1ijMNCu5FWHP3RNTDwcoL9Z5j6x3o1kEXj27HpS6cjo8qOcXAhw
ZugDaxBu0qHMuWUq5ewUTBrduZiY3ujf9ujf9ujf9KMmzABqmMNYCmz.WnJ6
IPj77+SqwTRwARxKEDC6AY.oN2dmCeJN7Oo2Hac60I6VIlNgyNwej4rzYWly
a16uYi.qbPhgEk+CeoGxhlfX40jHL2CxIt.wnntxXmnCehKRO2rVU+XzqCug
CKSVsWXbs6v4oA5P7OmDF6PFxZVFOZSnlniQEC8nt.fTG0jmZqWi0Lsiy1xA
VQp6Z7u4x5VEmiyaJ1v8O93q2s60YCcR.SAGYokFCG+7oBsTWBl+Topq7Dvj
kIUFXNxj5jlsYNbpwH3XiQy9IZowQW4L0nqxM.RwkljS2RFZwMYl3YVevCPl
vTBx.mSo.7beVCxlzSTLqOWylRc9GQrTa6ISdo.YaoXRO.Vv1VJlpC4MnssH
8bxK7a7cLCf7Z.boGDaQZjzjfZzTBa9LoD1WVw+RD.a2+US.r6U5fWcXa7ts
e9hDK3tTKXm2bTBhSCxdJN+wSuAsJR8lf3OT4ZpOi9q5v9MRVcCOWEGtHiwT
TyndH5CViQPvgwKdFVBIsJ+2Rbxe9VAIB+YZEjT6a09Vy.Oy53NELRKUDhdQ
07qaIj55byPN.Ts9b+avgwFwO8P0jiaHBNRq2.BLxKnhTf9jIS3yCWuDS7Q0
GW2Yg6Uoxr9Q+3jsakonrl9saRYl7TxkkZmAKvHsXFz9RXZsqUSOrLlDsFXG
yPeJ3vGzF87BkbmWSr40J22wfTppj6UZITu85S5R2vzLDthlmVgt4Ui2Ecc.
AWQP0ZYBX6Sx0yhmDdLdPtV7fb4iwSxFvy0ardRvAdRb2w3IwrsNQtxmD2l1
I9XfdDOKPOWzX8jFpNwFk5jM1dkU7q6IQr3Io1DwWckBissRc0OIab6UGhqS
TwI8IycqYv17TJUsK.HdjkhHApsY3gLWy2pck.bUUBD0hZABLBMTHfMlDvQ3
IAsZbiwvKAzl9tzw3AQlspDvlwMHjQ4QYkCI9n7nvV66CNJOKnUOKTsmkNHv
FL6Q9XZvnmFr4oMSd5lEOMYviJMcMzPWc0c.sysgB+7KurKwxqLAel0I2KQi
buP8w8B0FWy5gW2ZgWG5fWYRHu6kcKBQ0Efnp1dFz7VyM.WiN2doZb6UqusV
Y1zTaEuLEN7B0x1KSGaGMSmimHamul0d45U60nUsVqSsWcGh15Qq4NFVnAsW
cYolBV1kCRqzW1yWqRuDckcTpvUnNvfHucZF6nTrpkrNyEryGiub+Tms9udg
Z+5E5UbNFiqduC6ZEqx4BKrtFRCWuZKq1Z0p4R0kpOqiUArpNrZcIzJsWcjJ
gU0XUqKfVoqpiTArl9oZeIb.MScjJb00FUyktAzC0KTKTu9JPKMO0Zv0JcN8
pKfszyTKFH2ZML8BzuzyqBomBYG6DiW9Oe4+O3n8TL.
-----------end_max5_patcher-----------

ok now I get what you try to do, and I don’t think you will find in the first toolbox, one of decomposition, the machine learning tool for dynamic fast browsing and mapping… As I said in the other thread, we will consider this (black box, copyrighted) object you are trying to reproduce, but I would have a few creative questions for you:

  1. what are you trying to achieve in reverse-engineering a black box that seem to do everything you want it to do? What does it not do that you want to do?

  2. what would you need us to explain better/differently to streamline your learning process? @weefuzzy will soon plough on a refactor of the KE website but it would be good for us to know where you stall. I am sure he (and I and @groma) will study your great forum entries, but if you could voice your pending questions that would be ace.

  3. for your commission, are you not trying to plough a field with a Ferrari? Or to race with a tractor? Anyway, the mandate is to use the tools, with their different features, to stretch your world… but you seem to start from a task which is not completely adapted to the provided tools… 100 ms of delay is only a problem for RT percussion. You can do soooooo many other things with this set of tools! Allow a bit of divergence and see where the latency leads you?

Thanks again for the candid feedback on your usage progress (and the bug reports - you are catching cool exceptions :wink:

2 Likes

Yeah, that’s where I’ve been going with this.

In a general way, it’s very limited in scope. As in, their specific hardware driving a sampler. So being able to do much more than that would be great. So things like (fast) “onset descriptors”, more broadly useful training and matching algorithms, with some openness and variability. (as in being able to feed it any (real-time) audio and have it return what it thinks it was, along with the relevant distances).

Additionally, because their whole thing is a blackbox, it means it’s not possible to use it in another context (easily), without having shit/MIDI resolution (and presumably additional latency).

Yeah, would be happy to talk about that, and the pedagogy of the toolbox in general. The helpfiles, not surprisingly(!), have been a bit difficult to understand and go through, and with many of the examples I have only understood what’s happening after speaking with someone. The examples are also geared around a specific working process.

My initial idea was not this at all, and actually I still don’t know what I plan on doing with it. From what I was initially thinking (and mentioned at the plenary), the toolset isn’t nearly far along what I had in mind (I had some database stuff in mind).

Most of my ideas are centered around real-time use as that’s a big part of my interest and what I do. The pure (and offline) decomposition stuff is interesting, but doesn’t especially appeal to me as I don’t really “compose” in that manner. Even when I do compose, I don’t work with “sonic materials” that way.

So at the moment, this is just me exploring the tools and seeing what’s possible, with a big slant towards real-time processes. This quite likely won’t be possible, and therefore won’t be part of what I do, but may trigger something else (100ms later anyways…).

( tl; dr: Part of the issue here might just be that the STFT is a hard thing to get working with percussive sounds. But all is not lost )

So, you’re trying to make a classifier. That is, indeed, a kind of machine-learning task, but for these purposes NMF is a kind of machine learning. Whether, in combination with the STFT, it’s the best thing to discriminate drum hits is another matter.

The way the workflow above works is to use the first passes of bufnumf @ rank 1 to create sort of spectral summaries of the things you want to match on. As such, any of the temporal dynamics are sort of washed away, and you’re at the mercy of how well a species of drum strike can be represented by a summary of its magnitude specturm across time. My hunch is that the answer might be ‘not very well’.

So, things we can ponder:
– would a different underying representation of the signal work bettter?
– would you get better differentiation from NMF + STFT if you generate your dictionaries just from the first few milliseconds of a strike?

(As an aside, in Chris Kiefer’s echo state network code, that I made a slightly crappy Max version of a while back, he demonstrates making a drum classifier by training it on raw audio. I suspect this apporach wouldn’t scale hugely well if you wanted a biggish vocabularly, but it’s interested nonetheless. Looks like the only way to play with it is to build from Chris’s repo @ https://github.com/chriskiefer/Fecho , but if you fancy, I’d be happy to compile the PD object for you to experiment with)

2 Likes

That’s good to know. I was hoping to get some good mileage out of that approach, and it was useful to patch around that, but it may not be the direct for this.

Curious what this means? Like some sort of pre-cooking, or something else?

Interesting indeed…
I might try and chop up some audio and train it on that, but is there an upper(/lower) limit and/or sweet spot on how many examples to feed the algorithm? In @tremblap’s example he uses 3 notes per dictionary I think, whereas I went with 30+.

This would be handy, if you wouldn’t mind. It may be (finally, and after much procrastination) to watch the Kadenze videos on ML as I’m not really up on the different algorithms, approaches, and their respective merits and strengths/weaknesses.