Training for real-time NMF (fluid.nmfmatch~)

Ok played with this some more today and got it “working”. And by working I mean that I got some reasonable data out of each of the steps, and got the real-time matching to spit stuff out.

I also made a ‘batch’ version of the patch that allows pointing it to a folder of arbitrary wave files and it will do its thing on them. (patch below)

Here are the trained dicts:
drums-dict.wav.zip (5.3 KB)

I’m still unsure about a few things though.

With this part here. I’m not sure I follow on the steps here. So I processed each file to create individual dicts, then buffcompose~'d them into a single 10-channel file. THEN I ran the fluid.bufnmf~ file on the audio from all of them combined. This produces the audio/envelope buffers just fine, but the dicts appear to be unchanged by this process.

The final, and more critical part is that I can’t see a good way to get the matching algorithm to spit out the closest match in a timely manner. In looking at the matching data some notes/attacks are pretty clear, but others not so much. And if I copy the exact algorithm from your piano patch, waiting 100ms is pretty useless for onset-based processing.

As we’ve (@tremblap and @a.harker) have spoken about a couple of times, it would be great to reverse engineer the Sensory Percussion ML/onset algorithm to be able to train it on arbitrary hits and get it to work without their software (which spits out only MIDI).

I think I’ll make a separate thread for this, but I found their patent for “Systems and Methods for Capturing and Interpreting Audio”, which goes into some detail about what/how they are doing things.

It’s definitely ML stuff:


----------begin_max5_patcher----------
7273.3oc6c00jaaik8Y6eErTs6Ky1QFeCR+x5LIyj7vjcREO6N0VdlpKJIpt
YLEoVRJa2Spw+1W7AoD+.jDRhjsRFmTta0jRh.GbwEWbwAG7Ku7EKVk7ofrE
Nu14cNu3E+xKewKTWRdgWT72uXwN+OsNxOS81VrNY2tf37E2ouWdvmxUWGtz
426mu9Qm+RpeXbX7Ckui8xqJ966SCVmqeRX1RvcNHxRWJGh8ntbB.gIDw0f.
4svvk.m+dwWP7gcgwQA4pmOr3haShy25uNn50DuwjC4kuSPwUC2nJeIq94uh
BVT4imE9OTebDcInxki82ot7huOH5CA4gq8WHu4+7kuT9i6rDlhC9n3I1Bk1
6rRBG+XZx5frr+XRzlfzAAJH.sjvfPDfAnDlKmJtlpT6fPcATHifBpMnP7JK
.52W9S6CzO8EKtSzdGE3mFk7wEGeLpxonXqq7E0dA1EFE7gfzrvj3JMJuXg+
98Ut7Kp7QjP1Omn9hbu63kBi0WBb7RoAeHr7yiNdU+TAZkKfpCo51qOwHKN8
0jHP13Cgphh9hhFuhhjpYR1NmsuvDR0ZVd6S3O0Si+D8uXP4ucYdm.cgcvCQ
Iqeevlpk4EI6CD8A1mFjI5q3mWT3Od6MAa8ODkeeUyPHZow6WZka7lGMV+5z
P+ni0fGRC2jDKKD0ZJjWt7w8NGnxBR9ySUF06H1eugOrvVQfKcbyLQk7P1J+
TYK0pnfpMUKxSRhpeqietnfs4E2deXbbCTLOYe22LM7gG64ytJQbyc88cqtS
18Gh028dgQQ98Y9enNZm6GEUzYt9W+m7iC24mK7PnaBPfi2LH1WTQeLacZRT
Ts5q9Nevvc1HLxWG7wvM4OpdPUMFDu8v8kFQKN1JuI7gfr75WK2+gr5WIK+I
MnW4RGVUzI997fc6iD0h5ugZiJTsGaU2d0tduiRT2G3Z+n0GjOSm7DA55r4P
ptGRk2bTXbv5jCw405ya1IIzS0KkSD8RIPnmKGCJ7R5pGMAWwDuqQT5dDjVi
hPWTdc0fBE9Vl.jZeZXbty5Cooh2gizA6h9gBBCuDIp2Ltqrh6o7Wg.ia0mL
WUeg28TkYxioIGd3QE.jM.Bf0FCh3JV5AnttXtqGfIwDHbRPCvbgFqSCjfgX
3zzDggg506jcQbR15r9QegOrnLmvXmUG1t8XTEmcuIQOHkID1kqGdXJ5BQ3y
Ep4uYiylv04BeMN6DiZFVfTWIJw4LcGMwn1LFgwQtTHsvNCMEH1r0qyO1O5o
+QfyGeLIJn.lbNjIp+NIwQO4rMMYmyexII04mJM6tTTzEqbSQDg6SPdDHxk.
4dEA3R4SALNacW+3iAwNhfvBbhR72HQuBf8hGli5tzCKgFBBoiGcJ5ZhmMCM
Itn7oKbZI5cdccHcUAyV3zRLJ3D.LyloyiIezYme7S5A79OuXCFToAChhVxa
z+hwl.LBQaVg94CY4gaEShtL7UzbE6j+gLYDlNqiRDuXiX5QIOb4PIPG6f6R
Hlh7DXHhA4PWWQbmSgeJn6MCPlEDIvAGemsUyVw4CgD1Rkecn6RWHmIbhIhU
GRDAmIBWCni4XjAQxMCHVF0gFLaLYGCnEysHJLcPXfoHzUHZtbnYz6tgZMEp
pt7hX3QSPk1ctpyllUqo4q.8l5p7rMc0U9hnbDs0EdLZ6uvP8GgQZCccHMZC
fQF.viI.TKwtMLycV4rZn1a.QOwAlvkGBSg.NAA0iHCalQ2yoti5ntypVfZk
gWYSlLKupe672mKbJT9+CgThP55BobuFjB2ARg5EoBkF8090bAWaBDdNkYfe
fgLn3tvKJ4ZvKXGCe.6Eut0fmxNdDwHqTWWDk4ho5vimD3AwuMfmba5pUhMX
1RulSDGoy.zH6WBwrs21bAT68euibwQBiy72sOyAJB6ZnzshAcYSAQVXTgOa
iJ7up5yUjYdDDuj0bZSSRmN9MB9nSb+eaQ8UD+usXP6Ic+Ptn+nvbRLBHhhK
SeM5p5F1QnQbzLEaTnkidQwxUwgCntviq0MxaIcnZN5rMUfC5.Z1rVbjqg3f
QJ6hjv.mHPilKyERM05QFgXtC0YpYA9Do.piDcRNftHHPiOtIhBzAYA5lv.C
SZftHNPMxCXAAB5iDAXseOLqQpNLyi.K3RfM7IXPNEXIuB5gaA1wufA3Xvf7
LXPtFL.eCFlyACx6.K3dfM7O3b3fPO7PXPtHzOeD5mSB8yKgd4lPW7SvLGE5
fmBVwUAy7UnoCjldoac+g7V2b78.gk4Fmrf7luGSgAIBODK9OHxSLptKTNNG
BKBYjCIHF1Cx.x0NT3aGYZH9dGlue24sREDtYwsuXjpMxW8Q+lX7ccxtUgwA
N6D+QlyRmcYNuYu+F0xngb.p+0ppzU1nOmlE40jsJbOHm3JGe0UlUJ8hKQGp
cg1a6Bp61ERqJSOdx6jpdKZU5L3J+4uw8P7OmDF6PromCGrzsYSDAn4MJzi5
B.bwU6J2YmaOGZ2sPHxBqZBp9uagdRpHKcdy1jTw3CNO93q2s60YYV.85Y8W
rFyRL2J7Eco8.fCiuaiR7ymOfUUorFnz.D93O6Gl.WpCbFXPXpHUtMdak4su
4Pmpu+3MAepQ7tO6vJjCmSbENk3JZFvUc48RiFgyTQi.gPbwkrEx62iJnGHG
sXrsQqOOCY7.CDTnBdjuu9A5rjCoqKsCJMWbZCLhngyCiONEs2cx0lC7B8Yc
wkPv4VBAybITFKp0kPowxrWBImSIrr5LqkPTgokUkPxyVIDcNkPzyQIDeNkP
xLWBgmSqLZ39xUtY0zZsPl2gM2qy+x8944ogqNjqcgZJmdWzb+eHJYkeTwL5
ON0jNROvKaWAl1LwpYoxmc1jdPLkSQvuCsBPPJZIQsfOXOzRDGP87vLJBqun
X3zIXsyn8u1YGiSdVWT+xThH9UjL6hViaD9RFDxoTLGBzKdlGcBVaHJ9VYAY
WI5etxQsw5bNt85raOt3sjIWnQHTXnQ4.LTZ3cU1XzNPKjU7FoXcZK+SUsow
tFbdM+D+da3mrCMgL2kdbFgC7HbDSSeeMChGYaOr2sgsmLe5ZlvpXmkkvDhr
D2jwldSBo.52n6YfnRp7I5DY45bScW1j8DzofTWn9W6RcRPu6FkjITDaIRzm
S58RkOOvTXJg42Z7KIKHxYnvIPbxR2Nilf3tDeMqyaGPkcLDbNwIa4PIjtjx
HLuiNusxV5rAH3MFBo5eOHeAN1eiCZtUH4VfRzNQIRG83f1ziq7eyYOuTQgI
vovqnk8+LDUJGNA1V0VWgNWVmQDmNrKH9vflN5b+gV5QkCn45QnZm0HFVCP1
6tNLOXWQZ3V.fNeky2DHsdW9Q+OHqc2I+A.It9eXyCA0uJVb0eJbW1iI4F+T
jJ2u8mlpuqyeIbe8avJtwaeL4fjh50uKWVDSSxxbdad352W+ltha9s961Grw
vCzSbS0mQVfpcKHPdqGChhTWtIvML4jgd0axREybV.G2qW69lD7pHHX4G9G7
WK2ccYO578e6qe0+cVPZ1qRS1jF9Pxqd6Swqe0Olln3awq9iQG9ljev+U+nZ
t4Yu5mB7ijqdjS7tsupjBahJet+qrtidsJp+g7j8I6OTtD9UMRjeds84e8q+
e9CKlKdWsJHXuktQMwIOBcJnjGcwW3UkM7pBo2yHXjdC3o2Gdxs53WHV0WHV
0WHV0sIwpDcsy7eHnS9J7s+4+q+fMKhomZK.4h05GvXPNgdVcXxuNXMkErNe
.7DJBuiI9OwrpDg8Q5k44i3RACuAXkzF+0e1A5fr.rv5stflPuT5XX70C3vw
2.nyu6yViKFsg7nlnv+H1EkOLANxBeHVLj6MA+FeZcTvmcbcAVCqZWcZUaXJ
wQvMENNzvER6MGpCvgZCRBg.oYXATRmXnj48qigMjqH8msG8fHrh+PDLEPcK
1taXxDyOPF2V6x1KRvs.a1zXGBOKrYiSdtXI3rw.KN4bXOjmw2bR5F8LqgyJ
cNNqhNo+RNXdK4mCs233aJP+rK5vaFT+bnxWY8bdIan24zYj+bTB4mKFN2D1
7bbJ.+BEut7kWTMK4sN60RvsiL2GBv7qfhe.cj+7eeil+WekT2Hyp9Za2Cyb
1RnH1HOpmGqXwivXzTnAGzaCdkrM5P3lkqNrUD.w9jrZgS1qVk.cWBaHel.2
of.NzgWB24SnIV+9ileZlFpr+BcfVBaF0oCnU6Bb14hbd2XVXw619Ym27Xxd
Y9vcXDm27wvX0qQTlya1tM+zenEPXgyKEWNcdSpe76Ecwey1vHwcDeYpd+Wp
h1cr0f6tTtUJP.Dj3VvpQfmosL30poLyrQbyY2XTIbcIUDvKySsoyo0XW0lf
rPpFp7NpNKldEdu1aPmYBzJILFe5vLrM1Jc.ZCJjasmK3H.bs1ASF2scbrlH
rBqN...E85JIRlM334KXv0F3nJHYTTCmDKJq.lR4AY1.F2t.FSJeXuFLmN9I
ZkvfSPko3pq.Xshmt0TR5Z+RTOV5datN2xP6o.2Ug.NkkBnkkBYzjSVoPplL
VUJHtSYon4D66pAgMgEBpskh11v6B2rOQLfWQmDDkdTSvoKk6WE.CBvkwRyX
5PBMbyoppYqEOcRAXTyMcVW4hvcHDFqW4JrVHdQZkmr9eww5Ay8PG+qordQr
oODcpQWaLekLBsQyPSzEnmUrqV8aKVcqZ+EUKsCXArplaMCNsvqUtqQSM7hs
wc8ftGHZjTdhT3JY.LjB.HBARTaankTi2ZhpVDq8LLkfKw1lXu56yyQsPfsM
1DNdBgBL2xRAFN0kBabl.Y0SJ63VJX11hLkNVw1FWfZXqICKn1Ft3jZWX6P3
bzTWJroEobIC5wWLTO.FmPaJOyxPyF7dT4ZG2PXLYbbG2aJwCvnL1DVLQUUc
PtSmpWmQnkp4s1ysPdvV6MA8HZsuyTgEVOspoLTWj0dw4SYovV+mnIuTX0r6
ZnyDiaofXIVLolE1FsEgNXPjh9U5cxJaonSKDSnmxyMUm0IC2appX15ApLHq
ooT.GM+fxyTnB2VPgutiyc+j2rJWdJqNvwo5P.O+UGaClaJCfQAU1foT7TWJ
rILplxFy3VJrc3ALapKEVk2hoLjRnsgWimxoZ.gm4j+5tGeYRFYZcHgq2bwM
9K.sRJxp+WkI3Ah0CyzYBdJtXIeVFhKK+xkb7I2O2UZxaEcg6LOY3qeBJ05b
Vq1hsYb8nMtVzE.6RLkC4b9IECnwYSmwU8w7J9X3by7Lqp0YuckMjcMMBx7o
Em97fRuCKItWxYcOrc03nJi1ISFFoZXgdqpjhIL5N0I4n7g47QgQ6cNeLML2
lpua6SwXOzXgEtiLVTi8FcnoV03QUO0bj5TBDxYbJuP1egv9q3lskQsq3tnN
q3ljNqQp6M.9ZYjdmPcS6zCl1nGujv8.fxsYLq0oB2E1Kl4cI8h26e5H9U7V
Oxe2IqdjJb3sQuqNI1V0NtuUJJgqShz6f42o1YkU9ArxiWtojq9EJZ0K11q7
i6GyEqdn1WljSebwTbfXNhR7tyxKU8wJOAuSROtgiWRufVEwXc4GWi8taDXD
kXYeZHghykHx4YLYv+AFe5g2obDzKKkNWesxiu6rnvJCOZ5f.1UuUyj+BHr8
.T3QeHXjdWNRp5DoQya0VM.5NqtBbHvTRKRQjUqBiByep1M7i27j3G5MScIn
kEjK9RDU9JFvM61XvG2wMntg9GTaZrDO2iaVXXmCNTUXGVH2n9gxVzW6fGuw
Lzr8Kd21cxF3O67FwGRSdOvEw8upT9yv3P1FkUosEFtjwcEiK6x357RVXYUk
ueWX2JHuyQmhByxG0gl2Wvfo9FPVd3cICKuqc6pQdpMr+ZHn9.gmj3hx316P
ZKLIqEkyzwfbVXPJKLKiE8KgEljuhiDqe.YqnqiBHZwo5KmUgXVFUqhATphg
TohdUnBKTmhNTlhgUkhdTjhdUihdUhhdTgh9UfhdUehATdhgTcBaUbhNTahd
UZhtUYhtUXhtUWhya9vsTTBCpIwfJIQaUjXBjrW093.MDIII5HSc4pnhHbDj
UxedW1DHCgPzykT81dmS207u6L.R8bP0GXYD2qQ7y5hjxy3dK35wiBiG8d3c
RviaDEJt1L1uW3vx4aUu5qKjuL6fIA1zpOFzkNAcxbo2hcxNiT.M.TR0yZwC
MAlbt2h1bekubi2YwAvaA.oy0LDOIFWjeiZbU3d2zPgSioFz8lzT6vlvDar0
J2.mlbqQlBY9+pU4+IYeU3ZKyBbmxUT7VfiE2D.Az8pQhSlIpX66vN4zTCkc
XdqZBZ2+C0zR1xoRZtnu5AgSpni4Zyz1Muz7dg9cUSYHqliNI6tftxLwoxBG
S8JY53nMksA8mCV9AIGSgGgg3Dp5UtRBInRqmgOJ5zyD.8zOIfmKfnek3RvZ
4+6EclG4WXL0uXuZetC4IOj5uIrXuTALt83uqh0Rp3cZRf+zYKs52gIqsiMp
+oCq8ssg73VorVaBGAXdp1DgeQjFOcA.HoE7zJg5.4lXV9Ab8nDNW8JBxiqZ
dnrZeVoWb0m99vXYZfBN19f8DdjkehS4oEoxSqoBf8lhGaN5wZ7LrLpaO5dz
JzUxdSW8qLVnaZOxn.MbK0ZMFW+J821rXNVay+UAMHbpHPLE56gEMn5VDnv3
nc6f+50hu4Zebb4BnfIkUPCqfhN4GQA6Z9w4tBrPCn.nqq9UhuIDp8muZBuD
NI+y6ChcdqeblyaC1EtRd7XT0c4Dzo685Ey3u9Xsf05uyWkZKrX0rN8iIt7J
BvhUSNdGxge2EVQ+Z65V20R20k8itKzQeA7hdjyC3zXqeacKonWiXLHjWUqc
o2KBENWE8VS1n+B+k0dMN9cq38rzyW6WMf2StTJuUdGZ+JyezeUEGP6dpmuE
Qycc9YXMqZ+qDqk.mvdygkrdVe+9nCAeEz1RtoAgjm1Q5gTOVITGTTvYrRjl
7w3qpVbprSO5SAOy0hu4I+qqR.PhXB0tx4XLR2GGIOmZmuJw2kFDbk0BUqfT
0Lw5HjkA9MeUf+2fnnjOZeMn55H9N0JIR8..2AG6UDmGW6IE5BUiAPEsdzYo
hdX2pfzuSD01U0PoNM3TlaTQHDTUsAV069jVIR0AD9c0NZ1Nifrrn72UX6LB
ig8J5cwKlDlnsya75m8ab96hW57CEYMrO9MBzK9OlpRiH1i5xI.DlHCRUmMa
7fzIp553CsmwEGYVR092H5QhtUiA.eePzGBxCW6eITFr5gLTODoAB7VhfDDk
gDwsgD+Iw3Y3kYTn5AKz29iesbCaCXfxibm2dH9wjCx.SMSZJrIDhcQTZq9I
dy3QOm+iiBBWeXHU.WXJGPoLVgZqi7Zp15WHGgIcyW5SGfWiSkUmUojzO6fr
pZ6AZmGdDwqeSGr0rvBzcMul9metboMx+IIMt9b+tHT6gJwTFZ04.qEECUej
ygnYTSUQ7PUw6L+J4+j7mS3Q1O8oScvVGEt+w.IOZzrP7DEZ23m62zut7c2Q
tl8WkkDIFKP.NJ1qr3sgxIR47G8ECFTeInUzD6nSqAdeuOLVW+UK5i7JUdCQ
II6quNMBW8w4BW82KoiTGIjOIM7gPAtHoeRRY7RMUZq+uC9kD+Tw.1rv00Ee
M0kxODWvjx24PH0+FJeLh55C4OV51q4yQZcUbmZWOaefltRuqUP.6CE1eqSR
SKTGOCe5sIxnG0irVoZ17MktyON27CQxoor7z.wipmOa+khRpZaF.yhNjVxb
p13RczqRP4KTmVaKZiHYOFtsZkousizHFCAZoy2jFHL1b9QIW+B137sm5m0y
t+QqifHhRphJ1zOSUTDvYKJhtYs5oisstY9NUunrD5krsQLTy8X+KHgU4T8Y
qlVbOc0QlwgegvpegvpegvpWH2cZbDmYjWSZcQSSIUaHWxMM0K6SKrKNxxZO
bWsisrdvHnHh31GYYSxIyIblNWRMbTj08ozKBv69XHyVijtNJRwyTM928Yqp
qFaqMezhckcO38eXV25HvZhOvuMcbg0MLocavXSfaiZGOXyCtzmqzhURt0A.
l4oUyVdjl+b5D.NraDsxu4A5UOnADwr8v75JOnyqc.dcFGdWykhrqwCDaxTj
cN4JTj8mWszlaMcKaqOPsOumFWws11hFo6RF34U1sMnR3SMncVEM3rhZfaAw
ahYqBsxlR8riwOGr.9LS0ZXODL92DK7mkJt.sPSqGKcVviOy5rvPKdGrHcib
PCok34ak6busV3tbmUNqFzPw..BYikF2zsJBnLLtJItwxNBEZhvn0Qv0cl6H
zU9mac9co1ijMNCu5FWHP3RNTDwcoL9Z5j6x3o1kEXj27HpS6cjo8qOcXAhw
ZugDaxBu0qHMuWUq5ewUTBrduZiY3ujf9ujf9ujf9KMmzABqmMNYCmz.WnJ6
IPj77+SqwTRwARxKEDC6AY.oN2dmCeJN7Oo2Hac60I6VIlNgyNwej4rzYWly
a16uYi.qbPhgEk+CeoGxhlfX40jHL2CxIt.wnntxXmnCehKRO2rVU+XzqCug
CKSVsWXbs6v4oA5P7OmDF6PFxZVFOZSnlniQEC8nt.fTG0jmZqWi0Lsiy1xA
VQp6Z7u4x5VEmiyaJ1v8O93q2s60YCcR.SAGYokFCG+7oBsTWBl+Topq7Dvj
kIUFXNxj5jlsYNbpwH3XiQy9IZowQW4L0nqxM.RwkljS2RFZwMYl3YVevCPl
vTBx.mSo.7beVCxlzSTLqOWylRc9GQrTa6ISdo.YaoXRO.Vv1VJlpC4MnssH
8bxK7a7cLCf7Z.boGDaQZjzjfZzTBa9LoD1WVw+RD.a2+US.r6U5fWcXa7ts
e9hDK3tTKXm2bTBhSCxdJN+wSuAsJR8lf3OT4ZpOi9q5v9MRVcCOWEGtHiwT
TyndH5CViQPvgwKdFVBIsJ+2Rbxe9VAIB+YZEjT6a09Vy.Oy53NELRKUDhdQ
07qaIj55byPN.Ts9b+avgwFwO8P0jiaHBNRq2.BLxKnhTf9jIS3yCWuDS7Q0
GW2Yg6Uoxr9Q+3jsakonrl9saRYl7TxkkZmAKvHsXFz9RXZsqUSOrLlDsFXG
yPeJ3vGzF87BkbmWSr40J22wfTppj6UZITu85S5R2vzLDthlmVgt4Ui2Ecc.
AWQP0ZYBX6Sx0yhmDdLdPtV7fb4iwSxFvy0ardRvAdRb2w3IwrsNQtxmD2l1
I9XfdDOKPOWzX8jFpNwFk5jM1dkU7q6IQr3Io1DwWckBissRc0OIab6UGhqS
TwI8IycqYv17TJUsK.HdjkhHApsY3gLWy2pck.bUUBD0hZABLBMTHfMlDvQ3
IAsZbiwvKAzl9tzw3AQlspDvlwMHjQ4QYkCI9n7nvV66CNJOKnUOKTsmkNHv
FL6Q9XZvnmFr4oMSd5lEOMYviJMcMzPWc0c.sysgB+7KurKwxqLAel0I2KQi
buP8w8B0FWy5gW2ZgWG5fWYRHu6kcKBQ0Efnp1dFz7VyM.WiN2doZb6UqusV
Y1zTaEuLEN7B0x1KSGaGMSmimHamul0d45U60nUsVqSsWcGh15Qq4NFVnAsW
cYolBV1kCRqzW1yWqRuDckcTpvUnNvfHucZF6nTrpkrNyEryGiub+Tms9udg
Z+5E5UbNFiqduC6ZEqx4BKrtFRCWuZKq1Z0p4R0kpOqiUArpNrZcIzJsWcjJ
gU0XUqKfVoqpiTArl9oZeIb.MScjJb00FUyktAzC0KTKTu9JPKMO0Zv0JcN8
pKfszyTKFH2ZML8BzuzyqBomBYG6DiW9Oe4+O3n8TL.
-----------end_max5_patcher-----------