Memory hole problem?

Just for now, yes, sorry. Strictly temporary.


----------begin_max5_patcher----------
509.3ocsTErabBCD8L7UX4bkFggPX2lS4Gnp8bUTkA7twQfAYa1x1nzu8ZOF
2xlcqhqBUHMRy3mYl2aFOOGGgq5mXJL5inuhhhdNNJBBYCDM6Gg6nS0sTE.C
WMp08Bbh6HwXGWzxzvYj4f7F.Ye0Sefjs.Y+n90PU5isL.sG2.UW+HWr+aRV
s1UX4oWmlfHDqM6FuE8v7Mb+V8wAlCNthJ1iQOXO8k3XqIIPxIXe2T09RQyl
fJ.qQUnpfob5kob1+NkK.xl6nb1aR4j2C06XJEcO6Ltqpk7AMRxFZo0LT2wO
K6qMX6kHiZwlPoluMEHRFZW6Hu45pwcCbCg94EErryErxUXDgjs0JRaKAAKK
.EacGPdjqfphICcLovCbWuPKncNJdujSaWg4mY8fjCBxsaCPOR.MYQQo3+.N
jXtmO2zCd7W8I6MNg3H7cVyUewZYhFWjUPmOPk+VgVL.dVaHjAvK8h8+ZqXd
6kqSroH72x+8twBAEvfa4hWuEGpDa7SUYU+nr1mI+FZzeplFlRyETM2rieAn
TGnK1ICNSogjo0JQj2HQkmTLc7lgdtPOKhlEuEllUo0Rx23cdm0UY.7u3L96
5vzggCLoZFLjByKlmLOBLtkIfKW3byAWI6.2i+FHBUZlM0lAyQoapcZysX2U
6aXRwHGFvhsY9k3eADJCjCI
-----------end_max5_patcher-----------
1 Like

Nice abstraction, thank you!

Not sure if this complicates thing but I cannot get plug the memory leak with your scripting solution using fluid.bufloudness~. The memory climbs each time I use it until I am forced to close the application.

My patching:


----------begin_max5_patcher----------
4411.3oc6crsiqhb7447UfbRda14z2ARjh1UJ4snHEEsIOrJZD1FOG1vXb.7
4xtZOe6ouAF7.3FnwlYNrW7XnocW25pqtppK902c2p0IeNLakyez4mbt6te8
c2cm7Vhabm956V8bvm2DGjIerU6C+Tx5ed08plxC+bt71Gb1EeLZ6CaCy1jF
cHOIM6qEOTzV4iv612AwE2b+wmi1GGlK+Ug5adHHeyGh1+ziogaxUPEC38.3
dGLBH9CDJ+CB8.v4+b5WJ4X94+Tpak+kCgpemUqC1+zpxNIGovTMhpwz6VsK
JN7igoYQI6q7ic2pfCGpb66pzEA44mSj+Pd2Wdqn8paAJuUZ3GiJ5Os7tAob
7MmirGSkv4pOyHqN8yjrMLc+wHInnt4u8tBPRxR1G7bX1gfMpNK3bEMehBJo
XDp3Seen3OdT5I5Gmg+Tbxl+a31p.7pjCg6i1eHMLKbedPtFxKada3tfiw4O
tKYedVzuHGeHmqzT66zvWiMJv.Iv+CoQAwkf+SoQaS1K.hZ7AwsKFtexAJQJ
XMjQ9D6CNzPm4BfbhRKMlwQxiYqCREro0wxQ.UzXdRRb8lJwk3vc45lODse+
YTw7jCs2XZzSeni9tNg23yc8aKaI6wi6Us9HWhH+wrfOVGByChi0yZq+y+4f
8QOGjGlGoXAHPYig6C3H5G3ykShiqQmTs7wFZYKWBeS3mh1l+A4.UUXPoTPK
DspjKuM5ovr752KO3or52IK+KJhdkacbsdF7i4gOeHliEJsSIBJxpJ33IsaU
mzVUKWs6el1N9rqrfmBKmRVUkmBmbRC4C+lPm0G287tMaDxQYNb5c3mc.7+E
C.7IFZsi7mQ19Wc99rjio7dk+nnS7KkcSc0CpK9dthM97+nOl4.qB.wQ6C2j
bbedUQzyzy5WsC0T0VsCMntk5KUWf8Hp+H0WfqNCqUUtMq1kqxsn0e6cuq3K
2e03GcvJDM+UGHtByHZuy2uKLPnNNqj2raWdFGm3DpL9bDhmy2AE+mf+vWTY
alCAHuXSR3tcblEtUtEtYtk2X4Vh0C4+wWpOj3+5jakcfiWoAwYeH3PX+mFk
7LegprJyjT2vBSlHij8fbk7Erq+7ZxTMS4jM+wfzx0iajizHq7lvTPU6PKVg
zlclk8oYiOZgaBYRqMXp4Z.fALSTWLSoAo2aal55ibqF1upYhFDfA0wxTNEH
muJpZY8Zlr1EwqIC0wJSLUh7jhOGpnt1b8IUbuTBN2Ysy51nYqFHAQIuPkTB
pxv6p6aYnhKWeBSWDGvPoNZciPOWI8AY.4gXL449aE45CQYE6rrYBF8pq1hx
TVEn9iKXrxgZUVUdjhcbva7282EMWiL3r5OI9328ODeFteq5NW2ExJMLuuqe
ME1liH2nktPddy5ktLhM1MGb1XRO06VwjQUYxWXu.yQlbMiNuv70pO5W6fmW
XFZyr89NAF5ei3s9ty54uRbsk08vsZm.vHqtUti.T74H1.Ukl2j7rPvnlqkj
v89sgethSWsB0QAKsPdXiyHSL02X5C3VQ.Fg2.5xQ.2Fc.ti0m.J0zDW2owm
.mbHt.aZwGnRvSzdybOEoTOLZuJ5TGN2FlwojkgG3mJMxU7bFI6zWXvaF.Ct
y.XfZJgPXt4jAEjY.k.aHL.mPX.YJgfMk.goLCuIDHflpj.Mk.gzOWlIU.lT
3vPnvepABnA.g2TCDXC.B2oFHPF.DSoJSnoBl9SLP.G2rC8MKRHAoWf19nJF
6OFjmmFs9XtxxipYXQuhE7SwIqCh0tRpbCSMDp32cB3jepnQVHeVhSNtka+T
yIyBnuIyBjfdfduCEPTYiwPykkk7XYIOVVxikk7XYIpfWSuRjcHJN9uwWP3k
tiP5HBdqU7+PwJGUhXvlODrOyQ4l4cBZ7.hd.DfniMcQTQaCqXGy47QXI.sF
Ff1IN7r9DCHN3dGd1uoCKqh.STo272fgksTaZirrqmB0aW3XAUiTG30QVgIX
B+yADI8Bl2onoWmc1Wt1MK9pPh4bsYW3Uiaa5VIiqb+tNPWlHH3sEOkSLzOE
seaxmDj+h97gjCxKIj5K.85I3pfkXqNTayeKFeUjx5v2LwWsTKdexvZaqBGB
G59XXHWgazvJ2ngYjWsGRg3F2M4MPaL1xg4loR2PDVwabwSyISXRBysEbN9U
LzQJGBb6idDDNGhbyzF9nBJ8MMd2ylfaBWB9+rIcPlGQUzeNjAByiLiwBYDx
ajHrV2bnSlBsINLHsgvp5RaLrpn1CqJgHsR20uLwXGbXUsYDjEVcUZY1i6hh
yCaDe6cMQPcdBIJOWCQjKfunNKIB2+MboQvCniKkLfHbqkkzUOvRTkWhp7RT
kGSHWxBi4qpAayiUs6xJ7EBahZBqZdqmusCK0UKjT+YYtm2F4YvdFQQdfEG4
OCHOcto6HgyArIEYWbBe3aCssSVHrKIkOuUbWlQDKkW8TIh+Xi.2UMrlOEl+
PVvyGxj9CpEhJxcvg.VQdTS0fHzXEltdNDGAFpGwo5fx912i3cjEIFIWfzIQ
xr1k3cSA7GJEviNoT.z0xQmhkgMx2dnozqAXS8gyj5mSrodPo.bmFnvTO6gm
zSyhodzBOk9WDYJs.41gKTpYlbKyaJ2kEW4AqxNEOsYrlA40OsIIVsaV91sj
KSV9A3AZMPOXyF9j9JOO3AOwS4g8QPl3at.eO.Q18Z8Trdure7E4E6zNrR5P
e5CYuNQAtuBYOUbvGewtDD5J4XlbGzfNXa0HNmspa2jmZHKAgoHeAvhcQTh7
aHWBEds.8WDUtKwa2EEGWhB28hmfyXzqzy2Ucv1HgJ86qzprqvRdMi+Ot223
2Nigq6JpnqtddbZ28M9sl6ZQOQ9.nuT1BqksDeieKXC8jaqhde8tfpIZAmMj
lbHIszIGOf8q0ui4IkDfJLjZrRqKE+xYp8Wh3bqC5gzrDdIbwX.R9MN0kSUr
kj7aDG+xkKDhTM3wyhh2UWaxquGpFWeVy4Xtod+8zlcsiCfeR31kFPdPecus
FA8TFdyfyB2aKR733FPOleuOTTtdBziQUVU6KtXLdydXnYaQo3SoQMxFYv9x
Fwprn1UWEdHyB1HWkWXZbxmZBCQ8kSp2QjmpVMSIyaAU7.QOWlx4ifwF1E6F
tozvCg625jUthVMjk1WjECwUjVcQyBd4OmT5mqZXGquyEKXkHoGj4Zdny.rK
Z+tjuV0Wd0vQ2ApW0EfT9wCeAVneaH4t3jfbg.abTl7uk2vjuH9eauz51nM4
NAG2FkTFe0lHYdcJVTeuXrp+iz7bj9iJVJ2wDEuhBT+kh9LoMxr.o3ltEj9k
BxVsyzgoVAF975yBEVWdRuRiOGb3PYyV+.U+KwNYwQaBKiITsr42ef1Doe6.
PoCUW7HzCejSnaJd9TlIV21bng6wZQvGPxbfk074GpODfJQsXhsZhh6KiluE
VoUuT1Lxpo0G2wMa5qskBz0v3dWj.7UV3qWaBhYCk0VpBd37Wiw+Fvbh+.wb
lpvH.AtyOLWmUOslLO3dyto5ElUUbel+Rx7Lvj4gozTrjLOKIyyRx7rjLOKI
yyRx7rjLOKIyisRlm2rIxi227Ixi2Rh7rjHOKIxyRh7rjHOKIxyRh7rjHOKI
xSMquJLz4bmgBYxzflq9idJYAZ38Zow9BknJuqPcwmqv+.s5LTv.PlZkBylp
AbVIojPH8aKPU0EDT74.eqUayfLm2XoLlz6Hv4qh.kFSuXd6b0x6pCNogAae
7rPSUkQC6elenda3ppkodKks4d3VdpNXNjEuxu3U9EuxawpZZR1WddcRbaNf
cv07Lk1Nh50UL0cd4pvNHHEo.mt3Z0FcgNN5BUEVZe5qF5xkDTHiTPA59JSP
I6PJ2nicNe88bUFueaPdviq4BNbj68+E9E+fHwx9w8Q+uigu+Oj0+ndYFUSG
PGl2qFxVJenCc5TVpCWwhLgpnMwxNwuYhiQQ0Bq5QB1C.H90CPSSICSuihgl
vn+yaqvXTiD56BkjP+1pQ2fQp316J9RRbRbyOw32sVSpa9Yl5T4I0M+FWJ9l
RWaiIyhfMXZfWvS5qeNSkNwjuQeoRcJsIO8pDuoDmDMzDmTk8fPHdNmxnUdK
dWCqgCEqw9pWFTyZjV85msUdNdv7bfNMgceEf9Mf3HuAh3TWsik7ugHda4+t
vqU426veVmeO7dGUFo8RbGOXmtVLSedTqCSJpjzMfi8dZcANpOZCTuavY6ne
GkPn2vbnr104t2ni7Z+NjcP29eH6TVZaFRdcjTkhoMl49vlODgzKxDUSEu3w
xlzU7Opbfurc3PT6ZO3vA4K4mWh1zgwVg9HMactL6bSRbryQoaadTd1.aBaI
CLzOp25KP.ZnbYKcl9lPaRadVOZXDLBBNtiyhEVbt0H8V9lmvNg5UEUvwU7I
ruIHhe7Lm+8O7u9qMPA5+4sTeJh04WMc1bz9E+aqUR5AJ6pTkSACctdiur6t
unhWaUt76+wrvzr2+yh.wtNMX65ioe4hd0toceMrUA7wJmSNOJO.6Rh213Z6
CTIlN6Mf9zGboCeUNqe52aulc.Af9WLwU1unJzCy7h1gu+PQOj2qth1gOquH
qtHDnOrzy6h1geuM7TyJop2TauFJZG98d1HyShVTWvath1Qayo87GJQBqLJG
4OqlT2h7tGpux6EnoJ0ofdrYg.emExCudWUrXDoiAHJCKuIExiVwSLxk457D
eGzGZBWACjgRTlO6485AWY819POJZ1hqcTHZ5OdBYyS425uYvqYqeucWotL6
fUn2k8BBtW6Lxxgb372azWJzKjdWazJ345SSqm+bonz7BTuyvL1ayKKzUCpd
vzu8kvvSECrf8AweIKpIrEdMpEXE1eqT2Agf2n0BLIbTOyVTLvyyY.Ma7Buc
KaKyMpjEKuPTwzQxs1Qi0f2ShCdjD6M8hijOyViD3h3jUPJlICkMFHpACjuM
jHDaJyw8RiD0FijAnDp1y7bz1C7sOjqmVAgPgBWHU4lM0QJp9E95xhFUewHA
YOiDhopGZ7izEY31PF1CYvH4YCQKOnATOOqfSln.0qSgKlJE1IppsV0qFqJW
SlJy7rkx8KNRHaLRlHC4ZCNqqIbVDrS0F55vr5vVoKWwUufPTtTPkm00uBCJ
qJStT8kiEkLQmhA5Ac096QUXhO8cn9vVp0VJtxBP7k3AjtfXkFYF6j1YlcfJ
zkVKlU6gNmP540EgDoOtpJWqxHVAhwWBh6jPhXztfXLAchDKuXjPLyjkkXtV
XhNyjkZEk15JOz4rSeUoyW6dE0ISUd0XgMWCrQhYCaYYlXfICaiQ5psrDiX5
hysxY8XnJKNKthZmEmYlreHlMr9kc0VblY7hyi0nUF7ZICQMR6fMLZU3yyKO
RDaLR3qzlIolvlf1PziZzNwsAwi3YhREPmV7CcU9jGT0he2pkkhjzspxR.b5
AVFwDfUkJFtPlJsxpcEQ+l3.2vUJjrPOZyHIX7H4kl3PXchj58u6oia+oqrg
zxkrUjP6B1b4V9SKgMOkM1VA1LYIeOanGfXxzSpUlcZtqdF03fAWMW+gv8Xi
niajLQcMsyIRkaWECJBVN3EWoylBVwNaw1WmfQnxY63wR5bMwHHnMLrCYjG0
IVvbKnIZXgt1Zj.lr25wR8ft8PBYbijQ9t2FakEZjYP1vrXnIJZo1PoDzDwb
WarwTnI5Lbur2lvU71Dd7dapzr0tW.vJT.v3c5oKS5KDe84gT4Ly5WcQWaph
q4YUaKAPeVU15rJr0KqtVsWYsNupZIOpymUTSUDulKlomUaC0TTCKfoCq3k1
2R9n9za2RsRrdcRrpzRaEnzlQ4dWTRsBXUKO7aiW7hBNZQ73asPiNvhL5.Kv
nCq3h1bgEs8hJZKETzxy1ukjppO6vLtX0JQhARWWpHg1OIKkRlVpedu62d2+
GeAEvi.
-----------end_max5_patcher-----------

wait for Alpha08 then. It is sorted (we think)

2 Likes

Just wanted to test the last version of your patch with Alpha08 and I’m getting the following errors:
newobj: bufspill: No such object
newobj: get.samps: No such object
newobj: onecount: No such object

Ah yes, sorry. Three abstractions/externals.

bufspill just outputs buffer data as lists prepended by the channel number.

get.samps is a patch that wraps getattr samps and a buffer~ as an argument

onecount is just a 0 - (n-1) counter that sends a bang when done.

abstr_n_externals.zip (429.4 KB)

Continuing to work on the ‘over-night’ analysis engine and made some interesting discovery today. As mentioned previously, I’m running several instances of the patch, as all the poly~ and parallel processing did not lead to faster calculations.
My setup for the past weeks was: copying the Max application several times and run 4 or 8 Max versions. Each of the apps loads one copy of the patch and I give each a portion of the large sound folder to analyze.
What I noticed was that memory creeped up considerably and the initial quick calculation would slow down to a point that one single short sound file would take 15-20 seconds to complete. The same file would analyze in less than one second if Max was just freshly started.

SOLUTION: I packed the entire analysis portion of the patch into an abstraction. With scripting I’m tossing it after each analyzed subfolder and re-instantiate it before the next folder gets loaded. Memory behaves much better. Calculation time went down from a full night to about 2 hours.

1 Like

@weefuzzy is it linked to what we were talking about yesterday?

I’m afriad not (unless I’ve missiagnosed that particular problem).

Probably the only way to account for @tutschku’s issue here is to do runs with his whole patch on data sets of a comprable size, using a build with debug info so that once performance has started to degrade we can attach a debugger and actually see what’s happening. When I last looked at this, here, I wasn’t able to reproduce on a smaller set of data, which I why I think we’ll need to go all in.

1 Like

Here is the analysis portion of my patch.
https://drive.google.com/file/d/1XvoBcb1JvEuGjtxpSa4FNI-dxB_Ii1r1/view?usp=sharing

Hope you get it to work as I needed to extract it from my larger work environment.

I also made a short video to explain.

1 Like

Hello, I’m bumping this discussion because I recently experienced something similar to what happens in this thread.

When analyzing a large buffer with [fluid.bufnoveltyslice~], the object seems to take a big chunk of memory and not releasing it, even when the analyzing process is done. I did a little testing patch, here’s an example of what happens:

  • I open the testing patch / Max takes 172 MB of memory
  • I load 337 MB of audio in a buffer / Max takes 860 MB of memory
  • I analyze the buffer with [fluid.bufnoveltyslice~] / Max takes 3652 MB of memory

Here’s my little chunk of code:


----------begin_max5_patcher----------
613.3oc0VssiaCBD8YGo7Of3Y2UwFma6S4+XU0JrMNgUDvBiSSzpte6EXbRb
2j33F4tU8EiXXFl4b7bg2GOJ.mp1ypvnmQufBBd2JIvKyII3nf.7V59LAsxq
HNs1XTRbXyYpZifYLGJYv0fSox0Xz2OdNO2akJ8sus3jQkTS1Ftb8qZVlArK
dwSSBQKl69Fmb7646QVukKsdxGDQmuHMcKyvzuxjzTgOFlzxDH5.a7R+43Qt
U6RXuQrj8Ca3eJ3Mr8F.SkLYNmJTqQEJQNSeaRAG1EwDcehIZVreYRjmZhuO
076nOdnP+NpVZob+QEhZd9SYJYF0Pqy4pBtvlNcAQcc8PqxYUFtjZ3JIJSoK
qqtMClyyb5Q0GtNGlbeNLlrDVHOFGN7YP.wblR3cTWccXS5ArgblkIe0nlZL
5Z9eJhl2IjlOyihkP0vzIWhImW8WlYilUswVXh+Zf6i2XLYdO9KBP9+jVio0
EEL8GnJAOiUp3RSGk1EBE0zcGxjocwPjo9JZBIA5QN8ebOxOUeaICoZGSXN3
oiOPqpT05LVSOOzJtcLRlsgXuXKOIE1h0tAi0iFhMDFI9AaHR9qk2buoA8Hk
gzMALizZjvhAJiArFK3xKdUkOFbG7IZAREZfUyqjPsBiVyHaoTDnzs4+d6NW
6166OW1Tas1xyaxS85X4uoPdjaY9oMCRD55N1+Hb.bX+XjAycQ8wcjq5tl7M
ZY4NltpwBvS15s2T9AgKBg8bIrOF1qY63GMARuwTssLwXqQp0vC71OqoHBuU
YeeqzNMuYbgCqNW6KocOHrpjBnxW5OdjUgeAm241gC
-----------end_max5_patcher-----------
1 Like

@weefuzzy @jamesbradbury @tremblap
FYI, @Gabriel is on Windows 10.
I can reproduce the problem on an Apple M1 Max running Monterey.

Dom

Can you reproduce with the new nightlies? If so I’ll investigate more.

The nightlies @tremblap talks about are here @DominicThibault

I just reproduced with the new nightlies on Windows 10 and I get the same results:

  • I open the testing patch / Max takes 185 MB of memory
  • I load 337 MB of audio in a buffer / Max takes 866 MB of memory
  • I analyze the buffer with [fluid.bufnoveltyslice~] / Max takes 3566 MB of memory

What happens if you repeat those steps and then:

  • Delete the buffer from your patch

or

  • Send it the the message clear, size 11

?

Ok so this time once the audio files are loaded into a buffer and [fluid.bufnoveltyslice~] has done the analysis on it, Max is taking 3554 MB of memory.

If I delete the audio files and/or slice points buffers objects, Max is still taking 3554 MB of memory.

If instead of deleting the buffers I send a “clear, size 11” message I get:

  • Send “clear, size 11” message to audio files buffer : Max takes 2883 MB of memory
  • and then, send “clear, size 11” message to slice points buffer : Max takes 2883 MB of memory

I also tried:

  • Send “clear, size 11” message to slice points buffer : Max takes 3554 MB of memory
  • and then, send “clear, size 11” message to audio files buffer : Max takes 2883 MB of memory

Hi @Gabriel

Thanks for the simple patch to look at this with. I’ve had a nose about in Visual Studio to see what’s going on memory wise. Happy to say that it’s not a leak; however, you will see the memory grow markedly, and quite possibly descend again although it still holds on to a bunch. This is because the internal buffers used by the fluid.buf~ objects aren’t junked after the first run, but kept alive for subsequent invocations, which is useful if hitting them often.

Medium-term I think I can reduce some of this memory usage, because it does seem excessive.

Just to clarify this is explicitly a windows-centric issue? I can’t seem to reproduce on my macOS end.

No, it’s everything, but the OSes might differ in how much memory gets returned from Max to the OS once it’s been freed. In both cases though, there will be a marked net increase.

1 Like