Fluid.umap~ dump crashing dict

I got the idea to build a thing where rather than having a whole mess of separate fits, I pack all of the fits into a single dict so I can save them all as a single file, then load them all again in a similar manner. I haven’t actually tested to see if this is possible on the loading end (I presume that’s what the load message is for, but not gotten that far), but en route to this I noticed that fluid.umap~ crashes dict really hard when I dump. Or more specifically, when I try to view the dict that is dump-'d.

I have managed to successfully view/dump it a couple times, but I get a hang/crash 90% of the time.

Here’s a code bit showing what I mean.


----------begin_max5_patcher----------
4359.3oc48j0aaijlO67qfv.6aNZp6i8oz8L6lFX6d1FSxfdWzXgAkTYG1lh
TfjJIdFL429VGjTjREonjKI6.agXylTj028cw9e9lqtdd9WUkWG8uG86QWc0
+7MWck8TlSbU8+8UWuJ9qKRiKsesqWjuZkJq55abWqR80J64+y+se3C+Tzu7
C+OMWIMISsHeSl8xnNe8E4o4EtkDLSJXHBCPjD.ARH2nOE.vgH.WH.RNVfrm
BwIXBTHDbDkytIBNCD8+U+LSVZAf74+wawhlUOaypjrTUkEngaOY9lplyBpO
6c4YUYwqT1mxOoR+rpJYQbyCZcb0hOkjc+sEpEUNnFBEjYzahHbzLIDxwRIP
+ABfvahnzYfahvfN.nYAtKdgcAflS8udyaL+5lIRxMDs4appxyZ.JM.4n5WO
DcESj.Koa2CFhxI2hvEZhQkp3VUV77TUWRkWp5766rxPC1Clo4dHL0bDgCnB
Q+0MUce7hG6AyMOrj6yx0T5zjEOLLmC6iy8CEIwoQ+Xd5xQYcLKDR3xYjcYc
Hf4RntrtVwaZyC0AFUOtV4dhZVf6eVVRy8UnE7WprjWl94cBL8gzyVtY05n6
RZuPWdH6Dj98PjDDgQ7FI.ynTNTRHT6GjV8jB2RidoJJyeUHJKPrA4ReWHIm
lm+PYT9C9jjIgQRV6GwRjzNTlgPRIBxQlORo9jT7KeQY5qDqxNGpdYSiIKS9
dvpLNPxxXHzPjzZ36655IZV9DTQQ7vfVbnSMjYYyeGnShdcDoDSB6vWFSID+
cfRHJPgFwAHiNHDB5YnhG.cvKgrKT75veBRHGzT4KJ+IWDl9qi3gYl.gGP07
Ek4qKhSpWErbB0xWg.7LhPP0Lcf4C7.Y.gdo3vR6qJR8YUwiUFDyiaKXfbaY
pT0.DJN+I40JS8EMftuq3jEUQkKh0BbdDNodwJjWrhLVZCHKZ4O6NoEuPcD.
1icafxj7r3hGaX71+0dCkweVs7VMHqWxaiqpJRz5stxkdUK83pqUql6DOp4.
ip008hqhWut8x1qZo9gjGrQuF9XAj.wB.R1f0SyUpsW8rfxp3rkwEKS9Gpk9
XE3vvJnRPm.1kudD+WoJKiuW4Mc.ezazQPugiFjoKkUFaeQe7DH+WeRFbOVr
EDFrU.4t9ZP1uZiuXPVnLLHKiasbAwj8Ck77grCXCQG9QkJZHL9XbkhNrqTF
yhdrofj2DZt55hjLeIuCIggsRb3FlNSG5LSG+r8GKSEcokfGBWCjDLV9LfqC
H.eW5ljkyVFWEWpp9lMhjYGvm3wzFW7AY3BOgGBErCSFlGahJ2fA4wUgWdWG
9eUQbV4c4EqbjkB0xMKTKqoQlvm8QcXAxAFvYKyqCLTcBTWZwjh74aJqrn92
7f57vHXzf5LnmvVgfIngjlTVE.QiKR8edUTK.tsyAPp3nJE.7kRo.tWkoJhM
N5c1IGcJZ5xbCU2kf1JlBoPssRAffEbo.gr001ppfwmXEB5Kd2AzImdcImrM
NI0IUv1WpfXkJHGz9e3ikqmc+gBrC32JO9TM0Qv88BxYbSDt3IFn2MmknZmF
kfFFJAsFY0YxrWT8W.BwziBnavQ8hKvW7hAJ13Zhi276PTxKfPFGgFfBSTAM
AM6MdQF3hEu3nDECs3aQuSiTKSzdRJSxyJiHQuKUEWjY8f.lghdWRkwch8hT
Pz6zjkk5HVbWSeuYpj6+z77ME5K6ihBBCEsQpRvmA6JUQz1dnH1EKLqiMKLQ
fRCCQrirAi.dAjx4dFZlVpX7.k7ccuZ7USEDf+RHmzgx2RPCi1PizfW6KTzK
C6KcDG7k.lfDnTyqEGDd5VqoMUWJKCCPKVGo0TVluxWJnxqCRnoHWEHH.nN1
TBgwgB6GFbZMuZaVI1GtpX25xeWRpo4hFODc.kqtNd85Nm9pN2hg17Gtr1D2
zdpjrdIxYRCR84j1GKn8zwEZbrRifaJbIt8UVizh44juTUjsIocJ3sroZXxx
PLI7UttdP4s7slK2sKGtFYZiHwXW0PBEcxsyj0ZZ9hG1okD4qUYIYqKTk5ru
rNF6c4kp6h2jVcqIuyRsnuEHMZjdtdyn768h8SZsEAtuHYYdlAH5wJLmtY4z
gpWmrRWjw9MxhW64lcMlYfKpUiq1TNOtvvopSoB0bwp77z9Wp89RU2UUe40I
YY6PEqxWO7EKzgTLx8NOWewUi8rsWo71MYtqdqVln5VSSn5+8hSSq0Z6+3+Z
bVxJc.PUINV.BzdQWZkepbQQdZZO70ckO64JK0B4KTeIYY0mrKTWgA8WOYci
Pz0sb4kI2qJq5etp36K6elxpGcD8NmZy7Zk3aqTqVmpwh9eAs1g1dW4mx+RY
8WrQPqKAX6N4oqRcWSf8N+XlB6aN7Oo03z+Lq6E6XRj1879BSXXKiCOsqVMb
HypSruyQ+1DcNCbNB11OwqZbHDXphyEgizL.kgbNnLtY4.ZcereYr8SYbkw5
xPWrARoUXhfyhdq1OejXPIm1ARbeBD6TIPtHKP5+T6Q09AY1bYv8CzbzfMCD
sxhTCIgLrxC3jEQLiQsjS0Ib39ga1sZflMlV+sm1nk9pyWooxkcsKYwkrkpu
1wodPnXN3Y.RFGMHICNHICLEgFrV2xIr.p+gMHI6jnGacraJr6.lqsfm459I
Tk5z1WzvmpM+F0G1zdnpRxZCy422prEAlnj8wBDjoBEjyIPLQXfdFggFp7AA
BiX7.PQ8IahOdpShyQEXhNJ44wo6DMgu3VB9TQUEMOZgodU9lWE3QjcE5PE9
GAfCNm9SJSaGXdQp4DITs0kvbaNYOs17h26+2FA8sknPgp7Z1cig2TnQ19YM
NptMnnPIXmDM2G9BBiPcM9hY5+f.BLnI1FyvVhlF9dyYnsVqUpGpKhlNG6Md
mgGzQzDmIvx0QyPHBH.6JnLCRaqJv3zftoHDFrey+HwjCfOjFdDx4iU3Pjsv
gXDd+lyP3S2ZVyeOKx8CaROPSyCTJrnpYuhJwTp.XsrADlo8dBTgtR+mGS6C
QA3vvXuy4Tyukc5E1x97M2cmpnqVez6JiWstbHcg.YCzMSWZ0fYXrD.QLr4i
.XmoK7T0+OGVAKsd1uKJd4x045ih92VNtMQZXJjbsdgWwBH+R25Vi0PpNwoH
eBAjvXOrN3NBcFqavcDaq1d9MG5o4R9BJHTVEo1s5HTqPvoXNqlV.LSlEj7j
apj8F6mspizradQ0Dn8xGR1KorAxExzv4nglunIuRlGB5.qjcvZd5K0TvIT.
VHwTVIAMDqDeJqDIDqDYR7oPfT7oP9LwlFAepqDbBqDNDnzTHdgf1wl.BwCv
5PuPDNxTDvQvPrRrKEJMAYAjH.KDFMU8nm7JAbJiiK1IC0JcPtTHvIjXBqDM
TKzgHdvPHhiPWHqCnoXWkDpEBdATZQfKU.QP4kxkNbJFwQgPDGNkvICgIOqZ
xgLthCBaBLE+ErfsRGRHGEDIBvTDIjgaoNHVsutqK2lclGJyprybPsyLP4Y9
mFd1m1ctmr8zwMCH6jP01QGZyxj7OXGfma+EU1FWhhMiYTehw76uKIMscy1z
saPM43cs6psiZU28RDRRfPgcODAwbDydj9.JsCAs9dfM2Dg1rwiHLWV25iL6
JHr8c05N2FZ6ZAfR2J.jB.wcj9Tvd6gHMsL695AVh2YpaVWjuNuncxslgkse
+MU42WDuLotcr82K42TKKUnu5tidy0qzDzjl6oqrWKy3m2z7tf0OCncuNcaR
lQLR0htXIBZQWTK8BwEDbezs+lrxLmvDHWZ9xBIkv41iHHaK7AynrN22g47s
TkAX9Sjgzm8KZY5h5c.l4n8PrcY+LpdQLeWl4Gt6H2S5rv88rSxzVB9uWqxh
9PbVYzGTqRla1SYaeeGnRU12HB8nRbDf4.bHiibh3BfcOD0kMFuXgF.5uQ8L
Ld29yqA2MpVly08N0q5pcuUtPShbzX.THbGoeJnc1le8tIBmBcaMOjDKjtin
Za9MKXPTH9k3ppwTH5NBklsez.uHj+8sYb+DAne8Gpt0dafi.rP6ra.8pBa2
viM+ZTcVmRiui5wsL8GX+sToTqlScpFZMJNd2056K87IeamWEc272Nl54zLH
0mTJoFOlNKwDNvRTMzTbPUw9sO9eNlnbabQ84xFO3VySPDjxbFdLL.qmGYOk
vNJBhgEqgLLiXoKPNgIk0GAjuPDPml80m8vP7Ke9SOtrH+dU1GsjnQDSo.Ny
8dwmQwPG6PygQSvKBgAg.W3YRDiXQZpDp0FGQDGZBDzErgQt1J8vaBILXh31
lnjr33EyGzlbe4ZYHsuKvRtyNFVVq9aNE8DrT6IVb3wsLmr77tnTczo6ezIo
ELpv96UYpOGOpXNo1JpuiFNhmCcSgTV8G6rQ18KwtuqcQKyATeTvAq+R69L4
EDP893jruMQnRK8zZnY+iBOnU9kDsgkw.t8DOg0FH19qyBLgdQAT+WkoIKUE
iW8goYa1elUSUk2qqsi2bw4ld8yMadT+zJeXAaHB19Js89VgDr04yUj70EUE
GoUjyG.YdSf7siPr6flOdJxQgDy9q4KUkSOIUY30Alfi+fhvG1DxALfDTnYy
p4iCL9rSsuSyy.n8qIKpNLs5YW.9WM6e1pWbQa7gE4qUGiEiSKXy1BaHwleN
WxBeLd9wJDLwn4eFCd9i67hNahw3LRLEWPO6eL+96adSVOclBVm2NG5bFQvP
ahYjdcH4YLJ3+9pCzXoKH482h+rx7RI4aGqo4wkgAynOmD5kwEO71Lydh+s1
JYb7k5fzpKyaafTaqjDVrb+Zbbjkg8IT8N3ADzCdMKBdk65WyBOFetLk68Sw
Yk..5H5dALjU2hwHDWEFYXMszQfEB326Mh3YpgidDi3l4x2g2Tlf4hbv75y7
YHY0+ndGeB.0aJkP7nPGuwssbm8NX3Z5RGrUZi1M.P.KR6zVluKL3saGylmp
zwm45N9Olm+v0ASjKI6gSQB4ld+1qQu1urlp.DsV7vLf1.w3E0ehNH2WRaaG
RrqBt6QAne.SGktfNhCrQR+QyMAlnu93d36xas2N7sM4uePTSdvU.feyLGYO
2QlmZha8v4qsag3.RR8XmHkXAu9HtXBEzk0FYKs0bZi7b3vqUwek09hE6HGn
qoEO0NrjtIg3Q1UzOmcdsRXXQ3Nu9YlPM3HHLEI66WyXoL37gd6h1vwKdxoy
HZCWb+iFwT51WL56ezEzE+SYnH6qcLMtX22TOSsBucFfzfmLkaOY9ioaTMu3
6mbolvPd87tr0XjL3lfpAvh7ujczP3kwHoCB+yOFe7.H.Iot4LgywXjS2AA0
QLEd.78EJ0I.gVpWaAbgLSbfgG39Esksrp3iF7DD.qNAQFPCbVRISBkzvCi+
M0xiF93sAVBay+AT6wHzv2+qJMM+KiCh6OVmTI.HF0mKh5FiJt4kmhEAnZoV
5YCAPeOh.1dk897zSPDosTYTcbMT2Dq00kVP.v0aJVmdZAOvaGRWBzTOFqQJ
iY.5nILhADfKvHDEvjVk.jfxPhPhWEpEpjOOdah8ISzlC6VbBzsZWgA3bon7
91+WwxjRK3fBDdS2hnsCJq8evq2GDZgbYP8jTpxVV9RyCxVhb3087NTy.NDv
cM1jv.x5sbBGFTYmpl2BFCaGrSwn943p7ne1jE50Aa8KqT2sIMs5fcya24fV
PoPW6WXXN1smdXZsqw5tprslur1P7MUPfKBKQ0gTGMNgD75BQiakjwHIVJBP
p5OizCyq4vWRaNkuXplyiGewOgRIvE8mVNiTO+DL..CGaZ9wrozXnItwG1Qi
7ujTT8Xz+w84GZCSALLZa5u3Z9q1tTcizNQNsaiZNvKI927udy+OWhqB0
-----------end_max5_patcher-----------

Ok, it seems to write fine, and it appears that the fluid.umap~ dump output is gigantic! Or rather, it scales up with the size of the dataset, whereas fluid.standardize~ and fluid.robustscale~ need less values to do what it does.

I can confirm Max is beach-balling whilst trying to view the dict from UMAP (but it’s not crashed). It seems to be attempting an gigantic memory allocation in the window that shows the contents

1.07 min 98.8% 0 s juce::AsyncUpdater::AsyncUpdaterMessage::messageCallback()
1.07 min 98.8% 6.00 ms juce::CodeEditorComponent::rebuildLineTokens()
1.07 min 98.8% 1.07 min juce::String::substring(int, int) const
18.00 ms 0.0% 3.00 ms operator new(unsigned long)
15.00 ms 0.0% 4.00 ms malloc

I don’t know what’s in the UMAP dump to make it so but, presumably whatever’s needed to recreate the fitted model from file. Given that UMAP hinges on a nearest-neighbour graph of the fitted dataset, that could conceivably end up being N2 storage in the size of the training set :grimacing:

1 Like

@groma will confirm this is what we want, but in the meantime, if you parse the data you care about before the GUI via a dict.unpack that should help, right?

Well in this case, it’s all the data that I need as I want to save the fits of UMAP to load later.

So far it appears to be working…as long as I don’t look at it.

yep, I don’t think they thought such large dictionaries would be viewable.

Yes umap needs to store all the data points in the reduced space, plus a kd-tree wich has all the poiints in the original space, so in the end it is less compact than the original dataset.
The “print” function was actually intended to view large datasets without crashing, but it is only implemented in dataset.

1 Like