Fluid.kdtree~ throwing up errors all the time

rodrigo.constanzo · July 18, 2020, 9:30am

It looks like if you delete, rename, or instantiate any fluid.dataset~ object, an unrelated fluid.kdtree~ throws up the following error:

fluid.kdtree~: DataSet is smaller than k
fluid.kdtree~: DataSet is smaller than k
fluid.kdtree~: DataSet is smaller than k
fluid.kdtree~: DataSet is smaller than k
fluid.kdtree~: DataSet is smaller than k
fluid.kdtree~: DataSet is smaller than k

It’s almost like when you disconnect a yellow cable and there’s something wrong in your MSP chain, that it throws a fresh error each time.

Actually, it looks like fluid.kdtree~ is losing it’s fit when I do this. So creating unrelated fluid.dataset~s is doing some funny business.

tremblap · July 18, 2020, 11:28am

I cannot reproduce. let’s move to slack for this. give me a clear example in 3 steps. we’re burning the last troubleshooting hours now so clarity is super important! Give me 15 for a shower first

rodrigo.constanzo · July 18, 2020, 11:41am

I can’t seem to narrow it down to a small set of objects, but I imagine this shouldn’t be happening:

rodrigo.constanzo · July 18, 2020, 11:54am

Again, I can’t seem to isolate this well, but when I duplicate, or instantiate (by manually creating a new fluid.standardize~ object by typing the name it), it clears the fit for that particular fluid.standardize~.

Furthermore, it seems to also clear the fit on a fluid.kdtree~ further down the path as well.

fluid.standardize~: No data fitted
fluid.standardize~: No data fitted
fluid.kdtree~: DataSet is smaller than k
fluid.kdtree~: DataSet is smaller than k

I’ll see if I can narrow things down further.

tremblap · July 18, 2020, 11:55am

don’t for now. what is better is to tell me if all your patches still work despite the errors.

rodrigo.constanzo · July 18, 2020, 12:03pm

It breaks a lot of stuff.

Without a defer in the chain, I get random error/invalid buffer~ messages, and creating any new objects wipes all the fits out.

Take a look at this:

(this is not crucial at the moment, but just trying to highlight the bug stuff)

weefuzzy · July 18, 2020, 12:11pm

Yeah, not able to reprodce here yet. Will perservere

rodrigo.constanzo · July 18, 2020, 12:25pm

I’ll try to create a reduced patch, but I’ve narrowed it down to an uzi reading values into one of the buffers that the fit is being called on.

So with the message transformpoint ---transient ---transient.standardized, I’m writing (via uzi + fluid.bufcompose~) immediately before this message arrives at fluid.standardize~.

tremblap · July 20, 2020, 9:13am

that’s ‘sorted’ with a deferlow right?

rodrigo.constanzo · July 20, 2020, 9:32am

In as much as defer “sorts” anything. Yes.

So no critical for the time being (I imagine more stuff is going on), but does mean it’s not useful in an actual performance setting.

tremblap · July 20, 2020, 9:46am

maybe it is the buffer reading clugging that @weefuzzy and @a.harker were talking about… I am trying to find a simple way to reproduce now that I’ve cleaned all the helpfiles that we doing it

rodrigo.constanzo · July 20, 2020, 10:16am

@tremblap @weefuzzy @a.harker

Here is a patch that isolates this problem (which I believe to be related to the other problems).


----------begin_max5_patcher----------
4654.3oc2cs0aiibk94t+UPnmsoq6W1W1MSxlI.6LYajtCFrXRfAsDsMmlhT
fjps8Fj92dNUUTTjRTRTxEY6NMvXygjUcNmu5bqNUwx+i2+tY2k+bb4rf+if
eM3cu6e7928N6sL23c0++ua1xnmmmFUZesYyyWtLNqZ1UtmUE+bk897vf6hx
dHn5wjxf7r4wAKixVGkl9RPd0iwEOkTFGjTE7H7Rk+maZdZRV7770Y19fVey
r0KSxRiqrDDWey6yypxhVFaI1eJN8KwUIyi1zOqhpl+XR1C2VDOuxINXLAEx
uJ.iTgnqBHHl4WBQHJ3uukP4qq1PITKJcez731TOYgkt428aWiIyZ8hkI++1
WDnExb2+46eu4GW8JQSYXPzhEAKhuOtHnJO39jm6ExHdExTHoEwHZCTgwTyu
XuZDixm.DSDFTVEUTY0BAwJHJaQvSFYLXddVYdZ7D.fRovBY7VZbd.+XS.9Q
CAkrpphnrx6yKVB1wQUAkOlTMAvFFwIVDiZMUwLsm.N5D.brvf4OFO+yAuju
tHXQTUTIvkAQEwAOjmuXBfOF0otYwPrF4IvaJ7yQBCRyiVDD+bzxUowAqxSx
lBMNlz4hSHcJbR+fYD0DfY3ZLqVUalmMEcPApEtPd83hzi3xcqqpxyNpX2qf
owsBtQXa94AjrMck6VUurJ10OyLwWl0zn1IFn1hqE.lWEWbabVzco1VdIRZV
7SPGumBvpHvaSimZqICHdX0sUKWEb80WaeTBn5DBQDyVDUr..7E8BXziDKiU
6OAa9E0ELiPNe.qWvhhlMgC8H6PuTxFogdhdpF5MIyTACskmKHvbAUIXaHVg
eGMI3NtThKpkrZQCL6SRi+RbQYBL5ss2e2rnUqZc620pIF732xscj5plakj4
tEt4VEweIYS64M2Mp.j+JP3WW3788rXSRTltIeQbQ15DaO4tILxTyR1w.iOy
xU0NzrCUad7VD05hD4PSWLWEZKdBCvOjlO+ywKZ4mDvyUwYIYqJhKgAwnpZF
u4wPB+QqSqtssaRrwNrmmuweauOrwo+uqHIJsg6enHYQdlgI5LLXt8FxA1K1
zWM+bqvXeirnU8zXPA.vjC7PvCT05x6hJLiR01CjMOrJOOs6iZZWZ78U0OdU
RV1NnXU9pC+vhjGd7Hs8tb3gKOVeaeR4sqybO8VPgn51xnuzEsqf41Vak1s6
eNJKYYTEDocYcXtlG57I7X47h7zzNxq6Ieommr.TvmG+TxhpGsDpsx.75Iq1
nDMqYTdQxCwkUcuWUzCkcuSY0KNPu0sVeWsA7sUwPBYfTz8E.SijxpxGyepr
9E2nn0F.1VQg1Fzsc4049635Cr7JidHtwbsStOrPhRvHx.Xxo.VDv.O6JDVJ
CtFEp0ZgRZLXnRAGqCfDxBQLhlhgYyFp3JtPEvBUTtBKB3gXLmpwAWyCoBNg
qfPEgbLCSX1tCZIzIxPMV.8Rv0fIgRpoJBjYTnjhkDR.EnuVH4PuIf2HT.Sh
QSCvxPImB8poiPXBgSCDPqoLp40Pgx52iAjgKzr.HLqTQ0AjPgTpBtF5X3+W
gglg.oBy.wEy4XAAdENSSof3EpoBFPDCrnIHN7NlWAvfPrPK4bCKgQJFFXTF
AinVFB.PAQaHIvOTooij.OhLbg6sgWhfzLACHlhB3jzfRZtDQgdhnITgExYv
iXBSmSYHfqUgZEhh416.CBlwfPjhq4XCaSf9F3.HsRtvJxVrFEpTFlGdSAkI
sxNVKgqX.6PkDSm.8llafGkfJDtNQnLCeFbAwoFgWp3TBLLFxPRfXFNDtEzR
63DQPDFCHlRA3pQygKA8.fbBAgHnPmSILIxI9ZDWofagMu.QxQDinqTZDfA.
No3HKPCCjTX.ynLBWy4vqAMmSzLqDhzBlxH.bJ.mbCugwRPR.RfAwUYGyjDG
dpwFsFKxnwLaOAiYLrz.RLJhRrinZkPZXBPyVa0LHRJxgnJl1nqhBEZszLrS
.xJXtdh.1JvKIAZHrxE.JDICdKjBTtZay0Y1WXIt0i5jyAo0C5IuCtalDH6u
3tYpRg7w1Fa4fIebhDP1MIDtb1l6aS1pNr9E5ExJfyNjPiFhP6xJf17yKUh6
j14dy5PzlI2Lmw1NssMHaQ7ysR2wKPjiQOHFgGBFAJm3AiRnC.ApWMDrMQGi
Z+ABeYEByy6GaJyWWLeyXVsBIjfXG4ABYWkj0j22utU.Lu3fFaNWtvnfLHtX
C61K7791IIOyjLzhacI9caTUUQBLyLGr0Nq+yJAEHa46hR2Iqh9xe48aYtIY
5i0ybBKUiUgC39d1icSgpU5SHjsJA8BAjiAApVEAPR85jGwzY9aBx2mtNYQX
JjjJ4t02eoSRlaq6kj3W4D6Q4DDt6iK9Z2x8.SNdYT5AK1ywJQBjQhUjkrPH
+VHIMIjwIjvgzTxLg9D.A4P.w8o4QPDhio7iFaToHdw54uBHgIbSGlMdffdr
wfSVHvg.Dt0CDKGQsA0HCDWprCS3vkFqb7jc4LO5oeUQxAj1i3lmnnspQH0y
t+73P6gktiLVR3ZSUfob79t3nhSHqndJ54EIPqWdWbwYqFp0gjCw60IReA5j
GUcjNQkzt.Xq3.S0GCN7vJc.XC3dZ+PWT6X9Q.G5w.G6+0G3v7ospsxqmqoJ
UOdlpbumPV8ZU95bFKE1AZhaYbvrSIzzilI9Us7I2KJH74XbQbzhf+1ra9qk
wEk2TjunH4g7a93KYyu4CE41BneyeLc8uO+mit4C14.UdymLUhN+4.xMeJYY
bvmJfYckdSC7caMn92lM7ELmb7U90.utrdotY6P4dRkRodapSsQpwT9jnToz
uUhxqPigqCI5MRXdMySg4k3oONeuL+HEmWR9NKPuFMgA5kz2HQ5GIyU1z3V9
UTZh5n9p5srvI2GJuRGzR9aEGzh5Zh64QbwaDGzRA0SNnkSuC5dY9wxAs56L
GzRBe5bPS3uQbPORlq5IxA8EVkzZuyBkqHoJ0Hm9L5sh2YlaBCdd3V8VoJYb
gxOdmUeCJSVuL+H4cVw9Ny6LG7SMYdmUuU7NORlqhIx67qY8apcQybajaBdj
SfVcQoiUk+vCowmsn4JWCW75VYdi0Tuk.GOQ11PWWja1jTmq1cM.nqy.4RUt
O7Fa2myOz90Ld9Zu0eJJtsIjvu6aceVsJmC5teDcc9HMNWO1ZmF8fW20KboL
755NtymkReoXdVEF3nErlTa86bsikdU2fIlJbosy8gmT9QPFsaG5PULWLO+B
LTuVB6N1Kio5hh59.vqqiFB42U2V58LAZD0udt9Lgj5BMY0IbqTlldoNNL6i
pSGwmomtukKsKbPsM+HrY7XSa0V7Y3B2lSgSF2vE3ow526NEUD22lu6WTres
+YDua+uZdzWC9u.lZQxx3Ly23VY.4rcE3zFXJrSo.MttBvSnq.EudkpGMWAn
uWcETqqSqWT6QxU.gLItBNyYBerYK4RHhTWJHge2GyT+mBPKg8qW3biItn+X
zHmD.oyBkZeiteKENvX2udgZHYuuZAkp8WsvA9hET5sewB6A6ClRlO6AxonD
2GTRLDYR5CJwGBkX9hR3SQIhOnDdxPOz.zHH9PiPpGhLg7AkDSFk3C.8jTeP
I1.jIoWFmnCgRDeQoSYOIw9fRnoB8DCPgvG9hFhSOgGnCSODAxGdhXCIJH1K
TRN.8NhOh2xFhmHlWnDYHnmWnDd.nGS3KJMMZDCw+.yG97FhabpmnyIUw8.c
n5oRuiJG.kH9.6nCQui3Cu3TzTISjgDvPp7EkNYTceX0RFjwjOxyyNDbxrj8
g8DgLUdhrT5TiSTeX4RFjebejmGdPYq3EJolJcOrbpxg.ODeDDufd3ghdaUP
yKV3Nq0PiOowz9IM90Q5KJmcW4k14viyPhcNz314.ia+CKtCePws6gDm8ruv
clYsSIs1dTqsdQR9GsG3Y29ywYqcUraywxVWf3tGtOIMcddZ9dmUdaJ81L2S
aNW517tlSdNhlgwJywOGESkDg8J3BNu0XTcavaZDiyzHh4UYBhjwsWoHTN0d
N1sSyHaoEBqcT.oUHl6J3V3qBZeN9LKJ6g5C3MYqSorUE4qxKZNo6Bo5l2ec
U9CEQKRpOtVPcNYOtpVOpvbZGtyQU1rk.flroMs06ZFL9o0aNmSu7AfFl6.i
ACDW5NJnZvdkzbBS4tRwn81rlQAAGHh4cEl+IcW45oQYPnmCPVvb7+cUbVvG
ixJC9X7xj6xSWrsXv28PG3QiXXo1JcZNSZ4XEinkVYmKZZWz74.s6zVJATUs
3IaiXaTtM2qkvNKNMd4tMUp.zwAuHrR4tB5EBoaa6zHljio1Wkno.6ZuhiQh
cHXIPw4FTrKIIvKZYRrPRbFSJDBy51XSs8ss61jLieq3FEIplfENZtwBkzVi
vKFD+bTU0wLHZejR9qsbz29nnzroa7FC8geW0s1lg92I6zA2rw0P0credXaS
m322UsUY6wzbX9glMOMFBstsgX2gkpFbDvcMGfGI8H1jZyInG14qfIQ1APy3
GUOPypMmNqt8OXqVriFMwaZz+xm9iGSUtImntpmlH3VmFXBlKbt8LZNV+.5V
CgeiLDFly3u4YMzucve5kEE4ODm8IKDc.yArfBpX1g.ISn00WgzmzZfijBqn
gDbJ10EvvHY.ApXBLF4xASSDLKTw0XvWwQLJvlr8bAPMVBVUD4l79NrJtxap
31EtNY94qlePqwKLS3s9ooZoyKKUW6vvbKN0OpmcDBUSj58u5hTpOpt6OFmE
+knCp0NLAuOsVVsaz9t5v4HMvF8Z8Gq8st5OzZ2E0uF69woTMpGn5qv9ls9C
MmK2ugXpeLJI6qCjqLeFQa7As+U9m0JeJAbrbLlaOUcbsp11eLJ7D4MES8+T
llrXyGZwgRpdX9l6Mt0467nOI9nyn5jzvm30OsYij0OV0GFHNDfsuQam2xmr
MLethjmmWUbldQFOFxrAI+5Yn1cR2GuFsPeJY+47EwkmqNxEEjcfIXL1gL+y
m1ExIbf3Uto0GCb+LSedQ1On4HvZeHYd0owpu4Jvev72ajp2bYa7w44qhOGO
FWVhqMUjQSM+arzE9TzcmqRvPM2+1o47IHa96NYBz8LGCzPCQNhYB8oVeTwC
ePgBSoWhcAiXTrcpgrNqPx2vrf+qKOwBKMgv6uD8kXyt02iSX3hyJDEx8jvs
Hp3yWmY9aHz01JYb9k5f0XKKaJmeSg8UVF1CU0CeB8UuW7CuWQutE+nGeHmc
IfGbQ38hlxuU+Gq.Dp9bBvGcE47021VR98tvekY6HUAF8cll1tKUxcowP7M2
xZ9C44ed1AWEigBC6keN2apbIYe9RzPtpyO8P8Wa5ZXXAoZB1SEHvN63qmv3
4uxyqkvtEbcXKsWuqZ1IQodz01Vre6qSae0ole7PH3Y9964kc.so+.4Gqgdw
L4yt4j8Kls1y25jkRMoRb5Tn2s1HHMqdeHn0Tkr9JoZ.qrjnIYCdiqoMFc9S
tVF8rn4OLlCPl1AfsIrcXCFU2IEIqs6uvEZvaBbq+pfMfzcYDJmn6Fiv3Bw6
iCc9lZ889nn0NhXyJ9u+UGw6sTY1iYW06USXN.Czmumzp5ZcLrQw1+8TanEc
q0d5y6k.v8oI+CoqiuFely9mhk0qn9VmQZu6BplAKxeJ6r4vowIoiC+8uDc9
LHhn4tcEfTRoDm0FACIE3eF7GKhiu.NzhdM0TCKLIH4el6mAOaYUQmM6oXHg
vUsOABXNKTJzXM2+73eIdwYyexlrewMyk.UGwv272+WbZZ9SmODR3tM0hTgU
1T1QbPqjep8SEWiPpwP.HeOJ.1ku3GySu.UjlxdvgLg3t8WT6ffdgAWstXUZ
7YkbJC4xqgvQBsUGln3BhZfYnIa10gLLGbaX8sY7d3USyh34wIe43qbWe5LM
StaKygZuEG8Cy4lhhQs3LP9SpPz6b6XfePcc7CY8NwFLBzdMRRYb1hx2ZQP1
Bx921q2MsJRhQR2ZMwDHc8ldWh8ptSUzpSsStaUeqeJpJO3mLyBcl2neYU78
qSSqN4Brr6NsTw4XWEwETI0so2Ef00wVvKy25hz9phlIEXpfPyNkzqB0YKSD
kT4xyf1nISIZpV4gop+MDOL+E58sz2KvSlp47x4W7SrVibY+A5Yr5kzVfPTb
2MX8NlM+gjhpWB9ueHe1IplExLZXmUKsdP.bdTuxECJZ3.2R8cAVyIg6k.rt
OcN6J3YD1xU0Cg1iop2+Oe++BfVGOwA
-----------end_max5_patcher-----------

You’ll need to load this dataset:
transient_dataset.zip (9.5 KB)

And follow the instructions.

While building this I did find a tangential hang, which I added an extra step to avoid. Try skipping step5 to see if you also get the hang.

Anyways, here is a hang report that I got for that (unrelated?) problem.

hang.zip (127.0 KB)

tremblap · July 20, 2020, 10:27am

so 2 bugs if not 3 here, nice catch

the json not instantiating the objects from fresh instantiation and load is one
I get indeed pca to moan here about invalid buffer and wrong point size…

thanks for the exemplary bug report (except for the curse but hey…)

rodrigo.constanzo · July 20, 2020, 10:30am

I seem to be able to load the dataset fine (as far as I can tell). The fluid.dataset~ transformations all seem to work fine if you look at them.

The transformpoint does complain, as you’ve seen.

Were you able to reproduce the hang? Or is that what you meant with the json thing?

tremblap · July 20, 2020, 4:37pm

i was able to reproduce it all. it now has 2 official tickets.

weefuzzy · July 31, 2020, 4:28pm

So, for the hanging with transformPoint and predictPoint, we can stop this by not attempting to resize when not in the main thread. This means that these will behave like immediate mode if you trigger from metro or similar, and you’ll see an error if the buffer isn’t pre-allocated. [There’s a secondary issue that the error string you’ll see isn’t yet very informative, but that’s an easier fix!]

rodrigo.constanzo · July 31, 2020, 5:22pm

That makes sense. And for whipping up tests and such defer can cover use cases where you don’t know (or care) the size of the buffer~s.

pasquetje · February 8, 2021, 9:23pm

Hello !!

I still have to insert a “defer” before ‘fluid.list2buf’ when using ‘fluid.kdtree~’.
I send a vector every 512 samples… I also get “fluid.kdtree~: Invalid buffer” when using a longer speedlim beforewards.

tremblap · February 9, 2021, 9:03am

This is quite slow for query but that depends on your vector and dataset sizes… but indeed if you look at the innard of list2buf you will have to think about threading in general - there is a uzi in there filling the buffer, the bang at the out should be in the same thread as the one you send in though.

I don’t have java on my machine now so I cannot test but mxj whichtread should help you assess your problem and see if that is a racing issue.

pasquetje · February 9, 2021, 11:51pm

Thanks @tremblap !

A ‘defer’ is not a problem for me. It is also easier than a good old mxj for which I am not sure how much longer it will be maintained in Max and on Mac (and visce versa).
I understand this thread can be an issue when querying sync information. But for me, I simply store the resulting list into a ‘zl reg’ and trigger it with a sync metro.
Not a problem… I personally prefer working with the constraint of another thread and use buffers as lists.

Thanks !!