Analyse large databases and store them

hello,
as the draft name suggests I want to slice and analyse more than a single audiofiles folder and store the analysis results so that I can restore it - instead of redoing the analysis all the time which takes quite some time. anyone here who has experience with and can elaborate on it?
my idea is to load all included audio files folders in a master folder or to select multiple folders from different directories …
what are the limits and are there general recommendations when handling big bunches of audio and analysisdata?
audiofiles with same bit and samplerate won’t hurt I guess :slight_smile:

btw iam working with max

Hi @johannes,

This is kind of what fluid.dataset~ is there to help with: because the write and read messages let you save analyses as JSON files, you can build up a collection without having to redo work.

Definitely having uniformity of sample rates can be important; the word length not so much. You might want to think about how (if) you want to handle multichannel sources as well.

What’s probably easiest at the moment with dataset – if you know that you’ll have a bunch of different categories of sounds – is to keep some sub-sets that you can merge if needed. Currently it’s easier to combine datasets than it is to filter them.

Another thing to consider is a convention for the data point IDs: using the source filename in the ID can be really useful for being able to get back to the source audio (e.g. for playback).

1 Like

thanks for your detailed and helpfull reply @weefuzzy !
so i would keep analysis data seperate if i would like to treat/process lets say
analysed animal sounds differently than car engines– instead of merge the whole datasasets right from the beginning. that makes sense.
regarding the datapoint id i just realized that i have to lean more about that and definitely also about what kind of dataset processing is possible beside merging.
what i am curious is if there is already some abstraction for loading/slicing/analysing multiple audio folders just to get started and expand on it… maybe in some tutorial/helppatcher?

i found a great intro on this topic of large databases by @tutschku
(or at least how he uses it for his work)
https://tutschku.com/composing-with-large-sound-collections/?lang=de
but as far as i can say he doesnt mentioned how he does the loading folders part…
if there is anything else related i would be happy to learn/read more about it.

Does the example code in this post help?

sure, thanks for pointing out.
in your patch you use a polybuffer to store audiofiles and address them separately. can polybuffer also handle a whole library of with lets say 50000 or even more audio files this way…? i ask because i assume that loading audio into a max buffer or polybuffer (from internal or external hd) basically loads audio into ram. so the number of files is limited to size of my ram?

back to the original question:
how would this “load folders in a folder” thing work?
i guess i need to ask for all folder items in a directory and store their name/path.
after that is known i can iterate their path and load the contained files (more ore less your patcher) one after another, right?

The limits will be a function of your RAM but also your free disk space (as once RAM is exhausted, the OS will swap out to disk). I don’t know if there’s a count-limit for polybuffer~ (which would depend on the size of the integer they use to index), but I’m pretty confident it would be > 50k.

So, that’s maybe an issue of recursively walking a folder tree, and the complexity depends largely on how complex the folder structure is. If you only need one level of nesting, then it might suffice to use something like dropfile with its filetype set to FOLD (iirc) to get a list of subfolders that you can then use with the readfolder message to polybuffer~ to load all the audio files in that particular directory. If it’s more complex, then @a.harker 's AHarker_Externals package has a recursivefolder external that will maybe help:

1 Like

For binaries of my externals go here:

1 Like

nice one! recusivefolder does exactly what i want… thanks.

in case anyone here is interested or wants to use it …
here is my expanded version of @weefuzzy s demo patcher that now does the load folders in a folder thing using @a.harker s recursivefolder object. I just tested it with a ca 15 gb audio archive, ca 25000 wav files, 4 levels deep. all files properly loaded into the poly buffer… next week I will test it with bigger archives and see how it works. prost!

----------begin_max5_patcher----------
1852.3ocuZkzaaiCE9ryuBBidzUQjhTK8TGL4PGfYAClomZKBnkYhYf1fVRS
5f1e6CWjcrqWDizKw.wNhR1jeuku2B4+cwr4KKePzLG8NzmPyl8eWLalYH8.
y5ud17b9CoY7FyiMOsLOWTzNeg8dshGZMiWK3qPq3s7Fw1aVwaSWKKt85ZQZ
qcRXrDO1BTDIvyeABy70ePTui9R+WpnKWVjIZMSG9oAK6Z2Lpe+nxUlotb4c
ukRlqG66Wbg9sESDLesV1JbGMgIDfQi+XPStnogeq3nplMCZm21GqDVDLe91
k5wvGNbGsU.y.OxofG4nvCeH7BHPBOixZR3qW+EDBE9XPZLtptrBwQ7tUxxa
jYBzMkYqD0nxZ0nqjZzHKqeD8UY6ZTdWVqrR8P+7i2fVKpEnM+5YxBQZYWgY
JBNsDBiiLhlHqANV+AMZxF33QYATH9p5Ken.RbinNq7qiyFfXbgoAXOFye6K
bLYAhQOuAA1UCBPs2adLeYYF5M3wgWpuwLmEZ9HL.Ja9P.0nU0hJQwJTSaY8
XcsowZ8JiZrbiC.RSFPADleKC0jISEnynJWLDN8wF6WV7QreGEgM4H1uwuDn
lLcTmXHjfCmQ.hyZ0bIPVu0wiTZTh0OMwKZW8KlphKGAEtofi6UkEhIqeY3X
C3CABmPpdaQKQYnzLAu9j3bIu3VCVU+YexyAZrOUCZbnIrCKZLTVAGQ4BZzm
OO+O3oxh1xl0nOb06t7iMpbKt7tKupLsSm9Ryk+ye8w+7J06+909dIW+g+9x
+stKuRs917Ou8WeaZW0k2jwaaE0eqq3Vwked93n42HyHFoEKvbE.gyBA2gvl
F13cIvpjv0XKvjqZxgtD2Tpj9xuY9Aw5G5L3W+rE7byyN+Wpk7sLTC5CwRfT
zHR6paj2KbP5H0Iq5f.x2DS.SLw+I3SIgTDLIJ9kvjMuNmS1DjW3QE6TmzuN
68IXuX8LLDn3HSDRrO64wlbD3D5Cn5OsLafPiKdFgHUF8I6Dhzmx1leavorB
BNtexyUuSORM7QadzF98hUWqFSshuVw4UKWpHDZ5EW8xqYyE4KEq1odoY5zf
SkMxxB8nglAMxcnprjWvydrQfD7z0nkc2nJiBIKPsc0EmooGjXCCjRRCaSOv
9PV07ccMsJN2Zj3AdtpfXOOugwTnpZAVxNu7SfDgwPBvNkhqcsJthhkPahh3
Mne6Js9a39V0C2XU5U3cPajOUGSACDbAsKHlRA2CvCiuHaweDJAHDE.H22MY
cxUdZuNSdPE+X7T8wlDFCiwdD5SZy33PUdB9iIKxiv6SgOiHKiyTgcDl5Eui
QbbntxHefpXHHFxjmqpkOYZ+baaiwikP2yiU4Au.YyHDhd3.Y8Q2UJKFGT60
q5F3rq4LEqLmiABpjPPKELCkMU6XVPfWHcW13nE8MmD.2WB6En8bC4.6.niB
OLms3PnZ6Z.n4pluTVnBAUykEJr3gJPuWkK2s2patNdxp+9zoh.qWVX.Ae22
jC1jiMeNTcZ8MbOztWBihn9Xc4.zsLSk9Tkh+pE8FxnaztUuFi8OjwNlBDMF
N4kHijlVdayOPuuQjoPCJWvKlbxIGMy5ws6CGqH6nWD4PkBE07rl07JgRdnV
Foq4E63su21nQlpLxeexPLSKir6JYvzqbGGBtLpuRierkT75cp6XrhgXVjW7
NABiRzM3iFBThrXF3IxZr.PYxl1galgCLi1FXEOP+eCbkYDSgjY7VQqAtKPp
+qYcYso+HKPq5xqlxFvF32WkMTaHIF+BjwCuR+wHwoBgZfRRNQCbGWRNPFAn
pL6QaRcO4ReVSZSLeGTtQFOaLFCkS7n6s1gcQyrTLL4+zAgx.G836KHaJ6pS
2HD528rEnmPzJQSqrf2122tOssfCk..8kiprbdlX1ejyNSAg6sbxkqLoxz7T
agMa+uuW.gseh2TLyrqaGduottwtHgX1GZ5RnAmo..loDGlH74zDDru8TGYD
5AIauXhKrHWrQn.HAB8cPDruwXYsdWdzd5uNyL9jtA3dFI6A3yRSYt5v0p+z
VqtXPR.PcvRbPwm.vDo2YiggDDVXTWDdAw.LSa9QFlCexyjKROLDbgAtH81n
Lm1Lwd0vD8UCSjWKLQhcvwk.AlHtX6AxD4hoGFBxnMwxO+LAgiqgjl3no2Ih
5vrGT7XUv9nc6sspPeZr4.Kbj6AcDICPFxdCGCkHaPkC90R4fcU2nRDq+DOu
2UAg1qR1dAzI1P7cI4MP7P8cPfYxa+j4PEZxek12tdps4E5ql5RykPw9.HCv
IuVT7XWRufAAeH1EJdJD7gXm33gvVEScI.Y747tiLsUhRh7h18Xzlf0GxT7I
tGDK6AMjgP9fcuVzoMQtLOGl3usCK7pp6E0aNNSl4XdN+tRCsY7BykxB6kl1
XNuVb+1i+jcDdc5ZYqHssq1dlrdHjN29UKUDvEcxdNXE5TSoogW59S1Tws.w
zWrK99E+Oh..oBA
-----------end_max5_patcher-----------

I was about to jump in - a less elegant but native way to do it is via umenu. i can send the patch later if that is of interest of this thread.

1 Like

ok dirty code because I have not time to edit, but it does load a folder of nested folder and produces the inventory as a file (which I then use to load in SuperCollider :slight_smile:

Click at the top, then look at the [text] at the bottom. filepath followed by index (frame) and nbchans.

I then write a first line with the total framecount and the maxchancount so I can resize a buffer to hold it all. I regularly load large soundbanks that way.


----------begin_max5_patcher----------
1371.3ocyYszjZiCD9L7qPkqbZKXPOrMvdKG1TU9Cj8vVolRXDy3I9UYKOgT
ox9ae0CCSlAIrLH7NGv.pab2ep+5GV7yoSB1Ttm0D.9Sv+.lL4mSmLQsjbgI
ceeRPNceRFsQoVPNqog9.KXlVFmsmqVewWJyZEBWjSSdLsnYQUc4SLdyhOWm
VvVzTJVpImlkUzle3GmtU8SK27z7vCqIDmVjI9gRQ3tEqnb4M8g6qYIbs2hi
H2AmAPpq3nH0a36ffu9xMprke3Ng5VUuD+GUL8sIH.7UojeMcp7xLG2EJXeW
30mrIvAb5FPpA7EaDeH63KhfjPJZcr7MxpdvG1F9DNTvLPPZA2qHMorXWYct
vweDHzGrorja.13gB6v3kZ7BUQW7Ei6qBz1H4euNkyLwdgCk9Rzz2XTnBtgu
Knux8qSAGY0PAWDb8cQy.gKwJXo9xk.NeSZ4fMfFvFSXb4PIpHrJlsRU3ID0
S7iXChanEOHopxWpO6S7VUypXEaAkhql.8vKJEgtaoHXRfJXuhLtzVaokMLN
.ZBfQCk4tTSYCWqqB89nqBMIwXOSBwH7H1gWbnpdZDQ0TI7hq53+DyclSKGb
+iNDFpyK6szi01G6xJo7aRRIaeUM3C6Pf+PbECV.PPH7NSfGMT1aLRydQw5l
mv2ICEg.IYLZs4PLbng3kPEtH5ry3KtzqpamXRFoqccQ5MsbdooJr3isUpn0
zbFmUeOqftIS4CvKC3Hjp3jt+i75Pit9uOCi9MSnmL3AG55uzwfCgWYr82dy
m3sYWpX52+0DjgWbKUMcdIpmJVwtA4i0uN9A0KetKzVTQS9FHEzXXi.M3ApB
Wph43UpD5XxkN3e2FfmKgkTCxLAyvACynt7V8DFnKElI09GkxOX.ijAiQhpu
arNHd4YwGqJeCxgqEViA11lWYBx3gV4JLRiRzZE6c807Xq9c3XGNXFS6.vK7
.Y5FpDhdWL6g7Ie1lRyJe.fLfxUC9o9zUo0fb4xqIJeCF2vqSafzSR2c.a+O
LrgMBMskWVUV0lQEIvnY.wS4tKc+84kaYBP18UvGDRrjcO3wR5h5HrduHdjm
qVj8VzZJG0kv8VVE+QkScXkee+6Ub.0F2fOL2EmMGRB9tyWN3ie9SeRx7+6O
9k+J33V2Y1u0y1S58zwNz.IkyxOXL77mKSSXMy2TJxVxmSghRRem9rzAlIuX
PAbeJP5Sgv9THxrBhHxbqNnVH9bBImQHY04Dt9UB2HIcr54PzItyqDgsKhXW
TncQQlEgs6FX6tA1tafs6F3SbileTveT5eYkEObhm7Vo3yJkbVogmUpQuBay
gv17ErM2.ayCrtkPrY7v4E46tunLsg4lJ39UgzuJg8qhIbfHPLhbOBQPV71W
qAtWMH8pQXuZn8zi05r9DNmLpn5mDjIJQ+l+ROUMV45utKSSYacxg6Z27PfW
Jxtk0vSKn7TwvFuni7nxkJYrQlqFZoCFZkGrSrC1Q97MdwPnwvPx+powARgi
EYf3BsK1GFxEZ2a7lx5srZ0DMifoIlMM75McebShu1dw8XHrWXLtjWig9vRQ
CHtccVhLVYAXWgDxGVBMJadHWvD1GVB5.I+.q45sTujbuDmfiU2KW33HOXGr
Ska8QABLw0dJWaTBCcu.geadgchy6k3lzRDWrznrahuIMjwi0DbHm57GeKvn
SlFeKHqx+Rl9Kl4ioPPgiVfzIJSnOrDbrl5G4PMEzocr0OiKsp5YVcSm1JaD
jSepTweVMS80zB8WUmNXPM64zC5qWgVm7XJmkvaqU9Uv9X8+lWf7njqKZS6H
iBzILo57XKn4rlJpFHpisc5ul9e.yN4Qt
-----------end_max5_patcher-----------

thanks @tremblap
seems that this patch doesn’t work here (mbp m1 2021 - monterey)
yes it fills the the text object but not with files of multiple folders. also the open dialog just eats single files?

something related. I know that some here tend to concatenate multiple
files in a single buffer and store slices into another single buffer. @jamesbradbury corpus explorer is based on that idea. is there a specific reason to not use polybuffers instead- especially when loading a huge amount of sound files?

I made a mistake and always used the path included. Change [opendialog fold] and it works like a charm

yes and no. It all depends what you want to do after with the sounds. The one-buffer approach has advantages, the polybuffer approach others. In my case, I use the former in Max because I like in these moments to segment across the boundaries of sounds. I use the latter in SC because I use it more as a ‘sampler’. you can do both in both it is really flexible. This is why we have the identifyier as a string, you can encode however you want what the numbers refer to…