I forgot to mention this in the original post, but there’s also a big distinction (for me) between online and offline versions of what this means. More specifically, “real-time” (JIT) vs pre-analyzed.
For real-time stuff, a lot of things aren’t available as analysis vectors at all (overall duration, meta-data/tags, etc…) and some things I don’t have a great idea of given the tiny analysis window (pitch, morphology). The latter might be mitigated some by the predictive thing, but that’s (literally) a separate discussion.
But there are some very useful things that can be present in a pre-analyzed file. All sorts of meta-data like names/labels/tags (which @tutschku makes good use of for his searching/querying) as well as some broader metadata like duration, amount of onsets, or “timeness”.
So all of that is to say is that having the perspective and context (,and time) there are more relevant (both conceptually and perceptually) dimensions for pre-analyzed files than what can be computed in realtime.