# Additional statistics (linear regression)

**URL:** <https://discourse.flucoma.org/t/additional-statistics-linear-regression/562>\
**Category:** New Ideas\
**Created:** [July 10, 2020, 10:53pm UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562 "2020-07-10T22:53:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [July 10, 2020, 10:53pm UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/1 "2020-07-10T22:53:03Z")

</div>

Based on many recent [discussions](https://discourse.flucoma.org/t/audioguide-talk/550/17) around [AudioGuide](http://www.benhackbarth.com/audioGuide/) and how things are [implemented](https://discourse.flucoma.org/t/weighted-averages-for-spectral-descriptors/556/15) [there](https://discourse.flucoma.org/t/converting-log-to-lin-and-vice-versa-for-stats/558/10) I got to experimenting with linear regression on every frame of a loudness analysis. I go into a lot more detail in [this post](https://discourse.flucoma.org/t/dimensionality-reduction-mixing-the-rational-and-irrational/534/20) in [_timeness_ thread](https://discourse.flucoma.org/t/dimensionality-reduction-mixing-the-rational-and-irrational/534), but I wanted to make a separate thread to act as a feature request.

To summarize, I did some testing and comparing between the slope of the linear regression function (from [this javascript implementation](https://stackoverflow.com/questions/6195335/linear-regression-in-javascript)) and in terms of “accurately” representing the morphology inherent in a time-series, it appears to perform much better than the mean of the derivative (of loudness).

There are more examples in [the other thread](https://discourse.flucoma.org/t/dimensionality-reduction-mixing-the-rational-and-irrational/534/20), but here are a couple just to show what I mean.

Pretty similar results here:  
 ![Screenshot 2020-07-10 at 11.21.13 pm](https://discourse.flucoma.org/uploads/default/original/1X/4484469c527f8945e01d5f5c218c2ec363251efa.png)

Here we see a bit more of the difference:  
 ![Screenshot 2020-07-10 at 11.21.04 pm](https://discourse.flucoma.org/uploads/default/original/1X/7a957624c38b9a4aeefb9f533414ab26d9a6498a.png)

_( **edit** : the results aren’t always this dramatic, turns out I had messed up some of the maths for the derivative, but they generally are better, and you get the benefit of a corresponding r2 value)_

Although I didn’t implement in my Max version (yet), there is the r2 output, which can also act as a kind of “slope confidence”, which we also don’t get from the derivatives.

So it would be great to see linear regression as one of the statistics available in `fluid.bufstats~`.

(I’ve implemented it in Max already but it’s a bit faffy to code, not to mention the workflow problems of dumping the contents of a `buffer~` out into list-land, only to shove it all back into a `buffer~` and then onto a `fluid.dataset~`. And then having to do this _per descriptor_ that you want the values for…)

---

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [July 11, 2020, 9:01am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/2 "2020-07-11T09:01:12Z")

</div>

I’ll bring it to the table and see what the wisdom says…

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [July 11, 2020, 10:22am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/3 "2020-07-11T10:22:18Z")

</div>

Ah turns out you can compute it on the dB itself too, which is even better. For some reason I assumed it had to be computed in a 0.0 to 1.0 range.

The resultant slopes are slightly different as one would imagine.

---

<div class="post-metadata">

**Author:** ![b.hackbarth](https://discourse.flucoma.org/letter_avatar_proxy/v4/letter/b/8e8cbc/32.png) [@b.hackbarth](https://discourse.flucoma.org/u/b.hackbarth)\
**Post date:** [July 11, 2020, 10:55am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/4 "2020-07-11T10:55:00Z")

</div>

When I was working on a method for descriptor linear regression for sound segments, I remember that someone (Norbert Schnell IIRC) was adamant that regressions be calculated on only the ‘steady state’ portion of the sound, excluding the starting and ending descriptor values. FWIW, in practice, we ended up taking only descriptor values from 20%-80% of the array to perform the regression calculation. Also, FWIW, I haven’t ended up using these all that often, and can’t really comment on their usefulness for similarity calculations.

> def descriptorSlope(descriptorArray):  
> # with Norbert Schnell  
> start = int((len(descriptorArray))\*0.2) # only take descriptor from %20-%80  
> end = int((len(descriptorArray))\*0.8)  
> values = descript[start:end]  
> if len(values) \< 2: return 0.  
> cmpLine = np.linspace(0, 1, num=len(values))  
> return np.polyfit(cmpLine, values, 1)[0]

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [July 11, 2020, 10:57am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/5 "2020-07-11T10:57:28Z")

</div>

> [@b.hackbarth](#):
>
> FWIW, in practice, we ended up taking only descriptor values from 20%-80% of the array to perform the regression calculation. Also, FWIW, I haven’t ended up using these all that often, and can’t really comment on their usefulness for similarity calculations.

While taking a peek the other day with @jamesbradbury we saw the 20-80% thing. Hence the experimenting with the option to drop the first frame in my examples. In my case this corresponds to 6ms of audio, so the slope is (ideally) giving some idea of how sharp the transient itself is, but I can totally see the usefulness for longer segments to look at the “steady” parts.

---

<div class="post-metadata">

**Author:** ![b.hackbarth](https://discourse.flucoma.org/letter_avatar_proxy/v4/letter/b/8e8cbc/32.png) [@b.hackbarth](https://discourse.flucoma.org/u/b.hackbarth)\
**Post date:** [July 11, 2020, 11:04am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/6 "2020-07-11T11:04:41Z")

</div>

Yeah, omitting part of the array as a time in seconds (rather that a percentage) might make more sense here.

BTW, I cannot see some of the links you posted in your initial post (“Oops! That page doesn’t exist or is private”). Maybe the discoursebot is still getting to know me. However can I earn its trust?

---

<div class="post-metadata">

**Author:** ![tremblap](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/tremblap/32/5_2.png) [@tremblap](https://discourse.flucoma.org/u/tremblap)\
**Post date:** [July 11, 2020, 11:20am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/7 "2020-07-11T11:20:30Z")

</div>

you can if you ask nicely… you are part of the research team indirectly now it seems 🙂 Try again.

---

<div class="post-metadata">

**Author:** ![rodrigo.constanzo](https://discourse.flucoma.org/user_avatar/discourse.flucoma.org/rodrigo.constanzo/32/12_2.png) [@rodrigo.constanzo](https://discourse.flucoma.org/u/rodrigo.constanzo)\
**Post date:** [July 11, 2020, 11:38am UTC](https://discourse.flucoma.org/t/additional-statistics-linear-regression/562/8 "2020-07-11T11:38:11Z")

</div>

> [@b.hackbarth](#):
>
> Yeah, omitting part of the array as a time in seconds (rather that a percentage) might make more sense here.

Or maybe a combination where it’s a fixed % unless the segment is above (or below) a certain duration, in which case it defaults to time values.

Or to get even fancy-pants-er can have it automagically find the “steady state” via a piecewise linear approximation, so if there are extreme(ish) trajectory changes at the edges, those get disregarded, and if there aren’t, it takes all the frames.

> [@b.hackbarth](#):
>
> I cannot see some of the links you posted in your initial post (“Oops! That page doesn’t exist or is private”). Maybe the discoursebot is still getting to know me. However can I earn its trust?

I guess @tremblap bumped you into the dev group, which is where some of those point (as it turns out).