Precision in CSV output

Bug Reports and Issues
1

I use the CSV output from command line to make files which LUA reads in ReaCoMa.

I ran into a problem with Ampgate today where there are two slices which have the same number in the CSV output (1.30933e+06). In SuperCollider though, with the same settings these two numbers are different. When I use these numbers to slice I get a crash because it tries to slice on the left boundary of a new segment which is a hard no from REAPER’s point of view.

I would prefer if the CSV output just had massive numbers rather than scientific notation. Is this a limitation of the output or a design choice? Perhaps I’m completely wrong here in my thinking. Attached is the data from CSV output and the SuperCollider code that I am testing with.

CSV

9664,42438,77609,106793,144826,191958,286947,322573,357046,379575,450034,491711,544577,593768,653132,675475,725943,767358,817698,840725,868551,890606,912667,934780,959290,981840,1.01967e+06,1.04174e+06,1.07054e+06,1.13702e+06,1.16368e+06,1.18746e+06,1.20954e+06,1.24283e+06,1.26492e+06,1.30933e+06

42435,77600,104297,144820,191956,286945,322570,357033,379345,449858,491706,544487,593735,653042,675473,725933,767349,817689,840723,868544,890601,912656,934771,958649,981825,1.01964e+06,1.04172e+06,1.07053e+06,1.13702e+06,1.16354e+06,1.18573e+06,1.20951e+06,1.2428e+06,1.2649e+06,1.30933e+06,1.323e+06

SC Code

(
~source = Buffer.read(
	s,
	File.realpath(
		FluidBufStats.class.filenameSymbol
    ).dirname.withTrailingSlash ++ "../AudioFiles/Tremblay-ASWINE-ScratchySynth-M.wav");
~idx = Buffer.new(s);
)

(
var thresh = -51;
Routine.run({
    FluidBufAmpGate.process(
        s,
        source: ~source,
        indices: ~idx,
        rampUp: 1,
        rampDown: 100,
        onThreshold: thresh,
        offThreshold: thresh,
        minSliceLength: 22050,
        minSilenceLength: 2,
        minLengthAbove: 0,
        minLengthBelow: 0,
        lookBack: 0,
        lookAhead: 0,
        highPassFreq: 0,
    );
    
    ~idx.loadToFloatArray(
		action:{arg x; 
            ~onsets = x.unlace(2)[0];
            ~offsets = x.unlace(2)[1];
        }
	);
    "----Raw Data----".postln;
    ~idx.getn(0, ~idx.numFrames, {|x|x.postln;});
    s.sync;
    "----Onsets----".postln;
    ~onsets.postln;
    "----Offsets----".postln;
    ~offsets.postln;
    "----Pairs----".postln;
    ~onsets.size.do{
        arg x;
        ~onsets[x].asInteger.post;
        "->".post;
        ~offsets[x].asInteger.postln;
    };
};)
)
2

Now, this will be a fun discussion between @a.harker and @weefuzzy

3

2 Likes
4

@tremblap stop stirring :wink: I happen to know that @a.harker will be practically uninterested in this, because it’s my code that does the printing to stream. That said, he may well have opinions to share.

@jamesbradbury I started to ask why you wouldn’t just use binary output, but presumably this gives you a different species of problem when dealing with slice indices (e.g. running up against the 32-bit limit).

We probably can do something about this, but at the cost of making CSV files larger. I might be tempted to leave it as an option specific to the CLI, rather than change all our ostream stuff.

Let me think.

5

I could use the binary output but it would require me either adding a dependency to reacoma or making my own WAV file reader ( no thanks ).

Also, isnt the 32 bit limit for slice indices around 13 hours?

2147483647 / 44100 / 60 / 60

Not sure I personally will ever hit that limit for slicing or perhaps I dum dummed the maths

6

now now, I’m just missing the office banter between you two in your brotherly-loving ways.

7

Not that simple:

TLTR: 16777216 is the max value for a wav 32bit float. so ‘only’ 6 minutes 20 seconds at CD resolution, which explain that the elderly Max users here will remember using Kit Clayton’s HR objects before Max7

8

This would be 32 bit integer though, but is the first bit for signing?

9

The output now is either a wav (float32) or a csv (which @weefuzzy will confirm the resolution at one point) but under the hood it is definitely not an issue indeed.

10

Whatever the default floating point formatting is for std::ostream, it might even be implementation defined.

@jamesbradbury would you really need a dependency? Does PCM_Source_CreateFromFile not allow you to read whatever audio files Reaper can?
https://www.reaper.fm/sdk/reascript/reascripthelp.html#PCM_Source_CreateFromFile

By the looks of it, Alex’s underlying audio file code is capable of doing 64-bit wav, we just don’t take advantage of it at the moment. So, if you can open audio files from Lua, it might be more robust all round if we extend the CLI to produce 64 bit audio on request.

11

That function could do it but I don’t think its the best option. What it will do is create a new take of the audio file you provide a path to which then would have to be read from and cleaned up. I know Lua can use audio accessors to look at the data but it would mean a significant overhaul of the plumbing I’ve put in. Making and deleting takes is also a pain because you can never guarantee you are going to be selecting the right one when you want to delete which means making some mechanism for finding them. Again, an easy problem to solve assuming you don’t nest your tracks or do anything complicated with takes but it makes more scenarios where the ergonomics are finicky. Can be done though, I think, in theory.

12

Worth clarifying, in case anbody panics, that isn’t a hard limit: 16777216 is the maximum integer that can be represented exactly. Thereafter, accuracy declines steadily. At 44.1kHz, the back of an envelope tells me
6:20 - 12:40: 1 sample error
12:40-25:21: 2 sample error
25:21-50:43: 4 sample error
50:43-1:41:26: 8 sample error
and so on

1 Like
13

So can I be fairly confident that CSV will in the near future contain high precision numbers not in scientific notation? The only drawback I see so far is that it can make CSV files big, but they’re never going to balloon anywhere close to even a megabyte. Otherwise, I will start writing a wav file reader in Lua.

Sorry to be a nag, I just want to plan ahead!

1 Like
14

as usual, xkcd comes to the rescue. Friday’s new addition:

1 Like
15

Not yet

For your use case :wink: I also need to think about what happens if someone wants, e.g., densely sampled features from long, multichannel files.

Hold off on that too, it sounds needlessly painful.

16

I am interested to see how bad it can get so I made my own script for testing.

Even with a theoretical 100 channel file that containers 100000 slices where each slice is greater than the value of 200 million (~1.25 hours of audio at 44100) the size difference between scientific notation and high precision numbers is ~10%. At that point you are looking at slicing a 75GB sound file which I don’t think will even complete in realistic time.

If we double the values and test again the difference remains ~10%. I think that the question isn’t do you want bigger files but do you want the slices to be the correct value at this point.

EDIT:

The output of the files DEFINITELY goes above 1mb in these uses cases (how wrong I was) but the two formats still track each other in size linearly.

1 Like
17

Ok. I’ve put something (CLI only) in that seems to work, so it should be in the next RC release.

18

But for longer files where the precision will start to slip, your lua should still guard against possibly zero length segments (it should anyway, on principle).

1 Like
19

Hmm yes, thats a really good point. I haven’t come across it till now but it is going to be a possibility I haven’t accounted for. Thanks for looking at this for me

20

BTW, I had a small poke at this. Whilst one can call this funciton directly from Lua, without needing to make items / takes, the object it returns is opqaue to the Lua API as far as I can tell. So, if one were using C++ API, then it would be possible to open up a binary audio file directly. But at that point, one may as well make a direct wrapper around the FluCoMa stuff and do without intermediate files altogether.

2 Likes