Fractional loop playback

[hburgund]

TODO:

• determine when the next loop point of each track is coming, if it is now or somewhere very close say in 100ms in future or past. then update the buffer and play it now. otherwise if it's end is in future, start a timeout, end it then, and then update the list with null. this might lead to some empty space. (time waiting for next loop point to come).
• rather than updating a certain fraction of the number of loops each time, we cycle through all loops that could be updated and only update ones based on a probability. Let's set that to 50% for now as that might be convenient for testing, though I will want it lower in deployment live most likely.

This is a feature add for the looping playback system, newly built to handle recordings made using the looping recording method.

Would it be possible to change some (non-base) loop lengths to fractions of the base loop length? In other words, could we play another loop at half the length so that the first part of that loop played twice while the base loop played once?
Or is there something fundamental about this playback method that requires all loops to always be the same length and sharing the same consistent loop point?

SJ - yes, its possible. if the some speaker is exactly half the length, then every 2nd loop point of this speaker would be at the same time as loop point of base speaker.
FYI, only the loop points of base speakers are considered for invoking the algorithm. (other than location change).
all the speakers are started at the same time, but they are looping individually. so if there’s requirement for same loop point for all speakers then i would say its necessary to have them of same length.
although nothing is stopping us from passing different lengths of loops. the but if lengths are different and doesn’t even have any common multiple, it will also sound to be out of sync musically. currently there’s no mechanism that prevent this. if thats what you wanted to know.

Thx for this. What we want to enable is to change the length of the loops to common fractions of the original loop size so they come back together at the same loop point eventually. It’s easier to think about in terms of musical measures/bars rather than time. If we have a base loop of 8 bars, it would be really great if we could playback other loops at 4 bars and 6 bars and maybe even 5 or 3 bars as well, but not any other random lengths, all common fractional lengths.
Glad to hear that this is possible without major changes to how you have built it. I’d love to hear your thoughts on introducing another parameter to our playback engine that modifies the loop lengths like this. I think limiting it to [3/8, 1/2, 5/8, 3/4] fractions is a great place to start. I’ll think more about it, but some randomization mechanism where each time new speakers are added to the mix, there is a chance that they play back at one of these different lengths instead of their full length.
Do you have any concerns about the loops drifting out of sync over time if we do this?

SJ - OK… sure. we can add the param. i think there was a length param earlier but it would not be helpful for the approach suggested here.
limiting it to [3/8, 1/2, 5/8, 3/4] fractions is a great place to start
So, if a new loop is selected, and random selection comes as 1/2 then i set the length of that loop to 0.5 * originalLoopLength or 0.5 * baseLoopLength ?
Do you have any concerns about the loops drifting out of sync over time if we do this?
i don’t think so. although i can’t guarantee, there can be few milliseconds if played for hours, it depends how code is executed on low level, but i guess it should be reasonably accurate.
if the it is not something like 0.333333… chances are less

Shouldn’t originalLoopLength and baseLoopLength be the same, so this would make no difference, at least initially? I guess maybe you are asking if it happens more than once to the same loop? In this case, we definitely want to use the static baseLoopLength so we don’t keep on getting shorter and shorter. In other words, each time a loop is shortened to one of the available fractions, the resulting length is that fraction times the static baseLoopLength . If a loop has been reduced to 0.5 for a little while, if it is to have its length changed again, the new length will not be a fraction of the 0.5 length, but rather will reset and then take the new fraction fresh from the full base loop length.

[hburgund]

@shreyas-jadhav I am trying to test this on the Test Project in the dev server and am not getting any audio from any speakers at all for some reason. I have run npm install and am using your most recent code. Not sure what's going on. LMK if it all works fine for you and if you have any suggestions. FWIW, assets are playing and the speakers are all displaying on the map, just no audio from any of them, either old ones or new ones, AFAICT.

[shreyas-jadhav]

I just checked the 'Test Project'.
I think there is an error parsing the GeoJSON shapes.

Uncaught (in promise) Error: line must contain coordinates
    at lineStringToPolygon (roundware-web-framew…v=1770bd60:10232:11)
    at lineToPolygon (roundware-web-framew…v=1770bd60:10223:14)
    at convertLinesToPolygon (roundware-web-framew…v=1770bd60:32035:40)
    at new SpeakerTrack (roundware-web-framew…v=1770bd60:32063:39)

Turns out some of the speakers have attenuation_border as type: 'Polygon' but has 0 coordinates.
I checked all the speakers available and specifically these ones have this type of data.

maybe we should automatically be able to ignore them, and also hide from map, and any other usages.

[hburgund]

ah, that makes sense...thx for checking. Good idea to handle this issue better. Ignoring and hiding them is a good idea, but we could also do better checks when the speaker is saved and throw an error then if something doesn't look right?

I actually had a hard time adding the speakers using the react admin for some reason. Not sure if this was indicative of the problem or if this is what created the problem, but I'll see if I can figure anything out there as well.

[hburgund]

UPDATE: I've had this issue before, I am now realizing. When you make a speaker, it is possible to set the attenuation border larger than is physically possible (eg you create a circle of radius 10 and set the attenuation border to something larger than 10). In these cases, no error is thrown, but it seems to result in an attenuation border shape with no points which is what causes the issues.

Can we add an error check on this somehow so that if this ever happens, it won't save the speaker and lets the user know why there is an issue?

This to me is much more important and useful than doing the ignore/hiding, but both would be the most robust, for sure.

[hburgund]

I believe you added microfades on the looping playback at the loop points to avoid the clicks and pops we used to hear and this is great. Can we add those to the new loop points that now exist when playing back a loop fractionally? I am hearing pops, so I assume this is not happening, but lmk.

[shreyas-jadhav]

sure. will add fades on start of loop points and ends.
or I think its better to modify the buffer data to have fades in built, instead of adjusting volumes by checking on every loop point.

[hburgund]

or I think its better to modify the buffer data to have fades in built, instead of adjusting volumes by checking on every loop point.

Great...thx. Totally fine with putting the fades into the buffer data; sounds more efficient. We would still need to re-do this each time the fractional loop length changes, but that's a lot less than every loop.

[hburgund]

Doing lots of testing and have some questions/comments:

• should I include 1/1 in the lengths array if I want full length loops to be able to play sometimes as well as the fractional ones?
• I think some of the reason I have thought there might be weird loop points being set is because a full set of changes is happening at each base loop point, right? I am feeling like this is too frequent, so if we could introduce another config param like loopPointUpdateProbability such that things update, but not at every single loop point, that might help.
• Relatedly, at each base loop point, what is being updated currently?
  • which tracks are playing?
  • what fractional loop length is selected for each loop?
  • anything else?
• I would like for the changes to feel more evolutionary than sudden, so I think considering how many updates happen at exactly the same time is important and to minimize those. @shreyas-jadhav lmk your thoughts on how to possibly handle this...maybe new percentage config params or maybe something else?

[hburgund]

Made a diagram to help me get my head around the fractional looping and the changes over time. Not sure if this is exactly right, so lmk your thoughts @shreyas-jadhav

[shreyas-jadhav]

Relatedly, at each base loop point, what is being updated currently?

at each base loop point, first new set of tracks are selected (some may include previous ones and order varies)
and for each 4, the buffer is recalculated
with trimmed length, microfade & effects added.
and started at the loop point.

But now that i see the case for 6/8, there are two options:
option 1: determine when the next loop point of each track is coming, if it is now or somewhere very close say in 100ms in future or past. then update the buffer and play it now.
otherwise if it's end is in future, start a timeout, end it then, and then update the list with null. this might lead to some empty space. (time waiting for next loop point to come).

option 2: ignore all tracks, i.e. keep them running, until their loop point is not coming exactly at the loop point of base loop.

[hburgund]

@shreyas-jadhav thx for this explanation. I think we might want to change a few things with this approach in order to have the audio evolve more gradually and less suddenly at loop points. I haven't tested scenarios with a lot more simultaneously available speakers, but if I understand the above correctly, in a situation where we have 10 available speakers, it would be possible that all four non-base loop playing speakers could stop and four new speakers could start at the same loop point?

Assuming this is correct, I have a few ideas for how to make the changes more evolutionary and less sudden:

• at each loop point, only one available speaker slot is re-evaluated instead of ALL speaker slots re-evaluated and potentially changed
• introduce a probability factor such that at each loop point all tracks are re-evaluated as currently happens, but apply a probability up front such that, say, if set to 0.5 each speaker slot would only have a 50% chance of actually making a change on any given loop point

LMK your thoughts on these ideas. Feels like the second one would be more flexible for aesthetic tuning, so I am inclined towards that at first blush, but maybe I'm not thinking of all implications. Also, very happy to consider other ideas for smoothing out the changes over time if you have any.

[hburgund]

But now that i see the case for 6/8, there are two options: option 1: determine when the next loop point of each track is coming, if it is now or somewhere very close say in 100ms in future or past. then update the buffer and play it now. otherwise if it's end is in future, start a timeout, end it then, and then update the list with null. this might lead to some empty space. (time waiting for next loop point to come).

option 2: ignore all tracks, i.e. keep them running, until their loop point is not coming exactly at the loop point of base loop.

Which of these feels like a simpler or more elegant approach to you from a coding perspective? I think both could work, but if I understand them correctly, Option 1 is what I indicated in my diagram? That, as you say, might lead to some silence between when a fractional loop ends and the next speaker loop is added, but the base loop will remain playing and I think this would be fine. Option 2 seems to me like if we had less common fractions, say 7/8, we could have to wait a long time for them to come back and align with the base loop.

TL;DR, unless I'm misunderstanding, I lean towards Option 1, but lmk what you think. Thx.

[shreyas-jadhav]

it would be possible that all four non-base loop playing speakers could stop and four new speakers could start at the same loop point?

i think it's unlikely. as you said how, number of chosen to replace are determined by this:

// determine how many to replace
    const amountToReplace = random(1, max - 3);

[shreyas-jadhav]

I think we should have a flowchart like diagram to visualize the algorithm, and we can collaborate on that instead. so we both could have easy understanding.

I used mermaid live for editing. But you can edit here on GitHub as well if you want to suggest anything.

flowchart TD
    A[Location Change] -->|Determine Base Speaker of all available| B{Is Base Speaker Same as previous?}
    B --> |YES| C[NO CHANGE] 
    B --> |NO| D[Select as currently playing] 
    C --> E 
    D --> |"[BASE, A:null, B:null, C:null, D: null]"| E[Wait for loop point]
    E --> |"[BASE, A,B,C,D]"| F[Loop Point]
    F --> G[Get current last N-1 speakers]
    G --> |"[A,B,C,D]"| J["Shuffle"]
    F --> I["How much to replace? ATR = random(1, 5 - 3)"]
    I --> |ATR = 2| K
    J --> |B,D,C,A| K["Replacement Algorithm"]
    K --> |"ATR=2, Shuffled=[B,D,C,A]"| L["let i = 0;"]
    L --> M{"i > ATR"}
     M --> |"Loop finished! Result = [C,A,E,F]"| N["Replace last N-1 from shuffled"]
    N --> |"[BASE, C, A, E, F]"| F 
    M --> |"Evaluate i"| O["Remove first element from Shuffled (Dequeue)"]
    O --> |"[D,C,A]"| P["Random Number"]
    P --> |"0.45"| Q{"Should replace with none?"}
    Q --> |"Yes"| R["Push null"]
    R --> |"[D,C,A null]"| S["Remove this element from available, and repeat"]
    Q --> |"No"| T["Push any random From Available"]
    T --> |"[D,C,A,E]"| S
    S --> |"[D,C,A,E] Here only E is new but after the loop is finished D will also be replaced!"| M
   

Loading