Skip to content

Aaudio diff tolerance param #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AdamVenn merged 3 commits into from
Jun 1, 2026
Merged

Aaudio diff tolerance param #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f5d0cbd
Readme - use slash for 'or'
AdamVenn May 29, 2026
907f0b4
AudioDiff: add tolerance parameter and simplify output
AdamVenn May 27, 2026
2d7d26e
AudioDiff: tolerance in dB
AdamVenn May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 26 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ writing the processed audio to an output file.
| Option | Description | Required |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| `--plugin=<path>` | Path to, or identifier of the plugin to use. | Yes |
| `--input=<path>` | Path to an audio input file.<br>To supply multiple inputs, provide the `--input` argument multiple times. | Yes, unless `--midiInput` is set |
| `--generatorInput=<path/json>` | Path to a JSON generator config file or a JSON generator config string. See [Generators](#generators) for specification.<br>To supply multiple inputs, provide the `--input` argument multiple times. | Yes, unless `--midiInput` is set |
| `--input=<path>` | Path to an audio input file.<br>To supply multiple inputs, provide the `--input` argument multiple times. | Yes, unless `--midiInput` is set |
| `--generatorInput=<path/json>` | Path to a JSON generator config file or a JSON generator config string. See [Generators](#generators) for specification.<br>To supply multiple inputs, provide the `--input` argument multiple times. | Yes, unless `--midiInput` is set |
| `--midiInput=<path>` | Path to a MIDI input file. | No |
| `--output=<path>` | Path to write the processed audio to. | Yes |
| `--overwrite` | Overwrite the output file if it exists.<br>If this option is not set, processing is aborted if the output file exists. | No |
Expand Down Expand Up @@ -164,21 +164,29 @@ You can pass the path to a JSON file or a JSON string.
## Compare audio files
The `audioDiff` command takes two input files, compares the values of each sample and returns the RMS of the difference. It can be used to compare the output of two plugins, or two versions of the same plugin for regression testing.

| Option | Description | Required |
| -------------------- | -------------------------------------------------------------------------------------------- | -------- |
| `--test <path>` | Path to an audio file to compare - the output being tested. | Yes |
| `--reference <path>` | Path to an audio file to compare - the reference audio against which the output is compared. | Yes |
| Option | Description | Required |
| ------------------------------- | ------------------------------------------------------------------------------------------------ | -------- |
| `--test <path>` | Path to an audio file to compare - the output being tested. | Yes |
| `--reference <path>` | Path to an audio file to compare - the reference audio against which the output is compared. | Yes |
| `--tolerance <number/dB value>` | The volume of the difference at which the two files will be considered different. Default -50dB. | No |

Example usage:
```shell
plugalyzer audioDiff \
--test=new_audio.wav \
--reference=audio_from_plugin_v1.wav \
--tolerance=-30dB
```

## List plugin parameters
The `listParameters` command lists all available plugin parameters and their value range, as well as whether they support parameter values in text form.

| Option | Description | Required |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `--plugin=<path>` | Path to the plugin to use. | Yes |
| `--output=<path>` | Path to output the automation.<br>If not supplied, will be output to stdout. | No |
| `--overwrite` | Overwrite the output file if it exists.<br>If this option is not set and the file exists, a new file with a different name (eg. 'outputfile2.json') will be created. | No |
| `--format=<text\|json>` | The format in which to output the parameter list. Default text. | No |
| Option | Description | Required |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `--plugin=<path>` | Path to the plugin to use. | Yes |
| `--output=<path>` | Path to output the automation.<br>If not supplied, will be output to stdout. | No |
| `--overwrite` | Overwrite the output file if it exists.<br>If this option is not set and the file exists, a new file with a different name (eg. 'outputfile2.json') will be created. | No |
| `--format=<text/json>` | The format in which to output the parameter list. Default text. | No |

Example usage:
```shell
Expand Down Expand Up @@ -217,12 +225,12 @@ Plugin parameters:

The `busLayouts` command outputs all of the plugin's supported bus layouts. This can be useful in development to make sure you won't be provided bus layouts you cannot handle, and are being provided the bus layouts you intend to handle.

| Option | Description | Required |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `--plugin=<path>` | Path to the plugin to use. | Yes |
| `--output=<path>` | Path to output the bus layouts.<br>If not supplied, will be output to stdout. | No |
| `--overwrite` | Overwrite the output file if it exists.<br>If this option is not set and the file exists, a new file with a different name (eg. 'outputfile2.json') will be created. | No |
| `--format=<text\|json>` | The format in which to output the parameter list. Default text. | No |
| Option | Description | Required |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `--plugin=<path>` | Path to the plugin to use. | Yes |
| `--output=<path>` | Path to output the bus layouts.<br>If not supplied, will be output to stdout. | No |
| `--overwrite` | Overwrite the output file if it exists.<br>If this option is not set and the file exists, a new file with a different name (eg. 'outputfile2.json') will be created. | No |
| `--format=<text/json>` | The format in which to output the parameter list. Default text. | No |

Example usage:

Expand Down
18 changes: 18 additions & 0 deletions Source/Validators.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
#include <string>
#include <string_view>


// Validators for CLI11
// Signature:
// std::string validate(const std::string& arg)
Expand Down Expand Up @@ -111,4 +112,21 @@ std::string generator(const std::string& str) {
return "";
}

std::string amplitude(const std::string& str) {
try {
auto [value, unit] = parse::numberAndUnits<double>(str);
if (unit != "" && string_utils::lowerCase(unit) != "db") {
return "If units are provided, it must be in dB, e.g. '1.5dB'";
}
} catch (const std::invalid_argument& e) {
return std::format("Can't get a number from: {}, error: {}", str, e.what());
} catch (const std::out_of_range& e) {
return std::format("Number too large: {}, error: {}", str, e.what());
} catch (const std::exception& e) {
return std::format("Couldn't parse: {}, error: {}", str, e.what());
}

return "";
}

} // namespace validate
9 changes: 9 additions & 0 deletions Source/Validators.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,4 +62,13 @@ std::string outputFormat(const std::string& str);
*/
std::string generator(const std::string& str);

/**
* Validates an amplitude.
* Can be linear or with a dB suffix.
*
* @param str The amplitude argument
* @return Empty string if valid, or an error message
*/
std::string amplitude(const std::string& str);

} // namespace validate
46 changes: 30 additions & 16 deletions Source/commands/AudioDiffCommand.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,26 @@
#include "AudioDiffCommand.h"

#include "Errors.h"
#include "Parsers.h"
#include "Validators.h"

#include <juce_audio_basics/juce_audio_basics.h>
#include <print>


std::shared_ptr<CLI::App> AudioDiffCommand::createApp() {
// don't break these lines, please
// clang-format off
std::shared_ptr<CLI::App> app = std::make_shared<CLI::App>("Compares two audio files", "audioDiff");

app->add_option("-t,--test", inputFilePaths.at(AudioFileRole::test), "Audio to test")->required()->check(CLI::ExistingFile);
app->add_option("-r,--reference", inputFilePaths.at(AudioFileRole::reference), "Reference audio")->required()->check(CLI::ExistingFile);
app->add_option("-t,--test", inputFilePaths.at(AudioFileRole::test), "Audio to test")
->required()
->check(CLI::ExistingFile);
app->add_option("-r,--reference", inputFilePaths.at(AudioFileRole::reference), "Reference audio")
->required()
->check(CLI::ExistingFile);
app->add_option("-d,--tolerance", argThreshold, "How different the audio can be before it's considered a failure, in RMS.")
->check(validate::amplitude)
->each([&](std::string arg){ rmsThreshold = parse::amplitude(arg); });

return app;
// clang-format on
Expand All @@ -25,31 +37,33 @@ void AudioDiffCommand::execute() {
verifiedInPath = juce::File::getCurrentWorkingDirectory().getChildFile(inPath);
}

audioFiles.insert({role, juce::File{verifiedInPath}});
audioFiles.insert({ role, juce::File{ verifiedInPath } });
}

std::println("Reading audio files");

auto differ = AudioDiff::create(audioFiles);

if (!differ) {
for (const auto& [role, errorMessage] : differ.error()) {
std::println(std::cerr, "{}: {}", stringFromRole(role), errorMessage.toStdString());
}
throw FileReadError("Couldn't read files. Aborting.", 1);
throw FileLoadError("Couldn't read files. Aborting.", 2);
}

const double rmsThresh = 0.005;

const auto actualRMS = differ->getDifferenceRMS();
const auto actualRMSdB = juce::Decibels::gainToDecibels(actualRMS, -96.f);
const auto actualRMSdBReadout =
juce::Decibels::toString(actualRMSdB, 6, -96.f, true, "-inf").toStdString();
const auto rmsThresholddB = juce::Decibels::gainToDecibels(rmsThreshold, -96.0);
const auto rmsThresholddBReadout =
juce::Decibels::toString(rmsThresholddB, 6, -96.0, true, "-inf").toStdString();

std::println("Difference in RMS between test audio and reference: {}", actualRMS);
std::println("{}", actualRMSdBReadout);

if (actualRMS > rmsThresh) {
std::println("Cancellation test failed: detected SNR of {}, which exceeds threshold of {}",
actualRMS, rmsThresh);
throw FailedDiffError("Cancellation test failed", 2);
if (actualRMS > rmsThreshold) {
std::println(
stderr, "Detected SNR of {}, which exceeds threshold of {}", actualRMSdBReadout,
rmsThresholddBReadout
);
throw FailedDiffError("Cancellation test failed", 1);
}

std::println("Cancellation test successful.");
}
15 changes: 7 additions & 8 deletions Source/commands/AudioDiffCommand.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,8 @@

#include "AudioDiff.h"
#include "CLICommand.h"
#include <juce_audio_formats/juce_audio_formats.h>

class FileReadError : public CLI::Error {
public:
FileReadError(std::string msg, int exit_code)
: CLI::Error("FileReadError", std::move(msg), exit_code) {}
};
#include <juce_audio_formats/juce_audio_formats.h>

class FailedDiffError : public CLI::Error {
public:
Expand All @@ -23,8 +18,12 @@ class AudioDiffCommand : public CLICommand {
void execute() override;

private:
// String from CLI to be parsed into a double
std::string argThreshold;

double rmsThreshold{ juce::Decibels::decibelsToGain(-50.0, -96.0) };
std::map<AudioFileRole, juce::String> inputFilePaths{
{AudioFileRole::test, {}},
{AudioFileRole::reference, {}},
{ AudioFileRole::test, {} },
{ AudioFileRole::reference, {} },
};
};
9 changes: 4 additions & 5 deletions test/e2e_tests/run_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import os
import subprocess
import shutil
from pathlib import Path
Expand Down Expand Up @@ -94,12 +93,12 @@ def main():

for fail in failures:
logstring = f"Failed: {fail.description}\n"
if isinstance(fail.correct_output, str):
logstring += f"Expected: {fail.correct_output}\n"
logstring += f"Got: {fail.output}\n"
elif isinstance(fail.correct_output, bytes):
if isinstance(fail.correct_output, bytes):
logstring += f"Expected: {fail.correct_output.hex()}\n"
logstring += f"Got: {fail.output.hex()}\n"
else:
logstring += f"Expected: {fail.correct_output}\n"
logstring += f"Got: {fail.output}\n"
if fail.correct_exit_code != 0:
logstring += f"Expected exit code {fail.correct_exit_code}, got {fail.exit_code}\n"

Expand Down
Loading
Loading