New Synthesis Engine

[spessasus]

Hi cifkao,
Thanks for this great project! The visualizer looks really nice and the simplicity of embedding the player on a website is really convenient.

Unfortunately I've noticed that a lot of MIDI features are missing. They have been reported as issues here and most of them are caused by the underlying magenta music engine.

That's why I propose replacing it with one I've been developing. It's called SpessaSynth.

Here's why I think it's a good idea and the issues it would resolve:

• More MIDI features: Pitch bend, sustain pedal, pan, volume, modulation and even more exotic features such as system exclusives, RPN, MIDI Tuning Standard, Multi-Port MIDIs and even portamento are all supported out of the box. So a lot of older and exotic MIDIs would be able to utilize fancy effects. (Support the Sustain pedal #11, Pitch bend not working #56)
• Custom soundfonts: I saw a discussion about it here. Spessasynth fully supports SF2, SF3 and even DLS soundfonts out of the box, which would make it very easy to allow users to use their own custom ones. (Custom soundfonts? #74, Add a looping option #51)
• Audio worklets: spessasynth uses audio worklets for the synthesis. This would not only solve Players cannot play simultaneously #1 but also ensure no lag on the main thread as the entire synthesis+playback process runs on a separate audio thread. (Are there any plan to replace ScriptProcessorNode with AudioWorkletNode? #71)
• Faster loading times: spessasynth supports SF3, which are OGG Vorbis compressed soundfonts. A high quality one such as GeneralUserGS can be compressed down to just 8MB, and low quality ones such as the msgs soundfont to even lower (less than a megabyte!). Not to mention that it won't have to download all samples as they would be ready out of the box with a single SF3 file.
• Loop points: automatically detected (Add a looping option #51)
• Enhancing Download button #13: not only writing out a MIDI file is supported natively, but you can also export a rendered WAV file as well. So the download button would allow users to both get the MIDI and a rendered audio file.
• Tempo control: there's a playbackRate property for exactly that. (Tempo control #21)

If replacing the synth engine wouldn't be too much work, I think these features are worth it.

What do you think?

Regards,
spessasus

[cifkao]

Hi @spessasus, thanks for reaching out! Switching to a different synthesis engine has been on my mind for a long time so it's definitely something I'd be open to, and it would be awesome to get all these issues solved.

So if you feel like giving this a shot, a PR would be very welcome! Perhaps a quick proof of concept first to see if this works at all and how much change it will require.

I just have the the following questions:

• How would this interact with the Magenta NoteSequence format? I suppose the player wouldn't use this format internally anymore (it can't represent a lot of the MIDI features you mentioned), but it would still need it for communication with the visualizers (unless they are re-implemented). Moreover, it's nice to have this note-oriented representation of MIDI, and I guess some existing applications of html-midi-player might rely on having access to it and manipulating it.
• How would this affect the JS bundle size? The size of the minified Tone.js+@magenta/music/core bundle is 576KB (and html-midi-player itself is just 13KB). If we keep using the Magenta visualizers, we'll have to keep these dependencies (at least the Magenta one, not sure about Tone.js), so the amount of code that needs to be shipped will only increase.

[spessasus]

Hi cifkao,
Thanks for your quick response!

To address your concerns:

• Yes, spessasynth uses a very "raw" note sequence format, storing the MIDI as an array of tracks, and each track being an array of MIDI events. Hovewer, the demo website uses a visualizer similar to what you have (rectangular notes). Here is the code that calculates each note's start time and duration in seconds, which should be very easy to visualize (html canvas?)
• I think it would be less, given that we could remove both tone.js and magenta music. spessasynth_lib has no dependencies. Keep in mind that you'll have to enable tree shaking during the minification though, as this lib is way more than just a synthesizer, and importing all that additional code (MIDI editing, sf2/dls conversions, articulator modification & compression, RMI writing, etc.) would be a waste of bandwidth.

I'll look into the source of this project and try to create a PR when I have time :-)