Use AudioEngine ADM for PlatformAudio on Apple platforms#1215
Draft
hiroshihorie wants to merge 4 commits into
Draft
Use AudioEngine ADM for PlatformAudio on Apple platforms#1215hiroshihorie wants to merge 4 commits into
hiroshihorie wants to merge 4 commits into
Conversation
Select the new kAppleAudioEngine audio layer when creating the platform ADM on iOS and macOS. The AVAudioEngine based ADM supports runtime switchable voice processing and device change handling. Construction happens in the proxy constructor on the worker thread, which the AudioEngine device binds its sequence checker to. Also forward the platform voice processing interface (topology, path availability and toggle, processing state) to the platform ADM so the coupled Apple AEC+NS path is reachable through the proxy. Requires a libwebrtc build that includes CreateAudioEngineDeviceModule.
Apple's AudioEngine ADM exposes AEC and NS through one coupled voice processing path rather than independent toggles. Expose the platform audio processing topology and voice processing path controls through AudioDeviceController, PeerConnectionFactory and LkRuntime, and add configure_platform_audio_processing which picks the right strategy from the reported topology: - Coupled topology (Apple): enable the platform voice processing path only when both AEC and NS are requested with hardware preferred, otherwise disable it and fall back to WebRTC software processing. - Independent topology: toggle builtin AEC/AGC/NS individually as before. PlatformAudio::configure_audio_processing now makes a single native call instead of toggling builtin effects from Rust, and the docs are updated from VPIO wording to the AudioEngine ADM.
The AudioSourceInterface hunk stopped applying after the fork added SetOptions right where the patch expected the class to end. The build script tolerates patch failures (git apply || true), so building a prebuilt from current m144 would have silently produced a libwebrtc without external audio source support. Content is unchanged, only the surrounding context is regenerated against m144.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #1212 (worker-thread-affine AdmProxy) — review only the top 3 commits. Retarget to
mainafter #1212 merges.What
Switches the platform ADM on iOS and macOS to the AVAudioEngine based
AudioEngineDeviceand plumbs its platform voice processing API through toPlatformAudio:AdmProxynow creates the platform ADM with thekAppleAudioEngineaudio layer on Apple platforms (kPlatformDefaultAudioelsewhere, Android unchanged). Creation stays in the proxy constructor on the worker thread, which the AudioEngine device binds its sequence checker to — the contract Make AdmProxy worker-thread-affine #1212 established.AdmProxy→AudioDeviceController→PeerConnectionFactory→LkRuntime: processing topology, voice processing path availability/toggle, and processing state.configure_platform_audio_processingpicks the strategy from the ADM's reported topology:PlatformAudio::configure_audio_processingnow makes a single native call instead of toggling builtin effects from Rust; docs updated from VPIO wording to the AudioEngine ADM.Why
The AudioEngine ADM replaces the AudioUnit/CoreAudio-HAL default on Apple platforms, bringing runtime switchable voice processing (no engine rebuild to toggle AEC), proper device change handling, and parity with the Swift SDK's audio stack. Apple exposes AEC+NS as one coupled path, which the previous independent enable_builtin_* plumbing could not represent.
Dependencies — why CI is red
This PR cannot build against the currently pinned webrtc prebuilt (
webrtc-51ef663):kAppleAudioEngineand the platform voice processing API only exist in newer webrtc. The chain to green:CreateAudioEngineDeviceModule/kAppleAudioEngine), ideally together with Fix macOS input device selection without voice processing in AudioEngineDevice webrtc-sdk/webrtc#261 (macOS device selection fix, see Testing below).WEBRTC_TAGbump commit is added here (webrtc-sys/build/src/lib.rs).Until then this is compile-verified against the webrtc-sdk source tree headers (
clang -fsyntax-onlyfor the touched C++) rather than by CI.Testing
cargo run -p livekit --example platform_audio(added in Make AdmProxy worker-thread-affine #1212) exercises the full control plane against the AudioEngine ADM unchanged — lifecycle, device enumeration/selection/hot-swap, recording, AEC/AGC/NS reconfiguration incl. the coupled voice processing path, concurrent access, and runtime teardown cycles.configure_audio_processing(prefer_hardware_processing: true)activates the Apple voice processing path (GetPlatformAudioProcessingState).E2E test results (local webrtc build via
LK_CUSTOM_WEBRTC)Verified on macOS arm64 against a local libwebrtc built from webrtc-sdk
hiroshi/audioengine-platformaudio:AudioEngineDeviceconfirmed active (not a fallback): afterconfigure_audio_processing(prefer_hardware_processing: true), builtin AEC/AGC/NS report available and active types flip to Hardware — the coupled Apple voice processing path enables end to end throughAdmProxyon the worker thread.platform_audioexerciser: lifecycle, enumeration, recording, processing reconfiguration, 16-thread concurrent hammering, and full runtime teardown cycles all pass.AudioEngineDevicebug found by the exerciser (not caused by this PR): on macOS with voice processing disabled (prefer_hardware_processing: false, the macOS default), selecting a non-default recording device failed atInitRecording, because the engine's input and output nodes share one HAL I/O unit in that mode. Fixed in Fix macOS input device selection without voice processing in AudioEngineDevice webrtc-sdk/webrtc#261 via a private aggregate device; with that fix in the local build, device switching works in both processing modes and the full exerciser passes all phases against the AudioEngine ADM. The prebuilt for this PR should therefore include ffi v0.3.15 #261.