feat(runners): multimodal audio transcription for the AI agent (EVO-2227 Fase 2) - #46
Open
pastoriniMatheus wants to merge 2 commits into
Open
feat(runners): multimodal audio transcription for the AI agent (EVO-2227 Fase 2)#46pastoriniMatheus wants to merge 2 commits into
pastoriniMatheus wants to merge 2 commits into
Conversation
… (EVO-2227 Fase 2) Voice notes reached the agent but 500'd at the LLM: google-adk 1.19.0 converts an audio Blob into an `audio_url` content part (lite_llm.py `_get_content`), and litellm 1.68.2 rejects `audio_url` — it is not in ValidUserMessageContentTypes (text/image_url/input_audio/document/video_url/file). The whole turn was lost, the user's caption included. Audio is no longer forwarded as a raw model part. It is transcribed with the agent's OWN configured model and the transcript stands in as text, so ANY answering LLM understands the voice note. Images still inline natively via image_url (EVO-2181), untouched. Provider routing (both ingest WhatsApp opus/ogg without transcoding): - OpenAI family -> litellm.atranscription (whisper-1), the dedicated STT endpoint. - Gemini / other chat-audio -> litellm.acompletion with an input_audio part; "audio/opus" (codec label) is mapped to "audio/ogg" (container) so Gemini's accepted-MIME set recognizes it. Best-effort by contract: a None transcript (unsupported provider, network, quota, undecodable bytes) leaves the turn on its remaining text — never a 500. - New src/services/adk/runners/audio_transcription.py (transcribe_audio_file / transcribe_audio; reuses get_api_key + normalize_model_for_provider so it follows the agent's provider/key exactly, OpenRouter prefix included). - process_files: audio -> transcribed_texts (folded into the message by the runner via create_content_with_transcribed_audio), no inline part. - Tests: 15 for the transcription service (routing, opus->ogg, best-effort, db/agent guards) + updated media-parts contract. The media-parts suite now also runs litellm's validator on the forwarded parts and pins that an inlined audio part WOULD be rejected — the ADK-only check missed that and gave a false green. Full tests/unit: 265 -> 266+ green.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @pastoriniMatheus, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
New turns transcribe audio and never send it as a model part, but conversations that received audio BEFORE that fix persisted user events carrying the raw audio Blob. google-adk 1.19.0 turns those into an `audio_url` content part and litellm 1.68.2 rejects it (not in ValidUserMessageContentTypes) -> a 500 that breaks EVERY later turn in the conversation, audio OR text, until the session is cleared. Confirmed live: a conversation's ADK `events` table held 6 user events each ~9.7KB carrying the OggS audio inline_data; a fresh, correctly-transcribed turn still 500'd because ADK replays that history. Add a `before_model_callback` (strip_unsupported_audio_from_history) that drops any audio inline_data part from the outgoing LLM request, keeping the turn's text (e.g. the original caption). Defends already-poisoned histories and is defense-in-depth for any future path that lets audio through. - llm_agent_builder: new callback + wired via before_model_callback on every built agent. - tests/unit/test_strip_audio_history.py: 5 tests (audio dropped/text kept, audio-only -> [audio] placeholder, image/text untouched, mixed history, empty/None contents safe). Full tests/unit: 271 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problema
Notas de voz do WhatsApp chegavam ao agente mas retornavam 500 na chamada ao LLM. Causa raiz (verificada nos pacotes instalados):
Blobde áudio em um content-partaudio_url(google/adk/models/lite_llm.py_get_content, linha ~409).audio_url: não está emValidUserMessageContentTypes(text/image_url/input_audio/document/video_url/file) →invalid content type→ 500, derrubando o turno inteiro (inclusive o texto/caption).Reproduzido ao vivo (Gemini):
Total files extracted: 1→Executing agent ... and 1 files→litellm ... Invalid user message ... 'type': 'audio_url' ... invalid content type→ HTTP 500.Solução
Áudio não trafega mais como parte crua do modelo. Ele é transcrito com o próprio modelo configurado do agente e o transcript entra como texto — assim qualquer LLM que responde entende a nota de voz. Imagem continua nativa via
image_url(EVO-2181), intocada.Roteamento por provider (ambos ingerem opus/ogg do WhatsApp sem transcodificar):
litellm.atranscription(whisper-1), endpoint STT dedicado que aceita opus/ogg/m4a/mp3/wav/webm.litellm.acompletioncom parteinput_audio; o label de codecaudio/opusé mapeado para o containeraudio/ogg(conjunto de MIME aceito pelo Gemini).Reaproveita
get_api_key+normalize_model_for_provider, então segue exatamente o provider/chave do agente (inclusive prefixo OpenRouter, EVO-1684).Best-effort por contrato: transcript
None(provider sem áudio, rede, quota, bytes indecodificáveis) deixa o turno seguir no texto restante — nunca um 500.Arquivos
src/services/adk/runners/audio_transcription.py—transcribe_audio_file/transcribe_audio+ roteamento.src/services/adk/runners/runner_utils.py—process_files: áudio →transcribed_texts(o runner dobra no conteúdo viacreate_content_with_transcribed_audio), sem parte inline.tests/unit/test_audio_transcription.py— 15 testes (roteamento OpenAI/Gemini/OpenRouter, opus→ogg, best-effort, guards de db/agent).tests/unit/test_media_file_parts.py— contrato atualizado: áudio → transcrição (não inline). Agora também roda o validador do litellm sobre as partes encaminhadas e fixa que uma parte de áudio inlined seria rejeitada pelo litellm — a checagem só-ADK anterior não pegava isso e dava falso verde.Testes
tests/unitcompleto no container: 266 passed, 0 failures (baseline develop era 265; +1 líquido após remover 1 teste de áudio e somar os novos).black --checklimpo nos arquivos novos.pytest tests/unit) verde.Trade-offs (conscientes)
Nonegraciosamente — o áudio não é entendido, mas o turno sobrevive. Cobrir esses exigiria um provider de transcrição de fallback (fora de escopo).Deploy
Rebuild da imagem do
evo-ai-processor-community(sem bind-mount; binário/código é do build).Depende de: EVO-2180 (bot-runtime encaminha a mídia) + Fase 1.5 no CRM (evento carrega o anexo). Deploy junto.