Feat/spatialwalk

ai_agents/agents/examples/rtm-transport/tenapp/manifest.json

@@ -16,7 +16,7 @@
     {
       "type": "extension",
       "name": "agora_rtm",
-      "version": "0.11.1"
+      "version": "=0.11.1"
     },
     {
       "type": "system",

ai_agents/agents/examples/spatialwalk-avatar/Dockerfile

@@ -0,0 +1,83 @@
+FROM ghcr.io/ten-framework/ten_agent_build:0.7.12 AS builder
+
+ARG USE_AGENT=agents/examples/spatialwalk-avatar
+
+WORKDIR /app
+
+COPY .env.example .env
+COPY server/ server/
+COPY agents/scripts/ agents/scripts/
+COPY agents/ten_packages/ agents/ten_packages/
+COPY ${USE_AGENT}/frontend/ ${USE_AGENT}/frontend/
+
+# Copy tenapp files explicitly to avoid symlink issues
+COPY ${USE_AGENT}/tenapp/go.* ${USE_AGENT}/tenapp/
+COPY ${USE_AGENT}/tenapp/main.go ${USE_AGENT}/tenapp/
+COPY ${USE_AGENT}/tenapp/manifest* ${USE_AGENT}/tenapp/
+COPY ${USE_AGENT}/tenapp/property.json ${USE_AGENT}/tenapp/
+COPY ${USE_AGENT}/tenapp/scripts/ ${USE_AGENT}/tenapp/scripts/
+
+# Copy extension directories that are actual directories (not symlinks)
+COPY ${USE_AGENT}/tenapp/ten_packages/extension/main_python/ ${USE_AGENT}/tenapp/ten_packages/extension/main_python/
+
+# Copy other example files
+COPY ${USE_AGENT}/README.md ${USE_AGENT}/
+COPY ${USE_AGENT}/Taskfile*.yml ${USE_AGENT}/
+
+RUN cd /app/${USE_AGENT} && \
+  task install && task release
+
+# Build frontend in builder stage (bun already done by task install)
+WORKDIR ${USE_AGENT}/frontend/
+RUN NEXT_PUBLIC_EDIT_GRAPH_MODE=false bun run build
+WORKDIR /app
+
+FROM ubuntu:22.04
+
+ARG USE_AGENT=agents/examples/spatialwalk-avatar
+
+RUN apt-get clean && apt-get update && apt-get install -y --no-install-recommends \
+  libasound2 \
+  libgstreamer1.0-dev \
+  libunwind-dev \
+  libc++1 \
+  libssl-dev \
+  python3 \
+  python3-venv \
+  python3-pip \
+  python3-dev \
+  jq vim \
+  ca-certificates \
+  curl \
+  unzip \
+  && apt-get clean && rm -rf /var/lib/apt/lists/* && rm -rf /tmp/*
+
+# Install Node.js 20
+RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
+  && apt-get install -y nodejs
+
+# Install Bun using the official install script
+RUN curl -fsSL https://bun.com/install | bash
+# Add Bun to the PATH (if not already added by the install script)
+ENV PATH="/root/.bun/bin:$PATH"
+
+# Install Task
+RUN sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b /usr/local/bin
+
+WORKDIR /app
+
+COPY --from=builder /app/${USE_AGENT}/tenapp/.release/ /app/agents/
+COPY --from=builder /app/server/bin/api /app/server/bin/api
+COPY --from=builder /usr/local/lib /usr/local/lib
+COPY --from=builder /usr/lib/python3 /usr/lib/python3
+COPY --from=builder /usr/local/bin/tman /usr/local/bin/tman
+
+# Copy built frontend from builder stage
+COPY --from=builder /app/${USE_AGENT}/frontend/ /app/frontend/
+
+# Copy Docker Taskfile
+COPY --from=builder /app/${USE_AGENT}/Taskfile.docker.yml /app/Taskfile.docker.yml
+
+EXPOSE 8080 3000
+
+ENTRYPOINT ["task", "-t", "Taskfile.docker.yml", "run-prod"]

ai_agents/agents/examples/spatialwalk-avatar/README.md

@@ -0,0 +1,202 @@
+# Voice Assistant
+
+A comprehensive voice assistant with real-time conversation capabilities using Agora RTC, Deepgram STT, OpenAI LLM, and ElevenLabs TTS.
+
+## Features
+
+- **Chained Model Real-time Voice Interaction**: Complete voice conversation pipeline with STT → LLM → TTS processing
+
+## Prerequisites
+
+### Required Environment Variables
+
+1. **Agora Account**: Get credentials from [Agora Console](https://console.agora.io/)
+   - `AGORA_APP_ID` - Your Agora App ID (required)
+
+2. **Deepgram Account**: Get credentials from [Deepgram Console](https://console.deepgram.com/)
+   - `DEEPGRAM_API_KEY` - Your Deepgram API key (required)
+
+3. **OpenAI Account**: Get credentials from [OpenAI Platform](https://platform.openai.com/)
+   - `OPENAI_API_KEY` - Your OpenAI API key (required)
+
+4. **ElevenLabs Account**: Get credentials from [ElevenLabs](https://elevenlabs.io/)
+   - `ELEVENLABS_TTS_KEY` - Your ElevenLabs API key (required)
+
+### Optional Environment Variables
+
+- `AGORA_APP_CERTIFICATE` - Agora App Certificate (optional)
+- `OPENAI_MODEL` - OpenAI model name (optional, defaults to configured model)
+- `OPENAI_PROXY_URL` - Proxy URL for OpenAI API (optional)
+- `WEATHERAPI_API_KEY` - Weather API key for weather tool (optional)
+
+## Setup
+
+### 1. Set Environment Variables
+
+Add to your `.env` file:
+
+```bash
+# Agora (required for audio streaming)
+AGORA_APP_ID=your_agora_app_id_here
+AGORA_APP_CERTIFICATE=your_agora_certificate_here
+
+# Deepgram (required for speech-to-text)
+DEEPGRAM_API_KEY=your_deepgram_api_key_here
+
+# OpenAI (required for language model)
+OPENAI_API_KEY=your_openai_api_key_here
+OPENAI_MODEL=gpt-4
+OPENAI_PROXY_URL=your_proxy_url_here
+
+# ElevenLabs (required for text-to-speech)
+ELEVENLABS_TTS_KEY=your_elevenlabs_api_key_here
+
+# Optional
+WEATHERAPI_API_KEY=your_weather_api_key_here
+```
+
+For the `spatialwalk-avatar` frontend, pass required Spatialwalk IDs in URL query params:
+
+```text
+http://localhost:3000/?appId=<SPATIALWALK_APP_ID>&avatarId=<SPATIALWALK_AVATAR_ID>
+```
+
+`appId` and `avatarId` are required. The settings sheet only controls environment and layout options.
+
+### 2. Install Dependencies
+
+```bash
+cd agents/examples/voice-assistant
+task install
+```
+
+This installs Python dependencies and frontend components.
+
+### 3. Run the Voice Assistant
+
+```bash
+cd agents/examples/voice-assistant
+task run
+```
+
+The voice assistant starts with all capabilities enabled.
+
+### 4. Access the Application
+
+- **Frontend**: http://localhost:3000
+- **API Server**: http://localhost:8080
+- **TMAN Designer**: http://localhost:49483
+
+## Configuration
+
+The voice assistant is configured in `tenapp/property.json`:
+
+```json
+{
+  "ten": {
+    "predefined_graphs": [
+      {
+        "name": "voice_assistant",
+        "auto_start": true,
+        "graph": {
+          "nodes": [
+            {
+              "name": "agora_rtc",
+              "addon": "agora_rtc",
+              "property": {
+                "app_id": "${env:AGORA_APP_ID}",
+                "app_certificate": "${env:AGORA_APP_CERTIFICATE|}",
+                "channel": "ten_agent_test",
+                "subscribe_audio": true,
+                "publish_audio": true,
+                "publish_data": true
+              }
+            },
+            {
+              "name": "stt",
+              "addon": "deepgram_asr_python",
+              "property": {
+                "params": {
+                  "api_key": "${env:DEEPGRAM_API_KEY}",
+                  "language": "en-US"
+                }
+              }
+            },
+            {
+              "name": "llm",
+              "addon": "openai_llm2_python",
+              "property": {
+                "api_key": "${env:OPENAI_API_KEY}",
+                "model": "${env:OPENAI_MODEL}",
+                "max_tokens": 512,
+                "greeting": "TEN Agent connected. How can I help you today?"
+              }
+            },
+            {
+              "name": "tts",
+              "addon": "elevenlabs_tts2_python",
+              "property": {
+                "params": {
+                  "key": "${env:ELEVENLABS_TTS_KEY}",
+                  "model_id": "eleven_multilingual_v2",
+                  "voice_id": "pNInz6obpgDQGcFmaJgB",
+                  "output_format": "pcm_16000"
+                }
+              }
+            }
+          ]
+        }
+      }
+    ]
+  }
+}
+```
+
+### Configuration Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `AGORA_APP_ID` | string | - | Your Agora App ID (required) |
+| `AGORA_APP_CERTIFICATE` | string | - | Your Agora App Certificate (optional) |
+| `DEEPGRAM_API_KEY` | string | - | Deepgram API key (required) |
+| `OPENAI_API_KEY` | string | - | OpenAI API key (required) |
+| `OPENAI_MODEL` | string | - | OpenAI model name (optional) |
+| `OPENAI_PROXY_URL` | string | - | Proxy URL for OpenAI API (optional) |
+| `ELEVENLABS_TTS_KEY` | string | - | ElevenLabs API key (required) |
+| `WEATHERAPI_API_KEY` | string | - | Weather API key (optional) |
+
+## Customization
+
+The voice assistant uses a modular design that allows you to easily replace STT, LLM, or TTS modules with other providers using TMAN Designer.
+
+Access the visual designer at http://localhost:49483 to customize your voice agent. For detailed usage instructions, see the [TMAN Designer documentation](https://theten.ai/docs/ten_agent/customize_agent/tman-designer).
+
+## Release as Docker image
+
+**Note**: The following commands need to be executed outside of any Docker container.
+
+### Build image
+
+```bash
+cd ai_agents
+docker build -f agents/examples/voice-assistant/Dockerfile -t voice-assistant-app .
+```
+
+### Run
+
+```bash
+docker run --rm -it --env-file .env -p 8080:8080 -p 3000:3000 voice-assistant-app
+```
+
+### Access
+
+- Frontend: http://localhost:3000
+- API Server: http://localhost:8080
+
+## Learn More
+
+- [Agora RTC Documentation](https://docs.agora.io/en/rtc/overview/product-overview)
+- [Deepgram API Documentation](https://developers.deepgram.com/)
+- [OpenAI API Documentation](https://platform.openai.com/docs)
+- [ElevenLabs API Documentation](https://docs.elevenlabs.io/)
+- [TEN Framework Documentation](https://doc.theten.ai)

ai_agents/agents/examples/spatialwalk-avatar/Taskfile.docker.yml

@@ -0,0 +1,29 @@
+version: "3"
+
+dotenv: [".env"]
+
+tasks:
+  run-api-prod:
+    desc: run api server in production
+    cmds:
+      - /app/server/bin/api -tenapp_dir=/app/agents
+    background: true
+
+  run-frontend-prod:
+    desc: run frontend in production
+    dir: ./frontend
+    cmds:
+      - bun start
+    background: true
+
+  run-prod:
+    desc: run both api and frontend in production
+    deps:
+      - task: run-api-prod
+      - task: run-frontend-prod
+
+  health-check:
+    desc: check if both services are running
+    cmds:
+      - curl -f http://localhost:8080/health || echo "API server not responding"
+      - curl -f http://localhost:3000 || echo "Frontend not responding"