feat: security, config, and documentation improvements

Dockerfile

@@ -12,7 +12,7 @@ ARG INSTALL_LATEX=false
 
 # Labels
 LABEL maintainer="DSAgent Contributors"
-LABEL version="0.7.0"
+LABEL version="0.9.0"
 LABEL description="AI-powered autonomous agent for data science"
 
 # Set environment variables
@@ -21,8 +21,7 @@ ENV PYTHONUNBUFFERED=1 \
     PIP_NO_CACHE_DIR=1 \
     PIP_DISABLE_PIP_VERSION_CHECK=1 \
     DSAGENT_WORKSPACE=/workspace \
-    DSAGENT_SESSIONS_DIR=/workspace \
-    LLM_MODEL=gpt-4o
+    DSAGENT_SESSIONS_DIR=/workspace
 
 # Install system dependencies (base)
 RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -72,4 +71,4 @@ EXPOSE 8000
 
 # Default command: start API server
 # Can be overridden to run CLI: docker run -it dsagent dsagent chat
-CMD ["dsagent", "serve", "--host", "0.0.0.0", "--port", "8000"]
+CMD ["sh", "-c", "dsagent serve --host 0.0.0.0 --port ${PORT:-8000}"]

README.md

@@ -50,20 +50,32 @@ uv sync --all-extras
 
 ### Docker
 
+Configuration uses the same environment variables as the CLI and server (see [Configuration](docs/getting-started/configuration.md)). The container listens on `PORT` (default 8000).
+
 ```bash
-# Run API server
-docker run -d -p 8000:8000 \
+# Run API server (default: port 8000)
+docker run -d -p 8080:8080 \
+  -e PORT=8080 \
+  -e DSAGENT_DEFAULT_MODEL=gpt-4o \
   -e OPENAI_API_KEY=sk-your-key \
   nmlemus/dsagent:latest
 
 # Run interactive CLI
 docker run -it \
   -e OPENAI_API_KEY=sk-your-key \
+  -v "$(pwd)/workspace:/workspace" \
   nmlemus/dsagent:latest \
   dsagent chat
+
+# One-shot task
+docker run --rm \
+  -e OPENAI_API_KEY=sk-your-key \
+  -v "$(pwd)/workspace:/workspace" \
+  nmlemus/dsagent:latest \
+  dsagent run "Analyze data/sales.csv" --data ./data/sales.csv
 ```
 
-For Docker deployment details, see [docs/DOCKER.md](docs/DOCKER.md).
+For Docker deployment details, see [docs/DOCKER.md](docs/DOCKER.md) and [docs/guide/docker.md](docs/guide/docker.md).
 
 ## Quick Start
 
@@ -85,7 +97,7 @@ This will:
   - Local → `ollama/llama3`
 - Optionally configure MCP tools (web search, etc.)
 
-To use a different model, edit `~/.dsagent/.env` or use the `--model` flag:
+To use a different model, set `DSAGENT_DEFAULT_MODEL` or `LLM_MODEL` in `~/.dsagent/.env`, or use the `--model` flag:
 ```bash
 dsagent --model gpt-4o-mini
 ```
@@ -118,11 +130,15 @@ dsagent run "Analyze sales trends" --data ./sales.csv
 | `dsagent` | Start interactive chat (default) |
 | `dsagent chat` | Same as above, with explicit options |
 | `dsagent run "task"` | Execute a one-shot task |
+| `dsagent serve` | Run REST + WebSocket API server |
 | `dsagent init` | Setup wizard for configuration |
 | `dsagent skills list` | List installed skills |
-| `dsagent skills install <source>` | Install a skill |
+| `dsagent skills install <source>` | Install a skill from GitHub or path |
+| `dsagent skills remove <name>` | Remove a skill |
+| `dsagent skills info <name>` | Show skill details |
 | `dsagent mcp list` | List configured MCP servers |
-| `dsagent mcp add <template>` | Add an MCP server |
+| `dsagent mcp add <template>` | Add an MCP server from template |
+| `dsagent mcp remove <name>` | Remove an MCP server |
 
 ### Examples
 
@@ -259,8 +275,10 @@ DSAgent comes with essential data science libraries pre-installed:
 
 ## Documentation
 
-- [CLI Reference](docs/CLI.md) - Complete command-line options
-- [Python API](docs/PYTHON_API.md) - Detailed API documentation
+- [CLI Reference](docs/CLI.md) - All commands: chat, run, serve, init, mcp, skills
+- [Configuration](docs/getting-started/configuration.md) - Environment variables and `.env`
+- [HTTP API](docs/api/http-api.md) - REST and WebSocket API reference
+- [Python API](docs/PYTHON_API.md) - ConversationalAgent and PlannerAgent
 - [Model Configuration](docs/MODELS.md) - LLM provider setup
 - [MCP Tools](docs/MCP.md) - External tools integration
 - [Agent Skills](docs/guide/skills.md) - Extensible skill system

docs/CLI.md

@@ -8,7 +8,12 @@ DSAgent provides a unified command-line interface with subcommands for different
 dsagent                          # Start interactive chat (default)
 dsagent chat                     # Same as above
 dsagent run "task"               # Execute one-shot task
+dsagent serve [--port 8000]      # Run REST + WebSocket API server
 dsagent init                     # Setup wizard
+dsagent skills list              # List installed skills
+dsagent skills install <source>  # Install a skill
+dsagent skills remove <name>     # Remove a skill
+dsagent skills info <name>       # Show skill details
 dsagent mcp list                 # List MCP servers
 dsagent mcp add <template>       # Add MCP server
 dsagent mcp remove <name>        # Remove MCP server
@@ -31,8 +36,8 @@ dsagent chat [options]
 
 | Option | Short | Description | Default |
 |--------|-------|-------------|---------|
-| `--model` | `-m` | LLM model to use | `gpt-4o` or `$LLM_MODEL` |
-| `--workspace` | `-w` | Workspace directory | `./workspace` |
+| `--model` | `-m` | LLM model to use | `DSAGENT_DEFAULT_MODEL` or `LLM_MODEL` or `gpt-4o` |
+| `--workspace` | `-w` | Workspace directory | `./workspace` or `$DSAGENT_WORKSPACE` |
 | `--session` | `-s` | Session ID to resume | New session |
 | `--hitl` | | Human-in-the-loop mode | `none` |
 | `--live-notebook` | | Save notebook after each execution | Off |
@@ -49,6 +54,8 @@ dsagent chat [options]
 | `plan_answer` | Pause for plan and final answer approval |
 | `on_error` | Pause only when errors occur |
 
+(For `dsagent run`, valid modes are `none`, `plan_only`, `on_error`, `plan_and_answer`, `full`.)
+
 ### Examples
 
 ```bash
@@ -88,10 +95,98 @@ Once in a chat session, you can use slash commands:
 | `/clear` | Clear the screen |
 | `/model <name>` | Change the model |
 | `/status` | Show current status |
+| `/vars` | Show kernel variables |
+| `/data` | List or copy data files |
+| `/workspace` | Show workspace paths |
 | `/quit` | Exit the session |
 
 ---
 
+## `dsagent serve`
+
+Run the REST and WebSocket API server. Uses the same configuration as the CLI (env vars and `.env`). Host and port are set by flags; API key, CORS, and upload limits come from config.
+
+```bash
+dsagent serve [options]
+```
+
+### Options
+
+| Option | Short | Description | Default |
+|--------|-------|-------------|---------|
+| `--host` | | Host to bind to | `0.0.0.0` |
+| `--port` | `-p` | Port to listen on | `8000` |
+| `--reload` | | Enable auto-reload (development) | Off |
+| `--log-level` | | Logging level | `info` |
+
+### Environment variables (server)
+
+| Variable | Description |
+|----------|-------------|
+| `DSAGENT_API_KEY` | If set, API and WebSocket require `X-API-Key` header |
+| `DSAGENT_CORS_ORIGINS` | Comma-separated origins or `*` |
+| `DSAGENT_REQUIRE_API_KEY` | If `true`, server refuses to start without `DSAGENT_API_KEY` |
+| `DSAGENT_MAX_UPLOAD_MB` | Max upload size per file in MB (`0` = no limit) |
+
+### Examples
+
+```bash
+# Default: 0.0.0.0:8000, no API key
+dsagent serve
+
+# Custom port
+dsagent serve --port 3000
+
+# With API key (set in env)
+export DSAGENT_API_KEY=my-secret-key
+dsagent serve
+
+# Development with reload
+dsagent serve --reload --log-level debug
+```
+
+API docs when running: `http://localhost:8000/docs`. Health: `http://localhost:8000/health`.
+
+---
+
+## `dsagent skills`
+
+Manage Agent Skills (extensible capabilities). Skills are stored in `~/.dsagent/skills/`.
+
+### `dsagent skills list`
+
+List installed skills.
+
+### `dsagent skills install <source>`
+
+Install a skill. Source can be:
+
+- `github:owner/repo` — install from GitHub
+- `github:owner/repo/path` — install from a subpath
+- `./path` or `/path` — install from local directory
+
+Options: `--force` / `-f` to overwrite existing.
+
+### `dsagent skills remove <name>`
+
+Remove an installed skill by name.
+
+### `dsagent skills info <name>`
+
+Show details for an installed skill.
+
+### Examples
+
+```bash
+dsagent skills list
+dsagent skills install github:dsagent-skills/eda
+dsagent skills install ./my-skill --force
+dsagent skills remove eda-analysis
+dsagent skills info eda-analysis
+```
+
+---
+
 ## `dsagent run`
 
 Execute a one-shot task without interactive mode.
@@ -269,25 +364,32 @@ dsagent mcp remove brave_search
 
 ## Environment Variables
 
-DSAgent reads configuration from environment variables and `.env` files.
+DSAgent uses a single config source: `dsagent.config`. All settings can be set via environment variables with the `DSAGENT_` prefix or via `.env` files.
+
+### Search Order for `.env` (CLI)
 
-### Search Order for `.env`
+1. Global: `~/.dsagent/.env` (loaded first)
+2. Local: `./.env` (overrides global)
 
-1. Current working directory: `./.env`
-2. Project root: `{project}/.env`
-3. User config: `~/.dsagent/.env`
+So you can set API keys in `~/.dsagent/.env` and override model or workspace per project in `./.env`.
 
 ### Configuration Variables
 
-| Variable | Description | Example |
+| Variable | Description | Default |
 |----------|-------------|---------|
-| `LLM_MODEL` | Default model | `gpt-4o` |
-| `LLM_API_BASE` | Custom API endpoint | `http://localhost:4000/v1` |
-| `OPENAI_API_KEY` | OpenAI API key | `sk-...` |
-| `ANTHROPIC_API_KEY` | Anthropic API key | `sk-ant-...` |
-| `GOOGLE_API_KEY` | Google API key | `...` |
-| `DEEPSEEK_API_KEY` | DeepSeek API key | `...` |
-| `BRAVE_API_KEY` | Brave Search API key | `...` |
+| `DSAGENT_DEFAULT_MODEL` | Default LLM model (recommended) | `gpt-4o` |
+| `LLM_MODEL` | Legacy default model (CLI compatible) | - |
+| `DSAGENT_WORKSPACE` | Workspace directory | `./workspace` |
+| `DSAGENT_SESSIONS_DIR` | Sessions directory under workspace | `workspace` |
+| `DSAGENT_MAX_ROUNDS` | Max agent iterations | `30` |
+| `DSAGENT_TEMPERATURE` | LLM temperature | `0.3` |
+| `LLM_API_BASE` | Custom API endpoint (e.g. LiteLLM proxy) | - |
+| `OPENAI_API_KEY` | OpenAI API key | - |
+| `ANTHROPIC_API_KEY` | Anthropic API key | - |
+| `GOOGLE_API_KEY` | Google API key | - |
+| `GROQ_API_KEY` | Groq API key | - |
+| `DEEPSEEK_API_KEY` | DeepSeek API key | - |
+| `BRAVE_API_KEY` | Brave Search (MCP) | - |
 
 ### Using LiteLLM Proxy
 

docs/DOCKER.md

@@ -35,9 +35,11 @@ DSAgent provides Docker images for easy deployment of both the CLI and API serve
 # Pull the image
 docker pull nmlemus/dsagent:latest
 
-# Run API server
+# Run API server (PORT defaults to 8000)
 docker run -d \
-  -p 8000:8000 \
+  -p 8080:8080 \
+  -e PORT=8080 \
+  -e DSAGENT_DEFAULT_MODEL=gpt-4o \
   -e OPENAI_API_KEY=sk-your-key \
   -v $(pwd)/workspace:/workspace \
   nmlemus/dsagent:latest
@@ -48,25 +50,36 @@ docker run -it \
   -v $(pwd)/workspace:/workspace \
   nmlemus/dsagent:latest \
   dsagent chat
+
+# One-shot task
+docker run --rm \
+  -e OPENAI_API_KEY=sk-your-key \
+  -v $(pwd)/workspace:/workspace \
+  nmlemus/dsagent:latest \
+  dsagent run "Analyze this dataset" --data ./data.csv
 ```
 
 ## Configuration
 
 ### Environment Variables
 
-#### LLM Configuration
+Configuration matches the rest of DSAgent: use `DSAGENT_*` and provider API keys. The default container command runs the API server and listens on `PORT` (default 8000).
+
+#### LLM and model
 
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `LLM_MODEL` | LLM model to use | `gpt-4o` |
-| `OPENAI_API_KEY` | OpenAI API key (for gpt-* models) | - |
-| `ANTHROPIC_API_KEY` | Anthropic API key (for claude-* models) | - |
-| `GOOGLE_API_KEY` | Google API key (for gemini/* models) | - |
+| `DSAGENT_DEFAULT_MODEL` | Default LLM model | `gpt-4o` |
+| `LLM_MODEL` | Legacy default model | - |
+| `OPENAI_API_KEY` | OpenAI API key | - |
+| `ANTHROPIC_API_KEY` | Anthropic API key | - |
+| `GOOGLE_API_KEY` | Google API key | - |
+| `GROQ_API_KEY` | Groq API key | - |
 | `DEEPSEEK_API_KEY` | DeepSeek API key | - |
-| `LLM_API_BASE` | Custom API endpoint (for proxies) | - |
+| `LLM_API_BASE` | Custom API endpoint | - |
 | `OLLAMA_API_BASE` | Ollama API endpoint | `http://host.docker.internal:11434` |
 
-#### Agent Settings
+#### Agent settings
 
 | Variable | Description | Default |
 |----------|-------------|---------|
@@ -75,12 +88,15 @@ docker run -it \
 | `DSAGENT_MAX_TOKENS` | Max tokens per response | `4096` |
 | `DSAGENT_CODE_TIMEOUT` | Code execution timeout (seconds) | `300` |
 
-#### API Server
+#### API server and port
 
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `DSAGENT_API_KEY` | API authentication key | - (disabled) |
+| `PORT` | Port the server listens on | `8000` |
+| `DSAGENT_API_KEY` | API key (enables auth when set) | - (disabled) |
 | `DSAGENT_CORS_ORIGINS` | CORS allowed origins | `*` |
+| `DSAGENT_REQUIRE_API_KEY` | Refuse start without API key | `false` |
+| `DSAGENT_MAX_UPLOAD_MB` | Max upload per file (MB); `0` = no limit | `50` |
 
 #### MCP Tools (Optional)