[Feature] Add web-fetch as build-in tool

AGENTS.md

@@ -44,6 +44,9 @@ This document is the LeapFlow engineering collaboration contract. It is not only
 - **Dependency Inversion**: Core logic depends on Protocol abstractions, never on concrete implementations
 - **Protocol over ABC**: Use `typing.Protocol` with `runtime_checkable` for all extension points
 - **Event-Driven Communication**: Modules interact through typed events on EventBus, not direct imports
+- **Session Engine is the Only Reporting Source (MANDATORY)**: conversation state lives on the per-session engines built by `SessionRegistry`. `ctx.engine` is only the template they are cloned from and **never accumulates turns, context, or history**. Any code reporting runtime state — stream chunk metadata, `status()`, session analysis, dashboards, status bar values — must resolve the engine through the single entry point (`RuntimeLeapService._active_engine()` / `SessionCoordinator.resolve_session_engine()`), never `getattr(ctx, "engine")`. Reading the template silently yields zeros, which is invisible in review and has surfaced repeatedly as an empty LeapBoard and a status bar frozen at `0/<limit>`. When a value is produced *by* a specific engine (e.g. a stream event), pass that engine explicitly instead of re-resolving, so concurrent sessions cannot be cross-reported.
+- **Client-Visible Runtime State Must Be Pushed, Not Inferred**: a daemon-mode TUI is a separate process; it seeds model, context length, and usage at startup and can only learn about later changes from metadata the daemon returns. Any runtime value the status bar renders must travel on ordinary status/stream metadata (and on the mutation payload for command RPCs) — never rely on a one-off change notification, which change-detection can legitimately skip.
+- **Terminal Output Must Wrap at the Console Layer**: `LeapConsole` owns wrapping (`soft_wrap=False`), because prompt_toolkit's renderer clips at the window edge rather than reflowing. Never enable `soft_wrap` on the shared console or hand it pre-formatted long lines; long answers silently lose their tail. A standalone `Console` for fixed-width art (e.g. the banner) may opt out, but must set an explicit `width`.
 - **Immutable Domain Types**: Use `@dataclass(frozen=True)` or `NamedTuple` for domain objects
 - **Config-Driven Behavior**: Thresholds, intervals, feature flags, model budgets, platform capabilities, hub backends, gateway manifests, and paths must be configurable through Settings/env/config layers.
 - **Graceful Degradation**: Every optional component (LLM, Hub) can be absent without crash

README.md

@@ -135,6 +135,45 @@ Config keys (all via `leap config set …` / TUI `/config`):
 
 ---
 
+## Web Access (`web_fetch`)
+
+Reading the internet is a first-class **read-only** capability, not a shell command. `web_fetch` is disclosed on every turn, so the agent never has to improvise `curl … | python3 -c …` — which is what turned an HTTP 429 into a Python traceback and dragged a plain GET through side-effect gating (batch stop, session-scoped dedup, "this may already have taken effect").
+
+```
+web_fetch(url, select=?, timeout=?, max_bytes=?)
+  → { ok, status, kind, data | text | cache_path, title, links, from_cache, error_type, retryable }
+```
+
+| Behavior | Detail |
+|---|---|
+| **JSON APIs** | Parsed, with an optional dotted `select` (`chart.result.0.meta.regularMarketPrice`) so only the needed branch enters context |
+| **Web pages** | Boilerplate removed, returned as readable text plus extracted links |
+| **Binary** (PDF/images) | Never inlined; written to the session cache and returned as `cache_path` |
+| **HTTP errors** | Reported as `status` + `error_type` + a body excerpt, and marked `retryable` for 429/5xx |
+| **Retry & failover** | Rate limits and 5xx are retried; a client refused 403/429 by one transport is retried on the other (httpx / system curl) |
+| **Egress safety** | URLs resolving to loopback, private, link-local, or cloud-metadata addresses require approval; the grant is scoped to the **origin**, so approving once trusts that host for the session. `file://` and other schemes are refused outright |
+| **Caching** | Session-scoped, TTL'd, never synced off the machine |
+
+Config keys: `web.transport`, `web.timeout_s`, `web.max_bytes`, `web.max_retries`, `web.max_redirects`, `web.user_agent`, `web.extractor`, `web.private_targets`, `web.cache_ttl_s` (`leap config list web`).
+
+**Better HTML extraction (optional).** The built-in reader is dependency-free and always available. For article-grade boilerplate removal:
+
+```bash
+pip install 'leapflow[web]'   # adds trafilatura; used automatically when present
+```
+
+**JavaScript-rendered or bot-protected pages (opt-in, not bundled).** `web_fetch` performs no browser rendering and no anti-bot bypass. If you need them, register a scraping MCP server — LeapFlow already loads MCP servers, so this needs no code change. Add to `~/.leapflow/config/mcp_servers.json`:
+
+```json
+{
+  "scrapling": { "command": "scrapling", "args": ["mcp"] }
+}
+```
+
+Then install it separately (`pip install 'scrapling[ai]'` followed by `scrapling install`, which downloads Chromium — several hundred MB). Its tools appear with an `mcp_` prefix. Deliberately kept outside the default install: it is ~85 MB of wheels plus browsers, and fingerprint spoofing / CAPTCHA bypass carry compliance implications for the sites you fetch, so that choice stays yours rather than shipping on by default.
+
+---
+
 ## Prerequisites
 
 | Component | Version | Purpose |

pyproject.toml

@@ -20,6 +20,10 @@ classifiers = [
 ]
 dependencies = [
     "openai>=1.40",
+    # Declared explicitly rather than relied on through openai: web_fetch imports
+    # it directly, so a transitive-only dependency would break the moment openai
+    # changed its own HTTP client.
+    "httpx>=0.27",
     "msgpack>=1.0.8",
     "duckdb>=1.0.0",
     "pyyaml>=6.0",
@@ -40,6 +44,9 @@ dev = [
 ]
 hub = ["modelscope-hub>=0.1.0"]
 dashboard = ["aiohttp>=3.9"]
+# Better main-content extraction for web_fetch. Optional because the stdlib
+# extractor always ships: this upgrades quality, it does not enable the feature.
+web = ["trafilatura>=2.2"]
 
 [project.scripts]
 leap = "leapflow.__main__:main"

src/leapflow/cli/context.py

@@ -1353,6 +1353,10 @@ async def _on_gateway_event_with_bridge(event: object) -> None:
         # so config_set must not be able to disable its own supervision.
         from leapflow.tools.config_tools import set_config_approval_gate
         set_config_approval_gate(self._approval_orchestrator)
+        # Outbound fetches to internal targets (loopback, private ranges, cloud
+        # instance metadata) need the same review; public reads stay unprompted.
+        from leapflow.tools.web_fetch import set_web_approval_gate
+        set_web_approval_gate(self._approval_orchestrator)
 
         self._register_gateway_normalizers(settings)
 

src/leapflow/cli/tui_app/console.py

@@ -132,7 +132,13 @@ def __init__(self, theme: Theme | ResolvedTheme) -> None:
         self._console = Console(
             theme=_build_rich_theme(theme),
             highlight=False,
-            soft_wrap=True,
+            # Rich must wrap to the console width itself. With soft_wrap=True it
+            # emits one long line and leaves wrapping to whoever owns the screen;
+            # under prompt_toolkit's patch_stdout that renderer clips at the
+            # window edge instead, so long answers lost their tail. Width is
+            # detected per access from the terminal file descriptors, so this
+            # keeps following terminal resizes.
+            soft_wrap=False,
         )
 
     @property

src/leapflow/cli/tui_app/stream.py

@@ -16,6 +16,7 @@
 import os
 import re
 import time
+from dataclasses import dataclass
 from typing import TYPE_CHECKING, Any
 
 from rich.text import Text
@@ -157,10 +158,19 @@ def _is_permission_recovery_metadata(metadata: dict[str, Any] | None) -> bool:
     )
 
 
-def _truncate_detail(text: str, *, limit: int = _TOOL_OUTPUT_LIMIT) -> str:
+def _truncate_detail(text: str, *, limit: int = _TOOL_OUTPUT_LIMIT, keep_tail: bool = False) -> str:
     compact = " ".join(text.split())
     if len(compact) <= limit:
         return compact
+    if keep_tail:
+        # Failure text names its cause at the end (a traceback's last line), so a
+        # head-only cut would leave the user reading "Traceback (most recent call
+        # last): File ..." and nothing about what actually went wrong. The head
+        # keeps just enough to identify the output; the tail gets the rest so a
+        # typical exception line survives intact rather than being cut mid-word.
+        head = max(1, (limit - 1) // 4)
+        tail = max(1, limit - 1 - head)
+        return compact[:head] + "…" + compact[-tail:]
     return compact[: limit - 1] + "…"
 
 
@@ -274,12 +284,16 @@ def _tool_result_detail(metadata: dict[str, Any] | None) -> str:
     if metadata.get("ok") is False:
         exit_code = metadata.get("exit_code")
         prefix = f"exit={exit_code} " if exit_code is not None else ""
+        # error_preview first: the tool already clipped it to the informative end
+        # of the output, while stderr_preview starts at the top of the dump.
         detail = (
-            _metadata_text(metadata, "stderr_preview")
-            or _metadata_text(metadata, "error_preview")
+            _metadata_text(metadata, "error_preview")
+            or _metadata_text(metadata, "stderr_preview")
             or _metadata_text(metadata, "result_preview")
         )
-        return _truncate_detail(prefix + detail, limit=_TOOL_OUTPUT_LIMIT) if detail or prefix else "failed"
+        if not (detail or prefix):
+            return "failed"
+        return _truncate_detail(prefix + detail, limit=_TOOL_OUTPUT_LIMIT, keep_tail=True)
     detail = (
         _metadata_text(metadata, "stdout_preview")
         or _metadata_text(metadata, "content_preview")
@@ -306,6 +320,15 @@ def _format_elapsed(seconds: float) -> str:
     return f"{minutes}m{secs:.0f}s"
 
 
+@dataclass
+class _ActiveTool:
+    """One in-flight tool call, tracked independently of its siblings."""
+
+    name: str
+    detail: str
+    started_at: float
+
+
 class StreamRenderer:
     """Accumulates streaming output and renders on finish.
 
@@ -326,9 +349,12 @@ def __init__(self, console: "LeapConsole") -> None:
         self._pending: str = ""
         self._thinking_buffer: str = ""
         self._start_time: float = 0.0
-        self._tool_start_time: float = 0.0
-        self._active_tool: str = ""
-        self._active_tool_detail: str = ""
+        # Keyed by tool_call_id so a parallel batch tracks each call separately;
+        # a single shared slot attributed every line in a batch to whichever call
+        # started last and dropped the siblings' lines entirely.
+        self._active_tools: dict[str, _ActiveTool] = {}
+        self._active_order: list[str] = []
+        self._tool_seq: int = 0
         self._tool_history: list[tuple[str, float]] = []
         self._permission_block_reason: str = ""
 
@@ -363,12 +389,12 @@ def start(self) -> None:
         self._buffer = ""
         self._pending = ""
         self._thinking_buffer = ""
-        self._active_tool = ""
-        self._active_tool_detail = ""
+        self._active_tools = {}
+        self._active_order = []
+        self._tool_seq = 0
         self._tool_history = []
         self._permission_block_reason = ""
         self._start_time = time.monotonic()
-        self._tool_start_time = 0.0
 
     def feed(self, chunk: str) -> None:
         """Append a text chunk to the pending buffer.
@@ -398,11 +424,57 @@ def tool_started(self, name: str, metadata: dict[str, Any] | None = None) -> str
         self._pending = ""
         metadata = metadata or {}
         tool_name = _metadata_text(metadata, "normalized_tool_name") or name
-        self._active_tool = tool_name
-        self._active_tool_detail = _tool_action_detail(metadata)
-        self._tool_start_time = time.monotonic()
+        key = self._new_tool_key(metadata, tool_name)
+        self._active_tools[key] = _ActiveTool(
+            name=tool_name,
+            detail=_tool_action_detail(metadata),
+            started_at=time.monotonic(),
+        )
+        self._active_order.append(key)
+        pending = len(self._active_order)
+        if pending > 1:
+            # A parallel batch really has N calls running; showing only the last
+            # one would under-report the work in flight.
+            return f"{_tool_icon(tool_name)} {tool_name} +{pending - 1}"
         return f"{_tool_icon(tool_name)} {tool_name}"
 
+    def _new_tool_key(self, metadata: dict[str, Any], tool_name: str) -> str:
+        """Return a unique tracking key for a starting tool call.
+
+        Prefers the engine's ``tool_call_id`` so a completion can find its own
+        start inside a parallel batch. Callers that emit no id get a synthetic
+        unique key: reusing the tool name would make a second concurrent call to
+        the same tool overwrite the first and lose its line.
+        """
+        call_id = _metadata_text(metadata, "tool_call_id")
+        if call_id:
+            return call_id
+        self._tool_seq += 1
+        return f"{tool_name}#{self._tool_seq}"
+
+    def _take_active(self, metadata: dict[str, Any], tool_name: str) -> _ActiveTool | None:
+        """Pop the tracked start matching this completion, if any."""
+        call_id = _metadata_text(metadata, "tool_call_id")
+        key = call_id if call_id in self._active_tools else ""
+        if not key:
+            # No id match: pair with the oldest in-flight call of the same name,
+            # which is the right choice for both sequential and unlabelled calls.
+            key = next(
+                (
+                    candidate
+                    for candidate in self._active_order
+                    if self._active_tools.get(candidate) is not None
+                    and self._active_tools[candidate].name == tool_name
+                ),
+                "",
+            )
+        if not key:
+            return None
+        active = self._active_tools.pop(key, None)
+        if key in self._active_order:
+            self._active_order.remove(key)
+        return active
+
     def tool_finished(
         self,
         name: str = "",
@@ -411,19 +483,30 @@ def tool_finished(
     ) -> None:
         """Mark a tool call as finished; print one compact audit line."""
         metadata = metadata or {}
-        tool_name = _metadata_text(metadata, "normalized_tool_name") or name or self._active_tool
+        tool_name = (
+            _metadata_text(metadata, "normalized_tool_name")
+            or name
+            or (self._active_tools[self._active_order[-1]].name if self._active_order else "")
+        )
         original_tool_name = _metadata_text(metadata, "original_tool_name")
         alias_detail = original_tool_name if original_tool_name and original_tool_name != tool_name else ""
+        active = self._take_active(metadata, tool_name)
         if tool_name and metadata.get("ui_hidden"):
-            self._active_tool = ""
-            self._active_tool_detail = ""
-            self._tool_start_time = 0.0
             return
-        if tool_name and self._tool_start_time > 0:
-            duration = time.monotonic() - self._tool_start_time
+        if active is None and tool_name:
+            # Correlation failed (a completion with no tracked start). Report it
+            # anyway with a zero duration: dropping the line would hide a tool the
+            # user's turn actually ran, and silence is the worse failure mode.
+            active = _ActiveTool(
+                name=tool_name,
+                detail=_tool_action_detail(metadata),
+                started_at=time.monotonic(),
+            )
+        if tool_name and active is not None:
+            duration = time.monotonic() - active.started_at
             self._tool_history.append((tool_name, duration))
             ok = metadata.get("ok", True)
-            action_detail = self._active_tool_detail or _tool_action_detail(metadata)
+            action_detail = active.detail or _tool_action_detail(metadata)
             result_detail = _tool_result_detail(metadata) or _truncate_detail(output, limit=_TOOL_OUTPUT_LIMIT)
             line = Text()
             status_style = "leap.tool" if ok else "leap.error"
@@ -454,9 +537,6 @@ def tool_finished(
                 recovery_line.append("    ↳ recovery: ", style="leap.tool")
                 recovery_line.append(_truncate_detail(recovery_hint, limit=_TOOL_OUTPUT_LIMIT), style="leap.tool")
                 self._console.print(recovery_line)
-        self._active_tool = ""
-        self._active_tool_detail = ""
-        self._tool_start_time = 0.0
 
     def finish(self, *, command: Any | None = None) -> None:
         """Render all accumulated content to the console."""

src/leapflow/config.py

@@ -356,6 +356,19 @@ class Settings:
     tools_lint_command: str = ""  # empty => auto-detect (ruff/eslint/go vet/clippy)
     tools_terminal_session_enabled: bool = False  # persistent shell sessions (opt-in, high risk)
     tools_verify_edits: bool = True  # post-edit syntax check (advisory) for edit_file/file_write
+
+    # web_fetch: first-class read-only HTTP access, so reading a public page is
+    # not a side-effecting shell command. Limits are config-driven because the
+    # right timeout/size depends on the network and the target, not on LeapFlow.
+    web_transport: str = "auto"            # auto | httpx | curl
+    web_timeout_s: float = 20.0
+    web_max_bytes: int = 2_000_000         # response body cap before truncation
+    web_max_retries: int = 2               # retries for 429/5xx/timeouts (read-only, so safe)
+    web_max_redirects: int = 5
+    web_user_agent: str = ""               # empty => built-in browser-style default
+    web_extractor: str = "auto"            # auto (trafilatura when installed) | stdlib
+    web_private_targets: str = "approval"  # approval | deny | allow
+    web_cache_ttl_s: float = 900.0
     agent_validate_tool_args: bool = True  # pre-execution required-argument validation + self-repair
     context_hard_limit_ratio: float = 0.92
     context_warning_ratio: float = 0.75
@@ -849,6 +862,15 @@ def _build_settings_from_env(
     tools_lint_command = os.getenv("LEAPFLOW_TOOLS_LINT_COMMAND", "").strip()
     tools_terminal_session_enabled = os.getenv("LEAPFLOW_TOOLS_TERMINAL_SESSION_ENABLED", "0").strip().lower() in ("1", "true", "yes")
     tools_verify_edits = os.getenv("LEAPFLOW_TOOLS_VERIFY_EDITS", "1").strip().lower() in ("1", "true", "yes")
+    web_transport = os.getenv("LEAPFLOW_WEB_TRANSPORT", "auto").strip().lower() or "auto"
+    web_timeout_s = float(os.getenv("LEAPFLOW_WEB_TIMEOUT_S", "20"))
+    web_max_bytes = int(os.getenv("LEAPFLOW_WEB_MAX_BYTES", "2000000"))
+    web_max_retries = int(os.getenv("LEAPFLOW_WEB_MAX_RETRIES", "2"))
+    web_max_redirects = int(os.getenv("LEAPFLOW_WEB_MAX_REDIRECTS", "5"))
+    web_user_agent = os.getenv("LEAPFLOW_WEB_USER_AGENT", "").strip()
+    web_extractor = os.getenv("LEAPFLOW_WEB_EXTRACTOR", "auto").strip().lower() or "auto"
+    web_private_targets = os.getenv("LEAPFLOW_WEB_PRIVATE_TARGETS", "approval").strip().lower() or "approval"
+    web_cache_ttl_s = float(os.getenv("LEAPFLOW_WEB_CACHE_TTL_S", "900"))
     agent_validate_tool_args = os.getenv("LEAPFLOW_AGENT_VALIDATE_TOOL_ARGS", "1").strip().lower() in ("1", "true", "yes")
     context_hard_limit_ratio = float(os.getenv("LEAPFLOW_CONTEXT_HARD_LIMIT_RATIO", "0.92"))
     context_warning_ratio = float(os.getenv("LEAPFLOW_CONTEXT_WARNING_RATIO", "0.75"))
@@ -1175,6 +1197,15 @@ def _build_settings_from_env(
         tools_lint_command=tools_lint_command,
         tools_terminal_session_enabled=tools_terminal_session_enabled,
         tools_verify_edits=tools_verify_edits,
+        web_transport=web_transport,
+        web_timeout_s=web_timeout_s,
+        web_max_bytes=web_max_bytes,
+        web_max_retries=web_max_retries,
+        web_max_redirects=web_max_redirects,
+        web_user_agent=web_user_agent,
+        web_extractor=web_extractor,
+        web_private_targets=web_private_targets,
+        web_cache_ttl_s=web_cache_ttl_s,
         agent_validate_tool_args=agent_validate_tool_args,
         context_hard_limit_ratio=context_hard_limit_ratio,
         context_warning_ratio=context_warning_ratio,