diff --git a/AGENTS.md b/AGENTS.md index 20a444b..aa79a1f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,6 +44,9 @@ This document is the LeapFlow engineering collaboration contract. It is not only - **Dependency Inversion**: Core logic depends on Protocol abstractions, never on concrete implementations - **Protocol over ABC**: Use `typing.Protocol` with `runtime_checkable` for all extension points - **Event-Driven Communication**: Modules interact through typed events on EventBus, not direct imports +- **Session Engine is the Only Reporting Source (MANDATORY)**: conversation state lives on the per-session engines built by `SessionRegistry`. `ctx.engine` is only the template they are cloned from and **never accumulates turns, context, or history**. Any code reporting runtime state — stream chunk metadata, `status()`, session analysis, dashboards, status bar values — must resolve the engine through the single entry point (`RuntimeLeapService._active_engine()` / `SessionCoordinator.resolve_session_engine()`), never `getattr(ctx, "engine")`. Reading the template silently yields zeros, which is invisible in review and has surfaced repeatedly as an empty LeapBoard and a status bar frozen at `0/`. When a value is produced *by* a specific engine (e.g. a stream event), pass that engine explicitly instead of re-resolving, so concurrent sessions cannot be cross-reported. +- **Client-Visible Runtime State Must Be Pushed, Not Inferred**: a daemon-mode TUI is a separate process; it seeds model, context length, and usage at startup and can only learn about later changes from metadata the daemon returns. Any runtime value the status bar renders must travel on ordinary status/stream metadata (and on the mutation payload for command RPCs) — never rely on a one-off change notification, which change-detection can legitimately skip. +- **Terminal Output Must Wrap at the Console Layer**: `LeapConsole` owns wrapping (`soft_wrap=False`), because prompt_toolkit's renderer clips at the window edge rather than reflowing. Never enable `soft_wrap` on the shared console or hand it pre-formatted long lines; long answers silently lose their tail. A standalone `Console` for fixed-width art (e.g. the banner) may opt out, but must set an explicit `width`. - **Immutable Domain Types**: Use `@dataclass(frozen=True)` or `NamedTuple` for domain objects - **Config-Driven Behavior**: Thresholds, intervals, feature flags, model budgets, platform capabilities, hub backends, gateway manifests, and paths must be configurable through Settings/env/config layers. - **Graceful Degradation**: Every optional component (LLM, Hub) can be absent without crash diff --git a/README.md b/README.md index 42b5619..1912b9c 100644 --- a/README.md +++ b/README.md @@ -135,6 +135,45 @@ Config keys (all via `leap config set …` / TUI `/config`): --- +## Web Access (`web_fetch`) + +Reading the internet is a first-class **read-only** capability, not a shell command. `web_fetch` is disclosed on every turn, so the agent never has to improvise `curl … | python3 -c …` — which is what turned an HTTP 429 into a Python traceback and dragged a plain GET through side-effect gating (batch stop, session-scoped dedup, "this may already have taken effect"). + +``` +web_fetch(url, select=?, timeout=?, max_bytes=?) + → { ok, status, kind, data | text | cache_path, title, links, from_cache, error_type, retryable } +``` + +| Behavior | Detail | +|---|---| +| **JSON APIs** | Parsed, with an optional dotted `select` (`chart.result.0.meta.regularMarketPrice`) so only the needed branch enters context | +| **Web pages** | Boilerplate removed, returned as readable text plus extracted links | +| **Binary** (PDF/images) | Never inlined; written to the session cache and returned as `cache_path` | +| **HTTP errors** | Reported as `status` + `error_type` + a body excerpt, and marked `retryable` for 429/5xx | +| **Retry & failover** | Rate limits and 5xx are retried; a client refused 403/429 by one transport is retried on the other (httpx / system curl) | +| **Egress safety** | URLs resolving to loopback, private, link-local, or cloud-metadata addresses require approval; the grant is scoped to the **origin**, so approving once trusts that host for the session. `file://` and other schemes are refused outright | +| **Caching** | Session-scoped, TTL'd, never synced off the machine | + +Config keys: `web.transport`, `web.timeout_s`, `web.max_bytes`, `web.max_retries`, `web.max_redirects`, `web.user_agent`, `web.extractor`, `web.private_targets`, `web.cache_ttl_s` (`leap config list web`). + +**Better HTML extraction (optional).** The built-in reader is dependency-free and always available. For article-grade boilerplate removal: + +```bash +pip install 'leapflow[web]' # adds trafilatura; used automatically when present +``` + +**JavaScript-rendered or bot-protected pages (opt-in, not bundled).** `web_fetch` performs no browser rendering and no anti-bot bypass. If you need them, register a scraping MCP server — LeapFlow already loads MCP servers, so this needs no code change. Add to `~/.leapflow/config/mcp_servers.json`: + +```json +{ + "scrapling": { "command": "scrapling", "args": ["mcp"] } +} +``` + +Then install it separately (`pip install 'scrapling[ai]'` followed by `scrapling install`, which downloads Chromium — several hundred MB). Its tools appear with an `mcp_` prefix. Deliberately kept outside the default install: it is ~85 MB of wheels plus browsers, and fingerprint spoofing / CAPTCHA bypass carry compliance implications for the sites you fetch, so that choice stays yours rather than shipping on by default. + +--- + ## Prerequisites | Component | Version | Purpose | diff --git a/pyproject.toml b/pyproject.toml index a03f9c6..7b1df3f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,6 +20,10 @@ classifiers = [ ] dependencies = [ "openai>=1.40", + # Declared explicitly rather than relied on through openai: web_fetch imports + # it directly, so a transitive-only dependency would break the moment openai + # changed its own HTTP client. + "httpx>=0.27", "msgpack>=1.0.8", "duckdb>=1.0.0", "pyyaml>=6.0", @@ -40,6 +44,9 @@ dev = [ ] hub = ["modelscope-hub>=0.1.0"] dashboard = ["aiohttp>=3.9"] +# Better main-content extraction for web_fetch. Optional because the stdlib +# extractor always ships: this upgrades quality, it does not enable the feature. +web = ["trafilatura>=2.2"] [project.scripts] leap = "leapflow.__main__:main" diff --git a/src/leapflow/cli/context.py b/src/leapflow/cli/context.py index 13f4196..92ef1d9 100644 --- a/src/leapflow/cli/context.py +++ b/src/leapflow/cli/context.py @@ -1353,6 +1353,10 @@ async def _on_gateway_event_with_bridge(event: object) -> None: # so config_set must not be able to disable its own supervision. from leapflow.tools.config_tools import set_config_approval_gate set_config_approval_gate(self._approval_orchestrator) + # Outbound fetches to internal targets (loopback, private ranges, cloud + # instance metadata) need the same review; public reads stay unprompted. + from leapflow.tools.web_fetch import set_web_approval_gate + set_web_approval_gate(self._approval_orchestrator) self._register_gateway_normalizers(settings) diff --git a/src/leapflow/cli/tui_app/console.py b/src/leapflow/cli/tui_app/console.py index c7cb7e0..fa1c43f 100644 --- a/src/leapflow/cli/tui_app/console.py +++ b/src/leapflow/cli/tui_app/console.py @@ -132,7 +132,13 @@ def __init__(self, theme: Theme | ResolvedTheme) -> None: self._console = Console( theme=_build_rich_theme(theme), highlight=False, - soft_wrap=True, + # Rich must wrap to the console width itself. With soft_wrap=True it + # emits one long line and leaves wrapping to whoever owns the screen; + # under prompt_toolkit's patch_stdout that renderer clips at the + # window edge instead, so long answers lost their tail. Width is + # detected per access from the terminal file descriptors, so this + # keeps following terminal resizes. + soft_wrap=False, ) @property diff --git a/src/leapflow/cli/tui_app/stream.py b/src/leapflow/cli/tui_app/stream.py index 7b2621a..c6e8efe 100644 --- a/src/leapflow/cli/tui_app/stream.py +++ b/src/leapflow/cli/tui_app/stream.py @@ -16,6 +16,7 @@ import os import re import time +from dataclasses import dataclass from typing import TYPE_CHECKING, Any from rich.text import Text @@ -157,10 +158,19 @@ def _is_permission_recovery_metadata(metadata: dict[str, Any] | None) -> bool: ) -def _truncate_detail(text: str, *, limit: int = _TOOL_OUTPUT_LIMIT) -> str: +def _truncate_detail(text: str, *, limit: int = _TOOL_OUTPUT_LIMIT, keep_tail: bool = False) -> str: compact = " ".join(text.split()) if len(compact) <= limit: return compact + if keep_tail: + # Failure text names its cause at the end (a traceback's last line), so a + # head-only cut would leave the user reading "Traceback (most recent call + # last): File ..." and nothing about what actually went wrong. The head + # keeps just enough to identify the output; the tail gets the rest so a + # typical exception line survives intact rather than being cut mid-word. + head = max(1, (limit - 1) // 4) + tail = max(1, limit - 1 - head) + return compact[:head] + "…" + compact[-tail:] return compact[: limit - 1] + "…" @@ -274,12 +284,16 @@ def _tool_result_detail(metadata: dict[str, Any] | None) -> str: if metadata.get("ok") is False: exit_code = metadata.get("exit_code") prefix = f"exit={exit_code} " if exit_code is not None else "" + # error_preview first: the tool already clipped it to the informative end + # of the output, while stderr_preview starts at the top of the dump. detail = ( - _metadata_text(metadata, "stderr_preview") - or _metadata_text(metadata, "error_preview") + _metadata_text(metadata, "error_preview") + or _metadata_text(metadata, "stderr_preview") or _metadata_text(metadata, "result_preview") ) - return _truncate_detail(prefix + detail, limit=_TOOL_OUTPUT_LIMIT) if detail or prefix else "failed" + if not (detail or prefix): + return "failed" + return _truncate_detail(prefix + detail, limit=_TOOL_OUTPUT_LIMIT, keep_tail=True) detail = ( _metadata_text(metadata, "stdout_preview") or _metadata_text(metadata, "content_preview") @@ -306,6 +320,15 @@ def _format_elapsed(seconds: float) -> str: return f"{minutes}m{secs:.0f}s" +@dataclass +class _ActiveTool: + """One in-flight tool call, tracked independently of its siblings.""" + + name: str + detail: str + started_at: float + + class StreamRenderer: """Accumulates streaming output and renders on finish. @@ -326,9 +349,12 @@ def __init__(self, console: "LeapConsole") -> None: self._pending: str = "" self._thinking_buffer: str = "" self._start_time: float = 0.0 - self._tool_start_time: float = 0.0 - self._active_tool: str = "" - self._active_tool_detail: str = "" + # Keyed by tool_call_id so a parallel batch tracks each call separately; + # a single shared slot attributed every line in a batch to whichever call + # started last and dropped the siblings' lines entirely. + self._active_tools: dict[str, _ActiveTool] = {} + self._active_order: list[str] = [] + self._tool_seq: int = 0 self._tool_history: list[tuple[str, float]] = [] self._permission_block_reason: str = "" @@ -363,12 +389,12 @@ def start(self) -> None: self._buffer = "" self._pending = "" self._thinking_buffer = "" - self._active_tool = "" - self._active_tool_detail = "" + self._active_tools = {} + self._active_order = [] + self._tool_seq = 0 self._tool_history = [] self._permission_block_reason = "" self._start_time = time.monotonic() - self._tool_start_time = 0.0 def feed(self, chunk: str) -> None: """Append a text chunk to the pending buffer. @@ -398,11 +424,57 @@ def tool_started(self, name: str, metadata: dict[str, Any] | None = None) -> str self._pending = "" metadata = metadata or {} tool_name = _metadata_text(metadata, "normalized_tool_name") or name - self._active_tool = tool_name - self._active_tool_detail = _tool_action_detail(metadata) - self._tool_start_time = time.monotonic() + key = self._new_tool_key(metadata, tool_name) + self._active_tools[key] = _ActiveTool( + name=tool_name, + detail=_tool_action_detail(metadata), + started_at=time.monotonic(), + ) + self._active_order.append(key) + pending = len(self._active_order) + if pending > 1: + # A parallel batch really has N calls running; showing only the last + # one would under-report the work in flight. + return f"{_tool_icon(tool_name)} {tool_name} +{pending - 1}" return f"{_tool_icon(tool_name)} {tool_name}" + def _new_tool_key(self, metadata: dict[str, Any], tool_name: str) -> str: + """Return a unique tracking key for a starting tool call. + + Prefers the engine's ``tool_call_id`` so a completion can find its own + start inside a parallel batch. Callers that emit no id get a synthetic + unique key: reusing the tool name would make a second concurrent call to + the same tool overwrite the first and lose its line. + """ + call_id = _metadata_text(metadata, "tool_call_id") + if call_id: + return call_id + self._tool_seq += 1 + return f"{tool_name}#{self._tool_seq}" + + def _take_active(self, metadata: dict[str, Any], tool_name: str) -> _ActiveTool | None: + """Pop the tracked start matching this completion, if any.""" + call_id = _metadata_text(metadata, "tool_call_id") + key = call_id if call_id in self._active_tools else "" + if not key: + # No id match: pair with the oldest in-flight call of the same name, + # which is the right choice for both sequential and unlabelled calls. + key = next( + ( + candidate + for candidate in self._active_order + if self._active_tools.get(candidate) is not None + and self._active_tools[candidate].name == tool_name + ), + "", + ) + if not key: + return None + active = self._active_tools.pop(key, None) + if key in self._active_order: + self._active_order.remove(key) + return active + def tool_finished( self, name: str = "", @@ -411,19 +483,30 @@ def tool_finished( ) -> None: """Mark a tool call as finished; print one compact audit line.""" metadata = metadata or {} - tool_name = _metadata_text(metadata, "normalized_tool_name") or name or self._active_tool + tool_name = ( + _metadata_text(metadata, "normalized_tool_name") + or name + or (self._active_tools[self._active_order[-1]].name if self._active_order else "") + ) original_tool_name = _metadata_text(metadata, "original_tool_name") alias_detail = original_tool_name if original_tool_name and original_tool_name != tool_name else "" + active = self._take_active(metadata, tool_name) if tool_name and metadata.get("ui_hidden"): - self._active_tool = "" - self._active_tool_detail = "" - self._tool_start_time = 0.0 return - if tool_name and self._tool_start_time > 0: - duration = time.monotonic() - self._tool_start_time + if active is None and tool_name: + # Correlation failed (a completion with no tracked start). Report it + # anyway with a zero duration: dropping the line would hide a tool the + # user's turn actually ran, and silence is the worse failure mode. + active = _ActiveTool( + name=tool_name, + detail=_tool_action_detail(metadata), + started_at=time.monotonic(), + ) + if tool_name and active is not None: + duration = time.monotonic() - active.started_at self._tool_history.append((tool_name, duration)) ok = metadata.get("ok", True) - action_detail = self._active_tool_detail or _tool_action_detail(metadata) + action_detail = active.detail or _tool_action_detail(metadata) result_detail = _tool_result_detail(metadata) or _truncate_detail(output, limit=_TOOL_OUTPUT_LIMIT) line = Text() status_style = "leap.tool" if ok else "leap.error" @@ -454,9 +537,6 @@ def tool_finished( recovery_line.append(" ↳ recovery: ", style="leap.tool") recovery_line.append(_truncate_detail(recovery_hint, limit=_TOOL_OUTPUT_LIMIT), style="leap.tool") self._console.print(recovery_line) - self._active_tool = "" - self._active_tool_detail = "" - self._tool_start_time = 0.0 def finish(self, *, command: Any | None = None) -> None: """Render all accumulated content to the console.""" diff --git a/src/leapflow/config.py b/src/leapflow/config.py index a2fb380..787313a 100644 --- a/src/leapflow/config.py +++ b/src/leapflow/config.py @@ -356,6 +356,19 @@ class Settings: tools_lint_command: str = "" # empty => auto-detect (ruff/eslint/go vet/clippy) tools_terminal_session_enabled: bool = False # persistent shell sessions (opt-in, high risk) tools_verify_edits: bool = True # post-edit syntax check (advisory) for edit_file/file_write + + # web_fetch: first-class read-only HTTP access, so reading a public page is + # not a side-effecting shell command. Limits are config-driven because the + # right timeout/size depends on the network and the target, not on LeapFlow. + web_transport: str = "auto" # auto | httpx | curl + web_timeout_s: float = 20.0 + web_max_bytes: int = 2_000_000 # response body cap before truncation + web_max_retries: int = 2 # retries for 429/5xx/timeouts (read-only, so safe) + web_max_redirects: int = 5 + web_user_agent: str = "" # empty => built-in browser-style default + web_extractor: str = "auto" # auto (trafilatura when installed) | stdlib + web_private_targets: str = "approval" # approval | deny | allow + web_cache_ttl_s: float = 900.0 agent_validate_tool_args: bool = True # pre-execution required-argument validation + self-repair context_hard_limit_ratio: float = 0.92 context_warning_ratio: float = 0.75 @@ -849,6 +862,15 @@ def _build_settings_from_env( tools_lint_command = os.getenv("LEAPFLOW_TOOLS_LINT_COMMAND", "").strip() tools_terminal_session_enabled = os.getenv("LEAPFLOW_TOOLS_TERMINAL_SESSION_ENABLED", "0").strip().lower() in ("1", "true", "yes") tools_verify_edits = os.getenv("LEAPFLOW_TOOLS_VERIFY_EDITS", "1").strip().lower() in ("1", "true", "yes") + web_transport = os.getenv("LEAPFLOW_WEB_TRANSPORT", "auto").strip().lower() or "auto" + web_timeout_s = float(os.getenv("LEAPFLOW_WEB_TIMEOUT_S", "20")) + web_max_bytes = int(os.getenv("LEAPFLOW_WEB_MAX_BYTES", "2000000")) + web_max_retries = int(os.getenv("LEAPFLOW_WEB_MAX_RETRIES", "2")) + web_max_redirects = int(os.getenv("LEAPFLOW_WEB_MAX_REDIRECTS", "5")) + web_user_agent = os.getenv("LEAPFLOW_WEB_USER_AGENT", "").strip() + web_extractor = os.getenv("LEAPFLOW_WEB_EXTRACTOR", "auto").strip().lower() or "auto" + web_private_targets = os.getenv("LEAPFLOW_WEB_PRIVATE_TARGETS", "approval").strip().lower() or "approval" + web_cache_ttl_s = float(os.getenv("LEAPFLOW_WEB_CACHE_TTL_S", "900")) agent_validate_tool_args = os.getenv("LEAPFLOW_AGENT_VALIDATE_TOOL_ARGS", "1").strip().lower() in ("1", "true", "yes") context_hard_limit_ratio = float(os.getenv("LEAPFLOW_CONTEXT_HARD_LIMIT_RATIO", "0.92")) context_warning_ratio = float(os.getenv("LEAPFLOW_CONTEXT_WARNING_RATIO", "0.75")) @@ -1175,6 +1197,15 @@ def _build_settings_from_env( tools_lint_command=tools_lint_command, tools_terminal_session_enabled=tools_terminal_session_enabled, tools_verify_edits=tools_verify_edits, + web_transport=web_transport, + web_timeout_s=web_timeout_s, + web_max_bytes=web_max_bytes, + web_max_retries=web_max_retries, + web_max_redirects=web_max_redirects, + web_user_agent=web_user_agent, + web_extractor=web_extractor, + web_private_targets=web_private_targets, + web_cache_ttl_s=web_cache_ttl_s, agent_validate_tool_args=agent_validate_tool_args, context_hard_limit_ratio=context_hard_limit_ratio, context_warning_ratio=context_warning_ratio, diff --git a/src/leapflow/config_loader.py b/src/leapflow/config_loader.py index 8240648..21ce648 100644 --- a/src/leapflow/config_loader.py +++ b/src/leapflow/config_loader.py @@ -239,7 +239,7 @@ def _validate_known_sections( "context", "error", "session", "stale", "default", "circuit", "rpc", "use", "cua", "copilot", "notification", "live", "gateway", "hub", "privacy", "approval", "cache", "logging", - "observer", "scheduler", + "observer", "scheduler", "web", } for key, value in parsed.items(): if key in known_sections and key != "version" and not isinstance(value, Mapping): diff --git a/src/leapflow/config_service.py b/src/leapflow/config_service.py index d03492b..9f0cf08 100644 --- a/src/leapflow/config_service.py +++ b/src/leapflow/config_service.py @@ -169,6 +169,15 @@ class ConfigSnapshot: "agent.reentry_send_rate_per_hour": "Max autonomous outbound sends per originating chat per hour (0 = unlimited); backstops send storms.", "agent.reentry_send_global_budget": "Lifetime cap on total autonomous outbound sends per daemon (0 = unlimited).", "agent.reentry_send_verified_at": "Number of human approvals in a send scope before it reaches VERIFIED trust and may be auto-approved (non-destructive replies only).", + "web.transport": "HTTP transport for web_fetch. auto tries httpx then the system curl; a client refused with 403/429 by one transport is retried on the next. Pin to httpx or curl to make behavior reproducible.", + "web.timeout_s": "Per-request timeout in seconds for web_fetch.", + "web.max_bytes": "Response body cap in bytes for web_fetch; larger bodies are truncated and flagged.", + "web.max_retries": "Retries for rate limits, 5xx, and timeouts. Safe by construction because web_fetch is a read.", + "web.max_redirects": "Maximum redirects web_fetch will follow before failing.", + "web.user_agent": "User agent sent by web_fetch. Empty uses a browser-style default, because many CDNs answer 429/403 to library agents; set your own string to identify honestly.", + "web.extractor": "HTML reader for web_fetch. auto prefers trafilatura when the `web` extra is installed (`pip install 'leapflow[web]'`) and falls back to the built-in stdlib reader; stdlib pins the dependency-free reader.", + "web.private_targets": "How web_fetch treats URLs resolving to loopback, private, link-local, or cloud-metadata addresses. approval asks the user each session (default), deny refuses without prompting (for unattended deployments), allow permits them silently and is only appropriate on a trusted network.", + "web.cache_ttl_s": "Seconds a fetched body is reused from the session cache (0 disables caching). Entries are session-scoped and never synced.", } _SECTION_CATEGORIES = { @@ -209,6 +218,7 @@ class ConfigSnapshot: "prediction": "World Model", "curiosity": "World Model", "replay": "World Model", + "web": "Web Access", } _VALUE_HINTS = { @@ -216,6 +226,9 @@ class ConfigSnapshot: "daemon.log_level": "DEBUG|INFO|WARNING|ERROR", "recording.mode": "video|default|vision_only", "signal.channels": "all or comma-separated channel names", + "web.transport": "auto|httpx|curl", + "web.extractor": "auto|stdlib", + "web.private_targets": "approval|deny|allow", } _PARTIAL_RELOAD_SECTIONS = frozenset({"runtime", "mock", "gateway", "hub", "scheduler", "observer", "cua", "use", "dashboard"}) diff --git a/src/leapflow/daemon/approval_coordinator.py b/src/leapflow/daemon/approval_coordinator.py index f12cb29..f9eca89 100644 --- a/src/leapflow/daemon/approval_coordinator.py +++ b/src/leapflow/daemon/approval_coordinator.py @@ -33,6 +33,7 @@ def install_gate(self, ctx: Any, service: Any) -> None: from leapflow.tools.gateway_tool import set_gateway_approval_gate from leapflow.tools.registry_bootstrap import set_file_read_gate, set_file_write_gate from leapflow.tools.shell_tools import set_approval_gate + from leapflow.tools.web_fetch import set_web_approval_gate existing = getattr(ctx, "_approval_orchestrator", None) gate = SessionAwareGate(_DaemonApprovalGate(self)) @@ -48,6 +49,8 @@ def install_gate(self, ctx: Any, service: Any) -> None: # Config writes go through the same daemon-side approval path, so a # daemon session cannot change settings unattended either. set_config_approval_gate(orchestrator) + # Same for outbound fetches that resolve to internal addresses. + set_web_approval_gate(orchestrator) class _FileReadGate: def __init__(self) -> None: diff --git a/src/leapflow/daemon/service.py b/src/leapflow/daemon/service.py index e0e6178..a96c412 100644 --- a/src/leapflow/daemon/service.py +++ b/src/leapflow/daemon/service.py @@ -296,7 +296,7 @@ async def engine_chat(self, message: str, **kwargs: Any) -> AsyncIterator[Stream content="Configuration reloaded in leapd.", event_type="status", metadata={ - **engine_context_metadata(getattr(ctx, "engine", None), ctx.settings), + **engine_context_metadata(self._active_engine(), ctx.settings), "llm_model": getattr(ctx.settings, "llm_model", ""), "request_id": request_id, }, @@ -365,7 +365,9 @@ async def engine_chat(self, message: str, **kwargs: Any) -> AsyncIterator[Stream stream = engine.run_stream(message, enable_thinking=enable_thinking, request_id=request_id) else: stream = engine.run_stream(message, enable_thinking=enable_thinking) - async for chunk in self._stream_engine_events(stream, approval_queue, request_id=request_id): + async for chunk in self._stream_engine_events( + stream, approval_queue, request_id=request_id, engine=engine, + ): request_record["chunks"].append(chunk) yield chunk request_record["status"] = "completed" @@ -414,6 +416,7 @@ async def _stream_engine_events( approval_queue: asyncio.Queue[StreamChunk], *, request_id: str = "", + engine: Any = None, ) -> AsyncIterator[StreamChunk]: engine_task: asyncio.Task[Any] | None = asyncio.create_task(anext(stream)) approval_task: asyncio.Task[StreamChunk] | None = asyncio.create_task(approval_queue.get()) @@ -432,7 +435,9 @@ async def _stream_engine_events( engine_task = None break stream_event = normalize_stream_event(event) - yield self._chunk_from_event(stream_event, request_id=request_id) + yield self._chunk_from_event( + stream_event, request_id=request_id, engine=engine, + ) engine_task = asyncio.create_task(anext(stream)) finally: for task in (engine_task, approval_task): @@ -445,17 +450,42 @@ async def _stream_engine_events( except Exception: logger.debug("daemon: failed to close engine stream", exc_info=True) - def _chunk_from_event(self, event: StreamEvent, *, request_id: str = "") -> StreamChunk: + def _active_engine(self, session_id: str = "") -> Any: + """Return the engine holding live conversation state. + + Single entry point for "the engine to report on". ``ctx.engine`` is only a + template used to build per-session engines and never accumulates a + conversation, so reading it yields zero turns and zero context — which + has surfaced repeatedly as an empty LeapBoard and a status bar stuck at + ``0/``. Anything assembling runtime metadata must come through + here rather than reaching for ``ctx.engine`` directly. + """ + ctx = self._ctx + if ctx is None: + return None + engine, _ = self._session_coordinator.resolve_session_engine(ctx, session_id) + return engine + + def _chunk_from_event( + self, event: StreamEvent, *, request_id: str = "", engine: Any = None, + ) -> StreamChunk: + """Wrap an engine stream event as an RPC chunk with runtime metadata. + + ``engine`` must be the engine that produced the event — the per-session + one. Falling back to ``ctx.engine`` reports the base engine, which never + carries a conversation, so context usage reads as 0 and the client's + status bar sits at ``0/`` for the whole session. + """ ctx = self.context - engine = getattr(ctx, "engine", None) + active = engine if engine is not None else self._active_engine() metadata = dict(event.metadata or {}) - session_id = getattr(engine, "_current_session_id", "") if engine else "" + session_id = getattr(active, "_current_session_id", "") if active else "" if request_id: metadata.setdefault("request_id", request_id) if session_id: metadata.setdefault("session_id", str(session_id)) - if engine is not None: - metadata.update(engine_context_metadata(engine, getattr(ctx, "settings", self._settings))) + if active is not None: + metadata.update(engine_context_metadata(active, getattr(ctx, "settings", self._settings))) return StreamChunk( request_id=request_id, content=event.content, done=False, event_type=event.type, metadata=metadata, @@ -669,7 +699,9 @@ async def subscribe_notifications(self) -> AsyncIterator[StreamChunk]: async def status(self) -> dict[str, Any]: ctx = self._ctx settings = getattr(ctx, "settings", self._settings) if ctx is not None else self._settings - engine = getattr(ctx, "engine", None) if ctx is not None else None + # Report on the session engine, not the base template: the latter has no + # conversation, so context usage would always read as zero. + engine = self._active_engine() db_holder = getattr(ctx, "_db_holder", None) if ctx is not None else None layout = settings.layout profile_layout = settings.profile_layout diff --git a/src/leapflow/engine/context_control.py b/src/leapflow/engine/context_control.py index 984f1a5..7eb3372 100644 --- a/src/leapflow/engine/context_control.py +++ b/src/leapflow/engine/context_control.py @@ -282,6 +282,8 @@ def build(self, tool_name: str, arguments: Dict[str, Any] | None, result: Any) - return self._file_list_evidence(result) if tool_name in {"shell_run", "gp_shell_run"}: return self._shell_evidence(result) + if tool_name in {"web_fetch", "gp_web_fetch"}: + return self._web_fetch_evidence(result) return self._compact_mapping(result) def _file_read_evidence(self, arguments: Dict[str, Any], result: Dict[str, Any]) -> Dict[str, Any]: @@ -380,14 +382,53 @@ def _flatten_tree_nodes( out.append(prefix + self._compact_entry(node)) def _shell_evidence(self, result: Dict[str, Any]) -> Dict[str, Any]: + from leapflow.engine.tool_execution import exit_code_from + return { "ok": bool(result.get("ok", True)), "kind": "shell_evidence", - "exit_code": result.get("exit_code"), + # Read under either key: shell tools mirror subprocess' `returncode`, + # so looking only for `exit_code` reported null on every success. + "exit_code": exit_code_from(result), "stdout": self._head_tail(str(result.get("stdout", "")), self._max_content_chars), "stderr": self._head_tail(str(result.get("stderr", "")), self._max_content_chars // 2), } + def _web_fetch_evidence(self, result: Dict[str, Any]) -> Dict[str, Any]: + """Compact a fetched page or payload down to task evidence. + + A single page routinely exceeds the whole turn's context budget, so the + raw body must never pass through. Status, final URL and content type stay + intact because they are what the model reasons about next; the body is + head/tail trimmed, and links are capped rather than dropped so follow-up + navigation is still possible. + """ + evidence: Dict[str, Any] = { + "ok": bool(result.get("ok", True)), + "kind": "web_fetch_evidence", + "status": result.get("status"), + "final_url": result.get("final_url") or result.get("url", ""), + "content_type": result.get("content_type", ""), + "truncated": bool(result.get("truncated", False)), + } + for key in ("title", "select", "extractor", "note", "from_cache", "cache_path"): + value = result.get(key) + if value: + evidence[key] = self._compact_value(value) + if "data" in result: + evidence["data"] = self._compact_value(result["data"]) + text = str(result.get("text") or "") + if text: + evidence["text"] = self._head_tail(text, self._max_content_chars) + links = result.get("links") + if isinstance(links, list) and links: + evidence["links"] = [ + {"text": str(item.get("text", ""))[:120], "url": str(item.get("url", ""))} + for item in links[: self._max_items] + if isinstance(item, dict) + ] + return evidence + def _platform_action_evidence(self, arguments: Dict[str, Any], result: Dict[str, Any]) -> Dict[str, Any]: """Compact evidence for platform_action with strong completion markers.""" if result.get("ok") is False: @@ -452,6 +493,14 @@ def _compact_error(self, result: Dict[str, Any]) -> Dict[str, Any]: for key in ("returncode", "exit_code"): if key in result: compact[key] = result[key] + # Network failures explain themselves in the response: the status code is + # the classification and the body says why (an API error payload, a rate + # limit notice), so both must survive compaction for the model to act on. + if "status" in result: + compact["status"] = result["status"] + body_excerpt = result.get("body_excerpt") + if body_excerpt: + compact["body_excerpt"] = self._head_tail(str(body_excerpt), self._max_content_chars // 2) return compact def _app_connector_evidence(self, result: Dict[str, Any]) -> Dict[str, Any]: diff --git a/src/leapflow/engine/engine.py b/src/leapflow/engine/engine.py index 74313ba..ae90cf6 100644 --- a/src/leapflow/engine/engine.py +++ b/src/leapflow/engine/engine.py @@ -66,6 +66,7 @@ ToolExecutionLedger, effect_is_uncertain_on_failure, execution_policy_for, + exit_code_from, ) from leapflow.engine.graph_planner import GraphPlanner from leapflow.engine.scheduler import TaskScheduler @@ -143,14 +144,23 @@ def _normalize_tool_call(tool_call: Dict[str, Any]) -> Dict[str, Any]: } -def _single_line_preview(value: Any, *, limit: int) -> str: - """Return a compact single-line preview for UI metadata.""" +def _single_line_preview(value: Any, *, limit: int, keep_tail: bool = False) -> str: + """Return a compact single-line preview for UI metadata. + + ``keep_tail`` preserves both ends. Diagnostic text states its cause last — a + traceback's final line, a compiler's error summary — so a head-only cut shows + the least informative part of exactly the output a user needs to read. + """ if value is None: return "" text = value if isinstance(value, str) else json.dumps(value, default=str, ensure_ascii=False) compact = " ".join(text.split()) if len(compact) <= limit: return compact + if keep_tail: + head = max(1, (limit - 1) * 2 // 5) + tail = max(1, limit - 1 - head) + return compact[:head] + "…" + compact[-tail:] return compact[: limit - 1] + "…" @@ -159,8 +169,15 @@ def _tool_args_metadata( arguments: Dict[str, Any] | None, *, original_tool_name: str | None = None, + tool_call_id: str = "", ) -> Dict[str, Any]: - """Build safe, compact tool-start metadata for streaming UIs.""" + """Build safe, compact tool-start metadata for streaming UIs. + + ``tool_call_id`` is included so a UI can correlate a start with its own + completion: a parallel batch emits several starts before any finishes, and + without the id a renderer can only track "the last tool", which mislabels + every line in the batch. + """ args = dict(arguments or {}) original_name = original_tool_name or tool_name metadata: Dict[str, Any] = { @@ -169,6 +186,8 @@ def _tool_args_metadata( "normalized_tool_name": tool_name, "args_summary": _single_line_preview(args, limit=_TOOL_ARGS_PREVIEW_LIMIT), } + if tool_call_id: + metadata["tool_call_id"] = tool_call_id resolution = _resolve_tool_name(original_name, args) metadata.update(resolution.to_metadata()) metadata["tool_name"] = tool_name @@ -188,9 +207,15 @@ def _tool_result_metadata( result: Any, *, original_tool_name: str | None = None, + tool_call_id: str = "", ) -> Dict[str, Any]: """Build safe, compact tool-completion metadata for streaming UIs.""" - metadata = _tool_args_metadata(tool_name, arguments, original_tool_name=original_tool_name) + metadata = _tool_args_metadata( + tool_name, + arguments, + original_tool_name=original_tool_name, + tool_call_id=tool_call_id, + ) if tool_name in {"platform_action", "gp_platform_action"} and arguments: for key in ("platform", "action"): value = arguments.get(key) @@ -199,7 +224,10 @@ def _tool_result_metadata( metadata["ok"] = True if isinstance(result, dict): metadata["ok"] = bool(result.get("ok", True)) - for key in ("exit_code", "path", "lines", "truncated", "bytes_written"): + exit_code = exit_code_from(result) + if exit_code is not None: + metadata["exit_code"] = exit_code + for key in ("path", "lines", "truncated", "bytes_written"): if key in result: metadata[key] = result[key] for key in ( @@ -232,6 +260,9 @@ def _tool_result_metadata( metadata[f"{key}_preview"] = _single_line_preview( value, limit=_TOOL_RESULT_PREVIEW_LIMIT, + # On failure these fields carry the diagnosis, and the cause is + # at the end of them. + keep_tail=metadata["ok"] is False and key in {"stderr", "error", "stdout"}, ) # App Connector recovery metadata for TUI transparency recovery_hint = result.get("recovery_hint") @@ -3549,6 +3580,7 @@ async def _unified_tool_loop_stream( normalized_name, tc.arguments, original_tool_name=original_name, + tool_call_id=str(tc.id), ), ) results = await self._execute_tools_concurrent( @@ -3578,6 +3610,7 @@ async def _unified_tool_loop_stream( tc.arguments, item.get("result"), original_tool_name=original_name, + tool_call_id=str(tc.id), ), **self._tool_context_metadata(normalized_name, tc.arguments, item.get("result")), }, diff --git a/src/leapflow/engine/tool_execution.py b/src/leapflow/engine/tool_execution.py index 98b287e..61bba7d 100644 --- a/src/leapflow/engine/tool_execution.py +++ b/src/leapflow/engine/tool_execution.py @@ -23,6 +23,25 @@ def effect_is_uncertain_on_failure(policy: str) -> bool: """Return whether a failed call under ``policy`` may still have taken effect.""" return str(policy or "") in UNCERTAIN_EFFECT_POLICIES + +def exit_code_from(result: Any) -> int | None: + """Return a process exit code from a tool result under either key name. + + Shell-shaped tools mirror ``subprocess``' own ``returncode`` attribute, while + the model-facing evidence and the TUI read ``exit_code``. Reading only one + name silently dropped the code from both surfaces, so the mapping lives here + once instead of being re-guessed per consumer. + """ + if not isinstance(result, Mapping): + return None + for key in ("exit_code", "returncode"): + value = result.get(key) + if isinstance(value, bool): + continue + if isinstance(value, int): + return value + return None + _EXTERNAL_TOOLS = frozenset({ "shell_run", "scm_sync", diff --git a/src/leapflow/security/actions.py b/src/leapflow/security/actions.py index c724c9b..f1507b8 100644 --- a/src/leapflow/security/actions.py +++ b/src/leapflow/security/actions.py @@ -24,6 +24,7 @@ class ActionKind(str, Enum): SKILL_PROMOTE = "skill.promote" APP_INSTALL = "app.install" RUNTIME_CONFIGURE = "runtime.configure" + NETWORK_FETCH = "network.fetch" EXTERNAL_ACTION = "external.action" @@ -177,6 +178,33 @@ def platform_action( metadata=merged, ) + @classmethod + def network_fetch( + cls, + url: str, + *, + origin: str, + method: str = "GET", + metadata: dict[str, Any] | None = None, + ) -> "ActionDescriptor": + """Describe an outbound HTTP read before it leaves the machine. + + ``resource`` is the origin rather than the full URL so a session grant + means "this host is trusted for now" instead of expiring on the next + path or query string, which would turn progressive trust into a prompt + per request. + """ + merged = dict(metadata or {}) + merged.update({"url": url, "method": method, "origin": origin}) + return cls( + kind=ActionKind.NETWORK_FETCH.value, + summary=f"Fetch {method} {origin}", + detail=url, + effect=ActionEffect.READ.value, + resource=origin, + metadata=merged, + ) + def signature(self) -> str: """Return a stable signature suitable for session/profile grants.""" payload = { @@ -237,4 +265,9 @@ def _normalize_detail(kind: str, detail: str) -> str: text = re.sub(r"\s+", " ", detail.strip()) if kind in {ActionKind.GATEWAY_SEND.value, ActionKind.PLATFORM_ACTION.value}: return "" + if kind == ActionKind.NETWORK_FETCH.value: + # Collapsed on purpose: the grant is scoped by origin (the resource), so + # keeping the full URL here would mint a separate grant per path and + # query string and re-prompt for every request to an approved host. + return "" return text[:4000] diff --git a/src/leapflow/security/network.py b/src/leapflow/security/network.py new file mode 100644 index 0000000..fd61cc7 --- /dev/null +++ b/src/leapflow/security/network.py @@ -0,0 +1,207 @@ +"""Outbound URL classification for the network egress gate. + +Splitting this out of ``risk.py`` keeps the risk classifier synchronous and +I/O-free: deciding whether a host is internal requires DNS resolution, which is +I/O and must not run inside an approval decision. The tool resolves the target +here (off the event loop), then passes the verdict as action metadata for the +classifier to judge. + +Name resolution matters, not just literal addresses: a public hostname can point +at ``127.0.0.1`` or a cloud metadata address, so a gate that only inspected the +literal host would pass exactly the requests worth stopping. +""" + +from __future__ import annotations + +import asyncio +import ipaddress +import socket +from dataclasses import dataclass +from typing import Iterable +from urllib.parse import urlsplit + +# Only these reach the network. Everything else (file://, gopher://, data:, ...) +# is refused as a malformed request rather than judged for risk: a fetch tool +# that could read local files would bypass the workspace boundary entirely. +ALLOWED_SCHEMES: frozenset[str] = frozenset({"http", "https"}) + +# Cloud instance metadata services. Named explicitly, separately from the ranges +# that enclose them, because the consequence is specific and worth stating in an +# approval prompt: these endpoints hand out instance credentials. +_METADATA_ADDRESSES: frozenset[str] = frozenset({ + "169.254.169.254", # AWS / GCP / Azure / OpenStack IMDS + "fd00:ec2::254", # AWS IMDSv2 over IPv6 + "100.100.100.200", # Alibaba Cloud ECS metadata +}) + +_DEFAULT_PORTS = {"http": 80, "https": 443} + + +class UrlRejected(ValueError): + """Raised when a URL cannot be fetched at all, regardless of approval.""" + + def __init__(self, reason: str, detail: str) -> None: + super().__init__(detail) + self.reason = reason + self.detail = detail + + +@dataclass(frozen=True) +class NetworkTarget: + """A resolved fetch target and the verdict the approval path needs.""" + + url: str + scheme: str + host: str + port: int + origin: str + category: str + addresses: tuple[str, ...] = () + has_credentials: bool = False + + @property + def is_internal(self) -> bool: + """Whether the target reaches something other than the public internet.""" + return self.category != "public" + + def to_metadata(self) -> dict[str, object]: + """Return the fields the risk classifier and audit trail consume.""" + return { + "scheme": self.scheme, + "host": self.host, + "port": self.port, + "origin": self.origin, + "target_category": self.category, + "has_credentials": self.has_credentials, + "resolved_addresses": list(self.addresses), + } + + +def _category_for_address(address: str) -> str: + """Classify one resolved IP address into a trust category. + + "public" is decided by ``is_global`` rather than by ``not is_private``: the + private-address test misses ranges that are unroutable but not RFC1918, most + consequentially the shared address space (100.64.0.0/10) that carries Alibaba + Cloud's metadata service. Anything the stdlib does not consider globally + routable is therefore treated as internal, which fails closed for ranges we + have not enumerated. + """ + if address in _METADATA_ADDRESSES: + return "metadata" + try: + ip = ipaddress.ip_address(address) + except ValueError: + # An address the resolver returned but ipaddress cannot parse is not + # something to optimistically treat as public. + return "reserved" + if ip.is_loopback: + return "loopback" + if ip.is_link_local: + return "link_local" + if ip.is_unspecified: + return "unspecified" + if ip.is_private: + return "private" + if not ip.is_global: + return "reserved" + return "public" + + +def _worst_category(categories: Iterable[str]) -> str: + """Return the most restrictive category among resolved addresses. + + A hostname with both a public and a loopback record must be treated as + loopback: the connection may take either, so the safe reading is the worse + one. + """ + # Materialized before the scan: ``categories`` may be a generator, and + # re-evaluating it inside the comprehension would consume it on the first + # probe and then read as empty for every remaining category. + present = set(categories) + order = ("metadata", "unspecified", "loopback", "link_local", "private", "reserved", "public") + ranked = [c for c in order if c in present] + return ranked[0] if ranked else "unknown" + + +def _split_url(url: str) -> tuple[str, str, int, bool]: + parts = urlsplit(url.strip()) + scheme = (parts.scheme or "").lower() + if scheme not in ALLOWED_SCHEMES: + raise UrlRejected( + "unsupported_scheme", + f"Only http and https URLs can be fetched; got {scheme or 'no scheme'!r}.", + ) + host = (parts.hostname or "").strip() + if not host: + raise UrlRejected("missing_host", "The URL has no host component.") + try: + port = parts.port or _DEFAULT_PORTS[scheme] + except ValueError as exc: # malformed port, e.g. https://host:notaport/ + raise UrlRejected("invalid_port", f"The URL port is not a number: {exc}") from exc + return host, scheme, port, bool(parts.username or parts.password) + + +async def classify_url(url: str, *, resolve: bool = True) -> NetworkTarget: + """Return the classified target for ``url``. + + ``resolve=False`` skips DNS and classifies from the literal host only. That + is for offline/unit contexts; leaving it on is what catches a public name + pointing at an internal address. + """ + host, scheme, port, has_credentials = _split_url(url) + default_port = _DEFAULT_PORTS[scheme] + origin = f"{scheme}://{host}" if port == default_port else f"{scheme}://{host}:{port}" + + literal = host.strip("[]") + try: + ipaddress.ip_address(literal) + except ValueError: + addresses: tuple[str, ...] = () + if resolve: + addresses = await _resolve(host, port) + else: + addresses = (literal,) + + if addresses: + category = _worst_category(_category_for_address(a) for a in addresses) + elif resolve: + # Resolution failed. Report it as a target we could not vet rather than + # letting the request through unclassified. + raise UrlRejected("dns_error", f"Could not resolve host {host!r}.") + else: + category = "unknown" + + return NetworkTarget( + url=url, + scheme=scheme, + host=host, + port=port, + origin=origin, + category=category, + addresses=addresses, + has_credentials=has_credentials, + ) + + +async def _resolve(host: str, port: int) -> tuple[str, ...]: + """Resolve ``host`` to its addresses without blocking the event loop.""" + loop = asyncio.get_running_loop() + try: + infos = await loop.getaddrinfo(host, port, proto=socket.IPPROTO_TCP) + except (socket.gaierror, OSError): + return () + seen: list[str] = [] + for info in infos: + address = str(info[4][0]) + if address not in seen: + seen.append(address) + return tuple(seen) + + +__all__ = [ + "ALLOWED_SCHEMES", + "NetworkTarget", + "UrlRejected", + "classify_url", +] diff --git a/src/leapflow/security/risk.py b/src/leapflow/security/risk.py index f62a0a8..ee2b7ba 100644 --- a/src/leapflow/security/risk.py +++ b/src/leapflow/security/risk.py @@ -102,6 +102,8 @@ def assess(self, action: ActionDescriptor) -> RiskAssessment: return self._assess_file_read(action) if action.kind == ActionKind.FILE_WRITE.value: return self._assess_file_write(action) + if action.kind == ActionKind.NETWORK_FETCH.value: + return self._assess_network_fetch(action) if action.kind == ActionKind.GATEWAY_SEND.value: return RiskAssessment( level=RiskLevel.HIGH, @@ -145,6 +147,71 @@ def assess(self, action: ActionDescriptor) -> RiskAssessment: ) return RiskAssessment(level=RiskLevel.MEDIUM, score=0.5, reasons=("external_action",)) + # Target categories that mean "this request would reach infrastructure the + # user did not intend to expose": loopback holds the daemon socket and the + # dashboard, link-local holds cloud instance metadata (credential theft), and + # private ranges hold the rest of the user's network. + _INTERNAL_TARGETS = frozenset({ + "loopback", "private", "link_local", "metadata", "unspecified", "reserved", + }) + + def _assess_network_fetch(self, action: ActionDescriptor) -> RiskAssessment: + """Assess an outbound HTTP read. + + Deliberately never CRITICAL and never hardline: policy turns both into an + outright block, and the product decision is that internal targets stay + reachable through an explicit human approval rather than being refused. + HIGH plus ``allow_permanent=False`` is what expresses "ask every session, + never remember forever". + """ + category = str(action.metadata.get("target_category") or "unknown") + scheme = str(action.metadata.get("scheme") or "").lower() + origin = str(action.resource or "") + + if action.metadata.get("has_credentials"): + return RiskAssessment( + level=RiskLevel.HIGH, + score=0.8, + reasons=("url_embedded_credentials",), + explanation=( + "The URL carries credentials, so fetching it would send them to " + f"{origin} and record them in this turn." + ), + allow_permanent=False, + metadata={"origin": origin, "target_category": category}, + ) + + if category in self._INTERNAL_TARGETS: + return RiskAssessment( + level=RiskLevel.HIGH, + score=0.82, + reasons=(f"{category}_network_target", "internal_service_access"), + explanation=( + f"{origin} resolves to a {category.replace('_', ' ')} address, which " + "can reach services on this machine or private network rather than " + "the public internet." + ), + allow_permanent=False, + metadata={"origin": origin, "target_category": category}, + ) + + if scheme and scheme != "https": + return RiskAssessment( + level=RiskLevel.LOW, + score=0.3, + reasons=("public_network_read", "plaintext_transport"), + explanation=f"Reads {origin} over plaintext {scheme}.", + metadata={"origin": origin, "target_category": category}, + ) + + return RiskAssessment( + level=RiskLevel.LOW, + score=0.15, + reasons=("public_network_read",), + explanation=f"Reads public web content from {origin}.", + metadata={"origin": origin, "target_category": category}, + ) + @staticmethod def _platform_risk_level(action: ActionDescriptor) -> RiskAssessment | None: raw = str(action.metadata.get("risk_level") or "").lower() diff --git a/src/leapflow/tools/execution_context.py b/src/leapflow/tools/execution_context.py index 57b8a88..be06653 100644 --- a/src/leapflow/tools/execution_context.py +++ b/src/leapflow/tools/execution_context.py @@ -90,13 +90,17 @@ def is_within_allowed_roots(path: Path, ctx: ToolExecutionContext | None = None) return False -def _leapflow_managed_hint(path: Path) -> str: +def leapflow_managed_hint(path: Path) -> str: """Return a redirect hint when ``path`` is LeapFlow's own managed state. A refusal that only says "outside the workspace" leaves the model to guess another path, which is how a config change turns into a sequence of blocked probes. Classification comes from the layout descriptor rather than string matching, so it follows the path tree instead of duplicating it. + + Public because the shell gate refuses the same targets through a different + entry point; a hint that only the file tools emit would leave the shell path + telling the model nothing about what to use instead. """ try: from leapflow.config import get_settings @@ -132,7 +136,7 @@ def workspace_scope_error(path: Path, *, operation: str) -> dict[str, Any] | Non ctx = current_tool_context() if ctx is None or is_within_allowed_roots(path, ctx): return None - hint = _leapflow_managed_hint(path) + hint = leapflow_managed_hint(path) return { "ok": False, "error": ( diff --git a/src/leapflow/tools/name_resolver.py b/src/leapflow/tools/name_resolver.py index 01a2389..0d1ad89 100644 --- a/src/leapflow/tools/name_resolver.py +++ b/src/leapflow/tools/name_resolver.py @@ -36,6 +36,10 @@ "skills_list", "skill_view", "memory_search", + # Network reads: a GET has no side effect, so the loop must not treat it as a + # mutation. Egress safety is enforced by the tool's own target gate, not by + # pretending the call mutates state. + "web_fetch", # Verification/inspection tools: read-oriented for the loop even though some # names contain a mutating signal (e.g. test_run) or execute via a gated # underlying tool (test_run/lint_check delegate to the shell_run gate). diff --git a/src/leapflow/tools/registry_bootstrap.py b/src/leapflow/tools/registry_bootstrap.py index cf16558..81fadac 100644 --- a/src/leapflow/tools/registry_bootstrap.py +++ b/src/leapflow/tools/registry_bootstrap.py @@ -245,7 +245,17 @@ "type": "function", "function": { "name": "shell_run", - "description": "Execute a shell command with timeout protection.", + "description": ( + "Execute a one-shot shell command with timeout protection. Runs in the " + "active workspace; paths resolving outside it are refused. Reach for a " + "structured tool first when one fits — web_fetch for anything over " + "HTTP(S), git_query/scm_sync for git, code_search/file_find/file_read for " + "the repo, config_get/config_set for LeapFlow's own settings — because " + "those report typed results, while a failed shell command can only be " + "diagnosed from its exit code and stderr. Every shell run counts as an " + "external side effect, so a failure stops the rest of the batch and is " + "not retried automatically." + ), "parameters": { "type": "object", "properties": { @@ -255,6 +265,21 @@ }, "required": ["command"], }, + "x_leapflow": { + # Declared rather than inferred: without this block the category and + # risk came from keyword matching on the name/description, and the + # execution policy came from a separate hardcoded name list. The most + # dangerous tool in the registry should state its own contract. + "category": "shell", + "risk_level": "external", + "schema_cost": "low", + "requires_approval": True, + "mutates_state": True, + "effect_scope": "external", + # An arbitrary command may not be replayable, so identity is scoped to + # the session rather than the turn. + "idempotency_scope": "session", + }, }, }, { @@ -765,6 +790,53 @@ }, }, }, + # ── Web access (read-only; never shell out to curl for this) ── + { + "type": "function", + "function": { + "name": "web_fetch", + "description": ( + "Read a URL over HTTP(S) and get back extracted, context-sized content: " + "parsed JSON for API endpoints, readable text plus links for web pages. " + "Use this for anything on the internet — prices, docs, releases, articles " + "— instead of running curl through shell_run: it reports real HTTP status " + "codes, retries rate limits on its own, and is a plain read so a retry is " + "always safe. For JSON APIs pass `select` with a dotted path (e.g. " + "'chart.result.0.meta') to return just that part instead of the whole " + "payload." + ), + "parameters": { + "type": "object", + "properties": { + "url": {"type": "string", "description": "http(s) URL to read"}, + "select": { + "type": "string", + "description": ( + "Optional dotted path into a JSON response, list indices " + "allowed, e.g. 'chart.result.0.meta.regularMarketPrice'" + ), + }, + "timeout": {"type": "number", "description": "Timeout in seconds (default from config)"}, + "max_bytes": {"type": "integer", "description": "Response size cap in bytes"}, + }, + "required": ["url"], + }, + "x_leapflow": { + "category": "network", + # read_only is the point of this tool: a GET is not a side effect, so + # it must not inherit shell's batch-stop, session-scoped dedup, or + # "may already have taken effect" retry guidance. + "risk_level": "read_only", + "schema_cost": "low", + # Public reads run unprompted; the handler still routes internal and + # credential-bearing targets through the approval gate, since this + # flag only informs disclosure and does not gate execution. + "requires_approval": False, + "mutates_state": False, + "idempotency_scope": "turn", + }, + }, + }, ] + HUB_TOOL_DEFINITIONS + GATEWAY_TOOL_DEFINITIONS @@ -1155,6 +1227,17 @@ async def _memory_add_handler(params: Dict[str, Any]) -> Dict[str, Any]: TOOL_HANDLERS[f"gp_{_cfg_name}"] = _cfg_handler +# ──────────────────────────────────────────────────────────────── +# Web access: a read-only HTTP capability so reaching the internet does not +# require an improvised `curl | python3 -c` pipeline through the shell gate. +# ──────────────────────────────────────────────────────────────── + +from leapflow.tools.web_fetch import web_fetch as _web_fetch_handler # noqa: E402 + +TOOL_HANDLERS["web_fetch"] = _web_fetch_handler +TOOL_HANDLERS["gp_web_fetch"] = _web_fetch_handler + + # ──────────────────────────────────────────────────── # Research-ledger tool late-binding: delegates to the engine's per-task # ResearchLedger when installed; fails gracefully when not. diff --git a/src/leapflow/tools/shell_tools.py b/src/leapflow/tools/shell_tools.py index ba995b7..be5bd66 100644 --- a/src/leapflow/tools/shell_tools.py +++ b/src/leapflow/tools/shell_tools.py @@ -21,6 +21,7 @@ from leapflow.tools.execution_context import ( current_tool_context, is_within_allowed_roots, + leapflow_managed_hint, resolve_workspace_path, workspace_scope_error, ) @@ -140,16 +141,53 @@ async def _approve_command(command: str, cwd: str | None) -> tuple[bool, str]: return False, "Dangerous command requires approval (denied)" -def _command_workspace_escape(command: str) -> dict[str, Any] | None: - """Reject absolute path operands outside the active workspace. +def _expand_operand(token: str) -> str: + """Return the inspectable path operand carried by a shell token. - Shell is intentionally a broad escape hatch, so this is a conservative P0 - guard rather than a full shell parser: it catches explicit absolute / ~ path - arguments that would let a daemon-backed turn read another TUI's workspace. + Strips a leading ``--flag=`` and expands variable references, because the + shell expands them at execution time: ``$HOME/x`` and ``/Users/me/x`` reach + the same file, so a gate comparing raw text would guard the spelling rather + than the target. Unset variables are left literal by ``expandvars`` and then + fail the prefix/traversal tests below, which is the safe direction. + + Expansions that are not a single filesystem operand are discarded: a variable + holding a search list (``$PATH``) expands to ``os.pathsep``-joined entries + that begin with ``/`` but name no file, so treating it as a path would block + ordinary commands like ``echo $PATH`` or ``PATH=$PATH:./bin npm test``. + """ + candidate = token.split("=", 1)[-1] + if "$" not in candidate: + return candidate + expanded = os.path.expandvars(candidate) + if os.pathsep in expanded or any(char.isspace() for char in expanded): + return "" + return expanded + + +def _has_parent_traversal(operand: str) -> bool: + """Return whether a relative operand walks upward out of its base directory.""" + return ".." in Path(operand).parts + + +def _command_workspace_escape(command: str, cwd: Path | None = None) -> dict[str, Any] | None: + """Reject path operands that resolve outside the active workspace. + + Shell is intentionally a broad escape hatch, so this stays a conservative + guard rather than a full shell parser. It does normalize what the shell + itself would expand before deciding: variable references (``$HOME``, + ``${HOME}``) and relative traversal (``../..``) address exactly the same + files as an absolute path, so inspecting only literal ``/`` or ``~`` tokens + would let a daemon-backed turn read another TUI's workspace through a + different spelling. + + Command substitution (``$(...)``, backticks) and similar indirection remain + out of reach by design; the hardline patterns, approval gate, and execution + ledger are the defenses there. """ ctx = current_tool_context() if ctx is None: return None + base = cwd if cwd is not None else ctx.workspace_root try: tokens = shlex.split(command) except ValueError: @@ -157,21 +195,31 @@ def _command_workspace_escape(command: str) -> dict[str, Any] | None: for token in tokens: if token.startswith(("http://", "https://")): continue - candidate = token.split("=", 1)[-1] - if not candidate.startswith(("/", "~")): + operand = _expand_operand(token) + if not operand: + continue + if operand.startswith(("/", "~")): + path = Path(operand) + elif _has_parent_traversal(operand): + path = base / operand + else: continue - path = Path(candidate).expanduser().resolve() - if not is_within_allowed_roots(path, ctx): + resolved = path.expanduser().resolve() + if not is_within_allowed_roots(resolved, ctx): return { "ok": False, "error": ( "shell_run command references a path outside the active workspace. " - f"Path: {path}; workspace root: {ctx.workspace_root}." + f"Path: {resolved}; workspace root: {ctx.workspace_root}. " + "This boundary cannot be lifted by approval; work inside the workspace, " + "or ask the user to open a session in that directory." + + leapflow_managed_hint(resolved) ), "error_type": "outside_workspace", "retryable": False, "workspace_root": str(ctx.workspace_root), - "resolved_path": str(path), + "resolved_path": str(resolved), + "operand": token, "session_id": ctx.session_id, } return None @@ -202,7 +250,7 @@ async def shell_run(params: Dict[str, Any]) -> Dict[str, Any]: if scope_error: return scope_error - command_scope_error = _command_workspace_escape(str(command)) + command_scope_error = _command_workspace_escape(str(command), cwd=cwd_path) if command_scope_error: return command_scope_error diff --git a/src/leapflow/tools/web_cache.py b/src/leapflow/tools/web_cache.py new file mode 100644 index 0000000..979a234 --- /dev/null +++ b/src/leapflow/tools/web_cache.py @@ -0,0 +1,193 @@ +"""Session-scoped body cache for ``web_fetch``. + +Split out of the tool because storage is a separate responsibility from transport +and gating: this module owns where a fetched body lives, how long it stays valid, +and how it is indexed — nothing about how it was retrieved. + +Every path comes from ``CacheLayout`` rather than being assembled here, and every +failure degrades to "no cache" instead of failing the fetch: caching is an +optimization, never a requirement. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +logger = logging.getLogger(__name__) + +CACHE_CATEGORY = "web_fetch" + + +@dataclass(frozen=True) +class CachedBody: + """A previously fetched body and the response facts needed to replay it.""" + + body: bytes + status: int + final_url: str + content_type: str + truncated: bool + transport: str + elapsed_ms: int + path: Path + + +@dataclass(frozen=True) +class _Slot: + body_path: Path + workspace_id: str + session_id: str + + +def _ttl(settings: Any) -> float: + try: + return max(0.0, float(getattr(settings, "web_cache_ttl_s", 0) or 0)) + except (TypeError, ValueError): + return 0.0 + + +def _slot(url: str, settings: Any) -> _Slot | None: + """Return the layout-owned location for ``url``, or ``None`` if unavailable. + + Session scope also requires a workspace id, which keeps one TUI's fetches out + of another's cache — the same isolation the rest of the cache tree follows. + """ + from leapflow.tools.execution_context import current_tool_context + + ctx = current_tool_context() + session_id = getattr(ctx, "session_id", "") or "default" + try: + from leapflow.cache.manager import CacheManager, CacheScope + from leapflow.layout import workspace_id_for_path + + workspace_root = getattr(ctx, "workspace_root", None) or settings.workspace_root + workspace_id = workspace_id_for_path(Path(workspace_root)) + manager = CacheManager(settings.profile_layout.cache, profile_id=settings.profile) + directory = manager.path( + scope=CacheScope.SESSION, + category=CACHE_CATEGORY, + workspace_id=workspace_id, + session_id=session_id, + source="body", + ) + except Exception: # noqa: BLE001 - caching must never break a fetch + logger.debug("web cache unavailable", exc_info=True) + return None + digest = hashlib.sha256(url.encode("utf-8")).hexdigest()[:32] + return _Slot(directory / f"{digest}.bin", workspace_id, session_id) + + +def _meta_path(body_path: Path) -> Path: + return body_path.with_suffix(".meta.json") + + +def cached_path(url: str, settings: Any) -> Path | None: + """Return where ``url``'s body is stored, without reading it.""" + slot = _slot(url, settings) + return slot.body_path if slot else None + + +def read(url: str, settings: Any) -> CachedBody | None: + """Return a fresh cached body for ``url``, or ``None``.""" + if _ttl(settings) <= 0: + return None + slot = _slot(url, settings) + if slot is None: + return None + meta_path = _meta_path(slot.body_path) + try: + if not (slot.body_path.exists() and meta_path.exists()): + return None + meta = json.loads(meta_path.read_text(encoding="utf-8")) + if time.time() - float(meta.get("fetched_at", 0)) > _ttl(settings): + return None + body = slot.body_path.read_bytes() + except (OSError, ValueError, json.JSONDecodeError): + logger.debug("web cache entry unreadable for %s", url, exc_info=True) + return None + return CachedBody( + body=body, + status=int(meta.get("status", 200)), + final_url=str(meta.get("final_url") or url), + content_type=str(meta.get("content_type") or ""), + truncated=bool(meta.get("truncated", False)), + transport=str(meta.get("transport") or "cache"), + elapsed_ms=int(meta.get("elapsed_ms", 0)), + path=slot.body_path, + ) + + +def write( + url: str, + body: bytes, + *, + status: int, + final_url: str, + content_type: str, + truncated: bool, + transport: str, + elapsed_ms: int, + origin: str, + sensitive: bool, + settings: Any, +) -> Path | None: + """Persist a body, index it, and return where it was stored.""" + ttl = _ttl(settings) + if ttl <= 0: + return None + slot = _slot(url, settings) + if slot is None: + return None + try: + slot.body_path.write_bytes(body) + _meta_path(slot.body_path).write_text( + json.dumps( + { + "url": url, + "final_url": final_url, + "status": status, + "content_type": content_type, + "truncated": truncated, + "transport": transport, + "elapsed_ms": elapsed_ms, + "fetched_at": time.time(), + }, + ensure_ascii=False, + ), + encoding="utf-8", + ) + except OSError: + logger.debug("web cache write failed for %s", url, exc_info=True) + return None + try: + from leapflow.cache.manager import CacheManager, CacheScope + + manager = CacheManager(settings.profile_layout.cache, profile_id=settings.profile) + manager.register( + path=slot.body_path, + scope=CacheScope.SESSION, + category=CACHE_CATEGORY, + source="body", + workspace_id=slot.workspace_id, + session_id=slot.session_id, + content_hash=hashlib.sha256(body).hexdigest(), + expires_at=time.time() + ttl, + # Fetched content is cheap to re-acquire and the URL itself can be the + # secret, so it never leaves the machine. + sensitive=sensitive, + syncable=False, + owner_component="web_fetch", + metadata={"url": url, "status": status, "origin": origin}, + ) + except Exception: # noqa: BLE001 - an unindexed cache file is still usable + logger.debug("web cache index update failed", exc_info=True) + return slot.body_path + + +__all__ = ["CACHE_CATEGORY", "CachedBody", "cached_path", "read", "write"] diff --git a/src/leapflow/tools/web_extract.py b/src/leapflow/tools/web_extract.py new file mode 100644 index 0000000..6afe496 --- /dev/null +++ b/src/leapflow/tools/web_extract.py @@ -0,0 +1,375 @@ +"""Content extraction for fetched web responses. + +A fetch tool that hands raw markup to the model is not usable: a single page can +exceed the whole turn's context budget, and the useful text is a small fraction +of it. Extraction is therefore part of the capability, not an afterthought. + +Routing is driven by the response's ``Content-Type`` header rather than by +sniffing the body, so the decision follows the protocol contract instead of +guessing from content. Two extractor implementations exist for HTML: a +dependency-free stdlib parser that always ships, and trafilatura when the +``web`` extra is installed. The stdlib one is not a placeholder — it stays the +fallback whenever trafilatura is absent *or* declines a page (it returns nothing +on layouts it cannot model), so both paths are load-bearing. +""" + +from __future__ import annotations + +import json +import logging +import re +from dataclasses import dataclass, field +from html.parser import HTMLParser +from typing import Any, Protocol, runtime_checkable +from urllib.parse import urljoin + +logger = logging.getLogger(__name__) + +# Kinds the tool reports back to the model. Binary never enters the transcript; +# it is written to cache and referenced by path instead. +KIND_JSON = "json" +KIND_TEXT = "text" +KIND_HTML = "html" +KIND_BINARY = "binary" + +_JSON_TYPES = ("application/json", "text/json") +_HTML_TYPES = ("text/html", "application/xhtml+xml") +_TEXT_TYPES = ( + "text/plain", "text/markdown", "text/csv", "text/tab-separated-values", + "application/xml", "text/xml", "application/javascript", "text/javascript", + "application/x-yaml", "text/yaml", +) + +# Elements whose text is chrome, navigation, or code that never belongs in an +# extracted reading of the page. +_SKIP_TAGS = frozenset({ + "script", "style", "noscript", "svg", "canvas", "iframe", "form", + "nav", "footer", "aside", "template", "select", "button", "head", +}) +_BLOCK_TAGS = frozenset({ + "p", "div", "section", "article", "main", "header", "br", "hr", "li", "tr", + "td", "th", "pre", "blockquote", "figcaption", "dt", "dd", "table", "ul", + "ol", "h1", "h2", "h3", "h4", "h5", "h6", +}) +_HEADING_LEVELS = {"h1": 1, "h2": 2, "h3": 3, "h4": 4, "h5": 5, "h6": 6} +_MAX_LINKS = 30 +_BLANK_RUN = re.compile(r"\n{3,}") + + +@dataclass(frozen=True) +class ExtractedContent: + """Normalized, context-ready view of a response body.""" + + kind: str + text: str = "" + data: Any = None + title: str = "" + links: tuple[tuple[str, str], ...] = () + truncated: bool = False + extractor: str = "" + notes: tuple[str, ...] = field(default_factory=tuple) + + +@runtime_checkable +class ContentExtractor(Protocol): + """Turns an HTML document into readable text.""" + + name: str + + def available(self) -> bool: + """Whether this extractor can run in the current environment.""" + ... + + def extract(self, html: str, *, url: str) -> ExtractedContent | None: + """Return extracted content, or ``None`` to defer to the next extractor.""" + ... + + +def kind_for_content_type(content_type: str) -> str: + """Map a Content-Type header to an extraction kind.""" + base = (content_type or "").split(";", 1)[0].strip().lower() + if not base: + return KIND_TEXT + if base in _JSON_TYPES or base.endswith("+json"): + return KIND_JSON + if base in _HTML_TYPES: + return KIND_HTML + if base in _TEXT_TYPES or base.startswith("text/"): + return KIND_TEXT + return KIND_BINARY + + +class _HtmlTextParser(HTMLParser): + """Collect readable text, a title, and links from an HTML document.""" + + def __init__(self, base_url: str) -> None: + super().__init__(convert_charrefs=True) + self._base_url = base_url + self._skip_depth = 0 + self._in_title = False + self._pending_heading = 0 + self._parts: list[str] = [] + self._links: list[tuple[str, str]] = [] + self._link_text: list[str] = [] + self._link_href = "" + self.title = "" + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + if tag in _SKIP_TAGS: + # lives inside <head>, so keep reading it while skipping the + # rest of the head's machinery. + self._skip_depth += 1 + return + if self._skip_depth and tag != "title": + return + if tag == "title": + self._in_title = True + return + if tag in _HEADING_LEVELS: + self._newline() + self._pending_heading = _HEADING_LEVELS[tag] + return + if tag == "li": + self._newline() + self._parts.append("- ") + return + if tag == "a": + self._link_href = next((v or "" for k, v in attrs if k == "href"), "") + self._link_text = [] + return + if tag in _BLOCK_TAGS: + self._newline() + + def handle_endtag(self, tag: str) -> None: + if tag in _SKIP_TAGS: + self._skip_depth = max(0, self._skip_depth - 1) + return + if tag == "title": + self._in_title = False + return + if self._skip_depth: + return + if tag == "a" and self._link_href: + text = " ".join("".join(self._link_text).split()) + url = urljoin(self._base_url, self._link_href) + if text and url.startswith(("http://", "https://")): + self._links.append((text, url)) + self._link_href = "" + self._link_text = [] + return + if tag in _BLOCK_TAGS or tag in _HEADING_LEVELS: + self._newline() + + def handle_data(self, data: str) -> None: + if self._in_title: + self.title = (self.title + data).strip() + return + if self._skip_depth or not data.strip(): + return + if self._pending_heading: + self._parts.append("#" * self._pending_heading + " ") + self._pending_heading = 0 + if self._link_href: + self._link_text.append(data) + self._parts.append(data) + + def _newline(self) -> None: + if self._parts and not self._parts[-1].endswith("\n"): + self._parts.append("\n") + + def text(self) -> str: + raw = "".join(self._parts) + lines = [" ".join(line.split()) for line in raw.splitlines()] + return _BLANK_RUN.sub("\n\n", "\n".join(lines)).strip() + + def links(self) -> tuple[tuple[str, str], ...]: + seen: set[str] = set() + unique: list[tuple[str, str]] = [] + for text, url in self._links: + if url in seen: + continue + seen.add(url) + unique.append((text, url)) + if len(unique) >= _MAX_LINKS: + break + return tuple(unique) + + +class StdlibHtmlExtractor: + """Dependency-free HTML reader built on ``html.parser``. + + Always available, so a LeapFlow install with no extras can still read the + web. It removes chrome by element type rather than scoring text density, so + expect more boilerplate than trafilatura on article pages — the trade is zero + dependencies and no failure mode where extraction is simply unavailable. + """ + + name = "stdlib" + + def available(self) -> bool: + return True + + def extract(self, html: str, *, url: str) -> ExtractedContent | None: + parser = _HtmlTextParser(url) + try: + parser.feed(html) + parser.close() + except (AssertionError, ValueError) as exc: + # Malformed markup: report what was collected rather than failing the + # whole fetch, since a partial reading is still useful. + logger.debug("stdlib html extraction incomplete for %s: %s", url, exc) + return ExtractedContent( + kind=KIND_HTML, + text=parser.text(), + title=parser.title, + links=parser.links(), + extractor=self.name, + ) + + +class TrafilaturaExtractor: + """Boilerplate-removing extractor backed by the optional ``web`` extra. + + Returns ``None`` when trafilatura is missing or produces nothing for a page, + which hands the document to the stdlib extractor instead of reporting an + empty body. + """ + + name = "trafilatura" + + def available(self) -> bool: + try: + import trafilatura # noqa: F401 + except ImportError: + return False + return True + + def extract(self, html: str, *, url: str) -> ExtractedContent | None: + try: + import trafilatura + except ImportError: + return None + try: + text = trafilatura.extract( + html, + url=url, + include_links=True, + include_tables=True, + output_format="markdown", + with_metadata=False, + ) + except Exception as exc: # noqa: BLE001 - never fail a fetch on extraction + logger.debug("trafilatura extraction failed for %s: %s", url, exc) + return None + if not text or not text.strip(): + return None + title = "" + links: tuple[tuple[str, str], ...] = () + # Title and links are not part of trafilatura's markdown output, so take + # them from the always-present structural reader. + structural = StdlibHtmlExtractor().extract(html, url=url) + if structural is not None: + title = structural.title + links = structural.links + return ExtractedContent( + kind=KIND_HTML, + text=text.strip(), + title=title, + links=links, + extractor=self.name, + ) + + +def html_extractors(prefer: str = "auto") -> tuple[ContentExtractor, ...]: + """Return the extractor chain, most capable first. + + ``prefer='stdlib'`` pins the dependency-free reader so behavior can be made + reproducible regardless of which extras happen to be installed. + """ + stdlib = StdlibHtmlExtractor() + if prefer == "stdlib": + return (stdlib,) + candidates: list[ContentExtractor] = [TrafilaturaExtractor(), stdlib] + return tuple(item for item in candidates if item.available()) + + +def extract_html(html: str, *, url: str, prefer: str = "auto") -> ExtractedContent: + """Extract readable content, walking the extractor chain until one answers.""" + for extractor in html_extractors(prefer): + result = extractor.extract(html, url=url) + if result is not None and result.text: + return result + return ExtractedContent(kind=KIND_HTML, text="", extractor="none") + + +def select_path(data: Any, path: str) -> tuple[Any, str]: + """Return ``(value, error)`` for a dotted path into decoded JSON. + + Dotted segments with integer indices (``chart.result.0.meta``) cover the + shape of real API payloads without pulling in a JSONPath dependency. The + error string names the segment that failed and what was available, so the + model can correct the path in the same turn instead of re-fetching blindly. + """ + if not path: + return data, "" + current = data + walked: list[str] = [] + for segment in [item for item in path.split(".") if item]: + if isinstance(current, dict): + if segment not in current: + available = ", ".join(sorted(current)[:12]) or "no keys" + return None, ( + f"select path {path!r} failed at {'.'.join(walked + [segment])!r}: " + f"available keys: {available}" + ) + current = current[segment] + elif isinstance(current, list): + try: + index = int(segment) + except ValueError: + return None, ( + f"select path {path!r} failed at {'.'.join(walked + [segment])!r}: " + f"expected a list index, list has {len(current)} items" + ) + if not -len(current) <= index < len(current): + return None, ( + f"select path {path!r} failed at {'.'.join(walked + [segment])!r}: " + f"index out of range, list has {len(current)} items" + ) + current = current[index] + else: + return None, ( + f"select path {path!r} failed at {'.'.join(walked + [segment])!r}: " + f"{type(current).__name__} is not indexable" + ) + walked.append(segment) + return current, "" + + +def decode_json(body: str) -> tuple[Any, str]: + """Decode a JSON body, returning ``(value, error)``.""" + try: + return json.loads(body), "" + except json.JSONDecodeError as exc: + preview = " ".join(body[:200].split()) + return None, ( + f"Response was declared as JSON but did not parse: {exc.msg} " + f"at line {exc.lineno} column {exc.colno}. Body starts with: {preview!r}" + ) + + +__all__ = [ + "KIND_BINARY", + "KIND_HTML", + "KIND_JSON", + "KIND_TEXT", + "ContentExtractor", + "ExtractedContent", + "StdlibHtmlExtractor", + "TrafilaturaExtractor", + "decode_json", + "extract_html", + "html_extractors", + "kind_for_content_type", + "select_path", +] diff --git a/src/leapflow/tools/web_fetch.py b/src/leapflow/tools/web_fetch.py new file mode 100644 index 0000000..6a9d902 --- /dev/null +++ b/src/leapflow/tools/web_fetch.py @@ -0,0 +1,788 @@ +"""web_fetch — first-class read-only HTTP access for the agent loop. + +Without this tool the only way to read a URL is ``shell_run`` with a hand-written +``curl | python3 -c`` pipeline. That path is classified ``external_side_effect``, +so a plain GET picks up side-effect batch stopping, session-scoped deduplication, +and "this may already have taken effect" retry guidance — none of which describe +reading a web page. Worse, HTTP failures surface as whatever the improvised +pipeline printed (typically a Python traceback) instead of a status code. + +``web_fetch`` is therefore declared ``read_only``: retries are safe, the batch +gate does not fire, and failures come back as structured status information. + +Two transports exist and both are load-bearing. httpx is the default (async +native, structured status/headers). curl is the fallback for environments where +the Python TLS stack cannot complete a handshake that the system curl can — most +commonly corporate TLS interception, where certifi lacks the intercepting CA but +the system trust store has it. curl is invoked with an argv list, never through a +shell, so the quoting and injection hazards of the pipeline it replaces are gone. +""" + +from __future__ import annotations + +import asyncio +import logging +import re +import time +import uuid +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, Mapping, Protocol, runtime_checkable +from urllib.parse import urljoin, urlsplit + +from leapflow.security.network import NetworkTarget, UrlRejected, classify_url +from leapflow.tools import web_cache +from leapflow.tools.web_extract import ( + KIND_BINARY, + KIND_HTML, + KIND_JSON, + decode_json, + extract_html, + kind_for_content_type, + select_path, +) + +logger = logging.getLogger(__name__) + +# A browser-style default: many CDNs answer 429/403 to curl's or a library's own +# user agent, which turns an ordinary read into a failure the model then has to +# debug. Overridable through `web.user_agent` for deployments that need to +# identify themselves honestly. +DEFAULT_USER_AGENT = ( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" +) +_DEFAULT_ACCEPT = ( + "text/html,application/xhtml+xml,application/xml;q=0.9," + "application/json;q=0.9,text/plain;q=0.8,*/*;q=0.5" +) +_RETRY_STATUSES = frozenset({408, 425, 429, 500, 502, 503, 504}) +# Statuses that mean "this client was refused" rather than "this resource does not +# exist". Observed against a real CDN edge: the same URL with the same user agent +# answers 429 intermittently, and which transport is refused varies between runs, +# so the refusal is not attributable to the user agent alone. Retrying the same +# transport is one lever; handing the request to the next transport is another, +# and it costs a single extra attempt. Anything outside this set is reported as-is. +_FAILOVER_STATUSES = frozenset({403, 429}) +_MAX_RETRY_SLEEP_S = 8.0 +# Ceilings on what a single tool call may request, independent of the configured +# defaults: the model can raise timeout/max_bytes per call, and these bound how +# far. Config sets the normal value; these stop one call from monopolizing a turn. +_MAX_TIMEOUT_S = 120.0 +_MAX_BYTES_CEILING = 20_000_000 + +# Module-level gate, installed by the CLI/daemon exactly like the shell and +# config gates. ``requires_approval`` in the tool's x_leapflow block only informs +# capability disclosure and does not gate execution, so the handler must consult +# this explicitly. +_approval_gate: Any = None + + +def set_web_approval_gate(gate: Any) -> None: + """Install the approval gate consulted before a sensitive fetch.""" + global _approval_gate + _approval_gate = gate + + +def get_web_approval_gate() -> Any: + """Return the installed web approval gate (or ``None``).""" + return _approval_gate + + +@dataclass(frozen=True) +class FetchRequest: + """One outbound single-hop read, fully resolved from params plus settings.""" + + url: str + timeout_s: float + max_bytes: int + user_agent: str + + +@dataclass(frozen=True) +class FetchOutcome: + """A raw single-hop transport result, before content extraction. + + ``location`` carries the redirect target when the status is 3xx. Transports + deliberately do not follow redirects themselves: each hop is a new egress + target that must be classified and gated again, which only the orchestrator + can do. + """ + + status: int + final_url: str + content_type: str + body: bytes + truncated: bool + transport: str + elapsed_ms: int + location: str = "" + + +class TransportUnavailable(RuntimeError): + """Raised when a transport cannot run in this environment.""" + + +class TransportFailure(RuntimeError): + """Raised when a request failed before any HTTP status was received.""" + + def __init__(self, error_type: str, message: str, *, retryable: bool) -> None: + super().__init__(message) + self.error_type = error_type + self.retryable = retryable + + +@runtime_checkable +class WebTransport(Protocol): + """Performs one bounded HTTP read.""" + + name: str + + def available(self) -> bool: + """Whether this transport can run in the current environment.""" + ... + + async def fetch(self, request: FetchRequest) -> FetchOutcome: + """Perform the read, raising ``TransportFailure`` on connection errors.""" + ... + + +def _headers(request: FetchRequest) -> dict[str, str]: + # No Accept-Language: pinning one would silently bias which localized version + # of a page the agent reads, and the right language is the user's, not ours. + return { + "User-Agent": request.user_agent, + "Accept": _DEFAULT_ACCEPT, + } + + +class HttpxTransport: + """Default transport: async-native with structured status and headers.""" + + name = "httpx" + + def available(self) -> bool: + try: + import httpx # noqa: F401 + except ImportError: + return False + return True + + async def fetch(self, request: FetchRequest) -> FetchOutcome: + try: + import httpx + except ImportError as exc: # pragma: no cover - guarded by available() + raise TransportUnavailable("httpx is not installed") from exc + + started = time.monotonic() + try: + async with httpx.AsyncClient( + # Never auto-follow: a redirect can point at loopback or cloud + # metadata, and a hop the egress gate never saw is a hole in it. + follow_redirects=False, + timeout=request.timeout_s, + ) as client: + async with client.stream("GET", request.url, headers=_headers(request)) as response: + chunks: list[bytes] = [] + size = 0 + truncated = False + # Streamed rather than read whole: the byte cap must hold even + # when the server sends no Content-Length. + async for chunk in response.aiter_bytes(): + remaining = request.max_bytes - size + if remaining <= 0: + truncated = True + break + if len(chunk) > remaining: + chunks.append(chunk[:remaining]) + truncated = True + break + chunks.append(chunk) + size += len(chunk) + return FetchOutcome( + status=response.status_code, + final_url=str(response.url), + content_type=response.headers.get("content-type", ""), + body=b"".join(chunks), + truncated=truncated, + transport=self.name, + elapsed_ms=int((time.monotonic() - started) * 1000), + location=response.headers.get("location", ""), + ) + except httpx.TimeoutException as exc: + raise TransportFailure( + "timeout", f"Request timed out after {request.timeout_s}s", retryable=True + ) from exc + except httpx.ConnectError as exc: + raise TransportFailure("connect_error", str(exc), retryable=True) from exc + except httpx.HTTPError as exc: + raise TransportFailure("transport_error", f"{type(exc).__name__}: {exc}", retryable=True) from exc + + +class CurlTransport: + """Fallback transport using the system curl through an argv list. + + Earns its place two ways. The Python and system TLS stacks fail differently — + behind TLS-intercepting proxies curl trusts the injected CA through the OS + store while httpx (certifi) rejects it — and a CDN that refuses one client + with 429 frequently serves the other, so the chain has a second thing to try. + Invoked with ``create_subprocess_exec`` and an argv list, so no shell parses + the URL. + """ + + name = "curl" + # curl exit codes worth naming; anything else becomes a generic failure. + _EXIT_ERRORS = { + 6: ("dns_error", False), + 7: ("connect_error", True), + 28: ("timeout", True), + 35: ("tls_error", False), + 47: ("too_many_redirects", False), + 60: ("tls_error", False), + 63: ("too_large", False), + } + + def available(self) -> bool: + import shutil + + return shutil.which("curl") is not None + + async def fetch(self, request: FetchRequest) -> FetchOutcome: + if not self.available(): + raise TransportUnavailable("curl is not installed") + # The sentinel separates the body from curl's --write-out metadata on one + # stream, avoiding a temp file for the body. Split from the right so a + # body that happens to contain the token cannot shift the metadata. + sentinel = f"--leapflow-{uuid.uuid4().hex}--" + argv = [ + "curl", "--silent", "--show-error", "--compressed", + # No --location: redirects are followed by the orchestrator so each + # hop is re-classified against the egress gate. %{redirect_url} still + # reports where curl would have gone. + "--max-time", str(int(max(1, request.timeout_s))), + "--max-filesize", str(request.max_bytes), + "--user-agent", request.user_agent, + "--header", f"Accept: {_DEFAULT_ACCEPT}", + "--write-out", + f"{sentinel}%{{http_code}}\t%{{content_type}}\t%{{url_effective}}\t%{{redirect_url}}", + request.url, + ] + started = time.monotonic() + try: + proc = await asyncio.create_subprocess_exec( + *argv, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + stdin=asyncio.subprocess.DEVNULL, + ) + stdout, stderr = await asyncio.wait_for( + proc.communicate(), timeout=request.timeout_s + 5 + ) + except asyncio.TimeoutError as exc: + raise TransportFailure( + "timeout", f"curl exceeded {request.timeout_s}s", retryable=True + ) from exc + except OSError as exc: + raise TransportUnavailable(f"curl could not be started: {exc}") from exc + + elapsed_ms = int((time.monotonic() - started) * 1000) + if proc.returncode != 0: + error_type, retryable = self._EXIT_ERRORS.get( + proc.returncode or 0, ("transport_error", True) + ) + detail = stderr.decode(errors="replace").strip() or f"curl exit {proc.returncode}" + raise TransportFailure(error_type, detail, retryable=retryable) + + marker = stdout.rfind(sentinel.encode()) + if marker < 0: + raise TransportFailure( + "transport_error", "curl produced no status metadata", retryable=True + ) + body = stdout[:marker] + meta = stdout[marker + len(sentinel.encode()):].decode(errors="replace").split("\t") + status = int(meta[0]) if meta and meta[0].strip().isdigit() else 0 + content_type = meta[1] if len(meta) > 1 else "" + final_url = meta[2].strip() if len(meta) > 2 else request.url + location = meta[3].strip() if len(meta) > 3 else "" + truncated = len(body) > request.max_bytes + return FetchOutcome( + status=status, + final_url=final_url or request.url, + content_type=content_type, + body=body[: request.max_bytes], + truncated=truncated, + transport=self.name, + elapsed_ms=elapsed_ms, + location=location, + ) + + +def transports_for(preference: str) -> tuple[WebTransport, ...]: + """Return the transport chain honoring the configured preference.""" + httpx_transport = HttpxTransport() + curl_transport = CurlTransport() + if preference == "httpx": + return (httpx_transport,) + if preference == "curl": + return (curl_transport,) + return tuple(t for t in (httpx_transport, curl_transport) if t.available()) + + +def _auditable_url(target: NetworkTarget) -> str: + """Return the URL with its query, fragment, and userinfo removed. + + The approval prompt and the audit log both persist an action's detail, and a + query string routinely carries API keys or signed tokens. Origin plus path is + enough for a human to judge the request and for the audit trail to be useful, + so the secret-bearing parts never get written down. + """ + path = urlsplit(target.url).path or "/" + return f"{target.origin}{path}" + + +async def _approve_fetch(target: NetworkTarget) -> str: + """Return a denial reason, or ``""`` when the fetch may proceed. + + Fails closed when no gate is installed: an internal target reachable without + review would let the model read the daemon socket's HTTP neighbors, the local + dashboard, or cloud instance metadata. The grant is keyed on the origin, so + approving one page trusts the host for the session rather than only that URL. + """ + gate = _approval_gate + if gate is None: + return ( + f"{target.origin} resolves to a {target.category} address and needs approval, " + "but no approval gate is available in this session. Ask the user to fetch it " + "manually or run this in an interactive session." + ) + try: + from leapflow.security.actions import ActionDescriptor + + action = ActionDescriptor.network_fetch( + _auditable_url(target), + origin=target.origin, + metadata={"tool": "web_fetch", **target.to_metadata()}, + ) + result = await gate.evaluate(action) + except Exception: # noqa: BLE001 - a broken gate must not become an open door + logger.warning("web_fetch: approval evaluation failed; denying", exc_info=True) + return "Fetch denied: the approval gate could not be consulted." + + if getattr(result, "approved", False): + return "" + return str( + getattr(result, "denial_message", "") + or getattr(result, "reason", "") + or f"Fetch denied by approval gate: {target.origin}" + ) + + +def _decode_text(body: bytes, content_type: str) -> str: + """Decode a body to text using the declared charset when present.""" + match = re.search(r"charset=([\w\-]+)", content_type or "", re.IGNORECASE) + encodings = [match.group(1)] if match else [] + encodings += ["utf-8", "latin-1"] + for encoding in encodings: + try: + return body.decode(encoding) + except (UnicodeDecodeError, LookupError): + continue + return body.decode("utf-8", errors="replace") + + +def _retry_delay(attempt: int) -> float: + return min(_MAX_RETRY_SLEEP_S, 0.5 * (2 ** attempt)) + + +async def _attempt( + transport: WebTransport, request: FetchRequest, max_retries: int +) -> tuple[FetchOutcome | None, TransportFailure | None]: + """Run one transport with bounded retries. + + Returns the outcome, or the failure that ended the attempts. Retrying at all + is only safe because the tool is read-only; an unrecoverable failure (bad TLS, + too many redirects) is not retried. + """ + last_failure: TransportFailure | None = None + for attempt in range(max_retries + 1): + try: + outcome = await transport.fetch(request) + except TransportUnavailable: + return None, last_failure + except TransportFailure as failure: + last_failure = failure + if not failure.retryable or attempt >= max_retries: + return None, failure + await asyncio.sleep(_retry_delay(attempt)) + continue + if outcome.status in _RETRY_STATUSES and attempt < max_retries: + await asyncio.sleep(_retry_delay(attempt)) + continue + return outcome, None + return None, last_failure + + +async def _run_transports( + request: FetchRequest, preference: str, max_retries: int +) -> FetchOutcome: + """Fetch through the transport chain, failing over when a client is refused. + + Two escalation paths, deliberately distinct: a *failure* (no HTTP response) + moves to the next transport, and a *client-rejection status* does too, since a + refusal that is intermittent per client often clears on the other stack. + Everything else is returned as-is — an ordinary 404 is the answer, not + something to re-ask a second transport. + """ + chain = [item for item in transports_for(preference) if item.available()] + if not chain: + raise TransportUnavailable( + "No usable HTTP transport: install httpx (`pip install httpx`) or curl." + ) + last_failure: TransportFailure | None = None + last_outcome: FetchOutcome | None = None + for index, transport in enumerate(chain): + outcome, failure = await _attempt(transport, request, max_retries) + if failure is not None: + last_failure = failure + continue + if outcome is None: + continue + last_outcome = outcome + if outcome.status in _FAILOVER_STATUSES and index + 1 < len(chain): + logger.info( + "web_fetch: %s refused with %d; trying %s", + transport.name, + outcome.status, + chain[index + 1].name, + ) + continue + return outcome + if last_outcome is not None: + return last_outcome + if last_failure is not None: + raise last_failure + raise TransportUnavailable("No usable HTTP transport for this request.") + + +def _redact(text: str) -> str: + try: + from leapflow.security.redact import redact_sensitive_text + + return redact_sensitive_text(text, force=True) + except ImportError: # pragma: no cover - redaction ships with the package + return text + + +def _settings() -> Any: + from leapflow.config import get_settings + + return get_settings() + + +async def _gate_target(target: NetworkTarget, settings: Any, *, url: str) -> Dict[str, Any] | None: + """Return a refusal payload when this target may not be reached. + + Applied to every hop, not just the first: a public URL that redirects to + loopback or cloud metadata would otherwise reach it with the gate having only + ever seen the public address. + """ + if not (target.is_internal or target.has_credentials): + return None + mode = str(getattr(settings, "web_private_targets", "approval") or "approval").lower() + if mode == "allow": + return None + if mode == "deny": + return { + "ok": False, + "error": ( + f"{target.origin} resolves to a {target.category} address and this " + "profile is configured to refuse internal targets " + "(`web.private_targets=deny`)." + ), + "error_type": "blocked_target", + "retryable": False, + "target_category": target.category, + "url": url, + "blocked_url": target.url, + } + denial = await _approve_fetch(target) + if not denial: + return None + return { + "ok": False, + "error": denial, + "error_type": "blocked_target", + "retryable": False, + "requires_approval": True, + "target_category": target.category, + "url": url, + "blocked_url": target.url, + } + + +async def web_fetch(params: Dict[str, Any]) -> Dict[str, Any]: + """Read a URL and return extracted, context-sized content.""" + url = str(params.get("url") or "").strip() + if not url: + return {"ok": False, "error": "web_fetch requires 'url'.", "retryable": True} + if "://" not in url: + url = f"https://{url}" + + settings = _settings() + try: + timeout_s = min(float(params.get("timeout") or settings.web_timeout_s), _MAX_TIMEOUT_S) + except (TypeError, ValueError): + timeout_s = float(settings.web_timeout_s) + try: + max_bytes = min(int(params.get("max_bytes") or settings.web_max_bytes), _MAX_BYTES_CEILING) + except (TypeError, ValueError): + max_bytes = int(settings.web_max_bytes) + max_redirects = max(0, int(getattr(settings, "web_max_redirects", 5))) + + # Redirects are followed here rather than inside a transport so that every hop + # is classified and gated. Following them in the HTTP client would let a public + # URL bounce the request to an internal address the gate never inspected. + current = url + visited: list[str] = [] + target: NetworkTarget | None = None + outcome: FetchOutcome | None = None + for hop in range(max_redirects + 1): + try: + target = await classify_url(current) + except UrlRejected as rejected: + return { + "ok": False, + "error": rejected.detail, + "error_type": rejected.reason, + "retryable": rejected.reason == "dns_error", + "url": url, + "blocked_url": current if current != url else "", + } + + refusal = await _gate_target(target, settings, url=url) + if refusal is not None: + if hop > 0: + refusal["error"] = ( + f"Redirected to {target.origin}, which was refused: {refusal['error']}" + ) + refusal["redirect_chain"] = [*visited, current] + return refusal + + if hop == 0: + # Cache lookup sits after the first gate check, never before: a cached + # body must not become a way to read a target approval would refuse. + cached = web_cache.read(url, settings) + if cached is not None: + replay = FetchOutcome( + status=cached.status, + final_url=cached.final_url, + content_type=cached.content_type, + body=cached.body, + truncated=cached.truncated, + transport=cached.transport, + elapsed_ms=cached.elapsed_ms, + ) + return _build_result(params, target, replay, settings, from_cache=True) + + request = FetchRequest( + url=current, + timeout_s=timeout_s, + max_bytes=max_bytes, + user_agent=str(settings.web_user_agent or DEFAULT_USER_AGENT), + ) + try: + outcome = await _run_transports( + request, + str(settings.web_transport or "auto").lower(), + max(0, int(settings.web_max_retries)), + ) + except TransportUnavailable as exc: + return { + "ok": False, + "error": str(exc), + "error_type": "transport_unavailable", + "retryable": False, + "url": url, + } + except TransportFailure as failure: + return { + "ok": False, + "error": str(failure), + "error_type": failure.error_type, + "retryable": failure.retryable, + "url": url, + "origin": target.origin, + } + + if not (300 <= outcome.status < 400 and outcome.location): + break + next_url = urljoin(current, outcome.location) + if next_url in visited or next_url == current: + return { + "ok": False, + "error": f"Redirect loop while fetching {url}.", + "error_type": "redirect_loop", + "retryable": False, + "url": url, + "redirect_chain": [*visited, current], + } + visited.append(current) + current = next_url + else: + return { + "ok": False, + "error": f"Exceeded {max_redirects} redirects while fetching {url}.", + "error_type": "too_many_redirects", + "retryable": False, + "url": url, + "redirect_chain": [*visited, current], + } + + result = _build_result(params, target, outcome, settings) + if visited: + result["redirect_chain"] = [*visited, current] + return result + + +def _build_result( + params: Mapping[str, Any], + target: NetworkTarget, + outcome: FetchOutcome, + settings: Any, + *, + from_cache: bool = False, +) -> Dict[str, Any]: + """Shape a transport outcome into the tool's structured result.""" + result: Dict[str, Any] = { + "ok": 200 <= outcome.status < 300, + "status": outcome.status, + "url": target.url, + "final_url": outcome.final_url, + "content_type": outcome.content_type, + "bytes": len(outcome.body), + "truncated": outcome.truncated, + "transport": outcome.transport, + "elapsed_ms": outcome.elapsed_ms, + "origin": target.origin, + } + if from_cache: + result["from_cache"] = True + kind = kind_for_content_type(outcome.content_type) + result["kind"] = kind + + if not result["ok"]: + # An HTTP error is the answer, not a crash: name the status and let the + # model decide, with a body excerpt because error pages explain why. + excerpt = "" + if kind != KIND_BINARY: + excerpt = _redact(_decode_text(outcome.body, outcome.content_type))[:600] + result["error"] = f"HTTP {outcome.status} from {target.origin}" + result["error_type"] = "http_error" + result["retryable"] = outcome.status in _RETRY_STATUSES + if excerpt.strip(): + result["body_excerpt"] = excerpt + return result + + stored: Path | None = None + if not from_cache: + stored = web_cache.write( + target.url, + outcome.body, + status=outcome.status, + final_url=outcome.final_url, + content_type=outcome.content_type, + truncated=outcome.truncated, + transport=outcome.transport, + elapsed_ms=outcome.elapsed_ms, + origin=target.origin, + # A URL with a query string or credentials can itself be the secret. + sensitive=bool(target.has_credentials or urlsplit(target.url).query), + settings=settings, + ) + + if kind == KIND_BINARY: + # Binary never enters the transcript. When it was cached, hand back the + # path so file-oriented tools can take over instead of a dead end. + result["text"] = "" + if stored is None and from_cache: + stored = web_cache.cached_path(target.url, settings) + if stored is not None: + result["cache_path"] = str(stored) + result["note"] = ( + "Binary content is not returned inline; it is saved at cache_path for " + "a file-oriented tool to handle." + ) + else: + result["note"] = ( + "Binary content is not returned inline. Re-request with a text or JSON " + "endpoint, or ask the user how this file should be handled." + ) + return result + + body_text = _decode_text(outcome.body, outcome.content_type) + + if kind == KIND_JSON: + data, error = decode_json(body_text) + if error: + # Declared JSON that does not parse: report it as a content problem + # with the real status attached, never as an opaque failure. + result["ok"] = False + result["error"] = error + result["error_type"] = "invalid_json" + result["retryable"] = False + result["body_excerpt"] = _redact(body_text)[:600] + return result + select = str(params.get("select") or "").strip() + if select: + value, select_error = select_path(data, select) + if select_error: + result["ok"] = False + result["error"] = select_error + result["error_type"] = "invalid_selector" + result["retryable"] = True + result["available_top_level_keys"] = ( + sorted(data)[:20] if isinstance(data, dict) else [] + ) + return result + result["select"] = select + result["data"] = value + else: + result["data"] = data + return result + + if kind == KIND_HTML: + extracted = extract_html( + body_text, + url=outcome.final_url or target.url, + prefer=str(getattr(settings, "web_extractor", "auto") or "auto").lower(), + ) + result["title"] = extracted.title + result["text"] = _redact(extracted.text) + result["extractor"] = extracted.extractor + if extracted.links: + result["links"] = [{"text": text, "url": link} for text, link in extracted.links] + if not extracted.text.strip(): + result["note"] = ( + "No readable text could be extracted; the page may render its content " + "with JavaScript." + ) + return result + + result["text"] = _redact(body_text)[: int(getattr(settings, "web_max_bytes", 2_000_000))] + return result + + +__all__ = [ + "DEFAULT_USER_AGENT", + "CurlTransport", + "FetchOutcome", + "FetchRequest", + "HttpxTransport", + "TransportFailure", + "TransportUnavailable", + "WebTransport", + "get_web_approval_gate", + "set_web_approval_gate", + "transports_for", + "web_fetch", +] diff --git a/tests/test_runtime_metadata_and_wrapping.py b/tests/test_runtime_metadata_and_wrapping.py new file mode 100644 index 0000000..74bfcb4 --- /dev/null +++ b/tests/test_runtime_metadata_and_wrapping.py @@ -0,0 +1,223 @@ +"""Guards for runtime metadata reporting and long-output rendering. + +Two failures that kept coming back, both because a value was read from the wrong +place rather than because the display was wrong: + +- The status bar sat at ``0/<limit>`` all session. Conversation state lives on + per-session engines from ``SessionRegistry``; ``ctx.engine`` is only the + template they are built from and never accumulates turns, so anything reading + it reports zero context. The same root cause produced an empty LeapBoard + earlier, which is why this file guards the *entry point* rather than one call + site: new metadata code must go through ``_active_engine()``. +- Long answers lost their tail. ``soft_wrap=True`` makes Rich emit one long line + and defer wrapping to whoever owns the screen; under ``patch_stdout`` that + renderer clips at the window edge. +""" + +from __future__ import annotations + +import asyncio +import io +from types import SimpleNamespace + +import pytest + +from leapflow.daemon.service import RuntimeLeapService +from leapflow.daemon.session_registry import SessionRegistry +from leapflow.engine import StreamEvent + +_CONTEXT_LENGTH = 1_000_000 +_USED_TOKENS = 48_000 + + +class _BaseEngine: + """The template engine: it is never handed a conversation.""" + + _current_session_id = "" + context_token_count = 0 + turn_count = 0 + + +class _SessionEngine: + """A per-session engine, i.e. the one that actually accrues context.""" + + def __init__(self, session_id: str, used: int) -> None: + self._current_session_id = session_id + self.context_token_count = used + self.turn_count = 3 + + +def _service_with_session(used: int = _USED_TOKENS) -> tuple[RuntimeLeapService, _BaseEngine]: + service = RuntimeLeapService(SimpleNamespace()) + base = _BaseEngine() + registry = SessionRegistry( + base_engine=base, + build_engine=lambda b, sid, wm, root: _SessionEngine(sid, used), + build_working_memory=lambda: None, + ) + asyncio.run(registry.acquire("s1", workspace_root="/tmp")) + service._ctx = SimpleNamespace( + engine=base, + settings=SimpleNamespace(llm_context_length=_CONTEXT_LENGTH, llm_model="qwen3.8-max"), + ) + service._session_coordinator._session_registry = registry + return service, base + + +# ── Runtime metadata must describe the session, not the template ───────── + + +def test_active_engine_resolves_the_session_engine() -> None: + service, base = _service_with_session() + + engine = service._active_engine() + + assert engine is not base, "the base engine carries no conversation" + assert engine.context_token_count == _USED_TOKENS + + +def test_stream_metadata_reports_real_context_usage() -> None: + """The status bar reads this; zero here is what showed as ``0/1M``.""" + service, _ = _service_with_session() + + chunk = service._chunk_from_event( + StreamEvent(type="content", content="hi"), request_id="r1", + ) + + assert chunk.metadata["context_used"] == _USED_TOKENS + assert chunk.metadata["llm_context_length"] == _CONTEXT_LENGTH + assert chunk.metadata["session_id"] == "s1" + + +def test_stream_metadata_prefers_the_engine_that_produced_the_event() -> None: + """An explicit engine wins, so a concurrent session cannot be misreported.""" + service, _ = _service_with_session() + other = _SessionEngine("s2", 12_345) + + chunk = service._chunk_from_event( + StreamEvent(type="content", content="hi"), request_id="r1", engine=other, + ) + + assert chunk.metadata["context_used"] == 12_345 + assert chunk.metadata["session_id"] == "s2" + + +def test_status_reads_context_from_the_session_engine() -> None: + """``status()`` must report on the session engine like the stream does. + + Asserted on the engine it resolves plus the metadata builder, rather than by + calling ``status()``: that needs a full Settings (layout, profile, paths) and + the value under test here is only which engine gets measured. + """ + from leapflow.daemon._service_helpers import engine_context_metadata + + service, base = _service_with_session() + + engine = service._active_engine() + metadata = engine_context_metadata(engine, service._ctx.settings) + + assert engine is not base + assert metadata["context_used"] == _USED_TOKENS + assert metadata["llm_context_length"] == _CONTEXT_LENGTH + + +def test_base_engine_would_have_reported_zero() -> None: + """Pins why this matters: the old path could only ever report 0.""" + from leapflow.daemon._service_helpers import engine_context_metadata + + service, base = _service_with_session() + + stale = engine_context_metadata(base, service._ctx.settings) + + assert stale["context_used"] == 0 + + +def test_active_engine_tolerates_a_missing_context() -> None: + service = RuntimeLeapService(SimpleNamespace()) + service._ctx = None + + assert service._active_engine() is None + + +# ── Link protection: no metadata path may read ctx.engine directly ─────── + + +def test_metadata_paths_do_not_read_ctx_engine_directly() -> None: + """Regression guard for the class of bug, not one instance of it. + + Each recurrence so far was a fresh ``getattr(ctx, "engine")`` next to a + metadata assembly. ``_active_engine()`` is the single entry point; anything + bypassing it silently reports a conversation-free engine, which is invisible + in review and only shows up as a zeroed status bar or an empty board. + """ + import inspect + + from leapflow.daemon import service as service_module + + source = inspect.getsource(service_module) + offenders: list[str] = [] + lines = source.splitlines() + for index, line in enumerate(lines): + if "engine_context_metadata(" not in line: + continue + # Look at the call and the few lines above it for a direct base-engine read. + window = "\n".join(lines[max(0, index - 6):index + 1]) + if 'getattr(ctx, "engine"' in window or "getattr(ctx, 'engine'" in window: + offenders.append(f"line {index + 1}: {line.strip()}") + + assert offenders == [], ( + "runtime metadata must come from _active_engine(); these read the base " + "engine directly:\n " + "\n ".join(offenders) + ) + + +# ── Long output must wrap, not clip ────────────────────────────────────── + + +def _console(): + from leapflow.cli.tui_app.console import LeapConsole + from leapflow.cli.tui_app.theme import _LIGHT, resolve_theme + + console = LeapConsole(resolve_theme(_LIGHT, terminal_bg="#FFFFFF")) + console._console.file = io.StringIO() + console._console.width = 100 + return console + + +def test_console_wraps_rather_than_emitting_one_long_line() -> None: + """Rich must do the wrapping; patch_stdout's renderer clips instead.""" + console = _console() + text = "经济增长理论与索洛模型的边际产出递减规律，" * 12 + + console.print(text) + lines = [line for line in console._console.file.getvalue().split("\n") if line.strip()] + + assert len(lines) > 1, "a long paragraph must be wrapped into several lines" + # Wide (CJK) glyphs count double, so bound on display width, not characters. + from rich.cells import cell_len + + assert max(cell_len(line) for line in lines) <= 100 + + +@pytest.mark.parametrize("renderer", ["print", "system", "markdown"]) +def test_long_text_wraps_across_output_surfaces(renderer: str) -> None: + """Wrapping is a console-level property, so every surface inherits it.""" + console = _console() + text = "capital accumulation and productivity growth compound over decades " * 6 + + getattr(console, renderer)(text) + lines = [line for line in console._console.file.getvalue().split("\n") if line.strip()] + + assert len(lines) > 1, f"{renderer}() must wrap long text" + + +def test_console_is_not_configured_to_defer_wrapping() -> None: + """Direct guard on the setting, since the symptom is invisible in tests. + + Clipping only happens inside prompt_toolkit's renderer, so a unit test cannot + observe the truncation itself — it can only pin the configuration that caused + it. + """ + console = _console() + + assert console._console.soft_wrap is False diff --git a/tests/test_tool_call_hardening.py b/tests/test_tool_call_hardening.py index c974835..c03dbe1 100644 --- a/tests/test_tool_call_hardening.py +++ b/tests/test_tool_call_hardening.py @@ -235,3 +235,122 @@ def test_workspace_context_resolves_relative_paths_and_blocks_cross_workspace(tm assert blocked_shell["workspace_root"] == str(workspace.resolve()) finally: reset_tool_context(token) + + +def test_shell_gate_blocks_expanded_and_relative_escapes(tmp_path, monkeypatch) -> None: + """The gate must judge the resolved target, not how the operand is spelled. + + A guard that only inspects literal ``/``/``~`` operands passes ``$HOME/...`` + and ``../../..`` straight through, so the same file is reachable by choosing + a different spelling and the workspace boundary stops being a boundary. + """ + from leapflow.tools.execution_context import ( + ToolExecutionContext, + reset_tool_context, + set_tool_context, + ) + from leapflow.tools.shell_tools import shell_run + + workspace = tmp_path / "work" + other = tmp_path / "other" + workspace.mkdir() + (workspace / "sub").mkdir() + other.mkdir() + (workspace / "alpha.txt").write_text("inside", encoding="utf-8") + (other / "secret.txt").write_text("outside", encoding="utf-8") + monkeypatch.setenv("LEAP_TEST_OUTSIDE", str(other)) + + token = set_tool_context( + ToolExecutionContext.from_strings(workspace_root=str(workspace), session_id="sess-gate") + ) + try: + for command in ( + "cat $LEAP_TEST_OUTSIDE/secret.txt", + "cat ${LEAP_TEST_OUTSIDE}/secret.txt", + "cat ../other/secret.txt", + "cat --file=$LEAP_TEST_OUTSIDE/secret.txt", + "cd $LEAP_TEST_OUTSIDE && cat secret.txt", + ): + result = asyncio.run(shell_run({"command": command})) + assert result["ok"] is False, command + assert result["error_type"] == "outside_workspace", command + assert result["resolved_path"].startswith(str(other.resolve())), command + + # Traversal that stays inside must still run: the gate judges the resolved + # target, so `sub/../alpha.txt` is an ordinary in-workspace read. + inside = asyncio.run(shell_run({"command": "cat sub/../alpha.txt"})) + assert inside["ok"] is True + assert inside["stdout"].strip() == "inside" + finally: + reset_tool_context(token) + + +def test_shell_gate_allows_search_list_variables(tmp_path, monkeypatch) -> None: + """Expanding a variable must not turn ordinary commands into refusals. + + ``$PATH`` expands to an ``os.pathsep``-joined list that begins with ``/`` but + names no file. Treating that as a path operand blocked ``echo $PATH`` and + ``PATH=$PATH:./bin npm test``, which have nothing to do with the workspace + boundary. + """ + from leapflow.tools.execution_context import ( + ToolExecutionContext, + reset_tool_context, + set_tool_context, + ) + from leapflow.tools.shell_tools import _command_workspace_escape + + workspace = tmp_path / "work" + workspace.mkdir() + monkeypatch.setenv("PATH", "/usr/local/bin:/usr/bin:/bin") + monkeypatch.setenv("LEAP_TEST_HOME", str(tmp_path / "outside")) + + token = set_tool_context( + ToolExecutionContext.from_strings(workspace_root=str(workspace), session_id="sess-list") + ) + try: + for command in ( + "echo $PATH", + "export PATH=$PATH:/usr/local/bin && make", + "PATH=$PATH:./node_modules/.bin npm test", + ): + assert _command_workspace_escape(command, cwd=workspace) is None, command + + # A single-path variable must still be gated. + blocked = _command_workspace_escape("cat $LEAP_TEST_HOME/secret", cwd=workspace) + assert blocked is not None + assert blocked["error_type"] == "outside_workspace" + finally: + reset_tool_context(token) + + +def test_shell_gate_redirects_leapflow_config_targets(tmp_path) -> None: + """Refusing a config path must name the capability that serves the goal. + + Without the redirect the model only learns "not here" and moves the same + probe to another spelling, which is the loop the config tools exist to end. + """ + from leapflow.tools.execution_context import ( + ToolExecutionContext, + reset_tool_context, + set_tool_context, + ) + from leapflow.tools.shell_tools import _command_workspace_escape + from leapflow.config import get_settings + + workspace = tmp_path / "work" + workspace.mkdir() + config_path = get_settings().layout.global_config_dir / "user.yaml" + + token = set_tool_context( + ToolExecutionContext.from_strings(workspace_root=str(workspace), session_id="sess-hint") + ) + try: + error = _command_workspace_escape(f"cat {config_path}", cwd=workspace) + finally: + reset_tool_context(token) + + assert error is not None + assert error["error_type"] == "outside_workspace" + assert "config_get" in error["error"] + assert "cannot be lifted by approval" in error["error"] diff --git a/tests/test_tui_tool_audit.py b/tests/test_tui_tool_audit.py new file mode 100644 index 0000000..f9fc9c9 --- /dev/null +++ b/tests/test_tui_tool_audit.py @@ -0,0 +1,203 @@ +"""Tests for TUI tool-audit rendering fidelity. + +These pin the three defects observed in a real session: a parallel batch printed +one line instead of two, that line showed a sibling call's arguments, and the +failure text shown was the head of a traceback rather than the exception it ended +with. +""" + +from __future__ import annotations + +import time + +from leapflow.cli.tui_app.stream import StreamRenderer +from leapflow.engine.engine import _tool_args_metadata, _tool_result_metadata + +CMD = ( + 'curl -s "https://query1.finance.yahoo.com/v8/finance/chart/BABA" 2>/dev/null | python3 -c "\n' + "import json, sys\ndata = json.load(sys.stdin)\n\"" +) +TRACEBACK = ( + "Traceback (most recent call last):\n" + ' File "<string>", line 3, in <module>\n' + " data = json.load(sys.stdin)\n" + ' File ".../json/__init__.py", line 298, in load\n' + " return loads(fp.read(),\n" + ' File ".../json/decoder.py", line 363, in raw_decode\n' + ' raise JSONDecodeError("Expecting value", s, err.value) from None\n' + "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n" +) +SHELL_FAILURE = { + "ok": False, + "returncode": 1, + "stdout": "", + "stderr": TRACEBACK, + "error": TRACEBACK[-800:], + "execution_policy": "external_side_effect", +} + + +class _CapturingConsole: + """Collects printed lines as plain text and ignores everything else.""" + + def __init__(self) -> None: + self.lines: list[str] = [] + + def print(self, renderable="") -> None: + self.lines.append(getattr(renderable, "plain", str(renderable))) + + def __getattr__(self, _name): + return lambda *args, **kwargs: None + + +def _render_batch() -> tuple[_CapturingConsole, StreamRenderer]: + """Replay the recovered batch: shell_run(command=...) then time_get({}).""" + console = _CapturingConsole() + renderer = StreamRenderer(console) + renderer.start() + renderer.tool_started( + "shell_run", + _tool_args_metadata("shell_run", {"command": CMD, "timeout": 15}, tool_call_id="call_shell"), + ) + renderer.tool_started("time_get", _tool_args_metadata("time_get", {}, tool_call_id="call_time")) + renderer.tool_finished( + "shell_run", + metadata=_tool_result_metadata( + "shell_run", {"command": CMD}, SHELL_FAILURE, tool_call_id="call_shell" + ), + ) + renderer.tool_finished( + "time_get", + metadata=_tool_result_metadata( + "time_get", {}, {"ok": True, "human": "2026-08-05 10:41:07"}, tool_call_id="call_time" + ), + ) + return console, renderer + + +def test_every_tool_in_a_batch_prints_its_own_line() -> None: + """A sibling's result must not disappear. + + With a single shared slot the second completion found the timer already + cleared and printed nothing, so a batch of two tools reported one. + """ + console, renderer = _render_batch() + + assert len(console.lines) == 2 + assert console.lines[0].strip().startswith("✗ shell_run") + assert "time_get" in console.lines[1] + assert renderer.tool_count == 2 + + +def test_failed_line_shows_its_own_command_not_a_siblings_arguments() -> None: + """The audit line must describe the call it belongs to. + + Previously the detail came from whichever start ran last, so a failing + shell_run was labelled with `time_get`'s empty argument dict. + """ + console, _ = _render_batch() + shell_line = console.lines[0] + + assert "$ curl -s" in shell_line + assert "{}" not in shell_line + + +def test_failed_line_shows_the_exception_and_exit_code() -> None: + """The cause is the last line of a traceback, so the preview must reach it.""" + console, _ = _render_batch() + shell_line = console.lines[0] + + assert "JSONDecodeError" in shell_line + assert "exit=1" in shell_line + + +def test_single_call_path_without_ids_still_renders() -> None: + """Callers that emit no tool_call_id must keep working.""" + console = _CapturingConsole() + renderer = StreamRenderer(console) + renderer.start() + renderer.tool_started("file_read") + renderer.tool_finished("file_read", metadata={"ok": True, "normalized_tool_name": "file_read"}) + + assert len(console.lines) == 1 + assert "file_read" in console.lines[0] + assert renderer.tool_count == 1 + + +def test_repeated_same_name_calls_pair_oldest_first() -> None: + """Two calls to one tool without ids must produce two lines, not one.""" + console = _CapturingConsole() + renderer = StreamRenderer(console) + renderer.start() + renderer.tool_started("shell_run") + renderer.tool_started("shell_run") + renderer.tool_finished("shell_run", metadata={"ok": True, "normalized_tool_name": "shell_run"}) + renderer.tool_finished("shell_run", metadata={"ok": True, "normalized_tool_name": "shell_run"}) + + assert len(console.lines) == 2 + assert renderer.tool_count == 2 + + +def test_durations_are_attributed_per_call() -> None: + """Elapsed time must come from the call's own start, not the batch's last.""" + console = _CapturingConsole() + renderer = StreamRenderer(console) + renderer.start() + renderer.tool_started("slow_tool", {"tool_call_id": "a", "normalized_tool_name": "slow_tool"}) + time.sleep(0.05) + renderer.tool_started("fast_tool", {"tool_call_id": "b", "normalized_tool_name": "fast_tool"}) + renderer.tool_finished("fast_tool", metadata={"ok": True, "normalized_tool_name": "fast_tool", "tool_call_id": "b"}) + renderer.tool_finished("slow_tool", metadata={"ok": True, "normalized_tool_name": "slow_tool", "tool_call_id": "a"}) + + names = [name for name, _duration in renderer._tool_history] + durations = dict(renderer._tool_history) + assert names == ["fast_tool", "slow_tool"] + assert durations["slow_tool"] > durations["fast_tool"] + + +def test_spinner_reports_batch_depth() -> None: + """Two running calls must not look like one.""" + renderer = StreamRenderer(_CapturingConsole()) + renderer.start() + first = renderer.tool_started("shell_run", {"tool_call_id": "a", "normalized_tool_name": "shell_run"}) + second = renderer.tool_started("time_get", {"tool_call_id": "b", "normalized_tool_name": "time_get"}) + + assert first.endswith("shell_run") + assert second.endswith("+1") + + +def test_hidden_tools_release_their_slot() -> None: + """A ui_hidden completion prints nothing and leaves no stale in-flight entry.""" + console = _CapturingConsole() + renderer = StreamRenderer(console) + renderer.start() + renderer.tool_started("skipped", {"tool_call_id": "a", "normalized_tool_name": "skipped"}) + renderer.tool_finished( + "skipped", + metadata={"ok": True, "normalized_tool_name": "skipped", "tool_call_id": "a", "ui_hidden": True}, + ) + + assert console.lines == [] + assert renderer.tool_count == 0 + assert renderer._active_tools == {} + + +def test_exit_code_is_read_under_either_key_name() -> None: + """Shell tools emit `returncode`; evidence and UI read `exit_code`.""" + from leapflow.engine.context_control import ToolEvidenceBuilder + from leapflow.engine.tool_execution import exit_code_from + + assert exit_code_from({"returncode": 2}) == 2 + assert exit_code_from({"exit_code": 3}) == 3 + assert exit_code_from({"exit_code": None, "returncode": 4}) == 4 + assert exit_code_from({"ok": True}) is None + # Booleans are not exit codes even though bool is an int subclass. + assert exit_code_from({"returncode": True}) is None + + evidence = ToolEvidenceBuilder(max_content_chars=200).build( + "shell_run", {}, {"ok": True, "returncode": 0, "stdout": "done", "stderr": ""} + ) + assert evidence["exit_code"] == 0 + + metadata = _tool_result_metadata("shell_run", {}, {"ok": False, "returncode": 1, "stderr": "boom"}) + assert metadata["exit_code"] == 1 diff --git a/tests/test_web_fetch.py b/tests/test_web_fetch.py new file mode 100644 index 0000000..15a0391 --- /dev/null +++ b/tests/test_web_fetch.py @@ -0,0 +1,986 @@ +"""Tests for web_fetch: transport contract, egress gating, and extraction. + +Hermetic by construction: no test performs a real request. Transports are +replaced with a fake, and URLs use IP literals so target classification never +touches DNS. +""" + +from __future__ import annotations + +import asyncio +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from leapflow.security.actions import ActionDescriptor, ActionKind +from leapflow.security.network import UrlRejected, classify_url +from leapflow.security.risk import DefaultRiskClassifier, RiskLevel +from leapflow.tools import web_fetch as wf +from leapflow.tools.web_extract import ( + KIND_BINARY, + KIND_HTML, + KIND_JSON, + KIND_TEXT, + StdlibHtmlExtractor, + decode_json, + kind_for_content_type, + select_path, +) + +PUBLIC_URL = "https://93.184.216.34/data" +LOOPBACK_URL = "http://127.0.0.1:8765/state" +METADATA_URL = "http://169.254.169.254/latest/meta-data/iam/security-credentials/" + + +def _settings(**overrides): + base = { + "web_transport": "auto", + "web_timeout_s": 5.0, + "web_max_bytes": 1_000_000, + "web_max_retries": 0, + "web_max_redirects": 3, + "web_user_agent": "", + "web_extractor": "stdlib", + "web_private_targets": "approval", + # Off by default so transport behavior is tested without cache interference; + # the cache tests below opt in with a real layout. + "web_cache_ttl_s": 0.0, + } + base.update(overrides) + return SimpleNamespace(**base) + + +def _settings_with_cache(tmp_path, **overrides): + """Settings backed by a real CacheLayout under ``tmp_path``.""" + from leapflow.layout import build_layout + + profile_layout = build_layout(tmp_path / "home").ensure(profile_id="default") + workspace = tmp_path / "ws" + workspace.mkdir(exist_ok=True) + return _settings( + web_cache_ttl_s=overrides.pop("web_cache_ttl_s", 600.0), + profile="default", + profile_layout=profile_layout, + workspace_root=str(workspace), + **overrides, + ) + + +def _outcome(*, status=200, content_type="application/json", body=b"{}", truncated=False, url=None): + return wf.FetchOutcome( + status=status, + final_url=url or PUBLIC_URL, + content_type=content_type, + body=body, + truncated=truncated, + transport="fake", + elapsed_ms=12, + ) + + +class _FakeTransport: + """Replays a scripted sequence of outcomes or raised failures.""" + + name = "fake" + + def __init__(self, *script): + self._script = list(script) + self.requests: list[wf.FetchRequest] = [] + + def available(self) -> bool: + return True + + async def fetch(self, request): + self.requests.append(request) + item = self._script.pop(0) if len(self._script) > 1 else self._script[0] + if isinstance(item, Exception): + raise item + return item + + +def _install(monkeypatch, transport, **settings_overrides): + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: _settings(**settings_overrides)) + + +def _run(params): + return asyncio.run(wf.web_fetch(params)) + + +# ── target classification (SSRF surface) ───────────────────────────── + +def test_classify_rejects_non_http_schemes() -> None: + """A fetch tool that could open file:// would bypass the workspace boundary.""" + for url in ("file:///etc/passwd", "gopher://x/1", "data:text/plain,hi"): + with pytest.raises(UrlRejected) as excinfo: + asyncio.run(classify_url(url, resolve=False)) + assert excinfo.value.reason == "unsupported_scheme" + + +def test_classify_literal_addresses_by_category() -> None: + cases = { + "http://127.0.0.1/x": "loopback", + "http://10.1.2.3/x": "private", + "http://192.168.1.1/x": "private", + "http://169.254.169.254/x": "metadata", + "http://100.100.100.200/x": "metadata", + "http://169.254.10.10/x": "link_local", + "http://0.0.0.0/x": "unspecified", + "https://93.184.216.34/x": "public", + "http://[::1]/x": "loopback", + "http://[fd00:ec2::254]/x": "metadata", + "https://[2606:4700::1111]/x": "public", + # Shared address space (CGNAT) is not RFC1918 private but is not globally + # routable either; treating it as public would expose a whole class of + # internal endpoints. + "http://100.64.0.1/x": "reserved", + "http://198.18.0.1/x": "private", + } + for url, expected in cases.items(): + target = asyncio.run(classify_url(url, resolve=False)) + assert target.category == expected, url + assert target.is_internal is (expected != "public"), url + + +def test_classify_extracts_origin_and_credentials() -> None: + target = asyncio.run(classify_url("https://user:pw@93.184.216.34:8443/p?q=1", resolve=False)) + assert target.origin == "https://93.184.216.34:8443" + assert target.has_credentials is True + # Default ports stay out of the origin so grants match the common spelling. + plain = asyncio.run(classify_url("https://93.184.216.34/p", resolve=False)) + assert plain.origin == "https://93.184.216.34" + + +# ── risk classification: approval, never a silent block ────────────── + +def test_public_read_is_low_enough_to_skip_prompting() -> None: + """The common path must not prompt, or the tool is worse than shell.""" + target = asyncio.run(classify_url(PUBLIC_URL, resolve=False)) + action = ActionDescriptor.network_fetch( + target.url, origin=target.origin, metadata=target.to_metadata() + ) + risk = DefaultRiskClassifier().assess(action) + assert risk.level == RiskLevel.LOW + # policy asks whenever level is HIGH/MEDIUM or score >= 0.35. + assert risk.score < 0.35 + + +def test_internal_targets_ask_and_are_never_hardline() -> None: + """Internal targets are gated by approval, not refused outright. + + CRITICAL or hardline would make policy deny the action with no prompt, which + is not the configured behavior: the user decides. + """ + for url in (LOOPBACK_URL, METADATA_URL, "http://10.0.0.5/admin"): + target = asyncio.run(classify_url(url, resolve=False)) + action = ActionDescriptor.network_fetch( + target.url, origin=target.origin, metadata=target.to_metadata() + ) + risk = DefaultRiskClassifier().assess(action) + assert risk.level == RiskLevel.HIGH, url + assert risk.hardline is False, url + assert risk.allow_permanent is False, url + assert risk.explanation, url + + +def test_credentialed_url_is_high_risk() -> None: + target = asyncio.run(classify_url("https://u:p@93.184.216.34/x", resolve=False)) + action = ActionDescriptor.network_fetch( + target.url, origin=target.origin, metadata=target.to_metadata() + ) + risk = DefaultRiskClassifier().assess(action) + assert risk.level == RiskLevel.HIGH + assert "url_embedded_credentials" in risk.reasons + + +def test_grant_signature_is_scoped_to_origin() -> None: + """One approval must cover a host, not a single URL. + + Keying the grant on the full URL would re-prompt on every path and query, + which turns progressive trust into an unusable prompt loop. + """ + first = ActionDescriptor.network_fetch( + "https://example.test/a?x=1", origin="https://example.test" + ) + second = ActionDescriptor.network_fetch( + "https://example.test/b/c?y=2", origin="https://example.test" + ) + other = ActionDescriptor.network_fetch( + "https://other.test/a", origin="https://other.test" + ) + assert first.signature() == second.signature() + assert first.signature() != other.signature() + assert first.kind == ActionKind.NETWORK_FETCH.value + + +# ── extraction ─────────────────────────────────────────────────────── + +def test_kind_routing_follows_content_type_header() -> None: + assert kind_for_content_type("application/json; charset=utf-8") == KIND_JSON + assert kind_for_content_type("application/vnd.api+json") == KIND_JSON + assert kind_for_content_type("text/html;charset=gbk") == KIND_HTML + assert kind_for_content_type("text/markdown") == KIND_TEXT + assert kind_for_content_type("application/pdf") == KIND_BINARY + assert kind_for_content_type("image/png") == KIND_BINARY + + +def test_stdlib_extractor_drops_chrome_and_keeps_structure() -> None: + html = """ + <html><head><title>Quarterly Report + + + +

Revenue

+

Revenue grew 12% this quarter.

+
  • Cloud up
  • Ads flat
+ Full breakdown + + + """ + result = StdlibHtmlExtractor().extract(html, url="https://site.test/report") + assert result is not None + assert result.title == "Quarterly Report" + assert "Revenue grew 12% this quarter." in result.text + assert "# Revenue" in result.text + assert "- Cloud up" in result.text + # Chrome and code must not leak into the reading. + assert "tracking" not in result.text + assert "color:red" not in result.text + urls = [url for _, url in result.links] + assert "https://site.test/detail/q3" in urls + assert not any(url.endswith("/home") or url.endswith("/legal") for url in urls) + + +def test_select_path_walks_dicts_and_list_indices() -> None: + payload = {"chart": {"result": [{"meta": {"price": 128.99}}]}} + value, error = select_path(payload, "chart.result.0.meta.price") + assert error == "" + assert value == 128.99 + + +def test_select_path_errors_name_the_failing_segment() -> None: + payload = {"chart": {"result": [{"meta": {"price": 128.99}}]}} + _, missing_key = select_path(payload, "chart.results") + assert "chart.results" in missing_key and "available keys" in missing_key + _, bad_index = select_path(payload, "chart.result.9") + assert "index out of range" in bad_index + _, not_indexable = select_path(payload, "chart.result.0.meta.price.deeper") + assert "not indexable" in not_indexable + _, bad_list_key = select_path(payload, "chart.result.first") + assert "expected a list index" in bad_list_key + + +def test_decode_json_reports_position_and_prefix() -> None: + data, error = decode_json("Edge: Too Many Requests") + assert data is None + assert "did not parse" in error + assert "Edge: Too Many Requests" in error + + +# ── tool behavior ──────────────────────────────────────────────────── + +def test_json_fetch_returns_selected_value(monkeypatch) -> None: + body = b'{"chart":{"result":[{"meta":{"regularMarketPrice":128.99}}]}}' + transport = _FakeTransport(_outcome(body=body)) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL, "select": "chart.result.0.meta.regularMarketPrice"}) + + assert result["ok"] is True + assert result["status"] == 200 + assert result["kind"] == KIND_JSON + assert result["data"] == 128.99 + assert result["select"] == "chart.result.0.meta.regularMarketPrice" + + +def test_rate_limited_response_reports_status_not_a_traceback(monkeypatch) -> None: + """The regression this tool exists to prevent. + + Fetching through a shell pipeline turned a 429 with a non-JSON body into a + Python traceback from `json.load`. Here the status is the answer, the body is + quoted as evidence, and the failure is marked retryable. + """ + transport = _FakeTransport( + _outcome(status=429, content_type="text/plain", body=b"Edge: Too Many Requests") + ) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["status"] == 429 + assert result["error_type"] == "http_error" + assert result["retryable"] is True + assert "Edge: Too Many Requests" in result["body_excerpt"] + assert "Traceback" not in result["error"] + + +def test_declared_json_that_does_not_parse_is_a_content_error(monkeypatch) -> None: + transport = _FakeTransport( + _outcome(status=200, content_type="application/json", body=b"nope") + ) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["error_type"] == "invalid_json" + assert result["status"] == 200 + assert "nope" in result["body_excerpt"] + + +def test_bad_selector_lists_available_keys(monkeypatch) -> None: + transport = _FakeTransport(_outcome(body=b'{"a":1,"b":2}')) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL, "select": "missing.key"}) + + assert result["ok"] is False + assert result["error_type"] == "invalid_selector" + assert result["retryable"] is True + assert result["available_top_level_keys"] == ["a", "b"] + + +def test_html_fetch_extracts_text_and_links(monkeypatch) -> None: + html = b"T

Hello world

" \ + b'Next' + transport = _FakeTransport(_outcome(content_type="text/html", body=html)) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is True + assert result["kind"] == KIND_HTML + assert result["title"] == "T" + assert "Hello world" in result["text"] + assert result["links"][0]["url"] == "https://site.test/next" + assert result["extractor"] == "stdlib" + + +def test_binary_content_is_not_returned_inline(monkeypatch) -> None: + transport = _FakeTransport( + _outcome(content_type="application/pdf", body=b"%PDF-1.7 binary...") + ) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["kind"] == KIND_BINARY + assert result["text"] == "" + assert "not returned inline" in result["note"] + + +def test_retryable_status_is_retried_then_succeeds(monkeypatch) -> None: + transport = _FakeTransport( + _outcome(status=503, content_type="text/plain", body=b"busy"), + _outcome(status=200, content_type="text/plain", body=b"ready"), + ) + _install(monkeypatch, transport, web_max_retries=1) + # Collapse the backoff instead of patching asyncio.sleep: the retry loop is + # what is under test, not the timer. + monkeypatch.setattr(wf, "_retry_delay", lambda attempt: 0.0) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is True + assert result["text"] == "ready" + assert len(transport.requests) == 2 + + +def test_transport_failure_is_typed_and_retryable(monkeypatch) -> None: + transport = _FakeTransport(wf.TransportFailure("timeout", "timed out", retryable=True)) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["error_type"] == "timeout" + assert result["retryable"] is True + + +def test_client_rejection_fails_over_to_the_next_transport(monkeypatch) -> None: + """A 429 from one stack must try the other before reporting failure. + + Observed against a real CDN edge: the same URL with the same user agent answers + 429 intermittently, and which transport gets refused changes between runs. One + extra attempt on the other transport is therefore worth it — without it the + model is pushed back to running curl through the shell, which is exactly what + this tool exists to replace. + """ + refused = _FakeTransport(_outcome(status=429, content_type="text/html", body=b"Too Many Requests")) + accepted = _FakeTransport(_outcome(status=200, body=b'{"price": 128.99}')) + monkeypatch.setattr(wf, "transports_for", lambda preference: (refused, accepted)) + monkeypatch.setattr(wf, "_settings", lambda: _settings()) + + result = _run({"url": PUBLIC_URL, "select": "price"}) + + assert result["ok"] is True + assert result["data"] == 128.99 + assert len(refused.requests) == 1 + assert len(accepted.requests) == 1 + + +def test_rejection_by_every_transport_reports_the_last_status(monkeypatch) -> None: + first = _FakeTransport(_outcome(status=403, content_type="text/plain", body=b"denied")) + second = _FakeTransport(_outcome(status=429, content_type="text/plain", body=b"slow down")) + monkeypatch.setattr(wf, "transports_for", lambda preference: (first, second)) + monkeypatch.setattr(wf, "_settings", lambda: _settings()) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["status"] == 429 + assert result["error_type"] == "http_error" + assert "slow down" in result["body_excerpt"] + + +def test_ordinary_error_status_does_not_fail_over(monkeypatch) -> None: + """A 404 is the answer; trying another stack would only waste a request.""" + first = _FakeTransport(_outcome(status=404, content_type="text/plain", body=b"nope")) + second = _FakeTransport(_outcome(status=200, body=b"{}")) + monkeypatch.setattr(wf, "transports_for", lambda preference: (first, second)) + monkeypatch.setattr(wf, "_settings", lambda: _settings()) + + result = _run({"url": PUBLIC_URL}) + + assert result["status"] == 404 + assert second.requests == [] + + +def test_failure_in_first_transport_falls_through(monkeypatch) -> None: + broken = _FakeTransport(wf.TransportFailure("tls_error", "bad cert", retryable=False)) + working = _FakeTransport(_outcome(status=200, content_type="text/plain", body=b"ok")) + monkeypatch.setattr(wf, "transports_for", lambda preference: (broken, working)) + monkeypatch.setattr(wf, "_settings", lambda: _settings()) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is True + assert result["text"] == "ok" + + +def test_unsupported_scheme_is_refused_before_any_request(monkeypatch) -> None: + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport) + + result = _run({"url": "file:///etc/passwd"}) + + assert result["ok"] is False + assert result["error_type"] == "unsupported_scheme" + assert result["retryable"] is False + assert transport.requests == [] + + +def test_internal_target_requires_approval_and_fails_closed(monkeypatch) -> None: + """With no gate installed, an internal target must not be reachable.""" + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport) + monkeypatch.setattr(wf, "_approval_gate", None) + + result = _run({"url": LOOPBACK_URL}) + + assert result["ok"] is False + assert result["error_type"] == "blocked_target" + assert result["requires_approval"] is True + assert result["target_category"] == "loopback" + assert transport.requests == [] + + +def test_internal_target_proceeds_once_approved(monkeypatch) -> None: + transport = _FakeTransport(_outcome(content_type="text/plain", body=b"internal ok")) + _install(monkeypatch, transport) + seen: list[ActionDescriptor] = [] + + class _Gate: + async def evaluate(self, action): + seen.append(action) + return SimpleNamespace(approved=True) + + monkeypatch.setattr(wf, "_approval_gate", _Gate()) + + result = _run({"url": LOOPBACK_URL}) + + assert result["ok"] is True + assert result["text"] == "internal ok" + assert seen[0].kind == ActionKind.NETWORK_FETCH.value + assert seen[0].resource == "http://127.0.0.1:8765" + + +def test_denied_approval_blocks_the_fetch(monkeypatch) -> None: + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport) + + class _Gate: + async def evaluate(self, action): + return SimpleNamespace(approved=False, denial_message="user said no") + + monkeypatch.setattr(wf, "_approval_gate", _Gate()) + + result = _run({"url": METADATA_URL}) + + assert result["ok"] is False + assert result["error"] == "user said no" + assert transport.requests == [] + + +def test_broken_gate_denies_rather_than_opening(monkeypatch) -> None: + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport) + + class _Gate: + async def evaluate(self, action): + raise RuntimeError("gate exploded") + + monkeypatch.setattr(wf, "_approval_gate", _Gate()) + + result = _run({"url": LOOPBACK_URL}) + + assert result["ok"] is False + assert "could not be consulted" in result["error"] + assert transport.requests == [] + + +def test_private_targets_deny_mode_skips_the_prompt(monkeypatch) -> None: + """Operators running unattended need a hard refusal without a prompt.""" + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport, web_private_targets="deny") + + class _Gate: + async def evaluate(self, action): # pragma: no cover - must not be reached + raise AssertionError("deny mode must not consult the gate") + + monkeypatch.setattr(wf, "_approval_gate", _Gate()) + + result = _run({"url": LOOPBACK_URL}) + + assert result["ok"] is False + assert result["error_type"] == "blocked_target" + assert result.get("requires_approval") is None + assert transport.requests == [] + + +def test_private_targets_allow_mode_needs_no_gate(monkeypatch) -> None: + transport = _FakeTransport(_outcome(content_type="text/plain", body=b"dashboard")) + _install(monkeypatch, transport, web_private_targets="allow") + monkeypatch.setattr(wf, "_approval_gate", None) + + result = _run({"url": LOOPBACK_URL}) + + assert result["ok"] is True + assert result["text"] == "dashboard" + + +def test_browser_user_agent_is_sent_and_overridable(monkeypatch) -> None: + """Default UA is browser-style because many CDNs reject library agents.""" + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport) + _run({"url": PUBLIC_URL}) + assert transport.requests[0].user_agent == wf.DEFAULT_USER_AGENT + assert "Mozilla/5.0" in wf.DEFAULT_USER_AGENT + + custom = _FakeTransport(_outcome()) + _install(monkeypatch, custom, web_user_agent="LeapFlow/test") + _run({"url": PUBLIC_URL}) + assert custom.requests[0].user_agent == "LeapFlow/test" + + +def test_limits_come_from_settings_and_params(monkeypatch) -> None: + transport = _FakeTransport(_outcome()) + _install(monkeypatch, transport, web_timeout_s=7.0, web_max_bytes=4096) + _run({"url": PUBLIC_URL}) + assert transport.requests[0].timeout_s == 7.0 + assert transport.requests[0].max_bytes == 4096 + + override = _FakeTransport(_outcome()) + _install(monkeypatch, override, web_timeout_s=7.0) + _run({"url": PUBLIC_URL, "timeout": 3, "max_bytes": 128}) + assert override.requests[0].timeout_s == 3 + assert override.requests[0].max_bytes == 128 + + +def test_missing_url_is_a_retryable_argument_error() -> None: + result = _run({}) + assert result["ok"] is False + assert result["retryable"] is True + + +def test_no_transport_available_is_reported_clearly(monkeypatch) -> None: + monkeypatch.setattr(wf, "transports_for", lambda preference: ()) + monkeypatch.setattr(wf, "_settings", lambda: _settings()) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["error_type"] == "transport_unavailable" + assert "httpx" in result["error"] + + +def _redirect(location: str, *, status: int = 302): + return wf.FetchOutcome( + status=status, + final_url=PUBLIC_URL, + content_type="text/html", + body=b"", + truncated=False, + transport="fake", + elapsed_ms=3, + location=location, + ) + + +# ── redirects: every hop must be re-gated ─────────────────────── + +def test_redirect_to_internal_target_is_gated(monkeypatch) -> None: + """A public URL must not be able to bounce the request into the network. + + If the HTTP client follows redirects itself, the egress gate only ever sees + the first hop, and any server able to answer 302 can read loopback services + or cloud instance metadata on the agent's behalf. + """ + transport = _FakeTransport( + _redirect("http://169.254.169.254/latest/meta-data/iam/security-credentials/"), + _outcome(content_type="text/plain", body=b"AWS_SECRET"), + ) + _install(monkeypatch, transport) + monkeypatch.setattr(wf, "_approval_gate", None) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["error_type"] == "blocked_target" + assert result["target_category"] == "metadata" + assert "Redirected to" in result["error"] + assert result["blocked_url"].startswith("http://169.254.169.254/") + # The second hop must never have been issued. + assert len(transport.requests) == 1 + + +def test_redirect_to_loopback_is_gated(monkeypatch) -> None: + transport = _FakeTransport( + _redirect("http://127.0.0.1:8765/admin"), + _outcome(content_type="text/plain", body=b"internal"), + ) + _install(monkeypatch, transport) + monkeypatch.setattr(wf, "_approval_gate", None) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["target_category"] == "loopback" + assert len(transport.requests) == 1 + + +def test_public_redirect_is_followed_and_reported(monkeypatch) -> None: + """Ordinary redirects still work, with the chain visible.""" + transport = _FakeTransport( + _redirect("https://93.184.216.34/moved"), + _outcome(body=b'{"price": 7}'), + ) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL, "select": "price"}) + + assert result["ok"] is True + assert result["data"] == 7 + assert result["redirect_chain"] == [PUBLIC_URL, "https://93.184.216.34/moved"] + assert len(transport.requests) == 2 + + +def test_relative_redirect_is_resolved(monkeypatch) -> None: + transport = _FakeTransport( + _redirect("/elsewhere"), + _outcome(content_type="text/plain", body=b"landed"), + ) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is True + assert transport.requests[1].url == "https://93.184.216.34/elsewhere" + + +def test_redirect_loop_is_reported(monkeypatch) -> None: + transport = _FakeTransport(_redirect(PUBLIC_URL)) + _install(monkeypatch, transport) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["error_type"] == "redirect_loop" + + +def test_redirect_budget_is_bounded(monkeypatch) -> None: + """An endless chain of distinct hops must stop at the configured budget.""" + hops = iter(range(1, 50)) + + class _Chain: + name = "chain" + + def __init__(self) -> None: + self.requests: list[wf.FetchRequest] = [] + + def available(self) -> bool: + return True + + async def fetch(self, request): + self.requests.append(request) + return _redirect(f"https://93.184.216.34/hop{next(hops)}") + + transport = _Chain() + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: _settings(web_max_redirects=3)) + + result = _run({"url": PUBLIC_URL}) + + assert result["ok"] is False + assert result["error_type"] == "too_many_redirects" + assert len(transport.requests) == 4 # initial request + 3 redirects + + +def test_transports_do_not_follow_redirects_themselves() -> None: + """Auto-follow in the client would bypass the per-hop gate entirely.""" + import inspect + + httpx_source = inspect.getsource(wf.HttpxTransport.fetch) + assert "follow_redirects=False" in httpx_source + + curl_source = inspect.getsource(wf.CurlTransport.fetch) + assert '"--location"' not in curl_source + assert "redirect_url" in curl_source + + +# ── caching ─────────────────────────────────────────────── + +def test_second_fetch_is_served_from_cache(monkeypatch, tmp_path) -> None: + """A repeated read must not re-hit the network within its TTL.""" + transport = _FakeTransport(_outcome(body=b'{"price": 1}')) + settings = _settings_with_cache(tmp_path) + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: settings) + + first = _run({"url": PUBLIC_URL, "select": "price"}) + second = _run({"url": PUBLIC_URL, "select": "price"}) + + assert first["ok"] is True and first.get("from_cache") is None + assert second["ok"] is True and second["from_cache"] is True + assert second["data"] == 1 + assert len(transport.requests) == 1 + + +def test_cache_entry_is_session_scoped_and_not_syncable(monkeypatch, tmp_path) -> None: + """Fetched content is reproducible and may be per-session: never sync it out.""" + from leapflow.cache.manager import CacheManager + + transport = _FakeTransport(_outcome(body=b'{"a": 1}')) + settings = _settings_with_cache(tmp_path) + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: settings) + + _run({"url": PUBLIC_URL}) + + manager = CacheManager(settings.profile_layout.cache, profile_id="default") + entries = [e for e in manager.list_entries() if e.owner_component == "web_fetch"] + assert entries, "fetch body was not indexed" + entry = entries[0] + assert entry.scope == "session" + assert entry.syncable is False + assert entry.category == "web_fetch" + assert entry.expires_at is not None + + +def test_zero_ttl_disables_caching(monkeypatch, tmp_path) -> None: + transport = _FakeTransport(_outcome(body=b"{}"), _outcome(body=b"{}")) + settings = _settings_with_cache(tmp_path, web_cache_ttl_s=0.0) + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: settings) + + _run({"url": PUBLIC_URL}) + second = _run({"url": PUBLIC_URL}) + + assert second.get("from_cache") is None + assert len(transport.requests) == 2 + + +def test_binary_content_is_saved_and_referenced_by_path(monkeypatch, tmp_path) -> None: + """Binary stays out of the transcript but must not become a dead end.""" + transport = _FakeTransport( + _outcome(content_type="application/pdf", body=b"%PDF-1.7 payload") + ) + settings = _settings_with_cache(tmp_path) + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: settings) + + result = _run({"url": PUBLIC_URL}) + + assert result["kind"] == KIND_BINARY + assert result["text"] == "" + stored = Path(result["cache_path"]) + assert stored.read_bytes() == b"%PDF-1.7 payload" + + +def test_cache_is_consulted_only_after_the_egress_gate(monkeypatch, tmp_path) -> None: + """A cached body must not become a bypass for an unapproved internal target.""" + transport = _FakeTransport(_outcome(content_type="text/plain", body=b"internal")) + settings = _settings_with_cache(tmp_path) + monkeypatch.setattr(wf, "transports_for", lambda preference: (transport,)) + monkeypatch.setattr(wf, "_settings", lambda: settings) + + class _Gate: + def __init__(self) -> None: + self.approve = True + + async def evaluate(self, action): + return SimpleNamespace(approved=self.approve, denial_message="denied now") + + gate = _Gate() + monkeypatch.setattr(wf, "_approval_gate", gate) + + first = _run({"url": LOOPBACK_URL}) + assert first["ok"] is True + + # Revoking approval must block the second read even though it is cached. + gate.approve = False + second = _run({"url": LOOPBACK_URL}) + assert second["ok"] is False + assert second["error_type"] == "blocked_target" + + +def test_approval_detail_never_carries_query_secrets(monkeypatch) -> None: + """The approval prompt and audit log persist the action detail. + + A query string routinely holds an API key or signed token, so origin+path is + the most that may be written down. + """ + transport = _FakeTransport(_outcome(content_type="text/plain", body=b"ok")) + _install(monkeypatch, transport) + seen: list[ActionDescriptor] = [] + + class _Gate: + async def evaluate(self, action): + seen.append(action) + return SimpleNamespace(approved=True) + + monkeypatch.setattr(wf, "_approval_gate", _Gate()) + + _run({"url": "http://127.0.0.1:8765/admin?token=SUPERSECRET&x=1#frag"}) + + action = seen[0] + assert "SUPERSECRET" not in action.detail + assert "SUPERSECRET" not in json.dumps(action.metadata) + assert action.detail == "http://127.0.0.1:8765/admin" + assert action.resource == "http://127.0.0.1:8765" + + +def test_credentialed_url_is_not_echoed_into_approval(monkeypatch) -> None: + transport = _FakeTransport(_outcome(content_type="text/plain", body=b"ok")) + _install(monkeypatch, transport) + seen: list[ActionDescriptor] = [] + + class _Gate: + async def evaluate(self, action): + seen.append(action) + return SimpleNamespace(approved=True) + + monkeypatch.setattr(wf, "_approval_gate", _Gate()) + + _run({"url": "https://user:hunter2@93.184.216.34/private"}) + + action = seen[0] + assert "hunter2" not in action.detail + assert "hunter2" not in json.dumps(action.metadata) + + +def test_failure_evidence_keeps_status_and_body(monkeypatch) -> None: + """A compacted HTTP failure must still explain itself to the model.""" + from leapflow.engine.context_control import ToolEvidenceBuilder + + failure = { + "ok": False, + "status": 429, + "error": "HTTP 429 from https://api.test", + "error_type": "http_error", + "retryable": True, + "body_excerpt": "Edge: Too Many Requests", + } + evidence = ToolEvidenceBuilder(max_content_chars=400).build("web_fetch", {}, failure) + + assert evidence["ok"] is False + assert evidence["status"] == 429 + assert evidence["retryable"] is True + assert "Edge: Too Many Requests" in evidence["body_excerpt"] + + +# ── loop integration contracts ─────────────────────────────────────── + +def test_web_fetch_is_read_only_for_the_execution_ledger() -> None: + """read_only is the whole point: retries stay safe and batches keep running.""" + from leapflow.engine.engine import _SIDE_EFFECT_STOP_POLICIES, _default_tool_registry + from leapflow.engine.tool_execution import ( + effect_is_uncertain_on_failure, + execution_policy_for, + ) + + registry = _default_tool_registry() + policy = execution_policy_for("web_fetch", registry.specs.get("web_fetch")) + assert policy == "read_only" + assert effect_is_uncertain_on_failure(policy) is False + assert policy not in _SIDE_EFFECT_STOP_POLICIES + + +def test_web_fetch_is_disclosed_in_the_core_tier() -> None: + """A network capability the model cannot see is why it fell back to shell.""" + from leapflow.engine.context_disclosure import DisclosurePlanner, DisclosureRuntimeState + from leapflow.tools.registry_bootstrap import TOOL_DEFINITIONS, TOOL_HANDLERS + + plan = DisclosurePlanner().plan( + TOOL_DEFINITIONS, DisclosureRuntimeState(native_tools_enabled=True) + ) + names = [item["function"]["name"] for item in plan.tool_definitions] + assert "web_fetch" in names + assert "web_fetch" in TOOL_HANDLERS + assert "gp_web_fetch" in TOOL_HANDLERS + + +def test_evidence_builder_caps_fetched_bodies() -> None: + from leapflow.engine.context_control import ToolEvidenceBuilder + + builder = ToolEvidenceBuilder(max_content_chars=400) + result = { + "ok": True, + "status": 200, + "url": PUBLIC_URL, + "final_url": PUBLIC_URL, + "content_type": "text/html", + "title": "Big page", + "text": "HEAD" + ("x" * 20_000) + "TAIL", + "links": [{"text": f"l{i}", "url": f"https://site.test/{i}"} for i in range(50)], + "extractor": "stdlib", + } + evidence = builder.build("web_fetch", {"url": PUBLIC_URL}, result) + + assert evidence["kind"] == "web_fetch_evidence" + assert evidence["status"] == 200 + assert evidence["title"] == "Big page" + assert len(evidence["text"]) < 1000 + assert "HEAD" in evidence["text"] and "TAIL" in evidence["text"] + assert len(evidence["links"]) <= 50 + + +def test_web_config_keys_are_user_visible() -> None: + """Every durable limit must be reachable from `leap config` / `/config`.""" + from leapflow.config import get_settings + from leapflow.config_service import ConfigService + + keys = set(ConfigService(get_settings()).writable_keys()) + for key in ( + "web.transport", + "web.timeout_s", + "web.max_bytes", + "web.max_retries", + "web.user_agent", + "web.private_targets", + "web.extractor", + ): + assert key in keys, key diff --git a/uv.lock b/uv.lock index 12bd8b2..48cf6c5 100644 --- a/uv.lock +++ b/uv.lock @@ -179,6 +179,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548 }, ] +[[package]] +name = "babel" +version = "2.18.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845 }, +] + [[package]] name = "certifi" version = "2026.4.22" @@ -260,91 +269,76 @@ wheels = [ [[package]] name = "charset-normalizer" -version = "3.4.7" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/e7/a1/67fe25fac3c7642725500a3f6cfe5821ad557c3abb11c9d20d12c7008d3e/charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5", size = 144271 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/c2/d7/b5b7020a0565c2e9fa8c09f4b5fa6232feb326b8c20081ccded47ea368fd/charset_normalizer-3.4.7-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:7641bb8895e77f921102f72833904dcd9901df5d6d72a2ab8f31d04b7e51e4e7", size = 309705 }, - { url = "https://files.pythonhosted.org/packages/5a/53/58c29116c340e5456724ecd2fff4196d236b98f3da97b404bc5e51ac3493/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:202389074300232baeb53ae2569a60901f7efadd4245cf3a3bf0617d60b439d7", size = 206419 }, - { url = "https://files.pythonhosted.org/packages/b2/02/e8146dc6591a37a00e5144c63f29fb7c97a734ea8a111190783c0e60ab63/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:30b8d1d8c52a48c2c5690e152c169b673487a2a58de1ec7393196753063fcd5e", size = 227901 }, - { url = "https://files.pythonhosted.org/packages/fb/73/77486c4cd58f1267bf17db420e930c9afa1b3be3fe8c8b8ebbebc9624359/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:532bc9bf33a68613fd7d65e4b1c71a6a38d7d42604ecf239c77392e9b4e8998c", size = 222742 }, - { url = "https://files.pythonhosted.org/packages/a1/fa/f74eb381a7d94ded44739e9d94de18dc5edc9c17fb8c11f0a6890696c0a9/charset_normalizer-3.4.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2fe249cb4651fd12605b7288b24751d8bfd46d35f12a20b1ba33dea122e690df", size = 214061 }, - { url = "https://files.pythonhosted.org/packages/dc/92/42bd3cefcf7687253fb86694b45f37b733c97f59af3724f356fa92b8c344/charset_normalizer-3.4.7-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:65bcd23054beab4d166035cabbc868a09c1a49d1efe458fe8e4361215df40265", size = 199239 }, - { url = "https://files.pythonhosted.org/packages/4c/3d/069e7184e2aa3b3cddc700e3dd267413dc259854adc3380421c805c6a17d/charset_normalizer-3.4.7-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:08e721811161356f97b4059a9ba7bafb23ea5ee2255402c42881c214e173c6b4", size = 210173 }, - { url = "https://files.pythonhosted.org/packages/62/51/9d56feb5f2e7074c46f93e0ebdbe61f0848ee246e2f0d89f8e20b89ebb8f/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:e060d01aec0a910bdccb8be71faf34e7799ce36950f8294c8bf612cba65a2c9e", size = 209841 }, - { url = "https://files.pythonhosted.org/packages/d2/59/893d8f99cc4c837dda1fe2f1139079703deb9f321aabcb032355de13b6c7/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:38c0109396c4cfc574d502df99742a45c72c08eff0a36158b6f04000043dbf38", size = 200304 }, - { url = "https://files.pythonhosted.org/packages/7d/1d/ee6f3be3464247578d1ed5c46de545ccc3d3ff933695395c402c21fa6b77/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:1c2a768fdd44ee4a9339a9b0b130049139b8ce3c01d2ce09f67f5a68048d477c", size = 229455 }, - { url = "https://files.pythonhosted.org/packages/54/bb/8fb0a946296ea96a488928bdce8ef99023998c48e4713af533e9bb98ef07/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:1a87ca9d5df6fe460483d9a5bbf2b18f620cbed41b432e2bddb686228282d10b", size = 210036 }, - { url = "https://files.pythonhosted.org/packages/9a/bc/015b2387f913749f82afd4fcba07846d05b6d784dd16123cb66860e0237d/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:d635aab80466bc95771bb78d5370e74d36d1fe31467b6b29b8b57b2a3cd7d22c", size = 224739 }, - { url = "https://files.pythonhosted.org/packages/17/ab/63133691f56baae417493cba6b7c641571a2130eb7bceba6773367ab9ec5/charset_normalizer-3.4.7-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ae196f021b5e7c78e918242d217db021ed2a6ace2bc6ae94c0fc596221c7f58d", size = 216277 }, - { url = "https://files.pythonhosted.org/packages/06/6d/3be70e827977f20db77c12a97e6a9f973631a45b8d186c084527e53e77a4/charset_normalizer-3.4.7-cp311-cp311-win32.whl", hash = "sha256:adb2597b428735679446b46c8badf467b4ca5f5056aae4d51a19f9570301b1ad", size = 147819 }, - { url = "https://files.pythonhosted.org/packages/20/d9/5f67790f06b735d7c7637171bbfd89882ad67201891b7275e51116ed8207/charset_normalizer-3.4.7-cp311-cp311-win_amd64.whl", hash = "sha256:8e385e4267ab76874ae30db04c627faaaf0b509e1ccc11a95b3fc3e83f855c00", size = 159281 }, - { url = "https://files.pythonhosted.org/packages/ca/83/6413f36c5a34afead88ce6f66684d943d91f233d76dd083798f9602b75ae/charset_normalizer-3.4.7-cp311-cp311-win_arm64.whl", hash = "sha256:d4a48e5b3c2a489fae013b7589308a40146ee081f6f509e047e0e096084ceca1", size = 147843 }, - { url = "https://files.pythonhosted.org/packages/0c/eb/4fc8d0a7110eb5fc9cc161723a34a8a6c200ce3b4fbf681bc86feee22308/charset_normalizer-3.4.7-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:eca9705049ad3c7345d574e3510665cb2cf844c2f2dcfe675332677f081cbd46", size = 311328 }, - { url = "https://files.pythonhosted.org/packages/f8/e3/0fadc706008ac9d7b9b5be6dc767c05f9d3e5df51744ce4cc9605de7b9f4/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6178f72c5508bfc5fd446a5905e698c6212932f25bcdd4b47a757a50605a90e2", size = 208061 }, - { url = "https://files.pythonhosted.org/packages/42/f0/3dd1045c47f4a4604df85ec18ad093912ae1344ac706993aff91d38773a2/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1421b502d83040e6d7fb2fb18dff63957f720da3d77b2fbd3187ceb63755d7b", size = 229031 }, - { url = "https://files.pythonhosted.org/packages/dc/67/675a46eb016118a2fbde5a277a5d15f4f69d5f3f5f338e5ee2f8948fcf43/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:edac0f1ab77644605be2cbba52e6b7f630731fc42b34cb0f634be1a6eface56a", size = 225239 }, - { url = "https://files.pythonhosted.org/packages/4b/f8/d0118a2f5f23b02cd166fa385c60f9b0d4f9194f574e2b31cef350ad7223/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5649fd1c7bade02f320a462fdefd0b4bd3ce036065836d4f42e0de958038e116", size = 216589 }, - { url = "https://files.pythonhosted.org/packages/b1/f1/6d2b0b261b6c4ceef0fcb0d17a01cc5bc53586c2d4796fa04b5c540bc13d/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:203104ed3e428044fd943bc4bf45fa73c0730391f9621e37fe39ecf477b128cb", size = 202733 }, - { url = "https://files.pythonhosted.org/packages/6f/c0/7b1f943f7e87cc3db9626ba17807d042c38645f0a1d4415c7a14afb5591f/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:298930cec56029e05497a76988377cbd7457ba864beeea92ad7e844fe74cd1f1", size = 212652 }, - { url = "https://files.pythonhosted.org/packages/38/dd/5a9ab159fe45c6e72079398f277b7d2b523e7f716acc489726115a910097/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:708838739abf24b2ceb208d0e22403dd018faeef86ddac04319a62ae884c4f15", size = 211229 }, - { url = "https://files.pythonhosted.org/packages/d5/ff/531a1cad5ca855d1c1a8b69cb71abfd6d85c0291580146fda7c82857caa1/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:0f7eb884681e3938906ed0434f20c63046eacd0111c4ba96f27b76084cd679f5", size = 203552 }, - { url = "https://files.pythonhosted.org/packages/c1/4c/a5fb52d528a8ca41f7598cb619409ece30a169fbdf9cdce592e53b46c3a6/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4dc1e73c36828f982bfe79fadf5919923f8a6f4df2860804db9a98c48824ce8d", size = 230806 }, - { url = "https://files.pythonhosted.org/packages/59/7a/071feed8124111a32b316b33ae4de83d36923039ef8cf48120266844285b/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:aed52fea0513bac0ccde438c188c8a471c4e0f457c2dd20cdbf6ea7a450046c7", size = 212316 }, - { url = "https://files.pythonhosted.org/packages/fd/35/f7dba3994312d7ba508e041eaac39a36b120f32d4c8662b8814dab876431/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464", size = 227274 }, - { url = "https://files.pythonhosted.org/packages/8a/2d/a572df5c9204ab7688ec1edc895a73ebded3b023bb07364710b05dd1c9be/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bb6d88045545b26da47aa879dd4a89a71d1dce0f0e549b1abcb31dfe4a8eac49", size = 218468 }, - { url = "https://files.pythonhosted.org/packages/86/eb/890922a8b03a568ca2f336c36585a4713c55d4d67bf0f0c78924be6315ca/charset_normalizer-3.4.7-cp312-cp312-win32.whl", hash = "sha256:2257141f39fe65a3fdf38aeccae4b953e5f3b3324f4ff0daf9f15b8518666a2c", size = 148460 }, - { url = "https://files.pythonhosted.org/packages/35/d9/0e7dffa06c5ab081f75b1b786f0aefc88365825dfcd0ac544bdb7b2b6853/charset_normalizer-3.4.7-cp312-cp312-win_amd64.whl", hash = "sha256:5ed6ab538499c8644b8a3e18debabcd7ce684f3fa91cf867521a7a0279cab2d6", size = 159330 }, - { url = "https://files.pythonhosted.org/packages/9e/5d/481bcc2a7c88ea6b0878c299547843b2521ccbc40980cb406267088bc701/charset_normalizer-3.4.7-cp312-cp312-win_arm64.whl", hash = "sha256:56be790f86bfb2c98fb742ce566dfb4816e5a83384616ab59c49e0604d49c51d", size = 147828 }, - { url = "https://files.pythonhosted.org/packages/c1/3b/66777e39d3ae1ddc77ee606be4ec6d8cbd4c801f65e5a1b6f2b11b8346dd/charset_normalizer-3.4.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f496c9c3cc02230093d8330875c4c3cdfc3b73612a5fd921c65d39cbcef08063", size = 309627 }, - { url = "https://files.pythonhosted.org/packages/2e/4e/b7f84e617b4854ade48a1b7915c8ccfadeba444d2a18c291f696e37f0d3b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea948db76d31190bf08bd371623927ee1339d5f2a0b4b1b4a4439a65298703c", size = 207008 }, - { url = "https://files.pythonhosted.org/packages/c4/bb/ec73c0257c9e11b268f018f068f5d00aa0ef8c8b09f7753ebd5f2880e248/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a277ab8928b9f299723bc1a2dabb1265911b1a76341f90a510368ca44ad9ab66", size = 228303 }, - { url = "https://files.pythonhosted.org/packages/85/fb/32d1f5033484494619f701e719429c69b766bfc4dbc61aa9e9c8c166528b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3bec022aec2c514d9cf199522a802bd007cd588ab17ab2525f20f9c34d067c18", size = 224282 }, - { url = "https://files.pythonhosted.org/packages/fa/07/330e3a0dda4c404d6da83b327270906e9654a24f6c546dc886a0eb0ffb23/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e044c39e41b92c845bc815e5ae4230804e8e7bc29e399b0437d64222d92809dd", size = 215595 }, - { url = "https://files.pythonhosted.org/packages/e3/7c/fc890655786e423f02556e0216d4b8c6bcb6bdfa890160dc66bf52dee468/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:f495a1652cf3fbab2eb0639776dad966c2fb874d79d87ca07f9d5f059b8bd215", size = 201986 }, - { url = "https://files.pythonhosted.org/packages/d8/97/bfb18b3db2aed3b90cf54dc292ad79fdd5ad65c4eae454099475cbeadd0d/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e712b419df8ba5e42b226c510472b37bd57b38e897d3eca5e8cfd410a29fa859", size = 211711 }, - { url = "https://files.pythonhosted.org/packages/6f/a5/a581c13798546a7fd557c82614a5c65a13df2157e9ad6373166d2a3e645d/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7804338df6fcc08105c7745f1502ba68d900f45fd770d5bdd5288ddccb8a42d8", size = 210036 }, - { url = "https://files.pythonhosted.org/packages/8c/bf/b3ab5bcb478e4193d517644b0fb2bf5497fbceeaa7a1bc0f4d5b50953861/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:481551899c856c704d58119b5025793fa6730adda3571971af568f66d2424bb5", size = 202998 }, - { url = "https://files.pythonhosted.org/packages/e7/4e/23efd79b65d314fa320ec6017b4b5834d5c12a58ba4610aa353af2e2f577/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:f59099f9b66f0d7145115e6f80dd8b1d847176df89b234a5a6b3f00437aa0832", size = 230056 }, - { url = "https://files.pythonhosted.org/packages/b9/9f/1e1941bc3f0e01df116e68dc37a55c4d249df5e6fa77f008841aef68264f/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6", size = 211537 }, - { url = "https://files.pythonhosted.org/packages/80/0f/088cbb3020d44428964a6c97fe1edfb1b9550396bf6d278330281e8b709c/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:3dedcc22d73ec993f42055eff4fcfed9318d1eeb9a6606c55892a26964964e48", size = 226176 }, - { url = "https://files.pythonhosted.org/packages/6a/9f/130394f9bbe06f4f63e22641d32fc9b202b7e251c9aef4db044324dac493/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:64f02c6841d7d83f832cd97ccf8eb8a906d06eb95d5276069175c696b024b60a", size = 217723 }, - { url = "https://files.pythonhosted.org/packages/73/55/c469897448a06e49f8fa03f6caae97074fde823f432a98f979cc42b90e69/charset_normalizer-3.4.7-cp313-cp313-win32.whl", hash = "sha256:4042d5c8f957e15221d423ba781e85d553722fc4113f523f2feb7b188cc34c5e", size = 148085 }, - { url = "https://files.pythonhosted.org/packages/5d/78/1b74c5bbb3f99b77a1715c91b3e0b5bdb6fe302d95ace4f5b1bec37b0167/charset_normalizer-3.4.7-cp313-cp313-win_amd64.whl", hash = "sha256:3946fa46a0cf3e4c8cb1cc52f56bb536310d34f25f01ca9b6c16afa767dab110", size = 158819 }, - { url = "https://files.pythonhosted.org/packages/68/86/46bd42279d323deb8687c4a5a811fd548cb7d1de10cf6535d099877a9a9f/charset_normalizer-3.4.7-cp313-cp313-win_arm64.whl", hash = "sha256:80d04837f55fc81da168b98de4f4b797ef007fc8a79ab71c6ec9bc4dd662b15b", size = 147915 }, - { url = "https://files.pythonhosted.org/packages/97/c8/c67cb8c70e19ef1960b97b22ed2a1567711de46c4ddf19799923adc836c2/charset_normalizer-3.4.7-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c36c333c39be2dbca264d7803333c896ab8fa7d4d6f0ab7edb7dfd7aea6e98c0", size = 309234 }, - { url = "https://files.pythonhosted.org/packages/99/85/c091fdee33f20de70d6c8b522743b6f831a2f1cd3ff86de4c6a827c48a76/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c2aed2e5e41f24ea8ef1590b8e848a79b56f3a5564a65ceec43c9d692dc7d8a", size = 208042 }, - { url = "https://files.pythonhosted.org/packages/87/1c/ab2ce611b984d2fd5d86a5a8a19c1ae26acac6bad967da4967562c75114d/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:54523e136b8948060c0fa0bc7b1b50c32c186f2fceee897a495406bb6e311d2b", size = 228706 }, - { url = "https://files.pythonhosted.org/packages/a8/29/2b1d2cb00bf085f59d29eb773ce58ec2d325430f8c216804a0a5cd83cbca/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:715479b9a2802ecac752a3b0efa2b0b60285cf962ee38414211abdfccc233b41", size = 224727 }, - { url = "https://files.pythonhosted.org/packages/47/5c/032c2d5a07fe4d4855fea851209cca2b6f03ebeb6d4e3afdb3358386a684/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bd6c2a1c7573c64738d716488d2cdd3c00e340e4835707d8fdb8dc1a66ef164e", size = 215882 }, - { url = "https://files.pythonhosted.org/packages/2c/c2/356065d5a8b78ed04499cae5f339f091946a6a74f91e03476c33f0ab7100/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:c45e9440fb78f8ddabcf714b68f936737a121355bf59f3907f4e17721b9d1aae", size = 200860 }, - { url = "https://files.pythonhosted.org/packages/0c/cd/a32a84217ced5039f53b29f460962abb2d4420def55afabe45b1c3c7483d/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3534e7dcbdcf757da6b85a0bbf5b6868786d5982dd959b065e65481644817a18", size = 211564 }, - { url = "https://files.pythonhosted.org/packages/44/86/58e6f13ce26cc3b8f4a36b94a0f22ae2f00a72534520f4ae6857c4b81f89/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e8ac484bf18ce6975760921bb6148041faa8fef0547200386ea0b52b5d27bf7b", size = 211276 }, - { url = "https://files.pythonhosted.org/packages/8f/fe/d17c32dc72e17e155e06883efa84514ca375f8a528ba2546bee73fc4df81/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:a5fe03b42827c13cdccd08e6c0247b6a6d4b5e3cdc53fd1749f5896adcdc2356", size = 201238 }, - { url = "https://files.pythonhosted.org/packages/6a/29/f33daa50b06525a237451cdb6c69da366c381a3dadcd833fa5676bc468b3/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:2d6eb928e13016cea4f1f21d1e10c1cebd5a421bc57ddf5b1142ae3f86824fab", size = 230189 }, - { url = "https://files.pythonhosted.org/packages/b6/6e/52c84015394a6a0bdcd435210a7e944c5f94ea1055f5cc5d56c5fe368e7b/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e74327fb75de8986940def6e8dee4f127cc9752bee7355bb323cc5b2659b6d46", size = 211352 }, - { url = "https://files.pythonhosted.org/packages/8c/d7/4353be581b373033fb9198bf1da3cf8f09c1082561e8e922aa7b39bf9fe8/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d6038d37043bced98a66e68d3aa2b6a35505dc01328cd65217cefe82f25def44", size = 227024 }, - { url = "https://files.pythonhosted.org/packages/30/45/99d18aa925bd1740098ccd3060e238e21115fffbfdcb8f3ece837d0ace6c/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7579e913a5339fb8fa133f6bbcfd8e6749696206cf05acdbdca71a1b436d8e72", size = 217869 }, - { url = "https://files.pythonhosted.org/packages/5c/05/5ee478aa53f4bb7996482153d4bfe1b89e0f087f0ab6b294fcf92d595873/charset_normalizer-3.4.7-cp314-cp314-win32.whl", hash = "sha256:5b77459df20e08151cd6f8b9ef8ef1f961ef73d85c21a555c7eed5b79410ec10", size = 148541 }, - { url = "https://files.pythonhosted.org/packages/48/77/72dcb0921b2ce86420b2d79d454c7022bf5be40202a2a07906b9f2a35c97/charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl", hash = "sha256:92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f", size = 159634 }, - { url = "https://files.pythonhosted.org/packages/c6/a3/c2369911cd72f02386e4e340770f6e158c7980267da16af8f668217abaa0/charset_normalizer-3.4.7-cp314-cp314-win_arm64.whl", hash = "sha256:67f6279d125ca0046a7fd386d01b311c6363844deac3e5b069b514ba3e63c246", size = 148384 }, - { url = "https://files.pythonhosted.org/packages/94/09/7e8a7f73d24dba1f0035fbbf014d2c36828fc1bf9c88f84093e57d315935/charset_normalizer-3.4.7-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:effc3f449787117233702311a1b7d8f59cba9ced946ba727bdc329ec69028e24", size = 330133 }, - { url = "https://files.pythonhosted.org/packages/8d/da/96975ddb11f8e977f706f45cddd8540fd8242f71ecdb5d18a80723dcf62c/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79", size = 216257 }, - { url = "https://files.pythonhosted.org/packages/e5/e8/1d63bf8ef2d388e95c64b2098f45f84758f6d102a087552da1485912637b/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:733784b6d6def852c814bce5f318d25da2ee65dd4839a0718641c696e09a2960", size = 234851 }, - { url = "https://files.pythonhosted.org/packages/9b/40/e5ff04233e70da2681fa43969ad6f66ca5611d7e669be0246c4c7aaf6dc8/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a89c23ef8d2c6b27fd200a42aa4ac72786e7c60d40efdc76e6011260b6e949c4", size = 233393 }, - { url = "https://files.pythonhosted.org/packages/be/c1/06c6c49d5a5450f76899992f1ee40b41d076aee9279b49cf9974d2f313d5/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c114670c45346afedc0d947faf3c7f701051d2518b943679c8ff88befe14f8e", size = 223251 }, - { url = "https://files.pythonhosted.org/packages/2b/9f/f2ff16fb050946169e3e1f82134d107e5d4ae72647ec8a1b1446c148480f/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:a180c5e59792af262bf263b21a3c49353f25945d8d9f70628e73de370d55e1e1", size = 206609 }, - { url = "https://files.pythonhosted.org/packages/69/d5/a527c0cd8d64d2eab7459784fb4169a0ac76e5a6fc5237337982fd61347e/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3c9a494bc5ec77d43cea229c4f6db1e4d8fe7e1bbffa8b6f0f0032430ff8ab44", size = 220014 }, - { url = "https://files.pythonhosted.org/packages/7e/80/8a7b8104a3e203074dc9aa2c613d4b726c0e136bad1cc734594b02867972/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8d828b6667a32a728a1ad1d93957cdf37489c57b97ae6c4de2860fa749b8fc1e", size = 218979 }, - { url = "https://files.pythonhosted.org/packages/02/9a/b759b503d507f375b2b5c153e4d2ee0a75aa215b7f2489cf314f4541f2c0/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf1493cd8607bec4d8a7b9b004e699fcf8f9103a9284cc94962cb73d20f9d4a3", size = 209238 }, - { url = "https://files.pythonhosted.org/packages/c2/4e/0f3f5d47b86bdb79256e7290b26ac847a2832d9a4033f7eb2cd4bcf4bb5b/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0c96c3b819b5c3e9e165495db84d41914d6894d55181d2d108cc1a69bfc9cce0", size = 236110 }, - { url = "https://files.pythonhosted.org/packages/96/23/bce28734eb3ed2c91dcf93abeb8a5cf393a7b2749725030bb630e554fdd8/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:752a45dc4a6934060b3b0dab47e04edc3326575f82be64bc4fc293914566503e", size = 219824 }, - { url = "https://files.pythonhosted.org/packages/2c/6f/6e897c6984cc4d41af319b077f2f600fc8214eb2fe2d6bcb79141b882400/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:8778f0c7a52e56f75d12dae53ae320fae900a8b9b4164b981b9c5ce059cd1fcb", size = 233103 }, - { url = "https://files.pythonhosted.org/packages/76/22/ef7bd0fe480a0ae9b656189ec00744b60933f68b4f42a7bb06589f6f576a/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ce3412fbe1e31eb81ea42f4169ed94861c56e643189e1e75f0041f3fe7020abe", size = 225194 }, - { url = "https://files.pythonhosted.org/packages/c5/a7/0e0ab3e0b5bc1219bd80a6a0d4d72ca74d9250cb2382b7c699c147e06017/charset_normalizer-3.4.7-cp314-cp314t-win32.whl", hash = "sha256:c03a41a8784091e67a39648f70c5f97b5b6a37f216896d44d2cdcb82615339a0", size = 159827 }, - { url = "https://files.pythonhosted.org/packages/7a/1d/29d32e0fb40864b1f878c7f5a0b343ae676c6e2b271a2d55cc3a152391da/charset_normalizer-3.4.7-cp314-cp314t-win_amd64.whl", hash = "sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c", size = 174168 }, - { url = "https://files.pythonhosted.org/packages/de/32/d92444ad05c7a6e41fb2036749777c163baf7a0301a040cb672d6b2b1ae9/charset_normalizer-3.4.7-cp314-cp314t-win_arm64.whl", hash = "sha256:c35abb8bfff0185efac5878da64c45dafd2b37fb0383add1be155a763c1f083d", size = 153018 }, - { url = "https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d", size = 61958 }, +version = "3.4.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bd/2a/23f34ec9d04624958e137efdc394888716353190e75f25dd22c7a2c7a8aa/charset_normalizer-3.4.9.tar.gz", hash = "sha256:673611bbd43f0810bec0b0f028ddeaaa501190339cac411f347ac76917c3ae7b", size = 152439 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/e3/85ec501f206fb049259288c1f3506e53876937fb00edb47009348e66756b/charset_normalizer-3.4.9-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:0e94703ec9684807f20cfb5eed95c70f67f2a8f21ad620146d7b5a13677b93e5", size = 317075 }, + { url = "https://files.pythonhosted.org/packages/c3/69/2a5385192e67175f7d8bd5ce4f57c24bc956439adeae5c13a99aa28a53d1/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2a441ea71902098ffe78c5abe6c494f44160b4af614ed16c3d9a3b1d17fd8ee2", size = 213837 }, + { url = "https://files.pythonhosted.org/packages/b3/46/03ddc7da576d814fe0a36dd1f0fd3258e95404b4b2e3c026b7923d7e133f/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:304b13570067b2547562e308af560b3963857b1fa90bd6afd978130130fe2d6a", size = 235503 }, + { url = "https://files.pythonhosted.org/packages/4e/6e/de0229a7ef40f6f9d28a837eebf4ec47bdca5dab4e900c84f22919af636a/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4773092f8019072343a7447203308b176e10199920eb02d6195e81bbb3274c29", size = 229944 }, + { url = "https://files.pythonhosted.org/packages/a5/34/49b9060e8418b14fb5cba9cf6bfb383111e2538a03a1fb18e66a95aeb3d5/charset_normalizer-3.4.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:04ce310cb89c15df659582aee80a0603788732a5e017d5bd5c81158106ce249c", size = 221276 }, + { url = "https://files.pythonhosted.org/packages/44/95/80282cce0fae9c3061203d723ee87da996aed79679e65d8935050ee7ca1f/charset_normalizer-3.4.9-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:c0323c9daef75ef2e5083624b4585018a0c9d5e3b40f607eed81a311270b934b", size = 205260 }, + { url = "https://files.pythonhosted.org/packages/0c/74/2f62c8821b969ea3bd67cc2e6976834f48ca5d12664d2559ebcd9bcfbed7/charset_normalizer-3.4.9-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:871ff67ea1aad4dfd91736464934d56b32dac49f9fbe16cddba36198a7b3a0db", size = 217786 }, + { url = "https://files.pythonhosted.org/packages/d9/8d/feabb82cb49fcad14515b1d7d1ca4787b0da7fc723a212bf89bc9e0fac52/charset_normalizer-3.4.9-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:67830fc78e67501f47bb950471b2dcb9b35b140084429318e862895a8e89c993", size = 216798 }, + { url = "https://files.pythonhosted.org/packages/a5/ff/c946d63bc3786d5b84d960b0f7ab7e25b828486a946b5aa997625bcaf6a6/charset_normalizer-3.4.9-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:3d92613ec25e43b05f042302531ec0f00b8445190e43325880cbd6ab7c2581da", size = 206429 }, + { url = "https://files.pythonhosted.org/packages/af/ba/5e5007c370702f85d2ef75791fac7943ed41e080364a673b20142e430e3e/charset_normalizer-3.4.9-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:280081916dc341820640489a66e4696049401ef1cf6dd672f672e70ad915aca3", size = 223066 }, + { url = "https://files.pythonhosted.org/packages/83/d5/9096aa3cf532dfad237861544eb47a0f20d5adbf1039760fed8eaae935d9/charset_normalizer-3.4.9-cp311-cp311-win32.whl", hash = "sha256:ac351b3b8014eead140e77e9717e2992c6bbe30b63bc3422422eb84865412e3d", size = 150456 }, + { url = "https://files.pythonhosted.org/packages/ed/a1/e29995109e455dc8eff8d0fac6ae509be39561318a7cfeac5d33ad029213/charset_normalizer-3.4.9-cp311-cp311-win_amd64.whl", hash = "sha256:6366a16e1a25018694d6a5d784d09b046edc9eac40ea2b54065c3052672516a1", size = 161410 }, + { url = "https://files.pythonhosted.org/packages/4f/8d/1569f4d0032d6ba2a4fe4591c35bf87868c600c41a71eb5c2e1ffa8464c2/charset_normalizer-3.4.9-cp311-cp311-win_arm64.whl", hash = "sha256:1d22856ffbe153a602df38e4a5464f0b748a54002e0d69ac6d2ad0a197cc99ec", size = 152649 }, + { url = "https://files.pythonhosted.org/packages/70/4a/ecbd131485c07fcdfad54e28946d513e3da22ef3b4bd854dcafae54ec739/charset_normalizer-3.4.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:45b0cc4e3556cd875e09102988d1ab8356c998b596c9fced84547c8138b487a0", size = 319300 }, + { url = "https://files.pythonhosted.org/packages/ec/96/5d9364e3342d69f3a045e1777bc47c85c383e6e9466d561b33fdb419d1f9/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9b2aff1c7b3884512b9512c3eaadd9bab39fb45042ffaaa1dd08ff2b9f8109d9", size = 215802 }, + { url = "https://files.pythonhosted.org/packages/4b/4c/5361f9aa7f2cb58d94f2ab831b3d493f69efb1d239654b4744e3c09527cb/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9104ed0bd76a429d46f9ec0dbc9b08ad1d2dcdf2b00a5a0daa1c145329b35b44", size = 237171 }, + { url = "https://files.pythonhosted.org/packages/50/78/ce342ca4ff30b2eb49fe6d9578df85974f90c67d294113e94efdd9664cbd/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7b86a2b16095d250c6f58b3d9b2eee6f4147754344f3dab0922f7c9bf7d226c9", size = 233075 }, + { url = "https://files.pythonhosted.org/packages/01/c4/4fa4c8b3097a11f3c5f09a35b72ed6855fb1d332469504962ab7bafcc702/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e226f6218febc71f6c1fc2fafb91c226f75bdc1d8fb12d66823716e891608fd", size = 224256 }, + { url = "https://files.pythonhosted.org/packages/87/3a/ad914516df7e358a81aae018caa5e0470ba827fa6d763b1d2e87d920a5f6/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:90c44bc373b7687f6948b693cceaea1348ae0975d7474746559494468e3c1d84", size = 208784 }, + { url = "https://files.pythonhosted.org/packages/d7/74/3c12f9755717dfe5c5c87da63f35d765fa0c00382ec26bf23f7fae34f2ba/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cdef90ae47919cae358d8ab15797a800ed41da7aba5d72419fb510729e2ed4b", size = 219928 }, + { url = "https://files.pythonhosted.org/packages/33/9a/895095b83e7907abd6d3d99aad3a38ad0d9686cc186cb0c94c24320fe63e/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:60f44ade2cf573dad7a277e6f8ca9a51a21dda572b13bd7d8539bb3cd5dbedde", size = 218489 }, + { url = "https://files.pythonhosted.org/packages/a1/34/ef5c05f412f42520d7709b7d3784d19640839eb7366ded1755511585429f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:a1786910334ed46ab1dd73222f2cd1e05c2c3bb39f6dddb4f8b36fc382058a39", size = 210267 }, + { url = "https://files.pythonhosted.org/packages/83/dc/9b29fa4412b318bf3bfea985c35d67eb55e04b59a7c3f2237168b0e0be6f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:03d07803992c6c7bbc976327f34b18b6160327fc81cb82c9d504720ac0be3b62", size = 226030 }, + { url = "https://files.pythonhosted.org/packages/0e/42/6dbc00b8cd16011691203e33570fa42ed5746599a2e878112d16eab403a3/charset_normalizer-3.4.9-cp312-cp312-win32.whl", hash = "sha256:78841cccf1af7b40f6f716338d50c0902dbe88d9f800b3c973b7a9a0a693a642", size = 151185 }, + { url = "https://files.pythonhosted.org/packages/80/cc/f920afd1a23c58ccd53c1d36085a71893a4737ff5e66e0371efab6809850/charset_normalizer-3.4.9-cp312-cp312-win_amd64.whl", hash = "sha256:4b3dac63058cc36820b0dd072f89898604e2d39686fe05321729d00d8ac185a0", size = 162557 }, + { url = "https://files.pythonhosted.org/packages/f0/e6/0386d43a261ff4e4b30c5857af7df877254b46bec7b9d1b74b6bf969a90b/charset_normalizer-3.4.9-cp312-cp312-win_arm64.whl", hash = "sha256:78fa18e436a1a0e58dbd7e02fc4473f3f32cceb12df9dfca542d075961c307d2", size = 152665 }, + { url = "https://files.pythonhosted.org/packages/b2/06/97ec2aeae780b31d742b6352218b43841a6871e2564578ca522dce4a45c3/charset_normalizer-3.4.9-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:440eede837960000d74978f0eba527be106b5b9aee0daf779d395276ed0b0614", size = 317688 }, + { url = "https://files.pythonhosted.org/packages/d0/39/8ff066c672434225f8d25f8b739f992af250944392173dcc88362681c9bf/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:21e764fd1e70b6a3e205a0e46f3051701f98a8cb3fad66eeb80e48bb502f8698", size = 214982 }, + { url = "https://files.pythonhosted.org/packages/92/8f/3a47a3667c83c2df9483d91644c6c107de3bf8874aa1793da9d3012eb986/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e4fd89cc178bced6ad29cb3e6dd4aa63fa5017c3524dbd0b25998fb64a87cc8b", size = 236460 }, + { url = "https://files.pythonhosted.org/packages/f1/60/b22cdbee7e4013dab8b0d7647fc6181120fbbbc8f7025c226d15bd5a47fc/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bd47ba7fc3ca94896759ea0109775132d3e7ab921fbf54038e1bab2e46c313c9", size = 232003 }, + { url = "https://files.pythonhosted.org/packages/ea/f8/72eb13dcabe7257035cea8aefd922caad2f110d252bf9f67c4c2ca763aee/charset_normalizer-3.4.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:84fd18bcc17526fc2b3c1af7d2b9217d32c9c04448c16ec693b9b4f1985c3d33", size = 223149 }, + { url = "https://files.pythonhosted.org/packages/b0/3e/faee8f9de92b14ee1198e9163252bb15efee7301b31256a3b6d9ebfdd0dd/charset_normalizer-3.4.9-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:5b10cd92fc5c498b35a8635df6d5a100207f88b63a4dc1de7ef9a548e1e2cd63", size = 207901 }, + { url = "https://files.pythonhosted.org/packages/3a/25/45f30093ae27dd7b92a793b61882a38685f993700113ca36e0c9c14965e1/charset_normalizer-3.4.9-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a4fbdde9dd4a9ce5fd52c2b3a347bb50cc89483ef783f1cb00d408c13f7a96c0", size = 219176 }, + { url = "https://files.pythonhosted.org/packages/48/18/c8f397329c35e32f6a837e488986f4ae03bd2abebc453b48714991630c2f/charset_normalizer-3.4.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:416c229f77e5ea25b3dfd4b582f8d73d7e43c22320302b9ab128a2d3a0b38efe", size = 217356 }, + { url = "https://files.pythonhosted.org/packages/86/7e/5ce0bba863470fd1902d5e5843968951bddf38abe4742fc97116ef4598b3/charset_normalizer-3.4.9-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:75286256590a6320cf106a0d28970d3560aad9ee09aa7b34fb40524792436d35", size = 209614 }, + { url = "https://files.pythonhosted.org/packages/6c/ef/2473d3c4d869155be4af1191111d59c4d5c4e0173026f7e85b176e23bf65/charset_normalizer-3.4.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:69b157c5d3292bcd443faca052f3096f637f1e074b98212a933c074ae23dc3b8", size = 224991 }, + { url = "https://files.pythonhosted.org/packages/d0/a3/53ddae3db108a088156aa8ddfafd411ebbc1340f48c5573f697b27f69a39/charset_normalizer-3.4.9-cp313-cp313-win32.whl", hash = "sha256:51307f5c71007673a2bf8232ad973483d281e74cb99c8c5a990af1eefa6277d9", size = 150622 }, + { url = "https://files.pythonhosted.org/packages/e8/ef/6953a77c7cf2c2ff9998e6f575ab3e380119f100223381565a4f94c1f836/charset_normalizer-3.4.9-cp313-cp313-win_amd64.whl", hash = "sha256:fe2c7201c642b7c308f1675355ad7ff7b66acfe3541625efe5a3ad38f29d6115", size = 161947 }, + { url = "https://files.pythonhosted.org/packages/6e/fb/d560d1d1555debbfe7849d9cac6145c1b537709d79576bf22557ed803b82/charset_normalizer-3.4.9-cp313-cp313-win_arm64.whl", hash = "sha256:611057cc5d5c0afc743ba8be6bd828c17e0aaa8643f9d0a9b9bb7dea80eb8012", size = 152594 }, + { url = "https://files.pythonhosted.org/packages/7e/8d/496817fa0944239ecae662dd57ea765cfeaec6a735f9f025d4b7b72e7143/charset_normalizer-3.4.9-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:0327fcd59a935777d83410750c50600ee9571af2846f71ce40f25b13da1ef380", size = 317253 }, + { url = "https://files.pythonhosted.org/packages/2b/f9/ef4a69ea338ad3c0deceea0f5f7d2380ae8b52132b06d652cb0d2cd86706/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8a79d9f4d8001473a30c163556b3c3bfebec837495a412dde78b51672f6134f9", size = 215898 }, + { url = "https://files.pythonhosted.org/packages/8c/e7/5ddfd76fc061eb52de219658a4aa431cbacadf0a0219c8854f00da50d289/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:33bdcc2a32c0a0e861f60841a512c8acc658c87c2ac59d89e3a46dacf7d866e4", size = 236718 }, + { url = "https://files.pythonhosted.org/packages/49/ba/768fa3f36048d81c477a0ce61f813bc1454d80917ccfe550abd9f44f5e24/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f840ed6d8ecba8255df8c42b87fadeda98ddfc6eeec05e2dc66e26d46dd6f58a", size = 232519 }, + { url = "https://files.pythonhosted.org/packages/f4/c4/b3e049d2aa3766180c78507110543d9d50894cc97f57de543f1be521dcdc/charset_normalizer-3.4.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c25fe15c70c59eb7c5ce8c06a1f3fa1da0ecc5ea1e7a5922c40fd2fa9b0d5046", size = 223143 }, + { url = "https://files.pythonhosted.org/packages/19/79/55c32d06d76ae4feafe053f061f3e3ab70bcf19f4007797ce8c3efda7830/charset_normalizer-3.4.9-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:f7fb7d750cfa0a070d2c24e831fd3481019a60dd317ea2b39acbcebc08b6ed81", size = 206742 }, + { url = "https://files.pythonhosted.org/packages/10/e0/47c079dd82d217c807479cd59ffd30af56307ea31c108b75758970459ad3/charset_normalizer-3.4.9-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4d1c96a7a18b9690a4d46df09e3e3382406ae3213727cd1019ebade1c4a81917", size = 219191 }, + { url = "https://files.pythonhosted.org/packages/42/ab/b9bc2e77d6b44a7e46ef62ec5cac1c9a6ba7b9135a5d560f002696ec9995/charset_normalizer-3.4.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a4cfde78a9f2880208d16a93b795726a3017d5977e08d1e162a7a31322479c41", size = 218328 }, + { url = "https://files.pythonhosted.org/packages/f1/78/c9c71d599f5aa2d42bcdd35cbbd46d7f535351a57e40ff7d8e5a7e219401/charset_normalizer-3.4.9-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:d4d6fcde76f94f5cb9e43e9e9a61f16dacefd228cbbf6f1a09bd9b219a92f1a1", size = 207406 }, + { url = "https://files.pythonhosted.org/packages/f6/39/c914445c321a845097ce4f6ac7de9a18228a77b766272125a1ce00d851eb/charset_normalizer-3.4.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:898f0e9068ca27d37f8e83a5b962821df851532e6c4a7d615c1c033f9da6eedf", size = 225157 }, + { url = "https://files.pythonhosted.org/packages/9b/f2/c0d4b8508565a36bc5c624e88ed297f5b0b1095011034d7f5b83a69908b5/charset_normalizer-3.4.9-cp314-cp314-win32.whl", hash = "sha256:c1c948747b03be832dceed96ca815cef7360de9aa19d37c730f8e3f6101aca48", size = 151095 }, + { url = "https://files.pythonhosted.org/packages/49/fd/a1d26144398c67486422a72bf5812cda22cb4ccfcd95a290fb41ceb4b8e2/charset_normalizer-3.4.9-cp314-cp314-win_amd64.whl", hash = "sha256:16b65ea0f2465b6fb52aa22de5eca612aa964ddfec00a912e26f4656cbef890b", size = 162796 }, + { url = "https://files.pythonhosted.org/packages/20/95/d75e82f8ce9fd323ebf059c16c9aadefb22a1ecde13b7840b35835e4886c/charset_normalizer-3.4.9-cp314-cp314-win_arm64.whl", hash = "sha256:40a126142a56b2dfc0aacbad1de8310cbf60da7656db0e6b16eebd48e3e93519", size = 153334 }, + { url = "https://files.pythonhosted.org/packages/00/5e/17398df3a139985ba9d11ed072531986f408c8fca952835ef1ab1820c02b/charset_normalizer-3.4.9-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:609b3ba8fcc0fb5ab7af00719d0fb6ad0cb518e48e7712d12fd68f1327951198", size = 338848 }, + { url = "https://files.pythonhosted.org/packages/cd/91/7253a32e86b7e1d1239b1b36ba6dd0f021a21107ab33054b53119cc083b9/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51447e9aa2684679af07ca5021c3db526e0284347ebf4ffcec1154c3350cfe32", size = 223022 }, + { url = "https://files.pythonhosted.org/packages/cb/32/2e64bd2be10e89c61e57ebe6a93fd98ae88eb7ebe414b5121f22c96c69eb/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:cc1b0fff8ead343dae06305f954eb8468ba0ec1a97881f42489d198e4ce3c632", size = 241590 }, + { url = "https://files.pythonhosted.org/packages/3d/ef/d96ec496cfea0c21db43b0ad03891308b02388d054cc902cf0e5a1ad6a88/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fa36ec09ef71d158186bc79e359ff5fdd6e7996fe8ab638f00d6b93139ba4fcf", size = 239584 }, + { url = "https://files.pythonhosted.org/packages/d4/ce/9af95f7876194bd7a14e3dfe4a4de2e0bff02666a3910d72beafd06cc297/charset_normalizer-3.4.9-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:df115d4d83168fdf2cae48ef1ff6d1cb4c466364e30861b37121de0f3bf1b990", size = 230224 }, + { url = "https://files.pythonhosted.org/packages/52/94/af74dde74a3996bd959c350709bfe50e297823d70a8c1cbd54b838880863/charset_normalizer-3.4.9-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f86c6358749bd4fda175388691e3ba8c46e24c5347d0afd20f9b7edfc9faf07d", size = 212667 }, + { url = "https://files.pythonhosted.org/packages/ee/f0/f1c4fe746c395922961b5916ed1d7d6e7d4c84851d19ed43cc89980ec953/charset_normalizer-3.4.9-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:32286a2c8d167e897177b673176c1e3e00d4057caf5d2b64eef9a3666b03018e", size = 227179 }, + { url = "https://files.pythonhosted.org/packages/e4/56/6c745619ac397e8871e2bcd3cea1eec86b877488f33888b3aef5c3ed506e/charset_normalizer-3.4.9-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:83aed2c10721ddd90f68140685391b50811a880af20654c59af6b6c66c40513c", size = 225372 }, + { url = "https://files.pythonhosted.org/packages/78/ad/98aae8630ac71f16711968e38a5acfecce41b778bf2f0312851020f565a8/charset_normalizer-3.4.9-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cd6c3d4b783c556fa00bf540854e42f135e2f256abd29669fcd0da0f2dec79c2", size = 215222 }, + { url = "https://files.pythonhosted.org/packages/f7/40/9593d54209765207a7f11073c06494c1721e4ca4a0a426c597679bf7f91e/charset_normalizer-3.4.9-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ee2f2a527e3c1a6e6411eb4209642e138b544a2d72fe5d0d76daf77b24063534", size = 231958 }, + { url = "https://files.pythonhosted.org/packages/b1/27/693ee5e8a18191eb38647360c51cd505013e2bd3b366aa43fd5344c21e3c/charset_normalizer-3.4.9-cp314-cp314t-win32.whl", hash = "sha256:0d861473f743244d349b50f850d10eb87aeb22bbdcc8e64f79273c94af5a8226", size = 155580 }, + { url = "https://files.pythonhosted.org/packages/80/3f/bd97d3d9c613013d07cb7733d299385b41df37f0471310f5a73dc359f0b8/charset_normalizer-3.4.9-cp314-cp314t-win_amd64.whl", hash = "sha256:9b8e0f3107e2200b76f6054de99016eac3ee6762713587b36baaa7e4bd2ae177", size = 167620 }, + { url = "https://files.pythonhosted.org/packages/3d/c6/eee9dca4439b1061f76373f06ea855678cc4a64c1c3c90b50e479edbb8eb/charset_normalizer-3.4.9-cp314-cp314t-win_arm64.whl", hash = "sha256:19ac87f93086ce37b86e098888555c4b4bc48102279bae3350098c0ed664b501", size = 158037 }, + { url = "https://files.pythonhosted.org/packages/98/2b/f97f1c193fb855c345d678f5077d6926034db0722df74c8f057020e05a25/charset_normalizer-3.4.9-py3-none-any.whl", hash = "sha256:68e5f26a1ad57ded6d1cfb85331d1c1a195314756471d97758c48498bb4dcdf5", size = 64538 }, ] [[package]] @@ -368,6 +362,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335 }, ] +[[package]] +name = "courlan" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "babel" }, + { name = "tld" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/bb/16/2a771612ee0b3acaa95ac21cc7e8a3319e815d6360f8ffc5987d1ce28499/courlan-1.4.0.tar.gz", hash = "sha256:fbbac7b7fcde2195ea08e707609503c81cf39c891e8d26cdb1fed4585782d63d", size = 208997 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/38/ce65091ff20a16e06d17418c4353af5f56d3190821b1a06983c79ae79274/courlan-1.4.0-py3-none-any.whl", hash = "sha256:ad1dbdefd912ca7238d4607dc855df5df097f56bac175dd662c84eed3802f49e", size = 34193 }, +] + [[package]] name = "cryptography" version = "49.0.0" @@ -424,6 +432,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/aa/50/a9caea39ad19c431c1a3f8a31114df65b260cdfe67786b6c7e7c040c4c44/cryptography-49.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:be9fcb48a55f023493482827d4f459bd263cc20efde64f204b97c123201850c6", size = 3783731 }, ] +[[package]] +name = "dateparser" +version = "1.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "python-dateutil" }, + { name = "pytz" }, + { name = "regex" }, + { name = "tzlocal" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/59/6a/9f06999c4f27e9192c5eb38bfffadc2e6752df8178e97e88b10b9eb4c682/dateparser-1.4.2.tar.gz", hash = "sha256:bed2a3fd9bad8f2fb2d72b57748bada260b3a9349a264c22ffc23c3249d7049a", size = 338363 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/1b/349e07ad184d64e81109e85a3557d7e05631fa3d05344169114ba743c4d3/dateparser-1.4.2-py3-none-any.whl", hash = "sha256:752f3d49d477cf7f60a7a9c8bcb19c882496ede0e377d5a3d80014cdfeca7050", size = 316546 }, +] + [[package]] name = "distro" version = "1.9.0" @@ -612,6 +635,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515 }, ] +[[package]] +name = "htmldate" +version = "1.10.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "charset-normalizer" }, + { name = "dateparser" }, + { name = "lxml" }, + { name = "python-dateutil" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ad/1f/e7cf83e23d7b68105de8b874a8b36ba23b450d6f71388583e4ca3ce475ca/htmldate-1.10.0.tar.gz", hash = "sha256:a38df10772ab5d7dbb11896e3f6a852a8491fb1b0965465bc174e23fc2baae58", size = 44455 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f7/17/d3356233c826c641f940983d9479eab27faec59d49f4070bc58e80fcc021/htmldate-1.10.0-py3-none-any.whl", hash = "sha256:9211dae35ab94147c8ed9e5fc2c9287a5cf31d2394cb7857e7f5dd814eb2aad6", size = 31561 }, +] + [[package]] name = "httpcore" version = "1.0.9" @@ -784,6 +823,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437 }, ] +[[package]] +name = "justext" +version = "3.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml", extra = ["html-clean"] }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/f3/45890c1b314f0d04e19c1c83d534e611513150939a7cf039664d9ab1e649/justext-3.0.2.tar.gz", hash = "sha256:13496a450c44c4cd5b5a75a5efcd9996066d2a189794ea99a49949685a0beb05", size = 828521 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f2/ac/52f4e86d1924a7fc05af3aeb34488570eccc39b4af90530dd6acecdf16b5/justext-3.0.2-py2.py3-none-any.whl", hash = "sha256:62b1c562b15c3c6265e121cc070874243a443bfd53060e869393f09d6b6cc9a7", size = 837940 }, +] + [[package]] name = "leapflow" source = { editable = "." } @@ -791,6 +842,7 @@ dependencies = [ { name = "cryptography" }, { name = "duckdb" }, { name = "gnureadline", marker = "sys_platform == 'darwin'" }, + { name = "httpx" }, { name = "mcp" }, { name = "msgpack" }, { name = "openai" }, @@ -813,6 +865,9 @@ dev = [ hub = [ { name = "modelscope-hub" }, ] +web = [ + { name = "trafilatura" }, +] [package.metadata] requires-dist = [ @@ -820,6 +875,7 @@ requires-dist = [ { name = "cryptography", specifier = ">=42.0" }, { name = "duckdb", specifier = ">=1.0.0" }, { name = "gnureadline", marker = "sys_platform == 'darwin'", specifier = ">=8.0" }, + { name = "httpx", specifier = ">=0.27" }, { name = "mcp", specifier = ">=1.26.0" }, { name = "modelscope-hub", marker = "extra == 'hub'", specifier = ">=0.1.0" }, { name = "msgpack", specifier = ">=1.0.8" }, @@ -831,9 +887,129 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0" }, { name = "rich", specifier = ">=13.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.4.0" }, + { name = "trafilatura", marker = "extra == 'web'", specifier = ">=2.2" }, { name = "watchdog", specifier = ">=3.0" }, ] -provides-extras = ["dev", "hub", "dashboard"] +provides-extras = ["dev", "hub", "dashboard", "web"] + +[[package]] +name = "lxml" +version = "6.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/3b/aab6728cae887456f409b4d75e8a01856e4f04bd510de38052a47768b680/lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40", size = 4197430 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/b0/83f481780d1548750b8ce2ec824073deef2f452d9cd1a6faff8507e3d16d/lxml-6.1.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:53b7d2b7a10b1c35c0a5e21e9224accf60c1bbfba523990732e521b2b73adef2", size = 8526461 }, + { url = "https://files.pythonhosted.org/packages/b9/d5/30fa0f808002c7329397bfbb24e306789c0b29f04aa5842c07b174b4216f/lxml-6.1.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ff3f333630ab480244a1bff72043e511a91eb22e7595dead8653ee5612dd8f3d", size = 4595375 }, + { url = "https://files.pythonhosted.org/packages/4f/d2/edb71cf0e561581a7c5eb2626244320eb04e9f8ce6d563184fd668b45073/lxml-6.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a4bbea04c97f6d78a48e3fbc1cb9116d2780b1b39e03a23f6eb9b603fd61f510", size = 4923654 }, + { url = "https://files.pythonhosted.org/packages/4c/77/1bc7eeb0de4577d783fb625aa092cc9357883bba35845a3666bf1259f3dc/lxml-6.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:db1d75f6617a49c1c01bc7023713e0ff59ab32c9579ae62a7674c0e34f3b0b0a", size = 5067921 }, + { url = "https://files.pythonhosted.org/packages/1b/3c/c0690d74bd2bc17bc03b5b0d093569ead597dd0bfa088bf99eef8c24e19c/lxml-6.1.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a12689be69a28ddaa0ab99a5a1137da2afd5f8f16df7b5680b66f616d3eda1d", size = 5002456 }, + { url = "https://files.pythonhosted.org/packages/66/8d/d1b3271af0c0f1e27e8472a849e4d2c65bc7766884b9ad2da9e76e145c88/lxml-6.1.1-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18b73c339ae29b90fd2d06e58ebd555a751bde9cd6bbd36cc0281b9a2c94e9d8", size = 5202776 }, + { url = "https://files.pythonhosted.org/packages/7a/45/689824ffb237fd10125ad273f32b28ff04dc6203c2822c85ff65a93df65e/lxml-6.1.1-cp311-cp311-manylinux_2_28_i686.whl", hash = "sha256:752d3bbfe874715ccd0aec7f88d7fc623c0f1fd7aa7b3238a084e017bad2a009", size = 5329945 }, + { url = "https://files.pythonhosted.org/packages/5d/c0/ef73af53767e958fd87d437c170f272e2f6e6c0f854939f133a895f1e711/lxml-6.1.1-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:6b1761fbf9ec984e2e9d9c589ef5f5fd684b7c19f92aadd567a26c5224958db6", size = 4659237 }, + { url = "https://files.pythonhosted.org/packages/a0/5e/e1158e40397585e91cb0472374a1f63d0926a1ddeaa92f13d1a1ffe306d5/lxml-6.1.1-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d680fbcb768404c601ecb43519ecd8461f6954cb11c06a78962f666832ccfca8", size = 5265904 }, + { url = "https://files.pythonhosted.org/packages/a0/16/8687e5d1400ed1c0bc41dace232ebb7553952b618ea1f2e5fb6e2cfbbe23/lxml-6.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:162af1091cd785f2f27e62d3547ae9bc58ec5c86dd314d67021fd02463708d83", size = 5045225 }, + { url = "https://files.pythonhosted.org/packages/ca/18/d877bd1ae2e5ffdfd4836565aba350db31feb2f2656d6ce70316ed66a05e/lxml-6.1.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:e9308ff8241c532df3f3e570f9a5aeed6c853f888512ba4b75638d7c11c95ef6", size = 4712721 }, + { url = "https://files.pythonhosted.org/packages/44/4d/1f44fd1d770b10dacbf6b5c6e520f4d6e0708744930f719dc04e67cab981/lxml-6.1.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5f6994074ebae6ffb04447268e37dc16edc304f9859cf91acb86e0af6c1b395c", size = 5252549 }, + { url = "https://files.pythonhosted.org/packages/64/5d/1d66b84f850089254c230ef6ea6b267a5a54e2e179a5d960036a05d501d7/lxml-6.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:80c2dfadb855da477cf73373ad29a333535dedb9b12bad02c9814c8e2b43bf08", size = 5226877 }, + { url = "https://files.pythonhosted.org/packages/ad/00/84c4b5302d42a2d0184f38d538c8a197f33b52a50bd4f7bcfe990bce3036/lxml-6.1.1-cp311-cp311-win32.whl", hash = "sha256:30a89d3ac8faec007453fb541f3f46807eeec88edd5826f6e3fe001752a2c621", size = 3594072 }, + { url = "https://files.pythonhosted.org/packages/61/9d/2e2f7d876349f45e0f3e29f72da311668853d59b58d473a2dea4f0160135/lxml-6.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:abbefa31eee84842140f67acef1c828e28bba8bbf0c3bc6e5492a9af88152c28", size = 4025469 }, + { url = "https://files.pythonhosted.org/packages/b0/d5/570e6390e4110331e6208b2ba83d1482cc9146808ee118b22824a34c1070/lxml-6.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:dcb292aa7fe485ceff7af4f92e46c5af397daec5dff64871a528f0fc47a3cc5b", size = 3667640 }, + { url = "https://files.pythonhosted.org/packages/6a/6e/c4add832b6fc1e887125b96f880d7b9b70aae5248718e046b1704bcac4b9/lxml-6.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:104c09bda8d2a562824c0e319d0768ce26a779b7601e0931d33b09b53c392ef7", size = 8570821 }, + { url = "https://files.pythonhosted.org/packages/22/00/ff3009c88e65de8011630acf8ab5a09cb2becd2aaf47fba2f3449f6224e9/lxml-6.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25c6997a9a534e016695a0ba06b2f07945de682731ff01065b6d5a4474179da1", size = 4624252 }, + { url = "https://files.pythonhosted.org/packages/42/95/bb63f0fd62e554fe078e1fb3c8fe9083c14ddc7ad7fa178d10e57e071ac7/lxml-6.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c921ba5c51e4e9f63b8b00267d06566e1f63407408a0496da2d1d0bfc819c7fc", size = 4930746 }, + { url = "https://files.pythonhosted.org/packages/eb/99/0013e8d9b5960f4f041cf0b73e2f80c23eb5205b1f7bfb20203243651359/lxml-6.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:54a7f95e4de5fb94e2f9f4b9055c6ba33bf3d628fd77a1d647c5923caa2cdcdc", size = 5093723 }, + { url = "https://files.pythonhosted.org/packages/29/91/317b332636bfc7bddcff828d41b3307f50043f4b237e40849c333d80fa1a/lxml-6.1.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f2ec43df44b1f76249ee0a615334f9b5b060e1c8bd90e706dad2d14d02f383", size = 5005557 }, + { url = "https://files.pythonhosted.org/packages/42/2f/cc9bf06afe70f9c9093ae60855d9759da9db601ec4080f7473319666ffd7/lxml-6.1.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:70ef8a7e102a1508f8121aae5b0867abd663f72c14f0a9c937e6554cb4587b7b", size = 5631036 }, + { url = "https://files.pythonhosted.org/packages/08/f6/af32e23e563971ffb0fb86be52bc5be5c2c118858ffc119bf6a9039b173d/lxml-6.1.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ebe6af670449830d6d9b752c256a983291c766a1365ba5d5460048f9e33a7818", size = 5240367 }, + { url = "https://files.pythonhosted.org/packages/78/83/8555d40948b09ce86f1bd0c68a7ac31d07b1929f92cc1b074006c97ef2d2/lxml-6.1.1-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:27acc820660aaffa4f7c087f29120e12980f7779d56d8492d263170111284740", size = 5350171 }, + { url = "https://files.pythonhosted.org/packages/63/75/5d92da93729b7bad783689e6496049fa40927b45bec7bf183c981de3ca70/lxml-6.1.1-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:1db753c9115ec7100d073b744d17e25e88a8f90f5c39b2f5dd878149af59671f", size = 4694874 }, + { url = "https://files.pythonhosted.org/packages/c5/b5/3aad415a9a25b822e783f15deeb4dffccf5113030f1afa2222dd929313d9/lxml-6.1.1-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4f469aebd783bb741c2ecb2a681008fd26bfe5c16a9a72ed5467f834e810df2", size = 5244492 }, + { url = "https://files.pythonhosted.org/packages/f1/a1/5fcf7eb9904b80086aa47dcf0027de07b1bb990afad2e6823144c368ae04/lxml-6.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:766b010012d59470072c1816b5b6c69f1d243e5db36ea5968e94accf430a4635", size = 5048232 }, + { url = "https://files.pythonhosted.org/packages/77/74/1f601b63c7a69fcdf10fa9b148c81da8442204194f6c55509cc485c786b9/lxml-6.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b8d812c6011c08b8111a15e54dd990b8923692d80adf35488bee34026c35accf", size = 4777023 }, + { url = "https://files.pythonhosted.org/packages/a2/b9/7a78f51aec95b1bf780d78e12705a9f6533284f8693dc5c0e6724fa53d3f/lxml-6.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:fe0306bd29505a9177aac19f1877174b0e7422c222a59f70b2cd41633448c3dc", size = 5645773 }, + { url = "https://files.pythonhosted.org/packages/a5/6e/98a7b7ad54e4e74fa1f20fff776913980619d0ebe5558232d7da6580bdd8/lxml-6.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5ba186ad207446c65d3bb3d3e0412b032b1d9f595e59861e2354798c5703d955", size = 5233088 }, + { url = "https://files.pythonhosted.org/packages/65/d1/bc0ed2427bf609f2ee10da303a6a226f9c8bce94f945dc29a32ce55de6e4/lxml-6.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aa366a1e55b8ebfe8ca8ddc3cfe75c8ebade181aeb0f661d0cb05986b647f72a", size = 5260995 }, + { url = "https://files.pythonhosted.org/packages/69/8b/6772e1a4b513fc50a8d931f19edde0e13ae6918510a1e13ff67864f3e5ed/lxml-6.1.1-cp312-cp312-win32.whl", hash = "sha256:126c93f7f56f0eda92f6d8c619edc463a4f23d9252f1c9d0405a76f25fa9f11a", size = 3596382 }, + { url = "https://files.pythonhosted.org/packages/1b/89/45198e9624762af2dfd2cb8782598477ceb29f6e59caab560388ae1f4ec1/lxml-6.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:26e6eda8d38c1fcab1090dd196ee87cbd13788e531937610e2589085de074e77", size = 3997255 }, + { url = "https://files.pythonhosted.org/packages/90/a9/7a54b6834088d9ae528a7b780584ba6a39a9457b0ac330479f20ffbc9449/lxml-6.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:6540377fbd53fe1b629172288c464fb18db11ce1fa7dc15891da10aa9dcc3e7f", size = 3659610 }, + { url = "https://files.pythonhosted.org/packages/a5/eb/7e6f37c5584ccbb2ff267f56fd0339016938c1c8684cfefab9b33ffc2f36/lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736", size = 8559780 }, + { url = "https://files.pythonhosted.org/packages/a1/36/587c2521cf23a2cd6c9c22108aa7528f683a1f195ed7ccd23a4b1786ad36/lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9", size = 4618006 }, + { url = "https://files.pythonhosted.org/packages/6e/ca/ab7bfe2bf4c972af5e7878262845ead3a24a929a9b04bc11c7c1ece6c82a/lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354", size = 4924139 }, + { url = "https://files.pythonhosted.org/packages/6b/55/a0c72851dfee5ecc689f949723a73dea457758912542cb955b108eaf0d8f/lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca", size = 5082329 }, + { url = "https://files.pythonhosted.org/packages/f0/b6/0608f7d61a3b96cc67e5648a3d906e31a5082093e10e7be65b3886289938/lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099", size = 4993564 }, + { url = "https://files.pythonhosted.org/packages/4c/66/ae227524b066d29d55bf0b453d93d2d793c40218657d643dcbbca13b8faf/lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6", size = 5613467 }, + { url = "https://files.pythonhosted.org/packages/a6/76/dbe4a00b50385e40194231dcfe5a12c059de7cf90e89c83407d2b085b719/lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085", size = 5228304 }, + { url = "https://files.pythonhosted.org/packages/1c/01/00b1b8442ed2041793336868ba0b9ea4b13d7da7c085c6404c207a63bf79/lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e", size = 5341607 }, + { url = "https://files.pythonhosted.org/packages/63/36/1ad29931e9a4638bb707869f01d423a6c815f82152138d1a40dfcfde2b95/lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f", size = 4700168 }, + { url = "https://files.pythonhosted.org/packages/3c/d1/a9536cecf9be18a0dc72d32bead283a2332d1ffebd2dd3ac70ce444686e5/lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c", size = 5232487 }, + { url = "https://files.pythonhosted.org/packages/0e/77/b4fb1e03bf5d130e879214d3100092e386418807fb74dd0adc4b0a48f351/lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b", size = 5044231 }, + { url = "https://files.pythonhosted.org/packages/26/4c/d00daeeb0a5530c4028a9232aa1b93db3ef4ed2158c116ea73c79a9765b3/lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2", size = 4769450 }, + { url = "https://files.pythonhosted.org/packages/ed/6a/715a3a8d156ce42f29cf014706f5410c2ff3b02267774110fc23266409fe/lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5", size = 5635874 }, + { url = "https://files.pythonhosted.org/packages/45/37/0544bc21dde2a88f3a17b504e6fc79c0e01d25a33c2f6079724e9e72b9c7/lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785", size = 5223987 }, + { url = "https://files.pythonhosted.org/packages/4d/f8/f6a5e8185bcb28c2befae3d31f8e3df3b811cb0f47746517a81279fcafe1/lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947", size = 5250276 }, + { url = "https://files.pythonhosted.org/packages/c7/f2/1a2b9f1b7a49d45495369be7ef9ad05b262930f2eab3e3145706fca8083f/lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca", size = 3596903 }, + { url = "https://files.pythonhosted.org/packages/e6/99/f4ffb024f238eec2131aaa09f3278fb6129cf892741bf68e1fc1afb8c100/lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660", size = 3995869 }, + { url = "https://files.pythonhosted.org/packages/d1/53/70eb8c5c6037f27448f1e3c54ebede9545a801ae63f0a7254afca4fe8e45/lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc", size = 3658490 }, + { url = "https://files.pythonhosted.org/packages/13/e2/2e325795566de01d0d7c3bb57d3c370616b2d07b01214e84eec5d3b10963/lxml-6.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:19b7ab10b210b0b3ad7985d9ac4eb66ab09a90b20fe6e2f7ba55d01a234345d0", size = 8577146 }, + { url = "https://files.pythonhosted.org/packages/93/cf/5630b5e4be7d2e6bee8efe83865c925221103cf0221303b104ce134b01e2/lxml-6.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c08e5c694306507275f2290073350c4f32e383db15213b2c69e7ff39c1193840", size = 4623866 }, + { url = "https://files.pythonhosted.org/packages/d2/51/3904907c063451cf8d4a5c9fe0cad95fa1f4ec57f4e3884fa0731bd7a305/lxml-6.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:74a9717fd0d82effef5c2854f0d917231d5324b5a3eb7275c43ac9fa32f97a14", size = 4950022 }, + { url = "https://files.pythonhosted.org/packages/94/cd/9c7611a51c37a2830928405817cc5d56a97f64fab83cc3f628748b135749/lxml-6.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:efe0374196335f93b53269acd811b944f2e6bdc88e8894f214bd636455484909", size = 5086695 }, + { url = "https://files.pythonhosted.org/packages/da/d6/24e3b5906abb0b674ff2ae195bc3ce59708df2bcd17cf17703b2d7dd643a/lxml-6.1.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac931cdc9442c1763b8a8f6cd62c0c938737eafc5be75eff88df55fc73bc0d00", size = 5031642 }, + { url = "https://files.pythonhosted.org/packages/2d/db/6ec54f99019838bff54785c51da07f189eb4676861c5f2730962b0d8d665/lxml-6.1.1-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:aee395f5d0927f947758b4ec119fd5fc8ec71f07a1c5c52077b30b04c0fa6955", size = 5647338 }, + { url = "https://files.pythonhosted.org/packages/42/3d/ef4dcfffd22d27a61805d8ed9f7fb888495bc6aa88648fa07c1eaa5586b6/lxml-6.1.1-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9395002973c827b3ed67db77e6ec09f092919a587022174554096a269378fb13", size = 5239528 }, + { url = "https://files.pythonhosted.org/packages/62/bb/37fb3f0dff146bdcfa78eec47879273820b2a0bf350ec236ce14bd0b1c26/lxml-6.1.1-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:73bc2086f141224ebddb7fc5c6a36ca58b31b94b561e1dfe8e073e3270fad1e7", size = 5350730 }, + { url = "https://files.pythonhosted.org/packages/90/42/43253f168388df4fae1f38c01df36ddb9bee39e2048167b54cdcbae85ea3/lxml-6.1.1-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:3779def59032b81e44a5f70096ef6bf2082f8d901937dca354474ba09782e245", size = 4697530 }, + { url = "https://files.pythonhosted.org/packages/eb/a8/c5a8504f81bbdfc8e7094c2c850cdb4ed6777fc4d5ddd9e5ab819f3b0d54/lxml-6.1.1-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:86c89b9d55ebf820ad7c90bc533410f0d098054f293351f10603c0c46ff598f5", size = 5250670 }, + { url = "https://files.pythonhosted.org/packages/77/b7/c7e76ab18744d75e21f320ebf9ff9d1ceae2b54dd431ea5a64caf26c9672/lxml-6.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19607c6bbff2a44cf3fe8250abccd20942d3462473e0a721d01d379ed017e462", size = 5084485 }, + { url = "https://files.pythonhosted.org/packages/31/31/b35c53f8ef7b7c31cacd23d3638652fff7bcd1deb6eedb709ab43b685908/lxml-6.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:c6ed5141a5c7507cf3ee76bd363b0d6f801e3321adc35b5d825a23115faa5465", size = 4737635 }, + { url = "https://files.pythonhosted.org/packages/d9/06/31f23c813a7fe8e0cb1b175e915b08c9bf4e86d225b210feadbdbe519667/lxml-6.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:62aeb7e85b5d60320b9d77eef2e773994e2c0ce10121b277e0a19804e1654a5a", size = 5670681 }, + { url = "https://files.pythonhosted.org/packages/1a/bc/ce619bccc89b1fd9ad8a8e1330ee3f3beff9f2ff95b712d7bbcdd6e22fc3/lxml-6.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b1b963fd8f5caa68e99dfae060d54de1fe9cba899b8718b44a00cdca53c3e590", size = 5238229 }, + { url = "https://files.pythonhosted.org/packages/2f/5d/b329acbbedc0b619ebc2be6cf7ee9ed07e80892c88d4dfd612c33805789a/lxml-6.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:63876be28efefa04a1df615b46770e82042cce445cfdce55160522f57b231ccb", size = 5264191 }, + { url = "https://files.pythonhosted.org/packages/d6/85/be36fb1425b30db3c3f9df75fe86343ebffb79e6320bd7f588e25bfeac39/lxml-6.1.1-cp314-cp314-win32.whl", hash = "sha256:7f7a92e8583f06b1fd49d01158143b8461cfcd135dcb10ec807270a3051bd603", size = 3657202 }, + { url = "https://files.pythonhosted.org/packages/b8/ce/3cf9a827342269f54d405a6202397de63f07c69cbd6ce7d183a3f0cba1e9/lxml-6.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:b2d444f2e66624d68e9c6b211e28a76e22fff5fcabcfff4deac18b529b7d4137", size = 4064497 }, + { url = "https://files.pythonhosted.org/packages/d9/3e/1a957bde8f0760039e627f94699f82caa782c9d838d86c3d28245ee67212/lxml-6.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:3fd9728a2735fda14f4e8235830c86b539e9661e849665bf926d3f867943b4bf", size = 3741991 }, + { url = "https://files.pythonhosted.org/packages/78/b2/00ed55b3a2efa4658fb795c38d1090ec9b3e8a6c3683d4441fa517f09c3b/lxml-6.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:787b2496d0dbe8cd180984e8d29e3a6f76e7ea34db781cb3bd55e4ba1ef8b4ee", size = 8827545 }, + { url = "https://files.pythonhosted.org/packages/c0/73/74573db19baa618d5f266f2407898b087ff6927115b00b71e5fc1b700847/lxml-6.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2c8daa471358dc2d6fcf02165e80ec68f77871a286df95bc5cc3816153b0fd2c", size = 4735736 }, + { url = "https://files.pythonhosted.org/packages/16/02/6f7061f4f95f51e545d48e87647c54791d204a4e881be4156e7a26ba5338/lxml-6.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acd7d70b64c0aae0c7922cca83d288a16f5f6da523637697872253415269baef", size = 4970291 }, + { url = "https://files.pythonhosted.org/packages/b0/02/55fc057d8283427dea7d6edb102e7a840239c77a64a983d92f62a304c0e9/lxml-6.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4f0dd2f01f9f8a89f565d000e03abcf0a13d692a346c8d22f628d49af098777a", size = 5102822 }, + { url = "https://files.pythonhosted.org/packages/e4/48/8e1cf78d89d66850121d9255a2a24414c98f775da93b90cf976956c24b14/lxml-6.1.1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b7e8a14c8634bf6f7a568634cb395305a6d964aeb5b7ee32248094bed3a7e2c", size = 5027923 }, + { url = "https://files.pythonhosted.org/packages/ed/00/0632a0647612c8af24d26997b3b961397daa9d5b2581444805933629a4cb/lxml-6.1.1-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86281fbdd6a8162756f8d603f37e3435bfa38043adb79c6dc6a2dfee065e7525", size = 5595843 }, + { url = "https://files.pythonhosted.org/packages/bc/86/ab008a7dc360711b66858d61c80a5979a70a09f2aa2b05d9698df80b803d/lxml-6.1.1-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5d7152ec39ca7c402d8fb9bad86140a15b9503bd0c54484e3f1bbe3dd37ceca", size = 5224515 }, + { url = "https://files.pythonhosted.org/packages/75/c6/2702ff375e728e34f56d9a45339a9cf7e4427e917f542225242d63a05afa/lxml-6.1.1-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:88d8cb75b9d82858497a5393e3c63cfbf03035225e4b35a49ed7ccb151e4dc0e", size = 5312511 }, + { url = "https://files.pythonhosted.org/packages/b7/57/a5807c98f87a86f10ef9ffab35516df7c0f0c4b6d5d33e9f608ab9c04a31/lxml-6.1.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f64ec5397ea6a41fc1b4af0380d79b44a755b5531dcaccd9940fb260dca93038", size = 4639206 }, + { url = "https://files.pythonhosted.org/packages/1f/e1/8a0a2c35734812395f4da4eaf33748a7e5705bfb2a58b128da764339d5ec/lxml-6.1.1-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d34bbf07dbc7ca5970671b1512e928991fb5e9d95365636c9b2d8b4f53af405e", size = 5232404 }, + { url = "https://files.pythonhosted.org/packages/c2/e2/0e6a4dd5ad84d01d99aa7bae7cfefd4a760a0e0f8176818241de17d9b6c0/lxml-6.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:17e0e18d4ad8adbd0399291bc44845b69d9dd68439a3cdebdf35ff902ec05072", size = 5083769 }, + { url = "https://files.pythonhosted.org/packages/a0/7e/161f33d463f6ffc1c7679104b65086dea120080d49dde4d238f015aaee2f/lxml-6.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:3ab541146f1f6968c462d6c2ac495148e8cdba2f8347700b2141b6ec5a75bf52", size = 4758936 }, + { url = "https://files.pythonhosted.org/packages/f1/fb/2369825e3f6ca99305bf9f7b7085fda91c8b0922a89e54d900974aa3ef85/lxml-6.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2a0217714657e023ef4293500f65aa20fce6164c8fd6b08fa5bd4a859fb14b9b", size = 5620296 }, + { url = "https://files.pythonhosted.org/packages/30/90/d61e383146f74c5ab683947ea14dc7b82778838ab9b95ea73a23b60d0191/lxml-6.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:05a82eb6e1530a64f26225b55cbd178113bd0b5af1c2b625f25e5296742c26d2", size = 5228598 }, + { url = "https://files.pythonhosted.org/packages/76/2d/2dafd8149e94b05bb070690efd5bb2680720681e03ff03fc57d2b70a1105/lxml-6.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9e36f163528fc50cbef305f02a5fd66d404edf7049cdaff211dbc2cba5a7013e", size = 5247845 }, + { url = "https://files.pythonhosted.org/packages/ce/68/b30e913340c380ddac9580c6e6230991fc37240ec4f64704833e4f3e2769/lxml-6.1.1-cp314-cp314t-win32.whl", hash = "sha256:649dda677cf3bd6ac9ae14007ba0c824ded8ce5808b53fc7431d9140399118c1", size = 3897345 }, + { url = "https://files.pythonhosted.org/packages/3c/4e/9eb2af5335545f9fbcd7af57bcf87c6025d31eaa31b14ec184a6c8675328/lxml-6.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:793033d6c5cdf33a573f910d9bea14ef8f5771820411d118da8e1182edb53d5e", size = 4393350 }, + { url = "https://files.pythonhosted.org/packages/7f/2c/0f1e93c636720e8a3eb59af2bfda99d98b55891e1c53bc30c2e0e865f01b/lxml-6.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:58bb955caba94e467d2a96da17660d2d704e0675894cba21ab8a775b8621fd1c", size = 3817223 }, + { url = "https://files.pythonhosted.org/packages/b5/32/86a3f0f724a3a402d4627937a7fc27b160e45e7012b4adf47f6e1e844511/lxml-6.1.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:31033dc34636ea6b7d5cc11b1ddbda78a14de858ba9d3e1ed4b69a3085bc521e", size = 3930127 }, + { url = "https://files.pythonhosted.org/packages/40/44/d832e82af08723761556d004b1d04d281c09f9a8cecd7d3148548c9941a3/lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3893c14c4b6ac5b2d54ba8cf03e99fe5104e592de491f19bd6b82756c09f8004", size = 4210769 }, + { url = "https://files.pythonhosted.org/packages/6d/39/0dc5949f759ed7d951e0bb8c2f2d9d7aca1908d22352fa84a8afd2ea54af/lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c07da4cebf6889f03ebac8d238f62318e29f495de0aa18a51ea14e61ae907e2e", size = 4318163 }, + { url = "https://files.pythonhosted.org/packages/e6/fb/8ab3845fe046ba4cbf74536bcf6801a774b7caf4350de1c5d37f1f0a9e90/lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6f0ce10945fab9c4c06ce14e22af9059d1a87493a9af4501a5b0b9187e21cf2", size = 4250945 }, + { url = "https://files.pythonhosted.org/packages/68/1b/7553ab136894374ffae8851ec06f98f511cd8e66246e41b6be059d0a7289/lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8844cd288697c6425c9beba919302241e3278871dc6519515e72b04e987abcf", size = 4401664 }, + { url = "https://files.pythonhosted.org/packages/db/a4/441aee36c6f6b249823d20fd91f9be9ab89d7c5a8ae542a4a4ca6d342d56/lxml-6.1.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:ed21202aec73cda4d55d1ce57b389aadb90ffb044e6cd1080b8347efe1b1ec84", size = 3508989 }, +] + +[package.optional-dependencies] +html-clean = [ + { name = "lxml-html-clean" }, +] + +[[package]] +name = "lxml-html-clean" +version = "0.4.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0a/63/195dfdde380a84df309e3bccf4384b034b745dba43426886f7ae623b4fba/lxml_html_clean-0.4.5.tar.gz", hash = "sha256:e2a4c7d5beedd17cd7b484d848a0571e54baa239a4f9df5546e3acba7f990560", size = 24142 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/bd/6e2b76a6c5dee10397db9c929f0c5066766ec1036046f0335b7ca7ca08b8/lxml_html_clean-0.4.5-py3-none-any.whl", hash = "sha256:c76fcadd1e5bfb9b8bafc2200d51e4e78eb0dad67f56881c21dfb6484c7e7746", size = 14573 }, +] [[package]] name = "markdown-it-py" @@ -1505,6 +1681,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e5/35/f8b19922b6a25bc0880171a2f1a003eaeb93657475193ab516fd87cac9da/pytest_asyncio-1.3.0-py3-none-any.whl", hash = "sha256:611e26147c7f77640e6d0a92a38ed17c3e9848063698d5c93d5aa7aa11cebff5", size = 15075 }, ] +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892 }, +] + [[package]] name = "python-dotenv" version = "1.2.2" @@ -1523,6 +1711,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e1/04/e8135ebd1ad02c56ec633277529b2602ff99ff634be76cdba5744cf554fd/python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23", size = 30042 }, ] +[[package]] +name = "pytz" +version = "2026.3.post1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fb/48/fb042503b6ca6cd271261dc559fd6432f7d8c713153e9ec5c591af4dfc1c/pytz-2026.3.post1.tar.gz", hash = "sha256:2211d3fcf9a797d3405cac96ac7f61d80e6a644f72a3309607282fe8a2010c5d", size = 319745 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0f/7b/39c34ca613b0b198cb866466651b26b045e2009864c5183c979a3b83f383/pytz-2026.3.post1-py2.py3-none-any.whl", hash = "sha256:dd95840dd199baea12d9cc096a1d452caa6596a1c1e4b5f3dbd1541855d5e815", size = 508283 }, +] + [[package]] name = "pywin32" version = "312" @@ -1614,6 +1811,110 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766 }, ] +[[package]] +name = "regex" +version = "2026.7.19" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/20/98/04b13f1ddfb63158025291c02e03eb42fbb7acb51d091d541050eb4e35e8/regex-2026.7.19.tar.gz", hash = "sha256:7e77b324909c1617cbb4c668677e2c6ae13f44d7c1de0d4f15f2e3c10f3315b5", size = 416440 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/e5/cef4de2bac939280b68d32adc659478845238a8274f2f79c465063f590ad/regex-2026.7.19-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ac777001cdfc28b72477d93c8564bb7583081ea8fb45cdca3d568e0a4f87183c", size = 494012 }, + { url = "https://files.pythonhosted.org/packages/ff/87/e86f51eb117457bb7803132ffe5cb6e2841e2b5bea4cc85d397f3c6e257d/regex-2026.7.19-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:59787bd5f8c70aa339084e961d2996b53fbdeab4d5393bba5c1fe1fc32e02bae", size = 295281 }, + { url = "https://files.pythonhosted.org/packages/41/2e/2360c41d8080a3d9ec7e5c90fad6eab3b50192869d10e9a5609e48c8177b/regex-2026.7.19-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:90c633e7e8d6bf4e992b8b36ce69e018f834b641dd6de8cea6d78c06ffa119c5", size = 290615 }, + { url = "https://files.pythonhosted.org/packages/cf/69/b65ba4344efbc771b28fe5dde84cbbb6c8f9551165952fe78def5b9dde6a/regex-2026.7.19-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:87ccab0db8d5f4fbb0272642113c1adb2ffc698c16d3a0944580222331fa7a20", size = 791804 }, + { url = "https://files.pythonhosted.org/packages/81/b6/a40dfa0dc6224b36f620c00296eacc830489cbf8c2837b6750dfe6170375/regex-2026.7.19-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9e50d748a32da622f256e8d505867f5d3c43a837c6a9f0efb149655fadd1042a", size = 861723 }, + { url = "https://files.pythonhosted.org/packages/e3/02/735991dee71abd83196a7962f7ed8bf5aa05720ff06e2d3ff896a85e2bbb/regex-2026.7.19-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bf1516fe58fc104f39b2d1dbe2d5e27d0cd45c4be2e42ba6ee0cc763701ec3c7", size = 905932 }, + { url = "https://files.pythonhosted.org/packages/45/6c/e7098d8b846ccdbf431d8c081b61e496526a27a28094ed09e0dce21b3f54/regex-2026.7.19-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:09f3e5287f94f17b709dc9a9e70865855feee835c861613be144218ce4ca82cc", size = 801407 }, + { url = "https://files.pythonhosted.org/packages/8a/18/34b69274e2649bcc7d9b089c2b2983fb2632d8ecf667e359593be9072e79/regex-2026.7.19-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6383cd2ed53a646c659ba1fe65727db76437fdaa069e697a0b44a51d5843d864", size = 774448 }, + { url = "https://files.pythonhosted.org/packages/bb/e6/0a72247d025585fd3800b98e040b84d562a88af6303347100484849f4f01/regex-2026.7.19-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:09d3007fc76249a83cdd33de160d50e6cb77f54e09d8fa9e7148e10607ce24af", size = 783297 }, + { url = "https://files.pythonhosted.org/packages/b1/aa/c4f65ae7dd02a36b323a70c4cff326e1f3442361aaebc9311100a130d54f/regex-2026.7.19-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:6f8c6e7a1cfa3dc9d0ee2de0e65e834537fa29992cc3976ffec914afc35c5dd5", size = 854736 }, + { url = "https://files.pythonhosted.org/packages/62/c3/668082bcc817b9e694189b84997aeba7385b7779faa6711788679c482e35/regex-2026.7.19-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:b2ea4a3e8357be8849e833beeae757ac3c7a6b3fc055c03c808a53c91ad30d82", size = 763298 }, + { url = "https://files.pythonhosted.org/packages/4b/fb/2d07ad555e7af88aa5f867fdafa47a8d945ee237c20af3ebceb46a820835/regex-2026.7.19-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:80115dd39481fd3a4b4080220799dbcacb921a844de4b827264ececacbe17c78", size = 844430 }, + { url = "https://files.pythonhosted.org/packages/51/15/c82a471fe3dce56f03745635b43aa456c40dc0db089e07ef148b331507d1/regex-2026.7.19-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d6ce43a0269d68cee79a7d1ade7def53c20f8f2a047b92d7b5d5bcc73ae88327", size = 789683 }, + { url = "https://files.pythonhosted.org/packages/b5/f4/7532a2c59d56f5398902c20de60f0c9a5d1cd364e42a051b48e1b210be7b/regex-2026.7.19-cp311-cp311-win32.whl", hash = "sha256:9be2a6647740dd3cca6acb24e87f03d7632cd280dbce9bbe40c26353a215a45d", size = 266778 }, + { url = "https://files.pythonhosted.org/packages/83/2b/cf1bc631db154eb95520d9d5dbc2371ff77a0f014bbf7d748fed8496aa63/regex-2026.7.19-cp311-cp311-win_amd64.whl", hash = "sha256:8d3469c91dd92ee41b7c95280edbd975ef1ba9195086686623a1c6e8935ce965", size = 277983 }, + { url = "https://files.pythonhosted.org/packages/8d/bd/56ceaf170e875d5a6761bf2bfd0d040f1cacc896850d5e40cb29b11bbd06/regex-2026.7.19-cp311-cp311-win_arm64.whl", hash = "sha256:36aacfb15faaff3ced55afbf35ec72f50d4aee22082c4f7fe0573a33e2fca92e", size = 276961 }, + { url = "https://files.pythonhosted.org/packages/3b/b9/d11d7e501ac8fd7d617684423ebb9561e0b998481c1e4cbc0cb212c5d74a/regex-2026.7.19-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:2cc3460cedf7579948486eab03bc9ad7089df4d7281c0f47f4afe03e8d13f02d", size = 496778 }, + { url = "https://files.pythonhosted.org/packages/3f/a9/a5ab6f312f24318019170dc485d5421fe4f89e43a98640da50d95a8a7041/regex-2026.7.19-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0e9554c8785eac5cffe6300f69a91f58ba72bc88a5f8d661235ad7c6aa5b8ccd", size = 297122 }, + { url = "https://files.pythonhosted.org/packages/b3/63/4cab4d7f2d384a144d420b763d97674cb70619c878ea6fcd7640d0e62143/regex-2026.7.19-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d7da47a0f248977f08e2cb659ff3c17ddc13a4d39b3a7baa0a81bf5b415430f6", size = 292009 }, + { url = "https://files.pythonhosted.org/packages/22/85/102a81b218298957d4ea7d2f084fae537a71add9d6ff93c8e67284c5f45e/regex-2026.7.19-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93db40c8de0815baab96a06e08a984bac71f989d13bab789e382158c5d426797", size = 796708 }, + { url = "https://files.pythonhosted.org/packages/78/b5/dc136af5629938a037cd2b304c12240e132ec92f38be8ff9cc89af2a1f2d/regex-2026.7.19-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:66bd62c59a5427746e8c44becae1d9b99d22fb13f30f492083dfb9ad7c45cc18", size = 865651 }, + { url = "https://files.pythonhosted.org/packages/e0/75/67402ae3cd9c8c988a4c805d15ee3eef015e7ca4cb112cf3e640fc1f4153/regex-2026.7.19-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1649eb39fcc9ea80c4d2f110fde2b8ab2aef3877b98f02ab9b14e961f418c511", size = 911756 }, + { url = "https://files.pythonhosted.org/packages/2a/8e/096d00c7c480ef2ff4265349b14e2261d4ab787ba1f74e2e80d1c58079c3/regex-2026.7.19-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9dce8ec9695f531a1b8a6f314fd4b393adcccf2ea861db480cdf97a301d01a68", size = 801798 }, + { url = "https://files.pythonhosted.org/packages/f0/41/e7ecac6edb5722417f85cc67eaf386322fbe8acf6918ec2fdc37c20dd9d0/regex-2026.7.19-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3080a7fd38ef049bd489e01c970c97dd84ff446a885b0f1f6b26d9b1ad13ce11", size = 776933 }, + { url = "https://files.pythonhosted.org/packages/6f/69/03c9b3f058d66403e0ca2c938696e81d51cd4c6d47ec5265f02f96948d9a/regex-2026.7.19-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1d793a7988e04fcb1e2e135567443d82173225d657419ec09414a9b5a145b986", size = 784338 }, + { url = "https://files.pythonhosted.org/packages/f6/f7/b38ab3d43f284afbb618fcd15d0e77eb786ae461ce1f6bc7494619ddc0f2/regex-2026.7.19-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e8b0abe7d870f53ca5143895fef7d1041a0c831a140d3dc2c760dd7ba25d4a8b", size = 860452 }, + { url = "https://files.pythonhosted.org/packages/15/5c/ff60ef0571121714f3cf9920bc183071e384a10b556d042e0fdb06cc07a5/regex-2026.7.19-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:4e5413bd5f13d3a4e3539ca98f70f75e7fca92518dd7f117f030ebedd10b60cb", size = 765958 }, + { url = "https://files.pythonhosted.org/packages/aa/0f/bd34021162c0ab47f9a315bd56cd5642e920c8e5668a75ef6c6a6fca590d/regex-2026.7.19-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:73b133a9e6fb512858e7f065e96f1180aa46646bc74a83aea62f1d314f3dd035", size = 851765 }, + { url = "https://files.pythonhosted.org/packages/2a/20/a2ca43edade0595cccfdc98636739f536d9e26898e7dbddc2b9e98898953/regex-2026.7.19-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:dbe6493fbd27321b1d1f2dd4f5c7e5bd4d8b1d7cab7f32fd67db3d0b2ed8248a", size = 789714 }, + { url = "https://files.pythonhosted.org/packages/5d/47/e02db4015d424fc83c00ea0ac8c5e5ec14397943de9abf909d5ce3a25931/regex-2026.7.19-cp312-cp312-win32.whl", hash = "sha256:ddd67571c10869f65a5d7dde536d1e066e306cc90de57d7de4d5f34802428bb5", size = 267157 }, + { url = "https://files.pythonhosted.org/packages/08/8e/c780c131f79b42ed22d1bd7da4096c2c35f813e835acd02ef0f018bd892c/regex-2026.7.19-cp312-cp312-win_amd64.whl", hash = "sha256:e30d40268a28d54ce0437031750497004c22602b8e3ab891f759b795a003b312", size = 277777 }, + { url = "https://files.pythonhosted.org/packages/3e/4c/e4d7e086449bdf379d89774bf1f89dc4a41943f3c5a6125a03905b34b5fb/regex-2026.7.19-cp312-cp312-win_arm64.whl", hash = "sha256:de9208bb427130c82a5dbfd104f92c8876fc9559278c880b3002755bbbe9c83d", size = 277136 }, + { url = "https://files.pythonhosted.org/packages/5d/3d/84165e4299ff76f3a40fe1f2abf939e976f693383a08d2beea6af62bd2c1/regex-2026.7.19-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f035d9dc1d25eff9d361456572231c7d27b5ccd473ca7dc0adfce732bd006d40", size = 496552 }, + { url = "https://files.pythonhosted.org/packages/02/a2/a65293e6e4cf28eb7ee1be5335a5386c40d6742e9f47fafc8fec785e16c7/regex-2026.7.19-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c42572142ed0b9d5d261ba727157c426510da78e20828b66bbb855098b8a4e38", size = 296983 }, + { url = "https://files.pythonhosted.org/packages/95/47/2d0564e93d87bc48618360ddca232a2ca612bbdf53ce8465d45ca5ce14ee/regex-2026.7.19-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:40b34dd88658e4fedd2fddbf0275ac970d00614b731357f425722a3ed1983d11", size = 291832 }, + { url = "https://files.pythonhosted.org/packages/07/cd/42dfbabff3dfc9603c501c0e2e2c5adbb09d127b267bf5348de0af338c15/regex-2026.7.19-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c41c63992bf1874cebb6e7f56fd7d3c007924659a604ae3d90e427d40d4fd13", size = 796775 }, + { url = "https://files.pythonhosted.org/packages/df/5d/f6a4839f2b934e3eed5973fd07f5929ee97d4c98939fb275ea23c274ee16/regex-2026.7.19-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d3372064506b94dd2c67c845f2db8062e9e9ba84d04e33cb96d7d33c11fe1ae", size = 865687 }, + { url = "https://files.pythonhosted.org/packages/14/b0/b47d6c36049bc59806a50bd4c86ced70bbe058d787f80281b1d7a9b0e024/regex-2026.7.19-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fce7760bf283405b2c7999cab3da4e72f7deca6396013115e3f7a955db9760da", size = 911962 }, + { url = "https://files.pythonhosted.org/packages/2a/be/ff61f28f9273658cfe23acbbac5217221f6519960ed401e61dfdab12bc35/regex-2026.7.19-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0d702548d89d572b2929879bc883bb7a4c4709efafe4512cadee56c55c9bd15", size = 801817 }, + { url = "https://files.pythonhosted.org/packages/c3/bb/8b4f7f26b333f9f79e1b453613c39bb4776f51d38ae66dd0ba31d6b354ca/regex-2026.7.19-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d446c6ac40bb6e05025ccee55b84d80fe9bf8e93010ffc4bb9484f13d498835f", size = 776908 }, + { url = "https://files.pythonhosted.org/packages/09/13/610110fc5921d380516d03c26b652555f08aa0d23ea78a771231873c3638/regex-2026.7.19-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4c3501bfa814ab07b5580741f9bf78dfdfe146a04057f82df9e2402d2a975939", size = 784426 }, + { url = "https://files.pythonhosted.org/packages/ca/f5/1ef9e2a83a5947c57ebff0b377cb5727c3d5ec1992317a320d035cd0dbb6/regex-2026.7.19-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:c4585c3e64b4f9e583b4d2683f18f5d5d872b3d71dcf24594b74ecc23602fa96", size = 860600 }, + { url = "https://files.pythonhosted.org/packages/a0/02/073af33a3ec149241d11c80acea91e722aa0adbf05addd50f251c4fe89c3/regex-2026.7.19-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:571fde9741eb0ccde23dd4e0c1d50fbae910e901fa7e629faf39b2dda740d220", size = 765950 }, + { url = "https://files.pythonhosted.org/packages/81/a9/d1e9f819dc394a568ef370cd56cf25394e957a2235f8370f23b576e5a475/regex-2026.7.19-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:15b364b9b98d6d2fe1a85034c23a3180ff913f46caddc3895f6fd65186255ccc", size = 851794 }, + { url = "https://files.pythonhosted.org/packages/03/3a/8ae83eda7579feacdf984e71fb9e70635fb6f832eeddca58427ec4fca926/regex-2026.7.19-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ffd8893ccc1c2fce6e0d6ca402d716fe1b29db70c7132609a05955e31b2aa8f2", size = 789845 }, + { url = "https://files.pythonhosted.org/packages/4b/23/c195cbfe5a75fdec64d8f6554fd15237b837919d2c61bdc141d7c807b08b/regex-2026.7.19-cp313-cp313-win32.whl", hash = "sha256:f0fa4fa9c3632d708742baf2282f2055c11d888a790362670a403cbf48a2c404", size = 267135 }, + { url = "https://files.pythonhosted.org/packages/b2/80/a11de8404b7272b70acb45c1c05987cce60b45d5693da2e176f0e390d564/regex-2026.7.19-cp313-cp313-win_amd64.whl", hash = "sha256:d51ffd3427640fa2da6ade574ceba932f210ad095f65fcc450a2b0a0d454868e", size = 277747 }, + { url = "https://files.pythonhosted.org/packages/d1/29/0f5c8eff1b4f1f3d83276d365fccecf666afcc7d947420943bf394d07adb/regex-2026.7.19-cp313-cp313-win_arm64.whl", hash = "sha256:c670fe7be5b6020b76bc6e8d2196074657e1327595bca93a389e1a76ab130ad8", size = 277129 }, + { url = "https://files.pythonhosted.org/packages/dc/4c/44b74742052cedda40f9ae469532a037112f7311a36669a891fba8984bb0/regex-2026.7.19-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:db47b561c9afd884baa1f96f797c9ca369872c4b65912bc691cfa99e68340af2", size = 501134 }, + { url = "https://files.pythonhosted.org/packages/f0/45/bbd038b5e39ee5613a5a689290145b40058cc152c41de9cc23639d2b9734/regex-2026.7.19-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:65dcd28d3eba2ab7c2fd906485cc301392b47cc2234790d27d4e4814e02cdfda", size = 299418 }, + { url = "https://files.pythonhosted.org/packages/65/38/c5bde94b4cedfd5850d64c3f08222d8e1600e84f6ee71d9b44b4b8163f74/regex-2026.7.19-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:f2e7f8e2ab6c2922be02c7ec45185aa5bd771e2e57b95455ee343a44d8130dff", size = 294486 }, + { url = "https://files.pythonhosted.org/packages/d7/6a/2f5e107cb26c960b781967178899daf2787a7ab151844ed3c01d6fc95474/regex-2026.7.19-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fe31f28c94402043161876a258a9c6f757cb485905c7614ce8d6cd40e6b7bdc1", size = 811643 }, + { url = "https://files.pythonhosted.org/packages/37/d4/a2f963406d7d73a62eed84ba05a258afb6cad1b21aa4517443ce40506b78/regex-2026.7.19-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f8f6fa298bb4f7f58a33334406218ba74716e68feddf5e4e54cd5d8082705abf", size = 871081 }, + { url = "https://files.pythonhosted.org/packages/45/a3/44be546340bedb15f13063f5e7fe16793ea4d9ea2e805d09bd174ac27724/regex-2026.7.19-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cc1b2440423a851fad781309dd87843868f4f66a6bcd1ddb9225cf4ec2c84732", size = 917372 }, + { url = "https://files.pythonhosted.org/packages/f8/f6/e0870b0fd2a40dba0074e4b76e514b21313d37946c9248453e34ec43923e/regex-2026.7.19-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ac59a0900474a52b7c04af8196affc22bd9842acb0950df12f7b813e983609a", size = 816089 }, + { url = "https://files.pythonhosted.org/packages/ae/27/957e8e22690ad6634572b39b71f130a6105f4d0718bb16849eac00fff147/regex-2026.7.19-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4896db1f4ce0576765b8272aa922df324e0f5b9bb2c3d03044ff32a7234a9aba", size = 785206 }, + { url = "https://files.pythonhosted.org/packages/76/a4/186e410941e731037c01166069ab86da9f65e8f8110c18009ccf4bd623ee/regex-2026.7.19-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4e6883a021db30511d9fb8cfb0f222ce1f2c369f7d4d8b0448f449a93ba0bdfc", size = 800431 }, + { url = "https://files.pythonhosted.org/packages/73/9f/e4e10e023d291d64a33e246610b724493bf1ce98e0e59c9b7c837e5acfb7/regex-2026.7.19-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:09523a592938aa9f587fb74467c63ff0cf88fc3df14c82ab0f0517dcf76aaa62", size = 864906 }, + { url = "https://files.pythonhosted.org/packages/24/57/ccb20b6be5f1f52a053d1ba2a8f7a077edb9d918248b8490d7506c6832b3/regex-2026.7.19-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:1ebac3474b8589fce2f9b225b650afd61448f7c73a5d0255a10cc6366471aed1", size = 773559 }, + { url = "https://files.pythonhosted.org/packages/a3/82/f3b263cf8fad927dc102891da8502e718b7ff9d19af7a2a07c03865d7188/regex-2026.7.19-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:4a0530bb1b8c1c985e7e2122e2b4d3aedd8a3c21c6bfddae6767c4405668b56e", size = 857739 }, + { url = "https://files.pythonhosted.org/packages/47/2e/1687bd1b6c2aed5e672ccf845fc11557821fe7366d921b50889ea5ce57bf/regex-2026.7.19-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2ef7eeb108c47ce7bcc9513e51bcb1bf57e8f483d52fce68a8642e3527141ae0", size = 804522 }, + { url = "https://files.pythonhosted.org/packages/76/7c/cc4e7655181b2d9235b704f2c5e19d8eff002bbc437bae59baee0e381aca/regex-2026.7.19-cp313-cp313t-win32.whl", hash = "sha256:64b6ca7391a1395c2638dd5c7456d67bea44fc6c5e8e92c5dc8aa6a8f23292b4", size = 269141 }, + { url = "https://files.pythonhosted.org/packages/bb/14/961b4c7b05a2391c32dbc85e27773076671ef8f97f36cec70fe414734c02/regex-2026.7.19-cp313-cp313t-win_amd64.whl", hash = "sha256:f04b9f56b0e0614c0126be12c2c2d9f8850c1e57af302bd0a63bed379d4af974", size = 280036 }, + { url = "https://files.pythonhosted.org/packages/ce/67/795644550d788ddbb6dc458c95895f8009978ea6d6ea76b005eb3f45e8c9/regex-2026.7.19-cp313-cp313t-win_arm64.whl", hash = "sha256:fcee38cd8e5089d6d4f048ba1233b3ad76e5954f545382180889112ff5cb712d", size = 279394 }, + { url = "https://files.pythonhosted.org/packages/d2/25/0c4c452f8ef3efe456745b2f33195f5904b573fb4c2ff3f0cb9ec188461e/regex-2026.7.19-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:a81758ed242b861b72e778ba34d41366441a2e10b16b472784c88da2dea7e2dd", size = 496750 }, + { url = "https://files.pythonhosted.org/packages/24/9e/b70ca6c1704f6c7cd32a9e143c86cc5968d10981eca284bad670c245ea7d/regex-2026.7.19-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4aa5435cdb3eb6f55fe98a171b05e3fbcd95fadaa4aa32acf62afd9b0cfdbcac", size = 297093 }, + { url = "https://files.pythonhosted.org/packages/87/74/0b692da2520d51fbff19c88b83d97e4c702909dd02386c585998b7e2dbed/regex-2026.7.19-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:60be8693a1dadc210bbcbc0db3e26da5f7d01d1d5a3da594e99b4fa42df404f5", size = 292043 }, + { url = "https://files.pythonhosted.org/packages/e3/a7/1d478e614016045a33feae57446215f9fd65b665a5ceb2f891fb3183bc52/regex-2026.7.19-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d19662dbedbe783d323196312d38f5ba53cf56296378252171985da6899887d3", size = 797214 }, + { url = "https://files.pythonhosted.org/packages/aa/ae/11b9c9411d92c30e3d2db32df5a31133e4a99a8fc397a604fd08f6c4bffb/regex-2026.7.19-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d15df07081d91b76ff20d43f94592ee110330152d617b730fdbe5ef9fb680053", size = 866433 }, + { url = "https://files.pythonhosted.org/packages/b1/62/2b2efc4992f91d6d204b24c647c9f9412e85379d92b7c0ab9fdae622327e/regex-2026.7.19-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:56ad4d9f77df871a99e25c37091052a02528ec0eb059de928ee33956b854b45b", size = 911360 }, + { url = "https://files.pythonhosted.org/packages/14/71/986ceea9aa3da548bf1357cad89b63915ec6d21ec957c8113b29ece567df/regex-2026.7.19-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7322ec6cc9fba9d49ab888bb82d67ac5625627aa168f0165139b17018df3fb8a", size = 801275 }, + { url = "https://files.pythonhosted.org/packages/15/be/ce9d9534b2cda96eab32c548261224b9b4e220a4126f098f60f42ae7b4cd/regex-2026.7.19-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9c7472192ebfad53a6be7c4a8bfb2d64b81c0e93a1fc8c57e1dd0b638297b5d1", size = 777131 }, + { url = "https://files.pythonhosted.org/packages/61/2b/58b5c710f2c3929515a25f3a1ca0dad0dcd4518d4fff3cf23bc7adb8dcd2/regex-2026.7.19-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c10b82c2634df08dfb13b1f04e38fe310d086ee092f4f69c0c8da234251e556e", size = 785020 }, + { url = "https://files.pythonhosted.org/packages/84/03/5fe091935b74f15fe0f97998c215cae418d1c0413f6258c7d4d2e83aa37f/regex-2026.7.19-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:17ed5692f6acc4183e98331101a5f9e4f64d72fe58b753da4d444a2c77d05b12", size = 861263 }, + { url = "https://files.pythonhosted.org/packages/d8/fa/d60bf82e10841eef62a9e32aac401468f05fddfbcb2942e342b1ba3d2433/regex-2026.7.19-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:22a992de9a0d91bda927bf02b94351d737a0302905432c88a53de7c4b9ce62e2", size = 766199 }, + { url = "https://files.pythonhosted.org/packages/bf/5d/11e64d151b0662b81d6bf644c74dc118d461df85bdf2577fadbbf751788a/regex-2026.7.19-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:618a0aed532be87294c4477b0481f3aa0f1520f4014a4374dd4cf789b4cd2c97", size = 851317 }, + { url = "https://files.pythonhosted.org/packages/7c/34/532efb87488d90807bae6a443d357ee5e2728a478c597619c8aaa17cc0bd/regex-2026.7.19-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2ce9e679f776649746729b6c86382da519ef649c8e34cc41df0d2e5e0f6c36d4", size = 789557 }, + { url = "https://files.pythonhosted.org/packages/d6/90/3a8d5ca977171ec3ae21a71207d2228b2663bde14d7f7ef0e6363ecf9290/regex-2026.7.19-cp314-cp314-win32.whl", hash = "sha256:73f272fba87b8ccfe70a137d02a54af386f6d27aa509fbffdd978f5947aae1aa", size = 272531 }, + { url = "https://files.pythonhosted.org/packages/96/e1/8862885e70409de70e8c005f57fb2e7be8d9ef0317250d60f4c9660a300d/regex-2026.7.19-cp314-cp314-win_amd64.whl", hash = "sha256:d721e53758b2cca74990185eb0671dd466d7a388a1a45d0c6f4c13cef41a68ac", size = 280831 }, + { url = "https://files.pythonhosted.org/packages/08/82/2693e53e29f9104d9de95d37ce4dd826bd32d5f9c0085d3aa6ac042675c4/regex-2026.7.19-cp314-cp314-win_arm64.whl", hash = "sha256:65fa6cb38ed5e9c3637e68e544f598b39c3b86b808ed0627a67b68320384b459", size = 281099 }, + { url = "https://files.pythonhosted.org/packages/92/b7/9a01aa16461a18cde9d7b9c3ab21e501db2ce33725f53014342b91df2b0a/regex-2026.7.19-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:5a2721c8720e2cb3c209925dfb9200199b4b07361c9e01d321719404b21458b3", size = 501121 }, + { url = "https://files.pythonhosted.org/packages/f3/5e/bbaeca815dc9191c424c94a4fdc5c87c75748a64a6271821212ebdd4e1a3/regex-2026.7.19-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:199535629f25caf89698039af3d1ad5fcae7f933e2112c73f1cdf49165c99518", size = 299415 }, + { url = "https://files.pythonhosted.org/packages/cd/d6/0dd1a321afaab95eb7ff44aa0f637301786f1dc71c6b797b9ed236ed8890/regex-2026.7.19-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:9b60d7814174f059e5de4ab98271cc5ba9259cfea55273a81544dceea32dc8d9", size = 294483 }, + { url = "https://files.pythonhosted.org/packages/92/5f/40bacf91d0904f812e13bbbab3864604c463eced8afdc54aeaa50492ea95/regex-2026.7.19-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dbece16025afda5e3031af0c4059207e61dcf73ef13af844964f57f387d1c435", size = 811833 }, + { url = "https://files.pythonhosted.org/packages/94/7c/4902744261f775aeede8b5627314b38482da29cf49a57b66a6fb753246c5/regex-2026.7.19-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d24ecb4f5e009ea0bd275ee37ad9953b32005e2e5e60f8bbae16da0dbbf0d3a0", size = 871270 }, + { url = "https://files.pythonhosted.org/packages/16/70/6980c9be6bf21c0a60ed3e0aea39cf419ecf3b08d1d9947bc56e196ef186/regex-2026.7.19-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8cae6fd77a5b72dae505084b1a2ee0360139faf72fedbab667cd7cc65aae7a6a", size = 917534 }, + { url = "https://files.pythonhosted.org/packages/52/92/8b2bd872782ce8c42691e39acb38eb8efe014e5ddb78ad7d943d6f197ce9/regex-2026.7.19-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9724e6cb5e478cd7d8cabf027826178739cb18cf0e117d0e32814d479fa02276", size = 816135 }, + { url = "https://files.pythonhosted.org/packages/de/2d/33a602f657bdc4041f17d79f92ab18261d255d91a06117a6e29df023e5e2/regex-2026.7.19-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:572fc57b0009c735ee56c175ea021b637a15551a312f56734277f923d6fd0f6c", size = 785492 }, + { url = "https://files.pythonhosted.org/packages/9e/36/0987cf4cb271680064a70d24a475873775a151d0b7058698a006cb0cae4a/regex-2026.7.19-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:20568e182eb82d39a6bf7cff3fd58566f14c75c6f74b2c8c96537eecf9010e3a", size = 800658 }, + { url = "https://files.pythonhosted.org/packages/a8/24/c14f31c135e1ba55fa4f9a58ca98d0842512bf6188230763c31c8f449e3b/regex-2026.7.19-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:1d58561843f0ff7dc78b4c28b5e2dc388f3eff94ebc8a232a3adba961fc00009", size = 865073 }, + { url = "https://files.pythonhosted.org/packages/14/85/181a12211f22469f24d2de1ebddfe397d2396e2c29013b9a58134a91069a/regex-2026.7.19-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:61bb1bd45520aacd56dd80943bd34991fb5350afdd1f36f2282230fd5154a218", size = 773684 }, + { url = "https://files.pythonhosted.org/packages/23/58/bd1a0c1a62251366f8d21f41b1ea3c76994962071b8b6ea42f72d505c0f0/regex-2026.7.19-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:cd3584591ea4429026cdb931b054342c2bcf189b44ff367f8d5c15bc092a2966", size = 857769 }, + { url = "https://files.pythonhosted.org/packages/e4/4f/f7e2dad6756b2fe1fe75dd90a628c3b45f249d39f948dd90cd2476325417/regex-2026.7.19-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5cc26a66e212fa5d6c6170c3a40d99d888db3020c6fdab1523250d4341382e44", size = 804546 }, + { url = "https://files.pythonhosted.org/packages/2b/d7/01d31d5bdb09bc026fab77f59a371fdf8f9b292e4810546c56182ca70498/regex-2026.7.19-cp314-cp314t-win32.whl", hash = "sha256:2c4e61e2e1be56f63ec3cc618aa9e0de81ef6f43d177205451840022e24f5b78", size = 274526 }, + { url = "https://files.pythonhosted.org/packages/52/0e/cea4ce73bc0a8247a0748228ae6669984c7e1f8134b6fa66e59c0572e0ea/regex-2026.7.19-cp314-cp314t-win_amd64.whl", hash = "sha256:c639ea314df70a7b2811e8020448c75af8c9445f5a60f8a4ced81c306a9380c2", size = 283763 }, + { url = "https://files.pythonhosted.org/packages/6f/b6/26e41975febae63b7a6e3e02f32cff6cff2e4f10d19c929082f56aebf7c6/regex-2026.7.19-cp314-cp314t-win_arm64.whl", hash = "sha256:9a15e785f244f3e07847b984ce8773fc3da10a9f3c131cc49a4c5b4d672b4547", size = 283451 }, +] + [[package]] name = "requests" version = "2.34.2" @@ -1790,6 +2091,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9b/36/9c015cd052fca743dae8cb2aeb16b551444787467db42ceab0fc968865af/ruff-0.15.13-py3-none-win_arm64.whl", hash = "sha256:2471da9bd1068c8c064b5fd9c0c4b6dddffd6369cb1cd68b29993b1709ff1b21", size = 11179336 }, ] +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050 }, +] + [[package]] name = "sniffio" version = "1.3.1" @@ -1825,6 +2135,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/bb/2799cc2ede3ed41131f8975621e7213dfc7ef4acbbaadfa440f32500c370/starlette-1.3.1-py3-none-any.whl", hash = "sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6", size = 73632 }, ] +[[package]] +name = "tld" +version = "0.13.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5c/5d/76b4383ac4e5b5e254e50c09807b3e13820bed6d6c11cd540264988d6802/tld-0.13.2.tar.gz", hash = "sha256:d983fa92b9d717400742fca844e29d5e18271079c7bcfabf66d01b39b4a14345", size = 467175 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/90/39a85a4b63c84213e78b3c17d22e1bf45328acf8ebb33ef93be30d0a3911/tld-0.13.2-py2.py3-none-any.whl", hash = "sha256:9b8fdbdb880e7ba65b216a4937f2c94c49a7226723783d5838fc958ac76f4e0c", size = 296743 }, +] + [[package]] name = "tqdm" version = "4.67.3" @@ -1837,6 +2156,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374 }, ] +[[package]] +name = "trafilatura" +version = "2.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "courlan" }, + { name = "htmldate" }, + { name = "justext" }, + { name = "lxml" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a3/96/737133a93e73e967f9c888e6cfb1f2c31b2083d27263edb19fd65a9aca02/trafilatura-2.2.0.tar.gz", hash = "sha256:8c2cabb84066465228d03183fb698ce0b1245b81c58140b8ae0de57fddf3aae7", size = 314748 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c3/01/af18878398102a5a5afa0811f4f8f2a8a94a60cc16e8e9cf54bc95f96808/trafilatura-2.2.0-py3-none-any.whl", hash = "sha256:ac43592a6201264dfc4f9c361cbe3eb3fea96e54437010a159d5e7365360ed98", size = 151906 }, +] + [[package]] name = "typing-extensions" version = "4.15.0" @@ -1858,6 +2195,27 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611 }, ] +[[package]] +name = "tzdata" +version = "2026.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/92/ff/5a28bdfd8c3ebec42564ac7d0e54ca3db65044a9314a97f9564fa7a1e926/tzdata-2026.3.tar.gz", hash = "sha256:4a1518b8993086a7982523e071643f3c0e5f213e75b21318e78bcabfff9d1415", size = 198674 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/6d/b53b99a9f2766d095985947a5782f1702cabb129a34f7a802d7197af832f/tzdata-2026.3-py2.py3-none-any.whl", hash = "sha256:dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931", size = 348168 }, +] + +[[package]] +name = "tzlocal" +version = "5.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tzdata", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/81/5b/879b2f932adfa7a053c360d50bc896c977fa6426109185f7c12ebdd0cb9d/tzlocal-5.4.4.tar.gz", hash = "sha256:8dbb8660838688a7b6ba4fed31d18dedf842afb4d47ca050d6d891c2c15f3be4", size = 31170 } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/a4/017a7a6cbe387d961a688ec31364ae60a5c4e22c96ae9921b79a947c855d/tzlocal-5.4.4-py3-none-any.whl", hash = "sha256:aae09f0126a8a86fa736be266eb4a471380d26a0de3bc14844e7821fee3e2a15", size = 18115 }, +] + [[package]] name = "urllib3" version = "2.7.0"