Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "mycel"
version = "0.2.0"
version = "0.3.0"
edition = "2024"
description = "A fast, decentralized web crawler, indexer, and search engine"
license = "AGPL-3.0-only"
Expand Down
21 changes: 20 additions & 1 deletion docs/MANUAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,12 @@ process start; nothing reloads live.
| `bind` | `"127.0.0.1:8080"` | HTTP listen address for `mycel run`. The API has no authentication or TLS; keep it on localhost or put a reverse proxy in front. |
| `page_size` | `10` | Results per page, for the API, the web UI, and `mycel search`. There is no per-request size parameter. |

### `[admin]`

| key | default | meaning |
|---|---|---|
| `allowed_hosts` | `[]` | Extra `Host` header values accepted on `/admin`, beyond `api.bind` and its loopback equivalents; needed when `api.bind` is a wildcard like `0.0.0.0:8080`. See [the admin page](#get-admin-the-admin-page). |

### `[federation]`

| key | default | meaning |
Expand Down Expand Up @@ -284,7 +290,8 @@ Allowlist changes require a restart.
### Validation

At startup mycel rejects: unknown keys anywhere; `crawl.scope` other than
`"host"`; empty `index.languages`; `crawl.concurrency = 0`; a
`"host"`; empty `index.languages`; `crawl.concurrency = 0`; an
`admin.allowed_hosts` entry that is empty or contains `/` or whitespace; a
`federation.preset` other than `"n0"`/`"empty"`; a peer `id` that is not 64
hex chars; a peer `addr` that is not `ip:port`.

Expand Down Expand Up @@ -544,6 +551,18 @@ not authentication: anything that can already send local HTTP can read the
token from the page. The API's trust model is unchanged (keep it on
localhost or behind a reverse proxy; a proxy must forward a matching `Host`).

If `api.bind` is a wildcard like `0.0.0.0:PORT`, reaching `/admin` by LAN
IP or hostname requires listing those Host values in `admin.allowed_hosts`:

```toml
[admin]
allowed_hosts = ["mynode.lan:8080", "192.168.1.50:8080"]
```

The match stays exact and case-insensitive, so a web page you happen to
visit still cannot drive the node. It is not authentication: anyone who
can reach the port under a listed name can.

### `GET /healthz`

Round-trips the storage pipeline (1 s budget). Healthy:
Expand Down
3 changes: 3 additions & 0 deletions docs/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,9 @@ shard_mb = 1024 # seal open shard at ~1 GiB
bind = "127.0.0.1:8080"
page_size = 10

[admin]
allowed_hosts = [] # extra Host headers accepted on /admin (post-v1 extension)

[federation]
enabled = false # peerless default: no socket bound, nothing published
fanout = true
Expand Down
4 changes: 2 additions & 2 deletions site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ <h3>stract</h3>
<p class="epitaph">Open-source web search in Rust. Essentially one developer. Archived by its owner; no successor.</p>
<div class="rebuttal">
<svg class="thread" viewBox="0 0 20 60" preserveAspectRatio="none" aria-hidden="true"><path d="M10,60 C4,48 16,40 9,28 C4,19 13,12 10,0" pathLength="100"/></svg>
<p><strong>what grew back:</strong> survivability as architecture. ~6,300 lines, 27 boring dependencies, and data formats that stay valuable even if development pauses for a year.</p>
<p><strong>what grew back:</strong> survivability as architecture. ~8,500 lines, 27 boring dependencies, and data formats that stay valuable even if development pauses for a year.</p>
</div>
</article>

Expand Down Expand Up @@ -239,7 +239,7 @@ <h3><span class="instno">instrument 03</span> the merge</h3>
<div class="numbers">
<div class="num"><strong data-count="1">1</strong><span>binary</span></div>
<div class="num"><strong data-count="27">27</strong><span>dependencies</span></div>
<div class="num"><strong data-count="6300" data-approx="~" data-format="comma">~6,300</strong><span>lines of rust</span></div>
<div class="num"><strong data-count="8500" data-approx="~" data-format="comma">~8,500</strong><span>lines of rust</span></div>
<div class="num"><strong data-count="0">0</strong><span>servers required</span></div>
</div>

Expand Down
6 changes: 6 additions & 0 deletions src/admin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ impl AdminState {
allowed_hosts.push(v);
}
}
for h in &cfg.admin.allowed_hosts {
let h = h.to_ascii_lowercase();
if !allowed_hosts.contains(&h) {
allowed_hosts.push(h);
}
}
Self {
cfg,
cfg_path,
Expand Down
38 changes: 38 additions & 0 deletions src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ pub const DEFAULT_CONFIG_TOML: &str = r#"# mycel configuration. Every value belo
# bind = "127.0.0.1:8080"
# page_size = 10

[admin]
# allowed_hosts = [] # extra Host headers accepted on /admin, e.g. ["mycel.lan:8080"]

[federation]
# enabled = false # peerless default: no socket bound, nothing published
# fanout = true
Expand Down Expand Up @@ -67,6 +70,7 @@ pub struct Config {
pub rank: RankCfg,
pub warc: WarcCfg,
pub api: ApiCfg,
pub admin: AdminCfg,
pub federation: FederationCfg,
pub sync: SyncCfg,
pub bootstrap: BootstrapCfg,
Expand Down Expand Up @@ -168,6 +172,14 @@ impl Default for ApiCfg {
}
}

#[derive(Debug, Clone, Default, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct AdminCfg {
/// Extra Host headers accepted on /admin, beyond api.bind and its
/// loopback aliases (the DNS-rebinding guard's allowlist).
pub allowed_hosts: Vec<String>,
}

#[derive(Debug, Clone, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct FederationCfg {
Expand Down Expand Up @@ -280,6 +292,13 @@ impl Config {
if self.crawl.concurrency == 0 {
return Err("crawl.concurrency must be > 0".into());
}
for h in &self.admin.allowed_hosts {
if h.is_empty() || h.contains('/') || h.contains(char::is_whitespace) {
return Err(
format!("admin.allowed_hosts entries must be host[:port] (got {h:?})").into(),
);
}
}
if !matches!(self.federation.preset.as_str(), "n0" | "empty") {
return Err(format!(
"federation.preset must be \"n0\" or \"empty\" (got {:?})",
Expand Down Expand Up @@ -359,6 +378,25 @@ mod tests {
assert!(toml::from_str::<Config>(bad).unwrap().validate().is_err());
}

#[test]
fn admin_allowed_hosts_defaults_empty() {
let cfg: Config = toml::from_str("").unwrap();
assert!(cfg.admin.allowed_hosts.is_empty());
let cfg: Config =
toml::from_str("[admin]\nallowed_hosts = [\"mycel.lan:8080\"]\n").unwrap();
assert_eq!(cfg.admin.allowed_hosts, vec!["mycel.lan:8080"]);
}

#[test]
fn admin_allowed_hosts_validated() {
let good = "[admin]\nallowed_hosts = [\"mycel.lan:8080\", \"[::1]:8080\"]\n";
toml::from_str::<Config>(good).unwrap().validate().unwrap();
for bad in ["", " ", "http://mycel.lan:8080", "mycel.lan:8080/admin"] {
let cfg = format!("[admin]\nallowed_hosts = [{bad:?}]\n");
assert!(toml::from_str::<Config>(&cfg).unwrap().validate().is_err());
}
}

#[test]
fn overrides_apply() {
let cfg: Config =
Expand Down
12 changes: 11 additions & 1 deletion tests/admin_ui.rs
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ fn admin_page_drives_the_node() {
let port = free_tcp_port();
let base_cfg = format!(
"data_dir = \"{}\"\n[crawl]\ncontact_url = \"http://example.com/test\"\n\
[index]\ncommit_secs = 1\n[api]\nbind = \"127.0.0.1:{port}\"\n",
[index]\ncommit_secs = 1\n\
[admin]\nallowed_hosts = [\"extra.example:{port}\"]\n\
[api]\nbind = \"127.0.0.1:{port}\"\n",
dir.join("data").display()
);
std::fs::write(dir.join("mycel.toml"), &base_cfg).unwrap();
Expand Down Expand Up @@ -107,6 +109,14 @@ fn admin_page_drives_the_node() {
let (code, _) = http(&["-H", "Host: evil.example:1", &format!("{api}/admin")]);
assert_eq!(code, 403, "foreign Host header must be refused");

// admin.allowed_hosts admits an extra Host value beyond api.bind + loopback.
let (code, _) = http(&[
"-H",
&format!("Host: extra.example:{port}"),
&format!("{api}/admin"),
]);
assert_eq!(code, 200, "admin.allowed_hosts entry must be accepted");

// Seed through the writer: hosts activate, roots enqueue (.invalid never
// resolves, so the crawler generates no real traffic).
let (code, _) = http(&[
Expand Down
Loading