Deliver add-to messages to all participant domains (SPEC v0.4.0)

SPEC.md

@@ -138,7 +138,7 @@ Single-value codes (sent as first/only byte):
 | 8 | future time | Timestamp too far in future. |
 | 9 | time travel | Timestamp before parent's timestamp. |
 | 10 | duplicate | Already received for all recipients. |
-| 11 | accept add to | Add-to accepted; parent already stored; no _add to_ recipients on this host. Stop. |
+| 11 | accept add to | Add-to accepted; parent already stored; no _add to_ recipients on this host (including notification-only participant domains). Stop. |
 | 64 | continue | Header accepted; send data. |
 | 65 | skip data | Add-to accepted; parent already stored; _add to_ recipients on this host. Skip data, per-recipient codes follow. |
 
@@ -176,7 +176,11 @@ One message per connection. Two TCP connections used: Connection 1 (message tran
 
 ### 10.2 Sending (Host A perspective)
 
-Host A delivers iff _from_ or _add to from_ belongs to Host A's domain. For each unique recipient domain:
+Host A delivers iff _from_ or _add to from_ belongs to Host A's domain.
+
+When _has add to_ is NOT set: perform the steps below for each unique recipient domain.
+
+When _has add to_ IS set: perform the steps below for each unique participant domain — the domains of _from_ and of every address in _to_ and _add to_. _from_'s domain is omitted when _from_ is the _add to from_ (the adder is the original sender, whose host is Host A). Domains having no address in this message's _to_ or _add to_ are **notification-only**: the exchange completes at the single response code in step 5 (code 11 on success, or code 6 when the domain's host does not hold the parent) and never reaches step 6.
 
 1. Resolve recipient domain IPs via ``fmsg.<domain>``. Connect to first responsive IP (Connection 1). Retry with backoff if unreachable.
 2. Register the message header hash and Host B's IP in an outgoing record (for matching challenges).
@@ -201,10 +205,11 @@ Host A delivers iff _from_ or _add to from_ belongs to Host A's domain. For each
 3. Validate (all must pass, else respond code 1 invalid and close):
    - _to_ has ≥ 1 distinct address.
    - If _has add to_: _add to from_ exists and is in _from_ or _to_; _add to_ has ≥ 1 distinct address.
-   - ≥ 1 recipient in _to_ or _add to_ belongs to Host B's domain.
+   - If _has add to_ not set: ≥ 1 recipient in _to_ belongs to Host B's domain. If _has add to_ set: ≥ 1 participant (_from_, _to_, _add to from_ or _add to_) belongs to Host B's domain.
    - Common type IDs (message and attachment) are mapped.
+   - _expanded size_ fields are present iff the corresponding zlib-deflate flag is set.
 4. DNS-verify sender IP: resolve `fmsg.<sender domain>`, check Connection 1 source IP is in result set. Fail → TERMINATE.
-5. If _size_ + attachment sizes > MAX_SIZE → respond code 4, close.
+5. If _size_ + attachment sizes > MAX_SIZE, or total expanded size > MAX_EXPANDED_SIZE → respond code 4, close. Total expanded size uses _expanded size_ for compressed parts and _size_ for uncompressed parts.
 6. Compute DELTA = now − _time_:
    - DELTA > MAX_MESSAGE_AGE → respond code 7, close.
    - DELTA < −MAX_TIME_SKEW → respond code 8, close.
@@ -218,17 +223,18 @@ Host A delivers iff _from_ or _add to from_ belongs to Host A's domain. For each
      - pid MUST also be set. Fail → respond code 1, close.
      - Check if parent stored (§11):
        - **Stored**: check time travel (code 9 if fail).
-       - **Not stored**: treat as full message delivery.
+       - **Not stored**: if ≥ 1 recipient in _to_ or _add to_ belongs to Host B's domain, treat as full message delivery. Otherwise (Host B hosts only non-recipient participants) respond code 6 (parent not found), close.
 8. Optionally issue a CHALLENGE on Connection 2 (see §10.5).
 
 ### 10.4 Receiving — ACCEPT Response, Data Download and Per-Recipient Response
 
 1. If _add to_ set and parent verified stored in step 7:
+   - If Host B has already recorded this exact add-to batch (§11) → respond code 10 (duplicate), close.
    - If any _add to_ recipient belongs to Host B's domain → respond 65 (skip data).
-   - Otherwise → record add-to fields, respond 11 (accept add to), close.
+   - Otherwise → record the add-to batch (_add to from_, _add to_, _time_) per §11, respond 11 (accept add to), close. This is the path notification-only participant domains take.
 2. If challenge was completed, use the message hash from the challenge response to check for duplicates across all recipients on Host B. If duplicate for all → respond code 10, close.
 3. Otherwise → respond 64 (continue).
-4. If code 65 was sent, skip to step 6 (data already stored). Otherwise download data + attachments (exactly declared sizes).
+4. If code 65 was sent, skip to step 6 (data already stored). Otherwise download data + attachments (exactly declared on-wire sizes). For each zlib-deflate part, decompress and verify output byte length exactly equals _expanded size_; failure or mismatch means invalid → TERMINATE.
 5. If challenge was completed, verify computed message hash matches the challenge response hash. For code 65, compute from received header + stored data. Mismatch → TERMINATE.
 6. For each recipient on Host B's domain (in _to_ order, then _add to_ order), send one response byte:
    - Already received → 103 (or 105).
@@ -278,10 +284,14 @@ An add-to message is a duplicate of the original message with these differences:
 - _time_ = new timestamp.
 - _topic_ is NOT present (pid is set).
 
+An add-to message MUST be sent to every participant domain per §10.2, so all participants of the message being added to — including the original sender, when not themselves the _add to from_ — learn of the added recipients, not only the domains hosting the new recipients. This is required because a subsequent reply may reference this add-to message via _pid_, and a host can only accept a reply whose parent it holds.
+
+Add-to batches do not chain: recipients are always added to the original message; an add-to message's _pid_ MUST NOT reference another add-to message. A message therefore has 0 or more add-to batches, each independently referencing it.
+
 ## 13. Security Requirements
 
-- Enforce MAX_SIZE before downloading data.
-- Enforce MAX_EXPANDED_SIZE: reject when declared expanded size exceeds limit.
+- Enforce MAX_SIZE and MAX_EXPANDED_SIZE before downloading data.
+- For zlib-deflate parts, bound decompression and require output length to exactly match _expanded size_; mismatch means invalid → TERMINATE.
 - Enforce per-connection and per-IP rate limits.
 - Apply idle/slow-connection timeouts.
 - Verify sender IP via DNS BEFORE issuing any challenge.

cmd/fmsgd/host.go

@@ -522,30 +522,44 @@ func verifySenderIP(c net.Conn, senderDomain string) error {
 	return fmt.Errorf("DNS verification failed")
 }
 
+// handleAddToParentNotStored resolves an add-to whose parent this host does
+// not hold (SPEC §10.3 step 7): with a local recipient the add-to is treated
+// as a full message delivery; without one nothing can be stored for anyone,
+// so respond code 6 (parent not found) and close.
+func handleAddToParentNotStored(c net.Conn, h *FMsgHeader, hasLocalRecipient bool) (*FMsgHeader, error) {
+	if hasLocalRecipient {
+		h.InitialResponseCode = AcceptCodeContinue
+		return h, nil
+	}
+	if err := sendCode(c, RejectCodeParentNotFound); err != nil {
+		return h, err
+	}
+	return h, fmt.Errorf("add-to: parent not stored and no recipients for domain %s", Domain)
+}
+
 func handleAddToPath(c net.Conn, h *FMsgHeader) (*FMsgHeader, error) {
 	if len(h.AddTo) == 0 {
 		return h, nil
 	}
 
 	addToHasOurDomain := hasDomainRecipient(h.AddTo, Domain)
+	hasLocalRecipient := addToHasOurDomain || hasDomainRecipient(h.To, Domain)
 
 	parentID, err := lookupMsgIdByHash(h.Pid)
 	if err != nil {
 		return h, err
 	}
 
 	if parentID == 0 {
-		h.InitialResponseCode = AcceptCodeContinue
-		return h, nil
+		return handleAddToParentNotStored(c, h, hasLocalRecipient)
 	}
 
 	parentMsg, err := getMsgByID(parentID)
 	if err != nil {
 		return h, err
 	}
 	if parentMsg == nil || !isMessageRetrievable(parentMsg) {
-		h.InitialResponseCode = AcceptCodeContinue
-		return h, nil
+		return handleAddToParentNotStored(c, h, hasLocalRecipient)
 	}
 
 	if parentMsg.Timestamp-FutureTimeDelta > h.Timestamp {
@@ -555,6 +569,18 @@ func handleAddToPath(c net.Conn, h *FMsgHeader) (*FMsgHeader, error) {
 		return h, fmt.Errorf("add-to: time travel detected (parent time %f, current %f)", parentMsg.Timestamp, h.Timestamp)
 	}
 
+	// A batch this host already recorded is a duplicate (SPEC §10.4 step 1).
+	recorded, err := addToBatchRecorded(parentID, h.AddTo)
+	if err != nil {
+		return h, err
+	}
+	if recorded {
+		if err := sendCode(c, RejectCodeDuplicate); err != nil {
+			return h, err
+		}
+		return h, fmt.Errorf("add-to: batch already recorded for msg %d", parentID)
+	}
+
 	if addToHasOurDomain {
 		h.InitialResponseCode = AcceptCodeSkipData
 		return h, nil
@@ -1022,10 +1048,17 @@ func readHeader(c net.Conn) (*FMsgHeader, *bufio.Reader, error) {
 	log.Printf("INFO: <-- MSG\n%s", h)
 
 	if !hasDomainRecipient(h.To, Domain) && !hasDomainRecipient(h.AddTo, Domain) {
-		if err := sendCode(c, RejectCodeInvalid); err != nil {
-			return h, r, err
+		// An add-to message is delivered to every participant domain, not only
+		// recipient domains (SPEC §10.2/§10.3): a domain hosting no recipient
+		// is being notified that recipients were added. add to from is always
+		// in from or to, so from is the only participant left to check.
+		participantOnly := len(h.AddTo) > 0 && strings.EqualFold(h.From.Domain, Domain)
+		if !participantOnly {
+			if err := sendCode(c, RejectCodeInvalid); err != nil {
+				return h, r, err
+			}
+			return h, r, fmt.Errorf("no participants for domain %s", Domain)
 		}
-		return h, r, fmt.Errorf("no recipients for domain %s", Domain)
 	}
 
 	if err := verifySenderIP(c, determineSenderDomain(h)); err != nil {

cmd/fmsgd/host_test.go

@@ -620,3 +620,30 @@ func TestReadAttachmentHeadersRejectsExpandedSizeExceedsMax(t *testing.T) {
 		t.Fatalf("expected reject code %d, got %v", RejectCodeTooBig, got)
 	}
 }
+
+func TestHandleAddToParentNotStoredWithLocalRecipient(t *testing.T) {
+	c := &testConn{}
+	h := &FMsgHeader{}
+	got, err := handleAddToParentNotStored(c, h, true)
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if got.InitialResponseCode != AcceptCodeContinue {
+		t.Fatalf("InitialResponseCode = %d, want %d (continue)", got.InitialResponseCode, AcceptCodeContinue)
+	}
+	if c.Buffer.Len() != 0 {
+		t.Fatalf("unexpected bytes written: %v", c.Buffer.Bytes())
+	}
+}
+
+func TestHandleAddToParentNotStoredParticipantOnly(t *testing.T) {
+	c := &testConn{}
+	h := &FMsgHeader{}
+	_, err := handleAddToParentNotStored(c, h, false)
+	if err == nil {
+		t.Fatal("expected error for participant-only add-to without stored parent")
+	}
+	if got := c.Buffer.Bytes(); len(got) != 1 || got[0] != RejectCodeParentNotFound {
+		t.Fatalf("wrote %v, want single code %d (parent not found)", got, RejectCodeParentNotFound)
+	}
+}