Release-Notes.md

Release Notes

0.1.11-SNAPSHOT

Upstream / Resend — NIC binding for broadcast sends

• Fix: nic setting not honoured when targetIP=255.255.255.255: Previously, the UDP sender socket was created with a plain net.Dial call, ignoring the configured nic. The OS would pick the outgoing interface freely, and the socket lacked SO_BROADCAST, causing sends to the limited-broadcast address to fail silently or go out the wrong interface.
• The new NewUDPConnWithNIC helper in src/udp/sender.go sets SO_BROADCAST when the target is 255.255.255.255, and binds the socket to the IPv4 address of the named NIC when nic is set. Both upstream and resend pass config.nic through their NewUDPAdapter constructors.
• No downstream change required: downstream targetIP=0.0.0.0 is correct for receiving broadcast; the nic field on downstream continues to serve only MTU auto-detection.

Security Hardening — Downstream Listening Port

Four denial-of-service vulnerabilities on the downstream listening port have been fixed. All mitigations are on by default with safe values and can be tuned via configuration.

• Fragment cache memory exhaustion (OOM): An attacker could flood the port with UDP/TCP packets bearing unique message IDs and nrMessages=65535, filling the fragment-reassembly cache indefinitely. The cache now enforces a configurable entry cap (maxCacheEntries, default 100 000). Fragments arriving when the cache is full are dropped with a warning rather than accepted. The limit is exposed as exported constants DefaultMaxCacheEntries / DefaultMaxNrMessages in the protocol package so the value is accessible without magic numbers.
• Oversized nrMessages field: Packets claiming an implausibly large fragment count (nrMessages > maxNrMessages, default 4 096) are now rejected at parse time before any cache memory is allocated. Legitimate fragmentation at MTU 1500 tops out at ~730 parts for a 1 MiB payload.
• TCP goroutine exhaustion: Without a connection cap, an attacker opening thousands of idle TCP connections could exhaust goroutine stacks and memory. A new maxTCPConnections limit (default 256) closes connections that exceed the cap immediately, with a warning log.
• KEY_EXCHANGE CPU exhaustion via RSA flooding: Any sender could submit TYPE_KEY_EXCHANGE packets to trigger rsa.DecryptOAEP against every loaded private key. A configurable rate limit (keyExchangeMinIntervalSecs, default 1 s) silently drops duplicate key-exchange requests that arrive before the interval expires. Legitimate key rotation runs every hundreds of seconds and is unaffected.

All four limits are configurable via .properties file and environment variable overrides — see README.md for the full parameter table and guidance on when to change the defaults.

Security Hardening — Dependency and Toolchain Upgrades

• Go toolchain upgraded from go1.24.7 to go1.24.13, resolving 14 actively reachable stdlib CVEs (including TLS DoS, ASN.1 memory exhaustion, PEM quadratic blowup, and x509 panic).
• jackson-databind upgraded from 2.15.1 (May 2023) to 2.18.3 in the Java deduplication module.

Security Hardening — Sensitive Value Masking in Logs

• keyPasswordFile was logged as a literal file path in the startup configuration dump for the resend, create, and downstream applications. It is now masked as [set], consistent with how upstream already handled it.

Documentation and README

• Added a "DEBUG Logging Caveat" section to doc/InputFilter.md: when logLevel=DEBUG is active, up to 80 bytes of raw Kafka payload and the full message key are logged before the input filter runs, bypassing content suppression.
• Added four new rows to the downstream parameter table in README.md documenting maxCacheEntries, maxNrMessages, maxTCPConnections, and keyExchangeMinIntervalSecs.
• Added a "Listening Port Security" section to README.md explaining the threat model and guidance on when to raise each limit above its default.

0.1.10-SNAPSHOT

Input Filtering — Empty-Payload Behaviour

• No more false gaps from filtered events: Previously, events matched by inputFilterRules were dropped entirely before transmission. This caused the downstream gap-detector to report them as missing and trigger unnecessary resend cycles. Filtered events are now sent across the diode with an empty payload instead of being dropped, so every sequence ID reaches the downstream and no gap is recorded.
• Deduplication discards zero-length events: PartitionDedupApp now checks the payload length before forwarding to the clean topic. Events with an empty payload are silently discarded, so they never appear in downstream Kafka — the effect is identical to the old drop behaviour from a consumer perspective, but without the side-effect of generating gaps.
• Resend honours inputFilterRules: The resend application now supports the same inputFilterRules, inputFilterDefaultAction, and inputFilterTimeout configuration options as upstream. Matched events are sent with empty payload using the same logic, keeping resend consistent with normal upstream operation. Environment variable overrides: AIRGAP_RESEND_INPUT_FILTER_RULES, AIRGAP_RESEND_INPUT_FILTER_DEFAULT_ACTION, AIRGAP_RESEND_INPUT_FILTER_TIMEOUT.

Documentation Updates

• Updated doc/InputFilter.md to describe the empty-payload behaviour, corrected statistics field descriptions and troubleshooting log message references, and added resend environment variable examples.
• Updated README.md upstream configuration table to reflect the empty-payload + dedup-discard design.
• Updated doc/Resend.md parameter list to include the three new inputFilter* options.

0.1.9-SNAPSHOT

TCP Transport Reliability

• Configurable TCP retry behaviour: When transport=tcp, upstream now retries failed sends with three new config parameters:
  • tcpRetryInterval (AIRGAP_UPSTREAM_TCP_RETRY_INTERVAL, default 1000 ms) — pause between retry attempts
  • tcpRetryTimes (AIRGAP_UPSTREAM_TCP_RETRY_TIMES, default 0) — maximum retries; 0 means infinite (never give up)
  • tcpRetryErrorLevel (AIRGAP_UPSTREAM_TCP_RETRY_ERROR_LEVEL, default WARN) — log level for per-attempt retry messages (DEBUG/INFO/WARN/ERROR)
  • With the default infinite-retry setting, upstream blocks until the downstream comes back and no messages are ever lost, fixing a bug where restarting the downstream caused the upstream to stop sending permanently
  • UDP retry behaviour is unchanged (30 × 100 ms, then Kafka reprocessing)

Deduplication (Java Streams)

• Fail-fast startup with retry backoff: Deduplication app now validates Kafka connectivity at startup and fails fast with clear error messages; retries with exponential backoff and jitter instead of entering a broken state
• Fixed restart loop: Resolved a bug where the deduplication app could get stuck in an infinite restart loop under certain error conditions
• Password leak fix: Fixed a bug where the Kafka password was exposed in log output (#11, contributed by Tobias Wennberg)
• Improved logging: More detailed event-level logging in PartitionDedupApp and GapDetector; tuned log levels for individual events to reduce noise at INFO level

Upstream / Downstream

• Decompression size limit: New maxDecompressedSize config option for downstream to cap the maximum size of a decompressed message, protecting against decompression bombs. Defaults to no limit
• Fix downstream assembly panic (#10): Resolved a panic that could occur during message reassembly when fragments arrived in unexpected order or were corrupted; replaced with graceful error handling
• UDP broadcast duplication remedy: Documented and added configuration guidance for handling duplicate messages that arise from UDP broadcast environments

Resend

• Fixed TLS for resend app: Resolved a bug that prevented the resend application from using TLS when connecting to Kafka; added certFile, keyFile, caFile, and keyPasswordFile support matching upstream/downstream configuration
• TLS integration tests: Added resend_tls_test.go with test coverage for TLS-authenticated Kafka connections in resend

Partition Merging

• Merge multiple topics into one: New partitionStartValue feature (upstream) allows merging events from several source clusters/topics into a single downstream topic by offsetting partition numbers. Full test case (testcase 21) with two upstream sources, one downstream, one deduplicator and full resend support
• Load balancing filter utility: New utils/loadbalance.py script to calculate the correct deliverFilter values for a given number of upstream instances and partitions

Internal / Testing

• Compression integration tests: Added src/test/compression_test.go
• Test configs for testcase 21: Complete set of properties/env files covering all components in the partition-merge scenario
• Kafka TLS helper: Extracted reusable TLS config setup in src/kafka/getkafka.go for use by resend and other clients
• Documentation: Updated README, Deduplication.md, FAQ.md, Redundancy and Load Balancing.md, Resend.md, and Transport Configuration.md with new features and troubleshooting guidance

0.1.8-SNAPSHOT

Transport Layer Improvements

• New TCP Transport Support: Added TCP as an alternative to UDP for more reliable delivery over unstable networks

  • Configure with transport=tcp in upstream and downstream config files
  • Environment variable overrides: AIRGAP_UPSTREAM_TRANSPORT, AIRGAP_DOWNSTREAM_TRANSPORT
  • Automatic connection health checking with 1ms read deadline test
  • Automatic reconnection on connection loss
  • Lazy connection initialization - upstream doesn't fail on startup if downstream is unavailable
  • See Transport Configuration.md for detailed TCP vs UDP comparison

• Enhanced Message Delivery Reliability:

  • Automatic retry logic: 30 attempts with fixed 100ms intervals (3+ seconds total) for transient failures
  • Messages not marked as consumed in Kafka if send fails - automatic reprocessing when receiver comes back up
  • Prevents message loss due to temporary network issues
  • Only marks Kafka message consumed on successful delivery
  • UDP: Transient failures log at WARN level to indicate delivery uncertainty
  • TCP: Automatic reconnection handling with detailed error reporting

• Transport Status Monitoring:

  • New transport_status field in periodic statistics showing transport health ("running" or error message)
  • Status change logging: ERROR level when status changes to error, INFO level when restored
  • Transport errors tracked and reported in structured statistics instead of separate log spam
  • Useful for monitoring and alerting on transport issues
  • UDP and TCP use same monitoring interface

Kafka Connection Monitoring

• Kafka Status Tracking: New kafka_status field in statistics for both upstream and downstream

  • Monitors Kafka cluster availability and broker connectivity
  • Upstream: Custom Sarama logger captures metadata errors (leaderless partitions, connection refused, etc.)
  • Downstream: Kafka producer error monitoring via callback mechanism
  • Status changes logged at appropriate levels (ERROR for failures, INFO for recovery)
  • Both consumer and producer errors reported via unified callback system
  • Helps identify cluster issues early before message loss occurs

• Error Handling Improvements:

  • Kafka consumer no longer panics on connection errors - retries with 5-second backoff instead
  • Producer errors monitored and reported to status system
  • Error messages include context (attempt number, error details) for debugging
  • Health check goroutines with 10-second periodic validation

Content-Based Filtering

• New Input Filtering feature: Content-based filtering of events at upstream before transmission
  • Filter events using regex patterns with allow/deny rules
  • Configure with inputFilterRules, inputFilterDefaultAction, inputFilterTimeout parameters
  • Environment variable overrides: AIRGAP_UPSTREAM_INPUT_FILTER_*
  • Useful for security (high-severity events only), privacy (block PII), compliance, and performance
  • First-match-wins rule evaluation with configurable default action (allow/deny)
  • Protection against ReDoS attacks with configurable regex timeout (default 100ms)
  • Dangerous pattern detection at startup (nested quantifiers)
  • New statistics counters: filtered, unfiltered, filter_timeouts per interval
  • Cumulative counters: total_filtered, total_unfiltered, total_filter_timeouts
  • See InputFilter.md for detailed documentation and use cases

Documentation

• Release Notes have been extracted to a separate document
• A FAQ (Frequently Asked Questions) document has been created
• Transport Configuration documentation added with TCP vs UDP comparison, tuning, and troubleshooting
• Input Filtering documentation with examples for security, privacy, and compliance use cases
• Configuration and Monitoring documentation updated with new transport and Kafka status fields

Internal Improvements

• Protocol parser: Removed panic statements, now returns errors gracefully
• Code formatting: Fixed spacing in message parsing logic
• Kafka adapter: Updated to pass error callbacks for status monitoring
• Configuration: Support for all environment variable overrides (AIRGAP_UPSTREAM_, AIRGAP_DOWNSTREAM_)
• Test configurations: Added test cases 19 and 20 for input filtering and TCP testing

0.1.7-SNAPSHOT

• Fixed the following issues:
  • Support encrypted key files for communication with Kafka #5

0.1.6-SNAPSHOT

• Removed topic configuration from downstream. Downstream uses upstream's topic name, or a translation of that name.
• Fixed the following issues:
  • resend will not accept payloadSize=auto #1
  • Separate internal logging from event stream from Upstream Kafka #2
  • Dedup can't use TLS to connect to Kafka #3

0.1.5-SNAPSHOT

• Multiple sockets with SO_REUSEPORT for faster and more reliable UDP receive in Linux and Mac for downstream. Fallback to single thread in Windows.
• create application to create resend bundle files downstream
• resend application to resend missing events from the resend bundle created by create
• compressWhenLengthExceeds setting for upstream and resend to compress messages when length exceeds this value. As of now gzip is the only supported algorithm.
• More configuration for upstream and downstream for buffer size optimizations
• Upstream and downstream can translate topic names to other names. Useful in multi source and/or target setups.
• Statistics logging in upstream, downstream and dedup

0.1.4-SNAPSHOT

• Changed the logging for the go applications to include log levels. Monitoring and log updates.
• Changed the logging for the go applications to include log levels. Monitoring and log updates.
• Documented redundancy and load balancing (see doc folder)
• Documented resend (future updates will implement the new resend algorithm)

0.1.3-SNAPSHOT

• Added a Kafka Streams Java Application for deduplication and gap detection. Gap detection not finished.
• Added upstreams filter to filter on the offset number for each partition (used in redundancy an load balancing setups)
• Added a topic name mapping in downstream so a topic with a specified name upstream can be written to another topic downstream (used in redundancy an load balancing setups)
• Added documentation for the new features.
• Added JMX monitoring of the deduplication application. Added system monitoring documentation

0.1.2-SNAPSHOT

• All configuration from files can be overridden by environment variables. See Configuration Upstream
• UDP sending have been made more robust
• Transfer of binary data from upstream to downstream is now supported
• Sending a sighup to upstream or downstream will now force a re-write of the log file, so you can rotate the log file and then sighup the application to make it log to a new file with the name specified in the upstream or downstream configuration.
• air-gap now supports TLS and mTLS to Kafka upstream and downstream.
• air-gap now supports TLS and mTLS to Kafka upstream and downstream.

0.1.1-SNAPSHOT

air-gap now supports several sending threads that all have a specified time offset, so you can start one thread that consumes everything from Kafka as soon as it's available, one that inspects Kafka content that was added for an hour ago and so on. See Automatic resend above.