sdk_profiles: fingerprint-gated, fidelity-tiered SDK profile bundles - #911
Draft
lacraig2 wants to merge 12 commits into
Draft
sdk_profiles: fingerprint-gated, fidelity-tiered SDK profile bundles#911lacraig2 wants to merge 12 commits into
lacraig2 wants to merge 12 commits into
Conversation
The SDK-keyed alias groups in defaults.py (atheros_broadcom, realtek, netgear_acos, zyxel_or_edimax, ralink) were flattened into the single default_lib_aliases map and applied by LibInjectTailoredAliases to every target, discarding the grouping. Keep the grouping and emit one named, disabled patch per SDK group (sdk.<group>), each filtered to the symbols the target actually exports. A disabled patch is a candidate the config search can toggle on, not a fact baked into the initial config, so an SDK bundle becomes a searchable vertex with provenance rather than an always-on default. The always-on LibInjectTailoredAliases now applies only the generic (non-SDK) base_names/base_aliases; its unmodeled-nvram logging still checks the full union so an SDK-only symbol is neither emitted there nor falsely flagged. defaults.py adds sdk_lib_aliases (the grouping) and generic_lib_aliases; default_lib_aliases is unchanged. Host test in tests/unit/test_sdk_aliases.py asserts each sdk.<group> patch is emitted disabled with the right exported subset and that no SDK symbol leaks into the generic patch; test_init_discovery updated for the five new classes.
…dcom_hnd/qualcomm_qsdk) Add SdkFinder, which scores each profile catalog entry (profiles/*.yaml) against static analyses that already run (InterfaceFinder, ClusterCollector, LibrarySymbols) -- the consumer those analyses were missing. Each profile has a patch class that emits its config bundle as sdk.<name>, ENABLED when the fingerprint corroborates at >= enable_if.min_signals (a confident baseline fact) and DISABLED (a search candidate) otherwise. Ships two source-confirmed profiles: broadcom_hnd (et/robo switch + CFE nvram) and qualcomm_qsdk (libnvram DT_NEEDED shim + uClibc-eager nvram aliases). Both deliberately carry only SDK-general bundles, not device-specific seeds.
Restructure profile catalog bundles under a fidelity tier key so one SDK can
offer two rungs of the same emulation off its single shared fingerprint:
libinject (Tier 0, shipping) -- shim the SDK's nvram/switch symbols at the
library layer and answer from config.
mtd (Tier 1, reserved) -- back the real /dev/mtdN nvram partition (native MTD
device, hyperfile/mtd.py) with the SDK's actual flash blob so the firmware's
own nvram library parses it. Built per-SDK once its flash format is reversed,
emitted as a separate disabled sdk.<name>.mtd candidate the search promotes.
_profile_patch now takes a tier (default libinject); returns None when a tier
has no bundle (e.g. an unbuilt mtd). Legacy single-bundle profiles map onto
Tier 0. broadcom_hnd/qualcomm_qsdk migrated to bundles.libinject; broadcom_hnd
documents the reserved mtd tier inline.
… broadcom_hnd The always-on nvram.04_defaults (_get_default_nvram_values) shipped Broadcom silicon nvram (et0macaddr, the wl per-radio %d:macaddr block) to every target. Move them into the broadcom_hnd SDK profile's libinject bundle, where they are gated by the profile fingerprint, and shrink the generic default accordingly. Genuinely-generic keys (lan_*/wan_*/time_zone/...) stay in nvram.04. First slice of the nvram fold; the Netgear ACOS cluster (sku_name, time_zone_x, usb_info_dev, wla_/wlg_*, ...) is deferred pending the profile/alias-group naming decision.
Complete the nvram fold: the always-on nvram.04_defaults shipped a large Netgear ACOS cluster to every target -- sku_name, wla_wlanstate, time_zone_x, rip_multicast, bs_trustedip_enable, filter_rule_tbl, the 102-entry ReadySHARE usb_info_dev block, and the wla_/wlg_ AP-isolation ranges. Move them into a new fingerprint-gated netgear_acos profile (acosNvramConfig_* symbols + libacos_shared.so/acos_nat.ko + acos_service). The profile subsumes the former Slice-0 sdk.netgear_acos alias group: retire SdkNetgearAcosAliases and drop netgear_acos from sdk_lib_aliases (it stays in the default_lib_aliases union for the LibInjectFixedAliases fallback), and carry its WAN_ith_CONFIG_GET shim in the profile bundle. This resolves the sdk.netgear_acos patch_name collision and demonstrates the target arch: one profile = fingerprint + full bundle (nvram + aliases). ACOS keys enumerated verbatim in the catalog YAML. Also fixes the nvram.04-vs-nvram.05 inconsistency (nvram.05 already gated time_zone_x/rip_multicast/bs_trustedip_enable/filter_rule_tbl on ACOS files).
…tiers Adds the Realtek RTL819x SDK profile and lands the accompanying uncommitted SDK-profile graduations as one coherent checkpoint. realtek_rtl819x (new): graduates the former Slice-0 sdk.realtek alias group into a full profile. Fingerprint (apmib.so + apmib_get/set/init symbols + iwcontrol/wscd) detects RTL819x; libinject tier carries the apmib_get/set aliases; a disabled `.boot` tier models the flash MTD surface (/dev/mtd*, /proc/mtd) and the RTL8367 switch. Verified on D-Link DIR-605L rev B (mipseb): the full Realtek userland boots (rcS -> startup.sh -> flash -> sysconf -> boa), boa binds :80, 0 panics. Known Tier-1 gap: apmib opens the bare path /dev/mtd, which penguin still drops before runtime, so the flash MIB is not yet served (box boots with an empty MIB); documented in the profile. hyperfile/pseudofiles: only defer /dev/mtd<N> and /proc/mtd to the native MTD subsystem (which adopts just `^/dev/mtd\d+$`). Bare /dev/mtd and /dev/mtdblockN were deferred-and-dropped by both handlers; now they fall through to the normal pseudofile machinery (verified: /dev/mtdblock0/3 now modeled). qualcomm_qsdk: drop the over-broad `ld-uClibc.so.0` fingerprint signal (present on any uClibc firmware; it false-fired on Realtek); the qca-ssdk.ko + QSDK nvram symbols remain and RT-AC58U still enables at 2-of-2. Also lands: asuswrt (silicon-independent boot/wan tiers) and mediatek_ralink (raeth mii_mgr/switch shim) profiles + their sdk_profiles classes; graduated interventions (bcm6300_switch, gpio_buttons, nvram_pin, reboot_skip); the broadcom_hnd fingerprint hardening + boot/mtd tiers; the gen_image mke2fs (/opt/e2fsprogs) fix. Tests updated for the hardened broadcom/qualcomm fingerprints and the new profile classes (test_sdk_finder, test_init_discovery green: 29/29).
…e on type change A patch that switches a pseudofile read/write model to a different variant (e.g. read:zero -> read:stateful, write:discard -> write:return_const) yields a different model type than the base. The field-by-field merge rebuilt the OLD variant class with the NEW discriminator value, failing schema validation. Treat a type change as a wholesale override (patch wins), like a scalar conflict. This unblocks SDK-profile boot tiers that upgrade a PseudofileFinder auto-detected node to read-after-write.
blocking (PollNeverReady) returns a zero mask so poll()/select()/epoll parks the waiter on the per-device wait queue instead of spinning -- for event-source nodes whose read() blocks on a hardware event that never occurs under emulation (an AVM-style /dev/watchdog). A later write wakes the parked waiter. periodic (PollPeriodic) reports the node readable on a fixed interval_ms cadence, driven by an igloo_driver kernel timer (carried down via POLL_INTERVAL_MS -> devfs poll_interval_ms), so an epoll(timeout=-1) main loop advances at a steady rate instead of spinning (always_ready) or deadlocking (blocking). dwarffi drops the key on drivers that predate the field, so it degrades gracefully.
Emit /etc/ld.so.preload as 'lib_inject.so' rather than the absolute /igloo/dylibs/lib_inject.so. The absolute path is symlinked to the default (64-bit) ABI, so on a mixed 32/64-bit target 32-bit daemons hit 'wrong ELF class ELFCLASS64: ignored' and get no interception. A slashless entry is class-filtered by the loader via the /lib symlinks next to each libc.so, matching the env LD_PRELOAD=lib_inject.so mechanism.
Fingerprint-gated profile for the AVM FRITZ!OS userland (Lantiq/Intel VR9-class): libboxlib/libtffs libraries, avmipc_*/tffs_* symbols, ctlmgr/multid/dsld daemons; require two signals to enable. Tier-0 (sdk.avm_fritzos) carries a PSEUDOFILE, not lib_inject aliases -- FRITZ!OS is uClibc (LD_PRELOAD-blind) so symbol aliases would not bind. It serves the MANDATORY urlader boot environment (/proc/sys/urlader/environment) as a const_buf: /etc/boot.d/1 hard-blocks the boot until that node exists, then reads the systemd.unit target from its kernel_args. The additive DISABLED candidate sdk.avm_fritzos.boot models the TFFS config-store char devices (/proc/tffs, /dev/tffs/mtdN, /dev/tffs_userlog, /dev/tffs_panic) as read-after-write so libtffs writes read back -- opt-in higher-fidelity flash.
The MTD/mtdblock nodes in the boot bundle paired read:stateful with write:default, so tffs-style read-after-write did not actually read back the written bytes. read-after-write needs write:discard, which records into the same buffer the stateful read serves (per the pseudofile schema). Matches the AVM boot tier.
Collaborator
Author
|
But the test still asks for the class by name: # tests/unit/test_sdk_aliases.py:70-78
lib_inject_classes = [
_builtin(n)
for n in (
"LibInjectTailoredAliases",
"SdkAtherosBroadcomAliases",
"SdkRealtekAliases", # <- no longer exists
"SdkZyxelOrEdimaxAliases",
"SdkRalinkAliases",
)
]
There's a second, dependent occurrence at line 95: cases = {
"sdk.realtek": realtek, # <- no plugin emits this patch name now
"sdk.ralink": ralink,
"sdk.atheros_broadcom": atheros_broadcom,
}
(Found while building corpus-scale tooling to measure these fingerprints against our firmware set — I'll follow up separately with data on the profiles themselves.) |
Each fingerprint was distilled from one or two reference rehosts, which is enough to make a profile fire on the device it was built from and says nothing about what else it fires on. Measured over every corpus rootfs, four of the seven were wrong, in three different ways: * asuswrt keyed on `rc`, which is the Broadcom SDK's service manager and lives in 76% of the corpus -- more than the generic `nvram_set` control at 73%. With libshared.so (also inherited by every Broadcom fork) that was two signals on any DD-WRT or Tomato image: 21,980 ENABLED, 59% of labelled hits not ASUSWRT. Now keyed on ASUSWRT's own daemons: 4,403 ENABLED, 96% asus. * broadcom_hnd conflated the bcm963xx CPE SDK its bundles actually model with the classic "wl" Broadcom SDK that DD-WRT and Tomato inherit. `wl_probe` is in 68% of the corpus and 36,847 of its hits are DD-WRT. 25,781 ENABLED against a true population of ~2,600 -- a 10x over-fire putting a CFE envram seed and a CDK switch model onto images that have neither. Retargeted on the bcm963xx tools and libraries, each measured at 100% precision. * realtek_rtl819x was wrong in both directions at once. Its `file` signal named apmib.so (56 rootfs) and missed libapmib.so (319) -- the same library, the other spelling. Its `executable` signal named iwcontrol, which is madwifi's, and wscd, which is everyone's. Replaced with the RTL819x config tools. * avm_fritzos had a DEAD signal: libboxlib.so and libtffs.so matched 0 of 67,315, because AVM ships them as libboxlib.so.0.0.0 and libtffs.so.1.0.0 and ClusterCollector stores exact basenames. A dead signal is not inert -- with min_signals=2 it silently promoted the other two to mandatory. mediatek_ralink gains recall (4,479 -> 5,170; the wireless modules ship without the standalone Ethernet one). netgear_acos and qualcomm_qsdk were already right -- 99% and 100% vendor concordance -- and are annotated rather than changed; they are the control showing the mechanism works and these were bad token choices, not a flaw in fingerprint gating. Every count above is recorded in the profile it belongs to, so the next person to touch a token can see what it was measured at. Also fixes tests/unit/test_sdk_aliases.py, which has been failing at branch tip: it still asks for SdkRealtekAliases, removed when Realtek graduated to a profile, so _builtin() raises KeyError in setUp.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces fingerprint-gated, fidelity-tiered SDK profiles for Penguin: reusable bundles of config (NVRAM defaults, pseudofiles, lib_inject alias groups) that are matched against a firmware's static analyses and emitted at the right fidelity tier instead of being hardcoded into always-on plugins.
Mechanism
SdkFinderscores each profile's fingerprint (symbols / files / executables) againstInterfaceFinder,ClusterCollector, andLibrarySymbols. A profile patch is emitted ENABLED whensignals >= enable_if.min_signals, DISABLED (a search candidate forexplore) below threshold, and not emitted with no signal.bundles.<tier>):libinject(Tier 0, auto-enabled boot-gate essentials) andboot(additive DISABLED candidates for deeper device modeling).Profiles added
broadcom_hnd,qualcomm_qsdk,netgear_acos,asuswrt,mediatek_ralink,realtek_rtl819x, andavm_fritzos(AVM FRITZ!OS — urlader boot-envconst_bufin Tier 0 since uClibc is LD_PRELOAD-blind, plus aboottier modeling the TFFS char devices).Supporting changes
patch_config: a patch now overrides a discriminated-union config variant wholesale when the model type differs (e.g. aboot-tier upgrade of an auto-detected pseudofile fromzero→stateful), instead of field-merging incompatible variants.blocking(never-ready, parks the waiter) andperiodic(interval-driven) pseudofile poll variants + schema./etc/ld.so.preloadnow names the bare soname (lib_inject.so) so the loader resolves it per-ABI — fixes 32-bit daemons on mixed 32/64-bit targets hitting "wrong ELF class".nvram.04into their respective SDK profiles.Tests
New unit coverage:
test_sdk_finder,test_sdk_aliases,test_config_merge,test_lib_inject_preload, plus additions totest_pseudofile_compositionandtest_init_discovery.docs/schema_doc.mdregenerated and confirmed drift-free.🤖 Draft — opened for review of the SDK-profiles series.