fix(config): benchmarks silently got zero tools because of a leftover setting in global.env

[haroldship]

Bug Fix Pull Request

Related Issue

Fixes #124

Description

Every benchmark (M3, AppWorld, BPO, Oak) was silently getting zero tools — the agent had nothing to work with, and every task failed looking like the agent made a mistake, when really the environment never gave it anything to call. This fixes it with a one-line removal.

It's an issue of precedence and overriding of settings. config/global.env has DYNACONF_ADVANCED_FEATURES__REGISTRY=false since this repo's very first commit. Every benchmark's own has, such as m3.env or appworld.env has DYNACONF_ADVANCED_FEATURES__REGISTRY=true. Until recently, the latter won. A recent, unrelated fix (#114) changed the reading order so the shared global.env wins instead.

Type of Changes

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix that would cause existing functionality to not work as expected)

Root Cause

config/global.env sets DYNACONF_ADVANCED_FEATURES__REGISTRY=false. Every benchmark's own .env (m3.env, appworld.env, bpo.env, oak_health_insurance.env) sets it to true. The bash-side loader (benchmarks/helpers/load_env.sh) loads global.env first and uses "no-override" semantics (first value wins) — so it has always resolved this contradiction in favor of false. This never showed up as a problem because the Python-side loader (config_loader/loader.py) used to apply settings with load_dotenv(file, override=True), so the benchmark-specific file — loaded second — always won there instead, correcting the bash-side mistake.

PR #114 (parameterizing hardcoded ports, for an unrelated reason — letting a caller's shell override a port setting before Python loads config) changed that Python loader to merge the files into a dict and apply it with os.environ.setdefault(key, value). setdefault only fills in a key if it isn't already set — and by the time Python runs, bash has already set it (to the wrong value). So the correction Python used to provide silently stopped happening.

Confirmed with a controlled before/after: checking out the commit right before #114, with the identical config/global.env content, reliably works (tools bind, task passes). Current main, same command, same config file, clean process table (ruling out any timing/concurrency explanation), reliably fails with CombinedToolProvider: Found 0 apps.

Solution

Remove the DYNACONF_ADVANCED_FEATURES__REGISTRY=false line (and its now-stale comment claiming M3 loads tools "directly from containers" — that mechanism doesn't exist anywhere in this repo's history) from config/global.env. No replacement default is set here, on purpose: every benchmark already declares the value it wants in its own .env file, so the shared config layer doesn't need an opinion on this key at all.

Testing

• I have tested this fix locally
• I have added tests that prove my fix works
• All new and existing tests passed (just lint — only unrelated failure is in an untracked file for a separate, not-yet-committed feature branch/PR)
• I have verified the bug no longer occurs

Verification performed:

• Traced both the bash loader (load_env.sh) and the Python loader (config_loader/loader.py) end-to-end for both m3 and appworld benchmarks; both now resolve DYNACONF_ADVANCED_FEATURES__REGISTRY to true.
• Live A/B: reverted this exact file to unfixed on current main and re-ran the failing M3 task with a clean process table (no concurrent runs) — reliably failed (Found 0 apps). Restored the fix — same task, same clean conditions — reliably passed (Found 1 apps, real tool calls, correct answer).
• Confirmed the same root cause explains AppWorld's variant of the symptom (registry process starts, but CombinedToolProvider still reports 0 apps because it checks the same flag before querying the running registry).

Checklist

• My code follows the code style of this project
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation if needed (none needed — this is a one-line config fix, no documented behavior changes)

Summary by CodeRabbit

• Bug Fixes

  • Benchmark-specific registry settings are no longer silently overridden by a global default.
  • Registry configuration must now be explicitly defined for each benchmark where needed.

• Documentation

  • Added guidance clarifying registry configuration behavior in no-override mode.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c5457b7c-d5d1-472e-b9f4-1dc2318d9ea3

📥 Commits

Reviewing files that changed from the base of the PR and between 5562399 and 8f1493e.

📒 Files selected for processing (1)

• config/global.env

---

📝 Walkthrough

Walkthrough

The global environment file no longer disables the registry. Comments now state that registry configuration belongs in each benchmark’s environment file.

Changes

Registry configuration

Layer / File(s)	Summary
Remove shared registry override config/global.env	Removes DYNACONF_ADVANCED_FEATURES__REGISTRY=false and documents that benchmarks must configure the registry themselves.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• cuga-project/cuga-eval#114: Changed environment loading behavior, exposing the conflicting global registry setting.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes removing the stale global.env registry setting that caused benchmarks to get zero tools.
Linked Issues check	✅ Passed	The change removes the conflicting global registry default and leaves benchmark configs to control registry behavior as requested in #124.
Out of Scope Changes check	✅ Passed	The PR only adjusts the shared registry setting and related comment in config/global.env, which stays within the bug fix scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/124-global-env-registry-override

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[offerakrabi]

this looks fine, however i would add a tests that confirms this