M3 config still set a removed flag (REQUIRE_TOOL_CALL_BEFORE_FINAL) instead of enabling CUGA_LITE_NL_AUTO_CONTINUE

[haroldship]

Summary

Imagine you have a rule "always double-check before finishing," but someone removes that rule from the app — except your test script still flips the switch for the rule that no longer exists. That's what was happening here.

M3's config was still setting REQUIRE_TOOL_CALL_BEFORE_FINAL=true, but that flag was removed from cuga-agent a while back (#416 / #451). So M3 wasn't actually testing anything real — it was flipping a switch that wasn't wired to anything, and the real mechanism that catches an agent stalling on planning-text instead of calling a tool was never being exercised.

What this PR does

• Turns on the switch that actually exists now: CUGA_LITE_NL_AUTO_CONTINUE=true. This is the live feature that nudges the agent to keep going instead of quitting early when it outputs planning text instead of a tool call (this already matches how AppWorld is configured).
• Adds three new test scenarios to actually exercise this: issue451_a_n1, issue451_a_n3, issue451_b_probe5 — probing both the planning-stall case and the "answers without grounding" case.

Fixes #116. Related: cuga-agent#451.

Testing

• Tested locally
• Verified the bug no longer occurs

issue451_b_probe5 loads and runs correctly under --eval-key with NL auto-continue on. Full retest of issue451_a_n1 / n3 is a follow-up once the agent's planning fast-path lands.

[coderabbitai]

📝 Walkthrough

Walkthrough

M3 replaces the removed tool-call finalization setting with natural-language auto-continue and adds three evaluation key lists covering planning-text smoke, confirmation, and consistency probes.

Changes

M3 NL auto-continue configuration

Layer / File(s)	Summary
Enable NL auto-continue benchmarks/m3/config/m3.env	Replaces the removed tool-call requirement setting with CUGA_LITE_NL_AUTO_CONTINUE and documents interim-versus-final natural-language handling.
Add evaluation probe sets benchmarks/m3/eval_config.toml	Documents two probe patterns and defines the issue451_a_n1, issue451_a_n3, and issue451_b_probe5 evaluation key lists.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• cuga-project/cuga-eval#76: Introduced the eval-key filtering mechanism used by the new M3 probe sets.
• cuga-project/cuga-eval#96: Previously added the removed tool-call finalization setting to the M3 configuration.

Suggested reviewers: sami-marreed, Sergey-Zeltyn

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR removes the stale env flag, adds CUGA_LITE_NL_AUTO_CONTINUE, and updates eval keys as requested in #116.
Out of Scope Changes check	✅ Passed	The eval_config changes support the stated M3 evaluation work and do not introduce unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: replacing the removed REQUIRE_TOOL_CALL_BEFORE_FINAL flag with CUGA_LITE_NL_AUTO_CONTINUE.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/116-m3-nl-auto-continue

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}