Removing Step From test.md Does Not Get Reflected in TestManager

[priyanshugahlot2004]

kane-cli version

0.4.8

Operating System

Windows (x64)

Installation method

npm

What happened?

When a step is removed from a _test.md file and the test is re-run, the changes are not reflect on the TestManager interface. The removed step still appears in TestManager, and no new version of the test case is generated.

Additional details

Bug: Removed Steps Not Reflected in TestManager

Steps to Reproduce

1. Create a test case using kane-cli run which has one step
2. Add a second step in the _test.md file
3. Run the test with kane-cli testmd run — both steps execute and TestManager updates correctly
4. Remove Step 2 from the _test.md file (leaving only Step 1)
5. Run the test again with kane-cli testmd run
6. Issue: TestManager still shows both steps; no new version is created

---

Expected Behavior

• After removing a step and re-running, TestManager should reflect only Step 1
• A new test case version should be generated with the updated step count

Actual Behavior

• TestManager continues to display the previous test structure (both Step 1 and Step 2)
• No new test version is generated
• The artifacts field shows "replaced":false in the test summary
• The commit field shows "reason":"readonly_fallback" instead of a successful commit

---

Test.md Files at Each Step

1. Initial Test Case Created

File:

---
mode: testing
max_steps: 30
target: chrome
headless: true
---

# Session: step_remove

## Step 1
open google.com

Command:

kane-cli run "open google.com" --name step_remove --headless --agent

Complete Logs:

{"type":"recording_state","enabled":true,"session_id":"2011f172-554a-4dbe-88b4-5e23acf62847","persist":true,"session_name":"step_remove","test_path":"C:\\Users\\priyanshug\\.testmuai\\tests\\step_remove_test.md","output_path":"C:\\Users\\priyanshug\\.testmuai\\tests\\output-step_remove"}
{"type":"bifurcation","flows":["Navigate to https://google.com."],"count":1}
{"step":2,"status":"running","remark":"Step 1"}
{"step":2,"status":"done","remark":"navigate: Navigate to https://google.com"}
{"step":3,"status":"running","remark":"Step 2"}
{"type":"run_end","status":"passed","summary":"The run finished and ended on the Google homepage (https://www.google.com/).\nNo specific objective, searches, or extracted details were provided or recorded, so no additional actions or results can be summarized.","one_liner":"opened Google on google.com","final_state":{"url":"https://www.google.com/"},"reason":"Objective completed","duration":27.2,"bifurcated":false,"total_runs":1,"context":{"memory":{},"variables":{},"pointer":"(passed) opened Google on google.com"},"credits_consumed":0.9557249999999999,"session_dir":"C:\\Users\\priyanshug/.testmuai/kaneai/sessions\\2011f172-554a-4dbe-88b4-5e23acf62847","run_dir":"C:\\Users\\priyanshug/.testmuai/kaneai/sessions\\2011f172-554a-4dbe-88b4-5e23acf62847\\runs\\0","result_code":100,"reason_code":"success.complete","per_flow_metadata":[{"result_code":"100","reason_code":"success.complete","error_message":null,"summary":"The run finished and ended on the Google homepage (https://www.google.com/).\nNo specific objective, searches, or extracted details were provided or recorded, so no additional actions or results can be summarized.","one_liner":"opened Google on google.com","credits_consumed":0.9557249999999999}],"test_url":"https://test-manager.lambdatest.com/projects/01KT4A6T62SKG096YANK1JXJFK/test-cases/01KW203VPPVAAR5FPGAS9VFRDF/dashboard/share/US_Q6XJR94EEHD2XQ6JDLXUI695P0CC7DRVCQEWPAJGM9A2E6KOW5FEIGXE5VDRHOOK?type=summary&agentView=true&fqdn=summary-page"}

Status: ✅ Test created successfully

---

2. New Step Added

File:

---
mode: testing
max_steps: 30
target: chrome
headless: true
---

# Session: step_remove

## Step 1
open google.com

## Step 2
open hello.com

Command:

kane-cli testmd run step_remove_test.md --agent --headless

Complete Logs:

{"type":"test_md_lock_state","phase":"acquired","scope":"preemptive"}
{"type":"test_md_step_start","step_index":1,"heading":"Step 1","ref":null}
{"type":"run_start","objective":"open google.com","timestamp":1782478855}
{"type":"step_event","index":0,"event":"replay_started","detail":"replaying 1 actions","recording_length":1}
{"type":"step_start","index":1,"objective":"Step 1"}
{"type":"step_event","index":1,"event":"screenshot","detail":"https://kaneai-playground.lambdatest.io/"}
{"type":"step_event","index":1,"event":"action","detail":"Navigate to https://google.com","action_type":"navigate","success":true}
{"type":"step_event","index":1,"event":"reasoning","detail":"Navigate to https://google.com","action_type":"navigate"}
{"type":"step_end","index":1,"status":"passed","duration":2.4,"summary":"navigate: Navigate to https://google.com"}
{"type":"run_end","status":"passed","summary":"","reason":"replay completed","duration":7.08,"final_url":"https://www.google.com/","actions_executed":1,"screenshot_path":"C:\\Users\\priyanshug\\.testmuai\\kaneai\\sessions\\4ebcc231-af2b-4c59-ad5f-b4bd3ba92d6d\\runs\\0\\run-test\\screenshots/step_001.png","run_dir":"C:\\Users\\priyanshug\\.testmuai\\kaneai\\sessions\\4ebcc231-af2b-4c59-ad5f-b4bd3ba92d6d\\runs\\0\\run-test","context":{"memory":{},"variables":{},"pointer":""},"variables_out":{},"store_out":{}}
{"type":"test_md_step_end","step_index":1,"status":"passed","duration_s":7.08,"ref_kind":null,"inlined_count":null,"failed_sub_step_index":null}
{"type":"test_md_step_start","step_index":2,"heading":"Step 2","ref":null}
{"type":"bifurcation","flows":["Navigate to https://hello.com."],"count":1,"is_single_flow":true,"names":["Open Hello Site"]}
{"type":"run_start","objective":"","timestamp":1782478885}
{"type":"step_start","index":1,"objective":"Step 1"}
{"type":"step_event","index":1,"event":"screenshot","detail":"https://hello.com/"}
{"type":"step_event","index":1,"event":"action","detail":"Navigate to https://hello.com","action_type":"navigate","success":true}
{"type":"step_event","index":1,"event":"reasoning","detail":"Navigate to https://hello.com","action_type":"navigate"}
{"type":"step_end","index":1,"status":"passed","duration":0,"summary":"navigate: Navigate to https://hello.com"}
{"type":"step_start","index":2,"objective":"Step 2"}
{"type":"step_event","index":2,"event":"screenshot","detail":"https://hello.com/"}
{"type":"describe_trigger","index":2,"screenshot_path":"C:\\Users\\priyanshug/.testmuai/kaneai/sessions\\4ebcc231-af2b-4c59-ad5f-b4bd3ba92d6d\\runs\\1\\run-test\\screenshots/step_002.png","objective":"","recent_actions":[{"step":1,"action_type":"navigate","instruction":"Navigate to https://hello.com","success":true}]}
{"type":"run_end","status":"passed","summary":"Opened the website hello.com.\nNo specific task or objective was provided, and no additional actions were recorded.\nEnded on https://hello.com/.","one_liner":"opened hello.com","final_state":{"url":"https://hello.com/"},"reason":"Objective completed","duration":17.7,"bifurcated":false,"total_runs":1,"context":{"memory":{},"variables":{},"pointer":"(passed) opened hello.com"},"credits_consumed":0.79315,"session_dir":"C:\\Users\\priyanshug/.testmuai/kaneai/sessions\\4ebcc231-af2b-4c59-ad5f-b4bd3ba92d6d","run_dir":"C:\\Users\\priyanshug/.testmuai/kaneai/sessions\\4ebcc231-af2b-4c59-ad5f-b4bd3ba92d6d\\runs\\1","result_code":100,"reason_code":"success.complete","per_flow_metadata":[{"result_code":"100","reason_code":"success.complete","error_message":null,"summary":"Opened the website hello.com.\nNo specific task or objective was provided, and no additional actions were recorded.\nEnded on https://hello.com/.","one_liner":"opened hello.com","credits_consumed":0.79315}]}
{"type":"test_md_step_end","step_index":2,"status":"passed","duration_s":17.7,"ref_kind":null,"inlined_count":null,"failed_sub_step_index":null}
{"type":"test_md_summary","overall_status":"passed","duration_s":57,"steps":{"total":2,"passed":2,"failed":0,"skipped":0,"replay_decisions":1,"author_decisions":1},"retries":{"triggered":false},"commit":{"committed":false,"reason":"not_committed"},"artifacts":{"replaced":true},"upload":{"performed":true,"succeeded":true},"cancelled":false}
{"type":"test_md_done","overall_status":"passed","duration_s":57,"session_id":"4ebcc231-af2b-4c59-ad5f-b4bd3ba92d6d"}

Status: ✅ Step 2 added successfully
Key Indicator: "artifacts":{"replaced":true} — Test updated in TestManager

---

3. After Deleting Step 2

File:

---
mode: testing
max_steps: 30
target: chrome
headless: true
---

# Session: step_remove

## Step 1
open google.com

Command:

kane-cli testmd run step_remove_test.md --agent --headless

Complete Logs:

{"type":"test_md_lock_state","phase":"acquired","scope":"preemptive"}
{"type":"test_md_step_start","step_index":1,"heading":"Step 1","ref":null}
{"type":"run_start","objective":"open google.com","timestamp":1782478978}
{"type":"step_event","index":0,"event":"replay_started","detail":"replaying 1 actions","recording_length":1}
{"type":"step_start","index":1,"objective":"Step 1"}
{"type":"step_event","index":1,"event":"screenshot","detail":"https://kaneai-playground.lambdatest.io/"}
{"type":"step_event","index":1,"event":"action","detail":"Navigate to https://google.com","action_type":"navigate","success":true}
{"type":"step_event","index":1,"event":"reasoning","detail":"Navigate to https://google.com","action_type":"navigate"}
{"type":"step_end","index":1,"status":"passed","duration":1.7,"summary":"navigate: Navigate to https://google.com"}
{"type":"run_end","status":"passed","summary":"","reason":"replay completed","duration":2.64,"final_url":"https://www.google.com/","actions_executed":1,"screenshot_path":"C:\\Users\\priyanshug\\.testmuai\\kaneai\\sessions\\91d4749d-0e04-45c8-be55-c800f3cb4974\\runs\\0\\run-test\\screenshots/step_001.png","run_dir":"C:\\Users\\priyanshug\\.testmuai\\kaneai\\sessions\\91d4749d-0e04-45c8-be55-c800f3cb4974\\runs\\0\\run-test","context":{"memory":{},"variables":{},"pointer":""},"variables_out":{},"store_out":{}}
{"type":"test_md_step_end","step_index":1,"status":"passed","duration_s":2.64,"ref_kind":null,"inlined_count":null,"failed_sub_step_index":null}
{"type":"test_md_summary","overall_status":"passed","duration_s":7,"steps":{"total":1,"passed":1,"failed":0,"skipped":0,"replay_decisions":1,"author_decisions":0},"retries":{"triggered":false},"commit":{"committed":false,"reason":"readonly_fallback"},"artifacts":{"replaced":false},"upload":{"performed":true,"succeeded":true},"cancelled":false}
{"type":"test_md_done","overall_status":"passed","duration_s":7,"session_id":"91d4749d-0e04-45c8-be55-c800f3cb4974"}

Status: ❌ BUG — Step removed locally, but NOT reflected in TestManager
Key Issue Indicators:

• "commit":{"committed":false,"reason":"readonly_fallback"} — Commit failed
• "artifacts":{"replaced":false} — No artifact replacement occurred
• "upload":{"performed":true,"succeeded":true} — Upload reported success, but NOT reflected in TestManager
• Session ID: 91d4749d-0e04-45c8-be55-c800f3cb4974

---

Root Cause

The readonly_fallback error on step removal indicates that the commit logic does not properly handle step deletions. While additions work fine ("artifacts":"replaced":true), removals fail silently with a fallback mode that prevents the test structure from being updated in TestManager.