Test cases that do not fail test (and should as indicated in test name testFailTransferFromNotAuthorized) #412
Description
I opened this issue earlier in foundry-rs regarding an issue running solmate's tests in the benchmarks
I think this should be addressed in solmate tests too, thing is that there are cases that do not fail test (and should as indicated in test name
testFailTransferFromNotAuthorized), like using counterexample above in an unit test
foundry-rs/foundry#7531 (comment)
function testFailTransferFromNotAuthorized_counterexample() public {
address sender = address(0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496);
address receiver = address(0x000000000000000000000000000000000000089a);
uint256 id = 31;
uint256 amount = 3642;
amount = bound(amount, 1, type(uint256).max);
token.mint(sender, id, amount);
token.transferFrom(sender, receiver, id, amount);
}yields failure because test passes
Ran 1 test for src/test/ERC6909.t.sol:ERC6909Test
[FAIL. Reason: assertion failed] testFailTransferFromNotAuthorized_counterexample() (gas: 43212)
Logs:
Bound Result: 3643
Traces:
[559113] ERC6909Test::setUp()
├─ [504137] → new MockERC6909@0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f
│ └─ ← [Return] 2518 bytes of code
└─ ← [Stop]
[47424] ERC6909Test::testFailTransferFromNotAuthorized_counterexample()
├─ emit log_named_uint(key: "Bound Result", val: 3643)
├─ [25249] MockERC6909::mint(ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], 31, 3643)
│ ├─ emit Transfer(caller: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], from: 0x0000000000000000000000000000000000000000, to: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], id: 31, amount: 3643)
│ └─ ← [Stop]
├─ [25892] MockERC6909::transferFrom(ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], 0x000000000000000000000000000000000000089a, 31, 3643)
│ ├─ emit Transfer(caller: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], from: ERC6909Test: [0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496], to: 0x000000000000000000000000000000000000089a, id: 31, amount: 3643)
│ └─ ← [Return] true
└─ ← [Stop]
Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 994.36µs (159.48µs CPU time)
Ran 1 test suite in 767.02ms (994.36µs CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)
Failing tests:
Encountered 1 failing test in src/test/ERC6909.t.sol:ERC6909Test
[FAIL. Reason: assertion failed] testFailTransferFromNotAuthorized_counterexample() (gas: 43212)whereas if rename test to testTransferFromNotAuthorized_counterexample it is passing
To prevent against panics when running forge benches probably here we should not assume execution is always success but just execute
https://github.com/foundry-rs/foundry/blob/d94e3c631e2da7756af46c70f8f58b75563b7013/crates/forge/benches/test.rs#L17-L19
The cache/fuzz/failures cannot be found because bench clones and runs project in a temp dir which is wiped out after execution
Originally posted by @grandizzy in foundry-rs/foundry#7531 (comment)