Flaky detect #1420
Flaky detect #1420
Conversation
|
|
||
| val executedMainActionToFile = "target/em-tests/FlakinessDetectEM/org/foo/FlakinessDetectEM.kt" | ||
|
|
||
| args.add("--minimize") |
There was a problem hiding this comment.
for new E2E tests, should use setOption
| '[' -> { | ||
| try{ | ||
| // This would be run if the JSON contains an array of objects. | ||
| val list = Gson().fromJson(bodyString, List::class.java) |
There was a problem hiding this comment.
whe should NOT use Gson for new code. Gson has major design flows (and really nasty "features" on by default...). use Jackson for new code (at a certain point, we will need to get rid off Gson from EM codebase, but it is not high priority)
| try { | ||
| val resContents = Gson().fromJson(bodyString, Map::class.java) | ||
| handleAssertionsOnObject(resContents as Map<String, *>, lines, "", bodyVarName) | ||
| val flakyMap = flakyBodyString?.let { Gson().fromJson(it, Map::class.java) as Map<String, *> } |
There was a problem hiding this comment.
see previous comment on Gson
| val value = bodyIsString(s,GeneUtils.EscapeMode.BODY, responseVariableName) | ||
|
|
||
| val fs = flakyBodyString?.trim() | ||
| if (fs == null || fs == s) |
There was a problem hiding this comment.
always use {} brackets, even for single statement in a if
| try { | ||
| val number = s.toDouble() | ||
| handleAssertionsOnField(number, lines, fieldPath, responseVariableName) | ||
| // TODO only support flaky for JVM |
There was a problem hiding this comment.
why only for JVM? base flakiness handling should be in Black-Box as well
| minimizer.simplifyActions() | ||
| val seconds = minimizer.passedTimeInSecond() | ||
| LoggingUtil.getInfoLogger().info("Minimization phase took $seconds seconds") | ||
| } else if (config.detectFlakiness){ |
There was a problem hiding this comment.
shouldn't be done here, as it will miss all the test cases generated after the "search".
Look at Main class with:
//apply new phases
solution = phaseHttpOracle(injector, config, solution)
solution = phaseSecurity(injector, config, epc, solution)
maybe flakiness should be a new "phase" (recorded in ExecutionPhaseController), which is executed last
| } | ||
| """.trimIndent() | ||
| assertEquals(expectedLines, lines.toString()) |
There was a problem hiding this comment.
this is a very brittle assertion. it will start failing as soon as we make any cosmetic change in the generation of test.
could rather check for presence of // Flaky value (which anyway should in a const val shared between production code and tests), and/or number of // in the generated code
| } | ||
| """.trimIndent() | ||
| assertEquals(expectedLines, lines.toString()) |
|
|
||
| @Experimental | ||
| @Cfg("Specify whether to detect flaky assertions during post handling of fuzzing.") | ||
| var detectFlakiness = false |
There was a problem hiding this comment.
not only detect, but also handle, isn't it? maybe rename into handleFlakiness. also, in Cfg documentation, specify that flaky assertions will be commented out
| @@ -0,0 +1,15 @@ | |||
| package com.foo.rest.examples.spring.openapi.v3.flakinessdetect | |||
There was a problem hiding this comment.
when dealing with PR where test output code is changed/upgraded, should also have 1 BB test under core-tests/e2e-tests/spring/spring-rest-bb, as those automatically check all output formats (eg Python and JS)
Detect flakiness during post-handling of fuzzing
Comment out assertions at the field level.
Currently support only for JUnit with white-box mode