Development of secret testing strategy

[JohnnyJayJay]

To ensure people are actually solving the problem rather than abusing the visibility of challenge test cases to write code that only works for the tests, there need to be secret test cases that are run upon submission.

There are different ways secret test cases could work:

• Enumerated – like public test cases, secret test cases would be specified explicitly in the challenge file, making them effectively transparent for anyone reading them.
• Generative – Challenges would somehow describe the format of their input and based on this, valid inputs could be generated. With these generated inputs, submissions could be tested in one of the following ways:
  • if we have a known solution available, we could compare the output of this solution with the output of the submission
  • if we have a validator (a function taking an (input, output) pair and deciding if output is a correct solution for input), we could do property-based testing

It would probably be most beneficial to have generative tests for our secret tests, as this would reduce the burden on challenge contributors and enable us to be more confident that a solution doesn't just work for the explicit test cases. As a "downside", we would have to add the requirement to specify the input format – however, this description could be used in other contexts as well (see #2).

Whether we should use validators (possibly a collection of validators per challenge that each only test a single property) or a complete solution is a different discussion. Specifying the relationship between inputs and outputs may be easier, more obvious and less prone to errors. It is also more akin to how we typically unit test actual code.
On the other hand, full-blown-solutions are very straight forward to integrate – we could send them to the execution service in the same way as all other submissions.
For validators/property definitions, it would likely make the most sense to write them in some embeddable language that we can call directly from our backend code.

[JohnnyJayJay]

For validators/property definitions, it would likely make the most sense to write them in some embeddable language that we can call directly from our backend code.

Here, it might make sense to look at GraalVM's embeddable javascript

[JohnnyJayJay]

Generally, for solution validators we should consider languages that are

• approachable by most users (at least in this small context)
• easily embeddable in Java
• data-oriented (ideal for specifying input-output relations)

Ticking all these boxes, especially the first one, may be rather difficult. It would certainly be a lot easier (albeit less clean) if people could just submit their own solutions along with challenges. Some proofs of concept are needed and some examples should be written for research.

[NyCodeGHG]

For validators/property definitions, it would likely make the most sense to write them in some embeddable language that we can call directly from our backend code.

Here, it might make sense to look at GraalVM's embeddable javascript

luajava could also be an alternative if we decide against using javascript

[JohnnyJayJay]

For validators/property definitions, it would likely make the most sense to write them in some embeddable language that we can call directly from our backend code.

Here, it might make sense to look at GraalVM's embeddable javascript

luajava could also be an alternative if we decide against using javascript

I think I've settled on not doing the whole validation thing. Could be cool from a programming perspective, but just letting contributors write a solution in a language of their choice is much, much more user friendly.