Add a format fuzzer for compiled dynamic

[magnumripper]

We could add a "format fuzzer" for testing, that randomizes a hundred or so things like keccak_512(md4($p).haval256_4($s)) from a list of primitives and runs a --test=0 on each. Should be super trivial to implement. We would use a fixed (optionally set) random seed.

[jfoug]

Why would we use a fixed seed ? Do you meed a seed within the perl random ?

[frank-dittrich]

A seed that can be specified is needed to reliably reproduce a problem that only occurs under certain conditions.
So, we need something similar to jtrts.pl's already existing -random -seed <n>

[jfoug]

This will be done initially (might be kept that way) as a stand alone perl. We can hook it into jtrts.pl, and simply have it spawn this .pl script. The I really do not want to add other items to the TS. The only reason it is good to do, is that the TS has some underlying code that would be nice, but here, I am not sure we need it. We are simply looking for failures to compile and failures at cmp_all() and of course crashes. That is the 2(3) points where dyna compiler will have failures.

I see the point about these 2 flags, and the code will be done that way.

[jfoug]

This simple 'fuzzer' start already found problems, lol Skein384 is not integrated properly.

#!/usr/bin/perl
# @dynamic=expr@ fuzzer

my @funcs = ();

add_funcs ('md4', 'md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'gost', 'tiger', 'whirlpool',
          'ripemd128', 'ripemd160', 'ripemd256', 'ripemd320', 'haval128_3', 'haval128_4', 'haval128_5',
          'haval160_3', 'haval160_4', 'haval160_5', 'haval192_3', 'haval192_4', 'haval192_5',
          'haval224_3', 'haval224_4', 'haval224_5', 'haval256_3', 'haval256_4', 'haval256_5', 'md2',
          'skein224', 'skein256', 'skein38', 'skein512', 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', 
          'keccak_256', 'keccak_512');

my $i; my $j; my $k;
for ($i = 0; $i < scalar(@funcs); ++$i) {
    my $f = $funcs[$i];
    next if $f =~ /_raw$/;
    my $s = `../run/john -test=0 -form=\'dynamic=$f(\$p)\'`;
    print "-form=\'dynamic=$f(\$p)\' failed\n" unless $s =~ / PASS/;
    # print $s;  # we need to handle this with -q type syntax.
}

sub add_funcs {
    my $f = shift;
    while (defined $f) {
        push (@funcs, $f);
        push (@funcs, uc $f);
        push (@funcs, $f.'_raw');
        push (@funcs, $f.'_64');
        push (@funcs, $f.'_64c');
        $f = shift;
    }
}

$ ./dyna_expr_fuzzer.pl
-form='dynamic=skein38($p)' failed
-form='dynamic=SKEIN38($p)' failed
-form='dynamic=skein38_64($p)' failed
-form='dynamic=skein38_64c($p)' failed

[jfoug]

Nope, skein384 is fine. I left the 4 off the add_funcs string in the fuzzer, lol.

[jfoug]

This script here will keep me fixing stuff for a while, lol. I will check this in as a generation-0 fuzzer for dyna. There is nothing random, it simply walks each type down a very order manner.

[magnumripper]

Cool. Lots of immediate RoI.

[jfoug]

I knew that these hash(hash(v) . something . hash(v)) were going to fail. I have not gotten code to properly parse those types yet. But they are in fuzzer, and it should now push me to start on some of the issues like those.

[frank-dittrich]

In addition to the fuzzer, could we add at least one hard coded --dynamic=... test to the default test suite run?