[feature request] callback returning wins (or winrate) vs another agent

[kevinu3d]

When training with different reward functions it's hard to compare 2 bots. A callback capable of running n games between current agent and another would prove useful to measure progress.

I will look into it but if someone knows how to do that help is welcome.

[glmcdona]

Good idea.

This here is probably the closest existing code to it:

LuxPythonEnvGym/luxai2021/env/lux_env.py

Line 14 in c5d4c16

class SaveReplayAndModelCallback(BaseCallback):

And example usage:

LuxPythonEnvGym/examples/train.py

Line 114 in c5d4c16

SaveReplayAndModelCallback(

You can also look at the built-in eval callback:

LuxPythonEnvGym/examples/train.py

Line 136 in c5d4c16

EvalCallback(env_eval, best_model_save_path=f'./logs_{run_id}/',

One last thing to look at is this old example code that logged custom game information to the tensorboard. It's was a bit of a hackjob because of the agent having been reset before tensorboard grabs data: PR #86.