Random Selfplay

[vwxyzjn]

This PR prototypes fictitious selfplay.

Known issue: still needs to change the following variables to exclude data from agent2

        b_obs = obs.reshape((-1,) + envs.observation_space.shape)
        b_logprobs = logprobs.reshape(-1)
        b_actions = actions.reshape((-1,) + action_space_shape)
        b_advantages = advantages.reshape(-1)
        b_returns = returns.reshape(-1)
        b_values = values.reshape(-1)
        b_invalid_action_masks = invalid_action_masks.reshape((-1,) + invalid_action_shape)

[vwxyzjn]

Running an experiment here: https://wandb.ai/costa-huang/gym-microrts/runs/x9y055w6

[kachayev]

Should we prune agents that have chance of winning < threshold? With the current implementation, it seems to that number of saved agent only grows over time, right? 🤔

[kachayev]

Okay, I have much large question here. Will formulate it properly and will ask on Discord, so we can discuss 😀

[vwxyzjn]

Haha yeah this first implementation is very crude. OpenAI Five does it by sampling opponents probabilistically according to their trueskill.

[vwxyzjn]

Per discussion with @kachayev, it turns out the implementation in this repo is definitely not fictitious selfplay, which also has a supervised learning component. Instead, this PR implements what I call "Random Selfplay" where the agent plays against a random past version of self.

[vwxyzjn]

Closed in favor of #57