PMC

We attempt to use the same data collected for MBS training and apply it to off-policy learning, mixing off-policy learning with on-policy exploration in the MBS pseudo-simulation environment. We show the results of running multiple agents in parallel. The graph shows the learning performance of (a) learning without an off-policy data environment, (b) with 1 off-policy environment and 2 on-policy environments, (c) with 2 off-policy and 1 on-policy, (d) with all 3 environments off-policy.

Each panel shows a simulation of 20 trials for which the first nine trials were off-policy and the subsequent trials were on-policy, during which the action with the highest probability was selected. Trials marked with green ribbons were off-policy (actions were fed to the model), whereas the trials marked with blue ribbons were on-policy (actions were selected by the model). The ribbon below each panel shows the actions that were fed to the model (for the first 9 trials), and the actions that were selected by the model (on the subsequent trials). During off-policy trials, the sequence of actions that was fed to the model was R, R, R, R, R, R, L, R, L. See the text for interpretation.

Experiment design with a framework for off-policy evaluation and model and policy selection through cross-OPE evaluation and clinical policy inspection

Ablation study on GP in on-policy GAIL. We see that the agent is still able to learn policies achieving peak performance even without GP, in contrast to the off-policy version of the algorithm. In the most difficult environment of the MuJoCo suite (cf. Table ), Humanoid, GP achieves best performance. Runtime is 12 hours