Notes

The world model might look like it trains/converges faster when we look at real steps, but if it really uses double that amount of step (or whatever it is) while training, does it really run faster? Or if it takes more real-time/computer power to converge, does it really run faster? Not really. So, how can I measure these other metrics? What's the standard way of measuring a world model against a model-free rl? Or even, what's the way to measure a world model against a different world model, but where one of the models might use more rollouts/computation than the other?



-------------------------------------



Quick question, do you know the Continual-Dreamer AI? I believe that this is very similar to ours. Though, looking at it, it doesn't seem like they use multiple world models, but rather 1 to learn all the different tasks. How are they able to do this without the world model forgetting? Is it because the actual physics/rewards aren't being flipped on their head, rather they are simply learning a new task? So the contiual-dreamer probably wouldn't be able to manage in our setup, right? Or do you think it could?

Wait, could you humor me for a second. Do you think there's a way that we could use one world model to predict multiple regimes?

My original thinking was that if we tried to do this, then it would simply suffer the catastrophic forgetting itself, and forget what the original regime was. My first thought was to make it so that the world model outputs a latent variable, which would be compatible with both regimes, and a decoder would decode based on the current regime before giving the result to the PPO planner. But then I realized that this is just pushing the problem further, as the latent value decoder would then be the one to suffer the catastrophic forgetting.

Though, this is where you need to humor me, as I have a couple of new thoughts. The first is, what if we use a Mixture of Latent Value Decoders instead of a MoWM? These wouldn't even have to be transition models, but simply a model that takes the latent value from the world model and predicts the one-hot state/scalar reward it represents for regime 0, then a second decoder that predicts the one-hot state/scalar reward it represents for regime 1. Would that be possible? And wouldn't creating a new decoder be cheaper than creating a new world model for every regime?

Then, my other new thought was, could we use the fact that we can guess the current regime PLUS our constant dreaming, so that the world model itself could learn through dreaming? Then we can give the world model the current predicted regime as an additional input, so that it too, just like the PPO planner, can learn multiple regimes. Or it can do this in other ways, like outputting a list where Output[0] is what it'd guess for regime 0, Output[1] for regime 1, etc. What do you think of this? It seems like it will be hard to fine tune and get working, but could it be possible? One issue would be that hallucinations would exponentially increase errors, but we can try to prevent this by using our current world model variable has_mastered, and we only train the world model on past regimes if it's mastered them (and not roll-out too long of horizons). I don't know, I feel like I might still be missing some flaw. Be 100% honest in your answer.