Notes

GitHub - M4cs/yourAI: GPT-2 Discord Bot And Steps To Train Something Like You

I finetuned gpt2 models on a RTX2080Super. I set the clock to 1550MHz and used all of the memory. I was able 10k iterations with the 355M model (Medium size). This will vary, but you can still fine tune enough to create a functional bot in 5k iterations by using a CPU.



For a chatbot like this I recommend you try to reach at least 8500-10000 iterations. If the model achieves an unchanging learning rate or NaN difference, it might have trained too much. You will then encounter weird bugs. This is why you should save every 500-1000 Iterations so that you have snapshots to go back to.



My discord bot runs with 8560 iterations using the 355M on a dataset with 1375352 tokens (about 500k individual chat messages from discord). The messages are trained from a single discord channel which included myself and 4 friends. This resulted in a very specific personality set the bot picked up where it would make jokes that only we would pick up and call out each other even when others spoke to it. This isn’t something you want unless it’s all on one server.



To counter this, you should have many conversations that you participate in with many people. Make sure you have lots and lots of backand forth conversation in your final dataset. This is a good place to have general and off-topic conversations.



Here are some things to keep an eye on



- Links. You probably want to parse these from your dataset or your bot will start to send randomly generated links that look real. Sometimes it can send a real link and this can be funny but it's a rare occasion. - Bot messages and server messages. Bot messages may sound like bot messages. If you don't wish for your life to sound robotic, you shouldn't allow bots into your life. Parse these out along with discord's Joined the Server. messages. - Language. To make your bot nice, avoid putting toxic stuff in the databank. GPT-2 doesn't care what your views are or how moral they are. AI doesn't have feelings. If you are toxic in discord your bot is going to be toxic.



Getting Started



VIRTUALENV - Use it! !



Locally, first clone gpt-2–simple's source codes.



Setup a virtualenv in the repo



Download the models that you need:



Models available in: 124M to 355M, 774M to 1.5B.



**You won't be able to run any more than the 355M on a graphics card. If you don't own a Titan or Quadro, don't waste time downloading higher memory models. The 774M may work occasionally if you're using Colab. **



Create a Dataset



You should use a discord chat exporter like this and export it to txt. The format for my dataset was as follows:



It was clear that it had some contextual conversation, but this is what it was in.
Discord servers


Perfect tuning the model



I recommend the smaller 124M model if you are doing finetuning on a processor. You can try the 355M model if you have a powerful GPU, such as an RTX card or a high-level GTX/R card.



Here is how to tweak a model:



This can be run repeatedly and it will train your machine and pick up from where you left it off. If you have a different run_name, it will start anew one.



You can fine tune your model until it is around 8k-10k iterations.



The Discord Bot Part



Create a discord bot on the discord site.



First, go to the Applications Section in the developer panel. You will need to create a brand new app.


Here's my website: https://discord-servers.co/