Quoted By:
Any sauce for how the guy rigged it up?
Seems like he's running the OpenAI minecraft model from last summer (the one that was trained on youtube and tries to make a diamond pickaxe) for the footage. Cool how he got it on a private server though, didn't know it could run outside their simulator. It also appears to sometimes do things beyond that (crafts unneeded things like pressure plates, punches players), which could be accidents or unreinforced behaviors sneaking in from the dataset, but could also be him feeding some inputs manually. He could also be swapping in different models from OpenAI because they also had one for building a house, and in some streams she seems to be trying to do that.
Then for the chat interaction, GPT-3 with some prompts plugged in. Or maybe finetuned it himself with some chat logs scraped from twitch. Then bolted that to TTS and Live2d and twitch api. It also reads the minecraft chat and responds to a few events in game (I saw it mention dying, getting punched), which might be coming from the game or he might be pushing buttons. Also has a SHUT IT DOWN button when she gets a non-kosher prompt from chat.
Sounds like the osu model is all his though.
Guess I'll join his discord though it doesn't seem like he posts anything technical.
