Quoted By:
Matrixfag here. Thought that I would give a bit of a Pyg update. Sorry for the radio silence, I've been too preoccupied to even lurk here at this point, let alone the /g/ threads. TL;DR is this: we're working on Pyg-70B and 70B merges and getting the website up and running. Two More Weeks(tm) on that, but more good news should be coming soon. In addition we've kind of been moving quietly in the background, helping other open source folks out with stuff and making inroads. We've also been working tirelessly on our own Aphrodite inference engine, which is looking pretty good right now. Not only that, but we're busy preparing for our own ventures into foundational models. Me, personally, I've been busy with trying to improve on LLM architectures. Pic related. I don't know if it's something I'd like to scream as an innovation of mine yet, but I think I found out something cool. The graph shows the loss curve (lower means better) of a tiny 100M model used for testing, so this isn't an official "we've got a model out and ready" but hopefully it does keep your smiles. I'll be back in a few hours to answer any questions.
(Also, jesus fuck, it's nearly been a year since the CAI shitshow first began - tomorrow will be the 1 year anniversary since 980's thread... and I guess this thread by proxy. Happy early birthday, everyone...)
