Quoted By:
About HLL4.
I said I'll post it soon, but instead I was experimenting with small test models. I haven't started retraining it yet.
I'll post a new model eventually, but probably not HLL4 as it was originally planned.
conclusions after testing some random stuff:
- Great results with Lion - it learns very fast and generalizes well if you use bs >=64, low encoder LR, high decay as recommended by authors, but it likes to fry everything at low batch sizes. 8-bit Lion should be even faster. With low batch sizes using old adamw is safer
- Looks like most efficient way to train a model like HLL is to chain 2-3 trainings: 1: high bs low res with lion to learn general features and style, 2: low bs higher res with adam to improve details.
- Model learns from other AI-generated images faster than from "real" images?
- Adding just 10% of thread archives as reg images is enough to teach model how to imitate AOM/pesto look. Should be easy to get more interesting '"default style" with different reg images.
