Looks like adding norms to LoRA is a good idea -
https://github.com/kohya-ss/sd-scripts/discussions/294#discussioncomment-9977651First row - without norm layers, 2 row - train norms in unet, 3 row - train norms in te and unet
Another example: LoRA with norms > BOFT without norms
https://litter.catbox.moe/3jth90.jpg