How to merge some anime style into Pony without losing too much of its composition?
Just pay attention.
https://files.catbox.moe/roq6ya.pngLong explanation: Self-attention and cross-attention determine most of the content of the gen
Merging only the Q and K ensures stability of the merge
Swapping Q and K of the outermost layers affects the style towards the other model
Merging the rest of the attention is subject to further experiments, but as long as only Q and K are merged, Pony's composition seems to come through
Merging Q and K adds partial understanding of the TE knowledge of the other model (quality tags from anime models work in synergy with Pony quality tags)
I was experimenting with this for a while, the B-LoRA paper gave me the final clue about style vs composition
