Hello friends, its your friendly mergeanon, here to tell you about model merging.
Basic tier:
This is your standard merging in WebUI. In this area you have access to Weighted Sum and Add Difference. This level of merging is actually pretty good for basic merging, and you can achieve good results here with experimentation, but you can only do very high level merging, and your results may end up being limited depending on what you are trying to do.
Simple Concept and Character models are usually merged this way by doing Add difference to remove the model it was trained on and then adding the remains to the new model.
This is actually how a lot of model merges used to be done, and a lot of the model recipes with HLL were done this way. NOTE: Only use WebUI based UIs for this, as certain implementations may not be done properly, and almost everything was done with this, so you may not be able to recreate model hashes with the merges otherwise.
Advanced Tier:
This is where you get into supermerger. Supermerger is like the standard merger but on steroids, you can do Block Weight merging as well with this, but that is for a later tier. With Supermerger, you get into more advanced merging modes and calculation modes. With things like Cosine Similarity allowing you to merge the models based on the similarity of the two models tensors, combined with the amount of the model you want as well. Or merging methods like Tensor sum, where instead of weighting one of the tensors and adding it in order to merge, you are instead replace part of the tensor of the first model with part of the tensor of the second model, where the percentage replaced is based on the merge percentage/weight chosen. These model merging methods provide a novel way to guarantee that the structure or artstyle of a model is maintained with the merge. This allows the easy creation of models that blend Photorealism and Anime. But this could be used for any kind of artstyle as well. For tensor sum, this is more about a a way to try and not over add to a model, which means you won't burn your model from repeated merges since you aren't doing an additive process, but rather a substitution process. (Note: Cosine Similarity is broken into an A and B version, both do the same thing, but prioritize the A model's structure for the A verison, and Second models structure for the B version.) You can find more about Cosine Similarity here:
https://github.com/hako-mikan/sd-webui-supermerger/issues/33 (There is some cool stuff about other merge modes like trainDifference, which seem interesting as well.) See pic related for what TensorSum is doing on a tensor level.
Next two Merge tiers will be related to block weight merging, with an aside on Aesthetic classifiers, which will be my next post (once I finish writing it of course). Block Weight merging can also be used with all of the advanced merging methods found in Supermerger, so what I said above could also apply there as well if you so choose.
