I think it depends on the collaborator. Basically, most of us don't know the Holostars. If this is a popular talent on the JP side, it won't be that much of a problem.
If it's a comedian like Koge, who has collaborated with many vtubers, or a male voice actor, it won't be that big of a problem.
Unlike the days when it was a vtuber management company, Hololive is now proud to be a vtuber idol production company. Therefore, they have a lot of idol fans and are very sensitive to heterosexual interactions.
It may not be such a big scratch on the part of Hololive, but it could be fatal for Holostars. It is the nature of idol fans to attack anyone they are involved with. Some of the members, Lucia and Pecola, have a lot of stiff competition because they are operating with their idol side all over the place, and the repercussions will be immense.
In general, JP idol fans are attracted to those who can interact with them, so they don't settle for EN or ID. JP members are more susceptible to the repercussions of heterosexual interactions.
Tokinosora, Matsuri, and Fubuki are generally less gachikoi and less susceptible to collaboration in terms of popularity. However, the number of views and viewers will be affected.
It may seem unreasonable, but one of the reasons why Hololive has become so big is undoubtedly influenced by the idol aspect. As long as that's what it stands for, there's no escaping its unreasonableness.
