>>1719805not him but that's not it anon. They're not inherently hard to use or sketchy, you just need practice using them. This only says something about you.
The real major downsides are specific things.
And there are specific major benefits too, like that you can soft-shift the front (shift the FD to a position where the cage would rub, then shift it over a little more when the shift is completed), so you will never drop a chain.
And that you can swoop the entire cassette with 1 throw of the shift lever.
And that you can shift front and rear with the same hand if you're holding food in the other hand.
And that you can switch from indexed to friction mode for the rear if your derailer/hanger is bent in a crash.
And that they stave off RSI and hand pain generally from giving you more different things to do with your hands.
The main downside is you can't shift riding out of the saddle, or really, when you're muscling the bike around at all. This badly affects sprinting, and carrying speed up the base of a climb. Which is a major downside.
What you think is the downside, that it sucks taking a hand off the bars to shift, can actually be a positive thing, because it's fun, and quite a natural movement when you're used to it. It can also be a good thing being limited in when you can or should shift to retain full control of the handling. Shifting less and just riding the gear you're in can be a good/fun habit.