What is a normal amount of "being assaulted" when it comes to street shooting?

Ok, so I've been getting more into street photography, so I'm pretty new and I'm not like going right up to people with a wide lens, using flash, or being super blatant at all. Nevertheless, I've had a number of bad experiences and on the average three or four hour excursion I'll probably get a few people not too politely confronting me. I pretty much smile and de-escalate and offer to delete pic etc.

However, I've also been chased down, screamed at, threatened, and today a guy came across the street and started like pushing me and getting in my face...and kicked me as I backed away. He was not even close to being in my frame, we was prob like 40 feet away anyway.

Like, I expected some pushback, but not being berated or assaulted on the reg.

I'm like a normal looking tall white dude in SF...a city I didn't know had a reputation for nastiness.