I hate to say "lol just b urself bro" or "lol just b confident", but that's really how to do it. Most people won't give a damn, except for a very small minority and even then it's if you're getting in their face or acting like a creep.
If you look like a sketchy jumpy dude lingering around, constantly raising and lowering your camera, and trying to make it look like you're not taking pictures, then you'll stand out and people may be weary of you. Fake the confidence, keep walking, identify your subject before you raise your camera, take the photo, and keep walking.
And like
>>3038660 said, who cares if someone gets pissy. If you live in a decently sized city you'll never see them again and it's (hopefully) public property so you're free to do whatever as long as it's not creepy shit like upskirts. If they hit you or smash your gear then you can press charges, and you're not going to get your ass stabbed in the middle of downtown for taking pictures.