Are you sure they are intending it in a diminutive sense and not a familiar or affectionate sense? Buddy is pretty common to call people you are being friendly with in American English at least. If you don’t like a nickname or term of endearment, you can tell your friends not to use it, but fair warning, friends of your age might just tease you about it instead of respecting your wishes.
I gotta say, responding to “hey I think women and minorities might not be safe here” with “then maybe that space isn’t for [women and minorities]” is quite the hot take