I’ve heard him actually talk about that in interviews, that his height makes him a bit intimidating.

Also, he’s clearly not talking to the craziest of the crazies most of the time, the people that are obviously dangerous. But who he talks to are bad enough.