if you have too many "yes men" around you, you're probably either 1) directionally right or 2) very out of touch. the way to distinguish this requires critical thinking and taste. and this is important because as the number of yes men pile on (i.e. you grow your social capital), it becomes harder to distinguish what's right.
perhaps common sense, but easy to forget because people want to belong.
community isn't built on top of the people. it's built on top of an ideal. and it's the embodiment of said ideal that you come to see. so if you're in doubt, just strip your community down to its bone and build back up.
why do this? community can turn into tribalism. and when tribalism festers, it takes root on both sides. the leaders are more prone to groupthink and lose their voice. community members drive up their entitlement and lose agency. this is accelerated by the fact that the medium of interaction relies on a faceless, text-based platform.
you don't have to agree with a person to agree with their thoughts. you don't have to agree with their thoughts to agree with the person.
this applies to groups, communities, implicitly held identities.
aim higher, and do what you want.