How about if it is less than 5 votes, no ban.
This can be abused by people using alt accounts.
Everyone can ban new players as they won't know anyone to defend them.