Because if you've already got the best, then /anyone else you can add/ will be less smart, less creative, less quirky, less considerate, than who's already in the group.
And at some point you'll notice. Maybe at 50 people, or 500, or 5,000, or 50k, or 500k, or 5m, or ....
For a few reasons.
* Gradation of capabilities. These are ordinalities, not cardinalities.
* Limits to common experience and interest.
* Differences of opinion. Or morals. Or philosophy.
* Just plain scale.