Deciding what to build is easier with fewer people!
There are fewer people to align.
The effort to align people scales, all else equal, proportional to the square of the number of people.
A downside: with fewer people in the group, it's less likely a person with a game-changing idea (or an important bit of disconfirming evidence) will be in the group.