The swarm evolves even if none of the particles that make up the swarm change.

The aggregate of the swarm is the summation of the velocities of the particles.

The swarm can change when the particles change… or when the set of particles included in the population changes.

For example, new particles joining the swarm.

Or existing particles dying or leaving the swarm.

As long as the swarm subsets particles (that is, the ones it keeps) with a consistent bias, the aggregate vector of the swarm will become coherent and strong.

You don't need active selection by some judgment; if the unfit particles are selected out by failing to survive then the swarm gets the velocity of the fittest in this environment, and the selection bias is emergent but also with a consistent bias.

The adaptation works as long as there's a consistent bias in the selection.

If the particles can change (e.g. the beliefs, skills, incentives of humans in an org) then the swarm can change even without particles dying.

But if the particles can't change (e.g. an organism's genome) then the only way for the swarm to adapt is for particles to continuously be dying with some bias.