Memories should have a half-life.

If they haven't been used, they fade.

Each time they are used, they jump to the top.

The longer they aren't use, the more they fade to the bottom, and ultimately get culled.

This is like the Swarm Sifting Sort that Kiva robots do in warehouses, automatically keeping the most useful things towards the front.