Search engines should coevolve with their ecosystem.
- Search engines in a late stage ecosystem are massive, complex beasts.
- But at the very beginning of the ecosystem, they can be simple and small.
- Then, as the ecosystem grows, you iteratively increase the complexity of the search engine too.
- Search engines are proprietary, opinionated guides that users seek out to help them navigate the wild, verdant, but overwhelming jungle of an ecosystem.
- The ecosystem will grow at a compounding rate, so the search engine must, too.
- If you build a search engine as a linear process, it will never catch up with the compounding momentum of the ecosystem.
- Linear process here means "An employee has to invest effort to get it to handle the increased scope of the ecosystem."
- A directory like Yahoo was would be an example of a linear process.
- By default we build things with linear processes, but this will get you stuck.
- My first year out of college I worked in the Search team on a precursor to the Knowledge Graph.
- In the product we were building we needed to know which attributes were actually synonyms (e.g. "date of birth" and "birthdate").
- There were only a few dozen equivalence classes that covered almost all use cases, so I wrote up a CSV of configuration and we tried to check it into version control.
- That meant that as the ecosystem grew[abi]–as more users used it for more attribute names we'd never thought of, or in different languages, even without human intervention the system would be able to adapt and heal, automatically.
- This iron rule was one reason that the search engine could coevolve with the underlying, open-ended ecosystem.
- A tiny bit of additional work now made the system antifragile and auto-catalyzing, a compounding process, not a linear one[abj].