The difficulty of a programming task now comes down almost entirely to novelty.
- The difficulty of a programming task now comes down almost entirely to novelty.
- It used to be that there was a difference between integration hard and algorithmically hard engineering.
- Integration hard is easy to do, just requires a long, detail-oriented slog.
- Can be parallelized relatively easily.
- Algorithmically hard is hard to understand, but then easy to execute once you do.
- Requires carefully reading papers, brainstorming at the whiteboard, going on long quiet walks.
- But once you write the code it's often 1000 or so lines.
- Very difficult to parallelize.
- But that distinction was pre-LLMs.
- LLMs are great at algorithmically hard problems… as long as there are a lot of examples of it in the training set.
- No matter how arcane those examples are to discover or reason about.
- So the difficulty of executing now comes down entirely to novelty.
- Less novelty: more likely the LLM's first guess works and it can iterate its way to the solution.