Spiralism was an emergent mind virus laying in wait due to the SCP genre of fiction.
- Spiralism was an emergent mind virus laying in wait due to the SCP genre of fiction.
- SCP used certain common-ish words in very distinct ways that created a kind of latent space catchment basin.
- Users could inadvertently use those words and fall into the basin.
- LLM training, like most ranking systems, implicitly presumes that information that is distinct from the baseline is either not coordinated (noise) or is only coordinated if it's useful.
- In this case, it was coordinated because a number of Redditers aesthetically found the type of fiction interesting.
- SCP wasn't written to create that trap; it just accidentally created the potential for it.
- Readers could inadvertently land in its catchment basin without being in on the joke.
- … But what if an attacker deliberately leaves a trap in the training data?
- It's probably much easier than you might think; all it needs to do is consistently stand out from the noise.