Building a new system is hard.
Instead of being able to take the foundation for granted and being able to add a single incremental extension at a time, you have to keep an approximate view of all of the moving pieces of the system in your head at the same time.
This is a rare skill!
The most important thing is finding a viable system that feels good.
As game developers would say: to "find the fun"
If you can't find a viable thing that's fun, then nothing else matters.
Instead of accreting individual hardened, well-considered layers on top of an existing foundation, you need to rough in an approximate and adaptable sketch of the whole system.
As long as you can squint at each piece you approximated and convince yourself there's at least one way to harden it into a production ready thing–that there are no miracles–then it's fine.
Once you have a thing that feels fun, you can start hardening.
Harden the parts that have changed the least in the experimentation first.
Then keep on hardening until you have a viable first version of the product.
Remember: what counts as good enough / viable might be lower than you think.
If you develop in the open, the motivated early users can tell you when you have something worth the pain.