Do not do retries. Retries are the sign of the weak. If it didn’t work in the first try, unless you’re pinging an unreliable third party service (and under very strict definitions of reliability, you can find times of flakiness), or dealing with poorly defined race conditions, it’s not going to magically work in a second attempt. You’re dealing with systems that obey the laws of physics, stop wasting time. You’re increasing the cost of “doing things”.
Worse is when there’s back-offs included, it is the sign of an LLM induced psychosis. Keep your code clean, and trust that most internal mechanisms will take place in one-go.
Monoliths are fine. The breakdown I see most often is a server for ephemeral tasks, and a server for normal co-ordination. The problem that happens here is you end up maintaining multiple test suites, and multiple payloads. This drift becomes significant and if ephemeral codebase cannot access normal DB, you’re fucked.
Restarting your prod in the middle of the night to fix leaks is fine. You made the mistake of writing in a duck-typing language, and now you’re worried about memory leaks from the interpreter itself. It’s not massive, but it adds up, and it becomes costly. Just restart in the middle of the night. Oh, what’s that? It’s not durable? What do you think MQs and DBs are for?
If it needs a feature flag that is backend dependent only for roll-outs, do not roll it out. You’re wasting everyone’s time with a fall sense of security. It’s going to cause issues eventually because of some remote usage of this feature flag, and anyway you’re going to forget about this feature flag. Have easier rollbacks, instead.
One big file is fine. Put all endpoints in one place. If it turns out instead of using a real framework like Gin, you’re using something fake, such as fastAPIs, and now your routes are defined as decorators - and look, I love decorators, but come on, man, that’s on you. That was your mistake #1.
If you’re opening the frontend for your cloud provider more than once a day, revisit the basics. Install a server in your home.
Are you thinking of using MongoDB? Unthink it. No data is unstructured enough for MongoDB. Use PostgreSQL. Did you think for some reason of using a time-series DB? Go back, and think about buying this service from someone. Oh no, you thought of embedded vectors and such? Use pg_vector in case Pinecone becomes too expensive.
Lesser the hands near your DB, the better.
Remember nightly restarts? Run a vacuum on your DB too.
If the drift between your migrations and DB is so massive, that every auto-generated migration has a bunch of non-sense involved, then this is a helpful guide on fixing it.
Does your auto-generation not produce downgrades? Fix it.
Is there inconsistent migration step for creating enums vs anything else? And you cannot downgrade enums effectively? Another helpful guide.
Do not set defaults for environment variables, instead write better Dockerfiles. Do not even think of having both .env and cloud secrets. If there’s more than one source for how your code gets environments, here’s a helpful guide I suggest. (Note: If one of our employees is reading this, and is about to bring up that we had this, no we don't. Not anymore. You fixed this, my friend.)
If you see your engineers adding in too many emojis or comments or abstractions, understand that this is bad practice and comes from over-reliance on LLMs. A great guide on dealing with this in your team is available here.