august

How to do (mostly) good full-stack engineering

Stacks, languages, databases

There is often an idea that the best stack wins. In that the answers to revenue, users, and such are inside tech stacks. This is false. I've built out backends in Rust, C++ and Go, I've built out frontends in HTMX and React, I've built my own timeseries DBs with a better insert time, and they've all had zero users. If users is something you're optimizing for (which I'm not), then whatever you know best is the answer. At Commure, we scaled up massively writing horrible Python backends. How? Just throw money. A very annoying person recently dropped this truthbomb: You can use Supabase, but what of scale? By the time you get to scale you need to worry about, you'll have people who will move it out for you.

Of course, he's rich but I think he's an idiot whose words you shouldn't listen to. Write good backends. My personal stack is Go, PSQL, React. I know I've built the fastest products in the niche that I've built it in. But the place for optimization wasn't at Golang level (there is, but not when you're starting off), but at PSQL JOIN optimization. It is the main bottleneck in most of the cases. Or atleast true in healthcare. Think ahead about the joins. Think 10 years into the future. You'll have a million more tables, you'll have to synchronize data across it - and your users will always hate you anyway. The question is - can you keep them paying. Do NOT defer this decision. When YC says, "build something that doesn't scale", they don't mean a technology that doesn't scale - they mean the process. Eg. DoorDash early team going to actually deliver things after posting an HTML static page with a Google form. There are some actual bad technologies. I think Python for backend is a language you'll come to regret fairly early. Not using React is also a problem. XCode isn't suggested. Try to avoid NoSQL. Your exceptions should have good reasons. Either you're extremely comfortable in Vue, Python or XCode or something to that effect. Or you actually have unstructured data. Consider these. Do not - obviously, make a purist mistake of writing backends in things you consider fun unless you plan to keep the project fun. I have many such projects. Erlang, C++, Rust. They are public domain. I wrote such garbage, that if you use it, it'll eat into my bank balance or just not work. I am bad at them, unless you're extremely good, do not try. Do not LARP, be honest.


Networks and CyberSecurity

Maintain a list of all the endpoints you add in a curl format. Which means, add the endpoint, base URL, auth required, parameters, response. Do not ever use non-standard status codes or response model. Think about websocket support before you write the code. Any environment configuration missing should be an immediate pager or non-starter. It should fail your actual code from even starting. Toy languages such as Python have these issues the most. I know a company where the JWT token could easily be decoded, you could change email, re-encode it, and login for someone else. This happened because the developers (100 of them) never upload the JWT secret key.

One of the most commonly used glib sayings in CompSci is, "Do not roll your own auth". This is wrong, it should be "do not roll your own Crypto". You can absolutely write your own auth and do it safely as long as you're not an idiot.

You should always frequently rotate your keys. Never have your keys' keys stored in just ENV. Store it in your DB, and lock with something in .env - it should be practically impossible to get any key if you gain access to one service. Service Managers are good too, but remember to loudly fail any lack of secrets.

Some other things to look out for, as a pragmatic, early full-stack engineer is: Make sure to enforce HTTPS everywhere, even internally. Even during testing locally. Do not have write access to your SQL, you should connect to it via a proxy. Most frameworks will now help you with this, but use HSTS and secure cookies by default, we want to prevent any downgrade attacks. In CI/CD, make sure to detect any sensitive keys. There are many ways to detect now, no need to roll your own CI here. Make sure to validate all inputs at the edge (API gateway or ingress). If you have internal processes that take in dynamic parameters, make sure to catch it early and return early. Reject non-TLS DNS queries and prefer DoH/DoT where possible. Apply least privilege to every API key, IAM role, or service account. If someone wants or needs access to anything, make sure to run it via a granting mechanism with a two man switch. If it's only you, make sure to at least keep it time sensitive. Make sure that your tests never point to a production or staging DB (more on this later). Log auth events (login, logout, failure) with IP, UA, and timestamp. Any internal endpoints should mark who touched what. I also love to build a patch system for DB entries made by admins for version control. Eg. there is a preferences column per organization - someone changes it, then in preferences_vc I store this and can roll back when needed. I've had idiots say these were overengineering, and sentry is good. But they're the manager-class. Do not listen to them. The worst will happen. Hash all PII with bcrypt/argon2—not SHA256. SHA256 is faster, but bad actors can brute force billions of it. The chances of a hit increase with every entry in your table. bcrypt/argon2 are key derivation functions (KDFs): slow, tunable, and memory-hard. Of course, encrypt PII that needs to be shown to a user. Deny outbound traffic by default in infra firewall - never trust opening a link that can do arbitrary code, even if you send no parameters and such. iPhone used to scan photos in your Photos App for any links and then open it. They said it was for CSAM, which if true - great, but it also led to attacks.

Some notes on general infra: TLS certs should auto-renew with short TTLs (via ACME). Audit all open ports; nothing should be public unless justified. Segment networks by trust level (prod, staging, dev). Patch all dependencies in CI/CD, monitor CVEs (via osv.dev, Snyk etc) - sometimes a package will be discontinued, and your CI will start failing, and you're in your first week explaining to supposedly senior developers that this is the problem, but Python engineers being Python engineers don't understand that. Not that it happened to me.

Remember a few extra things: All infra must be reproducible via IaC (Terraform, Pulumi, etc). The environment must be completely reproducible with Docker and Make/Just files. A less than median engineer on Twitter said against "Stop Killing Games" that these games cannot just be released to the public because their environments implicitly trust in existence of many variables. Ok - write better code, and never work for game studios. That's another life lesson.

I am learning this now, but tag every resource (owner, env, cost center) for easy cost breakdown. Do not fall for the managed bullshit, roll your own where you need to, but generally AWS/GCP are good.

Enable S3 versioning and bucket encryption by default. Encrypt your DB even if HIPAA not required. Use staging environments that mirror prod infra 1:1. Take frequent dumps (heh, life lesson?). Auto-scale infra by metric, not just CPU (e.g., queue depth, latency). If your internal tools are using up too much DB (eg. using retool), move it to only allow DB access via APIs.

Use image scanning (Trivy, Grype) for every container build. Implement Blue/Green or Canary deploys with automated rollback. Never build containers in prod—use a build pipeline and signed images. If you're using RabbitMQ, remember that if it goes offline and comes back online, tasks in the middle may still hold on to your resources. I prefer managed queues because of this primarily. Use minimal base images (distroless, alpine) to reduce attack surface. I personally love alpine. Monitor infra cost daily; set alerts for sudden spikes. DNS entries should be auto-managed with IaC, not hand-written (looking at both you Namecheap users, and also at me). Health checks must go through the same load balancer path as users. Do not override - you may miss a service. Use connection pooling aggressively, especially with PSQL. Prefer IPv6 compatibility even if unused—futureproofing and compliance. Rate-limit at edge (Cloudflare/API Gateway), not in app logic, cloudflare is surprisingly good and I don't like AWS so prefer Cloudflare. All infra config should be in Git, versioned, peer-reviewed.

Mostly theoretical unless you're designing everything yourself, but learnt this one in a networks class: Design subnets to allow for future AZ expansion (e.g., /19 not /24).

I am getting a little tired, so for the important bits, DB backup, I'll just dump the bullet points from my notes. * Daily full backup (encrypted, timestamped)
* Store in off-site and/or cloud bucket (e.g., S3 with versioning)
* Use pg_dump for Postgres; avoid raw volume snapshot as primary
* Automate via cron or job scheduler
* Test restore regularly (automated + manual)
* Retention policy (e.g., 30 daily, 12 monthly)
* Monitor for failures and alert on backup miss
* Encrypt with KMS or GPG before upload
* Keep checksum/hash for integrity verification
* Don’t back up secrets in plaintext


Testing