systemd's DynamicUser feature could save some time here. It can allocate a uid, then create directories for logs/state with the correct permissions.

https://0pointer.net/blog/dynamic-users-with-systemd.html

You just drop a small text file (often a single line) into /etc/sysusers.d/ with the information about the user, like username, home directory and whatever, and then invoke the sysusers command or service!

Kerberos much? =)

https://istio.io/latest/docs/concepts/security/#principals

Performance is fantastic (using isolated app service plans in Azure), and I am completely over the ideological principals against per-request billing. Absolutely, you can do it cheaper if you own the literal real estate the servers are running inside of. Paying for flat colo fees makes per-request costs look ludicrous on the surface. But, achieving all of the other attributes of simple HTTP triggered functions in a DIY context is very challenging without also spinning up a billion dollar corporation and hiring 1k more people. Compliance, audits, etc are where it gets real super fast.

The "what about lock-in?" argument doesn't work for me anymore. HTTP triggers are a pretty natural interface to code against: "I've got a HTTP request for you to review, give me some HTTP response please". The only stuff that is vendor-specific are the actual trigger method signatures and specific contexts beyond HTTP, such as OIDC or SAML claims. You'd really have to go out of your way to design an HTTP trigger web app/API solution that is impossible to refactor for another vendor within a week or so.

If your business is a personal blog, then yeah I get it. It's more fun to buy a VM in Hetzner and get all artisanal about it. Also, if you are operating in a totally unregulated industry, perhaps you can make some stronger arguments against completely outsourcing the servers in favor of splitting hairs on margin vs complexity.

I literally rolled my eyes reading that. How do you think we did before cloud computing?

I am currently in charge of multiple teams scaling and deploying innovative applications for a large industrial company. We are using Azure for everything. Our cloud costs are insane for our number of users. I used to manage applications with ten times more user for one hundredth of the cost and less complexity. It’s billed to another part of the company which is responsible for this dubious choice (I really hope someone is getting nice business trips paid by MS so it’s not a complete waste) so I don’t care but how people can blindly put faith in the cloud is beside me.

For all of Microsoft's faults, you can get a person on the phone when you pay money.

There is a reason Microsoft kicks Google's ass all over the room in the enterprise space.

Which one? We are using this stack in fintech, and are subject to PCI-DSS, SOX, AML, SOC2, etc. Many of our customers (small US banks & credit unions) are very interested in this kind of cookie-cutter, cloud-native stack in 2023.

> We can't even take pictures in the office, there's an absolute 0% chance we're trusting a cloud provider with anything.

Sounds like you work for an F100. Our IT budget is 5 figures and we are doing business with clients who have 6 figure IT budgets at the high end. Forcing an on-prem architecture would make our solution less secure in most situations, especially for those customers who do not have the confidence to run a complex SAAS solution on their premises. Many of our customers actively understand this reality and are very open to the idea of offloading complexity to the cloud.

Yeah. I spent a couple of years at a billion-$ company in the telco industry which was subject to all sorts of US federal and foreign regulations (because they operated in 20+ countries.) They ran almost everything onprem, but seeing how that was managed, cloud would have 1000% more secure for them. At one point, the entire senior staff of the IT department was fired because of a security breach that was pretty clearly due to their poor decisions.

Companies do exist where their onprem operations do seem very secure, but you need really big budgets and good management to do that properly. Most places are not like that, even in the highly regulated spaces.

I suppose in the end they achieve a kind of security with this behavior, but it would be a lot better to avoid such incidents in the first place - which would be perfectly possible, with good decision-making.

> which add cost

We aren't interested in free. You get what you pay for.

I would say it's a somewhat weak correlation that breaks down completely if you consider a wider range of architectures. Back when Google was a startup "you get what you pay for" is what people running Oracle on Solaris would tell you.

And there was a sense in which it was true. There was no more reliable and feature rich way of scaling up. But starting something like Google on top of this egregiously expensive platform would have been completely uneconomical. It just wouldn't have happened.

And I think it's the same thing today. There are good reasons for using those egregiously expensive offerings from the largest incumbants but if you are starting something where servers are a significant cost factor then you are well advised to look at other options that are orders of magnitude cheaper, even accounting for labour.

What exactly are you talking about? Request handlers in HTTP servers? Function-as-a-service cloud computing services? Plain old RPC?

* I use Linode VMs ($5/month).

* I too use Debian GNU/Linux.

* I use Common Lisp to write the software.

* In case of a personal website or blog, a static website is generated by a Common Lisp program. In case of an online service or web application, the service is written as a Common Lisp program that uses Hunchentoot to process HTTP requests and return HTTP responses.

* I too use systemd unit files to ensure that the website/service starts automatically when the VM starts or restarts. Most of my unit files are about 10-15 lines long.

* The initial configuration of the VM is coded as a shell script: https://github.com/susam/dotfiles/blob/main/linode.sh

* Project-specific or service-specific configuration is coded as individual Makefiles. Examples: https://github.com/susam/susam.net/blob/main/Makefile and https://github.com/susam/mathb/blob/main/Makefile

* I do not use containers. These websites have been running since several years before containers were popular. I have found that the initialization script and a Makefile have been sufficient for my needs so far.

* I use Nginx too. Nginx serves the static files as well as functions as a reverse proxy when there are backend services involved. Indeed TLS termination is an important benefit it offers. Other benefits include rate limiting requests, configuring an allowlist for HTTP headers to protect the backend service, etc.

* I have a little private playbook with a handful of commands like:

  curl LINK -o linode.sh && sh linode.sh
  git clone LINK && cd PROJECT && sudo make setup https

I use a VPS with 512 MB of RAM, but each SBCL instance uses roughly 100 MB of RAM, so I can only have a couple services at once.

I’ve considered moving the lowest traffic services to CLISP, but it’s missing at least one feature I use (package—local nicknames).

This has some downsides: Firstly they run in the same address space, so global state is shared and serious bugs can affect another service. Secondly they run as a single systemd service, so they're not easy to manage individually. Still I've found it to work quite nicely for things that are simple and don't really need attention.

Maybe it’s my imagination but it seems like CL isn’t as suited to sharing code and read-only data pages across processes (e.g. in shared libraries). Or maybe there’s a solution I just haven’t found yet…

For long running services, it does consume about 100 MB of memory. For example, I have a service running right now with an uptime of 270 days. SBCL is currently consuming 108 MB of memory. This has not been a problem either because for low traffic websites like mine, this memory consumption size remains fairly stable. I have not found it to be varying much.

One of the many benefits of using docker is that I can use the same setup to run 3rd party software. I've been using this setup for a few years now and it's awesome. It's robust like the author mentioned. But if you need the flexibility, you can also do whatever you want.

The only pain point I have right now is on rolling deployment. As my software scales, a few second of downtime every deployment is becoming an issue. I don't have a simple solution yet but perhaps docker swarm is the way to go.

To avoid downtime try using:

    health_uri /health
    lb_try_duration 30s

    api.xxx.se {
      encode gzip
      reverse_proxy api:8089 {
        health_uri /health
        lb_try_duration 30s
      }
    }

Ideally, during deployment of a new version the new version should go live and healthy before caddy starts using it (and kills the old container). I've looked at https://github.com/Wowu/docker-rollout and https://github.com/lucaslorentz/caddy-docker-proxy but haven't had time to prioritize it yet.

edit: closest I found is this manual way, using Lua: https://serverfault.com/questions/259665/nginx-proxy-retry-w...

How do you orchestrate the spinning up and down? Just a script to start service B, wait until service B is healthy, wait 10 seconds, stop service A, and caddy just smooths out the deployment?

It's not perfect but it means rather than getting connection errors, browsers will just spin for a couple seconds.

The same technique is used by https://mrsk.dev/

So instead, we went for Swarm mode. Oh, what a journey that is. Sometimes Jekyll, sometimes Hide. There are some bugs that simply nobody cares fixing, some parts of the Docker spec that simply don’t get implemented (but nobody tells you), implementation choices so dumb you’ll rip your hair out in anger over, and the nagging feeling that Docker Inc employees seem incapable to talk to each other, think things through, or stay focused on a single bloody task for once.

But! There is also much beauty to it. Your compose stacks simply work, while giving you opportunities to grow in the right places. Zero-downtime deployments, upgrades, load balancing, and rollbacks work really well if you care to configure them properly. Raft is as reliable in keeping the cluster working as everywhere else. And if you put in some work, you’ll get a flexible, secure, and automatically distributed, self-service platform for every workload you want to run - for a fraction of the maintenance budget of K8s.

Prepare, however, for getting your deployment scripts right. I’ve spent quite a while to build something in Python to convert valid Docker-spec compose files to valid Swarm specs, update and clean up secrets, and expand environment variables.

Also, depending on your VPS provider, make sure you configure network MTU correctly (this has shortened my life considerably, I’m sure of it).

Let me know if that helps, or if you need more guidance. Maybe I could open source the whole thing properly, if that is useful to someone :)

There's also "Jumbo Frames" though you're not likely to encounter that day to day in a VPS.

Podman Pods (which contains PostgreSQL database and the app) all running in localhost on ports > 5000 and Caddy running on 443 as reverse proxy.

I use systemd Timer to dump all the databases at 4:55 PM in a directory. Then there is DejaDup [1] which automatically backs up $HOME (with no cache files of course) at 5 PM daily to external HDD. This backup includes the database dumps.

The OS is Debian with GNOME Core [2] and a firewalld rule to allow only 80, 443 and a customized SSH port. The SSH is key based with no password auth.

The most boring way but it just works :D

1 - https://flathub.org/apps/org.gnome.DejaDup

2 - https://packages.debian.org/bookworm/gnome-core

GUI is needed because the office staff doesn't know SSH or CLI or Linux at all ^^

(She liked GNOME once I showed it to her tho)

I think I would use Ansible to setup the servers and use it for the deployment script as well.

This would document the servers and make deployment script perhaps simpler.

I wouldn't shy away from accessing the servers manually when debugging or checking things, though.

I do use ansible but I'm reconsidering whether it is worth it, while my scripts have barely changed in the last 7 years, they tend to be slow and require me keeping multiple files around.

Related to debugging, you can expose the app logs through a password-protected endpoint by nginx.

I’ve recently taken to doing this in Go and absolutely love how easy it makes writing and deploying software that depends on static files.

Having the static frontend assets baked in along with a default config is a huge boon though.

It supports embedding the encrypted secrets in the binary or loading them from a file. The secrets would actually be stored (encrypted) alongside the code, even versioned in git.

Eg this is the rust version on GitHub: https://github.com/neosmart/securestore-rs/tree/master

See https://github.com/bbkane/grabbit for an example

Makes it trivial to run `brew/scoop update myapp` from another computer

I have used cargo-dist and the process is quite smooth, it could be worth giving it a try.

But there is one question. The article says:

> I get my HTTPS certs from Let's Encrypt via certbot — this handles automatic renewal so I don't have to do anything to keep it working.

But I'm using cross-region setup with two servers and a geo-DNS. With this setup, the certbot only works for the server, located in the US, and I have to manually copy the certificates to the server in Europe. Any idea how to overcome this?

PS. Read about ClickHouse Playground here: https://ghe.clickhouse.tech/

I have been running a similar setup for many years with some differences:

1. Use `EnvironmentFile` on systemd to load environment variables instead of bundling secrets into the binary.

2. Set `LimitNOFILE=65535` on the service to avoid reaching the file open limit on the app.

3. Set `StandardError=journal` and `StandardOutput=journal` so that `journalctl` can display the app logs.

4. Use postgres instead of sqlite, DO is taking regular backups for me and postgres maintenance is almost null for simple apps.

5. Nginx can have password-protected endpoints, which are useful to expose the app logs without requiring to ssh into the VM.

6. Nginx can also do pretty good caching for static assets + API responses that barely change, this is very helpful for scaling the app.

At last, I use ansible but I'm considering if its worth it, replacing it seems simple and I'd be able to keep a single deploy file that runs faster than ansible.

Also, none of the solutions discussed here gracefully handle new connections on the new service while waiting for all connections to terminate before shutting down the old service. Maybe some of the more esoteric Ansible do idk.

I TRULY want the simplicity of setups like discussed here, but I can’t help but think it’s irresponsible to recommend them in non hobbyist scenarios.

From personal experience in B2B web apps, a lot of sales/business MBA type's will say they need 100% uptime, but what they actually mean is it needs to be available whenever their customer's users want to access it, and their users are business users that work 9-5 so there's plenty of scope for the system to be down (either due to genuine outage or maintenance/upgrades).

You've possibly also got the bonus of the people that use the app are different to the people that pay for it, so you've also got some leeway in that your system can blip for a minute and have requests fail (as long as there's no data loss), and that won't get reported up the management chain of the customer, because hitting F5 30 seconds later springs it back into life and so they carry on with their day without bother firing an email off or walking over to their bosses desk to complain the website was broken for a second.

At a previous company we deployed each customer on their own VM in either AWS or Azure, with the app and database deployed. It was pretty rare for a VM to fail, and when it did the cloud provider automatically reprovisioned it on new hardware, so as long as you configure your startup scripts correctly and they work quickly then you might be down for a few minutes. It was incredibly rare for an engineer to have to manually intervene, but because our setup was very simple we could nuke a VM, spin up another one and deploy the software back onto it in and be up and running again in under 30 minutes, which to us was worth the reduced costs.

Not really, there are many kinds of apps that don't need such redundancy.

> Also, none of the solutions discussed here gracefully handle new connections on the new service while waiting for all connections to terminate before shutting down the old service. Maybe some of the more esoteric Ansible do idk.

I have dealt with this in the code with shutdown hooks on the server, waiting for existing requests to finish its processing and reject new requests, clients will just end up retrying, not all apps can accept this but many can.

Performance is not an issue in most docker setups you would ever use, correct.

I think it mostly comes down to what layer of abstraction you like working at.

But it doesn’t have system-level dependencies. For example, in systemd I can wait for a network interface to be up and have an IP assigned by DHCP. As far as I am aware, docker compose knows about the docker network and its own containers, but not the system more broadly.

Also, you will likely want it to run for a long time, so something has to trigger the docker-compose process to start and restart it. You might want it to restart in case the OOM killer knocks it over. That daemom stuff is what systemd is good for.

I should really put my homelab setup somewhere.

Unnecessary overhead gets introduced with docker, for example, now you need to depend on a container-registry + the authentication to pull those images from the server + other stuff.

This seems like a perfect application of the init system.

I used to have quite a few of them, then I shifted to K8s (or k3os, specifically), so now the only VMs other than the K8s nodes are my NAS, backup target, and a dev server. However, since Rancher has abandoned k3os and it’s forever stuck at upstream 1.21, I’m in the process of switching to Talos Linux for K8s. I have Proxmox running Ceph for me to provide block storage to pods.

My blog was running in a tiny EC2 with a Bitnami multi-WordPress AMI, but since everyone else sharing it with me quit, I shifted that out to GitHub Pages + Hugo. So far I like that quite a bit better, plus it’s free.

So you have a container with Alpine Linux and nginx and that’s hosted in Cloud Run and mapped to your domain?

When you say “visits” do you mean the container is active 40 times per day (presumably for not very long)?

Edit: what determines when to suspend the container?

Edit again: answering the previous question: https://cloud.google.com/blog/topics/developers-practitioner...

Don't underestimate cold starts though. Will be a long time until you need to worry about that if you're backend is written in Go or Rust, but you can hit painful starts pretty quickly when it's Nodejs. Really comes down to (unsurprisingly) image size.

If coldstarts are a problem, things like fly.io are incredible. Just set the minimum amount of instances to 1, pay 5$ a month and you don't need to deal with anything.

It wasn't immediately obvious to me how fly and Cloudrun compare, but the way I think about it now after having used both:

In Cloudrun you don't think about any servers, you give them your image and they'll do the rest. It feels serverless in the sense that you don't think about machines.

Fly meanwhile is much closer to the infrastructure. It feels more like "we deploy a docker image to a VM for you", which let's you much more granuarly control what's going on beyond that.

Less magic, but in a good way because more predictable results. Cloudrun has a few fun corners (like no CPU between requests unless you explicitly enable it) that can lead to fun side effects and just make it harder to reason about.

If you're at the point where you're concerned about cold starts and using a service like fly.io why not just skip the (unbelievable, no-one-knows-this-stuff) complexity and use a $5/month VPS?

"Simply" using a 5$ VPS sounds great until you need to start writing systemd files, need to keep the box updated, want to deploy straight from GitHub,...

Nowadays all linux distros offer an unattended way to do packages security updates and reboot the node.

The 0.5G instance has a minimum cost of $10 a month, before any kind of traffic metering, or what have you.

Digital Ocean, Rocky Linux or Fedora, systemd services, Bash. I usually run Rails with PostgreSQL. I might use containers more going forwards although I haven't so far.

I wrote Deployment from Scratch exactly for showing how to deploy with just Bash.

A single instance container that runs 24*7 for a month with similar cpu/memory as the $6 droplet is $30/month, before you factor in network costs.

One of the motivating factors is I had a cheap VPS as my syncthing node and it just stopped working one day and won't boot. I haven't had time to debug it and find out exactly why.

Setting up a python server environment seems to be hard work, with lots of steps (gunicorn and all that) but that said they make the point about using Docker. So maybe docker compose could take a lot of the pain out of it.

https://libcloud.readthedocs.io/en/stable/compute/drivers/di...

So as long as you don't mind writing your deployment scripts using Python you can make them reasonably portable (though honestly every provider has a little bit of quirkiness that needs to be worked around but it's usually trivial stuff).

Should solve 95% of that "lock in" problem.

One advantage from DO is the regular backups.

Like you said, you can go for executing docker compose on the server if you want to do it fast.

I find it just as easy as Heroku, but cheaper. As someone who hates messing with servers and prefers to focus on my product, render is perfect for me.

Nowadays I'd generally go for hosting providers that have been around for some time.

Hetzner: has both great features and affordable costs (I needed to verify my ID, though)

Contabo: the prices are great, but the performance is a bit worse than other options

Time4VPS: a Lithuanian provider that I used due to them being cheaper than Hetzner, unfortunately their prices have increased noticeably, only the yearly plans are worth it

DigitalOcean, Scaleway, Vultr: all have good features, but can be expensive

Azure, AWS, GCP: too complex and enterprisey for my private needs

That's not to discourage anyone from using them; in fact, LEB VPS servers are pretty much all I use (each being ~$25 / year), and I've gotten quite good at keeping useful, tested backups, keeping redundancy in my services, and being able to stand up a new server quickly.

On the one hand, it's a security thing: if one service gets breached by a catastrophic security hole, the rest of the servers should hopefully be unaffected.

But the main reason for it is ease of administration. I don't need to bother with Python virtual envs or RVM for Ruby or juggle multiple PHP versions, I can just install the version I want with apt and everything just works.

When I pass a project on to someone else, I can just give them access to the VM so they can easily migrate it.

That convenience is worth a few Euros per month. (my total hosting bill varies but I don't think it was ever more than 50€ per month)

In the past I would have templates, analytics and the works. Today I just want to have the shit up there and that's it.

One thing I noticed while messing with it tho is that the effort to do the basics today is much bigger than 10 years ago. Not because things are more convoluted but because the standards are much higher. You have to work on mobile and desktop, expectation of behaviour is also much higher. Animations are expected, etc.

I also solve this problems everyday for work, last thing I want is to do it for me I guess. Funny is that Im also doing way more than I needed, because I could pay for squarespace or something and be done with it but I also want to control it too much?

Moved to Ansible playbooks for all server config, checked into the same Git repo as the code for the project. Sure, I can set it up by hand fine, but then there's no documentation about how the server is currently set up. It's no fun to have to set it all up again by hand if you need to switch to a new server, or try to remember/figure out exactly what you did 3 years ago to set it up when something goes wrong.

Docker for the apps. Not as big of a deal with compiled languages, but for interpreted languages, getting a non-ancient version of the interpreter on the bare metal server is a major headache, as is updating that version. Updating the dockerfile to point to Ruby 3.2 by contrast takes just a moment. Docker is also at least as good as bare systemd, probably better, at auto-restarting services, holding configuration secrets, isolating app access, managing logs, etc.

I use Ansible to set up and launch the containers because it fits in with all of the other setup steps and it works well with running multiple independent apps on the same server.

I've experimented with the super-integrated stuff like Cloud Run. Seems okay if you just want to run an image somewhere, but it seems to me like the complexity and potential for issues multiplies fast once you need supporting services like DBs or caches, periodic background tasks, multiple services running, etc. Might be okay in say AWS with CloudFormation or something, but I'd honestly rather go right to managed K8s instead. It's a little more complex, but quite capable and can be run on a bunch of different cloud services.

1. I'm using a single postgresql database for all apps (each with a different user) on a different server; each app has a different db user

2. I use a minio instance for file/media uploads/serving

3. I mostly use nginx but i'm transitioning new apps to caddy because of automatic integration with let's encrypt and much smaller config for common purposes

4. I use a fab-classic (fabric 1x) script to deploy new versions: https://github.com/spapas/etsd/blob/master/fabfile.py

5. For backup I do a logical db backup once per day via cron (using a script similar to this https://spapas.github.io/2016/11/02/postgresql-backup/)

6. One memcache instance of all apps

7. Each app gets a redis instance (if redis is needed): https://gist.github.com/akhdaniel/04e4bb2df76ef534b0cb982c1d...

8. Use systemd for app control

- add build script using Github Action that fails the entire build pipeline if code doesn't build

- add deploy script (essentially a few commands ssh into your VPS and pulling, install, building and restarting)

I was surprised how easy it was. Naturally you need to get your hands a bit dirty compared to managed solutions, but using AWS with Lambda, Gateway, NAT (since I required a static IP) would have taken way time and costed significantly more.

[1]: https://aihelperbot.com/

[2]: https://gist.github.com/danielwetan/4f4db933531db5dd1af2e69e...

Depends a lot on what you're doing this for; in particularly for personal stuff then you do you - but this particular item does give me pause.

But then I use AWS for pretty much all my personal stuff, so I guess I have the overkill mindset already?

I'd guess that the author's idea is that if someone gains access to the server, the secrets would be either dumped from the binary or the environment file, if so, why to bother with another file?

I’ve never used LightSail but it looks like a good deal.

When I'm playing with more complex stuff I spin up an ECS with Fargate hosted containers. That's mostly for fun though.

- NixOS:

Was hard to figure out at first but is the MOST no-brained deployment setup after +20 years doing things. Is super easy to upgrade, add-remove things in a predictable way. Also, I love how ALL the config + security tweaks are in a single place (I Just do a big NixOS config to see everything at once).

Another big win with nixos is that you can do a very constrained setup and if for some reason need to do something that require a install you can do it once without polluting the install and it disappear after it (like: Install node, run things, get out and node and their dirt gone!)

- PG

I wish to use only Sqlite but it lack vital features for me (like proper stored procedures) and have the ability to access the DB with TailScale is a big plus when tracing a problem with a customer.

The author has snapshots faced every 6 hours. Even if something did happen, I feel like the time to recovery would be not bad. And the time & bad energy saved not worrying about all this obnoxious & almost certainly irrelevant enterprise grade security concerns seems greatly relieving.

I suppose I could git pull on a cron job, copy over the systemd unit files, and restart, but I'd like to have just a little more smarts than that. I've been working on my own tool to accomplish this but it's not super stable just yet.

It’s a super easy setup and saves managing SSH keys in env variables etc while still being really quick!

I'm pretty happy with that set up. Ansible has just enough smarts to be less tedious than shell scripting, but dumb enough that it's behavior is easy to figure out in most cases.

Now I'm tempted to try running it nightly from cron.

How complex... You know that you can simply git push to just about any ssh account, right?

I use Apache for the same purpose, works great, always has.

Oh, you sweet summer child.

For personal things I just use an EC2 instance w/ docker compose.

SQLite is a library you use in your application process that writes to a file. There's a bit of care you have to do to ensure you back up that file safely, but there's no extra monitoring or maintenance above what you do for your app anyway.

I agree that running Postgres isn't terribly difficult, but no matter how simple you try to make a small Postgres instance, SQLite is simpler.

For a simple setup, postgres maintenance is practically null.

If on AWS, and if your servers are behind a load balancer, just install a certificate on those. Isn't that a little better?

My take from the post is that author's goal is simplicity, AWS isn't simple, also AWS load balancers are expensive.