Not a solo founder, but I do a lot of ops professionally. Unless you have a specific reason, I say it’s best to avoid complicated tooling until later.
k8s, ci, etc are all really useful and solve a lot of hard problems, but god are they a bitch to set up and then monitor, fix when something weird happens, secure from nasty people outside, patch when some update becomes necessary, resist the urge to “improve”, etc. This doesn’t include the time it takes to work these tools into your project, like k8s for example really requires you grok its model of how stuff it runs is supposed to look. There’s a reason entire companies exist that simply offer managed versions of these services!
If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running) and honestly it’s pretty fun to just run a command and poof you’re deployed. Then you can just wait until it starts making sense to use fancier tooling (more load, more sites/DCs, more services, more people/teams, etc). Monitoring is always good to start with, for other stuff a good litmus test is that price points for hosted k8s & ops stuff will start to make sense, you’ll be paying in time in any case. Who knows, maybe when that happens something new will pop up that makes even more sense. Until then, reducing the amount of stuff you need to worry about is always a good strategy.
> spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script
I never realised I was using spooky arcane oldhat stuff! I feel wizardly now.
My projects (for small clients and myself) basically use this.
- A "build.sh" script that does a local build of back end and front end
- A "deploy.sh" script that scp's everything to the server (either a digital ocean VPS or an EC2 instance), runs npm install, runs database migrations and restarts with pm2
So running my entire CI pipeline in terminal is: ./build.sh && ./deploy.sh
If you like this style of deployment, and you haven't looked into Ansible yet, give it a try, you may enjoy it. I had similar build processes before using Ansible and I have found it a really positive step up.
My favorite deploy is using a simple `git pull` instead of scp. I also avoid complex build tools when I can. You either need your built files in the repo or you need to avoid them. Either are fine for my small personal projects. The only real exception I make is keeping any necessary secrets out of my repo. Those get dropped on the server manually.
This also solves the roll-back problem mentioned elsewhere. Just checkout the previous version (as long as you don't make any side effects like DB updates that are incompatible).
I admit, this requires relatively simple software.
Naive question: what's the correct way of doing this when you need to cp your content to a dir like /var/www/html that your user doesn't own (when logging in as root is prohibited)? My "spooky" (and probably very stupid) method is an Expect script that lets me supply a password to sudo.
Ideally you want some atomic deployment strategy, rather than having to deal with whatever crap might have been in the directory previously. Something like - extract your deployment artefact to a new, unique directory - and then read-only bind-mount that directory as /var/www/html.
> and probably very stupid) method is an Expect script that lets me supply a password to sudo.
The old school sysadmin way of doing this would be to have a dedicated deployment user (ssh pubkey auth only - possibly restricting only specific commands), with a sudoers[.d] configuration to allow that user to run an explicit group commands without a password (NOPASSWD)
Usually these directories are owned by another group (www or www-data) so that your web server program (nginx, apache, etc) can access them without running as root. If you add your user to that group, you should be able to manage files in your web root without sudo. Be careful with permissions, though - you may need to chown to set the group after copying so that the files are readable by your web server.
a pattern I have seen before is to clone the branch commit of each release to a folder /var/releases/{release_commit} and then html is not a folder but a soft link so /var/www/html -> /var/releases/latest_release_commit
this is useful if you need to revert back quickly.
But then releases folder needs to be cleaned up, or you can run out of space.
Safer/easier to keep the .git somewhere else entirely, rather than in the working directory. Git's got arguments that let you do that. Then you can check out to a dir that doesn't have the .git dir in it at all.
Alternatively, it's pretty easy to use git-archive (optionally, with an exclusions list to keep out files you don't want to serve publicly) to do something similar without ever cloning the remote repo. You can fetch a zip or tar.gz with a given commit's files, straight from a remote repo.
This is a great setup if you work with a lot of clients. Every client we work with wants everything to run on their infrastructure that they provide, and then they provide the weirdest jankiest setup you could imagine. Because of this I keep everything basic if it cant run on a fresh linux install or less (looking at you docker) then its going to cause issues down the line.
Basically same here. In cases the deployment has to be zipped and manually deployed to elastic beanstalk. On some EC2 instances for projects running a PHP stack, I don't even bother with this, just SFTP script changes to the server straight out of my IDE. Take care of any rollback needs with local Git versioning.
I had a similar setup a couple of years ago, where I had to deploy without downtime. My solution was to simply have the old version running in parallel until I was certain the new version was ok
----------
Static website:
1. Setup NginX using a symlink to the current version
2. Copy the new files to a separate folder
3. Change the symlink to point to the new version
----------
REST service (behind NgninX reverse proxy):
1. Current version runs on port X.
2. Deploy new version and run it on port Y.
3. Update NginX config to use port Y and reload
4. If you need rollback, just reverse step 3.
This can be done using scripts or Ansible too if necessary.
Keep a simple tool like Ansible around for those spooky admin tricks and you can take advantage of the Ansistrano plugins for smooth deploy and rollback (Ruby's great Capistrano tool ported to Ansible).
I've been writing some Ansible playbooks recently for the first time in years, and came upon geerlingguy's work. That guy is a powerhouse when it comes to writing Ansible roles/modules!
That's sort of pet peeve of mine: Migration are done separate from code deploys. Version 1 of your code runs on schema version 1. Schema version 2 does not make chances that will break code version 1. Code version 2 can utilize the chances made in schema version 2, but you're still able to rollback the code.
Each schema migration should also come with its own rollback script.
The downside is that you might need three migrations for some operation, but at least you won't break stuff.
The assumption that you can do a schema migration while deploying new code is only valid when you have very limited database sizes. I've seen Flyway migrations break so many times, because developers assumed it was fine to just do complicated migrations on a 200GB database. Or a Django database migration just lock up everything for hours because no one cared to think about the difference between migrating 100MB and 100GB. And I've never seen anyone seriously considering rolling back a Flyway migration.
Agree with this and have practiced and advocated for it. Make the schema changes to support the new feature first, then verify existing software still works. Deploy the schema change. Then develop the new feature, test, and deploy the program. That way you can deploy and rollback without needing to synchronously run a bunch of migration routines.
where `${remote}` is defined in `.ssh/config` and `${path}` is in the project config to move files over to a new deployment directory. Then, a quick command over SSH changes the symlink to the site's configuration file and restarts the web server.
The sites are either on Linode or SSDNodes. Has been working for me for decades and don't need to change it for these hobby things. After all, the log files prove that these things get more traffic from exploitation probes than real people ;-)
Also a hobbyist sysadmin at home, but after reading a few comments on the internet, I found that the rsync -z compress flag was bottlenecked on a maxed-out CPU thread of my Raspberry Pi 4 NAS transfers.
Admittedly, these are mostly "local" drive-to-drive transfers over USB rather than server-to-server transfers over house-wide gigabit. But consider trying the transfer without compression.
I'll definitely have to read up on your other flags though, if your rsync works from just one machine rather than both that might solve a minor problem case I have where the destination machine lacks rsync...
yah, this (or something similar) is the most straightforward solution. the whole point of heroku is to abstract away devops for solo/small dev groups and make deployments “one-click”. it’s unclear why that option was explicitly ruled out here.
I cannot imagine not setting up at least a basic CI. Even if you remove all unit/integration test alarms, a CI is still very useful. The 20 minutes of build -> upload -> deploy script -> check prod vs a 1/2 day to set up a CI and forget it.
I'm not sure you save much time but you save a lot of headache
I think you mean automation is very useful. I've worked in a lot of places where we had a single `make deploy` that could compile, test, upload the artifacts, deploy the artifacts to hosts, start the new services, and slowly transition the load balancer to point at the new instances while automatically backing out if error metrics started lighting up. This is just automation, it doesn't have to happen on every commit. Especially when working solo or on a small team, I might not want to maintain a lot of VCS branches and instead, just deploy when I feel it's ready. You have to do all of that stuff anyway, so if you later decide to hook it up to a VCS trigger, you're all set.
I guess that's fair, I'm probably coming into this having only seen it done a certain way and blind to other possibilities.
That being said, it would take a lot to convince me a standard CI/CD pipeline is not worth the investment. My only experience with non CI/CD deployment process was a very manual process at a startup, then when I saw my first CI/CD it felt like magic.
A good old makefile is a CI/CD pipeline. All the new fangled tools are re-inventing a lot of capabilities make already has. I blame the new kids on the block who have only ever known Windows :-)
> If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running)
If you have multiple people on the team that have access to servers, this will shoot you in the foot a year or two later. Good luck replicating that server setup when no one has any clue whatsoever about all the small tweaks that were performed ad-hoc over the years.
Yes. Then you realize that people may still SSH into your servers and manually tweak things when they need the fix "now". So then you start restricting SSH access and forcing everything through Ansible. And now you're deep in deployment best practices again ;)
I think you’re describing the growing pains that justify paying the costs of more complicated tooling. The big difference is that instead of needing to figure it out as a solo founder with bigger problems, in this scenario you get to figure it out as a solo founder with a team and presumably some extra $$.
I'm fully on team spooky admin - however on my latest project, I've been loving being on the free tier of tools like vercel, which deploys from git commit, and firebase, which is mostly maintenance free. That really reduces the amount of devops in my day.
At this point CI is pretty easy to setup and monitor and it's integrated with practically every development/deployment tool, so I have to disagree about waiting to implement it.
>you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running)
You would be surprised at how many old enterprises still do this. Sure, it's ugly, but it can be simpler than k8s, and anyone with scripting chops can understand it.
k8s, ci, etc are all really useful and solve a lot of hard problems, but god are they a bitch to set up and then monitor, fix when something weird happens, secure from nasty people outside, patch when some update becomes necessary, resist the urge to “improve”, etc. This doesn’t include the time it takes to work these tools into your project, like k8s for example really requires you grok its model of how stuff it runs is supposed to look. There’s a reason entire companies exist that simply offer managed versions of these services!
If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running) and honestly it’s pretty fun to just run a command and poof you’re deployed. Then you can just wait until it starts making sense to use fancier tooling (more load, more sites/DCs, more services, more people/teams, etc). Monitoring is always good to start with, for other stuff a good litmus test is that price points for hosted k8s & ops stuff will start to make sense, you’ll be paying in time in any case. Who knows, maybe when that happens something new will pop up that makes even more sense. Until then, reducing the amount of stuff you need to worry about is always a good strategy.