I have both and the one offered is really weak. If you pay you can use Gemini but it doesn’t have agentic access to smart home controls meaning you can no longer ask it to turn on or off the lights
What kind of issues did you have with streaming? I also set up ollama on fly.io, and had no issues getting streaming to work.
For the LLM itself, I just used a custom startup script that downloaded the model once ollama was up. It's the same thing I'd do on a local cluster though. I'm not sure how fly could make it better unless they offered direct integration with ollama or some other inference server?
I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(
Hmm, having a machine for every process is part of it but I actually like that kind of isolation. Storage and bandwidth also add up fast.
For example, I could get a digitalocean vm with 2gb ram, 1vcpu, 50gb storage, 2tb bandwidth for $12/mo.
For the same specs at fly.io, it'd be ~$22/mo not including any bandwidth. It could be less if it scales to zero/auto stops.
I recently tried experimenting with two different projects at fly. One was an attic server to cache packages for NixOS. Only used by me and my own vms. Even with auto scaling to zero, I think it was still around $15-20/mo.
The other was a fly gpu machine with Ollama on it. The cold start time + downloading a model each time was kind of painful, so I opted for just adding a 100gb volume. I don't actually remember what I was paying for that, but probably another 20/mo? I used it heavily for a few days to play around and then not so much later. I do remember doing the math and thinking it wouldn't be sustainable if I wanted to use it for stuff like home-assistant voice assistant or going through pdfs/etc with paperless.
On their own, neither of these are super expensive. But if I want to run multiple home services, the cost is just going to skyrocket with every new app I run. If I can rent a decent dedicated server for $100-$200/mo, then I at least don't have to worry about the cost increasing on me if a machine never scales to zero due to a healthcheck I forgot about or something like that.
Sorry if it's a bit rambly, happy to answer questions!
I would be curious how the Attic server would have gone with a Tigris bucket and local caching. Not sure how hard that is to pull off, but Tigris should be substantially cheaper than our NVMes and if you don't really NEED the io performance you're not getting anything for that money. Which is a long winded way of saying "we aren't great at block storage for anything but OLTP workloads and caches".
One thing we've struggled to communicate is how _cheap_ autosuspend/autostop make things. If that Machine is alive for 8 hours per day you're probably below $8/mo for that config. And it's so fast that it's viable for it to start/stop 45 times per day.
It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?
We are suboptimal for things that make more sense as a bunch of containers on one host.
Tbh I haven't looked at Tigris at all. I still have my attic server deployed (just disabled/not in use) so I might give it a shot just to compare pricing. I do remember a decent portion of the cost being storage-related, so it's a good idea.
I'll have to look at autosuspend again too. I remember having autostop configured, but not autosuspend. I could see that helping with start times a lot for some stuff. It's not supported on GPU machines though, right? I thought I read that but don't see it in the docs at a quick glance.
> It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?
Sorry, I did mean external healthchecks. Something like zabbix/uptimekuma. For something public facing, I'd want a health check just to make sure it's alive. With any type of serverless/functions, I'd probably want to reduce the healthcheck frequency to avoid the machine constantly running if it is normally low-traffic.
> We are suboptimal for things that make more sense as a bunch of containers on one host.
I think my ideal offering would be something where I could install a fly.io management/control plane on my own hardware for a small monthly fee and use that until it runs out of resources. I imagine it's a pretty niche case for enterprise unless you can get a bunch of customers with on-prem hardware, but homelabbers would probably be happy.
For instance this is a python flask app, it uses an sqlite DB, but I wouldn't really call it dynamic nor needing state because the DB is read only (gets updated daily from an authoritative source, but a website user can only search the db). It costs about $3.50 a month, but could be less because I have two instances at the ready (although only one is ever active/up).
But if you start needing state or persistence on an instance down/up I'd probably not go with Fly myself personally..
But I think it fits the bill perfectly for like that sort of in-between static and true dynamic. Eg not exactly static (for truly static I'd just use shared hosting or CF Pages, etc, et al), but not really truly dynamic either or needing state. That's sort of the sweet spot in my view. Package up your flask or fastapi or FastHTML into a Dockerfile, send to fly, easy peasy, love it.
I can't wait to play around with FastHTML and fly btw, haven't done so yet.. also would like to play around with a Datasette DB instance on fly..
That sounds kinda weird to me. I use fly.io exactly because the pricing works out for hobby use. I can enable my machine for a few hours, run a bunch of inference, and turn it off again. The whole auto start/stop thing makes it seamless too.
Either you're familiar with GPU pricing and being willfully ignorant, or you're not familiar with the pricing in which case let someone who is point out:
- "Datacenter" means it's comparable to Runpod's secure cloud pricing.
- A spot instance of an H200 under someone's living room media console wouldn't go for A100 rates.
$3.50 will also get you an H100 at a laundry list of providers people build real businesses on.
Certainly all better track records than fly.io, especially on a post where they explain it's not working out for them as an offering and then promise they'll keep it shambling along.
You seem like you're familiar with vast. Have you used their autoscaler/serverless offering before? I haven't tried it yet, but it wasn't immediately obvious if I could have something like ollama running and scaled to zero instances when not in use.
Not them, but Runpod and Vast are my gotos. Runpod cost slightly more, but is in turn more reliable, so for "hobby pro" I'd go with them, otherwise Vast.
Salad Cloud is also very interesting if your models can fit on a consumer GPU, but it's a different model than typical GPU providers.
I have a decent sized homelab in my basement that I use for most stuff, and then a couple cheap-ish dedicated servers for public-facing things. Nothing has GPUs though, so I don't have a good solution for llm/ai projects yet.
I used to use cheap vms/vps from lowendtalk deals, but usually they're on over-subscribed hosts and can't do anything heavy.
Actual host recommendations: I like Racknerd and Hivelocity currently. OVH too, but I've read a lot of horror stories so I guess ymmv.
> It seems with the git command line the way to do it is to switch to main then pull then switch back to my feature branch then rebase.
In case you haven't figured it out yet, you can do `git fetch origin` to fetch the latest branches from origin. And then `git rebase origin/main` to rebase the current branch against origin/main. origin/main doesn't have to be the same as local `main` so you don't need to switch branches at all.
> ... For a Yubikey to act as a poor man's HSM you have to store the PIN in plaintext on the disk. ...
I haven't read the article fully yet, but it's not a bad idea to store the Root CA on the yubikey, and then generate a separate intermediate CA that is not stored on the yubikey. This way, all your day-to-day certs are issued using the intermediate and you only need to touch the root ca if you need to re-issue/revoke/etc the intermediate.
This is what I do as well. I use acme.sh on one linux server to generate a cert with a few SANs on it, then copy that cert to things like opnsense/truenas/etc either using ssh or their api if there is one.
