Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Distributed SQLite with Elixir (silbernagel.dev)
77 points by clessg on Feb 16, 2023 | hide | past | favorite | 18 comments


Manually replicating writes to all nodes seems pretty fragile. At some point you're going to run into classic distributed systems issues, like netsplits, and reconciling the latest count. Maybe a CRDT implementation would avoid these issues alltogether?

But it sounds like LiteFS (with a single writer?) might be a pretty nice setup if it handles replication for you, as you probably don't want to build a Raft or Paxos system yourself.


Litestream isn't for those kinds of use cases. It's for DR, not availability and/or scale.


Litestream is different from LiteFS: same author, different projects.

LiteFS provides replication with the ability to serve read traffic from those replicas. Litestream provides disaster recovery by streaming a constant backup to an S3 bucket (or similar).


Replicating writes to all nodes does not sound scalable. Is it?


Whats DR?


Disaster Recovery


This is such a weird post.

They build a distributed counter that is consistent across a cluster using the Elixir message passing thing, but there's this whole thing about using the Fly Distributed SQLite thing as well.

But if they wanted to build a distributed counter they could choose to use Postgres at the first step and then it will just work.


For elixir+phoenix a cockroachDB store abstraction is natively supported.

I like the design in many ways, but still need to find something noncritical to test the solution on before clearing it for production. =)


cockroachDB uses Postgres wire protocol and SQL


Cross compatibility is a huge plus from my perspective.

However, there are some designs one should avoid... like auto increment indices etc. =)



> This is because we are not replicating the data between the instances. This is a problem Elixir is built for…

What feature is this and does it not really exist in C# or Java?


The feature is distributed message passing (basically an Actor model) https://www.erlang.org/doc/reference_manual/distributed.html

You can get this in other languages with the help of middleware like RabbitMQ or Kafka, but not many languages have it built in.


You can’t get it in C# or Java. The BEAM virtual machine is build upon the idea of lightweight processes. You might get an imitation in other mainstream places but not the real thing. Take a look at the Actor Model if you haven’t heard of it.


> might get an imitation in other mainstream places but not the real thing.

Could you further elaborate on this?

I am familiar with Actor Model. I had work related to Akka in Scala that works on the principles of Actor Model.


Erlang uses a preemptive scheduler for process scheduling, allowing for parallelization on multiple CPUs, while Scala distinguishes between thread-based and event-based actors. Thread-based actors can perform long-running computations without hindering other actors, but use more memory and scheduling overhead. Event-based actors are more lightweight but not suitable for parallelism as they execute on the same thread.


I think the cleaner thing would have been doing event sourcing utilizing OTP and rendered to local SQLite instances.


Doesn't event-sourcing require a central datastore where all events are persisted before building the sqlite readmodels?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: