As someone who has been doing local-first for the last 3 years with Notesnook, let me tell you: it's not all gardens and roses. Local first has its own very unique set of problems:
1. What to do with stale user data? What happens if a user doesn't open the app for a year? How do you handle migrations?
2. What about data corruption? What happens if the user has a network interruption during a sync? How do you handle partial states?
3. What happens when you have merge conflicts during a sync? CRDT structures are not even close to enough for this.
4. What happens when the user has millions of items? How do you handle sync and storage for that? How do you handle backups? How do you handle exports?
One would imagine that having all your data locally would make things fast and easy, but oh boy! Not everyone has a high end machine. Mobiles are really bad with memory. iOS and Android have insane level of restrictions on how much memory an app can consume, and for good reason because most consumer mobile phones have 4-6 gbs of RAM.
All these problems do not exist in a non-local-first situation (but other problems do). Things are actually simpler in a server-first environment because all the heavy lifting is done by you instead of the user.
> 1. What to do with stale user data? What happens if a user doesn't open the app for a year? How do you handle migrations?
version = db.query("select value from config where key='version'").fetch_one()
switch (version) {
case 1:
db.migrate_to_version_2()
fallthrough
case 2:
db.migrate_to_version_3()
// ... and so on
}
assert(version == 3)
start_sync()
Just don't delete the old cases. Refuse to run sync if device is not on the latest schema version.
One of my Django projects started in 2018 and has over 150 migration files, some involving major schema refactors (including introducing multi-tenancy). I can take a DB dump from 2018, migrate it, and have the app run against master, without any manual fixes. I don't think it's an unsolved problem.
> 2. What about data corruption? What happens if the user has a network interruption during a sync? How do you handle partial states?
Run the sync in a transaction.
> 3. What happens when you have merge conflicts during a sync? CRDT structures are not even close to enough for this.
CRDTs are probably the best we have so far, but what you should do depends on the application. You may have to ask the user to pick one of the possible resolutions. "Keep version A, version B, or both?"
> 4. What happens when the user has millions of items? How do you handle sync and storage for that?
Every system has practical limits. Set a soft limit to warn people about pushing the app too far. Find out who your user with a million items is. Talk to them about their use cases. Figure out if you can improve the product, maybe offer a pro/higher-priced tier.
> Mobiles are really bad with memory. iOS and Android have insane level of restrictions on how much memory an app can consume, and for good reason because most consumer mobile phones have 4-6 gbs of RAM.
You don't load up your entire DB into memory on the backend either. (Well your database server is probably powerful enough to keep the entire working set in memory, but you don't start your request handler with "select * from users".)
You're asking very broad questions, and I know these are very simplistic answers - every product will be slightly different and face unique trade-offs. But I don't think the solutions are outside of reach for an average semi-competent engineer.
> You may have to ask the user to pick one of the possible resolutions. "Keep version A, version B, or both?"
For structured data, with compound entities, linked entities, both, or even both in the same entity, that can be a lot more complicated.
If a user has updated an object and some of its children, is that an atomic change or might they want the child/descendent/parent/ancestor/linked updates to go through even if the others don't? All of them or some? If you can't automatically decide this (which you possibly can't in a way that will satisfy a large enough majority of use cases) how do you present the question to the user (baring in mind this might be a very non-technical user)?
Also what if another user wants to override an update that invalidates part/all of their own? Or try to merge them? Depending on your app this might not matter (the user might always be me on different devices, likely using one at once, that is easier to understand than the user interacting with others potentially making many overlapping updates).
I think you misunderstand. My intention was not to say local-first is bad or impossible; it's not. We have been local-first at Notesnook since the beginning and it has been going alright so far.
But anyone looking to go local-first or build a local-first solution should have a clear idea of what problems can arise. As I said in the original comment: it's not all gardens and roses.
Just a few weeks back a user came to us after failing to migrate GBs of their data off of Evernote. This, of course, included attachments. They had successfully imported & synced 80K items, but when they tried to login on their iPhone, the sync was really, really slow. They had to wait 5 hours just to get the count up to 20K items. And that's when the app crashed resetting the whole sync progress to 0.
In short, we had not considered someone syncing 80K items. To be clear, 80K is not a lot of items even for a local-first sync system, but you do have to optimize for it. The solution consisted of extensively utilizing batching & parallelization on both the backend & the users' device.
The result? Now their 80K items sync within 2 minutes.
The problem wouldn't exist. This was about the phone fetching 80k new items from the server. If the phone just shows the item you're looking at, one at a time, and doesn't try to sync everything, there's no such problem.
I've been working on mobile apps developed for education in Afghanistan, rural Rwanda, etc for the last 9 years. I used to think that sync was the way, but I have learned from painful experience.
4 (extended): What happens when the user has access to millions of items, but they probably only want a few (e.g. an ebook library catalog)? Do you waste huge amounts of bandwidth and storage to transfer data, of which 99.9% will be useless? We had a situation where the admin user logging in on a project that had been running for years resulted in a sync of 500MB+, 99.99% of that data the admin would never directly use.
Also: do you make users wait for all that old data to sync before they can do anything when they login?
Relying on sync is basically relying on push-only data access. I think in reality most offline/local first applications will work best when they push what a user probably wants (e.g. because they selected it for offline sync, starred it, etc) and can pull any other data on demand (in a way that it can be accessed later offline).
Query-based replication works when you know what the user probably wants to have in advance (e.g. a device in a warehouse needs records for that stock in that warehouse, not others). But that's still push.
You still need pull on demand access when a user opens any random item where we don't know in advance what they probably want (e.g. a discussion board scenario).
I'd say you're spot on except for point (3). There's a number of crdt and event log approaches that, when combined properly in order to preserve user intent, can solve almost all merge issues of applications that do not require strong consistency.
> 4. What happens when the user has millions of items?
Partial replication is a problem I haven't seen many people solving but it is definitely the next frontier in this space.
I am the developer of RxDB, a local-first javascript database, and I made multiple apps with it and worked with many people creating local first apps. The problems you describe are basically solved.
> What to do with stale user data?
Version/Schema migration in RxDB (same for IndexedDB and watermelonDB) can be done in simple javascript functions. This just works. https://rxdb.info/data-migration.html
> What about data corruption?
Offline first apps are built on the premise that internet connections drop or do not exist at all. The replication protocols are build with exact that purpose so they do not have any problems with that. https://rxdb.info/replication.html
> What happens when you have merge conflicts during a sync?
> What happens when the user has millions of items?
There is a limit on how much you can store on a client device. If you need to store gigabytes of data, it will just not work. You are right at that point.
> How do you handle backups? How do you handle exports?
Some domains just don't have these issues, like note taking where data is small and there are many acceptable ways to handle conflicts, like backup files and three way merges.
Maybe the question is less "How to make this work local first" and more "How to squash down the problem domain into something that just naturally fits with local first"?
I wish we had something like an embeddable version of SyncThing, that had incoming file change notifications, conflict handler plugins, and partial sync capabilities, and an easy cloud storage option.
I think most everything I've ever wanted to p2pify could probably be built on top of that, except social media which is a separate really hard thing.
I don't think even you understand what you just said.
Consumer devices are notorious for their reliability problems. Compared to a full blown server that you have 100% control over with almost insane amounts of RAM & CPU power & a lot of guarantees.
Running a migrations on a server is far, far different to running it on every users' device. The sheer scale of it is different.
> Using 4 GB per user on your backend works?
That was a comment on the average RAM on a consumer device - not the total RAM required per user.
> Running a migrations on a server is far, far different to running it on every users' device. The sheer scale of it is different.
Well, it’s not only just that. Among the other things, some of the application instances would be outdated but still need to work, so you would need to support _all_ the DB schemes you have ever had for your app.
I know I understand what I said, and I am not convinced by anything you said.
What reliability problem would prevent you from running local-first software but doesn't interfere with running a thin client?
Why would the business logic part of your app require more RAM on the end-user device that it requires per-user (or per-document, etc) on a server?
Why do you claim that running migrations is so fundamentally different here and there?
If you want to argue I would appreciate you doing it with real arguments and experiences rather than even more unsubstantiated claims and statements like "you don't understand what you just said".
I was exploring an interesting local-first component earlier today: cr-sqlite, which adds CRDTs to SQLite so you can have multiple databases that receive writes and then sync up and merge their changes later on.
They're syncing client-side SQLite to server-side Postgres on backend via an Erlang service that enables bi-directional active-active replication. They have a little more work to go until they solve the RLS concerns, but its been an incredible project to follow.
Here's another local-first collaborative component: SQLSync [0].
"SQLSync is a collaborative offline-first wrapper around SQLite. It is designed to synchronize web application state between users, devices, and the edge."
I’m a big proponent of being able to run the code locally, but this seems to be more along the lines of, first update some local DB, then sync it.
I see nothing fundamentally different here than using a well designed API for ‘sync’, and none of it does anything to revive the web browsing experience. Gone are the days where “page load” meant that you could safely begin reading without interruption. This whole “web application” thing was a foobar’d mess. Yes, we probably want some level of interactivity, but I fail to see how casting the web application problem as a database sync issue addresses the fundamental issues.
That being said, I can see the appeal. It’s a clean interface to work with and it allows for new frameworks to displace existing problematic frameworks.
I just worry that in 5 years we’ll all be complaining about poorly designed schema and overloaded table updates again. The solution? Microservices with their own smaller tables which batch updates to the larger system of course!
It's local first because you write your data locally, first, then you sync it.
As opposed to writing it remotely first.
Also how many well designed APIs for sync really exist in the wild? I feel like you need a decent knowledge of distributed systems theory to even understand the problem space. "Last write wins" is a cop out and just means you are throwing away data semi-arbitrarily.
Exactly. A great working example is Obsidian. Every version of it - desktop, android, iOS, writes locally first and then syncs to remote (if you’re subscribed to the Sync service, otherwise just writes locally). Other than Sync (and some other optional services like Publish), it is 100% functional without network connectivity.
Call me old fashioned, but why would anybody ever think doing it differently was a good idea?
Having it run locally and only sending requests when really needed is a way to make your app feel reliable and snappy. If every ailly button of your app sends out requests that means the snappiness of your whole UI will depend on the mercy of the network gods. Maybe it is the thrill of the gamble?
> Call me old fashioned, but why would anybody ever think doing it differently was a good idea?
Most would probably agree, but then you're adding a layer of complexity on top of what could be a very simple client/server application that satisfies 80% of your users. There is more "cheap" software out in the world than robust, complex software because most software doesn't make it far from MVP.
I don’t think there’s any one reason. In the early web days there were no client-side data stores, just server middleware and database that did all the work, generated and updated web pages with a little bit of JavaScript on them.
It wasn’t until Web2.0 and iPhone that web and mobile began to have the ability to do the write local then sync model. But the old client-server model was still easier to develop for, had the momentum, and internet speeds were fast enough.
Nowadays there are services like Firebase that make it super fast and easy to develop full-featured apps. They provide everything an app needs - user accounts, sessions, data store, etc. All of which is still more work to replicate in the local first model. Afaik there’s nothing like Firebase for local-first.
> Call me old fashioned, but why would anybody ever think doing it differently was a good idea?
Simplicity. Ignoring the issues of broken conectivity, going straight to the server to write and reading back from it, removes the need for both store/retreive and sync - you only have to think about store/retreive. There are unavoidable sync issues (two users load the same object, edit for a while, then save) that you have to deal with but they can be more simply dealt with because you can assume a small number rather than each user potentially having been offline for days and written a novel over many updates to an overlapping set of documents.
Also historically people just haven't had good offline options and many still don't. If you have to contend with users on more ancient browsers/devices (which many of us unfortunately do) then you have to support online working too so if you want to priorities that needs to come first (it supports all possible users) and offline after (as an improvement for those that have devices to support the tech and storage space your solution needs).
Locking issues, while not easy in an online client/server setup, are simpler than offline-first. And by simpler I mean actually possible to deal with - with offline first you instead have to deal with conflict resolution.
Local-first / desktop apps were really easy back in early 2000's and before since users pretty much only had a single device.
Today, users have many devices with many different storage and compute constraints. They also expect their data to be available on all devices and, to top it off, be able to invite outside collaborators.
Handling this heterogenous landscape of devices and collaboration is much simpler in a cloud model. Trying to put more data and compute locally suddenly means worrying about a multitude of device profiles and multi-way sync.
That surely.. depends. I agree that the syncing requirements can make a centralized cloud architecture benefitial.
But there are many services where this is not the case, where a local first architecture would be benefitial.
If the snappiness of your application is of any priority you will have to do a lot of local caching anyways. I don't say local first is always the right approach (it isn't), but that there are many remote first apps out there which would be better if they had been done local first (e.g. because the state they are syncing is trivially simple).
I really like this way of thinking about it. I try to advocate for internetless development, so it meshes nicely with that.
It also helps me understand the scope of the frameworks. They need to maintain local state akin to saving files locally.
It’s just funny to me how we had this as the default mode of operation many years ago, then lost it because everything went to the cloud, and now we’re realizing that was kinda a mistake and we’re trying to fix it in the new environment.
The reasons why everything left an "internetless model" seem to be getting lost to time but I explained some reasoning here in this comment: https://news.ycombinator.com/item?id=37500449
Local first development is a misnomer. It should be local first persistence. All it does it trade-off the complexity of API management against synchronization of distributed persistence.
The post writes about multiplayer, especially at the end. In games, but also in basically every other application, there is a state known to players (the cards in hand, the map around, etc) and a state that must be secret (the cards of the other players, the rest of the map, etc.) The server with the knowledge of all the state can be a client like the others but it has access to the real database. It can have the code to resolve interactions between players or that code runs locally to players too.
In complex turn based games there is a lot to do locally before sending the move to the server. Think about moving tens of units on a map. The problem here is of there are unknown parts of the map to explore. You need a connection to the server. However anything else will run faster because it doesn't have to hit the network and a round trip to the database on the server.
The post writes about Figma being developed local first. You don't have to hit the network while drawing, if you don't have to collaborate with someone on a shared document. Then you send the state to the server and your private state becomes known to all the other coworkers.
I think this article is focused on "multiplayer" in the contemporary non-game sense (which I still find weird).
However, I think an interesting thought experiment is a system that DOES share that secret state between players in a way where some facts are verifiable, but others are not. For example, when playing cards, I have the state of your hand in such a way that I can verify the card you played is valid, but cannot reasonably see (or find out) your hand.
I just want to say that I spent a long time re-reading your comment to understand your criticism before I saw that you're actually complimenting this idea, lol. I think that's a great way to highlight the strength of this approach: adding access control to such an app is about as easy as adding it to a simple DB.
This is a great article. I've been working on tooling related to local-first/offline-first for the last 10 years, and the benefits both for end-users and developers are great.
The biggest benefit for developers that I've seen is that state management gets _much_ simpler when it is all handled by a local database. Just run queries directly on every app/page, with live queries if the framework supports it. No need for maintaining in-memory app state and propagating changes to the different parts, no need for caching and associated cache invalidation issues.
The caveat is that keeping the local database in sync with the server is not easy, and it's best to use other tooling for it. I'm working on PowerSync [1] myself, and ElectricSQL [2] also looks quite promising (both also mentioned in OP).
I would love it if it could also address the question of data sovereignty for individual users somewhere down the line. I know it's not a priority when you need the product to first hit the market, start competing, become profitable, etc but it would be such a breath of fresh air in a world where everything you think you own is actually rented.
Is it a great article? I have to read several paragraphs before I can begin to piece together a vague idea of what it's talking about. Starting with a brief definition would massively improve it.
About 12+ years ago, i "invented" similar RPC-over-couchDB architecture as there wasn't much other distributed ways.
Worked wonders - each user reading/posting requests/responses in own "channel", the channel being replicated to server and any of user's devices, central server responding to requests and eventually cross-polinating the channels if need be.
Did a web client (pouchDB), android (touchDB), ios (forgot the name).. essentially each platform has its own implementation of the relatively simple protocol of Couchdb. So all one does is listen/read/write to a single local db.. and then db is doing all the communication, whenever possible. Simplified the clients sooooo much - reduced work and complexity like 50%.
Eh, the thing didn't go anywhere as product but nevermind :/
A whole new generation of developers is learning what the previous generation termed "briefcase applications". When client-server applications were the thing late 90s and early 2Ks, internet speeds were a serious limitation. This forced many architectures to work with "local-first", disconnected dataset, eventually-synchronised desktop applications. Microsoft ADO famously touted the "disconnected dataset" for this after Borland's "client dataset" pioneered this concept for the Windows desktop. Eventually even Java got disconnected datasets. All these techs more than two decades ago had real practical problems they solved: one of them I worked on involved collecting water flow data from several thousands of rivers in Southern Africa. Hundreds of users had mobile laptops that fired up desktop apps with authentication and business rules. Users then came back to head office to synchronise or used a branch with network to upload.
They worked and they were necessary.
Things changed when internet connectivity eventually became cheap and ubiquitous and the complexity of developing such applications then didn't merit the effort.
Now, the swing back to "local-first" is mainly for user-experience reasons. The same theoretical complexities of "eventually synchronised" that existed for the "briefcase app" are still present.
Counter-point: you don't need local-first because you're privileged. The crappier and less reliable your internet, the more valuable local-first is - and of the ~6 billion smartphone owners, most of them live somewhere with spotty connections.
For most people in the first world though, you're probably right.
Ah yes, let's ask people on spotty connections to download a (likely) megabytes-large JavaScript bundle. What could go wrong?
Most of my users have old phones and bad connections. I've tried this JS-heavy bundle-first approach. It doesn't work.
The solution is way simpler than local-first. Just shrink every page and interaction. Fewer requests, little JavaScript (if any at all), low latency. Use static pages when possible. Even the oldest phones on the most remote connections can usually deal with a sub-50kb page all-in. It feels like people forget how simple web interactions can be.
I'm sure local-first can be great for highly interactive tools like Figma. But the grandparent is right. Most sites don't need anything close to that level of complexity.
I feel like you're conflating a few different things here.
Small pages and interactions are good, sure, but I don't really see a tension between this and local first. My homepage is static HTML (minus google analytics) and it works offline.
The multi-mb JS bundle is also a red herring. The only multi-mb JS bundle I ever worked with did not work offline at all. Feel this is orthogonal.
Also connectivity for people is usually something that changes with time. You have it in the farmhouse, but not out in the field. You have it at the office, but not on the road. So downloading stuff when you have connectivity and still being able to read/write what you don't is the real aim of the game here.
If you're using browser APIs to do lean local interactions, then good for you. Gold star. Fully approved. That's not what I've been seeing recently from the people around me who are most excited about this stuff.
And you're right about how connectivity changes over time. But how many people are on their bikes when applying for unemployment insurance? I just don't think most business apps benefit from this level of offline support. There are of course use cases for this! But it's not the common case.
And you're right, if it's OK to not be able to read or write during a network partition - then you don't need this. But I would encourage everyone out there to figure it out first as bolting it on after the fact is a real challenge.
Yes, but "local-first" and "Big JS" aren't related. Sure you can build any monstrosity and make it local-first. And my local-first software can be a Tauri app, installed-once, or any kind of local software for that matter.
Hmm I just stumbled on this and am loving it so far, and your objection interests me: I'd be curious to hear what you see as the cycle? Lets say in WebDev only.
If so, I don't really see how this is promoting thicker apps. This whole approach assumes from the jump that you have complex state interactions on the client, and is just providing a wrapper around that to simplify syncing that state with the server. Couldn't you write a very thin SSR app that uses this paradigm to offload as much business logic (e.g. update cascades) to the server as possible? I suppose it would lose the offline capabilities, but otherwise it seems like the same fundamental approach.
All of this is assuming we're not going with what Hacker News wishes was true: that pure static sites are becoming the standard again ;)
Let's not say in WedDev only, because the cycle predates web.
Thin client - dumb terminals connected to a powerful central computer; 1960s, 1970s onwards
Thick client - multi-tier architecture, 1990s onwards, with client typically running on Windows, communicating with an application server (three tier) or database (two tier); for two tier, almost all the application logic is in the client.
Thin client - web browser communicating with web server; ~2000 onwards. The early versions are OCX controls hosted inside IE, basically hosting a Thick client as a downloadable control inside a web page, but it gradually migrates to web native as the browser gets more capable (AJAX revolution). Web native is a step change back to thin client compared to a client app or OCX control.
Thick client - this article here, a continuation of a trend over the past couple of decades, as browsers increasingly resemble operating systems in the breadth of APIs, permitting building something closer to a classic two-tier architecture, with not much more than basic validation and data backup / sync support on the back end.
I work for Ably, a realtime messaging platform. I'm also the author of SQLEdge (a replication tool mentioned in the article). One of our teams at Ably just launched "spaces"[1] to make it easy to build collaborative experiences. While it's not strictly in the local-first camp, I think it's in a similar 'space' (pun intended).
I think the collaboration experiences that you see in Figma, Miro, Notion, etc should _just be possible_ for all apps. And for a lot of product development teams they probably look at what would be involved to try and build a realtime collaboration experience and think it's just gonna be either too hard, or too much work. But the 'art of the possible' as definitely moved on (as evidenced by the list of companies in the blog post that are addressing these challenges).
As a bunch of the other commenters point out, the hard-part isn't just building the code for the UI that will show collaboration experiences, the hard-part is often handling the stateful connections (websockets) and making sure the messages are delivered to the right clients. You either build a stateless/polling mechanism over HTTP, or you have to deal with trying to scale and load-balance websocket connections. Neither approach is particularly easy.
We created Linear[1] as local-first product. Back in to 2019 there wasn't much anything available so we created our sync engine. Today we have several thousands companies, from early stage to public companies as paying customers (companies such as Vercel, Replit, Substack, Square..)
Our co-founder did talk about sync engine back in 2020 and recently did updated talk about scaling it [3]. The scaling has definitely taken lot of work and optimizing as datasets for company wide workspace with potentially thousands of users ands object they created can become quite large.
It was off the back of that talk that I was able to finally convince my team to go this route (back in 2019). They'd been trying to handle optimistic updates through Apollo and I was pushing the to make our own sync system over our mobx models to fix the DX.
After watching the talk we picked apart Linear's minified code to better understand the structure and then jumped on a call with Tuomas where he kindly filled any holes in our understanding.
Initially we tried to use IndexDB to give us more of the local-first features by caching between loads, but it was more hassle than it was worth. Instead we settled on live queries using Hasrua (we were a very early user / paying customer). We preload all the data that the user is going to need on app boot and then selectively load large chunks as they open their projects. These are then keeping mobx models up to date.
For mutating data we have a simple transaction system that you wrap around models to update them. It records and sends the mutations and makes sure that outstanding mutations are replayed over model changes locally.
We're not very "local-first" feature-wise; we don't handle conflicts perfectly, really relying on the fact that people are online and receive updates quickly. In practice it works.
Most importantly developers get a familiar graph of objects locally that they can just manipulate and not think about:
I built Mochi [0] from the ground up to be local first. The architecture is built around pouchdb for the local database which syncs to and from a remote couchdb database.
It's been a challenge to implement and in hindsight I wonder if it was even worth it. Unfortunately neither of these technologies are very widely used any more (if they ever were).
I am glad there is a lot of development and research in this area though. I think with the right DX and architecture local first makes a lot more sense than the status quo of continuously fetching everything from a remote database.
I made an open-source Pashto dictionary with a (local-first) word list (with spaced repititon review) that syncs with PouchDb/CouchDb. It's a free dictionary but you pay to get the word list sync. The whole dictionary is designed to be an offline-first SPA so a local-first DB/sync for the word made sense.
Ditto (www.ditto.live) is deployed on all Alaska Airlines, Lufthansa and many other airlines.
It also powers point of sale and kitchen systems at Chick fil A.
The major thing that Ditto is that it can sync without internet using Bluetooth low energy, peer to peer WiFi and Local Area network. It can even do multi-hop.
I like the idea but I've not tried it, it just seems quite a bit more complicated than just accepting the need to be online.
I'm talking hobby projects here where my main problem is getting it to a usable state so that it works (for me) without quirks and actually solves my problem instead of being a proof of concept.
Assuming the data model is kinda boring I wonder if there's a way to leverage some "framework" to offload the syncing part, for a simple CRUD app. There should be no resolution strategies, it has like a users table, a posts table and a comments table (classic blog example), this should be easily abstractable, or not? But then most people are talking about Websites as frontends, I guess, which changes my current "I'm gonna write a Qt native client" idea.
Hearing the name, I first thought this was about reducing complexity and removing unnecessary dependencies on cloud services. But it seems rather about adding yet another layer of complexity: You still have to track the state on the server, but now you want to also track it on the client.
There’s a big risk that it moves in that direction, indeed. But that depends on what kind of tech gains traction. Imo, a great local-first app has a rich client and a dumb server that serves simple functions like relaying, backing up and auth.
The main challenge is social. Once you have true local first, lots of apps have no plausible excuse to store all your private data in plain text. I can already see the disappointed faces of the VCs and other bean counters when they understand this. But for good or bad, that also means that there will be very little money behind these efforts.
I'm curious what kind of app doesn't already require tracking state on the client...? Very confused about your objection, but probably just because of inexperience on my part. It seems to me that basically every web app I can think of (e.g. Facebook, Figma, ChatGPT, Spotify, Discord) revolves around the server and client sharing some responsibility for tracking and updating the state.
If your objection is that we should just have state as a DB table on a server and just sync it to the client as needed, then that's basically what this is -- just with a client DB instead of a client Redux store/bespoke javascript mess/etc.
Tried to get pouchdb / rxdb working on a number of projects over the years. Hoodie also. Ran into "too hard" problems every time and fell back to something else :/
I think what I liked the least was the complexity introduced to the client (a fun statement considering some of the apps I've worked on recently). Instead of being able to abstract away business logic behind a simple API, it was there in all it's "haute logique" glory - hardwired into my otherwise dead simple todo app.
That said, still love the concept. Keen to try some of the new names. Tho the idea of it living in a worker, where it can be queried / controlled / deployed (or replaced) as if remote, has a nice ring to it?
The JavaScript/webapp world will soon have reached its goal and have a complete operating system running the browser. Then someone will come along and develop some kind of browser in that operating system and the whole game starts again …
This is an awesome resource! Glad Triplit made the list of hosted services (we're also open-source)
One common misconception with local-first, is that it's essentially local-only. But I believe the best experience is where the server and client are equally capable and only a partial set of data is stored on the client. The server still maintains authority but the client is always snappy when updated data or changing queries/views. We basically call this a full-stack database and we're building that at [Triplit](www.triplit.dev).
The analogy is Traditional backend development is thin client software, reactive frameworks are networked software, and local first is the native software (keep data with the client). The browser is the OS.
Why not skip the whole App Store generation that’s to come and build a browser extension today?
Lots of old school desktop applications are built in a way where they query a server for every single query result/item detail page/etc. Practically every industrial application for boring things like warehouse inventory is like that. That's not "local-first".
Let's take Excel as an example: Excel as a desktop app is either local-only (editing a local file), or remote-only (editing a file on a shared network drive). Excel as it has existed as a desktop app for a long time is not local-first.
An email client using IMAP, fetching mail ahead of time and having a queue of outgoing mails, is local-first.
Despite the name it's not really a web exclusive community.
Actually the web isn't a great way to do this, there's alternate stacks that maybe can work better. The Android stack has been rewritten in recent years by Google and parts are now usable on desktop and iOS. Someone posted this to the Kotlin slack earlier today:
I’ve been working on apps for limited connectivity environments such as Afghanistan and rural Rwanda for many years. I think we have to make offline-first a lot easier than it is now. Our own app Ustad Mobile has more than 60 tables in the database. Writing manual logic to run all of those offline would 1) exceed the resources we have and 2) be prone to human error.
This article introduces Door: our open-source way to automate the generation of an offline-first data layer using Kotlin Multiplatform and Kotlin Symbol Processing (KSP). Door is still a work-in-progress and not ready for production use in other apps yet, but we think it’s a game-changer for offline-first development. Feedback on the concept and API is welcome. Door can automatically generate HTTP server endpoints and an offline first client based on a Room database.
The idea is that you can use Kotlin for both server, Android and also a desktop client, and with KMP you're also able to use it on iOS for the backend (UI is still in Swift). So you can share your domain model across all devices and the underlying sync protocols and logic can be auto generated using a compiler plugin. The underlying DB is SQLite wrapped using a library from Google called Room which makes SQLite a bit easier to use.
Deploying such an app on Android is obvious, for iOS you need to learn about Kotlin/Multiplatform but obviously SQLite can run on iOS just fine and Kotlin can interop with Objective-C/Swift. For desktop, deployment is easy if you use Conveyor, which the Ustad guys are planning to do, as it supports Kotlin desktop apps out of the box. You do need to buy signing certificates (or distribute via the app stores) but that's a one time cost.
The main issue here is if you're wedded to the web. Probably stuff can be done with WASM but it's not going to be as natural as just using the tech on the JVM.
Thanks for sharing that! On the Web: we are using Kotlin/JS and SQLite.js (which uses WASM and talks to the HTML via a web worker). But you're right, Kotlin and Javascript definitely do not connect as naturally as Kotlin and the JVM. Its possible, but it will be trickier.
1. What to do with stale user data? What happens if a user doesn't open the app for a year? How do you handle migrations?
2. What about data corruption? What happens if the user has a network interruption during a sync? How do you handle partial states?
3. What happens when you have merge conflicts during a sync? CRDT structures are not even close to enough for this.
4. What happens when the user has millions of items? How do you handle sync and storage for that? How do you handle backups? How do you handle exports?
One would imagine that having all your data locally would make things fast and easy, but oh boy! Not everyone has a high end machine. Mobiles are really bad with memory. iOS and Android have insane level of restrictions on how much memory an app can consume, and for good reason because most consumer mobile phones have 4-6 gbs of RAM.
All these problems do not exist in a non-local-first situation (but other problems do). Things are actually simpler in a server-first environment because all the heavy lifting is done by you instead of the user.