The current trend of using async logic via message passing to maintain state is extremely error-prone and I personally consider it nondeterministic if it uses timeouts (which is pretty much the entire internet currently).
A much better system works more like Firebase, where clients can subscribe to change events but also have access to a single tree that holds the latest agreed-upon state. I'd recommend something like a distributed hash table (DHT) so that the entire state doesn't have to be downloaded by each client. The state itself should probably be binary JSON to start with, but the standard should be loose and allow for XML or anything else. This would take us to something more UNIX-like, where devices can be thought of as processes interacting via a single shared memory, and abstract away the tedious/brittle/insecure interconnect layer. It should also support unreliable message passing like UDP for realtime support, which is required for simulations and gaming (there is a higher than 50% risk of the protocol omitting this or getting it wrong).
After writing this out, I just realized that what I really want is an Actor model for IoT, where each device is completely isolated but communicates over channels like Elixir/Erlang or Go. I've come to reject both async and concurrent shared mutable memory because they each have major drawbacks. But I think that synchronous functional programming with primarily immutable variables and no shared mutable memory, building up a last known-good state inspectable by anyone with privileges is the future. The state could/should have something like an access control lists (ACL) and/or a permissions system. Ideally this should be hierarchical just like Firebase, and probably use a similar rule-based logic, to facilitate assigning roles to each client.
It's trivial to demonstrate that once you have this, you can build any other seemingly complex abstraction above it like a database or game server. We have been missing this abstraction for several decades so we keep witnessing company after company reinvent the wheel. As far as I know, the only attempt to do this is RethinkDB:
As usual, I would gladly work on a project like this, but there is currently no funding mechanism to support foundational open source projects like this with market-rate pay.
How would you solve access permissions in DHT? What if a node in DHT holds data that it is not granted to read? This would imply encryption of the data to prevent unauthorized reads - this does not really seem as a complicated problem - the data could be encrypted with only the nodes granted the read access having the decryption key(s). What seems harder to me is write permissions - if the data is encrypted/signed, only those nodes that have been granted permissions (and thus have copies of keys) would be able to overwrite/change that data. So the node that holds the data would not be able to overwrite it if it does not have r/w access to that piece of data. Nothing prevents from removing this data though. (I'm assuming the possibility of some nodes being malicious)
I'm not sure for DHT. But in Firebase the permissions are handled as user access rules written in Javascript, returning boolean true or false if the user has the ability to read/write the node and if the data being written passes a validation check:
So Firebase permissions really have more to do with the identity layer than keeping the data secret.
I think that probably everything should be encrypted anyway. This was a largely solved problem even in the 90s, and embedded devices are about as powerful as the 386 and 68030 computers popular then.
We really need a distributed SSL identity layer though. Something like letsencrypt.org except the client's identity would come from the social proof of its peers instead of a central organization. Maybe this is already a solved problem in OpenSSL? I only understand symmetric key encryption, but not the man in the middle protections that SSL certificates provide. To me, p2p identity and encryption should have been a foundational pillar of the internet, not some controversial afterthought like we think of it today.
The usual solution for this is that the data must be signed by a specific private key; possession of that key grants write permission. A generation count or timestamp is embedded in the signed data to ensure that newer data cannot be replaced with obsolete data. Malicious nodes can still block updates from being distributed, though, so you need more than one path to seed the data into the network.
https://en.wikipedia.org/wiki/Distributed_shared_memory
https://en.wikipedia.org/wiki/Software_transactional_memory
https://en.wikipedia.org/wiki/Consensus_algorithm
https://en.wikipedia.org/wiki/Raft_(computer_science)
https://en.wikipedia.org/wiki/Distributed_hash_table
The current trend of using async logic via message passing to maintain state is extremely error-prone and I personally consider it nondeterministic if it uses timeouts (which is pretty much the entire internet currently).
A much better system works more like Firebase, where clients can subscribe to change events but also have access to a single tree that holds the latest agreed-upon state. I'd recommend something like a distributed hash table (DHT) so that the entire state doesn't have to be downloaded by each client. The state itself should probably be binary JSON to start with, but the standard should be loose and allow for XML or anything else. This would take us to something more UNIX-like, where devices can be thought of as processes interacting via a single shared memory, and abstract away the tedious/brittle/insecure interconnect layer. It should also support unreliable message passing like UDP for realtime support, which is required for simulations and gaming (there is a higher than 50% risk of the protocol omitting this or getting it wrong).
After writing this out, I just realized that what I really want is an Actor model for IoT, where each device is completely isolated but communicates over channels like Elixir/Erlang or Go. I've come to reject both async and concurrent shared mutable memory because they each have major drawbacks. But I think that synchronous functional programming with primarily immutable variables and no shared mutable memory, building up a last known-good state inspectable by anyone with privileges is the future. The state could/should have something like an access control lists (ACL) and/or a permissions system. Ideally this should be hierarchical just like Firebase, and probably use a similar rule-based logic, to facilitate assigning roles to each client.
It's trivial to demonstrate that once you have this, you can build any other seemingly complex abstraction above it like a database or game server. We have been missing this abstraction for several decades so we keep witnessing company after company reinvent the wheel. As far as I know, the only attempt to do this is RethinkDB:
https://en.wikipedia.org/wiki/RethinkDB
Unfortunately, I'm not sure if it's still supported:
https://rethinkdb.com/blog/rethinkdb-shutdown/
https://lwn.net/Articles/713716/
As usual, I would gladly work on a project like this, but there is currently no funding mechanism to support foundational open source projects like this with market-rate pay.