This looks simple and yet elegantly built. Nice work! I'm definitely a fan of the use of ListenableFutures as a way to integrate this into applications. Blocking, non-blocking, callbacks? Pick your poison.

I haven't seen Apache's HttpAsyncClient used in many places. The API looks sane enough. I'm not sure how stable it is compared to Netty; performance seems to be less of an issue given that it uses NIO.

How's jetcd on thread-safety?

I'd be interested to cook up a version that uses Netty or Vert.x, although there's not quite an excuse to do so...

Possibly off topic, this is the first I'm hearing of etcd. Any particular reason anyone would choose etcd over doozer (https://github.com/ha/doozer)?

Etcd, Zookeeper and Doozer are all supposed to serve similar purposes. I believe Doozer is no longer actively maintained though.

There are also three major features etcd has over doozer:

  - time to live keys
  - persistence to disk
  - TLS support for client and backend

Looks like I need to add TLS support to my client library!

There's a small misconception in the article: the overhead of zookeeper is not the reason why you tend to have small zookeeper clusters (3, 5, or 7 are recommended). The issue is that zookeeper requires consensus, so a write generally needs a response from a majority of nodes before it can finish. The more nodes you have, the longer it will take to reach consensus. There's also limited benefit in terms of HA of running more than 7 nodes.

The main issue is that you can overload it with reads (writes are always going to be slow in such a system). To deal with this, ZK is adding read slaves which do not participate in Paxos.

I didn't mean to imply that, and you're absolutely right.

The point I was trying to make is that, if you're just doing leader election for a single service (so you essentially have one key, probably 1KB of state), then spawning up even 3 JVMs is difficult to justify. That's Go's sweet spot for me - small daemons that you would otherwise be tempted to write in C.

I am excited by the new code in Zookeeper trunk which allows dynamic cluster reconfiguration; along with observers then you can have one zookeeper instance per machine if you want to, and the cluster can self-heal. But you still wouldn't want to do that unless you were actually sharing a non-trivial amount of data :-)

> spawning up even 3 JVMs is difficult to justify

Well, you could compile to native code instead.

Are you talking about RoboVM?

There are a few native compilers for Java, RoboVM is just one of them.

Which would you recommend trying out?

From what I have heard, Excelsior is a good one, http://www.excelsior-usa.com/jet.html

Additionally you have,

- Avian (http://oss.readytalk.com/avian/)

- Codename One (http://www.codenameone.com/), just for mobiles

- Aonix (http://www.atego.com/products/aonix-perc/)

- WebSphere Real Time (http://www-03.ibm.com/software/products/us/en/real-time)

- Jikes RVM (http://jikesrvm.org/)

> To deal with this, ZK is adding read slaves which do not participate in Paxos.

Are you referring to observers ? They've been in ZK since 3.3.*, if not before.

edit: actually you mentioned they're _read_ slaves, whereas you can write to zookeeper observers (they just don't participate in elections, which is as you said)

Also, ZooKeeper doesn't implement Paxos.

> Content Encoding Error

>

> The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.

Thanks for reporting... which browser/OS?

On Windows, all the usual browsers.

Tried Windows 7, Chrome/IE/Firefox - can't reproduce. If it affects IE and Chrome (which have almost no code in common), I'm guessing you are behind a proxy server of some sort, so I'm going to check my Caching / Vary headers. Thanks for the pointers!

Corporate firewall when it happened to me.

Thanks for confirming. I think you've found an issue with haproxy compression; compression support is quite new, and there are still issues being fixed. I'm going to move compression into the web server layer (it probably belongs there anyway!)