This looks promising! We use the go-grpc SDK in conjunction with gogoprotobuf, and it's been a rocky road.
While the article identifies some operational issues (e.g. the reliance on HTTP/2), there are several considerable deficiencies with gRPC today, at least when using it with Go:
1. The JSON mapping (jsonpb.go) is clumsy at best, and by this I mean that it produces JSON that often doesn't look anything like how you'd hand-design your structures. "oneof" structs, for example, generate an additional level of indirection that you typically wouldn't have. Proto3's decision to forego Proto2's optional values (in Proto3 everything is optional) cause Go's zero value semantics to leak into gRPC [1]. (We had to fork jsonpb.go to fix some of these issues, but as far as I can tell, upstream is still very awkward.)
2. The Go code generator usually produces highly unidiomatic Go. "oneof" is yet again an offender here. The gogoprotobuf [2] project tries to fix some of go-grpc's deficiencies, but it's still not sufficient. Ideally you should be able to use the Proto structs directly, but our biggest gRPC project we basically gave up here, and decided to limit Proto usage to the server layer, with a translation layer in between that translates all the Proto structs to/from native structs. That keeps things clean, but it's pretty exhausting work, which lots of type switches (which are hampered by Go's lack of switch exhaustiveness checking; we use BurntSushi's go-sumtype [3] a lot, but I don't think it can work for Proto structs, as it requires that a struct also implements an interface).
3. Proto3 has very limited support for expressing "free-form" data. By this I mean if you need to express a Protobuf field that contains a structured set of data such as {"foo": {"bar": 42}}. For this, you have the extension google.protobuf.Value [4], which supports some basic primitives, but not all (no timestamps, for example) and cannot be used to serialize actual gRPC messages; you can't serialize {"foo": MyProtoMessage{...}}. Free-form structured data is important for systems that accept foreign data where the schema isn't known; for example, a system that indexes analytics data.
From what I can tell, though, Twirp doesn't "disrupt" gRPC as much as I'd like, since it appears to rely on the existing JSON mapping.
Yeah, I agree with pretty much everything you've written here.
> 1. The JSON mapping (jsonpb.go) is clumsy at best
The best thing for optional fields in jsonpb is to use the protobuf wrapper types [1]. They have special support in jsonpb to serialize and deserialize as you would expect, without the indirection. But the Go structs you get on the other end are a little weird, so its a tradeoff.
> 2. The Go code generator usually produces highly unidiomatic Go.
Yeah, using the generated structs as the main domain types in your code can be up-and-down. I agree that gogoprotobuf can help, but it's rough. We definitely use Getter methods on generated structs quite a bit for stuff like oneofs.
> 3. Proto3 has very limited support for expressing "free-form" data.
There's always `repeated byte` :) It sounds like a joke, but we've used it in some spots where the input is totally schema-less.
The Any type is also designed for this sort of thing. Still clumsy, though.
> > 2. The Go code generator usually produces highly unidiomatic Go.
> using the generated structs as the main domain types in your code can be up-and-down
At $DAYJOB we solve this by doing code generation outward from our domain types. The RPC layer is idiomatic Go because that's what we began with.
Some go/token and regexes take our structs and produce a server-side router implementation for net/http (endpoints from magic comments), some client-side libraries for Go / C++ (Qt) / PHP / JS, and documentation in markdown.
Our system is in a pretty reusable state, but nobody has the free cycles to open it. If Twirp had been available 24mo ago our project might have been different.
You mentioned problems with gRPC, but I think every one of your problems is with protobuf. Is that correct?
Also, regarding point 3, I'm confused with two things:
- You want "free form" data, but you're talking about protos in the context of Go. How would you define this "free form" data in Go?
- You explain that "free form structured data is important for systems that accept foreign data ... where the schema isnt known". Why are you using protobufs for this usecase? Protobufs are specifically meant to make the schema known, and be enforced by serialization.
True, but gRPC inherits these problems as it's based on Protobuf.
As for free-form data, it should be representable as map[string]interface{}. Our specific use case is a document store that stores documents on behalf of clients. The format of documents cannot be not known by the store, but the API to the store is gRPC. Also, we have a desire for documents to contain higher-level types such as dates, but we're forced to use google.protobuf.Value for this, and treat dates as strings, since Value cannot contain Proto types.
(Our next step is probably to model this explicitly, by defining our own Value message that uses "oneof" to represent all the possible types we need to support, and then using a map of these. But it would be nicer if Protobuf had first-class support.)
While the article identifies some operational issues (e.g. the reliance on HTTP/2), there are several considerable deficiencies with gRPC today, at least when using it with Go:
1. The JSON mapping (jsonpb.go) is clumsy at best, and by this I mean that it produces JSON that often doesn't look anything like how you'd hand-design your structures. "oneof" structs, for example, generate an additional level of indirection that you typically wouldn't have. Proto3's decision to forego Proto2's optional values (in Proto3 everything is optional) cause Go's zero value semantics to leak into gRPC [1]. (We had to fork jsonpb.go to fix some of these issues, but as far as I can tell, upstream is still very awkward.)
2. The Go code generator usually produces highly unidiomatic Go. "oneof" is yet again an offender here. The gogoprotobuf [2] project tries to fix some of go-grpc's deficiencies, but it's still not sufficient. Ideally you should be able to use the Proto structs directly, but our biggest gRPC project we basically gave up here, and decided to limit Proto usage to the server layer, with a translation layer in between that translates all the Proto structs to/from native structs. That keeps things clean, but it's pretty exhausting work, which lots of type switches (which are hampered by Go's lack of switch exhaustiveness checking; we use BurntSushi's go-sumtype [3] a lot, but I don't think it can work for Proto structs, as it requires that a struct also implements an interface).
3. Proto3 has very limited support for expressing "free-form" data. By this I mean if you need to express a Protobuf field that contains a structured set of data such as {"foo": {"bar": 42}}. For this, you have the extension google.protobuf.Value [4], which supports some basic primitives, but not all (no timestamps, for example) and cannot be used to serialize actual gRPC messages; you can't serialize {"foo": MyProtoMessage{...}}. Free-form structured data is important for systems that accept foreign data where the schema isn't known; for example, a system that indexes analytics data.
From what I can tell, though, Twirp doesn't "disrupt" gRPC as much as I'd like, since it appears to rely on the existing JSON mapping.
[1] https://github.com/gogo/protobuf/issues/218
[2] https://github.com/gogo/protobuf
[3] https://github.com/BurntSushi/go-sumtype
[4] https://developers.google.com/protocol-buffers/docs/referenc...