Without any explicit admission of guilt on my part; what are some good options here? Protobuf is cool but I really don't want to mess with a special compiler and all that.
The responses you got I think literally answer your question but probably aren't what you're going to reach for any kind of HTTP based thing. Your go-to should probably be multipart/form-data. It's well supported in every language and HTTP library and you can send both JSON and the file in the same payload.
There seems to be a common trend of people writing "JSON APIs" thinking that every other part of HTTP is off-limits.
It’s not even an HTTP invention, it’s RFC2046 MIME from 1996. RFC2388 standardised its use in HTTP in 1998.
The elegant thing about MIME is it allows multiple encodings and cross-references, so you can have your HTML and the images displayed in the HTML both optimally encoded in the same document, which was handy back in the time when HTML emails were taking off and marketing insisted that the fancy image signature they designed had to show every time, even when the person was reading the email offline…
Of course back then we had to encode the image in base64 anyway because of non-8-bit-clean email servers. But I digress and will go back to my rocking chair.
Avro has some really cool features like inbuilt schemas, schema versioning and migration (e.g. deprecating or renaming fields) but you pay for them with more overhead than MessagePack.
Protocol Buffers have schemas too (though versioning and ensuring compatibility is quite messy and requires understanding internals). And it has less overhead than MesssagePack.
I'm not sure what Avro is doing, but as a rule schema enables you to have less overhead, rather than more. The main advantage of MessagePack over schema-based formats is that it's dead-simple and mostly compatible with JSON. Schema-based formats usually need either a code generator or maintaining an annotated version of your data classes and making sure they match the schema.
(Of course, with JSON or MessagePack you might still end up using a serialization library and something like JSON Schema).
Oh, as I understand it Avro's schemas aren't just "built in" as in supported as a first-class part of the protocol, but rather that each message includes with it the schema needed to interpret it. This adds overhead to every message (it's still a binary protocol, though) but crucially it avoids a whole category of hassles around schema distribution and updating.
I just can't get over Cabo's choice to put 65 (yes 65) Bit fixed sized ints in the wire format and to top it off by making it ones complement (the sign bit is part of the type and the longest fixed sized ints are 64 bit range)...