Very interesting discussion.
I work on a 2D MMORPG for Android.
This is extremely relevant to me. I have a few questions though.
What if you take compression and deserialization out of the picture?
For example, in my server I have a hash like data structure that gets turned into JSON for browsers and byte array for mobile clients.
For example, because the data has to be transferred at fast rates and will be going over mobile networks. The size of the packet matters because every millisecond counts.
Then to read the data, I simply read the stream of bytes and build the objects I need on the client.
This has to happen mostly without allocations for example on Android to avoid the GC.
So a few questions:
Does deserializing JSON cause any memory allocations?
If you're not tokenizing the data and don't need to parse it, will it be a significant gain over s serialized byte protocol or JSON?
In any case, I'll experiment on my end and perhaps blog about my own findings.
Disclaimer: I'm authoer of MessagePack for C++/Ruby and committer of one for Java.
As for strings, JSON has to allocate memory and copy to deserialize strings because strings are escaped.
MessagePack does't have to allocate/copy because the serialized format of strings is same as the format in memory. But it depends on the implementation whether actually it doesn't allocate/copy.
C++ and Ruby implementations try to suppress allocation and copying (zero-copy). But Java implementation doesn't support zero-copy feature so far (we have plan to do so. Here is "TODO" comment: https://github.com/msgpack/msgpack-java/blob/master/src/main...).
As for the other types, C++ implementation (and new Ruby implementation which is under development) has memory pool which optimizes those memory allocations petterns. But it's hard to implement such optimizations for Java because JVM (and Dalvik VM) doesn't allow to hook object allocation.
Interesting, thanks so much for the response.
I'll keep this in mind as I continue to develop my app. It's still between custom byte stream, json, thrift, etc.
But MsgPack looks interesting as well and, if anything, these blog posts have brought it into the light for me.
I looked at the java class and what might help is if you can set a buffer size and use that buffer to store the data in the buffer and expand it if necessary. But that seems like a lot of work.
But yeah, not sure if you can optimize based on usage patterns due to the constraint you said.
In any case, great stuff and thanks for the info.
What if you take compression and deserialization out of the picture? For example, in my server I have a hash like data structure that gets turned into JSON for browsers and byte array for mobile clients.
For example, because the data has to be transferred at fast rates and will be going over mobile networks. The size of the packet matters because every millisecond counts.
Then to read the data, I simply read the stream of bytes and build the objects I need on the client. This has to happen mostly without allocations for example on Android to avoid the GC.
So a few questions: Does deserializing JSON cause any memory allocations? If you're not tokenizing the data and don't need to parse it, will it be a significant gain over s serialized byte protocol or JSON?
In any case, I'll experiment on my end and perhaps blog about my own findings.