> The complaint is that, somewhere, under the hood, simplejson is doing that token parsing, but because of their API, a user can't plug into it.
I read it exactly the other way around: simplejson skips an internal tokenization step, so even if you fork the library it is pretty much impossible to make it streaming, because there's no token stream to handle stream state.
That's more or less right, masklinn. To the above writers, look at the Python source (I can't vouch for how the C implementation does things). It's not beautiful, but it's also not hard to understand.
The effective work is all done inside a big function. To customize it, you'd have to split that function into pieces and then glue it back together using a class or your own function coordinating the parts.
I read it exactly the other way around: simplejson skips an internal tokenization step, so even if you fork the library it is pretty much impossible to make it streaming, because there's no token stream to handle stream state.