+1. So many cool desktop app ideas showing up in HN every now and then, yet most of them Electron-based web stuff that just feels horrible. At least Qt would be a lot more appreciated. So much missed potential
It's clearly a matter of taste. I, on the other hand, can't stand Qt or apps that try to look "native" in general. At least on Windows, it feels like going back to the previous century. The modern look, known from websites, is the only one that works for me, hence I use Electron. This way I also have full control over the UI and it looks the same on every OS.
> I, on the other hand, can't stand Qt or apps that try to look "native" in general.
100%. That's why I said "at least" and it's the feeling I have with Electron too. Electron apps (nor Qt ones) do not really feel native, and in that case, better to either go full native (so it doesn't feel like an imperfect approximation) or just deliver a web app that you can use on a browser?
The in-between ends up in a gray area that never feels quite right. But I agree it is in part a matter of style and expectations.
Though I also agree the Win32 look is terrible and outdated. GTK and Cocoa on Linux and macOS are really great and good looking native technologies. I've seen more and more projects target GTK on Windows instead of Win32 for this reason.
I feel comfortable with web technologies, and a large part of this project was using Monaco Editor as a powerful diff viewer, so I'll stay with my current stack. I'd be happy to make a pure web app, but I don't think it's possible with something like a Git client, since it needs to call system commands. Maybe there are some hacks available for local apps, but why would that be better than just using Electron. Yeah, it takes some space, but what is this compared to, for example, any modern computer game.
Can you explain why? I feel like more people are familiar with web technology and from my own subjective experience it's much simpler than qt for example
It depends from which point of view you are talking about.
A lot of people are familiar with web technologies, therefore using something like Electron is way easier for them. That makes a lot of sense.
However, from an end user point of view, Electron (while potentially easier to developer with for a large pool of developers) doesn't feel native. You can tell you are running a web app inside that doesn't obey the OS conventions, the standardised OS shortcuts, looks different than the rest, etc. It's like it doesn't quite match and all the muscle memory you have for working with other native apps (mainly for keyword-heavy users like myself) just doesn't work, making it a frustrating experience. Plus many (not all!) Electron apps are super heavy weight and feel slow when you contrast them with other truly native apps.
Overall, I think you will see a lot of people that don't really mind Electron, but many do. I think it largely comes down to whether you want to develop a desktop app faster yourself, or deliver a desktop app that would satisfy almost every user out there (which might be harder to build).
And BTW, this is coming from somebody that worked a LOT with Electron, as the original author of Etcher (https://github.com/balena-io/etcher), plus I led the Desktop Team at Postman (https://www.postman.com, which is arguable one of the worst Electron apps out there, mostly due to really bad early architecture/business decisions) for a while. I tried everything but I gave up on it. It can never even be a good enough approximation to a native experience that many users expect.
In any case, great job with GitQuill. It does look pretty cool and I wish it was a native app!
Developing apps with Qt and QML is super easy these days. I wrote a post about my experience[1]. QML is such a joy to program the UI in, and then you can use a compiled language–C++/Rust/etc for amazing performance gains. Also, most Qt Quick components (those exposed via QML) are written in C++, so using these native components means that even working with these abstractions is very performant.
To the article's point, many/most JavaScript projects are not optimised and better performance can be achieved with just JavaScript, and yes, JavaScript engines are becoming faster. However, no matter how much faster JavaScript can get, you can still always get faster with other system languages.
I work on high-performance stuff as a C++ engineer, currently working on an ultra fast JSON Schema validator. We are benchmarking against AJV (https://ajv.js.org), a project with a TON of VERY crazy optimisations to squeeze out as much performance as possible (REALLY well optimised compared to other JavaScript libraries), and we still get numbers like 200x faster than it with C++.
Not building an app myself right now, but I worked a lot on the desktop app space, leading development at various companies and been thinking of writing a book on the topic (I'm an O'Reilly author in another space). I tend to blog on macOS stuff every now and then, like https://www.jviotti.com/2022/02/21/emitting-signposts-to-ins....
Quick questions for everybody here:
- How do you develop your apps right now, mainly cross-platform ones? QT? Would you enjoy a C++ cross-platform framework that binds directly (and well) to Cocoa?
- Are you using Electron? If so, would you appreciate premium modules for Electron that bring a lot more "native" Cocoa functionality instead of reinventing the wheel in JavaScript?
What would you wish a book on modern desktop app development would cover?
Great stuff! I'm starting to work a lot on the satellite space (https://www.sourcemeta.com), building a binary serialization format around JSON called JSON BinPack (https://jsonbinpack.sourcemeta.com) that is extremely space-efficient to pack more documents in the same Iridium uplink/downlink operation (up to 74% more compact than Protocol Buffers. See reproducible benchmark here: https://arxiv.org/abs/2211.12799).
It is still a heavy work in progress, but if anybody here is suffering from expensive Iridium bills, I would love to connect and discuss to make sure JSON BinPack is built the right way!
Hey Juan, this looks promising - using a pre-shared schema would allow you to reduce payload sizes in the same way using pre-shared dictionaries makes compression more efficient (such as with zstd or brotli). Having worked on exactly this space in the past (sending and receiving messages from a microcontroller that eventually go through an Iridium SBD modem), some feedback:
After spending a few minutes poking around your landing page, I'm not sure how JSON BinPack works. I see a JSON Schema, a JSON document and then a payload, but I'd be interested in the low level details of how I get from a schema to a payload. When I'm on a microcontroller, I'm going to care quite a bit about the code that has to run to make that, and also the code that's receiving it. Is this something I could hand-jam on the micro, and then decode with a nice schema on the receiving side? Understanding the path from code and data to serialized payload would be important to me.
One thing that was nice about using ProtocolBuffers for this was that I had libraries at hand to encode/decode messages in several different languages (which were used) - what is the roadmap or current language support?
I can understand how ProtocolBuffers handles schema evolution, but I'm still not sure how JSON Schema evolution would work with JSON BinPack. An example would move mountains.
Finally, if I were digging into this and found it had switched from Apache to AGPL, and required a commercial license, it would be a hard sell vs all the permissively licensed alternatives. At the end of the day, Iridium SBD messages are like 270 bytes and even hand rolling a binary format is pretty manageable. I think I could swing budget for support, consulting, maintenance or some other service. But if I were evaluating this back when I needed a serialization format, and ran into AGPL, I would bounce almost immediately.
Thanks for the very valuable and honest feedback. All really good points, mainly how lacking the website is in terms of documentation at the moment. Making notes and plan to address all of it!
I'm working on JSON BinPack (https://jsonbinpack.sourcemeta.com), a binary serialization format for JSON (think of it as a Protobuf alternative) with a strong focus on space-efficiency for reducing costs when transferring structured data over 5G and satellite transceivers for robotics, IoT, automotive, etc.
If you work at any of those industries and pay a lot for data transfer, please reach out! I'm trying to talk to as many people as possible to make sure JSON BinPack fits their use case well (I'm trying to build a business around it).
It was originally designed during my (award-winning!) research at the University of Oxford (https://www.jviotti.com/dissertation.pdf), and it was proven to be more space-efficient than any tested alternative in every single tested case (https://arxiv.org/abs/2211.12799), beating Protocol Buffers for up to ~75%.
While designing it was already difficult, implementing a C++ production-ready version has proven to be very tricky, leading me to branch off to various other pre-requisite projects, like an ultra-fast JSON Schema compiler for nano-second schema validation (https://github.com/sourcemeta/jsontoolkit) (for which I'm publishing a paper soon).
Don't know if you'll see this, but I really like your work. I read your dissertation at least twice, it's well written and informative.
In my day job I have multiple large json files, used in an internal in-house application, into which I serialize a database, and working with them is much faster than using the database (for some workloads). Although the main reason to do so is just for using git for source control.
In the future I plan on migrating all the code to working off the json files, but that won't happen for some time yet.
I was looking for a fast json library and that's why I discovered your paper and your project, although that was some months after I have started working on that project and it was too late to change.
Currently I use the (very slow) json.net library for most serialization/deserialization, it's fast enough for the specific places where I use them, but for some specific workload that has to be fast I use RapidJson (the code in that part is already c++), but only its streaming parser, which blazingly fast, at the cost of some development. I tried using simdjson but had trouble compiling it at the time.
By the way, in your dissertation you don't mention RapidJson, nor json.net, nor simdjson. Is there a reason why you didn't compare them?
Hey olvyo, thanks for your nice comments! We should definitely connect!
> By the way, in your dissertation you don't mention RapidJson, nor json.net, nor simdjson. Is there a reason why you didn't compare them?
My research was focused on space-efficiency more than parsing performance, so I didn't talk about that problem. However the long-term goal is that by using JSON BinPack's binary encoding, you should have a better time parsing it, as you don't have to deal with the JSON grammar.
The json storage we currently use is kind of "secondary" to the database, and so there currently we don't verify it. The relational database we use does have a schema, of course.
Not verifying the json schema is something that is sorely missing, since the files get corrupted from time to time and this is discovered only during deserialization. Unfortunately this is something I don't have enough time to add, as we're swamped with other work (our app is huge and we are a very small time). We just tell our internal users to re-serialize the database to fix these corruptions, which is unfortunate and costly, but the best we can do at the moment.
I wrote more about that aspect of my internal app here.
Like I wrote, when in a future version we'll drop the database completely, and work only off those json files, I'll also introduce schema validation. That won't happen for some time though.
Thanks for your answer, I understand now why you didn't measure the performance of simdjson etc. into account.
I have thought in the past about using BinPack for my json documents, but: I want them to remain as human readable as possible, since the reason to move to json from a DB, was to make database diffs into readable diffs, and BinPack isn't readable.
I also want users to be able to use existing tools on the json files (e.g. the jq tool), but existing tools don't understand BinPack (yet?).
I think BinPack would shine in an RPC/IPC setting. Just recently there was this big discussion here about systemd replacing D-BUS with a json-based IPC and a huge discussion around the waste of using plain json to do that.
Currently experimenting with programmatic generation of json schemas via https://github.com/sinclairzx81/typebox. Trying to maximize reuse of schema components.
Was wondering if JSON BinPack is a good serialization format to sign json documents?
I Googled a bit (I'm not a consumer of these types of YouTube videos) and apparently this guy is pretty rich by now. He identified a path to wealth that clearly works and exploited it well, so he is definitely smart. Well deserved from that point of view.
However, it makes me sad that we as a society even let what he does (a zero-net as many pointed out) be a path to wealth on the first place. The fact that we reward so much something that is so void of value is our fault and IMHO what needs to be corrected somehow. It's sad to think we throw so much money at people covered by ketchup, but many PhD's barely make any money and we cannot figure out how to make open-source sustainable.
If our reward schemes were more aligned with actually providing value to society, this guy might have applied his intelligence and amassed his wealth doing something more worthwhile for everybody.
Mr Beasts YouTube videos make significant amounts of ad revenue. Some of that money goes to YouTube. But there is nothing wrong with having most of that money go to the creator of the videos.
You can apply this logic to other forms of entertainment too. You can say LeBron James or Michael Jordan just play basketball. But their basketball games generated enormous ad revenue. Shouldn’t a sizable chunk of that go to them and other NBA players that play the game?
Same logic for late night show comedians, all other big entertainment YouTube channels, tv stars, movie stars, movie directors, podcasters like Joe Rogan and so on.
All of these have a thing in common. They release content in some media like traditional tv or newer media like YouTube and they are primarily supported by advertising in their content and through merchandise sales.
I think it’s actually more moral for these people to be making lots of money than otherwise. Because otherwise the money generated by their content just goes to YouTube executives or tv executives or Spotify executives or some other corporate big wigs.
If ANY entertainment makes a ton of money shouldn’t the money go to the ones who create it?
In tv and movies, the Writer’s Guild and the Directors Guild routinely go on strikes for this issue.
Also I think it’s great George Lucas made billions of dollars on Star Wars. Many adults in the 1970’s thought Star Wars was junk for the mind…
JSON Schema is the industry standard schema language for JSON, used by big players like OpenAPI and by an insanely high number of APIs and products out there.
The TSC core team stays at ~6 people (I'm one of them)
Very cool! How are the balloons transferring telemetry back to earth for analysis, etc?
Asking because my research at the University of Oxford was around hyper space-efficient data transfer from remote locations for a fraction of the price.
The result was an award-winning technology (https://jsonbinpack.sourcemeta.com) to serialise plain JSON that was proven to be more space-efficient than every tested alternative (including Protocol Buffers, Apache Avro, ASN.1, etc) in every tested case (https://arxiv.org/abs/2211.12799).
If it's interesting, I'd love to connect and discuss (jv@jviotti.com) how at least the open-source offering could help.
It surprised me how popular this message got. I love nerding out about binary serialization and space-efficiency and great to see I'm not the only one :)
If you want to get deeper, I published two (publicly available) deep papers studying the current state of JSON-compatible binary serialization that you might enjoy. They study in a lot of detail technologies like Protocol Buffers, CBOR, MessagePack, and others that were mentioned in the thread:
> JSON BinPack is space-efficient, but what about runtime-efficiency?
> When transmitting data over the Internet, time is the bottleneck, making computation essentially free in comparison.
i thought this was an odd sales pitch from the jsonbinpack site, given that a central use-case is IoT, which frequently runs on batteries or power-constrained environments where there's no such thing as "essentially free"
Fair point! "Embedded" and "IoT" are overloaded terms. For example, you find "IoT" devices all the way from extremely low powered micro-controllers to Linux-based ones with plenty of power and they are all considered "embedded". I'll take notes to improve the wording.
That said, the production-ready implementation of JSON BinPack is designed to run on low powered devices and still provide those same benefits.
A lot of the current work is happening at https://github.com/sourcemeta/jsontoolkit, a dependency of JSON BinPack that implements a state-of-the-art JSON Schema compiler (I'm a TSC member of JSON Schema btw) to do fast and efficient schema evaluation within JSON BinPack on low powered devices compared to the current prototype (which requires schema evaluation for resolving logical schema operators). Just an example of the complex runtime-efficiency tracks we are pursuing.
I would imagine that CPUs are much more efficient than a satellite transmitter, probably? I guess you'd have to balance the additional computational energy required vs. the savings in energy from less transmitting.
Yeah, it all depends very much, given how huge the "embedded/IoT" spectrum is. Each use case has its own unique constraints, which makes it very hard to give general advice.
For sure, but radio transmitter time is almost always much more expensive than CPU time! It’s 4mA-20mA vs 180mA on an esp32; having the radio on is a 160mA load! As long as every seven milliseconds compressing saves a millisecond of transmission, your compression algorithm comes out ahead.
Sounds like you are pretty familiar with satellite transmission at the hardware level. If so, I would love to chat to get your brains on it. I don't know much of the hardware constraints myself.
This looks promising! One of the important aspects of protocol buffers, avro etc is how they deal with evolving schemas and backwards/forward compatibility. I don't see anything in the docs addressing that. Is it possible for old services to handle new payloads / new services to handle old payloads or do senders and receivers need to be rewritten each time the schema changes?
Good question! Compared to Protocol Buffers and Apache Avro, that each have their own specialised schema languages created by them, for them, JSON BinPack taps into the popular and industry-standard JSON Schema language.
That means that you can use any tooling/approach from the wide JSON Schema ecosystem to manage schema evolution. A popular one from the decentralised systems world is Cambria (https://www.inkandswitch.com/cambria/).
That said, I do recognise that schema evolution tech in the JSON Schema world is not as great as it should be. I'm a TSC member of JSON Schema and a few of us are definitely thinking hard on this problem too and trying to make it even better that the competition.
A lot of people already think about this problem with respect to API compatibility for REST services using the OpenAPI spec for example. It's possible to have a JSON Schema which is backwards compatible with previous versions. I'm not sure how backwards-compatible the resulting JSON BinPack schemas are however.
Great seeing you over here Michael :) For other people reading this thread, Michael and I are collaborating on a paper covering the schema compiler I've been working on for JSON BinPack. Funny coincidence!
Asking because we use msgpack in production at work and it can sometimes be a bit slower to encode/decode than is ideal when dealing with real-time data.
The TLDR is that is that if you use JSON BinPack on schema-less mode, its still more space-efficient than MessagePack but not by a huge margin (depends on the type of data of course). But if you start passing a JSON Schema along with your data, the results become way smaller.
Please reach out to jv@jviotti.com. I would love to discuss your use case more.
Why this over a compact, data-specific format? JSON feels like an unnecessary limitation for this company's use case. I am having a hard time believing it is more space-efficient than a purpose-built format.
Compared to other serialisation formats, JSON BinPack analyses your data and derives custom encoding rules that are specific to the data at hand given all the context it had on it. That's why on JSON BinPack, the static analysis part is the most complex one by far, and why I'm building so much JSON Schema advanced tooling in https://github.com/sourcemeta/jsontoolkit (i.e. check the huge API surface for JSON Schema in the docs: https://jsontoolkit.sourcemeta.com/group__jsonschema.html)
Of course there is still a lot to do, but the idea being that what you get with JSON BinPack is extremely close to what you would have done for manually encoding your data, except that you don't have to worry about encoding things yourself :) Thus you get the best of both worlds: the nicety of JSON and the space-efficiency of manual encoding.
That's the hardware. I meant on the software side through the transceiver. If you transfer less bits through the satellite transceiver, I believe you can probably reduce costs.
CBOR is a schema-less binary format. JSON BinPack supports both schema-less (like CBOR) and schema-driven (with JSON Schema) modes. Even on the schema-less mode, JSON BinPack is more space-efficient than CBOR. See https://benchmark.sourcemeta.com for a live benchmark and https://arxiv.org/abs/2211.12799 for a more detailed academic benchmark
Thanks for linking the benchmarks. I appreciate the work on shaving additional bytes especially in cases where every byte matters. Real savings seem to be in the schema-driven mode. Comparing a "realistic", schemaless payload for a general storage use-case (eg. the config examples), it looks pretty even with CBOR. E: my bad, BinPack is getting more efficient with larger payloads https://benchmark.sourcemeta.com/#jsonresume
Exactly! The real hardcore savings will always be when you pass a schema, as JSON BinPack uses that to derive smarter encoding rules.
However, schema-less is very useful too. The idea with supporting both is that clients can start with schema-less, without bothering with schemas, and already get some space-efficiency. But once they are using JSON BinPack, they can start incrementally adding schema information on the messages they care about, etc to start squeezing out more performance.
Compare that with i.e. Protocol Buffers, which pretty much forces you to use schemas from the beginning and it can be a bit of a barrier for some projects, mainly at the beginning.
As a note, while cbor is schemaless, there do exist tools to make it work with schemas. In rust cborium will generate rust types from a json schema that serde can use.
I never used cborium, but if I'm understanding it correctly, I think it adds types at the language deserialisation stage and not over the wire. Which means that it makes it a lot more ergonomic to use within Rust, but doesn't use typings for space-efficiency over the wire.
> As a TSC member of JSON Schema, the best DX thing you can push through would be the xsi:schemaLocation replacement because right now mapping files to their schema is some damn tomfoolery
I'm not an XML Schema expert, and I might be missing something, but I believe what you are looking for is the "$schema" keyword (https://www.learnjsonschema.com/2020-12/core/schema/). You set it to a URI reference that can either point to a remote resource (i.e. over HTTP) or a local one using the "file://" scheme.
reply