That's the only image with a non-zero EXIF orientation. Which probably means you're using an older browser (e.g. Chrome started respecting EXIF orientation in version 81+, which I think came out 3 years ago?). You'd have to update your browser for it to display correctly.
Hello! I made this. People are talking about not wanting pictures to be initially blurry before they finish loading. I understand that too, and I'm not sure how I feel about it myself (I could go either way).
But for what it's worth, I actually made this for another use case: I have a grid of images that I want to be able to zoom really far out. It'd be nice to show something better than the average color when you do this, but it would be too expensive to fetch a lot of really small images all at once. ThumbHash is my way of showing something more accurate than a solid color but without the performance cost of fetching an image. In this scenario you'd only ever see the ThumbHash. You would have to zoom back in to see the full image.
I have a bigger budget and would love higher quality, would it be possible easily to adapt the code to ouput 50-100 bytes strings in a similar fashion (2x-4x), or it'd be a complete rewrite? I read the JS code but unfortunately I'm really unfamiliar with low level byte manipulation and could not make heads or tails out of it.
Given the difference in quality seen, I'd guess Thumbhash at 50 bytes would be similar quality to webp at 100-150 bytes, so basically 1/3rd of the size or 3x the quality for "the same price". The examples of webp are way worse quality, and almost 2x the size.
That's a whole field of study on its own, called perceptual hashing. I surveyed these a while for amusement and the TL;DR is that all immediately obvious approaches tend to have particularly bad corner cases.
I just wrote a quick function to compare visual similarity using the thumbhash and just adding up the difference at each byte position seems to work really well! (As long as the images are the same aspect ratio. I want to do more tests.)
I was thinking about that too. Can't answer the question, but I did come across this just the other day: https://github.com/cloudinary/ssimulacra2 Supposedly good for comparing image similarity. Might depend on your use-case, I think it's geared towards image quality moreso than similar photos.
No kidding. Just to consider a single project, my total time saved using esbuild instead of pure JS bundlers could probably be measured wallclock months, and I really like the way it's written: simple, approachable, handwritten lexer/parser, etc. The way that esbuild organizes its lexer state has infected my handrolled lexers, too.
There are two performance implications of "modularization": initialization-time and run-time.
You are correct that initializing many modules is usually slower than initializing one module [1]. However, bundling puts all modules into one file, so this PR doesn't actually change anything here. Both before and after this PR, the TypeScript compiler will be published as a single file.
At run-time, switching to ES modules from another JavaScript module system can be a significant performance improvement because it removes the overhead of communicating between them. Other module systems (e.g. TypeScript namespaces, CommonJS modules) use dynamic property accesses to reference identifiers in other modules while ES modules use static binding to reference the identifiers in other modules directly. Dynamic property access can be a big performance penalty in a large code base. Here's an example of the performance improvement that switching to ES modules alone can bring: https://github.com/microsoft/TypeScript/issues/39247.
[1] This is almost always true. A random exception to this is that some buggy compilers have O(n^2) behavior with respect to the number of certain kinds of symbols in a scope, so having too many of those symbols in a single scope can get really slow (and thus splitting your code into separate modules may actually improve initialization time). This issue is most severe in old versions of JavaScriptCore: https://github.com/evanw/esbuild/issues/478. When bundling, esbuild deliberately modifies the code to avoid the JavaScript features that cause this behavior.
I came across one of these once without knowing about the phenomenon. It’s in Golden Gate Park near the intersection of JFK Drive and Transverse Drive. Water in the stream next to the road appears to flow uphill. Very surreal.
I'm the author of esbuild. I think this is a very smart, practical choice and is one of the approaches that has the best chance of succeeding (the other being a semi-automated porting tool instead of a purely manual port). I'm very excited to see someone take this approach. This is how I would do it if I were to do it. Web developers around the world are hoping this works out :)
The hard part about this project isn't doing the port but keeping up with the constant changes from the TypeScript team, especially if the TypeScript team isn't invested in helping you out. Staying as close as possible to the source language like this will be a big win there.
One of the big benefits of a native language like Go aside from lack of a JIT is the ability to easily add shared-memory parallelism. I haven't studied the TypeScript compiler internals so I'm not sure how easy it is to parallelize their type checker. However, I assume that it would be relatively straightforward to parallelize the parsing of JavaScript files, so I'd hope for at least be a big speedup there.
Unfortunately TypeScript is doing all of type-checking synchronously mostly due how it needs to build a global symbols list first. There are some hints that they might move towards more parallel processing but it requires a lot of major changes
Yeah I was afraid of that. Ideally it would be possible to do type checking in parallel at the function level or something, but TypeScript allows you to omit the return type and then infers the return type from the function body which adds additional dependencies and makes that more complicated to parallelize. I wonder if it would still be possible though. Maybe just for functions where the return type and all argument types are present? Then the approach would be sort of outside-in where you type check the unparallelizable global stuff serially first and then type check the remaining isolated function body stuff in parallel afterward.
The problem is that you can't use shared-memory parallelism in TypeScript. If you could, then the most straightforward way of speeding up the TypeScript compiler would be to parallelize the compiler, not to port it.
I just don’t see how Go is more uniquely suited to do this than any other language with the same feature and admittedly I may have read too much into your comment.
That comment wasn't about Go exclusively. My point was that porting single-threaded code to a native language doesn't necessarily cause huge order-of-magnitude speed gains, especially if you plan on sticking close to the original structure of the code to make porting future changes easy, but that in this specific case a port sounds likely to realize significant gains (which is very exciting!).
Shared-memory parallelism is necessary but not sufficient. Ideally you'd be able to only make some small changes in a few spots to greatly increase parallelism. I'm hopeful that TypeScript's AST parsing is a good candidate since ASTs can typically be constructed independently from the rest of the program state. Lowering and printing ASTs is also likely parallelizable. It would also be great if it was also possible to parallelize parts of the type checker somehow, but that sounds harder. I had some ideas about that in another comment. Anyway I think this means that it's likely that a port of the TypeScript compiler could realistically lead to significant speedups and may even lead to extreme speedups.
As far as language choice, it sounds like due to maintenance reasons the decision space is narrowed to native, garbage collected languages similar in object model to TypeScript but with parallelism primitives. Some candidates could be at least Go, JVM-based languages, or .NET-based languages. I think those can all be ahead-of-time compiled to a static binary? Another consideration is how easy/compact/fast/cross-platform binaries are. Go makes this trivial but I haven't used JVM or .NET AOT builds myself so I don't know how they compare in e.g. executable size or startup overhead. One advantage of picking Go is that esbuild has already demonstrated that Go is an effective choice for this problem domain, which may mean less work/research/unknowns. A disadvantage of Go here might be that the object model is different than TypeScript vs. something like Java or C#. But all else being equal the choice comes down to personal preference.
Yes that allows for shared memory. But the TypeScript compiler is written in TypeScript, and you can't put JavaScript objects (such as the TypeScript AST) in a SharedArrayBuffer. So you'd have to port the TypeScript compiler to another language that can target multithreaded WASM to do this.
You could also try sending JS objects between workers instead of using SharedArrayBuffer. But that copies the memory, which can take a significant amount of time, and of course also excludes certain kinds of advanced parallelism that require shared memory (instead of copied memory). Another drawback of using the JS worker model like this is that having multiple JS VMs probably means fewer CPU cores available due to there being multiple independent garbage collectors. I observed an effect like this when I tried to parallelize esbuild's JavaScript plugins. Running them in separate workers seemed to cause the gains from parallelism to stop when there were half as many workers as CPUs. I'm guessing is because half of the CPUs are busy collecting garbage for the other half. Using a language like Go with real shared memory presumably makes it easier for fewer resources to be spent on garbage collection.
Folks at Parcel wrote serializers that uses SharedArrayBuffer for their core stuff like graph[1]. I agree with parent comment that this didn't have to happen in Go, JS would've been fine if they were willing to depart from the source enough.
I'm the author of esbuild. I hadn't written Go before I started esbuild and I thought I would miss generics more, but I didn't. There is literally only one situation in the whole esbuild code base where I've found myself wishing for generics: sorting an array. The workaround is easy and involves implementing an interface with three methods, each of which is typically a single line: https://pkg.go.dev/sort#Interface. And I only needed a few of these interfaces for a few different types. That wasn't nearly enough of a reason to not use Go, especially considering the other advantages.
I agree. I haven't missed generics at all since switching to Go. However that being said, do you think that the upcoming Go 1.18 with generic support is going to increase performance in Esbuild? Just from not having to Unbox every interface{}, or by writing less to the heap? Have you experimented with generics and esbuild?
No, I haven't experimented with generics and esbuild. I hadn't considered whether generics could improve performance or not. Just thinking about it quickly now. I'm not convinced it would because esbuild hardly makes use of interface{}. If someone can demonstrate a noticeable performance improvement then I'd be happy to start using generics for that reason.
The main pattern esbuild uses is an interface with a single dummy method to denote a union type like this: https://github.com/evanw/esbuild/blob/34899aaa1d76acd3b4adc5.... It's used several times and is basically esbuild's core data structure. I'd like to be able to optimize this pattern. Specifically I'd like to avoid each reference to a union type taking up two whole pointers in memory (Go represents interfaces as a pointer to the object and a separate pointer to the method table).
I'm only using the method table as a tag for the tagged union, so ideally it'd be a single byte or something even less expensive like part of the pointer. I don't think generics can help with this? But Go doesn't let you do fancy stuff like this so I'm just leaving it be. A low-level language like C++/Rust could do better here, but that comes at the cost of a significant decrease in productivity, so I'm ok with this trade-off.
It is also all custom: https://github.com/esbuild/esbuild.github.io. YAML files that get converted to plain static HTML pages. It's designed to work well without any JavaScript. Even the animated graphs on the home page are done without JavaScript.
I'm the author of esbuild. Rest assured that I am very interested in supporting it and am actively working on it. It's just unfortunately very complicated to correctly, especially without affecting any of the other existing features. There are some recent updates at the bottom of this thread: https://github.com/evanw/esbuild/issues/253.
It would be great and in the one case I will when I have space to come back to it, since I could isolate it. In the other case idk what to file other than “open source project I contribute to has a file with a question mark that your compiler doesn’t like”.
Both would be good to file. I’m happy to investigate myself if you don’t have the time.
One thing to be aware of is that some other tools that integrate esbuild do so incorrectly. For example, the 3rd-party integration of esbuild into Webpack mis-configures esbuild in a way that causes issues with JSX: https://github.com/privatenumber/esbuild-loader/pull/107. This isn’t a problem with esbuild itself.
