This solution pulls in 155 packages from npm, including multiple versions of bullshit packages like "kind-of" that it can't deduplicate. One of the direct dependencies is a library that renders a loading spinner in command line interfaces, which itself pulls in over 20 transitive dependencies.
The reason that JavaScript build tooling is so complex and introduces so much overhead is that the ecosystem's culture is fundamentally broken. Every single one of these tools is a raging dumpster fire because it's built on top of all of these layers of low-quality interdependent crap.
I'm increasingly moving towards module-type script tags and standard ES modules everywhere, with no build step during development. It's still challenging to use node-targeted modules in this fashion, so I really do appreciate that people are working on finding ways to make it work. I just wish that we could do it without pulling in so much third-party code and the large surface area for failure and security problems that go with it.
> One of the direct dependencies is a library that renders a loading spinner in command line interfaces, which itself pulls in over 20 transitive dependencies.
Just trying to understand, is this a bad thing?
Someone else made an open source CLI spinner library which also uses other people's existing open source libraries. This saves a lot of time and gives developers many good options.
Should Pika write and maintain its own custom CLI spinner animations? Are you saying the CLI spinners should be standardized in the next version of ECMAScript itself?
How is this worse than the same thing written in Python, for example? (I mainly use javascript, so maybe I haven't been exposed to the kinds of alternatives you're thinking about.)
It’s bad because this leads to a standard React project having over 2,000 dependencies.
The real question is: do you -really- need an external lib with 20 dependencies just to show a freakin’ loading spinner? Remaking the wheel is bad but so is never making truly simple things yourself, or just not using them.
What happens when a common package breaks? What happens if it gets hijacked and becomes a security vector that’s impossible to spot because it’s loaded as the 567th package in a dependency tree?
The answer here is to have a strong stdlib where do you don’t need to pull in 3rd party packages all the time for trivial things, and not including a million small packages in every single project.
> The real question is: do you -really- need an external lib with 20 dependencies just to show a freakin’ loading spinner? Remaking the wheel is bad but so is never making truly simple things yourself, or just not using them.
So the problem is the sheer number of dependencies? What is a reasonable upper limit?
Yes, javascript should continue to standardize commonly used features, but avoiding dependencies doesn't seem to be a solution.
If anything, more dependencies are a good sign because they imply that other people have spent more time and effort on a solution than anything you'll be able to hand-roll for single-use.
It sounds like the root issue here is just dependency management. If our package managers were solving this issue well enough, there should be no practical difference between 2 big dependencies with significant functionality (and more code to review) or 20 tiny, easy-to-review dependencies.
From a security perspective, minimising dependencies is preferred. I have to review at least monthly all our dependencies for published vulnerabilities and new versions.
We don’t allow automatic upgrading of packages/dependencies due to the risk of malicious code making it in (see https://www.npmjs.com/advisories for examples). Yeah there are companies that will help manage your vulnerability process but it’s still a lot of overhead and only grows as the number of dependencies grows.
There’s also the whole left-pad mess from a few years back which shows you always need local archived copies of any dependencies you use.
If you care about security, you should evaluate your dependencies.
This preferably means reading the code you're pulling in but it's unrealistic that we're going to read 2000 constantly changing dependencies for every deploy, so you need to establish trust some how.
Reputation of maintainers, dep CVE scanning, SAST and protective monitoring can all add additional assurance, but they won't protect you from a random hijacked npm module, and the more you include and from more people, the more likely you'll be affected by a zero day.
Having an entire community depend on tiny libraries that do nearly nothing exaperates this problem, and if you use something that almost nobody else does, you're unlikely to be saved by npm audit.
I don't use node daily driver, but I assume npm audit growing to include reputations and frameworks owning more of the dependency tree will help, but the users of the system also need to be considerate of the risks they take and the trust they place.
I used to have all npm modules in source control (SCM), until npm introduced tree shaking. I'm using ZFS which have both de-duplication and compression, so I gain very little by tree shaking. I wish there was a way to disable tree shaking in npm, it's really the source of all evil.
Anyway, I reviewed all code diffs after each npm update, very little changed, eg. it wasn't that much work. But it's now impossible as npm moves files around.
I also deleted a lot of unnecessary files. About 90% of the content of each NPM packages are not needed.
Another reason why I stopped hosting dependencies in SCM is native modules. I wish Node.JS itself could become a platform layer so that I wouln't have to use these native modules.
Another thing is compile to JS, where a tiny change in the source might cause a huge diff in the generated JS.
I committed my dependencies until recently too, and I've been trying to figure out a better alternative. Are you doing something else now rather than committing dependencies? (Right now I'm just using a hodgepodge of npm scripts to lock things down.)
There is an enormous difference between 2 large packages and 20 small ones. I can see the main authors more clearly, and not have to worry about 20 different packages being compromised.
Your statement makes no sense. More packages does not, in any way, shape, or form, correlate to “better quality”. The sole thing it shows is that the author pulled in more packages. That’s it. Whether it’s good or bad or secure or not secure is only determined by analysis. The author could easily have been super lazy and pulled in 18 small packages instead of writing small helper functions manually, etc.
There is no defensible argument for more packages and bloat. None. As I said above, the core lib should focus on providing functionality so pulling in a fancy spinner library means you pull in ONE library, not 20.
It’s fine from a development perspective where the job is to make something work.
It’s horrifying from an operations perspective where the job is to make sure everything works.
Developers can afford to ignore looking into dependencies, operations need to make sure every dependency is functional and safe.
If you write a piece of C# using the standard .Net library you can be fairly sure it’s safe and sound. If you write something using 2000 JS packages, you have to read through every one of them to be sure.
The way it seems to work with JS is that the community develops a bunch of stuff and the best rises to the top. This then influences the official language spec. ES6 basically made a lot of the popular tools redundant because it was heavily inspired by them.
This is so incredibly important because the main development of the language is so poorly handled.
And despite adding arrow functions, async/await, modules etc. we still don't have sprintf and a Math.round that can specify the number of places to round to.
I think that modern javascript is pretty good and I'm grateful for the syntax extensions that have effected that, but the standard library is crying out for some love.
Agree. JS standard library has been very immature and was so slow to improve as it was primarily targeted for browser and people thought that wasn't the place to do complicated stuff, not even md5.
I still have some array helper methods just to shuffle array or randomly pick one.
We had several other languages once upon a time, but all were proprietary and most only worked through plugins. All were eventally removed from default installs in browsers due to awful security and mobile performance.
I was a Flash dev for some 10 years and I can count with one hand the number of third party libraries I regularly used.
I don't disagree about the security issues, mobile performance, and energy consumption though, but as a dev I preferred the experience of writing AS3 apps by a long shot.
Library rot is a well known thing in the python/ruby/java/php std libs, although I always hear good things about Go's std lib, they had the advantage of a fresh start and have the resources to keep the std lib well maintained.
I like the philosophy of no required build step. Once you make that decision, it's like a guardrail keeping you away from the tooling circus.
Do you know of any listings of third-party ES Modules that are designed to be used directly? i.e. their full URLs being present in import statements that the browser sees at runtime. Even toy stuff would be interesting to look at.
At this point, ES6 support is sufficiently widespread that I don't really have a problem using it natively. Major features like the const, classes, template literals, and async functions are all at least at 90% market penetration.
I can get away with not supporting Internet Explorer, though I understand that it's not a trade-off that everybody is comfortable making.
I think (thought) also like you, just use the latest es6 code to do everything. The workflow is very fast, I can write my app 20%-50% faster than es5. I develop mostly on Chrome. My application/website works.
Then we test the website on different browsers (we all work with OSX), FireFox works (of course), Opera works (is Chromium), Safari, naah, it's the new IE. We do some minor fixes for Safari.
When we send it to the client, nothing works, of course the client uses IE so we need to rewrite a lot of stuff to make it work for "most" people. (I live in Belgium, there are "a lot of" IE users still. And even if IE is not widely used, our clients always use IE :( )
But wait, why rewrite our application, this can be fully automated! We run our code through babel/webpack. It automagically works everywhere!
Now I understand there is some performance penalty by using transpilers, but using them leaves our code mostly bugfree on all browsers and the client is happy.
The big problem with those fancy new features is browsersupport. If every browser followed specs correctly, we wouldn't need these tools. They make the job of the developer easy, the bosses wallet full and the client happy.
I'm not sure why this would be desirable. Sure, bundlers are slow, but that's what projects like pax [1] are for. Bundlers create one single file instead of the browser having to download your entire possibly huge dependency tree. Sure, with HTTP/2.0 and the upcoming QUIC or HTTP/3.0 overhead of this is minimized, but it's still not zero. There is also a cognitive overhead of having to configure the server to push all the deps to clients otherwise you have constant back & forth where the client is fetching a file, then the server is answering, then the client is finding out it needs those other files, etc etc. Unless you are using global CDNs, expect latencies to be > 50ms for each such step, if not hundreds of ms, and this times the depth of your dependency graph. Bundlers also do dead code analysis.
There are really two solutions to this: either avoid using js dependencies, at which point you won't need npm or pika or whatever, or use bundlers.
Under ESM, each of your dependencies could go back to being a (CDN or /vendor-directory) hosted "release." The release would have been packed together into a single JS file + a single CSS file by its developer, when it was published. There would be no reason to further run a bundler just to turn ~5 such pairs of files into one pair. Over HTTP2/3 that difference is negligible.
It's non-negligible for "thousands of dependencies", of course, but nobody would be doing that. "Thousands of dependencies" is for library development, not for library release packaging.
NPM (or Pika) is still useful under such a paradigm, for the same reason Cargo or Bundler or Pip or Mix or whatever other tools are useful: these tools let you specify your direct dependencies using symbolic version constraints, and then retrieve+update+lock those dependencies. This is helpful whether your direct dependencies are "baked-down" release packages or raw deep-dep-tree source.
That's worse because you lose the ability to "tree-shake".
It would go back to importing all of Underscore on a CDN versus bundling the few functions you need into the main bundle.
That never really works out in practice when there's so many versions and then you're at the disposal of a public CDN, which can be the bottleneck, especially when it's critical code.
Sure you can fallback to local copies, but that just means your app hung there for X amount of time, and the added complexity of adding fallback logic.
Plus if you look at the size of these libraries, especially utility libraries (date-fns, underscore, etc) they are huge, and you usually only need a few functions. Relying on cache will never be a bigger benefit than tree-shaking.
Sure you could manually try to extract those functions, but that's a much much worse developer experience.
So glad this was such dominant practice and advice for the last 10 years. I wonder for how many of those influential practictioners have known it's more of a lazy include than clever cache reuse. Cargo culting writ larger than most.
It really depends. I think CDNs are still great if you are making smaller content website with few libs for things like lightboxes, sliders etc. It works out better than bundling. There is much less complexity without build process. Many websites don't need too much JS and devs tend to overengineer them nowdays.
If you are talking about products with lot of dependencies then of course but don't forget that bundler situation used to be completely different. There is not that much of difference (besides number of requests) between concating libs in sequence and loading libs in sequence.
I'm speaking to the jquery example. 99% of howtos have instructed people to pull from CDN because everybody (probably) already has it. It has been conventional wisdom for a long, long time.
I just checked, and googling "jquery cdn faster" brings up plenty of examples.
One major benefit is that for any given page in your application, you probably don't need the entire bundle.
This is not really true for SPA's, but I hope that's a trend that will eventually sizzle out again except for specialized cases.
Just grabbing what you need can have significant benefits. I agree with the latency problem though, it would be nice to have a HTTP/2 or HTTP/3 server that has a basic understanding of the JS dependency tree and preemptively push the entire tree.
1. Typically, you only ever care about minimizing bundle size in cases where latency is consistently questionable
2. In those cases, whatever UI you're supporting is likely then not meant for latency-sensitive operations, and thus the overall workflow tends to have low state complexity
3. Just how much actual value is added in those scenarios by having the ESM workflow specifically in conjunction with importable NPM modules?
I'm sure you can think up some contrived use cases but I'm just not seeing any particularly compelling ones.
No it's not just latency. In fact, that's not even the most important factor. On mobile devices if the script is too large, it will freeze while parsing.
> This is not really true for SPA's, but I hope that's a trend that will eventually sizzle out again except for specialized cases.
I suspect we'll see the opposite, with people building desktop type applications and deploying through the browser (especially with the growth of wasm)
"Today, it’s nearly impossible to build for the web without using a bundler like Webpack."
Is it? I've gotten by just fine, and webpack is a pretty all or nothing approach. Did everyone just forget how things were done before webpack and NPM overnight? What memo did I miss?
I think we're in a kind of awkward place right now, where browser support for ES6 is inconsistent, and TypeScript has stepped up to fill the "type gap" left by JavaScript in general.
Sure, if you only use ES5 then you don't need Babel or Webpack - but devs are tired of this shit and really want to use ES6 or Typescript.
So, "nearly impossible" is an exaggeration, but "would rather gouge my own eyes out" is perhaps not.
Tired of what shit exactly, though? typing the word "function"?, hacky workarounds to traditional "classy" style programming? Having to learn what deprecated features to avoid? Is it asynchronous programming? ES6 is nice, but the only real pain point I think it solves is standardising on a module system, which as this article proves, hasn't truly been solved yet, because now we're in the situation where we still have the legacy of competing mutually incompatible module systems hanging around.
Everything else is programmers coming from other languages and believing that actually learning javascript is beneath their dignity, and persistently insisting javascript be more like whatever their fave is.
I will say this though about build systems/bundling systems in general: Combined with linting and type checking, it can stop you from accidntally shipping a large class of bugs/mistakes. An out of place semicolon breaking a build is a better outcome than breaking a production website.
If you're actually looking for an answer, then you hit the nail on the head for me personally. For someone who spends 90% of their day working with Firebase/Cloud funcs, I really don't want to go back to promise handling. Call me lazy if it fits your narrative - but you can pry my async ES6 from my cold dead hands.
Yes and no. For many use cases, this might be justified.
* You are hosting an internal enterprise application and you are forcing all of your employee desktops to use Chrome any case.
* You are not serving a web site, but a business web app, which only works on a modern browser - because it is a business oriented app people are willing to switch the browser to use it. E.g. 3d modeler.
But for srs, couldn't I achieve more or less the same effect by not putting node_modules in my .gitignore and then making aforementioned modules accessible via static content dir pathing?
Yeah that’s the general idea - except that in this case, re-use of the same public URLs means browsers can cache the dependencies. (So we finally won’t have to download 400kb of the same React library code on site we visit!)
Yeah that was always a "benefit" of using a CDN for jQuery except it never really works out in practice when there's so many versions and then you're at the disposal of a public CDN, which can be the bottleneck, especially when it's critical code.
Sure you can fallback to local copies, but that just means your app hung there for X amount of time, and the added complexity of adding fallback logic.
srs -> SSR (server-side rendering), just a typo, right?
Anyway, committing -- and directly serving -- all of a given project's transitive dependencies sounds like a tough sell; without decent tooling to dedup and tree-shake, it might go from inefficient to implausible.
What does the SSR story look like with pika/web? If I have code that has something like `import x from 'lib'`, can I make `lib` point to node-specific code on the server, and browser-specific code on the client?
Similar to what would happen today when a package exports both ESM (web) & CJS (node): frontend gets ESM via @pika/web, while node would use CJS via require. A Babel plugin comes bundled with @pika/web which can help you resolve imports correctly in the web build.
Although at that point, where your app needs to reliably run in both Node & the browser, there’s a good case to be made for bundling. All that complexity is giving you something valuable, which is the point that the article tries to make:
in 2019, use bundling because you want to and not because you have to.
I found his opening analogy to be broken. In 1941 there were no clips and other things on the TV news because the tech to record and replay television signals hadn't been created. Until the 60's the only way to record TV was "telecine" which was, as the name implies to record the tv image on film. (The term also implied the reverse process of (re)broadcasting the filmed source material of course.)
It wasn't that no one had thought of showing the dumpster fire (to use a quote from a earlier comment!) but simply that the tech wasn't there. A lot of the problems here are of the same nature. The right level of tech is yet to built and hopefully when it is it may also improve the nature of the media that gets bundled (much as telecine was pretty poor looking when compared to the video tape recording that supplanted it later). So trying new tools is good. But I haven't seen that paradigm altering shift yet.
I get the idea. But the title of this article is "how" not "why" yet the article doesn't ever really get to the how. I'm convinced of the why. I'm still excited to try it.
I read through the linked page and it's not totally clear to me why you can't just import directly from node_modules. What Pika/web appears to do is find node_modules that use valid export syntax and copy them to a different dir. Is that correct? Am I missing something?
Because many modules don't use ES exports, especially the tiny annoying ones from people like sindresorhus that everything seems to rely on. ES modules are compatible with the browser, CommonJS modules are not.
So I basically don't exclude node_modules, and set up a simple webpack config to suck in my deps and output them as ESM modules to a given static dir of my choice
The reason that JavaScript build tooling is so complex and introduces so much overhead is that the ecosystem's culture is fundamentally broken. Every single one of these tools is a raging dumpster fire because it's built on top of all of these layers of low-quality interdependent crap.
I'm increasingly moving towards module-type script tags and standard ES modules everywhere, with no build step during development. It's still challenging to use node-targeted modules in this fashion, so I really do appreciate that people are working on finding ways to make it work. I just wish that we could do it without pulling in so much third-party code and the large surface area for failure and security problems that go with it.