This isn’t “better” than your tool of choice. This is a tool for JavaScript developers to write non-trivial scripts in a portable way without getting lost in POSIX hell. Who came up with this stuff?
That used to be a shell script. But this I can way more easily read (even after not touching it for a long time), I get autocompletion, and I can use the APIs I'm already familiar with. It's a pretty neat QoL improvement for me specifically.
Do you mean the `await fs.move` on line 13? Because yes, I do need that to complete before I can run the next command, the build, on line 40.
I could have appended it to the Promises that I'm creating on line 9 to run it in parallel with those moves, but this is not performance-critical code, and moving the two conceptually-unrelated concepts into the same expression wouldn't aid legibility, in my opinion. And since I'm the only one who needs to read the code, that opinion wins by default :)
Above: You have `await Promise.all()`. Remove the `await` in front of `fs.move()`.
You don't need the second `await` for blocking purposes.
Sure, it is your code to do as you like, but if you're going to post it as an example, you might want to correct it first so that others don't make the same mistakes.
That is absolutely correct! I wasn't intending it as an example of how to write flawless JS code though, just as an example of the type of script that ZX is useful for - but if I remember next time I open my dev env, I'll update it :)
Edit: though maybe I'm misunderstanding given that you said it would make them run sync, because as far as I know, the functions would still run in parallel? Essentially, I believe it would roughly be the equivalent of this:
That allows other code to run while e.g. files are being moved. It indeed can be somewhat painful ([1] is the classic criticism) and in shell scripts is not as beneficial. I think there are even sync alternatives (i.e. equivalent functions that don't need `await`) for most of them, but again: these are the APIs I'm already familiar with, which is the entire point - and thus the await-stuff comes naturally for me, because I intuitively know those APIs are async.
Sure, but again: I know and recognise this syntax and the APIs. What's boilerplate to me, I hardly see, whereas compact Bash operators are noise to my eyes.
(Though to be honest, I'm also somewhat sceptical that a single & in Bash would be sufficient to move a bunch of files in a particular order, and restore them properly if the process gets terminated somewhere halfway along the process. But what I'm sure about is that I wouldn't have been able to write that in a reasonable amount of time!)
This looks really cool, actually, and I'm surprised I haven't heard of this before.
My incorrect gut intuition is that JavaScript devs usually don't do very much in depth terminal level command work, but that's of course false on a Bayesian level. There are far more JavaScript developers in the world than there are even Python developers, to say nothing of the long tail of even less popular languages. If 30% of JS devs are heavy terminal users, and 80% of (say) Perl devs, that 30% will still swamp the 80% in terms of absolute numbers.
Hell, that's half the point of Node. Can't believe my gut isn't keeping up with my head here!
> There are far more JavaScript developers in the world than there are even Python developers
I don’t think that’s true. Anecdotally when I was interviewing at a recruiting company (and we let candidates pick any language to be assessed in), Python was by far the most popular choice, chosen by about 60% of candidates or something like that. Java and JavaScript came after, followed by a long tail of languages like C# and go.
I assumed JavaScript would be the most popular too - but that didn't seem to be the case. I have no idea where the demand for all this Python programming is coming from, but it seems Python programmers are everywhere. Maybe it’s just popular in schools.
If you need a programming language for you job, but you're not primarily a developer, and two languages would probably be overkill, then your one language is probably python.
I do digital marketing for clients. There’s a large number of activities that can be made more efficient, quicker, cheaper, or better using code.
For example taking a large number of text documents and turning them into video outlines with illustrations using OpenAI APIs and a quick and dirty Django app to organise everything.
Or connecting up various marketing APIs using Python instead of Zapier - a lot of companies spend thousands of dollars on Zapier each month when a script would literally cost a dollar to run.
Python is just fastest to write. No semicolons, fewer parens, no braces, no const / var / let. List / dict / set comprehensions. Lambdas are clunkier, but `def` is shorter than `function`. Etc.
Perl and Ruby are comparably compact and expressive, but relatively few people know Ruby, end even fewer people know Perl, especially among those younger than 50.
BTW Haskell is also syntactically compact and expressive; I once solved a toy interview problem using it. But even fewer people know it at a practical level, among interviewees and interviewers alike.
I’m proficient and confident in many languages because I like solving problems with computers. However in a high-stakes and low-tooling situation like a job interview I wouldn’t choose JavaScript any day of the week because of the uncertainty, inconsistency, and footguns it likes to sprinkle everywhere.
There is absolutely a role for JavaScript, it truly does run everywhere which is no small feat. But that’s about all it’s good at.
What I’m getting at is that when you give people a choice they are going to optimize for the scenario, so you have a heavy selection bias in your anecdote.
Is it a recruiting company, as it's main business is to recruit candidates ? Or was the core business something else and you were recruiting devs for it ?
I think it's always difficult to get the bigger picture of the dev market through candidates. Many (rightfully) handle their career moves by themselves and will directly apply to companies that fit them, heavily biasing the movement observed from the company or third party recruiters.
I could probably say "you're moving the goalposts here, I'm taking all developers not all working developers" or something, but I think these days you're basically correct. Python was relatively more niche when I picked it up in 2009 or so, but it's really become the lingua franca of our industry.
Maybe a lot of things start in Python, because so many people know it, and no one wants to change languages after a project starts?
Python also has a very wide range of applications so while there are many languages more popular in a particular area, Python is extremely widely used in aggregate. By contrast it is unusual to write numerical code in PHP or web apps in C.
Here's an anecdote to help your gut. I'm a platform engineer who's favorite language is Typescript. I LIVE in the terminal. The only reason I use vscode is cause it's easier to put him into vscode than it is to put vscode into vim. Otherwise I'd be in the terminal for everything but browsing.
Likewise! I use TypeScript everywhere possible, and zx is a godsend. Unfortunately, I've recently dealt with gobs of Python for ML software. After years of writing TypeScript, it's been a...sobering reminder of many painful coding issues that I'd forgotten even existed. Made me deeply appreciate what TS gives us haha.
I worked at a Python shop for a bit. I spent more time than I wish to admit trying to make all of my Python behave like TypeScript (adding typings, etc.).
Ditto. Python(3.13) types are so crudimentary that it becomes a Sisyphian struggle. Library type support is patchy at best. **kwargs is an abomination. Python is incredibly slow. Async code and anonymous functions are awkward and verbose. Hell, Python is verbose.[1] I could complain endlessly, but I'm mainly just disappointed in how little has changed since I last professionally wrote Python. The biggest benefits were basically a) familiarity, and b) powerful stats/ML libs.
I hadn't seen analogs of `AttributeError`s and `TypeError`s for years, until revisiting Python and remembering that those are still unsolved issues outside of TypeScript. Ugh.
If they'd go to JavaScript, they'd feel a sense of unfamiliarity similar to what JS developers going to Python would feel.
But the Python equivalent of TypeScript is mypy, and I don't think it's controversial to say (even among Python developers) that it's way behind TypeScript, generally.
My big use case for this is that it's quite good for glue scripting when you're already in the context of a JS shop. My common use cases are tooling setup, CI scripts, and API response sanity checking.
yea same here. I wrote nodejs programs for years. the focus on the eventloop and everything being asynchronous is great for servers and network IO but does NOT make it the perfect choice for scripts. Its a different use case entirely.
scripts are supposed to be small simple programs that run in sequence and terminate. I shouldn't need to deal with concurrency primitives at all in that particular situation.
async/await are dead simple. And many Node libs have synchronous versions. It's pretty straightforward if you are familiar with the language. If you aren't, well that isn't really the fault of JS nor any different from bash or python, which can both be just as painful for those unfamiliar.
> scripts are supposed to be small simple programs that run in sequence and terminate. I shouldn't need to deal with concurrency primitives at all in that particular situation.
I suppose bash pipes are a concurrency primitive, but I don't object to them. I think it's more about appropriate primitives.
> And it's sooooo much faster that python and Ruby.
That advantage becomes vanishingly small when the language task is to glue together a bunch of spawn/fork/exec to external processes.
If the glue code somehow needs actual heavy processing, that processing can be factored out into another program dedicated to that, leaving only glue code. Side effect: the heavy processing code immediately becomes composable!
Especially because these languages are only one package/install away and not two. I don‘t really get for which audience is targeted here. Usage in JS projects maybe, but then why not write it as npm tasks. ..
I‘m playing around with dotnet-scripts [1] at the moment (C# shop mainly) and this has the same issue imho. The reason why I looked into it was because we have developers not accustomed to bash etc. I still find it silly and would rather use ruby so…
Do you mean writing the entire script in a single line in a string in a JSON file (package.json)? Of do you mean that string just calling out to a non-inline script? Because the use case for this is very much that latter script - it will often still be started via `npm run`.
That’s a completely arbitrary limitation which should not exist. When my scripts contain independent non-instant subprocesses, I run them in parallel. I’m also using js/ts to write all my scripts, because why wouldn’t I use a proper programming language instead of that “bash” thing that can’t even handle its single datatype well and resorts to all sorts of gibberish to perform trivial operations on it.
PHP is a surprisingly capable scripting language. Some relatively recent version is installed by default on a lot of Linux distros and its standard library has tons of functions for dealing with strings, files, and the network.
As a dev it lets me get back to a good scripting language, where I process input, run other programs, emit output. Powerful & succinct.
As a tool maker, it helps me show my work. Many of the common tasks like running an script or calling fetch will by default log what's happening in clear format, pass through the activity. It's fantastic to write these devtools that do a bunch of tasks, but are showing their work at every step. If something goes wrong, the person immediately sees the evidence of what was ran & what the output was. It helps close the iteration loop very effectively.
Zx makes such a huge difference, while being little more than a couple small good ideas to help node.js be more shell-scipty. Good stuff.
An immediately visible problem with zx is that wrapping commands in $`` makes it harder to copy and paste them back and forth to the shell to test them. Did that slowed you down sometimes or added an occasional bug?
When my shell scripts get too large, I usually switch to Perl. Still best in class at handling text processing with a few shell commands here and there.
No way. If your script requires something more complex than what is difficult to implement or maintain in bash, then it is no longer just a script. Regardless of how small the resulting application may be, you should treat it as an application and not merely a script. Node.js is not typically used in the system layer where Bash scripts are commonly employed. No single administrator would replace Bash with Node.js. For parallel execution and asynchronous operations, tools like ansible exist, along with python, which administrators are already familiar with.
Yup, it seems the perfect example to wonder if a language with async by default is appropriate for the use cases of bash scripts.
The example runs 7 commands and has to explicitly await for 4 of them. The other 3 can run in parallel and it awaits for the end of all of them, which is fair.
The typical bash script is sequential and I fear that its zx replacement will be an exercise of writing await, await, await.
I'd go with an await by default language and wrap parallel code in whatever that language offers to wait for the end of child processes.
This feels to me like it might be a really useful tool to do systems-integration work around an app that is built in Node.js. For example, to integrate with incrond or cron, or get a supervisor task into JS. The smallest sensible "shell-ish" wrapper that gets you access to your codebase in JS and stops the coding effort being divided so much between technologies.
Sure. But I can easily see this becoming the default way to write shell scripts in the near future in the same way Electron is becoming the default way to write desktop applications.
If I want to write better shell scripts I usually run shellcheck and adjust accordingly or if I need facilities not provided by the shell I switch to a full fledged programming language. And oh yes, `sh` is present almost on every BSD and Linux box for free so I consider it an important thing to at least be comfortable with.
>Write your scripts in a file with an .mjs extension in order to use await at the top level. If you prefer the .js extension, wrap your scripts in something like void async function () {...}().
Shebang scripts shouldn't have a file extension in the first place, so requiring one here is a dealbreaker. And having the file extension be used to switch between two different incompatible source formats is really a terrible design.
Should not have or should not need to have? You can perfectly write a script with a shebang and an extension, stick it in your path, and use it without extension.
Therefore, I fail to see how this is a "dealbreaker". Or is it just because "I don't like it!"?
So how am I supposed to put such a script in $PATH? e.g. if I hypothetically want to make a replacement for say 'ls', what am I supposed to do, force people to type 'ls.mjs' rather than 'ls'?
I've very rarely seen, say, image viewers which can't handle a JPEG with a .png extension or vice versa, but they're badly designed. Most handle this from the file magic automatically. Keying on extension is definitely an antipattern IMO.
I don't see people naming C++ files .c++03, .c++11, .c++17. I don't see people naming Perl files .pl-strict-mode.
Yes, it's normal for people to use file extensions, but having programs impose forced usage and semantics to them is a crutch and a sign of bad design.
How am I supposed to put a script written using this tool in $PATH?
This bad design seems to be a thing in the TS/JS ecosystem now. At least some language ecosystems (Deno? Not sure.) seem to be encouraging people to do 'import ".../Foo.ts"'. Which is abysmal: Firstly these are URLs, yet file extensions are against web principles and good URL design. Secondly they couple a consumer of a module to what language the module was written in and expose this as part of an API, which is terrible design.
> How am I supposed to put a script written using this tool in $PATH?
Separate file that invokes sh or bash and calls the script file with zx and the path to the mjs file.
Should do it.
Though I agree with your objection. The requirement for specific extension while also suggesting a shebang, breaks conventions and expectations. From what I can tell from comments made, it's done that way because they prefer it. Which... is a bit silly. But to each their own.
Extensions are predefined mapping to the executing program. There is no real concept of compatibility there. Useful for media files where you e.g. pick your image-viewer-of-choice, less useful when the script file already has a very strong opinion as to the expected executing binary. Compatibility can be hinted at, but really only determined on the application level.
If the script runtime binary then looks at the filename and does something differently based on it, it's likely not correct, and if it is done, it'd better be because of some necessity. The "we prefer it this way" is not a necessity. Plus it breaks conventions and expectations.
Enforcing file name extensions at execution, because you want to help an editor figure out how to lint or highlight the file, is also, arguably, letting a specific use case affect something it shouldn't.
GPT4 makes it so easy to write or translate scripts, I don't care if you write it in js or bash or powershell or batch files or python or ruby or whatever. Whatever is most convenient at the time is best, and it can be very easily translated if something else is preferred later.
Bun Shell implements a cross-platform bash-like shell, using builtins for popular commands like `rm`, `cd`, `which`, `ls` etc which support the same flags on Linux, macOS, and Windows. A lot of why we made it was for `bun install` to work on Windows without needing polyfill packages like `rimraf` or `cross-env` in package.json scripts.
zx uses the system-provided shell, which means it isn't cross-platform
For the commands Bun Shell implements as builtins, they typically run something like 10x - 20x faster than in zx on mac/linux because spawning processes is expensive (even more expensive on Windows)
heh.. this is the engineering culture at Google. You identify a problem that bothers you but perks your interest to solve, and then dedicate 1 day a week to solving it, and then releasing it as an internal library. then get approval to open source it and do so.
I get the utility of it, and probably having everything in the repo of a project in the same programming language is a benefit, when onboarding new people.
but...
Shell scripting has a long and glorious history, filled with incremental
improvement in the tooling that have proven themselves over decades.
The shell scripting languages themselves and all the awesome tools that
come along with them on UNIX.
You are 100% guaranteed to be reinventing the wheel over and over (and poorly) if you write shell scripts without a good understanding of what is available
"Off the shelf"
There are great benefits to had by using am appropriate language for
the task at hand.
* I love the idea, and I'm interested to see more progress in this direction.
* I'm not a huge fan of the syntax - for me, the commands are the star of the show, while the async/await is a glue. This syntax emphasizes the async/await while the important stuff get shuffled into quotes.
* Don't make me install node - and especially not system/user-wide packages - to run a script.
Having said all that, this is a neat idea and a cool hack on top of javascript language features.
If you rightfully find bash inadequate at some point, just switch to the next most natural tool - Python. Nobody's going to be surprised when they see a Python script.
If you start feeling Python is inadequate too, then:
1) You're wrong, just stick to Python
2) If you're so bent on doing away with familiarity, at least get as much leverage as you can in return and use a "proper" language, like Go / Rust. You get static typing and easy to deploy binaries, which are solid benefits.
Trying to use JS here is the worst of both worlds. Nobody expects to see JS in this context and it doesn't bring much to the table.
Yes, but this isn't made for general scripting. This project seems to be aimed at JavaScript projects, for example calling zx from package.json/scripts/start
If I had a JavaScript project, and I wanted to migrate my scripts away from Bash, I wouldn't pick Python. Not only would it be another language, I would also have to configure Python to run correctly, ie add it to my Docker image.
Simply installing another NPM package, rather than include an entire other language is a much better and simpler solution.
Python's ok for this as long as you need 0 third party dependencies and it's a single file. As soon as you want to split into multiple files or import something that isn't part of the standard library then it becomes a total disaster.
I think Deno is a really good option here - better than ZX. You can use third party libraries easily, it works with IDEs, and it's trivial to install.
Python has been banned in many places for this sort of thing in my experience. There is just too much shite and baggage associated with python environments for it to be reliably portable between systems. Anaconda that, pip this, 2.x, 3.x, string support. It's a dumpster fire.
I am not saying JavaScript is any better, but python is actively considered harmful for this use case in many places I have been.
in my case im usign groovy scripts with grape dependencies. script is always selfcontained single file. max portability in JVM world. no need to fiddle with dependencies.
Honestly this looks better than Python. I hate all the subprocess.run([...], check=True, universal_newlines=True) and then you have the manage the dependency hell of virtualenvs. At least with node the node_modules is right there.
I read this and think "1. Bash is not great. 2. Why assume that I want to write complex scripts."
I like simple scripts. I am embarassed by any complexity. It is the mark of a cluttered mind.
As it happens, there have been and still are others that dislike complexity, though they may be greatly outnumbered.
Right now there is another submission in /active on HN titled, "A 2024 Plea for Lean Software. Why Bloat is Still Software's Biggest Vulnerability"
"I want to end this post with some observations from Niklaus Wirth's 1995 paper:
"To some, complexity equals power. (...) Increasingly, people seem to misinterpret complexity as sophistication, which is baffling-the incomprehensible should cause suspicion rather than admiration."
I've similarly observed that some people prefer complicated systems. As Tony Hoare noted long ago, "[T]here are two methods in software design. One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors." If you can't do the first variant, the second way starts looking awfully attractive perhaps."
And scripts are not even systems. The idea of intentionally creating "complex scripts" is indeed baffling.
The HN title could be something like "ZX - A tool for writing complex scripts". Or "ZX - A tool for writing better complex scripts". Instead it's "ZX - A tool for writing better scripts".
> I am embarassed by any complexity. It is the mark of a cluttered mind.
It is the mark of a cluttered world.
Sometimes, there is no simple solution. You have to parse this file, you have to use software to do it, and you can't fix up every little quirk in the file's actual grammar versus the grammar you wish the file had. Tag soup HTML is one example, but there are thousands of others, around the world, doing important things for business and education.
Sometimes you get to parse JSON. Sometimes you have to parse SQL.
Just another language Google tries to push, it doesn't really solve anything Python/JS/Bash et all can already do.
Remember Carbon who was supposed to replace C/C++ ?
I may be wrong, but I think it will be the same pattern as Carbon here.
“ Bash is great, but when it comes to writing more complex scripts, many people prefer a more convenient programming language. JavaScript is a perfect choice, ”
Hello everyone, I think I happen to have some experiences to share:
1. Five years ago, I developed my own version of zx/dax version using TypeScript. I call this project TypeShell.
2. In the past three years, I have written x-cmd, implemented by posix shell.
Please visit x-cmd at https://x-cmd.com. It is now close to the public beta version. It is still under construction, and the demo can tell some stories.
It tries to enhance posix shell in various ways. Its installation tar.gz package is only 1.02MB (which can be optimized). Even with sh/awk/curl, it can do a lot of things. These features are called modules.
And, through the self-contained pkg manager (do not require priviledge to install), x-cmd can gain capabilities from existing binaries and other script projects, becoming increasingly powerful.
---
The main reason I gave up TypeShell project was that the startup time became a problem. At that time, node 8 was mainstream, and node 10 had just become LTS. I remember that even using node 10, the startup time was poor compared to python/perl.
In addition to startup performance, size matters. I work on different servers and must carry a 20MB node/typescript zip package to run js script. This was acceptable to me, but I couldn't help thinking, what if the solution was only 5mb. Then I tried to rewrite it in golang, and got a version with a smaller startup time and size. Unfortunately, as I understood bash more deeply, I found that basic functions could be easily accomplished through bash and curl. When I was halfway through, I realized that bash was not available in some places. I challenge to rewrite it with posix shell and awk. After years of effort, x-cmd is almost ready now.
---
Friends, I'm not saying that tools like zx are not good. I do like to write some scripts using js/ts. I believe pythoners prefer https://xon.sh/ . Perl is interesting. Fish is friendly.
However, I believe posix-shell has its own advantages. The balance among size, code length, and expressiveness. I think the only possible competitors are tcl and perl, maybe lua.
But none of the above is included by default in all
server/container environments.
---
I wouldn't advise using posix-shell due to its numerous potential issues. I think people should write scripts using the language and library they are familiar with, and do it for fun.
Friends of js/ts use zx/bun-sh/..., pythoners use xon.sh, etc.
Engineers can use x-cmd like `x <your-script>.[py|mjs|lua|...]`. Then x-cmd will automatically help you download the script runtime, scripts, and install dependencies before running.
This is the future I have always wanted to achieve.
The first command of the first example is a cat|grep... How can I assume the author is an experienced script writer?
In any case, ZX replaces some of the shell's usual quirks with its own different quirks: you have to manually expand * with glob() and ~ with os.homedir(), you can't build commands as strings without repeated calls to push(), etc.
> example is a cat|grep... How can I assume the author is an experienced script writer?
Perhaps because they are writing documentation that can be understood be less experienced script writers as well as themselves, using a very common example seen everywhere?
I find "cat full | ..." keeps the left-to-right flow of a command nicely, and is better understood by those less familiar with shell work than prepending "< file " which I've seen cause confusion.⁰
Yes, it is an extra process. Yes this is less efficient, both due to the fork and the extra IPC along the pipe. But even under Windows where spawning a process is massively more costly than other OSs¹, if you are that worried about this bit of inefficiency² then perhaps you have picked the wrong tool for the job in using a shell/script in the first place?³
> with its own different quirks
I agree there. Though I can see that for some, starting from already using JS via node, this might be an acceptable trade-off.
--
[0] I sometimes also use it in part to wind up the cult of demogification, because they seem to both want to be wound up and deserve to be wound up :-)
[1] Complain to MS about that, not me and my script/one-liner.
[2] Are you using it in a tight loop? Maybe if you are doing something over thousands of files, but then the resources used by the something likely dwarf the cost of cat into irrelevance.
[3] A question that suggests the supplementary: are you an experienced <whatever>?
For a single command acting on a file, especially with no other parameters, like your less example, you are right and I wouldn't, though I have seen it often. There is practically no "left-to-right" flow to break. Longer examples is where I would use it.
Also not in circumstances where explicitly acting on a file like your GCC example. It is unlikely that GCC will commonly be used as part of a more collect pipeline of commands.
Note that I said "I use it", not "I always use it, without fail, no other option makes sense ever". Different tools, different jobs (this is what the Church of Latter Day Demogifiers usual don't accept in this context).
This isn’t “better” than your tool of choice. This is a tool for JavaScript developers to write non-trivial scripts in a portable way without getting lost in POSIX hell. Who came up with this stuff?
https://stackoverflow.com/a/13373256
If you can write shell code, Perl, Python, COBOL, suit yourself. This is for JavaScript developers in JavaScript projects.