Wow, didn't expect to see @xfennec pop up on hacker news while drinking my coffee this morning! I don't know if he'll see this, to be honest didn't know he was still doing things - but this person basically got me into programming and game development-I really can't believe it.
xfennec (and some friends?) I think built a game engine called Raydium, and one of their games called Mania Drive-a Track Mania clone-got distributed with OpenSuse installation CDs back in the day. When I was just like 12 years old, my dad installed that on the family computer and it was all we had, Mania Drive was one of the coolest games on there. Me and my siblings played that for literally days and months on end, making crazy levels we couldn't beat without knowing every turn. It was a huge part of our childhood.
Their game engine was in C with PHP scripting, I remember posting some levels to their forums and asking, in retrospect super dumb, questions and they were so polite and friendly. I remember us joking at the time that the French seemed like these god-like game developers, it had such a profound impact on us, I even wrote about it last year and linked a video of Mania Drive first[0]. I went on to learn Python and then lower-level languages as a result. I'm not sure I'd be coding today without them, to be honest.
Sorry it's off-topic, just really blown away to see a username like that pop up in my feed. Really goes to show that kindness + some cool open source software can have profound effect on people.
Xfennec here, thank you so much for this message. ManiaDrive was a small game made with a bunch of friends, I'm so glad it had an impact on you. I'm now a dad, and it makes me very emotional to read this. Thanks again.
The internet used to feel so human most of the time back in mid-90's because there was somehow an awareness that the person you where talking to was a human in a way we've lost.
Those where the days you could email a webmaster and get a response, when you hung around with like minded people in self-organised communities around a thing that interested them - like reddit without the massive negativity and astroturfing.
The modern internet (or web really) has lost much of that flavour, there are a few places that still maintain it with anonymity - HN is one, specific subreddits, the IRC server I've hung around forever, I've friends on there going back more than 15 years, we where all early 20's computer geeks, now a lot of them are married with kids or have kids on the way - I know a tonne of details about them personally and sometimes don't know their real name.
I do see some echoes of that world in some modern platforms (discord for specific games (in my case Arma) captures some of it).
This is one thing I really miss on Linux when compared with the BSDs, especially with dd(1). On BSD you can press ^t to see the status of a command. All you need to do is issue this command to activate it:
% stty status '^t'
btw, for Linux dd(1) I know about "status=progress", but for me it is a bit hard to remember and specific to dd(1). But, nice little utility :)
This is also handy when using socat to pipe across a network. You can use pv on both ends, one looking at compressed and the other looking at uncompressed data, in order to observe the real-time compression ratio.
tar c foo | pv | gzip | socat - tcp-listen:9999
socat tcp:bar:9999 - | pv > foo.tar.gz
If pv shows that you aren’t saturating your network and are cpu limited, replace gzip with lzop. If vice versa, replace gzip with something more aggressive.
'pv' is great, but you need to have the foresight to use it in your command before you run it. 'progress' seems great for those cases where you didn't realize your job was going to take so long, and you don't want to start it all over again.
Indeed. I created some aliases a few years ago to use pv in these cases. At least until I developed the habit of thinking about pv while composing the command.
I just backgrounded the process for pedagogical purposes. Usually a process I'm concerned about is running in the foreground anyway, so I signal the process from another terminal.
Probably not used as much by programmers, but it's a great swiss army knife type tool for copying stuff around with more finegrained control. Sysadmins use it a ton. Probably more often through scripts than directly.
The nice thing with ^T versus status=progress is that ^T will work even when dd was invoked from some script that you don't necessarily want to edit, etc.
Yeah, and backing up the MBR, creating files of a given size like keys, yadda yadda. Anyone getting nitty gritty with storage media, legacy hardware etc, should know their dd.
I first came to love it learning how to fix my bootloader and undo all sorts of horrible things I'd done to my system when I was a teenager exploring Linux well over a decade ago. I also worked in repair for a while and there it was indispensable for data recovery purposes along with its cousin ddrescue.
Now I just use it for my constant tinkering needs.
^t (SIGINFO) works with a lot more than just dd. It also triggers kernel behavior (printing the current waitchannel, e.g., named lock, if one is relevant) which can be useful for diagnosing why a command (any command) is hanging. (Is it busy in userspace, or in the kernel? And if in the kernel, where?)
I used it a lot (didn't miss the progress bar though). With it's options to control caching, it can be used as bare-bones single-threaded performance test for sequential access. Won't replace more elaborate test suites, but has its use as sanity-check.
I feel that dd is never the correct tool to write to SD cards. At least cp can figure out the block size itself. Or the more elaborated bmaptool can even skip empty blocks which are often found in disk images.
> At least cp can figure out the block size itself.
Why does it matter? Use `bs=$((4 * 1024 * 1024))`. It'll work perfectly for any imaginable block size.
My issue with dd is that it's possible to write corrupted data with some weird flags which I did once. Something with conv=sync I believe which does unexpected things. But if you're not trying to be too smart, dd works fine.
If you get into anything Raspberry Pi-based, you'll do it a lot. They are shockingly stable once booted, but an M.2 port is my dearest with for the Pi5.
I occasionally have a new stupid idea for which I want to use a fresh install on a Raspberry Pi, so I flash a new SD card with dd maybe 4-5 times per year.
Inevitably it turns out my idea was not as useful as it seemed when it first popped in my head, so after a few weeks/months that Pi is turned off and returned to the pi drawer.... ready for the next brilliant idea.
(ps. I typically use the `pv` command to see progress with stuff other than dd)
dd is the command to use when transferring bootable images onto USB dongles for installations; when working with embedded boards, especially for testing, one can use it like 20 times or more in a single day. Having a progress feedback isn't vital per se, but becomes useful when using by mistake a slow USB dongle, or plugging it in the wrong, slower, port.
I very rarely use dd. And thank $DEITY for that, because my typical use case is creating a USB boot drive and then backing up or recovering what I can from a failing disk.
The good reason is apparently that scrollback in vgacon was broken and there was no-one to fix it. No-one has stepped up to volunteer to keep it fixed in the future in the two and a half years since then.
Long-term we'll probably end up with the console in userspace, which should naturally normalize scrollback behavior across all the kms drivers.
There's already kmscon but I haven't looked closely into what prevented that transition from moving forward as the default. Presumably there's a bunch of edge cases a kernel based console doesn't suffer from, like having to avoid oom killer or getting paged out/becoming unresponsive on systems falling over etc.
The console process will have to be something exceptionally configured, pinned memory, realtime priority, oom-immune, all that jazz.
Not really. Most of such a system can just be ordinary processes, and there's no requirement for pinned memory anywhere. Pretty much only the terminal emulator processes are problems if they die, as that effectively becomes a hangup signal, so only they need the oom immunity, if that.
Sure, but the BSD convention meets that until you hit ^T, and then it does the thing the user wants (give a report) and then it goes back to being silent. There's nothing wrong with that, and the BSDs are Unixen, so... It's not like ^Z (SIGTSTOP) is un-Unix-like just because Unix VII (and System V) didn't have it!
UNIX-like is having your session or app time out because it gets no output. going all the way back to HP Omniback 2 running BCV split pre-exec scripts on an EMC array to back up a database to tape off a split mirror. Then aborting the backup because the array-side mirror split took over 15min, with no output.
UNIX-like is not a convention of UNIX - it's a convention of "we didn't get to making that useful thing yet, there's other stuff to do."
Perhaps an alias in your .bashrc for dd would solve it? I usually just use ddrescue most of the time (primarily because I prefer it's usage syntax, but it also reports status).
Similarly you could alias rsync instead of copy and move:
alias pcp='rsync -au --info=progress2'
alias pmv='rsync -aP --info=progress2 --remove-source-files'
status=LEVEL
The LEVEL of information to print to stderr; 'none'
suppresses everything but error messages, 'noxfer'
suppresses the final transfer statistics, 'progress' shows
periodic transfer statistics
I am aware of both of those, but in my case I prefer ddrescue for the syntax. For backup scripts I use dd and pv (and pigz).
I recall I started using ddrescue because one of my distros would not autocomplete with the if= prefix, but I can't seem to replicate it currently - so it's probably been fixed (or my memory is failing me).
I think I'm mistaken. Tested on a few Ubuntu machines and it didn't work. I must be conflating my experiences between local on MacOS vs remoting to Linux machines.
I use the pv “Pipe Viewer” tool to do the same. You can either put it in the middle of a pipe, or pass it a PID using -p
http://www.ivarch.com/programs/pv.shtml
To stop from saturating a slow shared connection for an extended period? This kind of transfer can interfere with what other users are trying to accomplish over the same link.
you want some kind of managed buffer (aqm/sqm/qos) in your bottleneck router.
openwrt or any linux can do it as the needed parts (fq_codel or cake qdisc) are in upstream for some years now and wrappers like firewall distros or libre-qos make it available via gui.
there is also some proprietary soho gear that allegedly works, big-iron is still on red/pie w/o fair queueing afaik
QoS is sort of a last resort, since it works by basically breaking TCP intermittently (dropping ACK segments). Any time you can throttle further up the stack is ideal.
tl;dr SQM is needed in most bottleneck routers as endpoints cannot be relied upon to know or respect resource constraints.
yes and no. "throttling up the stack" is crude but effective in a narrow set of circumstances (just takes an additional transfer from another device to render it ineffective).
ack thinning (dropping ack segments) is supported in cake but it's not a mechanism of throttling but for very asymmetric lines such as docsis where surplus ack's can saturate the upstream.
dropping of regular packets is the only way to reliably communicate "slow down" between tcp-endpoints (yes ecn exists) but this is not the main selling point here. every queue has to drop packets at some point, codel does it in a smart way.
fair queueing dissects diverse packet streams into flows (same set of src/dst ip/port) and separates them in order to minimize interference. this is what you want basically everywhere as ip-networks are by definition "best effort" and no amount of userspace whishmakeing (dscp,l4s) is going to fix it.
On a VMware virtualized infra, when you migrate a virtual machine from one (ssd, very fast) datastore to another (mecha, much slower) datastore over a very fast link, sometimes you get a corrupted VM on the target.
I've used pv before to down-throttle a massive wipe of old database entries in production. Doing it full-speed would have killed the site for active users.
in my case because the drives it was writing too did not like going fast (SMR) and would fail, replugging them in caused zfs to fix them, but I'd rather it slows down until I'm physically there so it's not got so much to resilver that it fails yet again
so that you can stop the activity during work / life hours and let it run while you're asleep, perhaps? so if your internet isn't great you don't lose all of your capacity to use the internet
That's interesting! I often found myself forgetting to turn on progress flags on many data transfer jobs and the occasional data transform batch job that I looked into something like this.
I found that `iotop` is great for this kind of thing. Sure, you have to either start it before your process starts or your accumulated total is off, but usually I'm not tracking progress for files less than 1GB so being off by kilobytes is fine.
My go-to's are `sudo iotop -aoP` for general monitoring, adding the `-p` flag if it's just a specific process, or `-u` if I'm monitoring something that is possibly transient.
One of my quick-and-dirty gotos for getting a rough idea of buffered-writes size + disk-write activity on random linux systems is: `watch -n1 grep -ie dirty -e writeback /proc/meminfo`.
You can invoke `sync` to watch the buffered-writes queue burn down when you have lots of pending writes.
> It simply scans /proc for interesting commands, and then looks at directories fd and fdinfo to find opened files and seek positions, and reports status for the largest file.
Wasn't expecting something as simple that at all. Bloody ingenious.
IMHO, it would be much better if linux implement SIGINFO. It has been present in BSD's since forever and there is a good linux implementation: https://lkml.org/lkml/2019/6/5/174
Random aside - I know formatting discussions border on the religious (and why something like gofmt is the only correct answer & yet I am also good with spaces for Python) but..
Did anyone else look at the code and ask themselves- what is the actual formatting standard being used?
Looked like a mix of “open brace on same line, 4 chars indent for code” then “open brace on new line, code at same zero indent”
Not a big deal obviously. Just something that tripped up my eyes scanning the code.
From a quick look to one file, seems fairly consistent. Perhaps most unorthodox but at the same time easiest to justify: function bodies skip one level of indentation. The curly brace of function body at column zero is actually a separate, very traditional style. It's just that both styles apply to function bodies, there is no other causal relationship. Finally, indentation is 4 spaces, and tabs are expanded. Of course a very few places may be miss-styled, as happens with hand crafted code. While definitely sinful for not indenting with tabs as God intended, the style is not messy
I've been adding status=progress to my dd commands and getting progress reports for years now, not sure when it started working like that but any current Linux should have it.
I used to have a habit of running 'killall -USR1 dd' to get the status of a dd process.
Then I switched to using sddm for my display manager/login screen. It didn't have a USR1 handler. If you kill your display manager, you're forcefully and suddenly logged out of your session.
Pipe Watch continues to read from the pipe even when backgrounded.
Pipe Watch shows you snapshots of the text that is passing through it. You can set triggers and filter and such. The triggers work even when it's in the background, not refreshing the display..
The method this uses is cute, hacky, and useful. Makes me want to write an osx background thing that uses the same scheme and pops up progress windows whenever I run a coreutils thing.
The "how does it work" section doesn't make sense for macOS since there is no /proc there. How does it work on macOS?! I tested it and it works like a charm!
Look at progress.c, start with "#ifdef __APPLE__" and keep looking from there. Something called libproc, there's some headers[1] I've found but I couldn't find any man pages unfortunately. You need some way to look at open FDs and every system will have such an API, even if it looks slightly (or very) different from the next one.
I think it is probably because of the UNIX philosophy of the virtue of silence: 'The program should say nothing if it has nothing interesting to say'.
It is an open question whether progress is 'interesting' or not. My own opinion is that it is not interesting if the operation is nearly instantaneous. If any operation can take more than 5 seconds, it should have a progress bar and an estimated time of completion.
Earlier versions of Windows did this very well. Often these progress bars were a joke, but sometimes they were useful. It gave us valuable input on whether there is enough time to go get a cup of coffee, which as we all know, is the most important question for us all.
By the way, regarding Windows progressbars - even if a particular implementation is garbage, like for example passing to 50% instantly, then sitting there for a minute, then passing again to 75% and staying there and son on, they are still useful if your are using the same tool or operation repeatedly. My VPN progress indicator is junk, but I can roughly guess where is the problem based on where it got stuck now, simply because I've already saw multiple failure modes of it. Or if I'm installing some CPP distributives repeatedly, I can guess how long remains.
If an operation is silent and hogging my prompt without any indication what is going on, is stuck, is the speed too slow etc. it is simply frustrating.
Another dimension is whether the (what is the correct noun here?) is a TTY or not. Arguably a progress display, especially one that deletes itself afterward, is a useful default when it's a TTY, but obviously it makes scripting annoying if you have to constantly add -q or -s (looking at you, CURL) to every command to shut it up.
True, I think my phrasing/usage of terms was incorrect. I think my usage of "TTY" was incorrect.
What I was thinking of is, a user invoking a script interactively which happens to call a tool (like CURL) may not want to see curl's big progress song and dance for every HTTP call the script makes, though a user invoking `curl` literally, would likely appreciate it. But I think in both cases curl sees a TTY, right? Oops.
You imply a great point that at least doing `2>/dev/null` in those scripts is at least consistent, and I just need to make a better habit of doing so.
Earlier OSes and applications with only cooperative multitasking had to do this well; if a misbehaving application/operation was running it could just monopolize the system-wide event loop and stall the entire system.
I think commands could have progress but only as explicit option. I wish curl and 'docker pull' didnt spew so much text by default, filling up so many jenkins server's disks.
I know you said “by default” so I’m assuming you know some ways around this, but in case you don’t, or for anyone who might find this useful … you can be really specific with curl outputs. It’s pretty sick.
Thanks! This sounds like a great tool for simple scripts where something like a quick response code (etc) would be useful to know. I'm imagining something like:
It's more likely that this is the entire "upstream downstream" development model in action again.
There are vast numbers of improvements to softwares that never get sent to the places where they will do the most good (or at least have a proper record of why they were rejected by their authors). A quick perusal of the bug trackers of Debian or Ubuntu will reveal tonnes of local patches that the original authors of the softwares often never even hear about.
Things don't "stick" often times simply because they get lost.
I was looking at something like that just the other day. Here's a bug report that describes a problem with "doas" not opening the controlling terminal to do its authentication dialogue. This is actually a problem with a package named LinuxPAM, and doesn't occur when "doas" uses OpenPAM or BSD Auth. It's LinuxPAM that's where the code in question is. Fixing LinuxPAM would improve the lives of everyone that uses LinuxPAM, because the behaviour of not allowing standard input through in a command pipeline is not confined to "doas" but affects everything that uses LinuxPAM to do login authentication.
But time and again stuff like this languishes in the wrong place, for years and decades.
The philosophy surrounding these tools wouldn't lend well to each implementing that. Fortunately `pv` exists, is perfect for this use-case, and included in many distributions. :)
`rsync` interprets paths a little differently than cp (such as trailing / or no trailing / on directories). Do you have a convention you normally call this with to avoid differences in behavior?
Great to see this here! I've never used it until now because instead I used a 20 year old shell script (later converted to Python by a friend) to do the same thing.
I gave it a try on a ddrescue job (unfortunately not recognized by default) and the estimated remaining time varies quite a bit between what ddrescue says and what progress says. I think ddrescue uses a larger moving average window and it seems to give more accurate estimates, although they are still far from perfect.
Cute, clever. There may be edge cases it doesn’t get right (parallel downloads/copies) but pretty useful when you’ve launched a job and didn’t think ahead of time to ask for progress.
I’ve used a signal handler (SIGHUP aka control-c) before as a “show progress” mechanism that I found very useful to monitor long running processes that were compute not io related (launched in screen fwiw to stay the active process).
Very cute and very clever. I think this could be extended with a watcher/main loop of its own, and keep a bit of context to spot any long transfers (say longer than 5s), and display a system-wide summary of all large transfer going on.
Yep I know. It was easier to do in the same window with the program in the foreground vs opening another window to send the kill -HUP to this one. Double control-c still killed the process.
pv is invaluable but my issue with it is it’s only useful if you think to include it before you run the command. Usually the thought doesn’t occur to me until I have some runaway cp or dd and I want to be reassured that it’s going to be done soon. For that case that this looks super interesting.
I'm sure it's nearly impossible, but this needs to be integrated with cp, mv and dd. Otherwise, usage will be a fraction of a percent.
I SSH into many many different machines (desktops, routers, servers, IoT devices etc..), and unless something is in the default install of RHEL, Debian, Arch etc..,I tend to not rely on it, as my muscle memory will cause me problems.
I have the same issue, but with the opposite conclusion: if this were integrated with cp/mv/dd, I wouldn't be able to rely on getting the new versions.
But since it is separate, I can install it where and when I need it, perhaps with the transfer already in progress. That might be hard for IoT/containers, but do I really need to see file copy progress in those kinds of places?
Good point on updates. But if Debian pulled in latest every 2 years (Debian release cycle), wouldn't that be enough. I can't imaging improvements being so urgent.
Also, agreed on the need for this on IoT stuff. But it's about muscle memory. I want to type the same way on. all devices. Not "oh, I'm on a server, use this command. Oh, I'm on an IoT device, use a different command"
On FreeBSD you can do CTRL+T to show progress, if I remember, that's a special shourtcut similar to CTRL+Z/C supported by BSD systems. There were some discussions a few years ago that Linux should introduce the same thing, but they never did.
xfennec (and some friends?) I think built a game engine called Raydium, and one of their games called Mania Drive-a Track Mania clone-got distributed with OpenSuse installation CDs back in the day. When I was just like 12 years old, my dad installed that on the family computer and it was all we had, Mania Drive was one of the coolest games on there. Me and my siblings played that for literally days and months on end, making crazy levels we couldn't beat without knowing every turn. It was a huge part of our childhood.
Their game engine was in C with PHP scripting, I remember posting some levels to their forums and asking, in retrospect super dumb, questions and they were so polite and friendly. I remember us joking at the time that the French seemed like these god-like game developers, it had such a profound impact on us, I even wrote about it last year and linked a video of Mania Drive first[0]. I went on to learn Python and then lower-level languages as a result. I'm not sure I'd be coding today without them, to be honest.
Sorry it's off-topic, just really blown away to see a username like that pop up in my feed. Really goes to show that kindness + some cool open source software can have profound effect on people.
[0] https://devlog.hexops.com/2021/increasing-my-contribution-to...