I'm not going to go into details - they are long, torturous and (of course) occurred very early in the morning - as they always did.
But, rsync.net used to run - exclusively - on UFS2 and, circa 2009/2010 we were pushing UFS2 to the very limits in terms of disk size, number of inodes per filesystem, size of memory used to fsck, etc.[1]
Several times we came very, very close to destroying a filesystem and losing customer data.
What made all the difference, every time, was:
- don't panic
- postulate what is happening
- do something non destructive that will prove or disprove the theory
- possibly repeat
Go slow and don't take any steps until you have proven that you understand the problem.
ALSO, when working with filesystems remember that you can run a filesystem, non-destructively, in read-only mode. There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive.
Your first steps should always be read-only ...
[1] I asked Kirk M., in 2010, what the long term solution to our problems with UFS was and he said "migrate to ZFS".
> There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive.
Some years ago (but not so long ago that I shouldn't have known better...), I wiped out by mistake my home directory were I had a few days of unpushed changes (yes, second mistake) to an important project I was working on.
When I realized what I had done, I immediately remounted read-only and I managed to preserve the data; a few hours later, ext4-undelete saved the day.
I did something like that recently, deleting a project directory with a mistyped git command intended to preserve the contents for safekeeping, ironically. (That's an interesting lesson in itself).
It was in a VM, so rather than remount read-only, I snapshotted the VM drive, then rebooted to perform recovery, confident I could try out things non-destructively by reverting to the snapshot.
Used ext4magic to undelete, and that tool did a really great job. Some files ended up with contents that should have been in other files, so it isn't perfect. But it was a project with only a few files, so I was able to reconstruct it easily enough because all the contents were at least recovered.
Did something similar for a customer a few years ago who had wiped a significant directory on a live system. That was on a RAID-1 (mirrored).
Rather than remounting read-only, to allow production to continue I detached one of the mirrored drives to preserve it for safekeeping, while allowing operations to continue using the other drive.
ntfsresize also destroyed my partition back then. When I searched it, I found the responsible bug in the bugtracker, closed several months ago, but it never made to the stable Ubuntu release. Fortunately, a manual fix guide was available. I stopped trusting Ubuntu and switched to SystemRescueCD (which is Gentoo based with newer packages) after that incident...
My trust in Ubuntu was greatly reduced when (about five years ago) the installer would put GRUB on the primary drive regardless of where the Linux system was installed, and even when you explicitly told it not to. This was fixed within a few months, but not before it trashed several Windows installations (using whole-disk-encryption so they were not recoverable).
It was linked to in an LWN article's comment section, titled "If you think you're really good, read this.", or something. The story proved very effective at humbling its reader for me.
That doesn't change the fact that I, too, once felt like the uncrowned teenage king of emergency system rescue operations back in 2002 or so, right in my parents' basement. I had started recursively deleting / by accident, and ended up with most directories, /bin and /sbin included, already turned to dust, and most of my as of then proudest achievement - a Gentoo Linux stage 1 installation - gone.
I dug myself (and the standing remains of my /home) out of that particular hole by virtue of still having an instance of a browser (I think firefox) running. Mildly panicking, I still left everything untouched ... and after a few hours of thinking hard about what to do, I arrived at the following: I set up a webserver on my mom's windows PC with am embedded PHP interpreter, and hacked up a script that enabled file uploads via HTTP over the LAN. After using that from within my still-running browser (fortunately, it could still list the contents of my home directory) to evac the most precious artifacts my ~ had in store, I rebooted my machine into the lightless nothingness of not having a kernel image in /boot any more. Another few hours later, I had Gentoo reinstalled, and copied the rescued files over to my own machine again - and felt like I had levelled up a little bit.
If I were him, the best I could imagine was doing documentation reading and ASM writing on a separate computer with working toolchains, testing it, and finally entering some machine code in hex to the broken machine by hand. But he literally wrote an ASM program and constructed an ELF on a totally broken machine itself on the fly, with no documentation, no toolchain. Absolutely wizard.
For example, it was how the syscall number was found without grep...
> The next step was getting the syscall numbers. grep? We don't need no stinkin' grep.
# while read i; do case $i in *__NR_link*) echo $i;; esac; done
</usr/include/asm/unistd.h
#define __NR_link 9
# while read i; do case $i in *__NR_exit*) echo $i;; esac; done
</usr/include/asm/unistd.h
#define __NR_exit 1
#define __NR__exit __NR_exit
At my first job right after graduation (less than 5 years ago), I managed to accidentally terminate production servers on a Friday night.
When I realized my mistake, my heart sunk. I got cold sweats. I was afraid I'd get fired. To this day I'm so glad my team leader was still at the office at the time. I told him about the blunder, and he asked me about the details. He sat down next to me, and we started working together, exchanging information and getting replacements back up. There was no CloudFormation, so everything was manual.
He never accused me in the postmortem, and I still feel so grateful for it. I learned from my mistakes and took action to prevent it from happening again.
Listen when you break something tell us. Explain what you did and we can get it fixed. You will make mistakes, we all still make mistakes.
That is what i always do and expect my coworkers to do. Yeah we talk a little smack afterwards (only jokingly), but were able to resolve most self inflicted wounds quickly. Nowadays what scares me more is a coworker who hides mistakes rather than one who confesses to them.
Another useful thing to remember: if you deleted a file and a process still has it open, you can copy it from the file descriptor in /proc! Just use lsof to find the PID and FD. This has saved my bacon a few times in the past ;P
> Windows won't let you delete a file that's still in use, so this scenario can't happen.
Windows lets you delete files which are open all you want, and Windows also lets the application which opens the file prevent that. A lot of things like this are blamed on Windows when it's actually Windows providing options to the application developer, and the application developer choosing to take advantage of options like that when they don't need to.
That is also problematic - and if I recall correctly partly one of the reasons behind why windows requires restarts so frequently when software updates are on the table.
Yes, that is indeed the major pain. Unix-likes have an actual VFS as an API and it happens (at least on every unix-like I've worked on for any bit of time, which is mostly Linux but I expect BSD has it as well) to allow for comparatively wondrous workflows.
Windows programs also often only support 'CDs' (ISO9660 / UDF), FAT, NTFS, and maybe SMB/CIFS (Samba / CIFS) network shares. Since Windows lacks (or at least historically lacked) a general VFS they could target. That's why any actual VFS that ever gets made, due to all the legacy software, is either going to directly support one of those interfaces, or have a compatibility mode which does.
I do wonder why they've not chosen to make progress in that regard to advancing the state of the system. Surely these are known good ideas that would improve the customer experience and the system from a technical level. I only speak as a very casual windows user, there is nuance I will miss with Microsoft's attitudes to it.
I would have thought that even a separation between OS and User applications could allow them to implement features like a VFS, while still maintaining compatibility. Are they not part of the way there with Hyper-V + WSL?
Microsoft would love to do these things, but backwards compatibility guarantees prevent advancement a lot of the time.
You would not believe some of the business critical software out in the field. They rely on very obscure bugs being present in order to function, and they sometimes can't be fixed because the source code is missing.
Apple has chosen the opposite path, which is to give the middle finger to long-term backwards compatibility, and advance their operating systems technically.
You can't run an OSX game from 2002 on the Macs of today, but you almost always can run a Windows game from 2002 on the Windows 10 PCs of today.
It has more to do with Windows services and applications (and even the kernel) only reading file contents on initialization which always happens on a reboot.
Sometimes you can simply log out and log back in, and the requirement to reboot is nowhere nearly as frequent as it used to be, believe it or not, but yes you still have to reboot for kernel patches and core service updates, and there are a lot of those things these days.
This includes executables. I've had Tmux incompatibly upgrade itself on my while I was running an important job; I recovered the old executable from the running Tmux server, then used it connect and finish up the job.
On day in my previous job our junior support guy started cursing loudly. He had done what he was asked for and did some some small, _almost_ day-to-day support request which was supposed to do some requested maintenance things for some credit card accounts our company was handling. He by some small accident ended up invalidating the cards with our CC operator and closing the accounts on our end.
He was so wrecked about it and we, the more senior guys started helping him and comforting him that "yeah, this kind of blunders can happen to anyone, no worries. Let's figure this out together.".
The other guys figured how to restore the data into correct setup on our end, and I found some unimplemented badly documented "undo invalidation" endpoint from the CC operator with which we could fix the stuff on the operators end.
So we spent nice and fun evening together figuring out the problem as a team, but damn the guy was so torn up about his small and completely understandable fumble that nobody else needed to critique him about it.
The funny thing is that with this kind of blunders one learns a lot more and a lot faster than usually. People need to make mistakes. It's usually the best way to learn.
“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. “No”, I replied. “I just spent $600,000 training him – why would I want somebody to hire his experience?”
Good point about the translation og "klok". I was about to use "wise" but ended with "knowledgeable" since it felt like the most literal translation.
For anyone still following I think "wise" maps to the Norwegian "vis".
BTW: I think old proverbs are really interesting - both local ones as well as international ones. Sooner or later someone should try to create a complete encyclopedia of proverbs :-)
In mid 2000s as a lead HPC sysadmin I accidentally pasted 'shutdown now' into the head node of a 200 CPU HPC cluster, which was 3 weeks into a 12 week simulation job! Fortunately that thing never shut down cleanly and it hung as usual, unable to unmount the NFS share used by all the compute nodes. For the next 9 weeks I was staring at that screen hoping it would keep going and I would not be fired. In the end nobody was ever the wiser.
I once fat-fingered crontab -r instead of crontab -e on a production accounting system running Solaris which immediately wiped out something on the order of about 200 cron entries. Had to restore the contents from off-site backup tapes.
The jokes circling the office for the next few weeks were all variations on "Remember alyandon - r doesn't mean 'revise'". :(
Same thing happened to a bloke in my old team, except of course we had no backups (this was on an unofficial ops server that ran business critical scripts). He spent the rest of the week putting the crons back together. Now the first cronjob in there is to create a backup of the cron file.
Oh wow. This reminds me of a game my friends and I created in college: System32 Roulette.
It was a drinking game where we would set up a Windows 98 machine and take turns randomly deleting a file found in the system32 folder, then rebooting.
If the computer fails to boot sufficiently to allow the game to continue, you lose and finish your drink (and have to reinstall windows).
It was pretty fun because windows was quite good and whining and complaining about countless problems but still being functional.
Seems to me that part of the whole ethos of Unix being "The Hole Hawg of Operating Systems" (See Neal Stephenson's "In the Beginning was the Command Line") was an almost actively cavalier disregard towards such issues, as if putting the ejector seat lever right next to the gearstick is an acceptably red-blooded thing to do. The best we can say is that attitudes have evolved somewhat since.
You'd think by now we'd have something like a vicron command similar to visudo that would safely allow you to edit and then verify the contents of the crontab.
crontab, like a few crusty old Unix programs, has this bizarre UX design where it's basically just launching vi, cat, or rm on a particular file. It probably shouldn't exist at all.
Most places don't. Cron is great for hobbyists and personal use, nobody should be using it in production. There are far easier and safer ways to schedule recurring tasks.
I've wished I worked at places that saw this as a more serious problem. Ideally, we would know there's a cron that stopped working and we'd have to figure out which machine and which user it was on...then try and access it. Even the people that create them forget these things (myself included).
The most reasonable solution seems to use something like Jenkins--but I don't think anyone is happy with that.
At one job they wrapped crontab and used RCS to write to a network location. Now you can grep a single folder, its now version controlled, and you can wire any re-image script to restore crontabs. I've pitched this approach at subsequent jobs, but was never in a seniority role to get it rolled out.
At most places I've been if there is something like Jenkins at all a single person sets up and troubleshoots the jobs. The average dev just looks at red/green, logs, and emails but only interacts with the code--maybe they manually run a job. So you're asking them to now author and modify things in Jenkins and I'm not even sure accounts and permissions were set up. So that's why the dev group isn't enthusiastic. Then you have sysadmins who otherwise have zero interest in Jenkins and aren't familiar with it at all.
Jenkins is a very capable app, but it's not "light" or approachable. Introducing it to new groups of people for an off-label purpose is a tough sell. If you're already heavily using Jenkins and that same group of people need it for periodic task scheduling it's an obvious choice.
If node.js: setInterval for short intervals, Bree for more complicated tasks [1]
If C: time_t
If Java: java.util.Timer, probably something 3rd party that better is wrapped around this
Anyways, the bigger point is that you should be expressing these interval settings in code, not system configurations. Everyone checks in source code but nobody checks in the crontab. It's better if you rely on the application to do these things rather than the system, because the system could always change beneath your feet (i.e. management says we're using Google Cloud now! Good luck porting a 200 line crontab to GCP) but you are in total control of your application's source code.
Honestly all joking aside systemd timer units have a lot of nice aspects. The only drag is that crontab entries are so much more terse and quick to add. Just one line instead of two unit files plus enabling/starting the timer. You can actually get the best of both with systemd-crontab-generator (not from the systemd project, just an unaffiliated third party).
It creates transient unit files based off of a standard crontab on boot and after any file modifications. If you want to extend one of your cron jobs with something that you can't express in a crontab you can just mv it from /run to /etc and modify it however you'd like.
By chance, do you happen to know of any program that will add (and enable / start / etc) a systemd entry based on a valid crontab line?
Extra bonus points if it yoinks things the other way and lets you edit them as a nice crontab file that makes sense before re-integrating it in to the nightmare syntax.
This happened to a coworker of mine, only it wasn't a prod accounting system, it was a prod customer facing system. We got the crontab restored, I though I forget from where. I think we ended up doing a blend of common stuff copied from a sister system, and bringing in other entries from the detritus of an ad-hoc backup that was fortunately still on the system.
My poor cowoker still feels embarrassed about it, even though it's been some years since.
Early in my Ops career I wrote a log handling script for Windows servers. It was a re-write of an existing one that pre-dated a lot of us. My version used an xml configuration file, but this iteration of the script did not properly validate the configuration items read from the file. Also, Powershell had this weird xml parsing bug that if a parent had a commented child element, it would show up as an empty child element.
So I came in one afternoon shift and my colleague said "your script destroyed two Production servers last night". These servers were critical for monitoring Police radio GPS coordinates.
"Impossible! It should only work on directories we configured it for!"
I sat there looking at horror at the debug logs, and sure enough, it read the empty child element, changed directory into "", looking for files with "" name pattern, which for Powershell resulted in C:\Windows\system32 and all files. It proceeded to zip and remove "old" files. On a reboot, they did not come back up.
To add insult to injury, when they rebuilt those servers manually, the same script with the commented out config was deployed, destroying the servers again. They realised that the script was the issue and decided not to deploy that the second time.
So I learned a lot about config validation and defensive coding in the months following the incident.
Could someone shed some light on the constraints of Unix & VAX at the time?
The decision making here does not make much sense to a youngster (36) like me.
Why couldn't they just mount their half deleted drive on an other machine and restore it from there?
I have the feeling all these other hundreds of seemingly easier solutions were out of the realm of possibilities back then, but I am unsure of what were the actual limitations / constraints of that era.
In the late 1970s/early 1980s VAX hard drives usually took cartridges which could be conveniently unplugged and carried across to a drive on another machine. But by 1986 drives were often "Winchesters" - sealed, rack width, a foot high, and weighing maybe 100kg. The cables were an inch thick and the plugs the size of a large ham and cheese sandwich. Hence the comment in the article that they'd need to pay a DEC technician to come out and physically do the move.
I'm not that familiar with the relevant systems, but this was before plug-and-play was a thing, and moving drives around wasn't a minor operation. Mario mentions a similar option in the article: "Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. "
They could have restored it on their current machine/disk FWIW. You could do a tape boot, boot to single user, and copy the OS and rebuild the devices. I think they were just unfamiliar with that process.
I managed to break a btrfs filesystem by exhausting metadata blocks. The filesystem would show as 50% occupied, but 0 bytes free. Everything started crashing. The resolution is to run btrfs balance to free unused metadata blocks. There is a catch - you need at least some free space to run the balance. I had none, but managed to remove some unnecessary files (system package manager cache) to have just enough free space to run the balance. In meantime I discovered this condition was created by npm creating thousands of files while installing packages.
Later I run into the same problem (again with npm), but this time there was not enough of unnecessary files to delete to be able run the balance (package manager cache was already purged). Normally this can only be solved by adding new partition to the btrfs filesystem and I had no unformatted or unused partiton on disk left. Instead I wanted to get rid of the btrfs that failed me twice and rsynced whole btrfs content into a loop-mounted ext4 disk image on a NTFS drive and then after unmounting I dd-ed the disk image over the btrfs. It booted! Had not any more filesystem trouble ever since.
This brings up an interesting issue in OS design, and that is:
What are the responsibilities of the OS's Kernel if/when all files are deleted, and all processes are similarly deleted?
In other words, the worst-case of all worst-case scenarios.
Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel?
Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only.
Also, we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty" -- for future recycling.
Between these two things, if they were implemented, we'd get a lot of leverage to resurrect a dead system, but there's more to this than that.
You see, the above scenario should become an active unit test (tested at each recompilation) of that future OS.
Also -- we want to know the fewest number of steps necessary (given the above scenario) to fix everything back to normal.
In other words, a very interesting topic for future OS designers to think about!
Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel?
Practically speaking, none. The responsibility of the kernel (in Linux terms, but I think it's the same across all Unixen) is to start the init system (PID 1), and that's where it ends. Should PID 1 ever terminate, the kernel is in an undefined state. The VFS semantics ensure that even when the init binary is removed from the filesystem, it is not actually unlinked until its reference count is zero -- and since the kernel holds a reference to it, the kernel and init are "safe" from harm through deletion.
Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only.
While not in-kernel, systemd is ahead of you, as it offers a shutdown-time ramfs to use as root filesystem so the real root filesystem can be safely unmounted. Previously, Linux didn't really unmount the root filesystem (it couldn't, because the kernel still had an open handle to the init binary), but mounted it read-only before shutdown.
we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty"
This is already the case. "removing" a file in Unix only removes its name from the directory listing, the inode isn't reused until it's been cleared. And even when it's been cleared, only the inode entry is zeroed -- not the data contents. That's why tools like "shred" exist -- to make sure a file's contents are removed along with the file.
a very interesting topic for future OS designers to think about
I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
Hi Tremon, first of all, your response is interesting, valuable, and it taught me some things I didn't previously know, that is the stuff about PID 1, etc.
So, I appreciate that!
Also, I have no question that you are a smart person.
But, that being said, I'm going to play Devil's Advocate a little bit here, if that's OK with you, as I don't think that people learn anything from agreeing with one another; but rather by the friendly and polite challenging of one another!
OK, so with that in mind, here we go:
>"Should PID 1 ever terminate, the kernel is in an undefined state."
Undefined by WHO? If there's nobody that takes the responsibility for defining something, then there is no rule, law, custom, or precedent there, and all of those things may now be implemented as seen fit by one or more implementing parties.
If that's the case -- then why can't it be implemented differently, why can't that be changed in the future?
>While not in-kernel, systemd is ahead of you
That might be true -- but I am NOT a member of the systemd religion! (You know, like you have Christians, Muslims, Buddhists, Hindus, Jews, and a smattering of other smaller groups in that bunch, like Wiccans, Scientologists, Deep Staters, and people that follow Joel Osteen! <g> Well, you have all of those, AND you have the people that believe in systemd! Of which I am not one of! <g>)
On a serious note however, systemd, while it might create a whole lot of "conveniences" -- creates equal-and-oppositely many drawbacks in terms of code complexity and the ability for someone even moderately technical to understand what's really going on on their system.
But, all of that is a "religious debate" -- of which we've already seen too many of on the Net...
>This is already the case. "removing" a file in Unix only removes...
Yes, this is well known... However, let's say we get a future OS student up to speed by building their own operating sysem... if they're building their own OS, then their file system will probably not do this. It a separate consideration and it adds to the complexity of the file system.
On the one hand, yes, you're quite right, such a student could just take pre-made source code for a filesystem and link it to their OS, but if they did that, then odds are they'd learn nothing of value...
>I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
I don't question this.
But, if you took the world's most technologically complex airliner, whether that's a Boeing 747, an Airbus A-380, or what-have-you... most people would rather fly on one of these things than even attempt to build their own plane, even if their own plane would be simple and crude by comparison, even if their own plane would be many future iterations removed from the current state-of-the-art...
Well, even if someone attempted something like this and failed(!) (e.g., they tried to build the equivalent of a Wright Flyer -- the first heavier-than-air plane built by the Wright Brothers) -- well, that is far, far superior educationally (in my opinion!) to someone who uses the latest and greatest technological marvel of the world -- and, yet, doesn't completely understand it...
You might say that that's my religion...<g>
It's the same thing with Operating Systems...
>I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
I completely agree that they might...
But you know, we might agree that Windows 10 does most of what 99% of computer users want -- so why use any Unix derived variant?
I completely agree that you're right; however, Windows 10 probably does most of what you're looking for OS-wise... yet you seem to be more inclined to use a Unix derrived OS...
I blindly followed a 'news' item's advice once on my home Gentoo Linux that said a particular version of glibc should be removed. Learned the hard way that you definitely want to make sure you have emerged another version of glibc first! Hardly anything works without it (can't even auth as root)
Unfortunately unlike the cool cucumbers in this story, I hastily rebooted, and it took a fair while for various reasons to get hold of a boot disk and get it repaired and running again. I love Gentoo but it does take a lot of patience generally, and this kind of thing seems to happen fairly continuously (which teaches you a lot but requires that you want to learn and solve the problems yourself).
Reminds me of a few years back when I was still new to Linux (and anything non-Windows in general tbqh), and I accidentally nuked /bin. That was a fun time. I happened to have Firefox and an IRC Client opened. I tried a few different things (on the advice of others in an IRC channel I happened to be in), but eventually a friend created a VM of the same Linux Distro and then compressed his /bin and sent it to me -- along with a few required programs to uncompress it just in case. That was a fun day or two. Taught me a few important things though!
I did something like this on a Sun SPARCStation ELC back in the early 90s. I was setting the machine up and was new to Unix and the shell and deleted the contents of the root directory. Not the sub-folders just the files.
That doesn't sound so bad right, there's nothing important just in the root directory - except in those days that's where the system kept the kernel binary! No problem, I'd learned how to do network file transfers the day before so I just copied the kernel across from another ELC and rebooted.
I was playing with raw disk access and overwrote selected parts of my kernel image once. oopsie indeed but it was a dev system and so figuring that one out wasn't that bad in terms of impact on others. which was good, cuz it took me a week to figure out.
I remember accidentally removing the shared libc on a SunOS box and having to use things in /sbin to recreate it...though I don't remember exactly what I did.
Hey, I did that exact thing (removing libc) on a QNX box years ago. I ended up restoring libc using a series of "echo -ne" commands (generated with a script, and then pasted into the SSH session I thankfully still had open).
Not quite the same, but one time a guy I worked with decided to "clean up" things on a NCR Unix system and set the ownership of various system files to whatever he thought made the most sense. This broke all the stuff that relied on setuid to function correctly. He then went to lunch while the machine slowly ground to a halt. I had to boot from tape and figure out how to fix things from there.
Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files.
I've not used Unix on an early VAX before, but every other OS I've used, including DOS, Windows, Linux, and macOS has the option to use a "recovery mode" shell if you boot from the OS installation media.
4.3BSD VAX was distributed on open-reel magnetic tape.
The second file of the first tape contained a miniroot filesystem which, at the very least, certainly would have included working copies of of the two utilities mknod(8) and restore(8) required to restore from his dump(8) tapes.
As with DOS, Windows, Linux, etc., 4.3 BSD could only mount filesystems stored on random-access block devices (so not sequential-access magtape).
Therefore, one of the first steps in "booting from distribution media" was to copy this miniroot filesystem to the swap partition of a formatted and labeled disk, at which point it could be mounted as the root filesystem (with swapping to this partition disabled, obviously).
In the (likely) case that the existing disk layout contained a swap partition of sufficient size on a device supported by the boot loaders and kernel supplied by the distribution, this would have indeed been a reasonable road to recovery, and, assuming the existing disk labels remained valid, the required swap space was available, and the dump(8) tapes were good, shouldn't have required anything beyond:
1. Copying the miniroot from tape to a preexisting swap partition using the standalone copy program included with the distribution media (typically on a console tape or diskette as appropriate for the VAX model in question).
2. Booting vmunix from the miniroot in the swap partition, taking care to RTFM and include the special syntax required to prevent swapping to the partition containing the miniroot.
3. Recreating required special files with mknod(8) (possibly using the /dev/MAKEDEV script from the miniroot).
4. Using restore(8) to restore from dump(8) tapes as usual.
If this happened to me, my gut reaction would be to connect the hard drive to another, working computer, and do the recovery in a fully functional environment. With all the physical work required, it probably would have taken longer. But if anything else unexpected happened, it would have been easier to deal with.
I remember fork-bombing a production server - the sheer panic as process table entries keep getting used up and commands getting slower and slower by second is unforgettable (which is probably why I'm super cautious about anything to do with Production now).
I once (accidentally) took down Netcom with a fork bomb. Not just one node, but all of them.
When I tried to login again after they rebooted everything, I found that my account had been banned. I got a NOC admin on the phone and apologized for my mistake, but also asked him why they did not enforce a maximum number of processes per user (via ulimit). Apparently they had never thought of doing that. He thanked me and my account was restored immediately.
Experiences like this and a bit of imagination are why I try to be not-root as quickly as possible, and sometimes consciously physically remove my hands from the keyboard instead of having them “idling” at their home position.
The thing about a fork-bomb is that you do not need to be root to spawn one. I never had root on Netcom, but still manged to take the whole thing off-line for about 15 minutes.
A key takeaway is "don't be root unless you have to be". Horrifies me the number of times at work, even on a production box, that "who" reveals a couple of roots logged on for no good reason. DBAs are the worst offenders.
But, rsync.net used to run - exclusively - on UFS2 and, circa 2009/2010 we were pushing UFS2 to the very limits in terms of disk size, number of inodes per filesystem, size of memory used to fsck, etc.[1]
Several times we came very, very close to destroying a filesystem and losing customer data.
What made all the difference, every time, was:
- don't panic
- postulate what is happening
- do something non destructive that will prove or disprove the theory
- possibly repeat
Go slow and don't take any steps until you have proven that you understand the problem.
ALSO, when working with filesystems remember that you can run a filesystem, non-destructively, in read-only mode. There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive.
Your first steps should always be read-only ...
[1] I asked Kirk M., in 2010, what the long term solution to our problems with UFS was and he said "migrate to ZFS".