>I didn’t know that I was functioning as an alarm clock for her! That’s why I was careful to be quiet, and why I didn’t even think to mention to her about the new videochat time.
>I suspect this is failure mode is more common than we realize: there is a process inside a system, and over time the process comes to fulfill some unintended, ancillary functional role, and there are people who participate in this process that aren’t even aware of this function.
Ah, Hyrum's Law showing up in an unexpected context, yet again.
With a sufficient number of users of an API,
it does not matter what you promise in the contract:
all observable behaviors of your system
will be depended on by somebody.
When a person leaves a company who has been there for a long time, a similar effect occurs.
These people often do a host of small tasks to that noone is really aware of (themselves included; they don't see how it's special or that nobody else knows about it). It can take the organization a long time to recover and realign.
Only 2 weeks? I work on a program at a Fortune 500 that is VERY reliant on tribal knowledge. Critical people take 2 week vacations at least once a year, we just grit our teeth and do our best to fill the gaps until the given SME returns.
At the end of the day the company has to be willing to make the investment in adequate documentation and training backups. The only time I've ever seen that happen is when people leave permanently, and of course it only happens AFTER they leave permanently. Documentation and training are cost centers, after all /s
More vacation is of course better, but you also have the opposite "problem" in Sweden, if you don't want to take 4 consecutive weeks, but prefer to spread it out, it also usually has to be negotiated and will be treated as an exception and possible denied.
OK, very few people I know get 4 weeks vacation at all, including myself, and of those even fewer could take it all at once without pre-negotiating that with their boss, it would be considered unusual.
But we know different people! I don't know a lot of people with six figure engineering jobs.
Apparently here's what the US Bureau of Labor Statistics last said about it:
> One reason for this is that American companies offer fewer vacation days. According to the Bureau of Labor Statistics, 76 percent of private industry workers (who make up 84.7 percent of all workers) receive paid vacation days. After one year of employment, these workers were granted 10 days of paid vacation, on average.
> This number grows modestly as years of tenure with an employer increase. In 2017, the average worker with five years of experience at a company was given 15 days of paid vacation and the average worker with 20 years of experience was given 20 paid vacation days.
Just in case it's not clear: when Swedes talk about 4 weeks of vacation, they also count the weekend that they have off anyway, so 15 days = 3 weeks and 20 days = 4 weeks
yep, same here. Apparently it's typical here in USA, for those who get paid vacation at all (ie not restaurant staff and other low-wage hourly workers), to get 4 weeks after... 20 years of tenure at the same employer!
I will note that here in Romania for example, maternal leave is commonly 6 months-1 year (legally you can request up to 2 years), and paternal leave is at least 4-6 weeks, at least for middle class jobs.
Usually possible, with stipulations such as, you are out of regular vacation, many mgmt approvals, and probably some kind of “risk assessment” saying the project you’re on will be fine without you… its rare and not usually part of the “culture” to do so, but ive seen it happen.
The only time I've ever seen that first-hand was a guy who was involved in a non-profit that did disaster relief in Haiti and was gone for a month and a half.
Sounds great to me, as a Swede I hate being forced to take 4 weeks consecutive vacation. Would love to instead have 2-4 weeks spread out through the year. It's a stressful burden for me as a single person who's life naturally revolves mostly around their job, to suddenly invent a new life for myself for 5 weeks. I just end up bored and restless, wasting time, even if I travel 4 weeks is just too long. 1 week at a time is perfect really
Edit: And you have to take those weeks in the summer as well, you can't take 4 consecutive weeks in fall, winter or spring. So it's really inflexible. If you want to go hunting in fall, and skiing in March, tough luck, in Sweden everyone "needs" to take all their vacation in the summer for some unknown reason.
It’s the same in Finland. I’d hazard a guess that the reason behind it is that without it being forced people would be subtly pressed not to have the vacation.
In USA/Canada boss just going to call you during your vacations, because it's an emergency. So those task won't be understood and will continue to be seen as an exception.
From a fraud perspective, the point is that a consecutive 2 week vacation is enough that someone probably needs to take over your financial roles, so it would become obvious to the newcomer that you're committing fraud. An example of a type of fraud this could expose (or prevent happening in the first place) is "kiting", where you steal a payment from a customer, then apply over customer's payments to that payment, then apply other customer's payments to those misallocated payments, etc. Doing this prevents those who look at the accounts receivable summary from seeing that payments from Company X have been outstanding for a significant amount of time. Making sure someone takes a vacation makes this much harder to do since someone else will be looking closely at your work in the meantime.
I left tech when the dotcom bubble collapsed because obviously tech was over. Then came crawling back during the legal recession following the housing collapse. Mistakes were made.
Even at Amazon, when I left work I left behind all ways to communicate and my bosses respected that. Nearly a decade there and never did a boss try to reach me on vacation.
Also been my experience doing technology in big banks for 20 years. Barring extraordinary circumstances, if you need to call someone who's on vacation that's an indicator of a broken process and unnecessary operational risk.
(I've also seen a person who believed he was indispensable and thought because of that he could behave like an asshole. Very satisfying to see him get fired.)
Why do they keep the tasks secret? I bet because they know they are important but they also know they are fragile in the sense no one else will care enough to do those things. And worse someone else will say don’t do them because they waste time.
During a summer not long ago, a client called with an urgent issue. Some critical messages were not being passed on, leading to a full stop on their end.
However I couldn't make heads or tails of what systems this was or how it related to us.
After a bit of back and forth, the customer said "ask Glenn, he always fixes this for us".
Glenn was of course on vacation and didn't respond to messages...
A few hours pass while we make zero progress on this issue and customer increasingly annoyed, when Glenn finally responds to my messages.
Turns out that on a different server from where our software is installed, there was a Java-based message broker service running that needed occasional kick in the pants (IIRC stopping, deleting a file, restarting). We're not a Java shop, we don't use message brokers.
For years, Glenn had been keeping this service running and updated for our customer, as the customer didn't have any internal resources that was capable of handling this. And nobody else at our shop knew this. Why? Well, he just did what had to be done and moved on to the next issue at hand, as he always did...
Not everything that someone does has to be "a deliverable" on some project manager's radar. There's joy in doing work that's satisfying and beneficial but that doesn't necessarily align with the immediate stated objectives of the manager, project or organization.
There's a word for this kind of thing: "covert agency".
I don't remember where I first saw it, but looking back at my own career, being able to exercise some covert agency made miserable jobs worthwhile in some ways.
> Now imagine declaring an extended period of time where, in the interest of reducing risk(!), no new code is deployed, and Chaos Monkey is disabled.
I remember reading about similar experiment in biology:
If you have a petri dish full of bacteria, and you gradually introduce antibiotics, not only will your bacteria population grow resistant to the antibiotics over time, but there metabolism even might become dependent on it.
You see, biology is a system as least as complicated as any computing infrastructure we have.
(Compare also most organisms these days being reliant on oxygen. Or how humans lost the ability to make vitamin C in their bodies.)
> You see, biology is a system as least as complicated as any computing infrastructure we have.
One of my first bosses describes the opposite relationship. We tend to think technology is very rationalized. He once told me that we should think about the system we were building as an organism. Build it, but also play with it and get to know it, because it will have its own personality.
Here is a video on the topic of bacteria in an experiment evolving to use a second source of food: https://www.youtube.com/watch?v=w4sLAQvEH-M. I'm not sure if they depend on it being present though.
This is why documentation matters, and why it can’t just live in a drawer.
Who would set up a virtual machine backup system that offers no alerts when it recycles something? I mean, I might do it myself, so I sort of know, but you shouldn’t do it.
Similarly I’m currently having the “pleasure” of documenting every function I’m responsible for as I’m leaving the public sector for the private sector, and man oh man, would the excel sheet of functions and directions to where they happen have saved myself from hundreds of hours “searching” or “refreshing” my knowledge on things when I had to revisit them after years of not working on them.
It’s also why MBA business process types are valuable in any enterprise sized organisation. Because they tend to both discover and document these hidden functions when they do their work. Of course, 9 out of 10 times nothing comes of their discoveries, because managers are notoriously poor at long term benefits realisation.
I've been thinking about this in regards to things like Idris vs TLA+ as well. I prefer using Idris than TLA+ simply because Idris is continually with you, type checking and part of the overall design. TLA+ can be used to verify a design, but then it is not really a living, breathing part of the end product. Having deduction built into the safety of the language is preferable to only verifying it externally then having it mutate under the next engineer.
It is the same with self-documenting code; if the function and variables names are clear with a sensible architecture in place, then it is easier to keep everything in sync and readable even as different engineers take over; rather than having external documentation which must be sought for, possibly discarded, and therefore not used as much as it ought.
One of the things I really like about Elixir is being able to put executable tests and examples inside the module documentation. If you run the tests and it fails, you know you need to update the examples in the documentation.
Documentation is also where things run, which resources they access and how and why they do it.
I’m not a big fan of executable testing or self-documenting code in smaller projects. Sure you solve a lot with good naming, but it’s usually the business logic that needs to be documented.
I can easily ready how a piece of Python or C# works, but I can’t easily tell why employees who only work less than 7 hours a week and are paid in advance are excluded from the function I’m looking at.
But I mean more generally. We selfhost some things, have others in azure, and have a lot of different databases and projects. It’s nice to have a reference sheet of which belong to each other.
A year (or two?) ago, I was looking at TLA+ when I stumbled across Coyote[1]. I haven't used Coyote, but it looks interesting in that it's embedded in the language itself, allowing you to "prove" that the algorithm still works when you change implementation details.
As far as I understand, Idris and dependent typing in general are only useable for small systems today - the largest systems successfully proved using dependent typing are the SEL4 kernel (which is ~10k lines of C) and the CompCert C compilers (which compiles a subset of the C language, and has only a minimalistic optimizer, if I remeber correctly). And these two have been 5-10 year PhD projects.
> their service might exhibit problems with long-lived instances that they never notice because Chaos Monkey tends to terminate instances before they hit those problems
Tangential to the main point of the article, but I've seen this exact failure before! I worked at a retailer that would reboot all point-of-sale devices daily, after stores closed. The day before biggest sales day of the year, the team decided to leave the devices online overnight - sometimes a device would fail to come back online after the reboot, so leaving them on would prevent that sort of issue.
Of course, one bug or another reared its head after 24 hours of uptime. We ended up rebooting all of the devices anyway.
New versions of our service got deployed once per week, barring delays and hotfixes. Sometimes we'd miss a release, and a version would stay in production for two weeks. Maybe there would be a code freeze and we'd miss a release, and a version would stay in production for three weeks.
We occasionally saw problems. Some cache would fill up over two weeks but wouldn't empty, or something like that. Something that you'd like to test, but which doesn't show up unless you run something in production for a month.
I advocated a simple solution: either check that the process runs for multiple weeks in testing, or restart the process every week in production. In other words, don't test it in production.
heh, my boss went on vacation. We were scrambling to do the technical part of their job. We never knew the boss did so much behind the scenes and we had no idea how difficult or time consuming it was.
When they came back, we had so much mad respect for the things they kept out of our day-to-day job.
It's not so much of a danger. Occasionally disrupting the system can reveal such weaknesses. Once you identify such "fragilities" you can compensate by adding redundancies. For example, add a second alarm clock at 6:30 and 7:00.
When it comes to programming, this is one of the good reasons why simplification matters. Systems that are complicated to the point where you can't understand how the intended functionality "arises" out of the interaction of its parts, is a system that is very difficult to debug and evolve.
Tangential rant, I recently learned it was illegal in some parts of North America that letting your child go to school by themselves is illegal. As in CPS gets involved if they get a call from a "concerned" citizen. For example in Ontario anyone under 16 has to be constantly under adult supervision. What the fuck, America? What the fuck, Canada?
Might just be anger-bait? I grew up in Washington (both in America and near Canada!) and I remember walking to school on my own at around eight years old or so I think, maybe a bit older (started somewhere in grades 1-6).
There's a notion in cybernetics of something like ‘the real org chart’ going all the way back to Stafford Beer's seminal work The Brain of the Firm. In any enterprise, org charts are fundamentally nominal, and you should always expect there to be gaps between that model and the way roles are actually structured in the organization.
Most my mates who are artists, designers, writers and scientists love working late at night... See? The argument can easily go the other way around. Builders, bus drivers, farmers, etc., all have to start early, and that's understandable.
Our field of work (I suppose most readers here are IT professionals) never had to normalize fluctuating schedules until relatively recently - when our profession has become fashionably decentralized.
And I think we need to be careful and make sure that harmful practices like having too many meetings or mandatory meetings at odd hours have not become a norm.
Programmers don't create things and solve problems at meetings. Most of the time, the meetings are there for other people in the business.
>I suspect this is failure mode is more common than we realize: there is a process inside a system, and over time the process comes to fulfill some unintended, ancillary functional role, and there are people who participate in this process that aren’t even aware of this function.
Ah, Hyrum's Law showing up in an unexpected context, yet again.
https://www.hyrumslaw.com/