I actually don't see any issue that isn't simply a matter of money.
Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company.
Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
Throwaway purely so that I can comment. I've had this happen before.
Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda.
Fast forward 6 months.. a contractual obligation existed to release the software. They still hadn't successfully built (yup autoconf, make and friends), let alone know where to build it. I got the call you always dream about, 30 days prior to the release being due.
My boss asked me to do him a favour - I reminded him that I quit due to a lack of promotion and raises. He asked what I want, and I asked for 125K, which clearly he scoffed at. Then I reminded him that it was a 10M deal they'd signed, because we had an all hands. The CEO got involved and screamed at me for 'torpedoing the company'. I walked out and calmly informed them both along the way the price was now 250K, non negotiable.
I ignored every single mail (and legal letter) sent me way. With 10 days left, they agreed. I had the company deposit a cheque in escrow with lawyers of my choosing. We signed contracts, and I did my thing. Four days of work. It was well worth it.
Dougie, if you're reading this I hope your cringing.
Hell. Yes. That's absolutely beautiful, and ethically completely correct.
I have bosses at previous companies that I'd help for free today because they were good to me and we parted on friendly terms. One boss helped me move my house once, and I'm not about to charge him to fix some random issue.
I have other bosses whom I'd require a cleared wire transfer of money before I'd lift a pinky to help them, because they were not nice people and I don't want to associate with them or assist them with anything. If someone in this camp approached me, I'd have a story just like yours afterward.
This is actually not a very rare situation. Unfortunately many of us don't really have the smarts or courage (overcoming guilt) to negotiate a price like the OP did.
Yea, this is one of those "I want to believe" stories! It's practically engineer-porn. Everyone who has quit a shitty employer whose technology was a house of cards resting on the point of a needle is cheering OP on. We all have that fantasy that the house will finally fall over, the company will have its comeuppance, and we're the one person in the world with the knowledge to fix it.
Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.
That an overworked employee who had already left the company would stick up for himself? Or that a company would go to such efforts to try to manipulate a former employee?
I realize my tone doesn't hide my feelings, but even if we disagree I'd still be curious to hear more.
IMO your edit didn't help and you're not getting a lot of sympathy. The point wasn't that the employee is psychopathic as you say, but that management was incompetent by not ensuring a transfer of knowledge and validating before the employee left. You as the buinsess owner should plan for people to not be available and no one person can derail a $10M contract. This is also called the "bus factor".
Also yelling at someone who is the only person who can solve your problem is not a good negotiating strategy, especially when it reminds them of why they left.
Walking away from a toxic situation is not "psychopathic." It's common sense and if we all did it more then the psychopaths who expect us to stick around through it all would finally learn how to treat other people with respect. It's a job, not a marriage.
I mean... what could the other side of the story actually be?
"We tried to hire back an employee to build something that we didn't train or try and figure out ourselves and a deadline forced us to pay a large lump sum to that employee".
No matter how you slice it, that's really shitty planning on the businesses side. They should have spent the time and money learning how to maintain their shit. By not doing it, they paid a huge price.
The other side of the story is it's fake, fake, fake. But it's a nice story.
Every negotiation is about the best alternative the parties have, and it's generally harder in real life than in fantasy because you underestimate the alternatives to a deal.
Applying maximum leverage has a way of creating "out of the box" alternatives. And vindictiveness.
I imagine if it came down to it they could try to pin it on him and say he didn't provide documentation he said he did, and that's why this entire 10M project now can't be delivered.
Luckily for him they decided it's easier to just pay the fee.
Even if he was a shitty employee that doesn't obligate him to work for a price favorable to the company. Hell, they could have fired him for not documenting shit.
Because here's the thing, even with how weak US labor laws are, one thing they don't allow is for a company to come back and sue your for incompetence. So long as they didn't go out of their way to sabotage things, they are free and clear.
A bit late to the party, but even when threatening with insolvency-via-lawsuit, this would probably not work for them:
- if he can prove the threat, the company will be in an even worse position
- given the rather lacking precedent (as pointed out by the sister comment), there's a good chance the suit would be thrown out pre-court and therefore be rather cheap
- given that the OP seems to be a valued engineer, it is reasonable to assume that he has the f-you-money to fight this out
- even if won, the drawn out battle would take far too long to fix the software in time and, given that they then lost a 10 million contract, they lack the motivation (and maybe money) to continue
- lastly, even if they won after loosing the contract, they'd probably not be able to extract much in damage payments
Maybe this is my naivety showing, but to my knowledge it's not a common occurrence for companies to sue employees for refusing to work for them, especially if (as the OP notes elsewhere in the thread) there's no actual contractual obligation.
That won't work in most reasonable countries. A company that is no longer employs someone cannot make that person work. In the situations similar to the ones that are being described, the time is not on the side of the company - they have a dead line, a former employees does not.
I don't know why you'd make such an assumption about me, please expand.
I think in general all of my comments on this thread have been horribly misunderstood - I was just trying to say that if the company was scummy, instead of paying him the 250k fee, they probably could have sued him instead. I'm not saying that should have done, I'm not saying it would have been the right thing to do, that he deserved it.....literally nothing of the sort. Just pointing out that the company could have made an argument against him in court. As many other people pointed out to me, it would have almost certainly failed. But I don't understand how you can deduct where moral considerations are on my priority list from this?
This seems absolutely unrealistic to me. Can you find a precedent? A single precedent of the company suing an engineer (individual contributor, not a manager) for quality of work after he left? I think if they tried they would have been laughed out of court; and out of all their future contracts, too.
sure it does. the company would have to prove malicious intent beyond a reasonable doubt and not leaving adequate documentation absolutely does not meet that requirement.
I think enough of us have been on the same side of this story that we feel fairly comfortable taking it at face value, even if we're only applying it to said personal experience in a wishful fashion.
Yeah, HN at it's finest I suppose. He's like you, he says what you want to hear, therefore you believe him. No need to consider the possibility that maybe there might actually be another point-of-view.
The point of view of managers and business owners is well known. Work is not a new concept. You get what you pay for. Nothing is owed to you. Work? Tough luck, you have to pay for it too. On a free market, no less. I don't understand why business owners think the rules are different for them than the rest.
This is not extortion. The company made choices that left significant leverage in the hands of another business. That business correctly perceived the value of the services it could render and priced them accordingly.
This is exactly the free market at work regarding labor. Have a problem with the free market if you like, many of us do but do not accuse this poster of criminal action equivalent to a protection racket.
The market answer to that is, the seller underpriced their services to begin with, regardless of the personal motivation for increasing the price.
To clarify, the market would likely not allow this specific work to be sold at this price normally. But the buyer wasn't just buying specific work they were buying specific work to be completed by a specific date. The work in isolation does not have this value but the work completed by a certain day does. Companies use this rationale all the time charging more for expedited services.
It's especially common in construction and manufacturing where costs double or even triple when expedited delivery is required.
"Everything is worth what its purchaser will pay for it." That includes labor. The business was free to try and renegotiate their contract with their customer. They were free to delay the release and offer a discount. They were free to turn down the 125k and the 250k payment on principle and deal with the consequences. They found the most economical solution that met their needs was to pay $250,000 to have the work done on time. Was this poster engaging in rent seeking behavior? Absolutely. But that is a consequence of the market and one companies exploit to their own benefit all the time.
> They found the most economical solution that met their needs was to pay $250,000 to have the work done on time.
I'd be inclined to believe that if there were some other providers but as it is told working with OP was the only solution. And OP's first offer was 125 000. Two screams later it's 250 000 ? It has nothing to do with economical theory or free market interpretation.
First, he owed no-one anything. He could set the price to 1 billion for all he cared. His former employees could take it or leave it.
Second, you speak as if his claims were irrational. The way I see it, however, you would set for 125k (probably much less judging from what you write) while he, the sentimental fool, scooped up double the money. Therefore, your decisions, which you personally claim are rational, would have left you with half the money the "irrational" player would get. Seems to me, the "irrational" is actually the rational one, as he netted more profits. Economy and market are not math games in a vacuum, and players are not formless ideas.
> Second, you speak as if his claims were irrational. The way I see it, however, you would set for 125k (probably much less judging from what you write) while he, the sentimental fool, scooped up double the money. Therefore, your decisions, which you personally claim are rational,
The way I see it you are putting words in my mouth and I am definitely not interested in having a discussion like that.
I never insinuated OP was a sentimental fool or wrong to raise the price. I am calling bullshit on the free market and econ 101 as an explanation of why the price got doubled and call it like it is: it's payback from OP because they were dicks to him first, everything else is pedantry and useless justifications.
edit: if anything, OP could be considered a fool since he obviously gave an under the market quote as his first proposal. Had he started with 250K from the beginning he would have gotten more since there likely would have been screams anyway and he would have risen the price all the same.
No one said the free market is "why" the price got doubled. Emo drama ended up being the mechanism of price discovery here. But the market accepted this price. It was not a swindle. It was free trade.
And yes. I completely agree with your edit and said as much elsewhere. Their acceptance of 250K showed that the OP under-priced their services at 125K.
And yes. Free markets produce monopolies. Especially when time is important. As I said above, the critical good here was not the delivery but the delivery within a certain time.
I find this explanation (from top comment) more in line with how the world spins:
> If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
So, “being a dick” has more to do with price raising than any free market justification (which I am okay with, just don't try to convince me it's the only thing at play. The offer would have been 250k from the beginning then. Or 500K to get 250K, whatever.).
This is such a silly and mundane bit of pedantry you're harping on here. Of course there are emotional factors in price discovery-- it happens all the time, and is a completely natural and rational part of the free market.
Any small or independent contractor in any business will eventually have a blacklist, or a list of troublesome customers who get charged extra. Most of my contractors charge me less because they like me (and I pay on time). My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole; the contractor has decided that it's only worth dealing with the man's unpleasantness if there's more money to be made. Apparently the man has been blacklisted by other HVAC companies, so he might be in a tight spot, but whatever the reason, he ends up paying a lot more money for the same service.
This is a completely market-based mechanism: the troublesome customer has limited options (in this case, maybe only one option), and is thus obviously disadvantaged in negotiating price. Regardless of the myriad motivations involved in price discovery, it all leads to the same place: if the price wasn't worth the service the customer was to receive, they would not have paid it.
Honestly, this isn't mysterious or profound. It's economics 101.
Not even that. It's payback, that's all it is. There's no need to call for a more elaborate explanation for what was at play here. Not that I disagree with it. It's humane nature.
> My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole;
Like you say. No need to even know about market dynamics to (wisely) choose that option. Because the other guy is an asshole.
This is literally the definition of price discovery in a free market. If the price was not worth the service the company was to receive, they would not have paid it.
The fact that they did pay it clearly indicates that the first price was far below what the "market" would bear.
"The market" has nothing to do with that because one single trade isn't a market; if you and I exchange 1g of gold for $1000 that doesn't say "gold is fairly priced at $1000/gram". If many people exchange gold for money, each with different information and desires, over time the gold will go to the people who pay more, sellers will generally trend their sale price down because higher prices won't get buyers, buyers will trend their buying price up because lower offers won't get accepted, and a range of "price gold is being traded at" values emerge, the spread of those values highest to lowest narrows and hones in on, and tracks a dynamic "price of gold" where it's "fairly traded" - not fair in some Deity decreed way, but fair in that if you trade at close to that price, you aren't buying unusually high or selling unusually low, you're getting no better or worse deal than anyone else, so it's not unfair in your favour or unfair against you, so if it's not unfair then it must be fair - you couldn't get a significantly better trade with anyone else.
Anything distorting those conditions - e.g. there's only one seller, only one buyer, a time constraint, a skill constraint, an information disadvantage, will distort the pricing as well. One trade isn't enough information for a market to exist or to settle on a fair price. I agree there that the market has nothing to do with it.
But I disagree with your implication of "the fair price is the one paid" is somehow unfair in an ethical sense. (I also think OP is in no way obliged to be "fair" in any ethical sense to the company, but that's another matter). The company got a $10M contract for $250k, versus losing the contract, is that unfair? If that really was the pivotal factor between losing the contract and not, the company ought to benefit by spending right up to the point where the contract becomes a money loser, to not lose it. Would it be unfair for the company to get an $8M contract by paying $2M? Or a $2M contract by paying $8M? Is OP obliged to help the company make as much profit on the contract as they can, by charging them less for his services?
> Anything distorting those conditions - e.g. there's only one seller, only one buyer, a time constraint, a skill constraint, an information disadvantage, will distort the pricing as well. One trade isn't enough information for a market to exist or to settle on a fair price. I agree there that the market has nothing to do with it.
This is non-sense. Those conditions specifically define a market and make it distinct from any other. To say a market with any of these constraints is "distorted" is to say all markets are distorted and render the term meaningless.
Also, I never said this one trade signified a market. My point was that this is a consequence of free markets. You can make any single trade "irrelevant" by choosing arbitrary criteria by which to separate it from its market. The company could have certainly hired another engineer in hopes they could complete the work in time while running the risk of blowing the deadline. But it was freely and fairly worth at least 250k to them to guarantee the job could be done correctly and on time.
This kind of thing is such a common part of the free market, that there is a whole aspect of contract law that pertains to ensuring companies don't get into this position. It's why contracting completed work is so important. Leaving things partly done leaves you vulnerable to others eating right up to the tiniest margin of your profit.
Time was a factor. Negotiations in such a situation waste time. Doubling the price associated the cost of wasted time with additional negotiation in a tangible way. In clarifying the situation, thread parent was really helping that former employer. Had that clarification not occurred, the contract might might have been signed too late.
The company can avoid this if they only care to maintain the documentation / wiki left behind by said person
When they failed to do even just basic proper operations, it's their own fault.
I'm a business owner and a consultant, and I too charge more for these kind of asshole clients, because of the extra stress.
But nowadays I just try to avoid them, it's just too damaging for your own sanity. Their kind of stupidity, self-entitlement, and ignorance is on a completely different level.
Citation? By that light, a DVD containing defamatory speech would be libelous, but that's generally not how courts see it.
From the (highly-influential) U.S. Court of Appeals for the Second Circuit, summarizing New York law (which is quite typical): "Defamation, consisting of the twin torts of libel and slander, is the invasion of the interest in a reputation and good name. Generally, spoken defamatory words are slander; written defamatory words are libel. Libel is a method of defamation expressed in writing or print." [0]
There is also "calumny", which is the action of slandering someone (and I assume also making writing libellous statements. After writing that parenthetical, I just googled, and libel is also a verb. So I guess I can say libelling someone)
Business is business. The right to fire people, deny them raises and form binding contracts in regards to their work comes at a price. GP did not act malicious, rather, they acted in the same self-interest most bosses, CEOs and companies act while trying to stand on the moral high ground.
Coding standards, documentation requirements, meetings and code reviews exist for a reason, too. It's not like companies are defenseless.
The fact that you'd refer to this as extortion makes me sick. The management/capital class loves the free market except for when it applies to labor, then suddenly it's extortion when supply and demand don't work in your favor.
The employee documented everything, but said documentation was not backed up by the company. Furthermore, the company did not train or hire an adequate replacement for said employee for _six months_.
This is a gross failure of leadership at the company, and the former employee has every right to negotiate compensation for the inconvenience of having to work for them again.
I owned a small company. After selling it, and giving 2+ more years of my life for the new owners to figure out how to replace me, they didn’t. On my last day I told them that if they ever needed anything my rate is $5,000/day, non negotiable. It’s kept them at bay for now. I have no ethical qualms with my decision. It’s business.
I see and understand your sentiment but, in my rather insignificant few years in the workforce - carrots, dipped in vanilla chocolate and honey, generally tend to work better than the stick.
The company could have apologised, and do a lot more ego stroking before asking a favour which they can pay for and maybe throw in an all expenses paid weekend with the SO as a thank you. Not only will you get goodwill but future problems are going to be significantly cheaper to solve. Even if you have a bad break up - as a business owner you should understand the value of mending bridges. It's cheaper than losing sleep at night and losing your steak dinner to a bad stomach.
"Lack of planning on your part does not constitute an emergency on my par--wait did you say $250,000?"
If your company were in the hands of incompetents, but still kept going in spite of the incompetents because it had gotten its foot wedged in the door or whatever, and then fell apart in ways you could only scorn, how would you view that sort of situation?
Sounds like your best next step is to become an exemple of software/infrastructure best practices - get everything in the best backup/recovery solution you can, document every detail any time an issue comes up, and above all treat your people like the vaaluable assets they can be.
You may be doing any of that already, since I don't know anything about you except your one comment.
But the answer to your fear of exploitation is avoid making people hate you/your environment enough to want to protect themselves from it.
Hopefully you wouldn't mismanage your own company and treat your employees like shit. Probably you are a better person than that, and can avoid this ever happening to you.
You're terrified that you might treat your employees so badly that you would need to pay a lot to resolve a situation? Seems like that's entirely within your control.
You forget it's all part of business. If you hired an external contractor you would not say a word for demanding their money. And the price of the contractor grows by a lot relative to how much and how soon you need their work. Remember, you need their help, not the other way around. A contractor on bad terms with you would not help at all. How is a former employee any different? Do you think you become family after paying for their work?
If you do not have the money to pa for someone's services you cannot complain. You say you have a company? Have you ever let a client get your products by paying less than their value or nothing at all? Work itself is a product too.
Then treat your employees better and adhere to best practices and you won't be subject to such "extortion". Once we have elft yoru company, especailly if it was because we were treated poorly, we are under NO compulsion to help you for free. If the idea of paying 1.25% of the value of a deal to make sure the deal happens seems excessive, you're greedy, and that's why his price went to 2.5%.
The idea of /you/ mismanaging your company in such a "blasé and psychopathic fashion" is more what should terrify you. Remember, /you/ manage a small company, not the person you hire. If you end up in such a mismanaged position, the manager who let it get there is you, not the employee.
I think this is just flamebait. Uses emotionally charged words without any other substance. Both the comment and responses to it (including this one) add nothing to the discussion.
He left detailed instructions how to release the software. The company managed to screw up so bad they lost the manual he made sure to write on their internal wiki.
I know this sounds like an empowering story, and it may even be true, but don't actually follow this example. If you're at the point where they'd rather send you legal threats than pay you, even escrow is not going to help you much. The escrow process may force them to pay, but there is no guarantee you'll be able to keep that money if they really decide to go after you.
People like that would much rather spend $2.5M on courts and lawyers than let a worker walk home with 250k, or even 25k. Agreeing to a deal like this is an incredible minefield that may blow up on you even years later.
The letters stated that I had a contractual obligation to assist. But I noticed at the top they were stamped 'without prejudice'. I asked my friend what that meant, and he said that technically it's not a legal letter - so I ignored them.
For a while, I made it game to light the various communications on fire in interesting ways, but it got boring.
Look up what "escrow" is. When 1/4 million dollars is on the line, you don't take the company's word that they'll pay you afterward, and they don't take your word for it that you'll actually deliver the service.
The idea is that you want a trusted third party because either of the two adversarial parties could screw the other over.
The company could say: we’re going to pay you 125 and you could sue us and spend 60k and two years or just settle for that, after he’s already done the work.
The guy could demand the payment directly to his account and then not deliver, and the firm would be in the same situation.
By depositing the funds in escrow, they’re making sure that the money is there and in the hands of people whose only incentive is to give it to the correct party upon fulfillment of the contract.
Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.
The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.
Here in america farmers hire illegals to harvest crops, claiming that no american would do the work. This has always struck me as a really terrible excuse. I'm sorry, you can't simply break the law because you'd have to pay an american $20 / hr to do the work. Sure, it means that your tomatoes are more expensive, but also, now there's a hell of an incentive to apply automation to the problem yes? If you don't like the law, maybe pressure your representatives to reform immigration instead of just hoping everyone turns a blind eye to your exploitation.
Here in america farmers hire illegals to harvest crops, claiming that no american would do the work
This is a political talking point that has been repeated so often that people believe it is true.
During the recession of 2008, there were stores on local TV news in many places showing unemployed Americans waiting in long lines for the opportunity do do those jobs that the politicians kept saying no American wanted to do for those wages. Meatpacking plants, especially, had more than enough Americans to choose from who were willing to work for low wages.
Because these things happened in "flyover" states, they were only very rarely shown on television on the coasts, and so the meme stuck.
A similar oft-repeated lie is that American farmers are all a bunch of Republican hillbillies who hate brown immigrants.
Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
The more you actually go places and do things and meet people and talk to them, the more you realize that the words coming out of the political organizations on the coasts are mostly for their own benefit, and do not reflect reality.
I don't think your points really contradict your first statement. First, people tend to drop their standards on the type of work they're willing to do and pay they'll accept when they're desperate. In 2008 a lot of people needed a job. Any job was better than no job. I'm not surprised that jobs that would normally be unpopular had lines of people applying at that time.
Your second point about farmers trying to increase legal immigration in 2010 clearly indicates most Americans won't regularly do this work for the wages being offered. Why else would farmers be looking to hire immigrant workers just 2 years after locals were lining up to get these jobs?
While it would be nice if there was a way to get enough immigrants working legally to fill those jobs that doesn't seem to have happened. Short of another large economic downturn I don't see Americans working those jobs either. That leaves people who are working in the US illegally to fill the gap.
> A similar oft-repeated lie is that American farmers are all a bunch of Republican hillbillies who hate brown immigrants.
> Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
Farmers are, for the most part, single issue voters: Whoever has the best plan to support agriculture and small businesses gets their vote.
>Because these things happened in "flyover" states, they were only very rarely shown on television on the coasts, and so the meme stuck.
How do you determine from TV that the people lining up for meatpacking jobs were not undocumented immigrants?
Like, it is generally accepted there aren't any in "flyover states" or something?
I never really thought of it until this moment, but you know how within 100 miles of any border, your constitutional rights are somewhat compromised so DHS can operate? That ought to mean a pretty substantial incentive for people who are undocumented to migrate inward. So I would bet they do.
My somewhat educated guess is that these farmers in the PNW don't share the same politics as farmers in the rural south or midwest.
That and Republicans have grown way more hostile to immigration since 2010, so it's an interesting anecdote but not sure how well it translates across the US, where polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not.
"When you say it's an "educated" guess, educated how? It only seems to reflect the usual bigotry that comes out of the coasts."
I lived first 28 years of my life in the rural south and now live in the midwest.
My entire family and most of my friends growing up are the people you claim I'm bigoted against...
"If you're going to make a statement that sweeping, you really should back it up with credible sources."
The fact that farmers are generally far more conservative and republican than the median voter or average American isn't exactly some shocking new insight.
You seem to have never spent much time around these folks or have done much research on them if this comes as a surprise to you.
A couple of years ago, a serial offender food producer was raided, multiple times. 600+ undocumented workers were found, and all spoke of management knowing their status.
The workers were detained, deported, and so on.
After several raids, not so much as a misdemeanor was levied against a single person, nor a fine to the company in general.
The point is to punish and cause pain to the immigrants. Regardless of the original justification of a system, the point of a system is what it does or will do. That is what this system does and will do unless changed.
I don't know anything about it, but from the Des Moines Register, it seems like a lot of people felt there was a miscarriage of justice:
"There was false testimony and willful manipulation, and that makes this a shocking case of prosecutorial misconduct."
"An unprecedented letter, recently sent to Kevin Techau, U.S. attorney for the Northern District of Iowa, signed by 107 former high level Department of Justice officials, FBI directors, federal judges and prestigious law professors attests to the extraordinary prosecutorial misconduct which exacted the most possible punitive sentence for Rubashkin"
Trump commuted Sholom's sentence in 2017, apparently.
I'd suggest a different phrase than "prominent Jews"; the family are "ultra-Orthodox Jews of the Lubavitcher hasidic movement" not representatives of some sort of Jewish privilege in general.
About half of US hired farmworkers are illegal immigrants, says the U.S. Department of Agriculture.[1] It was only 12% in 1991, but passed 50% by 2000.
That's easier said than done. The problem is that in order to reform immigration you have to do one of two things:
1. Start treating people fairly, with the result being higher food prices, which will make some of your constituents very unhhappy
2. Codify the current class structure into law so that you can continue to exploit cheap labor legally. The optics on that are really bad.
Exploiting people off the books is the path of least resistance for everyone, including the workers (manifestly so, or they wouldn't risk life and limb to be here).
Of course it isn't that simple. If your farm's tomatoes are more expensive than your neighbors' tomatoes, then nobody will buy them. You'll quickly go bankrupt.
It is also difficult because of the two tiers of labor. A farmer will try increasing wages by $1/hour, and still struggle to find enough workers. Wages would probably have to be doubled before new workers—Americans—would start to enter the industry.
Or, you know, you let unprofitable endeavors be substituted by profitable ones, and you just import food from different places where it is profitable. Just make sure not to wage war with them all.
That's always an alternative to subsidies and tariffs, but it's not as politically useful.
"gave up" sounds like an unnecessarily ominous take on "delegate". But what's your exact point? Do you believe this is an irremediable situation or that know-how has been lost to time in the age of information?
I'm assuming you're implying a conversation about China and their manufacturing capabilities. But worst case scenario short of nuclear war, assuming US-China burns bridges and somehow for some reason now US is fully unable to import Chinese manufactured goods; how does this eliminate the US and all other countries from investing in an area that has now become more profitable? You could argue that prices would increase, but then again you aren't owed cheap manufacturing, and subsidies and tariffs are paid by either the taxpayer or the consumer (which tends to have a knock-off effect on the rest of economy, anyway). There was never a free lunch.
It's also in general how we end up with monopolies. Well-funded companies operating at a loss until they're the only noteworthy players ain't exactly unheard of.
If farmers paid enough to get Americans to do seasonal farm labor their produce would cost twice as much as central/south American produce and they would immediately go out of business as everyone buys cheap imported fruit.
People get really pissed off when food gets more expensive. Politicians realize that and will therefore never pass legislation that would cause such drastic price increases on food.
If they enforced the laws already on the books, they'd have to pass new laws to fix things. I see that as a net positive. If you selectively enforce against illegal immigrants, well, that's very much a 'rules for thee, not for me' situation isn't it?
> complain that young adults are refusing to relocate for $5 an hour hard labour jobs.
USD $5/hour is what I earned as a teenager working the fields by hand (picking corn, beans, and fruit for sale same-day at the farmers small stands throughout the area) in the early 1990s... That's insane. Is that USD $5 or AD?
I think people will one day demand higher pay for desk jobs over outdoor work. Looking back on my early life, the hard outdoor work was incredibly valuable to me. Desk jobs sometimes were soul-sucking, depression inducing. Not always and my dev work now is desk work that doesn't suck.
Now, outdoor work with a high injury rate is a different story.
Sounds like exactly the same problem British farmers are having post-Brexit with things like fruit-picking, they're scratching their heads wondering why British workers aren't keen on having half their wages taken for "rent" to live six people to a crappy caravan on-site and do backbreaking labour day in day out. Their only choices are to improve their wages, conditions, both, or go bankrupt.
Farming is low margin. Farmers don’t get to set the price of their products, and they aren’t handed billions in capital to burn through or 1000x multiples on their profit in the public markets. When most businesses say they can’t hire someone and the reason is because they don’t pay enough, the underlying cause of their low pay is usually because that’s what the company can afford.
Sure, farmers could pay $10/hr, but unless your Doritos were $10 a bag, they’d lose money.
To me it always seemed a bit similar to justifications used for the "peculiar institution" - the system can't exist without slavery, therefore slavery must exist. The benefits of cheaper labor was well appreciated; Carnegie called immigration "the golden stream which flows into the country every year."[1]
Being paid $5/hour farming in Australia vs $0.50/hour assembling shoes in Malaysia. Wow, such exploitation.
"Exploitation" is a relative term, some people would rather be exploited in the countryside of a first-world nation rather than the countryside of their homeland.
> Being paid $5/hour farming in Australia vs $0.50/hour assembling shoes in Malaysia. Wow, such exploitation.
The Australian national minimum wage is $19.84/hr.
Getting paid about a quarter of minimum wage is very clearly exploitation. There may be other benefits for actually working in Australia, but that's not really relevant here when you're trying to attract native Australians to work for you.
In particular, when the wage is low enough you have slavery in all but name. People working for your shitty pay staying in your shitty dormitory eating your shitty food because they can't afford not. The end result people leaving when visas run out with more debt, and headlines about people forced to pick fruit in bikinis and allegations of sexual assault, because that happens when some people gain that sort of power.
Which is more exploitative, working for $5/hour in agriculture in a wealthy country, 50¢/hour in an urban factory in a poor country, or $1/day in a rural peasant village somewhere or having no work at all?
Doritos aren't going to get more expensive, because like all the real staple crops, corn is completely mechanized, and you have one guy driving an air-conditioned tractor or combine, following GPS-plotted courses.
Not sure that’s the best example to use, since food costs are a trivial fraction of the retail price of a bag of Doritos, and harvesting labor is a trivial fraction of the price of food.
Edit: that was just based on my general memory of non-perishable food economics. But I googled it and found this (sorry, quora) from someone who would know, a 2 oz bag of potato chips costs 15 cents to produce and bring to (retail) market. Of that, a fraction is potato prices.
The reason I buy expensive eggs, milk and cheese is that I know that the supermarkets fuck over the farmers. I never buy the supermarket brand, unless it's the last one on the shelf. The price difference is sometimes 1.5x but unless you're really "broke" broke, you don't notice the difference. It's a few cents a week.
While, like most things in life, it's not a guarantee, doing basic research about the farms and the cooperatives that produce the goods will provide some insight. Almost anything is better than a CAFO (Concentrated animal feeding operation).
I always buy the supermarket brand, even though I can easily afford more. It's not because I want to save a few pennies on each purchase; it's helping to drive prices down on all overpriced goods, helping out those really in need.
Same reason I avoid organic whenever possible. We don't need boutique food.
That doesn't even make sense in neoclassical economics. One person's effects on the market are necessarily tiny, but if buying more of Brand X has any effect it will be to drive up the price of Brand X. You're actually harming "those really in need", and you should feel bad about it.
This is offset by imported food. If 100% of American farmers colluded to fix prices, they'd have their lunch eaten(hah) by food importers. Obviously not everything is importable(eg rice), but there is more to the market than the American farmers.
You're right, there is more to the market but not how you describe it. Tariffs are available to any American cartel willing to buy enough politicians. Take a look at sugar, for example. [0]
People have to eat food. Food in aggregate has one of the most inelastic demand curves imaginable. If the price rises because the production costs increase, people will still buy food. They might buy different food, and you'll see a decrease in acreage given over to labor-intensive crops, but that's just the free market at work.
The thing is, if farmers in Australia increase wages and thus their product becomes more expensive, the product on supermarket shelves will not become more expensive. Rather, supermarkets will import food from another country. Food is, with few exceptions, a commodity. The producer does not set the price of the commodity, the market does.
I am telling this as an engineer working in the Canada's oil and gas sector. Since 2014, salaries in our sector have decreased and firms are struggling. "Just raise the price of oil" is not an actionable advice. The price of oil is not ours to set. It's the market's
It just shows most people working in Tech Industry, and specifically software ( or web ) development ( as oppose to Tech / software within Oil Industry or any other industry ) have absolutely zero idea on market or commodity trading.
I consider food security a national security issue. Produce is one of the few areas where I would consider a tarif justified. Every country should be able to feed it's citizens with the food grown there, or they risk hunger when cut off by war or natural disaster.
And countries do that, but for a few strategic ones, not everything. Canada produces lots of wheat and dairy and maple syrup. It even has a strategic maple syrup reserve. But it does not produce a lot of citrus fruits or pistachios or dates. It may be possible for US to produce everything within the country—it is a gigantic country and has just about every climate there is. But it would be an ecological disaster if countries without those attributes do the same. Think of all the heating required to produce warm-climate fruits in Canada.
You're absolutely right that perhaps not everything should be grown everywhere, but I strongly believe that a country should have a local source of all the calories and vitamins / minerals they need. In Canada, for example, maybe they encourage farmers to grow greens, brocolli or potatoes instead of citrus for vitamin c, but they absolutely need a source.
Apples and Oranges. If you include global supply, then you have to increase salaries "globally". We all (I assume) should know that demand/supply set the price.
But imported products don't fly right into the country. There is customs, and denying/tariffing them is a policy that countries are very actively using. So at the end of the day, it's a political decision.
Edit: On the other hand, if you are exporting stuff (oil for Canada), then you can't really do that for the external market. Might explain why the government will let some sectors run with illegals (to remain competitive); the alternative being their ultimate demise.
If you ban import X from a country, the country will likely ban import Y from your country in retaliation. Sometimes the net effect would still be in your favour, but often times it would not be. Do you know why American brands dominate the US truck market? It started with a tariff on American chicken by some European countries and it quickly snowballed into much more [0][1].
What tariff does is that it moves the price point at which the retailers switch to imports a bit higher. Tariff does not eliminate the ceiling. There still would be a ceiling on price, determined by international commodity price + X% tariff.
So the tariff needs to be high enough to be absolutely ruinous. See, for example, Canada's tariff on dairy. Since Canada runs a cartel on dairy producers, Canada does set the domestic price of milk at $4/gal. This is incompatible with the US subsidy model, where it prices out to around $1.50 a gallon -- so there's a 270% tariff on dairy imports. The great thing about numbers is there's always a bigger one.
Yes, you can do that. It's not something the farmers strapped for workers can do, but the government certainly can. It would not be particularly popular in most political climates, so passing such laws in democracies are always difficult. But as the Canada example shows, not impossible.
We reduce production, US, Russia, and Saudi Arabia would be more than happy to fill the gap. Actually Saudi Arabia prefers prices that are lower than the strike point for non-traditional oil, so it can get rid of competition from likes of Canada. How should we convince Russia and Saudi Arabia to take a hit to their exports in order to help us out, when we cannot even convince our biggest ally, the US?
> The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
This. There is no amount of money anyone sane can be given to stay in Romania. Most recently i’ve read they (hospitals) were tying covid patients to their beds.
Couldn't help but comment on this as some of things i discovered are shocking for an eu member country (or any country for that matter), such as EU leading human trafficking, police corruption, collapsing health care, and the list goes on.
But sure, the simple solution is to give people money, not the constant abuse taking place at work and outside that makes people want to flee that place.
That's every poor country in existence. Not everyone has the same mindset as you, however. Provided with basic incentives, many, if not most, would not want to leave family, friends, common culture, etc, just for a wealthy life - they just want basic stability.
Companies paying more (offering more at least) would begin to slow this domino effect, which raises standards of living and provides more funding for the government to provide services and ultimately pressure for increases in quality of life by the voting public. Everyone wins.
Ultimately not everyone can emigrate - it's an impossibility. The brain drain will continue to be real, so it's incumbent upon these companies hiring to raise their wages and retain talent.
That was my point. The simplest solutions are sometimes the most effective, as well. Incentivizing brain power to remain the country via pay increases seems like the most basic approach one can take, considering the GP was talking about Romanian studios complaining they can't find/hold talent and they are competing with the rest of the EU.
Step in as a multi-national company, open a software engineering office, pay well over "local market value" and you get good software engineers, at a lower price than moving them elsewhere. They, in turn, get a better quality of life, don't have to emigrate (which for some is a bonus, and for some the opposite (a malus?)) and relatively speaking have more spending power than they would have had, if they moved.
It is really not as easy as go "this is the exchange rate" and simply use that to compare pay in different countries.
Well, the problem is this: it might not make economic sense for those specific companies to hire people at a higher rate.
Companies aren't going to say "I guess we're inefficient and should just die." They're going to fight to survive and thrive, which is what you're seeing.
Then again, some companies ARE just cheap. You never know.
I was at a hotel with crappy service and asked about the problems with house keeping. The manager told me everyone in town (Traverse City MI) was trying to bring in foreigners to do housekeeping because they couldn't get local labor.
They said the standard is 30 minutes per room per day. I figure at a very generous $50 per hour (100k / year) it should run $25 to have my room taken care of. Now that doesnt cover the shared costs - doing laundry, cleaning the pool, front desk, etc... but my room was something like $150 per night. How the fuck can they not find people to clean rooms?
Answer: they're too cheap to pay a decent wage. But why? My suspicion is that all these places are actually leveraged far too much and paying rent/interest/dividends so much they can barely operate.
I suspect low interest rates and other incentives to "stimulate" economies are actually causing these problems.
The service industry might be labor intensive but is hardly the only cost for the owner(s). What about mortgage payments, marketing, integrating with third parties (expedia), gas & electricity, general maintenance / accounting / security etc. I am sure there are hundreds of other costs that I have no visibility into
You make no sense. No one will pay $25 to clean a single room in a hotel. Just because your room is $150 it doesn’t mean that “there’s plenty to pay everyone 100k/year.
I assume the cost of a room cleaning (including costs other than direct labor) is closer to $5. That's about what hotels will often give you in some sort of funny money to skip a cleaning.
This kinda sounds like saying "why are poor countries poor? Can't they just pay everyone more and then they won't be poor?" I get what you are saying but I don't think this will work if the problem is country-wide. You can't just magically increase everyone's wages in a country.
1. The problem isn't country wide. 2. Don't have to increase everyone's - just those with a "shortage". Companies want a market economy - except when it comes to salaries.
At college I had a professor who I worked for who grew up in Romania. Apparently the song that kids had back in the day was: "We pretend to work and they pretend to pay us."
Don't forget the whining from bureaucrats when American tech takes over the local market, and ships profits back to it's headquarter where... they employ Romanians expats!
It's slightly more complicated than that, depending on how much time has passed since the creators of the system left, but you're correct in general.
Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.
Yes. I had an ancient system where the holders of domain knowledge kept either dying or retiring, one by one. I was frantic, but nobody listened to me. The system began to fail on the day I announced my departure. I would later find that one of the previous architects had reached out to my manager to offer assistance, but this was never made known to us. It wouldn't have even cost anything.
The person who started the thread added this comment:
> The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..).
Sounds like finding their old employees wouldn't help. They need to find old employees of the company that wrote the system and hope they kept a copy of the source code.
Which from a legal perspective might be fraught with peril, at least in theory. In practice, if the company folded then it'd depend entirely on the litigiousness of the current owner of the copyright, which could range from "nobody" to "some holding company consisting almost entirely of lawyers chomping at the bit to eat you for dinner".
There's the possibility that those employees are happily retired and don't care. Or they're just not available in many ways.
I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.
I bet if the hospital is to pay Bob/Charlie $1M for 6 months, post tax to fix it they would gladly do it. Note: this does not mean "We will pay you up to a million dollars, Net 90, after an invoice is approved and signed off by fifteen people". This means "We will wire a deposit of $250,000 today. The rest will be wired every month in 24 hours after an invoice is issued"
The reality is company execs like to talk about fixing problems with money but rarely are they actually willing to fix a problem with money. In my career I dealt with the exact three companies where execs decided to throw money at the problem bypassing all the fiefdoms and bureaucracies created to avoid rapidly solving problems using money in favor of slow moving, don't step on Jackson the IT manager's toes, approaches. All three solved their problems, destroying the local fiefdoms in a process.
This modern-day feudalism with corporations is really the root of the problem. Managers "patch" broken IT systems with lots of manual labor to actively avoid pointing out how some other manager's system is borked. They doom the company to spend hundreds or even thousands of man hours a year to work around these processes, refusing to allow it to be fixed, keeping the "power" and influence of the broken system with some other manager. They do this hoping to curry favor for the sake of their career prospects.
After 25 years, I'm an expert at making small, efficient software tools to fix local IT workflows. God knows, in the Fortune 250's I've worked at, there are endless opportunities. And, while I've written many solutions, I've seen this political game of thrones thwart my efforts many times. This is how you get the kind of 20-year legacy mess described in the original post: People sacrificing long-term success of the company for short-term success in their careers.
I guess there are 2 kinds of people. Those that think this is just how the world works, and find this to be a perfectly-cromulent way of getting through it, and those, like me, who find it odious. Luckily, I now work for people like me. We do what we can.
This applies to similar problems in government as well. Some people care, some just kick the can to whoever is unfortunate enough to inherit the problem in 5-10 years. If you get a few of the latter people in political power, often the best thought out solutions are offered many times and denied because of internal political wrangling.
I have personally turned down an offer very much like this. I could have paid my (at the time) salary and benefits with about 2 days work a week with the option to bill as much time over that as I wanted as long as I agreed to work at least 2 days a week on thier top issues.
The ERP system was a nightmare, though, and the company was difficult to work with (key, long term employees). It was a system I inherited, built out of garbage, that I tried to improve. Before I left, I found a very experienced contractor to take over, on-boarded him, and stayed as a support resource for a couple months. Six months later, he's retired and I'm getting panicked phone calls. I valued (and still do) my current job far more, even though I would probably have done better financially in the short term with the alternative.
A few years ago, I heard they migrated to a new ERP at long last.
For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want. Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day.
In context of this case - if they don't have money to fix their backups (https://news.ycombinator.com/item?id=26539988) they likely couldn't drop lots of cash without a serious credit line.
>For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want.
A new system isn't going to do them much good when they don't even know who their customers are. I doubt any customer is going to hang around for a year while you rebuild your system from scratch, ignoring the fact that if they're an MSP the customers may not even have their own copy of the data.
All of that is ignoring the likely contractual obligations they're likely under to maintain the data on their systems.
>Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day.
Hospital? He said they're an MSP on reddit:
>Yeah we're an awful MSP. I've been trying to find a new job for quite some time.
That's called priorities. I'm sure as every public hospital it pays a lot money in salaries to those protecting and growing their personal fiefdoms within the hospital.
I knew someone who was a specialist in doing calibrations on some very specific industrial measurement equipment, much of which was out-of-date but massively expensive to replace.
He had retired, but took another 12 or 18 month contract flying around the country servicing these things. Apparently they asked, and not wanting to do it he quoted what he thought was an insane "go away and leave me alone" fee, but then they said yes. (I have no idea what that fee was or what multiplier he applied, though it definitely brings to mind patio11's standard advice [1]).
There's a pretty big gap between "happily retired with enough money to not be worried" and having zero interest in ever getting more money regardless of the rate. After a decade a happily retired person who left amicably and didn't hate their job might even enjoy a short-term contract if you can isolate them from all the petty annoyances that come with a normal job. Charlie leaving the country shouldn't be a dealbreaker either unless you literally have tried to track them down and couldn't.
If the long-gone ex-employees hated their employer and would be happy to see if go out of business then you're in quite a bit more trouble.
Is there another industry where a few employees with "domain knowledge" can sink a company?
Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation?
Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
At the highest level, this is a leadership issue. There's a lack of someone at a sufficiently high level (eg: C-suite) that properly understands technology, and that's probably caused by the rest of the leadership team not understanding their limitations.
At a lower level, it's allowing individuals to become knowledge silos (or having a bus factor [1] of 1); lack of documentation; not having+testing disaster recovery plans; neglecting systems, code bases, and platforms; letting technical debt accrue endlessly (or worse: not even realize that's what you're doing).
How to avoid? Make sure there's a strong, competent technology leader with a vested interest in the company succeeding, and empower them to make the changes they need to.
What should that person be doing? Making sure all the stuff I just said doesn't happen.
There are a LOT of industries with this problem. Specialist stuff like military shipbuilding is an example, but also things like precision ball bearings. Knowledge gets lost all the time and old timers have to be brought back in after the fact. Or the technology is just lost and a replacement has to be reinvented (famously, the production of tamper material for the US thermonuclear nuclear weapons).
> To anyone outside of the ballpoint pen manufacturing world, it might seem hard to understand what, exactly, is so surprising about this development. China already produces 38 billion ballpoint pens a year, according to China Daily, which is about 80 percent of all ballpoint pens in the world. That's a lot of pens, but there was a catch: China had long been unable to produce a high-quality version of the most important part of the pen, its tip.
> The tip of a ballpoint pen is what makes it a ballpoint pen. At the tip, a freely rotating ball is held in a small socket which connects it to an ink reservoir that allows the pen to write or draw lines. Manufacturing a ballpoint pen tip that can write comfortably for a long period of time requires high-precision machinery and precisely thin steel, but for years China was unable to match those crafted by foreign companies.
> Or rather, how does one avoid getting into this situation.
IMO, the problem is technical debt. Most likely, in the majority of these cases, the engineers working on the system knew perfectly well that it was a mess, was very difficult to learn how to maintain, and many parts were dangerous to modify. Most likely they told management multiple times that this was a problem and what the consequences of it would be. They would have refused to budget resources for improving it, so everyone either saw the writing on the wall and quit, or gutted it out and learned how to manage the system eventually. Maybe the system limps along for a while longer, but it's only getting worse and the knowledge needed to maintain it more siloed and specialized. Eventually you lose a key person, something breaks that nobody left can figure out, or maybe the whole thing just implodes like the main story here.
To avoid it, you must understand the pattern, hire good technical people, and give them the resources needed to improve the system in an orderly fashion.
>Is there another industry where a few employees with "domain knowledge" can sink a company?
I guess any company where leadership got there without progress on the technical side. Law firm partners get there after being lawyers. Consultant partners get there after consulting.
But if you are the CEO of an IT company without ever having to code, then you may never know the things you must take care to prevent a washout.
> Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
Mentorship and documentation.
You need to set aside time and resources for these two things, and make contributing to them worth something in the performance review.
If all you do is count JIRA tickets closed, you'll forever end up in this situation.
The issue with "documentation" is there are never enough resources devoted to it at most companies. When given a schedule it only covers the main project goal(s) and no time for documentation or refactoring or any quality of life improvements.
Significant documentation is also not something easily written in slack time. With any non-trivial project good documentation is a full time effort. It also needs to be updated as the project evolves lest it become out of date and incongruous with the behavior of the system.
There's dreams of in-line documentation but that's only going to cover individual methods and classes and not necessarily whole modules or major subsystems of a project. It's also not going to necessarily cover in-line documentation from other parts of a project in different languages e.g. comments in config files.
There's lots of technical ways to help documentation but there's no replacement for documentation being a high level project goal with resources allotted to it.
Part of a way around this is broadening your definition of "documentation".
For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.
A fully scripted build and ideally also deployment (CI/CD pipeline) beats a checklist with tens of items that have to manually followed.
An issue system with good descriptions of changes (including why and what), reproduction steps, expected outcomes, links to related issues, etc also can go a long way.
All of this stuff has the side effect of making day-to-day work simpler, overall, while actually just being a part of that work.
> For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.
1. Naming things is hard. It's a bit of a joke but good descriptive yet usable naming is hard. It can get harder after refactoring or adding in features you didn't expect the last time you were in the module.
2. Project structures start out simple. Then you need to add some feature or fix some major issue with a due date of last week. As a project gets bigger the simple structure of yesterday might not serve the needs of today. Resources for refactoring are usually somewhere around the priority level of "Good Documentation".
3. Comments need maintenance like explicit documentation. Tight schedules lead to "# TODO add documentation here". Then you get the assholes that believe comments or in-line documentation is something you put in check in notes.
4. A "unit test" that covers a "real scenario" is not a damned unit test. It's an integration test (or however you label it) and it's not likely something your CI/CD system can run on the regular. If a real scenario is touching a production-load DB or something it's a big deal to run the test. Tests like that need resources to run and non-trivial resources to write. You're not just making sure a method returns a float or throws an error correctly.
I'm not saying your suggestions are bad or shouldn't be used. The issue is management rarely sees any value with documentation, tests, or even code cleanliness. They're rarely incentivized to care about those things so they don't. They get their bonus on feature checkboxes or hitting a deadline. At the end of a project they'll schedule a "documentation" sprint or some dumbass thing that accomplished little useful output. It's worse when you've got a "move fast break things" chanting moron doing the scheduling.
It doesn't help that "Technical Writer" is now an incredibly rare job title, it was one of the first in a long line of specializations that were pushed on to general developers. As you mentioned, they were never allocated the extra time but also don't have the training and experience that technical writers once did. It was also a very poor target for outsourcing.
The specializations common on a team even 20 years ago was much closure to the "Surgical Team" model from the mythical man month than what we have today.
you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.
This. I actually think just documentation is, well not completely worthless but close to. Imagine telling someone they could learn to weld airframes by reading. Just no.
For software, it’s the maintenance task that’s helped by documentation not building. They still need to know how to build but it’s a lot easier with documentation. I know that when we use software packages we all rely on the documentation to understand how to use it and code our desired behaviors. It’s obvious that our own code can benefit from documentation in the same way.
documentation is required but not sufficient. also, documentation is usually outdated really fast if the person maintaining the system leaves and/or does not care
> customers will jump ship and the company will probably sink anyway.
Depends, they might have a captive market or a customer base that's non technical and doesn't understand how bad this is.
If their systems are like this, chances are their offering isn't anything groundbreaking and the competition already provides a better service for possibly even cheaper, so the customers who are willing to jump ship would've done so long ago and the fact they're still in business suggests they have a complacent customer base that's likely to stick to them even despite this incident.
The same reason this technical debt was left unchecked for ages applies to customers. The task of migrating to another provider will most likely rot forever in a Jira board somewhere and they will keep paying their bill in the meantime.
Even if you open a checkbook wide, there's no guarantee that someone remembers the specifics of some kludgy system they worked on years ago. And if things are really snafu-d, there's no guarantee that even someone who does have domain knowledge can recover things.
Let's not forget about the Mythical Man Month. Sure, specific people amd money can fix the problem, but we can't just assume that throwing people and money can fix this.
I would anticipate a mix of: the people who know where the right bodies are buried, the right incentive (yes, that includes money), but also people to look over the entire program, a designer to make sure it is designed correctly, an organized person to make sure it is properly documented, and someone to make sure it is adhered to in the future.
How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference.
Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).
If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things.
Also, make sure everyone knows you are going on vacation once you resolve things.
These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
I remember stuff like that, once the COO called the entire company to the common room to present me a bottle of champagne for my efforts (another time I was given a fancy bottle of whiskey that some marketing guys drank 3/4 of before I ever got around to it). I rather not work any overtime instead.
I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10"
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
I don't know of a company that does this sort of work, but I know of some technologies that experts in receive these types of calls. The one that comes to mind is an ERP-focused database system. It's called "Progress" by "OpenEdge". IMHO, it's awful, but this has no hindered adoption in the slightest. I wrote Progress/4gl (their query language) often enough in a prior position to have it on my resume. Every 2 or 3 months, I'll get an email/call, asking if I could be available for short-term contracting upwards of $200/hr for Progress emergencies. I have declined all of these, because I found it soul crushing to work with in the past. However, if you could enjoy that sort of thing, that's one example of a very lucrative field to dabble in.
I've gotten a few pings along similar lines for my HighJump experience (warehouse management system, for those not acquainted). And much like OpenEdge, it's pretty... rough around the edges (at its core it's basically a runtime for a VM that's programmed with a "language" (if you can call it that) driven entirely by conditional GOTOs developed entirely in a half-baked SQL-backed IDE called "Architect"; this is paired with a DB schema from hell for all data storage, and it's filled to the brim with sprocs because even fucking T-SQL is more ergonomic than anything doable in Architect). And yet, it was actually kind of fun (in the twisted, Dwarf-Fortress-esque sense of the word) to hack on that system and abuse the hell out of it.
And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.
I've worked such a job at a large enterprise. It really does feel like firefighting (minus all the smoke inhalation and physical strain and death risk).
However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.
An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.
But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
I spoke with a CISO of a large F500, his biggest gripe was that he has a team of 30 that can barely keep their heads above water, let alone respond to incidents
It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.
CEO: We are doing project X. We have engaged so and so to do it. You are to support this.
<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>
...
CEO: Yes, tell them to do this.
...
Jack, head of IT in charge of backups: Ok, you can have this access.
Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?
Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.
Damn that's so true. We hired a network expert to figure latency somewhere between our app and the end user. Spent a month asking autorisations for each gdmn net boxes which were owned by several organizations which were working in the same building but not able to cooperate because they were not paid by the same institution.
The guy did very creative reporting though :-)
And the problem was not fixed, of course.
I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.
When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.
In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.
When you are in a firefighting role, your job is to fix the fire, not appease Jack and build relationship with him, the person whose practices created the situation to begin with. Unfortunately, even in those cases the highest level of stakeholder who engaged your firefighting service may not be willing to tell Jack's manager or Jack that he will do what he is told or he will be removed.
Jack's gatekeeping is what keeps Jack employed. The company's ultimate bosses will need to make decision if they want to remove Jack and solve a problem or if they do not care that much that the problem is quickly solved. In the vast majority of cases no one in the company who can fire Jack wants to be seen as a bad guy.
If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?
I would say the analysis is 90-99% of the work, based on my experience. Especially the more jacked up systems where cheapest devs found work on the codebase and it's a mess. 90% only comes in when it's so jacked up it requires a lot of refactoring.
This is my ideal job, I think. Charge a big rate to be the consultant you call when everything dies from ransomware or OPs problem or whatever, I calmly fix it to normal standards (as possible), take three weeks off after.
The first manager I ever respected was this goofball who worked at a fast food restaurant. I don't think the other managers took him seriously. He didn't really take himself seriously either (told the worst jokes), but when there was a rush he was the one you wanted running the kitchen. The other 2-3 people were kind of assholes under stress (they were assholes all the time if I'm honest). He was a machine, and it didn't hurt that he good at predicting orders before they were even made. We always made too much food in these situations but more of his would sell and the wait times were lower.
In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.
I've heard this is a kind of symptom of people who have ADHD, some of them have grace under pressure.
I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD.
Looking through the comments, it appears concerns were raised for years and management refused to allow the necessary changes to save money.
While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.
Reading comments https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... it sounds like he ran into a kernel panic, maybe because cosmic radiation messed with their full disk encryption because they went cheap on the ecc ram, and then he got discouraged, he probably asked management for the resources to rewrite the whole system from scratch, they said no, and then he sat on his hands collecting paychecks for two years until the thing he knew would happen finally happened. How else can you explain an MSP not having a copy of their zone file? That's their core business. It's also no secret in the MSP world how frequently these DNS servers get hacked. I read about it all the time. Especially on Reddit where you can just sense the pervasive fear. So you'd think that'd give a sysadmin a moment's pause to think, OK, have I scp'd my zone file in case that happens to me?
One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..
>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
> Management said we don't have enough money to fix any of this.
It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost.
I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner.
An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years.
I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
> I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
Afford in the strict sense of have enough money, yes of course. But backup is not the same as archive even when the backup exists. It is usually not easily searchable, it is hard to be sure which version is authoritative and so on.
Deletion requests affect monthly backups as well, although it's sort of up in the air how many months the EU will accept as a reasonable business practice for keeping should-be-deleted user data.
The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
I have no reason to doubt that this is factual. However, I have serious doubt that management will acknowledge they made that decision.
Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was.
Thinking you're fucked is unhelpful, but it's pretty normal to go full Alien Hudson in a dire situation and need someone else to assure you that things will be okay.
This reminds me of when the Danish mega-corp Maersk was attacked by ransomware and had all of their computers locked and decrypted with no access.
All computers except one, which due to a power failure had been offline the whole time.
This computer had the critical DNS information needed to restore the network.
Only problem: it was in Africa, in a country that required Visa which would take weeks to apply for and receive. So someone from the African office had to bring the hard drive to an airport that someone from the London office could travel to and pick up the critical hard drive, then bring it on a plane back to London.
Not just DNS, it was their entire Active Directory Global Catalog. If it weren't for that server, they would have needed to re-create their entire user database, all of their endpoint & server management policies, and it would have been almost impossible to safely restore their Exchange email database. They got very very lucky.
Sounds very interesting, but the story seems to be behind a paywall. I found some other articles[0][1] related to the incident, however none of them seem to mention the flying of a hard drive across continents.
These kind of posts are awesome tabletop scenarios. (and fuel for https://twitter.com/badthingsdaily ) Coming up with potential fixes is a good exercise.
Things I would do that I haven't seen mentioned yet:
- Grab images from the relevant servers to have current backups before messing with anything.
- Try tcpdump-ing and decoding the custom KV protocol - other databases are complicated, but things like memcached/redis/... have trivial wire protocol. Maybe it's possible to recover some data by listing all entries.
- Run `strings` on the KV binary or `file` on the data to make sure it's not just some open source project with a proprietary interface. With any luck it's just something like bdb that can be accessed externally.
- Hook up a debugger to the KV and try to recover the keys (or maybe just strace to see if they keys are in any files)
- Start tcpdump on the old DNS IPs. They mentioned they have no idea which domains they own anymore - as long as NS on customer domains wasn't trashed as well, they could collect queries coming in and recover a list from there.
- See if the customer domains can be collected from billing data / some email store of notifications.
(as much as I try to design for no disasters in production, the rush of "this is so FUBAR, what insane thing I can do to fix it" for me is an amazing feeling :) )
Tangent Alert!
I have a 3 months resignation period, so I can't even leave...
Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)?
Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?
Not that the American system (no cause firing/quitting with no notice) is perfect, but 3 months? Yikes.
In my EU-centric worldview, the US system of at-will employment, and of two-week notices as the standard courtesy, is crazy.
Three months is the typical notice period for tech workers and other highly-skilled jobs. One month is the legal minimum in Sweden, and certainly several other countries. Of course it goes both ways, in fact it's even more difficult for the employer to get rid of an employee. An employee doesn't need a reason to resign, an employer needs a valid reason to fire an employee, in addition to observing the notice period.
To complete the picture, it's necessary to say that probation periods exist. Employees may be initially hired on probation (typical for fresh graduates), which has a shorter notice period, and either side may unilaterally decide not to continue past probation. That is, an employer unhappy with a probation employee doesn't need additional legal grounds to say "we're not proceeding with full employment after the probation period is over".
In reality, you can walk out of any job in EU as well. Unless you have some specific clauses in your contract, the worst they can do is simply not pay you anything. There may come a lawsuit after, but I doubt it. You can also get a doctors note and go on sick leave if you feel some serious stress on the job.
It's true that they probably won't sue you for breach of contract, though they can, but if you just walk out of a job, you're at the risk of ruining your professional reputation, arguably the most precious resource. There aren't many other things that would be as much of a red flag when hiring as an employee that just decided to walk out of a job.
Of course if you have good reasons to stop working, there are good options. If it's stress or similar mental health issues, you can resign and spend your notice period on sick leave. If your employer assaults you or just stops paying, or otherwise grossly neglects their duties as an employer, you're legally entitled to quit immediately.
Typically a company owes you money when you leave, like the started part of the month, pending holidays, and the annual bonus/extra pay that you already earned.
If you don't respect your notice period they'll remove your salary over the missed time from that final payment. So in that case they'd remove 3-month salary. Sometimes it goes into the negative, and some companies actually send a bill, but there's basically no consequence for ignoring it. Usually companies don't push the matter.
I remember reading many years ago that employees in the U.S. have the right to quit at any time because being penalized for quitting would make the employment “involuntary servitude,” which is forbidden by the Thirteenth Amendment to the U.S. Constitution. I can’t find any confirmation of that explanation on the web now, though.
Yup. Most states are "at will", which effectively means both parties can leave the arrangement without notice or repercussion.
There are federal laws that limit this, typically applied to larger employers making entire business lines redundant (closing a factory, etc). State laws very, some with zero extra protection, some with a bit more, but AFAIK none with European-style rules.
In professional jobs, it's traditional for an employee to give 2+ weeks notice. Most employees will try to stay long enough for a reasonable transition - depending on position/project status, that could be 2 weeks or a month or two.
It's traditional for employers to give 2+ weeks severance. Most white-collar employers have periods that start at 2 weeks and grow with tenure. Including benefits is hit or miss, and a HUGE issue because health insurance/care in the US is so expensive.
Interestingly the case can be made that the military is not exempt from the 13th amendment, which only makes one single exception for prisoners. It's just that the courts have ruled consistently that OBVIOUSLY the military wasn't meant to be included in the ban against involuntary servitude. Obviously.
Just like they have also ruled that mandatory Jury duty is not in violation of the 13th amendment either. Funny how the system protects and perpetuates itself.
But with respect to your point, I am nearly 100% sure that it is not the case that employers can use financial penalties to enforce such a contract. Anything you are paid as wages is pretty much untouchable after it is paid out, unless it was paid in mistake or you intentionally damaged the company or something. The best the company can do is withhold potential future earnings or bonuses, or take back conditional non-wage payments like bonuses within a certain window of time.
This is part of why golden parachutes exist. Your contract has very generous severance clauses which award you lots of money if you follow a certain procedures for leaving, which involves advance notice and such. Fail to follow that and your paycheck just stops the day you walk off. So there are incentives, but they can't go after your already earned wages, paid or still due to you.
> Your contract has very generous severance clauses
That's pretty much what I meant although, as noted in another comment, I assume there are cases where they would try to claw back relocation or signing bonuses. Also, of course, things like RSU vesting.
>they can't go after your already earned wages, paid or still due to you.
Certainly, and depending on how paid time off accrues, anything earned there--which is one of the drivers with "unlimited" PTO because that's not owed when you leave.
In my EU-centric worldview, the US system of at-will employment, and of two-week notices as the standard courtesy, is crazy.
Oh, I agree. But 3 months seems crazy to me. If I have an open position and it isn't filled in ~4-6 weeks, it often gets given to some other team (on the basis that if I really needed that position filled, I'd do so). I can't plan hiring over that timeframe.
And when hiring employees in India, we've had them claim a 2-month resignation period, but they just end up using that time to find some other opportunity on the previous employer's time. It's a real mess.
> Oh, I agree. But 3 months seems crazy to me. If I have an open position and it isn't filled in ~4-6 weeks, it often gets given to some other team (on the basis that if I really needed that position filled, I'd do so). I can't plan hiring over that timeframe.
If you're in a culture (locality + industry) where a position can be filled in 4 weeks, that's one thing. If you're in a culture where every experienced person has 3 months of notice, then nobody would take your inability to fill a position within weeks to mean that you don't really need it.
I have some experience with part-time contracting with US customers. Seeing how quickly people can join or leave there is certainly a culture shock!
Assuming that you’re talking about senior people, who are all employed, I don’t see how this helps. Realistically employee X resigns, and we find a replacement; that replacement is working and needs to resign. Their end dates are still serialized.
We have a bunch of employees in Europe and everyone involved except the very marginal employees sees this situation as strongly negative (and, in fact, an argument to not hire in Europe at all).
Are those Europeans who don't value their rights fairly young?
I don't need my 3 months now. They might be useful if my employer axes a whole department. They're useful for people with family commitments giving them less mobility.
What you might not realise is that these three months would be the default legal position in case of disagreement. If you want to leave earlier and the employer agrees, you can usually leave earlier, too (with them being off the financial hook).
I had a "to the next quarter" period in my last job, and when moving to the new job the deadline was missed, making the contract valid for a whole quarter.
It wasn't a problem. I was on good terms with the previous company (I'm still getting invited to their anniversary parties and stuff like that), so there was no issue in working something out.
Also, remember that ~6 weeks of vacation per year is common outside the US, so often it boils down to taking your remaining vacation. (Though I think you need to be careful with taxes and other regulations if the contracts overlap.)
Since the same base rules apply for everyone this usually doesn't happen, plans are built accordingly and recruitments take into consideration those time scales.
I agree (I am from the UK) when first saw US professional workers complaining about > two weeks my first thought was wow you must have a really low status job.
In Poland it scales over time: you start off with 2 weeks notice (for both sides), after a year or so this jumps to one month, and after 3 years it jumps to 3 months, or something like that. I've seen people quit right before their 3 year anniversary just so that they don't fall into the 3-months-notice period.
I haven't worked much on regular job agreements (been contracting or "contracting" most of my life), but generally if you want to quit you go to your boss and say "hey, I wanna quit" and you negotiate when, regardless of the default requirements (which only come into play if either side doesn't agree). There's no business in keeping an unproductive employee around for 3 months.
European here. Having 3 months written in your contract (and in many countries by law, like France) is pretty common.
You can still quit if both parties agree. I've done it before.
The clause is there to avoid this type of scenario where a key employee just says fuck it and leave.
It works both ways. So a company can say fuck it and fire you just because they want to.
It's a fantastic system. It can be exploited, but usually works really well for employees and it gives you a huge safety net you can (usually) rely on.
> If they are at the "fuck it" point, what's going to force them to be useful and productive during the forced 3 month notice period?
Nothing - beyond professionalism. It is common for knowledge transfer to happen in this time, and any outstanding vacation days are usually taken at this time.
There's a huge cultural difference between US and Europe (rest of world?) when it comes to going to work after serving notice. It appears the employer/employee relationship is more adversarial, so much so that the employee is not trusted/expected to do the right thing after being fired (or after they quit) - the whole "immediately getting escorted out of the building by security" is unheard of in Europe.
It's not unheard of, but it happens quite rarely. Violence, or threats of violence, typically leads to being escorted out immediately here in Europe too.
You are correct - when I was typing my previous comment, I had considered adding "unless criminal behavior is involved", but I thought it would make the sentence too unwieldy. If I could edit, I'd remove "unheard of" and replace it with "ridiculously uncommon"
The countries in Western Europe specifically are much much smaller pools in the tech space than anything an American could conceive. The reputational damage of getting to the “fuck it” point and becoming a pain in the ass has a lot more blow back when the pools of both workers and employers are tiny even compared to a second or third tier US geo.
If you reach that you just put in the notice, take whatever vacations you have and then put in sick leave.
But 3 months notice is used for, so the company can hire a replacement for you, and you can help train him, and transfer knowledge.
A lot of people reading this site work in companies that have huge teams, but most people in IT I know from EU (where I live) are one or two person IT teams. 3 or 4 person team is considered almost medium sized.
In such situations you need yo hand of your task responsibly.
And that the one thing that will get you burned, if you leave your previous employer hanging.
At least in Germany this goes both ways. The period can also be longer depending on your seniority. This is to give the company time to find a proper replacement for you and allow for a smoother handoff. Of course if both parties agree the grace period can be skipped or shortened.
You usually don't get stuck with duds because there's a probationary period of (max.) 6 months in the beginning in which these protections are not in place.
Also (after the probationary period) the company needs a legal reason to fire you. If they wanna fire you for incompetence oder insubordination they need to have reprimanded you at least once officially, in written form. Only after getting a 2nd reprimand for the SAME THING can they act on it. Meaning if you told the boss to go fuck themselves once and shown up drunk to work once they wouldn't have legal grounds to fire you and if they did you'd drag their ass to labour court and get some kind of compensation.
There are certain things that you can outright be fired for - such as stealing, I believe. I think that showing up drunk for work might also be one of them?
There are more, and they are immediate. And nearly always lead to a lawsuit, because it is only for grave errors in judgement. i.e. theft, violence, endangering co-workers, fraud, misrepresentations to the board and such. So when this is triggered it is never a clean break. Lawsuits can come from both sides as the reason for the firing is grave and the argument doesn't end when a person was kicked out.
AFAIK showing up drunk isn't enough but if you fuck up (I think that's the legal term) and say you drop the production DB while drunk that you can be fired immediately.
To give an example of a sysadmin from a job long ago:
Turning up drunk to a meeting is enough to start disciplinary procedures -- HR handle this, and work out how to proceed. (On [1], I think they would have been proceeding through the "Misconduct" section, second chance etc.)
Rotating the backup tapes, then carrying said tapes about 1km to the other building to put in the safe, was part of the job, but he was watched while he did it.
Stopping at a bar along the way was not fine.
Being oblivious to the point that a manager was able to take the bag containing the tapes from the bar back to the office was not fine.
Coming back to the office (drunk) and not realizing what had happened was not fine.
This was gross misconduct, and led to immediate dismissal.
IANAL: In the US, doing something destructive [at work] while intoxicated can be "gross negligence", which is criminal if I remember correctly. Example... you show up to work drunk, get in a forklift, and drive through a wall causing damage.
I don't think it goes both ways. As an employee, your maximum notice period is always 3 months - which is also the legal minimum required period for employers. Now depending on your position, your employer will probably even have longer period (i.e. if covered by a union negotioted tarif). Additionally, you can't even be fired at will - the employer always has to give a reason. "Shrinking down" due to business requirements would be a valid reason, but that would mean that the company cannot fire you and hire someone else for the same position as long as you got the formal qualification.
> This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)?
In theory, the business could sue an ex-employee who just left for damages caused by the breach of contract, but in the real world it would be pretty much impossible to prove that the damages were caused by the employee leaving. In most cases it's simply considered a dick move, and you'll end up with a hole in your CV as you won't ever get a decent reference if you just get up and leave.
> Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?
Usually yes. And unlike the situation above, employees could probably drag the employer to court for it, unless the employee did something illegal or completely irresponsible during working hours.
Yes (in France at least) it is on both sides, unless both sides agree to shorten it
> face some sort of repercussions (financial, I assume)?
I'm speaking for France. This is a tricky subject, you almost can't get any repercussion. But it can sucks, like the employer can refuse to fire you and suspend your contract, in the mean time, you are not paid but are not allowed to work for someone else.
I once had a colleague "go on holiday" and never come back to work.
If you don't work your notice period, you obviously won't be paid for it, and any reference will state that you broke your contract.
(Unless your employer asks that you don't come into work any more. This is called "gardening leave" in Britain: https://en.wikipedia.org/wiki/Garden_leave , and it's normal in some industries and circumstances.)
I'm looking to leave my current job but if voluntarily leave in my first year I have to surrender my signing bonus.
It's certainly a crushing feeling to be chained to a job. Even worse are these stupid 2 weeks or unimaginable 3 months notice. As everyone says- you'll never get notice you're being fired. It'll be quick like a mob hit.
For many positions, the person isn't a positively contributing member for three to six months. A positive ROI on the expense of hiring them isn't realized until much later. Having signing and relocation bonuses be paid back if the person doesn't stay for a year is quite reasonable in that context.
"We didn't pay for your relocation to X just so that you could change jobs after a few months."
Two weeks notice is a courtesy.
The much maligned PIP is often a hint that you're going to be let go. Rarely is there a "you're being let go today, pack your desk" unless there is cause (the one time I saw it was because someone was using their desktop support credentials to snoop on the CIO's computer - that one they weren't even given the pack your desk) or the company is in trouble (and in which case, things like the WARN period kick in).
In a previous job it slowly dawned on me that the job was making me quite unhappy. This was about six months in. Only six more months until the "vesting cliff"! I managed to power through and get my stock options for the year.
So, while we don't have 3 months notice in the US, we certainly have similar ways to get stuck.
I am in the Netherlands, and my current contract stipulates that should they want to get rid of me, the company has to notify me 4 months in advance, whereas should I want to leave, I need to notify them 2 months in advance.
Do you have any sources on the legality of different notice periods in Europe?
The legal situation in Switzerland would (probably) allow for this. What it wouldn't allow is the reverse agreement. I.e. you can be fired in 2 month, but have to give 4 month notice.
There are three types of employment laws herearound:
- Statutes can be changed freely if both agree
- They can be changed only to the employees advantage. For example: minimum vacation days are 20 days. A contract can stipulate 30 days, but not 10, or 15 days
- Some statutes cannot be changed at all.
I can obviously not speak for all of Europe (plus INAL, etc), but would assume that it's comparable in most of Europe, or even more stringent in terms of employee protection. Employee laws here are quite liberal compared to other European countries.
Switzerland, default rules regarding termination (if I recall correctly).
* first month of work, 0 second notice
* rest of the first year of work, 1 month notice
* second year of work, 2 months notice
* years after third, 3 months notice
Impossible to terminate: pregnant women, before and also during the legal maternity leave. Common termination is the day upon returning to the office.
Personnel called up for the army or civil defence.
People on a fixed contract.
Employer must give notice but may be without reasons. Without reasons often leads to lawsuits, because terminated will claim one of the illegal (discrimination etc) reasons applies. Still cost of a lawsuit is normally not prohibitive.
In case of critical financial issue termination maybe shorter (mass-layoffs).
In Sweden (don't know where this company is located) it's usual with collective bargain agreement and if you've been employed for along time (5+ years) it's not uncommon with 3 months notice. It is usually accompanied by 6 months if your employer wants to let you go (but it's not easy if you're the employer to let people go). Most of the time, these periods are negotiated when someone resigns though.
For Finland 3 months is bit special, mine is 1 month and previous job was 2 weeks(up to 5 years). From employer side legal minimums are: under 1 year 2 weeks, under 5 years 1 month, under 8 years 2 months, under 8 years 4 months and over 12 years 6 months.
Penalty from employee side seems to be monthly wage, so in this case 3 month salary. Ofc, seeing how company is probably doing I wouldn't be too worried of them coming after him...
I guess US fire-at-will system should make contracting even more attractive option for good engineers: you do not get any social protection from being fired at any time unlike say Germany. One solid benefit less to justify pay difference.
Not quite. If you are let go "without cause", meaning they fire you for no fault of your own, you are entitled to unemployment pay from the government. In my case it was around $800/week for about 4 months while I looked for a new position. It didn't replace my salary, but it was more than enough to pay my bills.
Most large US companies also have a policy of not giving references beyond answering if the person worked there for the time claimed in the position claimed.
There are a number of benefits to full time employment that good engineers appreciate.
Healthcare (in the US), known reliable income (this can be a big factor for financing a house mortgage/car), various other benefits (I get massage credit, dental, vision/glasses, acupuncture, physiotherapy). The biggest for me is just avoiding the stress of feast or famine, managing business/taxes, etc..
First of all, you get benefits as a full-time employee that contractors don't get, like healthcare, disability insurance, and paid time off. (This all mostly boils down to money of course.)
Probably more importantly, yes, a company can in general simply tell you your services are no longer needed one morning. However, in reality, absent either company downsizing or a documented performance issue, that is unlikely to happen out of the blue. Whereas not renewing a contract at the end of some project is commonplace.
Employee status vs. independent contractors is actually something of a political football at the moment as contractors include both the stereotypical gig economy workers and freelance writers who want to freelance as well as contractors of other sorts.
Health-care and other benefits would be priced into any contractor's fees. IE, for a full-time job that paid $100k + benefits, you'd need ~$160k on contract to make up for the additional taxes, self-paid benefits, etc.
Looking at peers who struck out on their own, you can make good money as a contractor, but you typically need to be top of your profession and able to command massive fees. And like any business owner, the pressure to always have too much work usually ruins any chance at using the money for longer, more extravagant vacations.
In my country, we have a 3 month notice aswell and yes it does also apply in reverse. You can negociate to lower the period but most of the time the company will decline your request.
For high-demand jobs it is a nightmare for everyone.
In a former job, because I was an employee for many years, the notice period the company had to give me was 6 months. My notice period in case I wanted to quit was still 1 month (the legal minimum).
In the US we have the similar crappy situations. No competes come to mind. Healthcare tied to employment (what if you wanted to start your own business?) is another.
UK here. Terminology may differ, I think what is being discussed is a notice period. Normally a resignation is either accepted or rejected pretty much on the spot and if accepted is officially seen as a more amicable departure.
Handing in your notice to leave is something the employer cannot reject. Depending on circumstances you might try to resign first. Sometimes you may be "asked to resign"[0] and made a nice offer to encourage you to do so, offer will almost always have an NDA attached to it.
[0] this can be preferable to the company as it can be quieter than sacking you
> This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)?
Yes. The worst they can do is take you to court for breach of contract though. That can be expensive and embarrassing for both parties so unless they are whiter than white[1] or bitter enough companies often don't bother.
Potentially a big risk to take though. And in an industry that is hardly free from nepotism, you probably don't want the reputation of being someone who walked that way.
Three months is most common in skilled jobs, one month otherwise. In retail work and other low-skilled professions two weeks is not uncommon, though here people often do just walk away as the hassle of chasing them for breach of contract is massively not worth it.
[1] i.e. there is nothing you can say on public record that looks bad for them going forward
> Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?
Yes. What usually happens if they want to have you gone without notice and don't want to make you an offer to resign "amicably" is that they put you on gardening leave. You are still employed and paid for the notice period but are told to stay home. They can't stop you looking for new work[2] as long as you don't do it "on their time"[3] but you can't start new work without resigning and them accepting that resignation until the end of notice period without the same risks as if you had just walked out without notice yourself[4].
Gardening leave can also happen when you hand in your notice, though if you want to leave and the company wants you to leave then they'll just agree to waive the notice period.
[2] Some contracts have clauses that say otherwise, those clauses are not legally enforcible. Some even have clauses saying you have to actively look for new work, these too are not legally enforceable.
[3] Which is impossible for them to prove unless perhaps they have you check-in regularly while you garden.
[4] Though they are likely accept the resignation as at that point they can stop paying you for the notice period, unless you are going to work for a competitor, or they otherwise feel like being dicks about it.
At places I've worked in the UK it is asymmetrical. The employer has to give three months' notice but the employee only has to give one month's. It is intended as an employee benefit, and it is. I've never been asked for more than one month's notice from my side.
I don't know where the OP is located but he should double-check his contract. The first time I heard of the three month period I was reluctant, until I realised it didn't apply to me.
This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains which should at least bring their internal services back online and allow them to proceed further.
"So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records."
Elsewhere, they say that two years ago, they found they were unable to restore from backups...
That isn't the failure of the business; backup failure is a screwup of the IT, which I think is not that uncommon.
But the fact that after this event the management did not get it fixed (and requested that IT demo it, three times, with no failures) is an existential failure.
I'd say if it was such a house of cards, my best bet is that this security chain is full of holes. Maybe the ERP used a fixed password, which can be guessed/bruteforced.
it doesn't sound like modern software. At my current employer (to remain unnamed) when I started there were a bunch of MS-SQL servers with an SA password of <blank> (i.e. no password). It took a long time to fix that because we had hundreds of local installs of the client app that had SA/<blank> in the app config file (yes, they were setup to access as system administrator), so fixing it meant reconfiguring hundreds of local installs. (at the time we didn't have a mechanism for automating it.) We still have "enterprise" software that does no password rules, and would happily allow a blank password.
We don’t know that the records are still in the database. This could be a matter of ON DELETE CASCADE followed by a delete of what looked like innocent data and then boom it’s all gone. Then the integration updates bind which helpfully erases all records and then once there are no domain names to talk to everything is down.
I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.
Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees).
Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other.
But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".
> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database..
It doesn't, but in my experience, the ciphers and key sizes commonly used were often not future-proof :). This rests on the assumption that nobody updated the keys/encryption components in 20 years - which I think is quite possible with such internal enterprise deployment.
I did something similar for a customer of ours who was a major UK cinema chain - they had had a falling out with some small company that had built an online booking system.
Highlights:
1 Bookings where made at individual cinemas
2 The network was a packet radio system that blocked :-) only Transaction could be in flight (Leicester square had two channels)
3 Core of system was Perl running on three windows NT machines which manipulated a shonky screen based system (Pacer Cats)
4 Trying to sell tickets online for Phantom menace system crashed at midnight (when tickets went on sale) we made 1 transaction at 6am
% And no credentials so I had to research how to crack NT - I did consider setting up or fleet of a dozen or so of our suns as a cluster to crack NT, but not sure if the internal security would have liked that (aka secret squirrels and the bit that Bruce Shenier worked for
I'd love to read a detailed write-up about this. Why packet radio - what was the real-time requirement and why couldn't it be fulfilled by an internet connection (I understand internet was primitive back then, but ISDN should've been able to give you enough bandwidth?).
We never understood why the where using packet radio as you say in 1999 ip via ISDN would have been fine.
Its complicated by at the time the total lock on ticket booking systems cinema and theatre by a particular company - who wanted 1/4 mill plus to update the system.
It was an interesting project I had to get cover from a v senior manger to investigate how to break into NT.
I agree. My first reaction was, "I think I could fix this". Worst case this may be be proceeded by a forensic recovery of the data. This doesn't seem impossible at all unless there is some other factor at play like ransomware/physical damage.
> > All domain information was wiped out and records became null.
> > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there.
To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath. A high enough rate that even if they are doing hours elsewhere some will be tempted to sign in after their work hours to help out.
A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.
To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath
Those people are probably employed somewhere else and would need very significant incentives. More than likely their current employers would take a dim view of them freelancing to help a competitor, so: enough to take a year off or even to retire on. Basically the idea of hiring back former employees as consultants is a non-starter.
It is also helpful to have at least two tape drives, and check the backup tapes against both.
Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine.
Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks.
A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.
I always wrap the actual cp/mv/rm etc in echo first before actually running it e.g. running echo "mv $src i" first would have immediately highlighted the problem visually.
It's dark, but I kindof love situations like this (although luckily I haven't had something like this happen to me in digital space since I was a teenager). Some places I would look:
1) Are you SURE that everything is actually gone? Is there a logfile, a billing trail, sent emails trail from your email provider, something like that? If it was me I'd be looking for a secondary system (like alerts, logfiles, billing, etc.) that had some of this information stored in it as secondary function, and writing some scripts to parse those things and start rebuilding what can be rebuilt.
2) You said you lost all the emails and SMS of your team. Do you have an HR department? Surely they have this information. Your team is likely going to start trying to send emails to each other about this. Can you dump logs off of your mail server to rebuild at least the naming schema? For phone numbers: can you try an old school phone tree style? Start with somebody you know, and ask them for everybody that THEY know and so on.
3) Was there a staging environment that might have had some of the data you're missing on it?
4) Are you really really sure that everything was wiped out? Is it possible that your ERP system is broken (lost its DB connection or something?) and that caused a cascading failure that makes it LOOK like everything is gone?
5) Can you get at the BIND server somehow? Are there logfiles there?
I'd love to know more specifics about what "wiped out" means in this case. It seems unlikely that it's properly "wiped out" unless something malicious happened, and even then I think you could rebuild a lot of this from sortof "secondary" data.
Think like a hacker, but your system is your own target. How can you steal the data you're looking for from places it isn't really "supposed" to be?
I get where you're coming from. It's a little bit like a real world, high stakes escape room.
Come to think of it, this could be a fun online challenge. I picture a series of VM images, each of which describes the design of the system and the terrible thing that has gone wrong, and you are challenged to get it fixed and/or get the data out of it.
It’s easy to imagine there are a lot of these type of time bombs out there, particularly in really old legacy systems (> 20 years for example). I was at one place where even building the application for deployment led to a one week outage when some core people were laid off.
Up until 5 years ago, my employer's primary system (government, property tax assessment, billing, and collection) was from a company that had gone belly-up in the early 90's. We fortunately had the source, but it ran on a VERY legacy database called Unidata because it had originally be developed on the Pick operating system. Unidata was/is a Multivalue Database with a built in BASIC programming language. By the time 2014 rolled around there was one programmer in the organization that understood the system, and he was responsible for all code changes. I half-joked with management that we needed an insurance policy on his life/ability to work, because if we lost him, it would put us in a very difficult position, and that software accounted for 90% of our revenue (as well as a lot of the revenue that went to all the local towns and school districts for their operations.) That employee was also nearing retirement (he is retired now.) Fortunately we were able to successfully migrate off of that application, though we keep a minimal license pool to the old system so people can validate pre-migration data against post migration data.
Interestingly, some of these legacy systems run on things like IBM i (AS/400) which have an easy backup story and vendor support for recovery. I can have a new i machine receiving a backup and deployed fairly quickly (day or two) and I'm out in a rural area.
50% of companies who lose all of their data go out of business within 6 months. They clearly failed to test their backups with regularly-tested restores. And, a separate alternative system should've had multiple revisions of critical pieces of information like customers, assets, inventory, bank accounts, debts, and employees.
A data recovery shop could attempt to forensically-reconstruct SMS numbers. Look for logs and other places critical deets might be found. Try your best to get things back and make alternatives processes to piece together what you can. A phone carrier or SMS send provider might have a log of phone numbers. The paycheck company may even have the address of everyone. Send them a $0.01 paycheck with a note on them to call in.
I would ride it out for a paycheck because it's not like it could get worse than going under. Just be sure to get paid.
Reaching people seems like the least of their problems to be honest. If they use a payroll service, they'll have addresses--don't know about Europe but does just about any company do their own payroll in the US at this point? In any case, I assume most people know how to reach most of their direct reports and many of their co-workers. Especially at a fairly small company, I would think you could reconstruct contact info on employees fairly quickly.
Of course, that may not help with the bigger problem.
Yep. It's going to be a massive problem figuring out what's what, who owns what, and who owes what. I don't even see how they'll keep enough customers to stay afloat.
So many beautiful details if you search for more comments by the original author of the post:
"The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..)."
Extremely poor planning/lawyering on the companies part. When larger entities deal with smaller entities, it's common to have a source code escrow as part of the agreement which grants the purchasing company rights to the source code in the event of this exact situation.
Obviously we all know that source code alone isn't enough, but it is something to start from.
If the ERP solution was not SaaS and installed on site, the purchasing companies IT should have gotten credentials as part of the contract. Again, normal type stuff when buying software from smaller companies that have somewhat higher risk of going out of business.
One of the comments in the thread said, simply, "Prepare 3 envelopes." Being a 25-year sysadmin, immersed in thinking about the problem, it caught me totally off-guard. I don't think I've ever seen a more perfect response in a Reddit thread.
New executive starts the job. On his desk are three envelopes, left by his predecessor, with a post it note saying "When shit hits the fan, and you don't know what to, open one of the envelopes."
Yadda yadda, exposition, things are normal and exec is successful for a year or two and then things go pear-shaped. Completely out of options, he opens the envelope, finds a letter which says "Blame me." He blames his predecessor for leaving the state of things in such a poor shape that this misfortunate was unavoidable, everything blows over, his success continues.
Blah blah, narration. Another shitstorm. He opens the second envelope, finds a letter which says "fire all the managers." He promises a significant shake-up, says he has the leadership to turn this around, purges the old guard. Things go well.
Hurly-burly story, and so forth. Catastrophe strikes. Having blamed his predecessor and fired everyone and installed his own managers, he has no one else to turn to for casing the blame. He opens the last envelope, hoping to find salvation.
Thanks. Reddit is such a good example of how "progress" hasn't really been progress in web development. The new front-end framework site is so much slower and more buggy.
They want people using their app, for some reason I don't understand. Maybe to make ad-blocking harder by getting people off computers where it's easy (most mobile browsers don't support plugins, so it can't be about them).
Division of responsibility can make this sort of thing very difficult to know.
If you aren’t running drills then transparency may not be in everybody’s self interest and someone somewhere is covering up for the fact that they only half know what they’re doing.
I’ve seen a few data disasters in my time but I think this takes the cake. It’s a good reminder of why you just need to prioritise the sort of work that has no immediate payoff.
On a personal level, this may be a little self centered, but I use cases like this to put my own problems in perspective. Recently I had to troubleshoot lots of IT issues during a winter storm. It was a bad outage for the organization during high load important scenario. I distinctly remember thinking during the thick of it “this is bad, but imagine being the poor engineers in TX responsible for the power grid.” This helped me not panic and focus on the problem at hand.
I feel like this ability to step back and not take a problem more seriously than necessary can be an asset. In this case, it’s pretty bad, but “at least people aren’t in immediate danger of their life.” Think it to yourself, but probably not best to share with others until after the problem is mitigated ;)
What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?
Not sure what country the person works in. In addition it could be a situation of a small town / tech backwater and word getting around, regardless of the legal question, so quitting out of contract could get you blacklisted.
Ultimately though a person could not show up for any reason at all (e.g. hit by the proverbial bus) and you need to plan for that. Sounds like a company that just flies by the seat of its pants so they probably have that 3-month clause for that reason.
In terms of personal reputation, once you're in charge of backups at a company that went bankrupt due to the failure of their backups, I'm not sure it makes much difference to your reputation whether you diligently work your notice period or not...
Depends on the contract and jurisdiction. I had one contract that stipulated a fixed penalty, which made any attempt of ignoring notice periods a calculated risk (the fine wasn't that steep). In any other case, in Germany, I would expect a law suite, incurring costs, stress and time, followed by some kind of agreement between the two parties. As an employee, you don't necessarily have the time nor the money (there's insurance for legal disputes like that so).
In that particular case, my experience tells me the guy won't get any credit for finding a solution, I would consider sick leave. Burnout, whatever. Doesn't have any monetary consequences in Europe (at least no harsh ones), and Germany covers up to 6 weeks of salary.
Any way, you can forget about getting goods references. Not that would get them anyway, with an employer like that.
At least in Germany, while it's in principle possible to sue the employee for damages, it is very difficult to succeed with that if it isn't downright sabotage. You can't force the employee to work, it quickly goes into negligence on your part if you can't handle an employee going missing, ..., so you'll probably not reach much more than the salary - and for that just terminating the contract early would be easier, either in agreement with the employee or one-sided after they stopped showing up.
In America absolutely nothing happens. He might have some type of agreement where he gets a severance payment upon resignation, that would be forfeited. If I was him, assuming upper management starts yelling or whatever, I would just leave. Stress can kill you, and stressing over a job is absolutely never worth it because if you die today , they'll post a req for your position by the end of business.
It looks like they could probably sort something out, wouldn't be a bad idea to call up any of the old vets who retired to see if they could come down to the office and contract for a week or so.
I guess it could be a problem in the US for COBRA (health insurance continuation)? apart from that, if the company is dead I guess there's little they can take away that you aren't going to lose anyway.
Well, for one, your work references probably won't be the best.
Even if you're not responsible for the problem itself, leaving your employer in the ditch, when there's a major outage - well, it just doesn't look good (barring some scenario where your employer is completely unreasonable - like not paying you).
It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?
But if they get in a situation where management is yelling at them and demanding they work 20 hours a day until this is fixed somehow and they have no idea how to. Management won't bring in consultants, etc. That ship has already sailed.
> It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?
The firefighter who recognizes when the building is about to collapse and gets out of there is a good firefighter.
I just checked for curiosity. The employer can sue for damages they'll have to show.
But there's some additional factors here. If the employer misses salary payments, then they're in breach of that employment contract, and unless they can't fix the situation really soon, that's grounds for immediate termination.
Also in Finland, salaries earned in a normal employment are backed by the state for up to 3 months of missed payments (the state will then try to collect from the employer). So it's relatively safe for an employee to stick with a sinking company. Just get that CV polished and start warming up the network.
It seems to me that they can sue for 3 months salary. Or the amount of days employee does not work. However in this situation I would imagine they have more immediate concerns so might not get around even to that.
Perfectly common in large parts of Europe. I also have a 3 months notice period, which is pretty much the standard here in Sweden for tech workers or other highly-qualified occupations. For very senior positions, 6 months isn't unheard of.
It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.
How does getting a new job work? You tell the new company that you have to give 3 months notice? That's some long term planning on the part of the hiring company.
Yes exactly. Sometimes they ask you if you can negotiate with your employer to shorten this period, which is often the case.
Of course, they would rather it was shorter, but hiring is already a multi-months long process in most cases, so a few more months is inconvenient, but not the end of the world. If you need someone now, you hire contractors.
You would have to prove cause to fire without notice.
Also, depending on the country, firing/layoffs can be difficult without cause (and often even with cause) (e.g. you can't layoff someone at will, you might need to prove that you have an economic downturn before downsizing). But this varies way more between countries as far as I know.
That was my thought. It's typical in Canada that an employer will pay you out for the rest if your two weeks or whatever, and in the U.S I don't know, but have it really favours the business. I'm still at abloss as to how this works in what is apparently most of Europe. Notice is more like 2 or 3 weeks.
Can't speak for Sweden, but we do pretty much exactly that with our India team. They have to give notice well in advance, and so when we hire a new team member we have to do so with the understanding that it will probably be a few months before they join the team.
Yes, but that's nothing unusual when the entire industry expects that. The last time I changed jobs, I signed a contract at the end of March, to start in August. That was the three month notice period plus most people are on vacation in July, so my new company would rather have me in August.
There are contractor agencies filling the niche of providing employees on short notice (and usually for a short period), but generally having people begin 3 months later is just something everyone expects and factors into their processes.
The average length of time people spend at a single job is also longer than in the US.
Yes, if both parties agree, you can leave at once, otherwise employer can sue you for damages or lost profits (if damages are small, they don't bother) and you don't have unemployment benefits.
I guess it just seemed so implausible to me. As in, what happens if 2 weeks or 4 months in you just decide that the job is sufficiently ill-suited? Is there typically a trial period of equal time that precedes such an agreement?
What would be the practical consequences to the employee if they break that notice period in UK, i.e. stop coming to work (or, especially now with Covid, simply stop doing the remote work) sooner than these 3 months have passed?
I suspect you could be sued by your employer. Mind you I've never heard of anyone being sued in that way - can't imagine it would be worth it for the employer unless they could demonstrate a significant financial loss.
I investigated this once (while I was sitting around doing not much during a 3 month notice period, funnily enough). It's as you say, the breach of a contract is something where the employer would have to sue you for damages, and if you're sitting around not doing much, it's going to be hard to argue that losing you has caused them harm, much less enough to make it worth suing over.
I didn't break the contract anyway out of some combination of feeling like it was breaking a promise and the worry that Cambridge is a small town.
while I agree that this is common in Europe, if the company ceases to exist, what good is the 3 months notice for the employee? If he/she has any options, I'd leave asap and not worry about amicable terms with a soon to be non-existent entity.
Fair point - in that case I suspect that would be highly dependent on whether there are any assets remaining in the company and how the local legal system treats employees as creditors.
Mind you - if the company is going down the tubes they are highly unlikely to enforce any job contracts.
Well, it looks like someone better track down the people who knew how it worked and offer some serious cash.
The only actual backup is one that has been tested. With the cost of hardware, buy 2 or have an agreement with your vendor to have a machine ready to receive a backup.
Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.
Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
It's up there with "unless you test power supply failover, don't rely on that generator to save you".
I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up.
When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
Yup. Have the problem when all the right people are awake and on-site to handle it.
I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment.
If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
They explicitly mention that they have backups, but that the "backups are fucked", whatever that means.
Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.
Can’t you just pull the drive from the failed server, extract the DNS entries and build a new DNS server? It doesn’t sound like everything is broken, just that it is failing because the DNS server isn’t answering. Just install Linux on a new box and set up a DNS server on it. It could be done in a day I would expect. Or am I missing something?
It would work if they knew what the DNS entries were but seems they lost all DNS records and don't even know what records they have/had, so spinning up a new DNS server won't help as they don't know what it should serve.
> All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in the ERP which is now busted
That's part of what bothers me about this whole story--it says they were running BIND, which configs on text files for goodness sake. These critical records were tiny and could fit anywhere & transfer in the blink of an eye. That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling.
> That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling
I find this to pretty common in many setups where the engineers don't focus on simplicity and removing layers of abstraction. If the workforce is young and inexperienced, over-engineering tends to happen everywhere and you end with situations like this.
I have seen worse in my years in the industry, that's for sure.
Best thing to do for this poor sysadmin, before all else: talk to a lawyer, understand what you should do from now on to minimize risk of being sued by company/shareholders, etc.
I'm a systems engineer and a software engineer. A couple tips that are relevant to this story:
1. Applications do not need DNS servers, ever. Hosting platforms do. The way I read this is that it's some odd form of split horizon where one DNS server seems to replicate to larger DNS servers, yet the single server is what carries most of the critical information.
2. State should be transferred from where you gather input to where it becomes actionable by an eventually consistent process. ie: If you store your customer CNAMEs in ERP, then something like webhooks to a service that maintains the state in machine readable form should verify, accept/deny, and do something with that change (like update DNS).
3. Customer systems and core/critical systems should be separate. The fact that critical contact systems for employees were not separate from
4. Have a DR plan and artificially enact it at cadence, doing so even in a sterile environment is better than nothing. Successful DR's in production are best. If the first time you live test your DR strategy is when all is lost, then all is most likely lost and at best you have a lot of hours of work ahead of you.
5. The holy grail is replayable audit frameworks. This could mean capturing diffs but it can be as simple as logging the current state of a given $thing and sending it to an off-local system.
I worked for a newspaper group back in the 1990s. I did all their programming in dBase III & IV LAN. I was putting in 80+ hours a week. I wasn't getting paid squat, but then they wanted to put me on salary (at less than I was currently making) and I said no way I'd give my notice first. So that's what I ended up doing.
They didn't believe me. But I left. After a few months of haggling I went back at $50 an hour on contract. I did this for a few weeks (about 25 hours a week), then started to not want to pay me. So I quit for good. Unless they made an upfront deposit.
Which they wouldn't do.
They ended up spending close to $250,000 on equipment and programmers and still couldn't do what I could. They were duplicating and tripling everything. Wasted so much money. Now I'm going to college for my BS in Computer Science with a specialty in Software Engineering. I'm currently running Windows 10 Pro Insiders Edition DEV and VMware with Kali Linux.
I keep wondering if there’s an underwriter angle here where you offer insurance and audit the company to set the premiums based on how broken their IT situation is.
If you don't have a backup of your zone files, a good way to quickly pick up the main domains being used would be to turn on logging for your DNS server and output into a log collector. That way you can quickly build queries on which IPs are asking for which FQDNs and start rebuilding a list of your lost zones and can maybe do some guess-work on which IPs those FQDNs were likely to resolve to.
I would expect irregular or extremely infrequent processes that use specific FQDNs will pop up from time to time as errors/failed processes and should prime IT teams on what to look out for.
I’m guessing the IT department reported to the CFO. Bean counters have zero IT knowledge but like to think they do. They also by default like to say no to any expenditures. Combine the two and you have a disaster waiting to happen.
IT reporting to finance/CFO is always the worst decision.
Choice 2: Put your head down and fix this. Your worth (to another company because you really should leave after you fix this) will increase dramatically.
As a system administrator, I never, EVER encrypted data at rest. Stupid policy I know, but encrypted stuff tends to be unrecoverable when you need it.
I had an Exchange Server that had issues, and I didn't have enough hardware to try a backup, I kept telling management... it died, and I was able to recover most, but not all of it.
Shortly thereafter they outsourced IT, and I assume the new crew was able to get the resources to actually fix it.
And that is why I choose to host "my company" at the big names: homepage directly with Wordpress instead of a smaller local provider, office directly with MS and so on.
Because, even there might be cheaper solutions, at least I can be sure to have a huge org in place to make sure stuff is up. And to find ways to get stuff back up and running in case something goes south. I value that peace of mind a lot.
Interesting. I choose to keep a copy of everything on my home machine (it's only personal stuff, not a company), plus backups to two removable disks, one in a fire-resistant box. Because I don't trust a big company to decide that my account is bad, or something, and I can't ever get my data back.
There would be a lot of value in independent organizations that can evaluate engineering practices. Non-technical businesses could just ask if you have cert XYZ before paying you.
It wouldn't have to get into the weeds too much. I mean really basic stuff like finding documents and following them to restore a backup or release a new version or fix a simple bug injected in the system.
> So, everything is fucked. We have (had) a custom DNS system built into our custom ERP software (don't ask). It had an integration to our old (bind) nameservers, which was stragith up awful and we've been trying to replace it for years. Well, on Friday it all broke down. All domain information was wiped out and records became null. Our company's domain is down, as well as our customer domains (we're a medium-sized MSP). Every person who knew how the custom DNS system worked has left the company years ago.
Unclear why HN wastes time on this type of 'story' with no attribution and scant details. (I have expertise in this area let's say).
To start this implies that they have not added a customer or made a change in several years by this statement:
"Every person who knew how the custom DNS system worked has left the company years ago."
This surprised me. If ever there was a use case for an organizational structure, this is surely it.. just have every manager reach out to their direct reports? Following the tree communication should happen at something like nlogn speed
> We tried really hard to make management aware and actually succeeded in that, only for management to follow up and say we don't have enough money to fix it, and they understood the possible outcomes.
Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company.
Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.