What do they do, walk into a Google datacenter and randomly yank out a 1U server that they feel like and rip out some ethernet cables? I really don't quite understand how this works.
Any good datacenter distributes everything geographically and encrypts everything.
You're thinking too big. "Normal size datacenters" is a small industrial building with 20 racks. And yes, they walk in, shut the power and start hauling drives. No joke.
> they walk in, shut the power and start hauling drives.
Or, if they suspect disk encryption, they don't shut the power. Instead, they dissect the power cords to add a special UPS, so that they can move servers without powering them off...
That's exactly how it works in Russia. This led to some innovative services and exotic contermeasures, like locating the datacentres inside the perimeter of military factories (hard for LEOs to breach unnoticed), or installing racks inside trucks - you have time to drive them out of a nearby building while the front doors are breached.
It led me down the path of imagining two data centers: one in your basement, and one in some office building with a reception area, some cubicles, etc. It seems like the former would see the government breaking down doors and yanking servers off the rack, ripping ethernet cables. The latter would probably see a phone call or perhaps a discussion with reception, followed by a security officer or something.
The suspicious workload could be the exact same in both cases, but this is one of those neat little spots where being a Real Business has massive advantages.
Datacenters retail space, power, and cooling -- sometimes bandwidth. Data privacy is up to the tenant, but datacenters have a process just like any ISP to facilitate the execution of legal warrants.
I don't think it's the first time. Ever since all these stupid idiotic SMS confirmations became a trend they've all effectively required you to accept a {Verizon, AT&T, T-mobile} EULA.
That said though EULAs aren't law. If it's just a contract between two parties, you are free to violate it and they are free to violate it as well.
They wouldn't be in Berlin, you want to go to cheaper labor places than Las Vegas, which are plentiful in the US, and even more plentiful in Mexico if you want reasonably low latency to the US.
I'd be more concerned about the remote driver's internet connection crapping out. The car probably has multiple simultaneous cellular connections (e.g. PepLink SpeedFusion hot failover type thing).
And maybe also always-on humans, which some companies seem to ridiculously expect.
I really don't understand this obsession with 24/7 uptime for non-critical systems. Requiring your engineers to be always on-call and debug something at 3am is a health hazard and should be treated like one.
If a photo-sharing app is down at 3am, I'm sure the users can go to sleep and wait till 10am. This isn't some oxygen life support system. If you have that many users in multiple time zones, then hire people in multiple time zones.
Even if TurboTax crashes on 4/15 at 11:35pm and the engineers don't fix it until the next workday, resulting in millions of people not being able to file their taxes, I'm sure the IRS might grumble a lot but would give people an extension. It'll all be good, and everyone will get to sleep .
> Even if TurboTax crashes on 4/15 at 11:35pm and the engineers don't fix it until the next workday, resulting in millions of people not being able to file their taxes, I'm sure the IRS might grumble a lot but would give people an extension. It'll all be good, and everyone will get to sleep .
That's way too big of a risk, and way too much stress to put on your customers.
For something like tax software, you should have people on call, or even 24/7 staffing, for that specific week. 2% of the year.
In general, big release dates or important deadline should often have extra resources. 0-10 days per year. Pay extra for the health hazard, but that doesn't mean don't do it.
> For something like tax software, you should have people on call, or even 24/7 staffing, for that specific week.
In my country, the tax system (EDS, Electronic Declaration System) is down pretty much every single year on the day when tax declaration submissions start.
So basically their "solution" for the longest time was to just tell people that it's too expensive to make it have high availability and that they shouldn't use the system on the first days of the period when you can submit the data and eventually just adding a queue in front of the system to manage the concurrent users.
It seems that taxes still get handled correctly and that nobody really cares that much. Found this to be an interesting example of going against the established culture of trying to go above and beyond for availability, even if I scoffed at it a few years ago.
It definitely wouldn't be horrible to live in a world where a prod outage doesn't mean "Sorry wife, I'm not coming home today, will be stuck in some random war room for hours and then fudge up the groceries massively due to sleep deprivation" but rather "Sorry boss, the system is down, what a bummer. I'll look into it tomorrow at 9 AM." for pretty much anything aside from truly critical and time sensitive systems (e.g. air traffic control, as opposed to your music streaming app).
If it's down on the day that submissions open, then don't rush it. But when the window is closing there are thousands of dollars at stake for millions of people and I consider that pretty critical. It's not a generic outage. And it's also not unexpected. There's a lot less "Sorry wife, I'm not coming home today" when you scheduled it three months in advance.
Millions of thousands of dollars. When "health and life" is talking about whether ten people have overtime for a week, it's far less important than billions of dollars. And you can easily easily pay them enough to compensate for the stress.
And as I already said, when it comes to missing the tax deadline, leaving things broken would have a huge impact on the customers' heal and life. The total stress levels they'd feel would be enough to kill your server engineers outright.
> If a million people can't file their taxes the IRS will wait and I'm okay with that.
Even if you're right, it would cause a fuckton of stress in the people filing taxes, many of them now undergoing a significantly higher risk of cardiac arrest than the guy who's on call 1 week per year.
And if there's even a few percent chance you're wrong, the fallout would be enormous. In both money lost and even more stress.
And how many people do you think it'll take to make the IRS wait? What if you're a bit under that threshold, still with a whole lot of very stressed customers?
As long as the amount of on-call time is very small, I don't think it needs to be restricted to a super critical subset of jobs.
I feel you're missing the point. Your angle is self perpetuating. People have a higher risk of cardiac arrest at the fear of the consequences of missing the IRS date. The argument being made is that there won't really be any major consequences - if millions of people miss it because of a TurboTax issue, an extension will be granted.
Why should the engineers be stressed and overworked because other people are scared of something that doesn't have to happen?
The world is less stressful and - I think - better without manufactured urgency like what you're defending.
Don't get me wrong, some things are life and death, like life support machines. Taxes are not.
Pretend it's a smaller company. 100k people late. That's small enough to make a special exception quite unlikely, but big enough to be a lot of very stressed people. It's not self-perpetuating logic, it's how deadlines work. Letting those engineers off the hook won't solve the deadline, those people will just be told they should have done it sooner and they will suffer the consequences.
Despite not being anywhere near life or death, the stress is real. And for most people it's not crippling stress, but neither is being on call for a single week out of the year. If we're going to blow that level of on-call into a "risk of cardiac arrest" then to be reasonable we have to do the same thing for tax filing failures.
There's no way for deadlines to not be moderately stressful. You can't decide to avoid urgency and stress.
"If a photo-sharing app is down at 3am, I'm sure the users can go to sleep and wait till 10am"
There will be people, who will feel it is critical important to post some pictures at 3 am and they will get stressed, if it is not working (say people preparing an event and the pictures should be online the next day).
But .. whether that is worth that engineers must be on call, is a differnt question. I never had a job like this and I know I would never accept it as default for myself.
just before going to sleep is the only time i do hobby stuff. in the morning i have to work. putting something off to the next day always means putting it off to the next evening. i don't care how long things take. i care that i can do them right now and then forget about them. if i can't do that then i have to keep the tasks in my mind. i can write them down, but that only helps if i have a habit of doing that.
sometimes i remember some important message i need to send colleagues or clients... for that i like the telegram feature where i can send a message with a delay. i can write it now and it will be sent the next morning or whenever i think is a good time. i wish my email client had that feature too.
"If nobody's life or health is at risk, it is not urgent enough to sacrifice someone else's health for it."
I agree. But I think being on call is possible in a healthy way, if there are long enough brakes in between. Occasionally being a night on call is possible. (Well and normal if you have small kids, but having small kids and having to worry about getting an emergency call any minute, that would be draining)
not if they already committed to this one. you can't make such a change in a few minutes.
limitations are so unusual that nobody would expect them and be prepared. and even if they were known ahead of time, you are also not expecting the kind of situation where you have to upload pictures in the middle of the night.
They are pictures. They can skip them. They can run the event without it. They can print the photos on paper and hand them out. They can hand-draw posters, like everyone did in the 1700s. They can cancel the event. They can postpone the event.
if they are a business then none of these alternatives are acceptable. they could loose customers over this. and surely you are joking with cancelling or postponing an event over this. please try to be realistic. you have to consider that in our society we have come to certain expectations. these may not be ideal, but closing an online service at night is a violation of that. things would look different if everyone did that, but then i'd be the one offering night service and use it to beat the competition.
If only 10% of Tax software users are affected, IRS may not give an extension and the customers would be cursing you. Probably worth it to keep those extra engineers on call during the tax season.
And it uses a Python concrete syntax tree (with https://pypi.org/project/libcst/) to apply changes to just the docstrings without risk of editing any other code.
I disagree, I think it's absolutely astounding that they've gotten this good in such a short time, and I think we'll get better models in the near future.
By the way, prompting models properly helps a lot for generating good code. They get lazy if you don't explicitly ask for well-written code (or put that in the system prompt).
It also helps immensely to have two contexts, one that generates the code and one that reviews it (and has a different system prompt).
> You're not wrong here, but there's a big difference in programming one-off tooling or prototype MVPs and programming things that need to be maintained for years and years.
Humans also worry about their jobs, especially in PIP-happy companies; they are very well known for writing intentionally over-complicated code that only they understand so that they are irreplaceable
The challenge is that sufficiently bad code could be intentional or it could be from a lack of skill.
For example, I've seen a C# application where every function takes in and outputs an array of objects, supposedly built that way so the internal code can be modified without ever having to worry about the contract breaking. It was just as bad as you are imagining, probably worse. Was that incompetence or building things to be so complicated that others would struggle to work on it?
If your TC is 500k-1M and you don’t feel like job hopping anymore, you’d certainly not want to get hit by a random layoff due to insufficient organizational masculinity or whatever. Maintaining a complex blob of mission critical code is one way of increasing your survival chances, though of course nothing is guaranteed.
hmm I have seen conda env with far too many packages and maybe a lot of current version bumping, and the dev says "who cares" and it naturally gets a bit more.. Intentionally complicated is more like an accusation of wrongdoing.
Currently, that is LiFePO4. It is cheaper than LiPo packs used in electronics, half the energy density, twice as many charge cycles, and doesn't burst into flame. The lithium is flammable but requires external ignition.
Larger batteries, including some electric cars, have switched.
It seems unlikely that there's any practical chemical batteries with 0 fire risk.
But I do think there should be home energy storage that doesn't involve chemical batteries. Where are all the pumped hydro, flywheels, and compressed air storage for consumer use?
There’s no perfectly safe energy storage. The danger comes from the concentration of energy. Water can cause flooding or you can drown in it. Flywheels can disintegrate into shrapnel. It’s always risk management.
We're talking on the order of millions of kilograms for the building materials that needed to be transported to build it. The batteries needed for backup power for its occupants won't come anywhere close to that, even at far lower energy density than lithium.
I've survived ventricular fibrillation 3 times and have an implanted ICD. I never really understood why delivering high voltage shocks is that effective, and always have anxiety about "the time it won't work"
> I have 192GB of CPU VRAM in my desktop and that was cheap to obtain.
How? Or what's "cheap" here? (Because I wouldn't call 192G of just regular RAM that's plugged into the motherboard cheap, I think everything else is more expensive, and if there's some hack here that I haven't caught I very much would like to know about it)
Which is pretty cheap compared to the cost of my whole build and whatever other things I've spent on. Cheap is relative, but I'm just saying that if you're going to spend $3000+ on a build, and you love to work with massive datasets, VMs, and things, $500 for a metric fuckton of RAM so that your system is never, ever swapping, is a very worthwhile thing to spend on.
192GB worth of GPU will cost you about $40000, for reference, and will be less performant if your goal is just a vramfs for CPU tasks.
* Beware that using 4 DDR5 slots will cut your memory bandwidth in half on consumer motherboards and CPUs. But I willingly made that tradeoff. Maybe at some point I'll upgrade to a server motherboard and CPU.
Couple of reasons.
1. You can use vram when you don't have massive amounts of ram for a ramdisk (or /dev/shm)
2. Depending on implementation, you might have faster random seek/write than normal ram.
3. You could presumably run certain gpu kernels on the vramfs.
What do they do, walk into a Google datacenter and randomly yank out a 1U server that they feel like and rip out some ethernet cables? I really don't quite understand how this works.
Any good datacenter distributes everything geographically and encrypts everything.
reply