why do they always happen to always be inaccurate on the bad side these days?
I don't have time to find it right now, but I remember reading a study a few years ago that showed most projects to in fact come in on time and on budget. It's the few major projects that make the headlines that have schedule & budget problems, and (to no surprise) those tend to be the projects that have to deal with the most complexities. It's not different than large software projects.
On the other hand, they also had serious challenges that we don't meet
Sure, but they could work around those problems by (literally) throwing bodies at that project and dumping crap into rivers. And some of those things you indentifiy as problems ("very bad transportation network," "less educated workforce") aren't really problems - "less educated" means "cheap": the project just needed to train you to hold a rivet in place or swing a hammer, today's workforce needs to know how to manage complex diesel equipment. "very bad transportation network" means "more room to work": building the Hoover Dam in the boonies is in many respects much less difficult than building the 2nd Avenue subway under a crowded Manhattan or building the Big Dig under a busy Boston. In fact most of the high cost of CA HSR isn't from building the sections in the boonies (which is why they're starting construction there first), it's building the connections into the heavily populated metro areas like San Francisco and LA.
If you ever have the chance (and you're an American), go to a public meeting organized by a huge capital project. Listen to all of the comments (ranging from measured to angry, reasonable to crazy) and keep in mind that the project has to address all of those concerns. That will give you a sense of scale of the problem. In fact I spend this past weekend dealing with concerns that frankly defy the laws of thermodynamics, but the project has to address, which means I spend billing hours debunking it.
edit: enko makes a good point, you could go ulta-conservative with cost estimation, but then every highway widening project becomes a $100 billion undertaking and nothing would ever get done. Projects to set aside 5-15% for continengy and most of the time that's enough. Of course there are outliers, and those are what tend to get attention. You also have times were a project may also set a relatively low budget, but they get back bids from contractors and/or engineering firms that way exceed the budget, so the budget gets increased.
I would like to see that study saying most projects come in under budget. I'm guessing it doesn't apply to the large and complex projects we're talking about here, which are probably harder to estimate. And, the fact that it's in California may matter, too. According to Musk yesterday, without a citation:
I don’t think we should do the high-speed
rail thing because it’s currently slated
to be roughly $70 billion but if one ratio
is the cost at approval time versus the
cost at completion time of most large
projects I think it’s probably going to be
close to $100 billion.
...the lowest (and winning) bidder for early work on the California HSR has a history of cost overruns, so things aren't starting out so hot.
Your last paragraph seems to say that we don't want accurate estimates because then we might not do the projects. That's the point of knowing the costs. We don't want to do projects that are too expensive.
Large projects, like Hyperloop or HSR, are probably the ones particularly likely to be "outliers" (if indeed projects with overruns are outliers) and big projects are the ones where that damage costs the most, so it's a double whammy.
Meanwhile, if we're now accepting that lowballing is a good idea, as your last paragraph says, then I don't understand why you seem to be defending the original article as Musk must just be lowballing and he expects overruns. It will take a lot of overruns to catch up with the ostensible California HSR budget.
I will say that, with SpaceX, Musk and the government have fixed-price contracts, so SpaceX eats cost overruns. They also have the lowest space cargo delivery cost for their class. This gives me some evidence that Musk could deliver more cheaply and on budget than the linked industry observer expects.
Your last paragraph seems to say that we don't want accurate estimates because then we might not do the projects.
No, my last paragraph says we don't want ultra-conservative estimates because we would definitely not do the projects.
Meanwhile, if we're now accepting that lowballing is a good idea, as your last paragraph says
My last paragraph doesn't say that.
This gives me some evidence that Musk could deliver more cheaply and on budget than the linked industry observer expects.
Perhaps. But if I were a betting man, I think Must will be in for a rude awakening upon discovering the mess that is the public environmental process, as have many other entrepreneurs before him.
I don't have time to find it right now, but I remember reading a study a few years ago that showed most projects to in fact come in on time and on budget. It's the few major projects that make the headlines that have schedule & budget problems, and (to no surprise) those tend to be the projects that have to deal with the most complexities. It's not different than large software projects.
On the other hand, they also had serious challenges that we don't meet
Sure, but they could work around those problems by (literally) throwing bodies at that project and dumping crap into rivers. And some of those things you indentifiy as problems ("very bad transportation network," "less educated workforce") aren't really problems - "less educated" means "cheap": the project just needed to train you to hold a rivet in place or swing a hammer, today's workforce needs to know how to manage complex diesel equipment. "very bad transportation network" means "more room to work": building the Hoover Dam in the boonies is in many respects much less difficult than building the 2nd Avenue subway under a crowded Manhattan or building the Big Dig under a busy Boston. In fact most of the high cost of CA HSR isn't from building the sections in the boonies (which is why they're starting construction there first), it's building the connections into the heavily populated metro areas like San Francisco and LA.
If you ever have the chance (and you're an American), go to a public meeting organized by a huge capital project. Listen to all of the comments (ranging from measured to angry, reasonable to crazy) and keep in mind that the project has to address all of those concerns. That will give you a sense of scale of the problem. In fact I spend this past weekend dealing with concerns that frankly defy the laws of thermodynamics, but the project has to address, which means I spend billing hours debunking it.
edit: enko makes a good point, you could go ulta-conservative with cost estimation, but then every highway widening project becomes a $100 billion undertaking and nothing would ever get done. Projects to set aside 5-15% for continengy and most of the time that's enough. Of course there are outliers, and those are what tend to get attention. You also have times were a project may also set a relatively low budget, but they get back bids from contractors and/or engineering firms that way exceed the budget, so the budget gets increased.