I actually looked at this in detail about a year ago for some automated driving compute work at my previous job, and I found that the detailed info you'd want from Nvidia was just 100% unavailable. There's pretty good proxies in some of the data you can get out of Nvidia tools, and there's some extra info you can glean from some of the function call stack in the open source Nvidia driver shim layer (because the actual main components are still binary blob, even with the "open source" driver), but over all you still can't get much useful info out.
Now that Brendan works for Intel, he can get a lot of this info from the much more open source Intel GPU driver, but that's only so useful since everyone is either Nvidia or AMD still. The more hopeful sign is that a lot of the major customers of Nvidia are going to start demanding this sort of access and there's a real chance that AMD's more accessible driver starts documenting what to actually look at, which will create the market competition to fill this space. In the meantime, take a look at the flamegraph capabilities in PyTorch and similar frameworks, up an abstraction level and eek what performance you can.
I just sent the link to a driver developer at Nvidia. If he shares the link with others at Nvidia, they should become aware of the idea tomorrow. That said, I have no idea if he will do that, but at least I tried.
The problem is that CUDA already does that, and they're not incentivized to really improve from that baseline, given the capability and ease of use of the ROCM or Intel solutions.
I'm not sure, it seems to me like this should be doable in Nvidia as well. This is a paper that uses instruction sampling (called CUPTI) in Nvidia to provide optimization advice:
The issue there is that that info is what Nvidia chooses to port out from the on-chip execution. Most of what we can do for observation is in the kernel driver space and not really on-chip or even low level transit to the chip. One of the other commenters pointed out that you can get huge benefits from avoiding busy waiting on the returned data from the chip, which makes total sense, but also increases latency, which didn't work for my near-realtime use case when I was investigating. Other than those types of low hanging fruit where you can accept a little latency for better power state management, it's hard to find low level optimizations specifically for Nvidia through the closed source parts of the CUDA stack or through the driver transit to chip when those are intentionally hidden.
A while ago, I read a paper on dissecting the Nvidia architecture using very specifically tuned microbenchmarking to understand things like cache structure on chip and the like [0]. Unfortunately, no one has done this for seriously in use, recent architectures, so it's hard to use this info today. Similarly, there isn't an eBPF VM running on the chip to summarize all of this and the Nvidia tools aren't intended to make this kind of info easy to get, probably specifically because of this paper...
I have two vehicles, my girlfriend has another one, and despite living in one of the higher insurance premium areas in one of the highest insurance premium states in the country (Oakland County, Michigan), our total bill with full coverage and relatively low deductibles is around $7k/yr. A family with 4 cars total (two driving age kids, two parents with cars to drive to different jobs) should not be massively more expensive, likely only $3~4k more than my bill, making it about half of the headline number. To do worse than that, you'd have to live in the worst part of inner city Detroit with the highest insurance rates and have full coverage or massively more expensive vehicles... I don't think the math checks out, really.
I do know people who have very very high bills, but they have one of 1) very high end cars (911 GT3 RS, Bentley, etc.) 2) something rare insured at full value of its rare nature or 3) actively stating to the insurer that it's a track car being raced on the weekends. All of those happen around me, but those aren't the modal case.
This is an interesting move given Hyundai's position in funding Motional, who didn't seem totally incapable of creating AV tech. Does anyone know what their future looks like after this partnership?
I generally agree with this, as a small-L libertarian. However, the results of the Philips Respironics problems killing people and badly damaging lungs would indicate that we need some amount of better tracking and observation of things like this. See this old HN discussion: https://news.ycombinator.com/item?id=39223982
That's a fine interpretation, but the observable evidence lately would indicate otherwise. The two terms to look into are "institutions as platforms" from Yuval Levin at the American Enterprise Institute [0], describing congressional representatives who spend more staff budget on comms than policy and more of their time on cable news hits than on committee meetings. The second is "public choice theory" economics which is the branch of economics describing why policy doesn't align with stated goals on a regular basis. Robin Hanson had a good substack on the second one just today, in fact [1].
Area Code 810 is the Thumb of Michigan, including some of central Michigan, so it's not all 8XX. Unsure how they draw the line, but 800 and 888 are the two that I see businesses use for toll free calling.
The funny thing is that they supposedly only made a couple million from this, total. I can't remember if that was an internal discussion from when I worked there a few months ago or a news article, but I'm pretty sure of it either way. That makes the whole stupid thing just dumb.
Software (controls) engineers at VW during the emissions scandal went to jail, engineers at GM were held liable for the ignition switch issue (not mostly in software, but still). I expect we'll eventually see some engineers/low level managers thrown under the bus with Boeing. It definitely happens, but not as frequently as it could. That said, I definitely prefer Amazon's response to the AWS East 1 outage back in 2016 -- the engineer wasn't blamed, despite the relatively simple screw up, but the processes/procedures were fixed so that it didn't happen again in the last 8 years. Crowdstrike is a little bit gray on that regard -- people should have known how bad the practice of zero testing on config updates was, but then again, I've seen some investigating saying that the initial description wasn't fully accurate, so I'm waiting for the final community after action report before I really pass judgement.
The General Motors CFO basically said as much on an investor call in 2022 or 2023. That didn't get rid of enough people, so now they've implemented stack ranking because that worked so well at MSFT. That said, they get a "free" 5% workforce reduction every year that they want it, no severance, no buyout, just let go. There will be more of that coming, particularly in US automotive after the UAW deal and the current sad state of vehicle sales and the write downs on EVs.
The sad state of vehicle sales has mostly to do with the outrageous increases in new car prices that go way beyond transmission of supply chain costs and dance right on in to price gouging territory. The prices still have not corrected to something sane according to the actual inflation curve.
The average price of a new car these days is $48K. This is insane. For that price tag to make any kind of sense at all, you have to have an income of nearly half a million per year. Median household income in the U.S. is around $75K meaning most of these households, if they're even purchasing a vehicle at all, would be buying used.
It would mean financial ruin to borrow two-thirds of your income for a car. The automakers have forgotten Henry Ford's lesson. He payed his workers enough so they could each afford the products he was selling. Our companies don't pay their employees enough to afford their products any more. <Surprised Pikachu!> Sales are down.
1000% agreed -- I worked for Ford and GM and now work for a different vehicle software company, and the salary I made starting out + the employee discount meant that vehicles were affordable, but not fun to afford. Now, I can't upgrade my F150 to something with more towing capacity and couldn't even while working at GM with their employee discount and somewhat lower truck prices in general compared to the other Big Three. The problem is corporate culture related, but it's also contributed to by the safety regs requiring every car to have a bunch of driver assist things that drive up total cost even on low end cars, meaning that the mid and high end need to be super profitable for overall fleet profit to be acceptable.
This comment laments about the rising prices of cars, but it doesn't offer anything to explain why they're getting more expensive.
"Greed" isn't the answer - we have greedy companies in every area of the economy, yet somehow the prices of phones, computers, desks, large appliances, televisions, and many other types of goods hasn't skyrocketed in the same way as cars.
What makes cars different? I know that the cost of labor has significantly increased (which partially motivates the stratospheric increases in education and healthcare costs), but I thought that cars were mostly manufactured in an automated manner.
Several things that aren't regularly talked about: to meet increasingly stringent emissions standards without massive losses in engine power (and resulting customer dissatisfaction), companies spend heavily on powertrain R&D to eke the remaining emissions improvements out (99.9% reduction to 99.99% reduction). To meet increasing safety standards, manufacturers have to have much more complicated base models than before, which drives up even the lowest car prices -- backup cameras, automatic emergency braking cameras and logic, immobilizer standards, etc. Then, for the products that everyone wants, some manufacturers aren't even making the new CAFE standards, so they buy "indulgences" (EV offset credits) from Tesla, which is literally a regulatory tax for the incorrect outcomes, but customers don't want GM and Ford EVs at the moment. Beyond that, some solutions to the above problems (hybrids) are simply insanely expensive (ballpark $20k extra manufacturing cost for a hybrid powertrain on some high end hybrid SUVs vs the identical engine without the electric components). All of that shifts parts of the curve to the right, some of it only shifts the high end, some of it shifts the entire curve.
Auto company profit margins provide further evidence supporting your hypothesis (modern car requirements are just inherently more expensive).
The auto companies whose profit margins I checked are in the single-digits, sometimes negative. If it was just greedy price gouging, we'd expect them to see much healthier profit margins.
I don't understand why people don't check these figures more often for these publicly traded companies when offering the 'companies are being greedy' narratives.
100% Agree, We could have $10k electric cars in the US, specifically made in China. It's not going to happen anytime soon for enough reasons that could fill a HN comment section.
What are the basics? The "front end" (demand side) is that cars are effectively subsidized by cheap and widely available financing, but at a high level where is that money actually going on the "back end" (supply side) that makes cars expensive to manufacture?
How is that average calculated though? If everyone switches from buying the $40k Camry to the $85k F150, prices didn't have to change for the "average price of a new car" to increase. This isn't a dismissal since we can compare every model year to the previous and see an increase YoY in prices without serious improvements to the underlying product. We have also seen everyone going from sedans to trucks though.
>It would mean financial ruin to borrow two-thirds of your income for a car. The automakers have forgotten Henry Ford's lesson. He payed his workers enough so they could each afford the products he was selling.
I agree but the problem is that business owners don't like how much money comes from that system and prefer competing for people's lines of credit, because an economy where everyone has $40k in debt literally has more money sloshing around it than one where everyone buys their car outright, and that means more profit, which is the entire goal of every CEO everywhere, so of course every CEO wants you to go into debt for them. Unfortunately, a lot of people can leverage themselves farther than they can actually afford for long enough that some CEO can extract profit before the entire system collapses.
A shitload of profit and CEO salary was made in the run up to 2008, and none of the people who extracted that wealth from the system went to jail, so why are we surprised they're trying to do it again?
Scientology managed to distill this down to the bones; They simply had people sign up for credit cards and max them out buying scientology stuff. Since the penalty for debt goes to the people who took it, instead of the place the money the debt went, there's infinite incentive to do this. CEOs look at that system and get extremely jealous.
> I agree but the problem is that business owners don't like how much money comes from that system and prefer competing for people's lines of credit, because an economy where everyone has $40k in debt literally has more money sloshing around it than one where everyone buys their car outright
It's even simpler than that, and "greed" doesn't come into the equation anywhere: any market where goods/services are subdized will lead to more expensive goods and services, by simple economics.
Even industries where companies are "good", and try to provide quality goods to customers at an affordable price point (while still turning a small profit) will see this happen, because the excess capital will push the apparent "affordable" point up - its effective, unwitting collusion for entire markets.
This is happening in healthcare (insurance that has transformed from being actual insurance in case of catastrophic injury, into cost-sharing), home prices (cheap mortgages), cars (cheap financing), and education (cheap student loans).
Debt needs to be more expensive. Insurance needs to only be for catastrophic events.
I listened to the entirety of the investor call and the GM CFO definitely does not call it a "free" workforce reduction.
The stack ranking is primarily about rewarding high-performing employees with higher bonuses. It also puts a low-performing employee at risk of getting let go, but it doesn't magically absolve GM of its unemployment law obligations, because any layoff pursuant to this planned workforce reduction would be grouped together and at GM's size the 5% reduction would trigger WARN Act obligations long before they got to 5%.
Seriously guys: if stack ranking were a magical way to avoid the WARN Act or other unemployment laws every company would be claiming that layoffs were performance based. That no company tries to claim performance as the basis for avoiding severance should tell you something about how the law works...
There is no entitlement in any country/state/province I've ever worked. You can quit at any time and your employer can fire you at any time - no reason needed either way
They have a new performance rating system that says that 5% of every team should be ranked as "needs improvement" meaning an automatic PIP. When MSFT did this in the 2000's, it was a requirement to fire that group. Firings for performance do not entitle an employee to severance, but official "layoffs" generally do outside of bankruptcy. Net cost of this move is hidden in employees spending time updating resumes and doing job interviews, not in direct bottom line dollars from actually paying out severance.
The other hidden cost is the paranoia and backstabbing that is encouraged when someone in every team has to be fired even if everyone did objectively excellent work.
What happens if stack rank->pip is too obvious is that people and teams will not cooperate with newer fresh meat. They're incentivized to keep knowledge protected for their job. New people struggle to get above the pip mark and it becomes a revolving door.
Firings for performance do not entitle an employee to severance,
This keeps getting repeated on this thread but its not true. U.S. WARN laws require severance or 60 days notice for mass layoffs (which would definitely include 5% of a company as large as GM). Only terminations for cause are exempted, and performance is only grounds for cause in a very small handful of "right to work" states.
No, my point is that it's irrelevant whether they're underperforming or not; what matters for WARN purposes is the amount of workers being laid off.
Only for-cause terminations are excluded the WARN Act, and for-cause in this context means specifically behavior that violates the law, or the employee handbook setting forth company rules (which are binding on both the company and employee, which is why companies make you read them every year). Also note that companies can't make poor performance a violation of company rules; this has been tried before and smacked down pretty harshly even in right-to-work states.
Thus, those 5% laid off due to stack ranking are entitled to 60 days of severance, or 60 days notice of an impending termination.
You're missing the part where the WARN Act doesn't require the employees to be let go all at once.
And the part where rolling layoffs at a company the size of GM will all involve multiple layoffs exceeding the WARN threshold.
And the part where no general counsel will let their company knowingly subject itself to a $500 daily fine per employee, in addition to full back pay for the mandated period.
And the part where GM satisfies its WARN obligations by paying the mandated severance (or more) in lieu of providing advance notice...which is what most companies do...
Which is exactly why they will try to bully you into quitting rather than fire you in jurisdictions with strong firing compensation/protection regulation.
Severance is a "we'll pay you to leave quietly" deal, not an entitlement. If the company plans to be continually laying people off (kind of a stupid idea because it makes your other employees nervous) they can find other ways to ensure silence from their victims.
perhaps this 5% is employees who were documented underperforming, breaking rules, getting into conflicts, gaming at work, etc. ehe company has every right to let them go without severance. usually referred to as "regretted attrition" which is ironic in this case because GM doesn't regret them leaving in the slightest.
Pitting your employees in a hunger games-esque charade is sure to have unintended consequences.
Surely, if I was an employee here, self-preservation would be of the utmost importance. If I fix something to make my job easier, I will keep it to myself. If I have the opportunity to sabotage a teammate, I would. If I can lie easily, I will.
And in my experience you run out of those individuals rather quickly and start cutting well-intended employees after only a few years. After that, you start firing hard working individuals who can’t or won’t play the game.
After that, you’re selecting for corporate psychopaths.
Not who you asked, but where I live, the law prescribes three levels of severance pay (one to three months of pay, depending on your time with the company) for all layoffs. Obviously not if you get fired.
None of that support staff is constitutionally allowed to make the call to respond to an imminent or ongoing attack. We need a commander in chief and we need to know who that person is -- which was the whole point of the Presidential Succession Act and the 25th amendment. This isn't even a Biden thing -- we had similar issues with Wilson, FDR, Kennedy and Reagan, and I'm sure that some administrations had similar issues that were never reported or written about afterwards.
Now that Brendan works for Intel, he can get a lot of this info from the much more open source Intel GPU driver, but that's only so useful since everyone is either Nvidia or AMD still. The more hopeful sign is that a lot of the major customers of Nvidia are going to start demanding this sort of access and there's a real chance that AMD's more accessible driver starts documenting what to actually look at, which will create the market competition to fill this space. In the meantime, take a look at the flamegraph capabilities in PyTorch and similar frameworks, up an abstraction level and eek what performance you can.