It's true in the informal sense. Normal people, when considering an "infinitely" bad thing happening (being killed, losing their home, etc) with a very low probability will round that probability to zero ("It won't happen to ME"), multiply the two and resultantly spend zero time worrying about it, planning for it, etc.

For instance, a serial killer could kill me (infinitely bad outcome) but the chance of that happening is so tiny I treat it as zero, and so when I leave my house every day I don't look into the bushes for a psycho murderer waiting there for me, I don't wear body armor, I am unarmed, I don't even think about the chance of being killed by a serial killer. For all practical intents and purposes I treat that possibility as zero.

Important to remember that different people gave different thresholds at which they round to zero. Some people run through dark parking garages and jump into their car because they don't round the risk of a killer under their car slashing their achilles tendons down to zero. Some people carry a gun everywhere they go, because they don't round the risk of encountering a mass shooter to zero. Some people invest their time and money pursuing spaceflight development because they don't round a dino-killing asteroid to zero. A lot of people don't round the chance of wrecking a motorcycle to zero, and therefore don't buy one even though they look like fun.

The lesswrong/rationalist people have a tendency to have very low thresholds at which they'll start to round to zero, at least when the potential harm would be met out to a large portion of humanity. Their unusually low threshold leads them to very unusual conclusions. They take seriously possibilities which most people consider to be essentially zero, giving rise to the perception that rationalists don't think that infinity * 0 = 0.

> It's true in the informal sense. Normal people, when considering an "infinitely" bad thing happening (being killed, losing their home, etc) with a very low probability will round that probability to zero ("It won't happen to ME"), multiply the two and resultantly spend zero time worrying about it, planning for it, etc.

Is this the kind of thing that is part of the Less Wrong cult? I see this "multiply" word being used which I understand is part of the religious technology of LW. It all seems very sophomoric. I don't know what talking about "Infinity * 0" means in an informal sense means. What I can tell you is that "Normal people" are not multiplying "infinitely bad" with a "very low probability rounded to 0". For one, this is conflating multiple senses of infinite. I'm not sure anyone thinks likes bad outcomes are "infinitely bad", maybe in a schoolyard silly-talk kind of way, they just think it is bad. I think that's basically what Less Wrong is, a lot of fancy words and Internet memes and loose-talk about AI all strewn together in a "goth for adults" or some other kind of nerd social club.

> "I don't know what talking about "Infinity * 0" means in an informal sense means"

I'm not a rationalist, I'm only using their language to make the mapping to their ideology simpler. A comet striking earth would be "infinitely bad". The chance of that happening is, as far as I'm concerned, zero (its not zero, but I round it down.) If you multiply the infinitely bad outcome by the zero percent chance of it happening, you result is that you shouldn't waste your time and emotional resources worrying about it.

Normal people don't phrase this kind of reasoning with math terminology as rationalists do, but that terminology isn't where the rationalists go wrong. Where the rationalists go wrong isn't the multiplication, it's the failure to ignore very unlikely outcomes as normal people would. They think themselves too rational to ignore the possibility of unlikely things, but ironically it is normal people who don't spend their time dwelling on extremely unlikely bullshit have a more rational approach to life.

The rationalists spend hours discussing scenarios like "What if a super AI manipulates people into engineering a super virus that wipes out humanity? Its technically possible; there's no law of physics which prevents this!", to which a normal person would respond by wondering if these people are on drugs, why would they spend so much time worrying about something which isn't going to happen?

Yeah pretty much. If I was to write it out further: "near infinity bad thing could happen but it has a near infinitesimal chance of it happening, what is the amount of finite resources you should spend to prevent it?". The numbers are probabilities and how much of an effect it is. It really is infinity * epsilon but that would confuse more people so I decided to say infinity * 0.

I was very explicit when I said "humanity decided". It doesn't matter if one or the other is the actual formal system math system result either way, it was chosen out of practicality that in this kind of philosophical issue, the more pragmatic thing was to axiomatically choose that "infinity * 0 = 0" when faced with things like this. The rationalists in a more meta/broader sense have decided that it's infinity * epsilon = infinity even if they say it is not on the surface. Their actions show they believe the other direction.

In math infinity * epsilon is indeterminate until you decide what the details of infinity & epsilon is, which I find quite fitting.

> That humans regard the expected value of (arbitrarily large impact) * (arbitrarily small probability) as zero?

There are many arguments that go something like this: We don't know the probability of <extinction-level event>, but because it is considered a maximally bad outcome, any means to prevent it are justified. You will see these types of arguments made to justify radical measures against climate change or AI research, but also in favor space colonization.

These types of arguments are "not even wrong", they can't be mathematically rigorous, because all terms in that equation are undefined, even if you move away from infinities. The nod to mathematics is purely for aesthetics.