Maybe he was bluffing because he knows dead hostages are worse than live hostages?
If you can stomach Quarich being okay with being reborn into the thing he hates the most, or everyone important that died in A1 being reborn, or the fact humans don't simply nuke the Navi from the orbit, or the fact that unobtanium is not as important as miracle whale oil, .... then Quarich bluffing and being called on it and then trying to repeat the bluff is not SO far fetched.
If you can stomach Quarich being okay with being reborn into the thing he hates the most, or everyone important that died in A1 being reborn, or the fact humans don't simply nuke the Navi from the orbit, or the fact that unobtanium is not as important as miracle whale oil, .... then Quarich bluffing and being called on it and then trying to repeat the bluff is not SO far fetched.