This is it an either/or: you can find the problem libraries and submit the work back upstream so you don’t have to maintain it. Then, for the upstream that don’t cooperate, fork temporarily until you find an alternative.
To their point, all of this is not trivial. You could just compile as x86 and be done with in in 5 minutes, assuming the performance hit is acceptable.
> long-term maintenance cost if upstream accepts the patch
When they said "maintain", I read it as short term, doing this if and when for all libraries down the dependency chain. Or, compile as x86, and wait until external resources push for the support.