Ultimately, a country can regardless of what any other country says... ban a company from doing business within their borders, confiscate property within their borders, arrest employees within their borders, and prohibit employees or products from entering their borders.
Arguably, tech companies could just disregard GDPR, at the expense of all business, offices, and access to the European Union. However, if they want to continue to have that access, they need to comply with the EU's laws, even if someone could argue that those laws extend in dubious ways outside the borders.
The thing about national sovereignty, of which each EU member state has, and to some degree, the EU as a whole is capable of exacting, is that you can't decide whether you think their laws are reasonable or not, you can only decide if you're willing to comply in exchange for access to that market.
Bear in mind, the US literally tells people "you can't do business in Iran", and everyone accepts it, because the alternative of "you can't do business in the US" is way worse for business.
You're right in that international law isn't real in the same way that national laws are. But countries do consider international law in determining whether to give effect to another country's actions. If EU member states started seizing assets of businesses that it applied the GDPR to in violation of international laws, other countries would take that as a hostile action and possibly stop recognizing EU member state judgments, for example, as presumptively valid. And they can take retaliatory actions against European companies.
Customary international law is best thought of as widely accepted norms of fairness between sovereign states. There is no international police to come get you when you don't play nice, but staying within the dictates of international law generally protects you from accusations of unfair play and the consequences that can bring.
And yeah, the US maybe does violate international law in the specifics of its sanctions policies, though they are usually carefully crafted to only apply to people who somehow participate in the US financial/payments system. It's very possible that one day the US will pay the price for this overreach/maximalism when countries eagerly jump ship to an alternative international financial system, should a viable one emerge. But for now, the US is the only real game in town. The strong take what they can, and the weak suffer what they must.
Arguably, tech companies could just disregard GDPR, at the expense of all business, offices, and access to the European Union. However, if they want to continue to have that access, they need to comply with the EU's laws, even if someone could argue that those laws extend in dubious ways outside the borders.
The thing about national sovereignty, of which each EU member state has, and to some degree, the EU as a whole is capable of exacting, is that you can't decide whether you think their laws are reasonable or not, you can only decide if you're willing to comply in exchange for access to that market.
Bear in mind, the US literally tells people "you can't do business in Iran", and everyone accepts it, because the alternative of "you can't do business in the US" is way worse for business.