what was the basis and the motive for ally countries unilaterally not to allow the plane to land? If the motives are not different, then the fighter jet could be seen as a similar "technical reason" in this case too.
It was US pressure obviously, for which the countries apologized. It was an asshole move by the US but this is on another level, it involves foreign civilians, KGB agents, fake bomb threats and fighter jets
Mental gymnastic you are doing there. Spain and France denied the airspace, meaning the flight couldn't gone through either of them. There is no "not to allow the plane to land". In fact an airplane calling emergency has priority to land anywhere, regardless of the political difference with the host country.
On this topic, this happened with a flight from a US's air base in Afghanistan, rerouted to Iran because of a "bureaucratic issue":
Quite contrary, I'm asking for the basis and the motive of denying one specific flight with one specific individual on board a routine procedure of air travel, something that was allowed to hundreds of other flights on that same day. "landing" vs "airspace" is a non-essential detail here, and depending on the context of a situation, either could be denied to achieve the desired goal. I'm asking about the motives of the goal.