But only a small portion of the interchange fee goes to the credit card processor. Most of it goes to the bank.

How else do you think credit cards are able to do things like pay 2% rewards?

So the argument still holds. Banks hold more risk on larger transactions.