Actually it's not (edit: only / mainly) a question of venue, it's a question of personal jurisdiction as defined by Rule 4(k) of the Federal Rules of Civil Procedure.
The general rule is that a federal court only has personal jurisdiction where a state court of general jurisdiction would have it. So both the constitutional due process requirements (International Shoe, et. al.) and the state long arm statute come into play. (This assumes the patent statutes don't have unique PJ rules, they might.)
All that said, without having actually looked into the case law, I'd imagine you couldn't defeat jurisdiction by not doing business in the forum because the Calder effects test would be construed to apply to infringing a patent owned by an entity in the forum.
The general rule is that a federal court only has personal jurisdiction where a state court of general jurisdiction would have it. So both the constitutional due process requirements (International Shoe, et. al.) and the state long arm statute come into play. (This assumes the patent statutes don't have unique PJ rules, they might.)
All that said, without having actually looked into the case law, I'd imagine you couldn't defeat jurisdiction by not doing business in the forum because the Calder effects test would be construed to apply to infringing a patent owned by an entity in the forum.