Sure, but in this context, unless you are able to use the same source port for TCP and UDP (and likely from the load balancer to the application server too) you'll still need another way to identify a client/session when switching from TCP to UDP.
Doing that with NAT is even trickier. Take a look at the way some firewalls need to configure a DMZ for gaming, or SIP for some examples.
I'd like to point out that getting SIP through firewalls is a massive headache even today. If you're embedding connection information into a control channel, then the firewall needs to do DPI to figure out what is a valid communication stream vs. some attacker. And then people start encrypting the control channel and it's game over unless you can hook your firewall up to whatever the controlling software is or your firewall is super fancy and can MITM the control channel traffic because you've installed the cert chain on the firewall.
Doing that with NAT is even trickier. Take a look at the way some firewalls need to configure a DMZ for gaming, or SIP for some examples.