Indeed, it should be the actual use that governs this. If you use it as a passenger vehicle, it's a passenger vehicle. If you use it on a farm then it's a farm vehicle.
There's already special registration for farm vehicles in a lot of states. The taxes are lower and that's one of the few cases where you can use pink diesel, which is also taxed differently.
Trucks used as passenger vehicles are already more expensive to register and run than farm vehicles.
