The use of students in place of professionals would have gone as #3 in my list if I had thought of it, thank you for adding it.
Generalizing from studies where all participants were students is one of the things that psychology and the social sciences have had to stop doing (except when they are studying students specifically, of course).
We are currently growing into it and may find we get the same mixed results.
I work with a lot of undergrad CS students and the difference between an undergrad senior and a second year junior developer is enormous.
That's not a problem in general, but it is if you want to test software engineering practices with students beyond just working the kinks out of your protocol.
It occurs to me that you might be able to actually run real tests, with real engineers.
Say you're in Iowa or Kansas or somewhere. You can probably hire software engineers, with 5 to 10 years experience, for $100K/year, maybe even less. Hire 10 of them, for two years. That's $2 million.
Assign them to random teams. Give them non-trivial projects that run, say, three months each. That gives you eight experiments you can run. Change the methodology, or the language, or whatever you're trying to study.
You say that you can't get funding for a $2 million experiment? Tell Microsoft, and Oracle, and IBM, and the federal government, that the things you learn will enable them to more efficiently create software. See if they'll fund at least part of it.
Generalizing from studies where all participants were students is one of the things that psychology and the social sciences have had to stop doing (except when they are studying students specifically, of course).
We are currently growing into it and may find we get the same mixed results.
I work with a lot of undergrad CS students and the difference between an undergrad senior and a second year junior developer is enormous.
That's not a problem in general, but it is if you want to test software engineering practices with students beyond just working the kinks out of your protocol.