It is hard to emphasize how bad overfitting on the test set is. Your cheating analogy is accurate, and cheating in science is a serious matter. If it is unintentional, you should have your work rejected or retracted. If it is intentional, you should have your work rejected or retracted and, frankly speaking, you should most likely leave the field.
Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting.
A good short description on several issues that plague the field is "Clever Methods of Overfitting".
Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting.
A good short description on several issues that plague the field is "Clever Methods of Overfitting".
http://hunch.net/?p=22