> “We found replicated evidence across multiple experiments that playing our games for two hours causes improvements in executive function skills as compared to a control group that plays an unrelated game,” said Mayer.
Christ what a low bar. That's like having your treatment group do jazzercise while the control watches TV, and then concluding that jazzercise is the most effective form of physical activity.
Your control group should be performing some sort of non-gaming intellectual exercise: reading a passage and answering some questions, having a group discussion on a serious topic, etc.
The joke when I was in academia was that there were two ways to get lots of citations: publish a state-of-the-art model that gets cited because it's state-of-the-art, or publish a bad model that gets cited a lot because everyone wants to show how much they beat it by.
Also worth questioning whether the test was sufficiently far removed from the game itself that improved scores mean anything. Unfortunately the article is paywalled, but here's a quote from the abstract:
> Students who played Alien Game at a high level of challenge (i.e., reaching a high level in the game) developed significantly better performance on cognitive shifting tests compared to controls when they played for 2 h (Experiment 1, d = 1.44), but not when they played for 1 h (Experiment 2).
Probably it was a card sorting test. This sounds to me like "people who practiced card sorting got better at card sorting."
I have access to the article [1] through my university, here is the description of the procedure:
The study took place in the students' classroom, once a week over an 8 week period. [...] On the first week, demographic questionnaires and the computerized DCCS and Flanker tasks were administered. The Alien Game was then introduced to the students [..] They then played through the first, tutorial level to learn the basics of the game. For the next 6 weeks, students were asked to sign in and play for 20 min per week. Each week, the next two levels of the game were unlocked. [...] On the final week, the DCCS and Flanker tasks were administered again, and students were given a posttest questionnaire asking about their experiences with the game.
So if I understand correctly, that was:
- 1st week: just the questionnaire and test, plus demo
- weeks 2 to 7: play the game 20 minutes
- week 8: test again
So it seems the performance improvement was measured one week after the last game session.
Christ what a low bar. That's like having your treatment group do jazzercise while the control watches TV, and then concluding that jazzercise is the most effective form of physical activity.
Your control group should be performing some sort of non-gaming intellectual exercise: reading a passage and answering some questions, having a group discussion on a serious topic, etc.