A text file can contain a malicious payload in exactly the same way a binary file can.
If you want to get paranoid about a test subdirectory, stash the payload in comments or formatting choices in the source code itself.
It's great to say "aha, this particular exploit would be deleted by not having binary files in a test directory!", but that totally misses the point that you can put the bytes you want in other places, such as in the source code which is itself a load of bytes. See also polyglot programs and the code is data premise.
What point are you arguing here? More binaries rather than less? Less readable source code rather than more readable? That we should just give up because some people manage to win underhanded C contests? Are you arguing that binaries are equally easy to understand and dissect as source code that has to follow some syntax?
folders containing test files could easily be removed before build just by name or should live in a separate repository altogether.