After having read about plenty of such cases over the years, I have a persistent dread of pulling something like that myself, to the point of being nervous with ‘*’ in the terminal, and generally checking everything twice. (And also have some kind of mild horror-high from corporate snafu stories, weirdly reminiscent of Ballard's ‘Crash’).
So: I never feed the data straight from the gathering script into the modifying script, at least not in the first runs. Instead, I dump the whole list of items into a file, count them in there, gawk at them to see that they're right, and compare with the source data by hand until I begin to annoy myself. Then I feed that file to the second script.
So: I never feed the data straight from the gathering script into the modifying script, at least not in the first runs. Instead, I dump the whole list of items into a file, count them in there, gawk at them to see that they're right, and compare with the source data by hand until I begin to annoy myself. Then I feed that file to the second script.