Ok, now I'm confused, because I was under the impression that this is the first thing I ever learned in statistics - that a p value is how likely it is that a value created by chance would have been as extreme as the one the experiment gave. Is this wrong?
You're assuming an underlying model for the data. You have a test statistic( that estimates a model parameter) and you have a hypothesis regarding a parameter. The p-value is the probability that you get a test statistic more extreme than the one observed assuming that your hypothesis is true.
Ex. You have a sample of 1000 men's heights. You compute the sample average height as 5'9 and a sample standard deviation of 3 inches.
(Unlikely) hypothesis: the average height is 4 feet. Your p-value is the probability of getting an sample average more extreme than 5'9 given that your 4 ft height hypothesis is true. Given that the sample standard deviation is 3 inches and 5'9 is 7 standard deviations from 4ft... the p-value is going to be small, so you'll reject that.
People can be easily lead to misinterpret p-values even if they can define them. Most often people assume that p values indicate something about the correctness of a model or an inference. This is the classic p(d|h) v p(h|d) debate.
It's not wrong, but it did open up some great questions as to what it means to have values created by chance and how that relates to "this experiment".
