We had this too. It somewhat frustrated me to be able to understand the theory and apply it but not being "good" enough in the eyes of the PE teacher. Excellent grades in other subjects though, just didn't see the fun in running after a ball or the worth of investing time and effort at being faster or better at it. College change that. PE was stil mandatory in the cursus but was more oriented towards a healthy lifestyle. Benefits of physical activity, research on muscle development and training methods, what consists of a healthy diet, how to enjoy physical activities and include them in your schedule etc.
Oh and in primary school, teachers would go in the teachers lounge to smoke while they were given a break by us being in PE.
I didn't see the fun in many subjects either, and got appropriately poor grades that reflected my level of knowledge or skill.
For PE we had normatives we were supposed to meet -- so many pullups, etc (based on old soviet army normatives I think :)), and if you don't you got corresponding grade.
I would make sense to break PE course into two -- one about theory and one about skill and have two separate grades for them.
Oh and in primary school, teachers would go in the teachers lounge to smoke while they were given a break by us being in PE.