I just recently learned that the note in Fermat's margin was published posthumously by his son! So Fermat never necessarily publicly claimed to have a proof. So I would imagine you're absolutely correct.