It's not a mystery, but after generations of lecturers teaching something they never learned themselves, physicists have forgotten the history of it.
Complex numbers turn up a lot because exp(2πiθ) is a convenient way to model any kind of rotation or cyclic process.
It turns out that there's lots of cyclic processes in physics, such as atomic orbitals or waves in fields.
Similarly, vector algebra is just terrible at modelling physical spaces, especially 2D or 4D. It just happens to work okay in 3D, and even there it has all sorts of sharp edges, such as gimbal-lock.
Geometric algebra is a lot better at modelling physical spaces of arbitrary dimension, and one of its strengths is the ability to take various interesting sub-spaces and use them without having to make changes to formulas.
Both the complex numbers and quarternions are such subspaces of matching geometric algebras, so it's no surprise that they "turn up" a lot in the algebras of geometric spaces.
Interestingly, some of the other subspaces of a GA are isomorphic to the various "complex valued matrices" used in Physics, such as the Pauli matrices and the Gell Mann matrices.
These matrices are introduced to students as essentially scalar-valued black boxes. "Don't worry about it, they just have the 'right' properties." is commonly heard in lectures.
This seems like mathematical magic, but no magic is necessary. A much more consistent and logical theory based around GA could have been used, but wasn't for merely historic reasons.
Unfortunately, lots of people will say that these alternative formulations are all isomorphic to each other, so who cares, just shut up and calculate. However one method yields an intuitive understanding, and the other one yields complex numbers all over the place with no clear picture of their origins...
Although they don't affect models due to the isomorphic quality you describe, these historical representational accidents seem to have a powerful effect on our intuition.
I am reminded of the conceptual differences between the Copenhagen and Bohmian interpretations. Using one or the other basically does not change our models or results, but what affect do these different perspectives have on the field?
Bohmian mechanics (pilot wave theory) is not the same as traditional QM, it is a stronger theory from which traditional QM can be derived (of course, we don't know if it's correct yet).
Examples of interpretations of QM that are mathematically equivalent are Copenhagen and Many Worlds. Examples of theories that are stronger (make more predictions than) QM are Objective Collapse theories, de Broglie - Bohm Pilot Wave theory, Superdeterminism.
Complex numbers turn up a lot because exp(2πiθ) is a convenient way to model any kind of rotation or cyclic process.
It turns out that there's lots of cyclic processes in physics, such as atomic orbitals or waves in fields.
Similarly, vector algebra is just terrible at modelling physical spaces, especially 2D or 4D. It just happens to work okay in 3D, and even there it has all sorts of sharp edges, such as gimbal-lock.
Geometric algebra is a lot better at modelling physical spaces of arbitrary dimension, and one of its strengths is the ability to take various interesting sub-spaces and use them without having to make changes to formulas.
Both the complex numbers and quarternions are such subspaces of matching geometric algebras, so it's no surprise that they "turn up" a lot in the algebras of geometric spaces.
Interestingly, some of the other subspaces of a GA are isomorphic to the various "complex valued matrices" used in Physics, such as the Pauli matrices and the Gell Mann matrices.
These matrices are introduced to students as essentially scalar-valued black boxes. "Don't worry about it, they just have the 'right' properties." is commonly heard in lectures.
This seems like mathematical magic, but no magic is necessary. A much more consistent and logical theory based around GA could have been used, but wasn't for merely historic reasons.
Unfortunately, lots of people will say that these alternative formulations are all isomorphic to each other, so who cares, just shut up and calculate. However one method yields an intuitive understanding, and the other one yields complex numbers all over the place with no clear picture of their origins...