I would say the field is even broader than just those two areas. Much design literature is on a third focus, the interaction between desiderata and materials, which has a significant overlap with engineering. This is perhaps most obvious in architecture (the kind involving buildings, but also software architecture / API design), which is as its core not primarily UX or visual design, though it has elements of both. Herb Simon's The Sciences of the Artificial is my favorite book in this area, though Donald Schön's The Reflective Practitioner is also a good book on design in this broader sense of producing human-usable artifacts out of materials.