Can you elaborate on that a bit or give an example maybe? It seems to me that mathematics does pay attention to what things are, since mathematicians often start their arguments with getting themselves and their readers to agree on rigorous definitions of mathematical structures, before they do anything with those.
It's basically duck-typing. If you can do arithmetic on it, it's a number. If you can do matrix operations, it's a matrix. If it satisfies the axioms of X, it's an X.
This is just something I thought of and I am not sure if this makes sense but mathematicians explore objects much in the same way particle physicists do. So the physicists bounces particles off each other and see what happens to understand how these particles work.
Similarly, mathematicians study objects by looking how they act on other objects. So if it turns out that two differently defined objects have the same actions on other objects, we call them isomorphic and don't really distinguish between them.
So for instance, we say that the set of rigid motions that preserve the triangle and it's orientation is the same as the set of permutations of the roots of say: x^3-3x+1 even if the two sets are absolutely not defined in the same way.
I think he means that mathematical objects are pure structure without substance. They are defined only in relation to other objects and there is no deeper meaning.
I guess you've been to a US school. In france the problem is the opposite. They generaly teach you theory first, and you pretty much have to guess by yourself what all those things are for (beyond the obvious problems you find in exercises).
By "do" he means the behaviour of the objects, not their use in real-world applications. In general, you consider a bunch of isomorphic things as one, rather than worrying about the differences between them which have no effect on their behaviour. This is sort of handwavy because it depends on context.
For example, if we are considering sets and functions between them, we generally don't care about the exact names of the elements of the set, only the fact that they are distinct elements. The important aspects of a function in this case are properties such as injectivity, surjectivity, etc, not that it sends one particular element of one set to another particular element of another set.
Another example is in linear algebra: we really care about linear transformations on an abstract vector space more than we care about what that linear transformation looks like relative to a specific set of coordinates.
This point of view is espoused in category theory, where the important information is carried in the morphisms between objects, not really the objects themselves.
Mathematics never pays attention to what objects ARE, but rather what they DO.