Have you read the 2017 paper Attention is All You Need? It's not a book, but it is long form, a few math formulas but not too many. 10 pages long.

https://arxiv.org/pdf/1706.03762.pdf

At the current speed of research, any book about transformers would be outdated before the ink it was printed on has dried. You got to read the papers if you want to keep up with the bleeding edge.

The attention is all you need paper that tikkun linked is great, but not exactly a gentle start. This might help a little: https://sebastianraschka.com/blog/2023/llm-reading-list.html