For crypto itself, read Applied Cryptography by Bruce Schneier. It begins with some history, high-level theory and pseudo-code and ends with C code and the nitty gritty.