I don't know specific AMD64 addressing modes, but you should always be able to detect overflow in signed 2s complement addition by checking to see if both addends are positive and the result is negative. There is a similar check for subtraction.
Yeah, but the point is addressing modes fuse addition and multiplication and using the result, so there's no opportunity to check anything after the operations.