Does anyone know why intel discontinued their tamper protection toolkit? They had an obfuscation compiler that would turn compiled C code into a self encrypting/decrypting code. The idea was if you dissassembled the code at any point you wpuld get mostly garbage instructions. I always wondered how a de compiler could get around that.
Google for anything Rolf Rolles has published on the topic, believe it or not there are general approaches to solving this. Someone already mentioned dumping the text segment, that only works for silly 90s-era obfuscators.
Contemporary obfuscators _rewrite_ the protected code as a series of instructions executed on a virtual machine whose bytecode (and bytecode semantics!) are randomly generated at build time. The solution (AIUI) is symbolic execution of the instructions to determine their underlying architectural effect, synthesize some compiler IR that is equivalent to those effects, run an optimization pass (like a regular compiler) over that IR, and finally generate x86 from the result.
The optimization passes are necessary to remove side effects that do not impact the state of the program ("noise"), which modern obfuscators like Themida insert a ton of into the instruction stream
In other words, rather than attempt to dump some particular part of the program, the binary as a whole is statically analysed to determine, regardless of the indirections inserted by any obfuscation pass, what machine instructions are ultimately executed for a given program input. The abstract representation is then compiled to an equivalent new program which is much easier to read, because all of the indirections and noise have been optimized away.
When I was reading about Rolles' work initially, I couldn't help but imagine this is the kind of approach Geordi La Forge would have come up with if cracking an encrypted binary were ever the plot for an episode of Star Trek :)
iirc it wasn't very advanced. They would 'mov' the decrypted instructions to a region in memory (always the same one), executed it, then save register state and go on to decrypt the next set of instructions.
Breaking it involved monitoring the memory for the decrypted instructions, and dumping them right before they were executed. I don't remember if there were any additional complications with stuff like conditional jumps.
There has been some malware that did just that - it was still possible to record the trace of instructions being executed along with the current instruction pointer to be able to reconstruct the binary quite well.
https://software.intel.com/en-us/articles/intel-tamper-prote...