We used to place an INT3 vectored exception handler on function entry points and do everything interesting inside the exception handler. This made the execution stack basically invisible to every debugger since it doesn't debug exception handlers. You can enable/disable interrupts and tracing and whatever you need to do inside the exception handler to guarantee that nobody can see what you are doing and/or verify that no other program has registered another exception handler before doing anything interesting.
If you need to hook functions in third party software, this trick can be used to hook the function without modifying any of the functions code. All you need to do is modify some pointer used by the function to zero, and it will raise an exception as soon as something like p-> is executed on that pointer, then your exception handler can execute whatever code you need (i.e. write over stack, write to memory, exfiltrate handles) and on exit all you need to do is restore the correct register containing the pointer and wind back the execution counter by the size of the de-reference instruction.
Please don't use this knowledge to hurt people ...
A debugger sees exceptions before the application does, and it knows which exceptions it should handle by itself (e.g. breakpoints set by the user) vs passing them to the application.
I don't have a windows machine rn to verify but I'd expect your assumption that it's impossible to debug an INT3 VEH to be incorrect.
Yes, however you can easily detect when a debugger is attached and avoid placing the VEH. There are only two mechanisms for a debugger to operate: by placing its own VEH (or UHE), or by populating one of the four hardware breakpoint registers, and both are very easy to detect.
Sure there's a gazillion ways of detecting a debugger (specifically on windows), but then we are back to detecting debuggers. My point was that VEH (alone) doesn't prevent any sort of debugging, specifically not the VEH handler itself.
Btw, debuggers (on windows) won't usually install VEH to support BPs, they'll use the win32 debugger infrastructure where the OS manages exceptions and delivers them to the attached debugger object (which again can be detected in several ways).
They also do not technically need HW bp registers, although often they will. A simple way to implement BPs is to write 0xCC (INT3) to the text section, then restore original bytes when the INT3 fires.
I don't disagree that it's possible, just in my experience the debugger VEH was fired after my application's own VEH and not before it, so the debugger did not step into the VEH. And if you want to hide something from static analysis it's not a bad way, even if it is meta to "security through obscurity". The barrier to entry for implementation is pretty low, like ~50 LOCs so it's not going to be on the level of like state sponsored malware or something like that.
The VEH target is visible if you know where to look. You can use something like Themida to virtualize/obsfucate the VEH if you really need the VEH to be encrypted.
It's actually not so easy to find the VEH because if you are injecting the VEH into a third party process from another process, then not only does the VEH not exist during static analysis of the binary at rest, but its program address changes on each execution. Moreover, the VEH can be encrypted at rest before it is injected into the second process.
Interestingly enough, VirtualBox does this too, and they call it "hardening", but IMHO it's quite an unexpected and hostile behaviour which is more characteristic of malware.
Thankfully this is easy to circumvent: have your debugger catch the ptrace syscall and lie about the result. Also, if antivirus programs haven't already added a signature for any programs that do that, they should.
The malware will then rely on actual ptrace behaviour as a check. You could instead use seccomp_unotif and let the target ptrace itself as much as it wants: https://man.archlinux.org/man/seccomp_unotify.2.en
Windows also has many similar evasion techniques, like checking if there is a top level exception handler. I use scyllahide, but even on gdb you can break at ptrace and patch it or for automated analysis, just flag anything that used ptrace but isn't a debugger and run it in a sandbox without ptracing it. Audit subsystem might be enough.
If I remember it right, only 1 ptrace can be attached. So this seems easy to fix:. Just ptrace yourself, and nobody else will.
As a bonus, write important state to memory supposed to be read-only. If someone hooked your ptrace, the hook has to reimplement ptrace in a lot more detail. Or use breakpoints as a mechanism to call subroutines.
In an old $DAY_JOB I used a variant of this as a way to make certain internal errors execute a breakpoint when debugged, and generate an error log if not debugged. iirc this did not happen until after an error had already occurred, so it was not very useful as anti-RE.
I've seen this used preemptively - have the process ptrace itself on startup (and then do nothing with it) to make it impossible (or at least far-from-trivial) for other interested parties to ptrace it.
Yes. Or if it's using dynamic libraries and not compiled static, you can use LD_PRELOAD and overwrite ptrace() to do nothing. You don't have to patch anything then, which might be easier.
int ptrace(int request, int pid, void *addr, void *data) {
return 0;
}
Thanks, both! This was used in a static build that decrypted and checksummed its binary before execution, which ruled out naive implementations of the attacks above. I agree there are ways round these too, but I believe it was just intended to discourage amateurs rather than protect against serious hacking.
If you need to hook functions in third party software, this trick can be used to hook the function without modifying any of the functions code. All you need to do is modify some pointer used by the function to zero, and it will raise an exception as soon as something like p-> is executed on that pointer, then your exception handler can execute whatever code you need (i.e. write over stack, write to memory, exfiltrate handles) and on exit all you need to do is restore the correct register containing the pointer and wind back the execution counter by the size of the de-reference instruction.
Please don't use this knowledge to hurt people ...