procstat can also do interesting things, like show current rusage state:
% procstat -r 5592
PID COMM RESOURCE VALUE
5592 less user time 00:00:00.010805
5592 less system time 00:00:00.002444
5592 less maximum RSS 3172 KB
5592 less integral shared memory 192 KB
5592 less integral unshared data 80 KB
5592 less integral unshared stack 256 KB
5592 less page reclaims 199
5592 less page faults 0
5592 less swaps 0
5592 less block reads 4
5592 less block writes 0
5592 less messages sent 0
5592 less messages received 0
5592 less signals received 0
5592 less voluntary context switches 59
5592 less involuntary context switches 0
BTW, I totally suck at formatting stuff in various forums. Is there a way to post pre-formatted text here? Eg, like triple back-ticks will do in slack? ``` stuff... ```
If you are running java instead of a c program the proc stacktrace shows you just the virtual machine state. You can still get a stacktrace of your java threads[1].
> If you are running java instead of a c program the proc stacktrace shows you just the virtual machine state.
No, this is a _kernel_ backtrace: what is happening in kernelspace on behalf of your process. If the work is being done in userspace (that is, in state R; the thread isn't in a syscall or page fault handler), you'll see essentially nothing here. I just tried it on a userspace busylooper and got this:
Java, Python, C++ nothings all look pretty similar.
If you want a userspace stack trace, you need a different tool. If you're using an interpreted (or perhaps JITted) language, yes, you probably want something language-specific.
Also note the current stack trace is a per-thread concept, not a per-process one. If you're looking at a multithreaded program, you want to target the thread(s) of interest with "/proc/<PID>/task/<TID>/stack".
Nicely formatted site. Anyone know if/what static site generator/theme is being used? Couldn't find anything on the footer, site tags, or his GitHub that would reveal that...
The /proc interface tells you what the process is doing in the kernel (eg, after a system call), which is orthogonal to getting a user-space stack trace. Both are helpful. The userspace trace is arguably more helpful. The nice thing about the kernel trace is that you're likely to have symbols (or be able to download them); that is unlikely to be true for userspace if you're running a commercial binary, for example.
The most widely distributed embedded ARM system software in the world, Android, offers this. They use mini debug info (normal debug info sections compressed, IIRC) and then you can signal a daemon in the background to do a stacktrace on your application.
I've done it, but it increases binary size by a whole lot.
What worked better was a last chance hardware fault handler that wrote a partial core dump out to flash, and a tool to create an ELF core dump file from that that GDB can accept.
Instead of teaching people how to do this portably across all UNIX-like systems, by sending SIGABRT to the process, the article is steeping them in GNU/Linux only way of doing things. This feels exactly like the '90's of the past century, where a lot of people with computer-related careers had no idea that there were other operating systems and other ways of doing things (better): an intel-based PC tin bucket with Windows was the one and only truth for them. Now it's exactly the same except Windows has been replaced with GNU/Linux. 28 years later and the only advancement some people have made is running the proverbial sed 's/Windows/Linux/g'.
That's a very strange response. This is an article about Linux specifically. The author tags the article with Linux and mentions that it's part of a series of articles about Linux. It's not in and of itself a bad idea to have articles that specialise in a specific unix flavour.
Secondly, SIGABRT will cause the process to abort and dump core, will it not? That would give you a userspace stack trace (if you load the core into a debugger) whereas this is how you get the kernel-side stack trace of a still-running process.
I don't know of a way to get the kernel stack trace of a process in a cross-platform way. Is there such a thing?
SIGABRT in most places is not going to both get the stack trace and keep the process running. In fact, I would argue that if you are running something that continues working after it receives a signal telling it to /abort/, it's a bug. What do the word abort mean to you?
As for SmartOS, as someone who ran it at home and in production for years: Keep flying that flag, I guess. I liked it. But I also realized that Joyent, even with the Samsung acquisition, does not have the resources to keep it going in any meaningful way for anyone beyond themselves and people who have the exact same usecase as them. Things like lx branded zones are clever. I miss SMF, and am not a fan of systemd. But bpf is better than dtrace. Container management is easier than zone management. I've got less bugs dealing with KVM on Linux than I ever did on SmartOS. I spend less time compiling things from source and having to find random patch files to make things work. I know plenty about Solaris and SmartOS and HPUX and AIX and the BSDs and I don't think anyone is making the incorrect choice in deciding to learn Linux over any other UNIX-like.
That ship has sailed, man. And there's no compelling reason that it shouldn't have.
"Container management is easier than zone management."
Linux has no containers, cgroups aren't anything conceptually close to zones. There are 56 different solutions to virtualization on Linux, all competing, mainly because everyone there is still flapping on deployment and lifecycle management. We'll just have to disagree and vehemently at that.
As for Joyent having no resources: you debug it and fix it yourself. That's precisely how Linux got to be the hegemony that it is today. Oh, how quickly we forget, how short our memory is...
> Instead of teaching people how to do this portably across all UNIX-like systems, by sending SIGABRT to the process
Sending SIGABRT doesn't do what the article is talking about, on any UNIX. Perhaps you would learn something from listening to the kids these days, like the distinction between a kernel stack and a userspace stack.
A kernel should always be compiled with symbols / source code inside of it -- that the Linux kernel doesn't have full support for CTF says more about it than it does about CTF.
Yes, a SIGABRT will get you a core file and will kill a process, but if your process is hanging in an endless loop (like the author's was), one already has far bigger problems, and keeping such a process running will not amount to much.
I can't think of a single OS that gives you kernel stack traces on SIGABRT.
Can you disambiguate CTF? I only know that as capture the flag, which doesn't really make sense here.
Edit: ok, I figured it out.
No theyre not going to give you that raw information because it's a kernel ASLR bypass. You can totally get all the same information with the dwarf symbols, but you're going to have to opt in on a kernel for it to mean anything.
Edit2: bitching about how people aren't using portable Unix techniques, and then citing features that are Solaris specific isn't a great look.
For that to be useful, you'd need the addresses in question, which is why it'd be an ASLR bypass. The kernel needs to give higher level information by default so it can sanitize the output.
And the debugging symbols exist on Linux anyway, they're just DWARF (which ironically is the more standards compliant way as opposed to CTF).
CTF is just a specially compressed format and specification of embedding the source code into the binary; it's a complement to DWARF format, not the format itself. The documentation you cite even says it's similar, meaning that it's not DWARF.
The main purpose of CTF is to provide a succinct representation of the graph of C types used in a program. It's generated from DWARF; anything encoded in the CTF section can also be found in DWARF info. CTF has nothing to do with "embedding source code" and isn't useful for stack unwinding or symbol resolution.
OK, it's a dtrace thing that got ported to FreeBSD with dtrace. That still doesn't make it "Unix" any more than Linux's proc access.
Embddeding debugging information in a way that allows to you to use it (ie. in a way that you'd care about the format as an end user) implies giving you kernel addresses, which implies an ASLR bypass. If it's an implementation detail, then DWARF works great.
/proc on Linux wasn't implemented like /proc is on other operating systems; it's the only /proc implemented that way. The interface, if it could even be called that, is completely proprietary to Linux; output is ad hoc with no consistency. As usual.
Could you write a blog post in response that describes the better way to achieve this? There are possibly related items which you could throw in for others on achieving better portability.
Yes, it feels strange to hear millennials talk about UNIX, when they are actually talking about GNU/Linux, and many things don't apply to e.g. OS X, Aix or many other variants.
Write for yourself -- I've no problem paying Apple Computer for a solution which I plop down on a table, turn on, and start using immediately after that without having to fiddle with it.
They regularly fuck up security so much that you aren't getting what you pay for. In the last major release they both allowed anyone to login as root by just not supplying a password, and writing the FDE password on the disk in plaintext as the "hint" instead of the actual hint.
What is the non alternative O/S that has complete hardware support? fwiw openbsd / freebsd have been painless to install on every machine I have tried in the last 5 years.
BSD user here. Of course there is porting occurring, but this is the nature of open source, is it not? Oftentimes, the BSD devs will take whatever hardware specs they can get and code something from that. OpenBSD frequently writes drivers that are orders of magnitude smaller than comparable Linux drivers, while getting the same functionality. NVIDIA drivers are but one example. Some hardware vendors are generous with hardware specs for OSS, most are not. OpenBSD, as an example, will not have any GPL'd code in the base OS, as it's not truly free. Ports are another issue.
> will not have any GPL'd code in the base OS, as it's not truly free
That's a disingenuous way to put it. The GPL is a free software license. It is "free" as in libre by all common definitions of "free" in regards to software.
The GPL is not compatible with the BSD dev's preferred license, which is much more likely the reason they avoid using such code.
By all conventional definitions, the BSD and GPL licenses are "truly free [software licenses]."
You're welcome to argue that the BSD license is better (because, for example, it lets sony create derivative playstation OSs without providing that source code to their users, ensuring their users have less freedom than if it were linux) or that the GPL is better (because it would prevent the previous), but they're both free licenses.
No idea about other BSDs, but in case of FreeBSD that’s generally just Intel and AMD graphics drivers. All the rest is native.
And this goes both ways - at one point Linux folks famously managed to violate the BSD license when porting some WiFi driver, iirc, which is notable, since it’s not that easy given the licenses nature :-)
They don't. However, they do reduce the ability of the target O/S of the port to claim that it isn't a clone of Linux. The top-level post was complaining that people don't consider non-Linux O/S, and then failed to offer compelling alternatives.
I did not, I stated that nothing has changed: before people only knew PC buckets and Windows, now PC buckets and Linux is the only truth and religion. Nothing else exists. So the more things changed, the more they stayed the same. That's progress!
I also listed several alternatives just a few comments below, but apparently you didn't read that. You know of the old saying "you can bring a horse to the water, but you can't make him drink"?
procstat can also do interesting things, like show current rusage state: