The big downside of using a traditional signal handler is that the only way to get your own data into the handler function is through global variables (or thread locals). While you can certainly make an exception just for that one thing, it feels gross to do so. And you can also just defer processing to your main loop by setting a flag or writing to a pipe, but those things still need to be global variables.
I didn't know about signalfd's limitations before reading your post, and was happy that signalfd could eliminate the need for global variables when doing signal handling. Shame that's not really the case.
In my case I use a thread_local pointer that I initialize right before epoll_pwait and set back to null immediately after. The pointer points to the same data structures that I would otherwise use to handle signalfd events. Yeah it's a little icky to use the global but I think it ends up semantically equivalent.
Unfortunately, thread-local storage is not async-signal safe. You're relying (knowingly, I presume, but others should be warned) on implementation details.
But, yeah, signalfd leaves much to be desired. *BSD kqueue EVFILT_SIGNAL has much saner semantics.
> Unfortunately, thread-local storage is not async-signal safe.
Doesn't matter, because the signal handler in this case is strictly called "during" invocation of epoll_pwait, so there's no risk of it interrupting the initialization of a TLS object. The usual rules about async signal safety do not need to be followed here; it's as if epoll_wait()'s implementation made a plain old function call to the signal handler.
(Also, since we're talking about epoll, we can assume Linux, which means we can assume ELF, which means it's pretty easy to use thread_local in a way that requires no initialization by allocating it in the ELF TLS section. But yes, that's relying on implementation details I suppose.)
> kqueue EVFILT_SIGNAL
Having recently implement kqueue support in my event loop I have to say I'm disappointed by EVFILT_SIGNAL. It does not play well with signals that target a specific thread (pthread_kill()) -- on FreeBSD, all threads will get the kqueue event, while on MacOS, none of them do. Fortunately EVFILT_USER provides a reasonable alternative for efficient inter-thread signaling.
(I don't like using a pipe or socketpair as that involves allocating a whole two file descriptors and a kernel buffer, and it requires a redundant syscall on the receiving end to read the dummy byte out of the buffer. If you're just trying to tell another thread "hey I added something to your work queue, please wake up and check", that's a waste.)
> Doesn't matter, because the signal handler in this case is strictly called "during" invocation of epoll_pwait, so there's no risk of it interrupting the initialization of a TLS object. The usual rules about async signal safety do not need to be followed here; it's as if epoll_wait()'s implementation made a plain old function call to the signal handler.
And if dlopen() is called in another thread, needing to (re)allocate TLS space and/or rebuild the index? You're relying on implementation details.
TLS is async-signal safe in musl libc, but musl libc also fudges (arguably) dlclose--it leaks memory as its async-signal safe data structures prevents it from deallocating per-module TLS space.
glibc may work by accident, or in the past 10 years or so they may have refactored things to work for your case. But, again, strictly speaking you're definitely relying on implementation details. (Which is perfectly acceptable as long as the dependency is transparent.)
> It does not play well with signals that target a specific thread (pthread_kill()) -- on FreeBSD, all threads will get the kqueue event, while on MacOS, none of them do.
That's because file descriptors are independent of threads--a thread doesn't "own" a descriptor--so the semantics of per-thread signals cannot cleanly map. (Multiple threads can wait on kqueue/epoll/signalfd descriptor, and moreover on both BSD and Linux you can install a kqueue/epoll descriptor as an event on another kqueue/epoll descriptor.) That Linux tries to fit a square peg into a round-hole with signalfd in this regard is probably related to the horrendous locking issues you encountered.
How are EVFILT_SIGNAL semantics better as a general matter? Among other things, it permits different libraries or components to listen for a signal without stepping on the toes of any other component. That's something that is simply impossible on Linux, period, where even signalfd is basically implemented as a signal handler--which is why you can't have both a signal handler and signalfd responding to a signal, not to mention multiple signalfd's responding to a signal.
There really aren't a lot of great solutions, here. OTOH, there aren't many use cases for per-thread signals, at least as originally conceived on Unix. EPIPE should be masked and handled in-band for anything with an event loop. SIGBUS and SIGSEGV must be handled by a signal handler, if at all. I'm sure you have your reasons, but at the end of the day it sounds like you're dealing with an issue at the nexus of signals, threading, and process semantics that cannot be correctly[1] resolved without a new, dedicated kernel API.
[1] Not strictly true. You could install a dedicated stack using sigaltstack for each thread, which could then be used to smuggle per-thread data to the signal handler, e.g. placing the data at a fixed offset from the stack given to sigaltstack. I did this once for an app that needed to longjmp from SIGSEGV--the SIGSEGV occurred by design when indexing off the end of an mmap'd array. Bounds checking and growing of the array inline was too slow as the constant inlined conditional checks destroyed pipelining. Catching SIGSEGV and longjmp'ing back to a safe point to regrow the array and then restarting the operation improved performance by some multiple > 2. Using the sigaltstack trick meant the library could be safely used from multiple threads, and without the rest of the application needing to know anything.
> And if dlopen() is called in another thread, needing to (re)allocate TLS space and/or rebuild the index? You're relying on implementation details.
This could also happen when I'm not in a signal handler, and the exact same mechanism that protects me in that case would also apply inside the signal handler (if the signal handler is restricted to running only during epoll_pwait()).
Again, a signal handler that is blocked from running except during epoll_pwait() has no special safety concerns different from those of a regular function call.
> That's because file descriptors are independent of threads--a thread doesn't "own" a descriptor--so the semantics of per-thread signals cannot cleanly map.
Well, file descriptors are independent of processes, too! You can send them to another process via parent->child inheritance or across a unix domain socket via SCM_RIGHTS. So what happens if you add EVFILT_SIGNAL in one process and then read the kqueue from another? Whose signals do you get?
Anyway, what they could have done is provided a way to specify, when adding EVFILT_SIGNAL, that it should notify only for signals deliverable to a particular thread.
> Again, a signal handler that is blocked from running except during epoll_pwait() has no special safety concerns different from those of a regular function call.
You're absolutely correct. Mea culpa for objecting too soon.
I would've deleted my comment immediately after submitting as discussing my sigaltstack made me realize my error. But I got locked out by the procrastination setting. :(
Makes sense, and is probably the "safest" you can get. Since, as you say, you know exactly the state of everything on that thread when you're in your handler, you can also know that your thread local was set properly before the epoll_pwait() call.
It's probably code I'd want to isolate somewhere, with big warnings so any future reader understands why it is how it is, but I agree it's probably the safest way to do it.
> The big downside of using a traditional signal handler is that the only way to get your own data into the handler function is through global variables (or thread locals). While you can certainly make an exception just for that one thing, it feels gross to do so. And you can also just defer processing to your main loop by setting a flag or writing to a pipe, but those things still need to be global variables.
That's not true. Ever since POSIX.4's real-time signals, we've had sigqueue(), which allows the you to attach an arbitrary sigval (integer/pointer union) that is passed to the signal handler, thereby allowing it to receive data without using globals/thread-local variables.
signalfd()'s signalfd_siginfo structure has two fields: ssi_int & ssi_ptr, which can receive said arbitrary data along with the signal. There was a brief period of time, when signalfd() was initially created, where those fields were not populated by the Linux kernel, but that hasn't been true since... (checks man page), 2.6.25 (and signalfd() has only been in the kernel since 2.6.22).
I didn't know about signalfd's limitations before reading your post, and was happy that signalfd could eliminate the need for global variables when doing signal handling. Shame that's not really the case.