There are a lot of misunderstandings and shallow information in the article (and in the comments).
I used the RT_PREEMPT patches in 2012/13, so my insight is most certainly rusty, dated and might even be wrong here or there. But still:
First of all, RT_PREEMPT is about realtime as in "hard realtime" for control systems. This has nothing to do with "small latency" (the cycle time could be minutes), but with deadline guarantees, allowing control loops over (realtime) network protocols, etc.
Then, almost all important additions of the RT_PREEMPT patches were mainlined one after the other. The last bits were never mainlined although essential for full preemptive kernel AFAIR, but:
Most "modern" (>15 years) OTS-platform (x86, x86-64) is increasingly unfit for realtime - no matter what kernel/OS you use, because your CPU is not in control. Beginning with DMA, later also "management engines", modern platforms might dethrone your CPU on the lowest ring entirely and for as long as they want, to serve a request/interrupt, e.g. fan control, while your CPU wakes up 500ms later realizing that it missed quite a few deadlines. (Fan control is a bad example, but DMA is not. Also Intel ME!)
This makes almost all Linux PCs inherently unable to reliably keep these guarantees.
This puts the whole "hard realtime on Linux"-project in a pickle - since it requires full control over the platform. And most shops that go this route, will take the small additional step of using something well tested and with vendor support.
And the rest buys platform and OS with support from a trusted vendor.
This hits on the core of why we still use what we do in military aerospace. COTS stuff just can’t hit the requirements nowadays.
Personally the newest X86 I’ve seen in this type of stuff was an 80286; everything newer has been POWER or SPARC of one flavor or another, typically running VXWORKS. I haven’t by any means seen everything though.
Agreed here. I do this type work for a career, and if you need safety critical + real time (which really just means known and bounded time) guarantees, anything with an ME type device that can usurp the cores is generally a no go. Especially if the source for the firmware for said ME is not accessible.
Actually, RedHawk Linux is widely used in the US “defense” industry. It supports modern hardware and even provides real-time performance when using CUDA. It is not exactly the PREEMPT_RT though.
I have actually run into that on one or two things, but there were only peripheral to what I deal with. Drones, aircraft, etc. have all been what I mentioned in my (limited) experience.
I agree that Linux with RT_PREEMPT cannot be hard real-time in the sense that it cannot formally guarantee meeting a deadline due to CPU firmware interrupts.
However in many applications (e.g. robotics) where missing a single deadline will not result in loss of life it works perfectly fine.
The unmaskable interrupts on x86 can be a problem but long terms benchmarking can help with possible edge cases.
Other Linux-based solutions such as Xenomai & RTAI which provide even better performance deserve a mention.
There's a general misunderstanding that RT_PREEMPT somehow lowers latency. This is not at all true as RT_PREEMPT is about assured determinism not latency. Non-RT Linux has demonstrably lower latency but not determinism: response times can be very spiky (sometimes fast - often faster than RT, sometimes very slow).
I'm a big Linux fan so having realtime support in Linux would be great. But in the absence of that, Zephyr RTOS is another great open source option with some Linux kernel inspiration.
Ah, Zephyr. Another project that Intel had and then let go.
Intel is on my list, alongside Google, under the category of "companies that do one great thing, everything else is a disposable experiment that is probably already cancelled".
Proceed at your own peril with the Linutronix stuff.
Cynically.. "everything else is a move designed to foul a market and disrupt a potential competitor that is then cancelled as soon as it's no longer needed."
Intel had the top performing ARM platform with StrongARM/Xscale. They threw it away. They had their legacy embedded platforms that were popular. They threw it away. Edison could have been a compelling gateway drug for embedded Atom products. They crippled it from the start and threw it away. The root cause is MBAs making bad decisions for short term gains.
Intel did not throw it away. They sold the business unit to Marvell for $600 million.
I was working for a different CPU company at the time but the thought was by doing this it would allow Intel to focus on the sales and marketing of the Atom and push x86 everywhere rather than have a mixed x86 / ARM marketing message.
On June 27, 2006, the sale of Intel's XScale PXA mobile processor assets was announced. Intel agreed to sell the XScale PXA business to Marvell Technology Group for an estimated $600 million in cash and the assumption of unspecified liabilities. The move was intended to permit Intel to focus its resources on its core x86 and server businesses.
Zephyr started out as a WindRiver acquisition. Then Intel bought WindRiver. Then Intel sold it off.
TLDR: Intel had a foothold in embedded/realtime operating systems and let it go.
Wanna talk about XScale? Intel had an ARM license and then decided, out of the blue, to sell the product line off.
TLDR: Intel had a foothold in low-power ARM platforms and let it go.
I never went near an Edison board so I don't even address that one. And now they're doing RISC-V. Don't go near it.
So, your question is "what is the root cause of Intel not getting a foothold in (em)bedded"? The answer is Intel itself.
Yes, I know they contribute to LF and whatnot. The point is that staking your embedded development pipeline on Intel itself is something I would never do (again). And I worked with Zephyr post-Intel, it wasn't a great system. I hope it's improved since then.
It doesn't strike me as abandoned. So if they no longer act as a curator while its community thrives, so what? Intel contributes to LF btw, that's some real support too.
Intel had an opportunity to develop Zephyr as a usable RTOS, internally* and externally. If they thought that the technology was compelling, why did they let it go?
Zephyr has nothing to do with the Linux Kernel -- especially from a technical standpoint. It is a project that is under the umbrella of the Linux Foundation, that's all.
Much of the core kernel elements definitely follow Linux kernel conventions as well. For example the `eventfd` api was copied from Linux and is quite handy. The POSIX compatibility layer constants match those in Linux, etc. The style guide is also borrowed from Linux.
And then there is stuff in Zephyr like Devicetree, which is designed to look familiar to Linux users but operates in a completely different and broken manner.
It’s also good to note that it’s really the only RTOS with a _real_ open source community and that has support from multiple hardware vendors. FreeRTOS is mostly whatever Amazon decides to publish. Mbed is kinda open but mostly driven by ARM. Zephyr has a community governance based on Linux’s.
What was the name of that anesthesiologist who about decade ago did a lot of work in desktop UI latency in Linux kernel? How did it went? Was any of it integrated to mainline kernel?
That's Con Kolivas. It wasn't merged, he maintained patches out of tree until recently[0]. A pity as latency doesn't seem like a high priority for Linux. Strictly speaking a real-time OS isn't about that though, that's about bounded latency, not low latency.
Nothing was merged, and last time I tried the nvidia driver didn't work with his scheduler. It's a shame, because BFS was the snappiest desktop experience I've had on Linux.
I remember quite a few small utilities and sites I've seen over the years were written by anesthesiologists, not career software developers. I wonder why.
can you elaborate, is it to track drug concentration profiles or work out some kind of body weight to dosage formulae or something? What's going on exactly that pushes them into the technical space.
I was under the impression that PREEMPT_RT in being merged into mainline piece by piece already. If you look at the patches, 4.4-rt patch is 987864 bytes in size while 5.15-rt patch is only 369182 bytes. So, crudely estimating 2/3rds of -rt has already been merged to mainline between 4.4 and 5.15.
You really have to define 'good enough'. E.g. Can handle 1000 Hz interrupt rate, and the time from firing of the interrupt timer to the time your first instruction executes is xxx microseconds. (this is important if you are doing e.g. velocity estimates from position information -- you need accurate 'dt'; therefore this time should be minimized. There are other ways to deal with jitter, but minimizing this is best.)
I turned BSD 4.3 unix into RT on a VAX 780 by using an external KW-11P clock on highest priority interrupt, have the RT code run in the kernel, and otherwise, timesharing was still going while we were running the robot -- although it was pretty sloooow. There was a big memory pool in kernel space that stored robot variables, so when the robot fell over, we would press a button which would 'freeze' the circular buffer, and then the user-space code would do an ioctl() to receive a copy of that memory pool (it was too hard to share the same pool between kernel / user space -- or maybe I was too lazy, and the solution found was 'good enough' ;-) )
>Is RT Linux a good (or at least a good enough) option for realtime OS's
Soft realtime, yes. It does keep scheduling latency under control, enabling e.g. pro audio with low latency (very small buffer). This is unlike without PREEMPT_RT, where with buffers under 20ms XRUNs happen frequently.
Hard realtime, no. It can't guarantee a thing, as the kernel is complicated (has millions of lines of code) and thus unpredictable.
Look at seL4 instead if your deadlines are hard. It has been designed for realtime, and comes with formal proofs of worst case execution time.
I used the RT_PREEMPT patches in 2012/13, so my insight is most certainly rusty, dated and might even be wrong here or there. But still:
First of all, RT_PREEMPT is about realtime as in "hard realtime" for control systems. This has nothing to do with "small latency" (the cycle time could be minutes), but with deadline guarantees, allowing control loops over (realtime) network protocols, etc.
Then, almost all important additions of the RT_PREEMPT patches were mainlined one after the other. The last bits were never mainlined although essential for full preemptive kernel AFAIR, but:
Most "modern" (>15 years) OTS-platform (x86, x86-64) is increasingly unfit for realtime - no matter what kernel/OS you use, because your CPU is not in control. Beginning with DMA, later also "management engines", modern platforms might dethrone your CPU on the lowest ring entirely and for as long as they want, to serve a request/interrupt, e.g. fan control, while your CPU wakes up 500ms later realizing that it missed quite a few deadlines. (Fan control is a bad example, but DMA is not. Also Intel ME!) This makes almost all Linux PCs inherently unable to reliably keep these guarantees.
This puts the whole "hard realtime on Linux"-project in a pickle - since it requires full control over the platform. And most shops that go this route, will take the small additional step of using something well tested and with vendor support.
And the rest buys platform and OS with support from a trusted vendor.