Hans-Jürgen Rauscher, System Architect at Wind River, compares the real-time capabilities of Linux and a real-time operating system.
Electronics Weekly’s Focus on Mobile Linux
Linux has become well-established in the world of embedded design, but there are often reservations about its real-time performance. There are significant advantages to using Linux. A careful analysis of the real-time performance requirements of a system design can reveal whether Linux can be used or a real-time operating system is necessary.
Hard and Soft Real-Time
The official definition of real-time computing is a task that is executed within a given time, giving a deterministic system. Some applications need only an average response time, while others require that every deadline is met every time.
There are two main approaches to real-time systems: “soft” real-time and “hard” real-time.
- Soft real-time computing provides a guarantee of a specific level of CPU bandwidth in a specific unit of time. For example, an application that needs 10 milliseconds of CPU bandwidth and must have that requirement met within 100 milliseconds has a soft real-time requirement. This is often where a general-purpose operating system such as Linux is used.
- Hard real-time computing looks at the response time rather than at a bandwidth guarantee, when an application must respond to an event within a specific time. An example would include responding to a periodic interrupt where the worst-case response time must be less than the interrupt time so that no interrupt events are lost.
Spelling out GNU and Linux stories
A more extreme hard real-time requirement would be a response to an event in a control application such as a vehicle antilock braking system, where failure to meet a guaranteed response time could lead to catastrophic results.
Hard real-time is a specialized approach that can be relaxed in some applications, such as rendering the audio in a computer. At the popular 44.1KHz frame rate, an application has to transfer 32 bits of data into hardware every 22.7 microseconds. Because other operating system services could delay this operation, buffering is used to compensate for any delays.
Adding a 4096 32-bit buffer reduces the application timing requirement to fill this buffer approximately every 0.1 seconds, solving a hard real-time problem with a soft real-time operating system.
The one thing all of the different applications in aerospace and defence, industrial control, instrumentation, robotics and telecommunications have in common is a requirement for single-microsecond level interrupt and scheduling latency.
Real-time responsiveness is usually measured by the interrupt latency: the interrupt blocking time during which the kernel is pending to respond to an interrupt, saving the task context, determining the interrupt source and invoking the interrupt handler.
In reality, the interrupt latency is based on the speed of the hardware because the real variable in software is the speed of the operating system to service the interrupt with the interrupt service routine.
But with most real-time applications, before the system can respond to the interrupt, some process, task or user thread must execute and respond to the interrupt. This is called scheduling or context switch latency.
The context switch latency is the amount of time it takes to switch control to a high-priority process in response to an event.
Part of this scheduling latency is the context switch from one process to the higher priority process, and the total real-time responsiveness is the combination of interrupt latency, scheduling and context switching.
There are many methods for achieving real-time responsiveness in operating systems. Real-time operating systems were designed specifically to solve this problem, whereas Linux was designed to be a general-purpose operating system.
Transforming Linux into an RTOS
There have been many attempts over the past several years to improve the real-time performance of Linux so that the benefits, such as open source software availability, can be used for real-time applications.
There are two basic approaches to this: a preempt kernel and the real-time core.
Preempt kernel approach
In the preempt kernel approach, the Linux kernel is modified to reduce the amount of time the kernel spends in nonpreemptible sections of code. Because this approach requires the modification of the Linux kernel (via a patch) to achieve minimized nonpreemptive paths, it is an ongoing effort and potentially requires time-consuming revalidation whenever a patch is made.
If code that exceeds the nonpreemptible path limit of the kernel is found, it must be restructured to run within the latency limits, and the process of finding nonpreemptive paths of excessive length is empirical and requires exhaustive testing of all the kernel paths.
This has to be done under all load conditions in order to guarantee discovery of the worst-case scenario and is realistically impossible. While competitive preemption latency figures are available, the complexity of the overall kernel is such that an absolute guarantee is not realistic.
Many open source projects have been formed to create a low-latency kernel using a variety of preemption approaches. The most significant project was started by Ingo Molnar, and is being included in the standard Linux kernel over time.
But there are certainly benefits to this approach. The most notable is the PREEMPT_RT project that is widely considered to be “mainstream Linux” and the quality and maturity is improving over time. Many code paths in the kernel (such as device drivers) are being modified in the mainstream code to support preemption.
However there are still some limitations, with PREEMPT_RT support – particularly in code – receiving less community coverage. This includes architecture-specific code as well as device drivers.
While in many cases device drivers will port seamlessly into the PREEMPT_RT model, the performance as well as correct operation of legacy and new device drivers present ongoing validation issues.
Real-time executive approach
With the real-time executive approach, a small real-time kernel coexists with the Linux kernel. This real-time core uses a simple real-time executive that runs the non-real-time Linux kernel as its lowest priority task and routes interrupts to the Linux kernel through a virtual interrupt layer.
All interrupts are initially handled by the core and are passed to standard Linux only when there are no real-time tasks to run. Real-time applications are loaded in kernel space and receive interrupts immediately, giving near hardware speeds for interrupt processing.
The core communicates with Linux user tasks through lock-free queues and shared memory; from the application programmer’s point of view, the queues look very much like standard Linux character devices, accessed via POSIX read/write/open/ioctl system calls. The shared memory is currently accessed via the POSIX mmap calls.
While the Linux kernel is largely untouched and does not need revalidation, the interrupt abstraction model does have some differences from “standard” Linux that developers should be aware of.
This model requires the real-time tasks to be implemented as Linux kernel loadable modules. This is the same format that is used for other kernel features, such as file systems, device drivers, alternate loaders and others. Essentially the real-time applications behave similarly to Linux device drivers and operate in kernel space.
While the PREEMPT_RT feature delivers the benefit of being part of the mainstream Linux kernel development and is well-suited to certain types of real-time application development such as audio or video streaming, it cannot give 100 percent guaranteed real-time performance.
On the other hand, a real-time core is not a part of the standard Linux operating system and is not licensed by the General Public License. It is immediately capable of providing a guaranteed hard real-time environment for appropriately written applications within a Linux environment.
A Different Approach
Both of these approaches to real-time Linux take a GPOS and make it an RTOS. A fundamentally different approach is to add non-real-time services to a basic real-time kernel.
An RTOS provides a real-time kernel that interleaves the execution of multiple tasks employing a scheduling algorithm. The scheduler determines which task will own the CPU time and by default runs a preemptive algorithm to guarantee that the highest priority task preempts a lower priority task. It also uses a single common address space for all tasks, avoiding virtual-to-physical memory mapping.
Historically, RTOSes have only offered kernel mode tasks, trading abstraction and protection of kernel components for direct access to hardware to give much higher responsiveness and performance.
This trade-off requires that kernel development operates at a more sophisticated programming level, where the developer is more aware of subtle interactions with the hardware and the risk of hitting unrecoverable fault conditions.
The latest RTOS technology introduces the real-time process model, or RTP, that creates both user and kernel modes. This creates a user mode application environment without eliminating the traditional model of RTOS programming and allows memory protection without compromising performance for traditional applications in the kernel space.
Applications based on the RTP model benefit from the high-performance RTOS kernel, with the addition of reliable resource reclamation and memory protection.
The RTP model also greatly increases the facilities available to the application programmer. User mode applications run in memory contexts even without memory management unit support and run in protected memory contexts with MMU support.
Comparing Linux and RTOS
A real-time core provides real-time in Linux by providing a fairly small real-time “microkernel” with its own scheduler. Applications that link into the microkernel will get hard real-time for interrupt and context switching.
For other services, the application must pass over to a Linux non-real-time task that will do the work. With an RTOS where the whole system is built to be real-time, the application may have more capabilities at its disposal in real-time mode. All of this depends on how independent the applications actually are.
The traditional RTOS still creates a much smaller footprint than the real-time core, and because Linux runs as a low-priority process alongside the real-time core, there are still footprint constraints.
There is also performance degradation for any non-real-time processes that may run in user mode in Linux. The user mode RTPs will have much better performance than standard Linux processes but probably about the same as Linux applications running in the real-time core executive in the Linux kernel space.
Existing RTOS solutions have been proven over decades of use while the PREEMPT_RT patches are relatively immature, and the RTOS is also more easily certified for military and government applications that require strict certifications. The size of the Linux operating system makes it very difficult and costly to certify.
Other considerations in determining the right solution are the availability of the third-party software for the application, the overall costs and the availability of the right hardware support.
The increasing diversity of real-time development tools and device application requirements has made life more complicated for developers. Determining whether to use an RTOS or Linux to meet real-time requirements should not be an additional source of risk.
Moving to Linux has some key advantages in cost and scale and can provide significant advantages in many situations. Depending on the application requirements, even for hard real-time applications, there are several different ways to access the open source advantages of Linux.
Hans-Jürgen Rauscher, System Architect. Wind River