Guest columnist David Kleidermacher, chief technology officer at Green Hills Software believes that the choice of operating systems and hypervisors plays an important part in unlocking the potential of multicore network processors
High-end multicore network processors, such as Cavium OCTEON II, Freescale QorIQ, and NetLogic XLP, provide developers with a dizzying array of functionality - numerous general purpose processing cores, accelerator engines for security and pattern matching, multi-level cache and memory controller hierarchies.
The only hope to maximising the potential of these processing behemoths is with some ridiculous software smarts. We’re not going to solve the entire problem in this article, but we’re going to talk about the software layer that controls the platform – the operating systems and hypervisors upon which everything else rides.
Symmetric multiprocessing (SMP)
A single SMP-capable real-time operating system (RTOS) or Linux can manage both the control plane and data plane workloads across the multicore platform (Figure 1).
Typically, control plane applications are time-sliced across a small number of cores, while packet processing threads use the majority of the cores.
These performance-critical threads may be bound to cores in order to simplify the data flow between accelerator engines and processing cores.
Figure 1 - Real-time operating system SMP
Lightweight executive (LWE)
SMP starts to become a performance challenge as we move into the many-core arena. OS services require protective kernel locking which can increase latency when many cores are trying to concurrently access those services. A lightweight executive (LWE) may be preferable for data processing tasks over a full-blown operating system.
In some cases, the LWE is nothing more than some initialization code, device drivers, and a superloop without threads. When LWEs are employed, the control plane can be handled either by Linux or an RTOS, resulting in an Asymmetric Multiprocessing (AMP) Environment. If the control plane OS is running Symmetric Multiprocessing (SMP) over a subset of the cores, you actually have a combination of SMP and AMP.
Linux and RTOS
It used to be that embedded designers needed to choose between Linux and an RTOS. However, multicore processors are now commonly hosting both. Similar to Option 2, Linux runs in SMP mode on the control plane while the RTOS manages the real-time workloads, either in an SMP or AMP fashion (Figure 2):
Figure 2 - Linux and RTOS hybrid
Virtualized model
With Options 2 and 3, the control plane OS has no direct control over the real-time cores and vice-versa. An errant DMA programmed by the LWE could corrupt the control plane OS. The system has a division of resources but lacks strict isolation and access control of those resources.
System virtualization can solve this problem. For example, the Freescale P4080 supports the hypervisor mode extensions in Power Architecture ISA 2.06, enabling full virtualization of guest operating systems.
Virtualization can transform Option 3 into a strictly partitioned system that still retains the flexibility of running different operating systems for control and data plane workloads. In fact, some real-time operating systems have virtualization built-in, precluding the need for a separate hypervisor layer.
What is the right architecture for your design?
If you want to deal with a single run-time software platform and less vendors, a single SMP operating system (Option 1) may be a good choice.
If the design requires the software ecosystem of Linux and the capabilities of an RTOS (or LWE), a hybrid AMP model (Options 2 or 3) may be a good choice.
However, if you prefer this hybrid architecture and the processor provides a hypervisor mode, then it would be silly not to take advantage of that (Option 4) for improved robustness, core management flexibility, and overall ease of use.
Many other factors may impact your architectural decision. The quality of the software development tools integrated with the applications, operating systems, hypervisors, and microprocessor is often overlooked by architects.
The multicore hardware debugging features (such as the aforementioned features of the P4080) must be harnessed by system analysis tools that convert the data into useful information for finding bugs, identifying performance bottlenecks, and visualizing sophisticated system behavior.
These tools may well make the difference between getting to market on time and drowning in a sea of complexity.
Stability and trustworthiness of systems software vendors is another key consideration. In today’s era of supplier consolidation and economic strife, it is increasingly difficult to make a safe long-term decision about vendors upon which you must bet the success of current and future projects.
In addition to the obvious financial and corporate stability concerns, architects must consider the availability of integrated and third party middleware stacks and drivers and the level of working relationship between the processor vendor and the software supplier.
Finally, a vendor’s adherence to open API standards and ability to provide long-term support for all leading multicore processor families enables an architect to retain flexibility of choice over time.
Modern multicore SoCs present software architects with incredible power that must be properly managed and controlled. The first step is to understand the major platform architecture choices and tradeoffs. Good decisions here can lead to successful designs for many product generations to come.