This article, written by IDT's interconnect and serial-switching team, is a review of how certain standards support specific applications

Traditional parallel bus architectures have served network, communications, server and storage equipment designers well over the last few decades.  Standards-based buses, from VME and PCI to the more recently deployed Compact PCI and PCI-X, have consistently offered a highly reliable, scaleable and cost-effective mechanism for data transport.

But recently it has become increasingly clear that these solutions have reached their performance limits.  As the speed of processors and I/O devices has relentlessly continued to escalate, so too has the demand for higher speed system buses to link those devices.

However, there are only two ways to accelerate the performance of a parallel bus:  increase its clock rate or widen the bus.  Higher clock rates present severe timing problems as line impedance, signal load and trace routes contribute to signal skew.  Widening the bus also aggravates the skew problem, increases power consumption and drives up the number of I/O pins on the bus interface IC.  That, in turn, drives up cost.

Serial approach

To address these limitations, over the last few years the industry has turned to a variety of serial-bus standards for the design of higher bandwidth communications, network, server and storage systems.  By eliminating skew between data and clock lines, these self-clocking bus structures support faster data rates and higher bandwidth per pin.  In the process, they lay the foundation for a new generation of highly scaleable and cost-effective systems.

However, the industry has not opted to rally around one specific serial bus technology for all applications.  Instead, developers have crafted a variety of different serial interconnect standards to deliver the most optimal solution for different applications.  Each technology brings its own unique performance attributes, cost structures, management capabilities and levels of service and reliability.  Each also offers distinct opportunities for expansion and its own roadmap to future product generations.

To help systems designers sort out their options as they move into this emerging technology, this article will assess the relative merits of leading serial interconnects and identify applications for which these new bus structures offer the best fit.

Communications and networking systems

For the large number of wireline and wireless communications applications where a data plane is needed, designers must weigh the relative merits of Gigabit Ethernet and the Advanced Switching Interconnect (ASI).  From a pure cost and ease-of-deployment standpoint, the widely deployed Gigabit Ethernet standard offers a very attractive solution. Ethernet’s tremendous manufacturing scale translates into lower costs than any other currently available solution.  Furthermore, over its long, extensive development history, the protocol has become widely familiar to a large population of designers and OEMs.   

But Ethernet’s inherent performance limitations restrict its applicability.  The standard cannot support carrier-class Quality of Service (QoS) or provide performance guarantees despite over-provisioning.  Its traffic-management capabilities are constrained by the limited capabilities of the Ethernet switch, and its origins in unreliable transmission mediums pose significant reliability issues.  It permits dropped packets, uses inefficient TCP/IP layers for end-to-end reliability and offers minimal flow-control capabilities.  Given these limitations, it will likely be used only in those access/aggregation systems employing native Ethernet uplinks. 

ASI, on the other hand, was designed from its inception to support carrier-class performance (See Figure 1).  To meet those requirements it provides QoS, a high level of reliability, a performance roadmap that extends to 10 Gigabit per second (Gbit/s) and beyond and extensive scaleability. 

Figure 1

Unlike competing serial-interconnect standards, ASI offers developers a comprehensive foundation for implementing multipoint, peer-to-peer switched interconnect links for both data plane and control plane communications.  Its congestion management and flow-control features are extensive.  For flow control, the architecture uses a credit-based mechanism between entities per each virtual channel.  Data traffic is differentiated and isolated through the use of different class identifiers, a virtual channel queuing mechanism and egress link scheduling.

To avoid congestion as traffic volume approaches fabric capacity, ASI offers more functions than any other serial bus standard, including a status-based flow control mechanism, a minimum bandwidth scheduler and end-point source limiting.  The standard maximizes availability by devoting a specific protocol interface for event handling and management.  Link layer, transaction layer header and payload CRCs ensure end-to-end packet integrity and provide error notification.  ASI also adds extensive security and multicasting capabilities.
 
Moreover, long term, ASI promises to offer a very competitive cost structure.  Designed from the ground up as an extension of the PCI Express (PCIe) architecture, ASI reuses the physical and data link layers of PCIe and adds a modified transaction layer to deliver the functionality required to meet the needs of a high-performance backplane interconnect.  This allows ASI to offer an extensive array of high-performance features, while leveraging the software base and economies of scale associated with the broad-deployed PCI infrastructure. 

Enterprise servers and storage

As system data rates continue to rise in the enterprise server and storage arena, demand will grow for higher speed, easier-to-implement and low-cost serial interconnects, first at the chip-to-chip level and eventually for the backplane. 

Offering seamless compatibility with legacy PCI systems, PCIe, a serial interconnect standard that retains software compatibility with the extensive installed base of PCI systems, has already been widely accepted at the chip-to-chip level.  The specification defines a packetized protocol and layered architecture capable of supporting initial data rates of 2.5Gbit/s.  In addition, it adds QoS, integrated power management, native hot plug capability, high bandwidth per pin efficiency, error reporting, and recovery and correction.

In server applications, PCIe provides additional I/O slots and on-board connectivity. This situation is more likely to occur on 2-way and 4-way servers. Legacy slots to PCI-X are supported with a PCIe to PCI-X bridge (see Figure 2).

Figure 2

As storage systems begin to migrate to PCIe, there will be an increased demand for PCIe switches capable of supporting I/O expansion for high-performance, data-intensive end-points, such as Fibre Channel and Raid Adapter Cards, as well as redundant 1:1 architectures through the use of non-transparent bridging (see Figure 3).

Figure 3

Eventually PCIe switches and bridges will offer designers the most cost effective way to deliver I/O expansion capability while retaining bridging to legacy PCI/PCI-X systems.

For high performance backplane interconnects in both mid- to high-end server and storage applications, ASI addresses the future needs for disaggregated architectures, I/O virtualisation and interprocessor communications.  Over the long term, ASI’s price/performance advantages will be driven by a rapid expanding PCIe infrastructure. 

Embedded processor and DSP cluster applications

Some embedded applications, such as DSP blades in wireless basestation applications, demand a leaner, simpler approach to data transfer and protocol management.  These computationally intense applications typically require the system to move data quickly between signal processors in a tightly coupled DSP farm.  Usually, they do not require the QoS or protocol-management functions associated with high-speed system backplanes.

For these embedded applications requiring tightly coupled compute elements, the Serial RapidIO (sRIO) standard offers a number of advantages.  A simple chip-to-chip serial link targeted at processor interconnect applications, sRIO eliminates the large transaction overhead associated with standards such as Ethernet.  Instead, it is expressly designed for short payload interprocessor communications, such as the use of AAL2-based ATM cells to carry packet voice in a 3G basestation.  Using point-to-point links, sRIO allows processors to communicate directly with one another via sourced-directed addressing where a transaction contains the explicit address of the destination device (see Figure 4).  Data is routed to its destination using routing tables embedded in switches.  To support real-time data flows across these links and minimize latency, sRIO adds error detection and recovery in hardware.

Figure 4

High-end computing

In a small number of high-end computing applications where system designers cluster large numbers of processors in highly parallel, multiprocessing-type configurations, Infiniband, the first standard to combine low latency, high reliability and extensive scaleability, offers some advantages.  Its software-intensive protocol, complex addressing and routing schemes support the scaling of chassis-to-chassis interconnects to hundreds or thousands of nodes.  But given its extensive software overhead and ASI’s ability to leverage the cost advantages of the PCIe ecosystem, Infiniband will likely be limited to a very small number of mid-to-high-end server cluster and SAN applications.

Conclusion

Clearly, designers using serial interconnects to build the next generation of high-speed systems must carefully weigh a number of difficult choices.  Each of the new serial interconnect standards discussed above offers designers a specific set of features. And each carries with those technical characteristics its own set of advantages and disadvantages.

It is important to note, however, that the ultimate success or failure of any standard in the electronics industry has historically been driven as much by industry adoption rates as by a specification’s technical merits.  Standards that offer not only superior technical capabilities, but can also exploit an extensive installed base of legacy systems and the economies of scale that infrastucture brings, are at a distinct advantage.  More often than not, those standards offer designers lower costs, reduced risk and a higher probability of product success.  

About the authors:
Harpinder Singh is principal engineer in the IDT Serial-Switching Division
John Chiang is a product manager for the IDT Serial-Switching Division
Mario Montana is general manager of the IDT Serial-Switching Division
Bill Beane is product marketing manager in the IDT Flow-Control Mangement Division

www.idt.com