The top serial switched fabrics specified in VITA technologies include Ethernet, PCI Express, Serial RapidIO, and Infini-Band. Each has features that make it more or less suitable for specific applications. Our focus in this discussion is to provide an update on recent advancements; to learn more about choosing the right switched fabric for your embedded application, check out the following white paper: Serial Switched Fabrics, written by GE Intelligent Platforms,
ge-ip.com/news-events/detail/2884.
Ethernet
Ethernet is under the IEEE’s guidance; specifically, the IEEE 802 LAN/MAN Standards Committee (www.ieee802.org)
develops and maintains the family of Ethernet specifications.
Hot off the press is the news that the IEEE 802.3-2012 “Standard for Ethernet” was approved. IEEE 802.3 defines wired connectivity for the Ethernet local area, provides access to metropolitan area networks around the world, and expands Ethernet to address new markets, bandwidth speeds, and media types.
“IEEE 802.3 technologies and the varied Ethernet networks that they enable are found everywhere, and the standard’s application horizon continues to expand,” explained David Law, Chair of the
IEEE 802.3 Ethernet Working Group and distinguished engineer with HP Networking. “When Ethernet networking was conceived in the 1970s and the
IEEE 802.3 standard was first published in 1985, its founders could not possibly have foreseen the global transformation that their ideas and efforts would ultimately set into motion. The standard has helped spawn whole new business models, industries, and ways of life. And that cycle of innovation continues today.”
IEEE 802.3’s relevance continues to grow multidimensionally to address additional media types, bandwidth speeds, and protocols. The new IEEE 802.3 revision incorporates various technical updates and enhancements and consolidates a host of amendments to the base standard that was approved since IEEE 802.3’s last full revision, in 2008. Amendments addressing 10 Gbps Ethernet Passive Optical Networks (EPONs), energy efficiency, extension to 40 Gbps and 100 Gbps speeds while maintaining compatibility with previously installed IEEE 802.3 interfaces, enhanced support for loss-sensitive applications, and time synchronization are among those incorporated into IEEE 802.3-2012.
“For decades now, Ethernet has provided the flexible connectivity foundation on which application innovation and the bandwidth explosion have been based,” said Brad Smith, Senior Vice President and Chief Analyst at LightCounting.com, an optical transceiver and high-speed interconnect market research firm. “The reason it has proven to be such an enduring foundational technology over the years is because of the IEEE 802.3 community’s vigilance to keep the standard current with its users’ real-world needs.”
Added Wael William Diab, Vice Chair of the IEEE 802.3 working group, Chair of the revision task force, and Senior Technical Director at Broadcom: “IEEE 802.3 is constantly being refined to address new challenges and applications. We see the standard being expanded horizontally to address the specific needs of new markets such as energy efficiency, in-car networking, data-center networking, and content delivery. At the same time, IEEE 802.3’s relevance is being expanded vertically in terms of bandwidth speeds and connection media (Figure 1). Work, in fact, is already underway on a variety of fronts that will have dramatic impact on the next generations of the world’s ubiquitous wired connectivity protocol of choice.”
|
|
(Click graphic to zoom by 1.3x)
|
The IEEE 802.3 Working Group has a number of active projects in a task force or study group:
- IEEE P802.3.1 (IEEE 802.3.1a) Revision to IEEE Std 802.3.1-2011 Ethernet MIBs
- IEEE P802.3bj 100 Gb/s Backplane and Copper Cable
- IEEE P802.3bk Extended EPON
- IEEE P802.3bm 40 Gb/s and
100 Gb/s Operation Over Fiber Optic Cables - IEEE P802.3bn EPON Protocol over Coax (EPoC)
- IEEE 802.3 Reduced Twisted Pair Gigabit Ethernet (RTPGE)
- IEEE 802.3 Next Generation BASE-T
PCI Express
The PCI-SIG owns and manages the PCI Express (www.pcisig.com/specifications/pciexpress/) specifications as open industry standards. The organization defines and implements new industry standard I/O specifications as the industry’s needs evolve.
The most recent release was in November 2010: PCI Express 3.0. The PCIe 3.0 architecture includes a 128b/130b encoding scheme and a data rate of 8 Gigatransfers per second (GTps), doubling the interconnect bandwidth over the PCIe 2.0 specification. PCIe 3.0 technology also maintains backward compatibility with previous PCIe architectures and provides the optimum design point for high-volume platform I/O implementations across a wide range of topologies.
Each new version of the PCIe spec has doubled the bandwidth of the prior generation. The PCIe 3.0 specification extended the data rate to 8 GTps in a manner compatible with the exist-ing PCIe 1.x and 2.x specifications and products that support 2.5 and 5 GTps signaling. This bit rate represents the optimum trade-off between manufacturability, cost, power, complexity, and compatibility. Based on this data rate expansion, it is possible for products designed to the PCIe 3.0 architecture to achieve bandwidth near 1 GBps in one direction on a single-lane (x1) configuration and scale to an aggregate approaching 32 GBps on a 16-lane (x16) configuration. The 128b/130b encoding scheme also allows near 100 percent efficiency, offering a 25 percent efficiency increase for 8 GTps as compared to the 8b/10b efficiency of previous versions, which enables the doubled bandwidth.
PCIe 4.0 is the next evolution of the PCI Express I/O specification. At 16 GTps bit rate, the interconnect performance bandwidth will be doubled over the PCIe 3.0 specification, while preserving compatibility with software and mechanical interfaces. The key requirement for evolving the PCIe architecture is to continue to provide performance scaling consistent with bandwidth demand from a variety of applications with low cost, low power, and minimal perturbations at the platform level. The final PCIe 4.0 specifications, including form factor specification updates, are expected to be available in late 2015.
Serial RapidIO
The RapidIO standard is finely tuned for embedded applications. The RapidIO Trade Association (RTA) enables, sup-ports, and drives development of the RapidIO specification and ecosystem (rapidio.org). Defense and aerospace customers continue to rely on RapidIO for mission-critical systems and increased automation of unmanned vehicles.
The latest enhancements to the RapidIO road map were announced last summer when the RTA announced a technology road map of enhancements with the Serial RapidIO 10xN set of specifications that moves the protocol to serial lane speed of 10 Gbps and higher supporting individual port speeds that scale beyond 100 Gbps (Figure 2). The new technology road map scales the needs of RapidIO customers in the wireless, defense, aero-space, imaging, video, and server markets beyond what can be implemented in other interconnect protocols and sets the stage for OEMs to develop systems with scalable backplanes.
|
|
(Click graphic to zoom by 1.9x)
|
The RapidIO 10xN specification is backward compatible with the RapidIO Gen 1 and Gen 2 systems deployed in the market today. Initially, the RapidIO 10xN specification will support greater than 10 Gbaud per serial lane with lane widths up to x16, resulting in data rates up to 160 Gbps per port. The RapidIO 10xN set of specifications will also scale to serial lane speeds of 25 Gbps and beyond as there are no limitations in the logical and transport layer, thus allowing for the standard to keep in lockstep with mainstream PHY technology. As key to supporting applications, there is a short-reach specification for local interconnect up to 20 cm over one connector on FR4 and a long-reach specification that will support up to 1 m over one connector on FR4. Protocol efficiency is further improved by using industry-leading coding schemes that move from the 25 percent overhead of 8b/10b encoding to schemes that have less than 5 percent encoding overhead.
“I am also extremely interested in finding ways to extend the performance of RapidIO technology from 40 Gbps to 100 Gbps and beyond in the coming years,” stated Sam Fuller, RTA Executive Director.
The RapidIO Trade Association has a technology road map that provides details about RapidIO Specification Gen 2, and previews the development of specification 10xN and 25xN. The embedded systems market has a stable and longer life cycle than PC and consumer markets. The needs of this market are reflected in the collaborative efforts of the association’s members to develop the appropriate technology, cost, and performance in a timeframe suitable for the needs of its members.
InfiniBand
The InfiniBand Trade Association (IBTA), a global organization dedicated to maintaining and furthering the InfiniBand specification (see www.infinibandta.org
and Figure 3), manages InfiniBand. In July, the IBTA announced the release of a report from the Taneja Group, demonstrating the continued market growth of InfiniBand in High Performance Computing (HPC) and its emergence as an attractive choice for the core of the enterprise data center.
|
|
(Click graphic to zoom by 1.4x)
|
The July 2012 report examines technology and solution trends where higher-speed switched fabrics are required for optimal performance, including solutions like big data, web-scale applications, scale-out storage, and virtual I/O for mission-critical applications. According to the report, “Data center architects need a high-speed switched fabric at the core of the new data center. Today that choice is clearly InfiniBand.”
“We’ve witnessed InfiniBand’s growth in … HPC over the last decade due to its ability to deliver application performance and data center efficiencies, and we anticipate InfiniBand to become increasingly relevant in the enterprise data center for the same reasons,” said Bill Lee, InfiniBand Trade Association. “Big Data analysis, virtualized data centers, and scale-out web services are only a few of the applications that have seen significant performance improvement when run on InfiniBand.”
The report also observes the overall health of the InfiniBand industry, citing Intel’s recent acquisition of QLogic’s InfiniBand portfolio and InfiniBand’s longevity as proof points for data center architects to consider InfiniBand in their next-generation data center. According to the report, “This isn’t a new technology that already-stressed IT staff have to puzzle out from scratch and pray that it ‘sticks.’ Rather, since it’s proven, reliable, constantly evolving, and tested at a massive scale, InfiniBand is here to stay.”
Summary
Each of these leading serial switched fabrics is being aggressively enhanced by its respective keeper to meet the growing demands for more bandwidth (Table 1). Because each does have unique advantages, many systems use a combination of fabrics: Ethernet for control and PCI Express or Serial RapidIO for the data plane. System architects will continue to be creative in their use of serial switched fabrics to develop next-generation critical embedded systems.