To that end, VITA switched fabric roadmaps are highly dependent on the roadmaps of industry established switched fabrics. In this article we will take a look at the current state of the art for the major serial fabrics and how they are being addressed in various VITA form factors.
A switched fabric is a network topology in which network nodes interconnect via one or more network switches in various topologies. Because a switched fabric network spreads network traffic across multiple physical links, it can yield higher total throughput than parallel buses or broadcast networks. Switched fabrics became popular in the early 2000’s and are now the dominant interconnect technique for high performance computing.
There are many tricks to extending bandwidth within given limitations. For example, making links wider and using a different modulation scheme to gain more per link bandwidth. All switched fabrics improve bandwidth performance in combinations of the following:
- Baud rates: Increasing baud rates increase potential bandwidth. Laws of physics limit performance, limits which we are quickly approaching.
- Increasing links: Adding more links or lanes to increases bandwidth. Spreading the traffic out into more links is like increasing lanes on a freeway, allowing more traffic to flow at the same speed.
- Modulation scheme: Implementing various modulation schemes enables more bandwidth per link configuration to improve pin efficiency.
- Transmission media: Moving from copper media to optical media gains the performance advantage of moving from slower electrons to much faster protons. Limitations in optical media make it hard to use this in all cases, but when it can be used, tremendous performance gains are achieved.
The good news for VITA working groups is that all of the serial fabrics are on a similar performance curve, challenged with the same advancements in transceiver and connector technologies. The VITA roadmap benefits from this in that we can make improvements across the appropriate VITA standards as determined by the target applications.
Ethernet
https://ethernetalliance.org/the-2018-ethernet-roadmap/
Ethernet is the industry workhorse. Having been around even longer than VITA, it has evolved to stay relevant in today’s most demanding applications. In March, the Ethernet Alliance, the Voice of Ethernet, released the 2018 version of its roadmap (See Figure 1). The map provides the industry with the directions needed to navigate the many roads making up today’s Ethernet ecosystem. With all the new standards released by the IEEE 802.3 Ethernet Working Group and the supporting technologies, navigating today’s Ethernet has grown increasingly complex.
|
|
Ethernet is used in a variety of ways in a typical system, from a backplane connection to tradition Ethernet ports, making cutting through all of the variations of Ethernet an interesting challenge. In the VITA technology application space, the relevant working groups consider the Ethernet roadmap before the appropriate speeds and solutions can be determined. The 2018 Ethernet Roadmap helps them understand how Ethernet is achieving 100 GbE [Gigabit Ethernet], 200 GbE, and 400 GbE. The roadmap also provides guidance in which Ethernet configurations are best suited for backplanes as utilized in many VITA standards.
PCIe
https://pcisig.com/
The Peripheral Component Interconnect Special Interest Group (PCI-SIG) roadmap has PCI 5.0 ready in 2019 at 128 Gigabit per second (GB/s). Early revisions of PCIe 5.0 have been made available to the SIG’s 700+ members to consider. Many of those members are also hard at work on products for version 4.0, which at version 0.9, is considered “feature complete” and awaits only final signoff. The SIG says it’s seeing “unprecedented interest in our PCIe 4.0 compliance testing and early adopters have already tested 16GT/s solutions.” The PCI-SIG is driven to double the bandwidth every three years, if only to keep up with other expected accelerations like the advent of 400 Gbps Ethernet.
Most implementations today are at PCIe 3.0. VITA applications are dependent on the integration of PCIe into processors, bridges and I/O devices. As these devices start to be used in next generation designs, the pressure will increase to improve the connector technology used in the various VITA standards. At some point, PCIe performance exceeds what is possible on a copper backplane, PCIe 5.0 may be that point. (See Table 1).
|
|
InfiniBand
https://www.infinibandta.org/infiniband-roadmap/
InfiniBand is used in a variety of VPX products. A contraction of “Infinite Bandwidth,” in theory, a designer can keep bundling links so that there is no theoretical bandwidth limit. InfiniBand is a favorite serial fabric for high performance computing and storage systems.
The InfiniBand Trade Association’s (IBTA’s) InfiniBand roadmap is continuously developed as a collaborative effort from the various IBTA working groups. Members of the IBTA working groups include leading enterprise IT vendors who are actively contributing to the advancement of InfiniBand. The InfiniBand roadmap details 1x, 2x, 4x, and 12x port widths with bandwidths reaching 600 Gb/s data rate HDR (High Data Rate) in the middle of 2018 and 1.2Tb/s data rate NDR (Next Data Rate) in 2020. The roadmap (See Figure 2) is intended to keep the rate of InfiniBand performance increase in line with systems-level performance gains.
|
|
RapidIO
http://www.rapidio.org/rapidio-roadmap/
Though not as widely implemented on VITA technology-based products, RapidIO is used with several VPX products, particularly those based on Power Architecture processors. The RapidIO Trade Association has mapped out a plan for future performance enhancements that keep it in step with the other switched fabrics (See Figure 3).
|
|
Current situation
Pin count and connector capability are the two major limiting factors in most VITA form factor standards. Where possible, the working groups project as far into the future as is reasonable. Here is a recap of where the VITA technologies stand today.
VPX
The VITA 46 working group, responsible for the core VPX standards, is currently documenting updates to the standard. Top on the list are performance enhancements to take VPX to 25 Gbps and beyond. New connector technology has been proposed and plans for testing are underway. This should be enough head room to accommodate bandwidth performance improvements on the near horizon. Expect to see an approved standard in 2019 with products to soon follow.
VPX has also defined a set of standards (VITA 66.x Optical Interconnect On VPX) that support optical interconnects in several different blind mate connector styles. While not a backplane distributed solution, VITA 66 provides a very high bandwidth channel for direct access to VPX slots. These optical interconnect options provide even more bandwidth for the most demanding applications. The same switched fabric protocols used on the copper interconnections can also be used on these optical links.
FMC
FMC+ ANSI/VITA 57.4-2018 was recently ratified, bringing the maximum multigigabit interfaces to 32 full duplex channels. Additionally, throughput per multigigabit interface has increased to 28 Gbps in each direction. This allows for higher data rates and expanded bandwidth to fit within the same form factor as FMC. The next steps here will likely lead to a new form factor and optical interconnections. FMCs with optical I/O are already in production, pushing the need for more bandwidth to the carrier even harder.
XMC
The popular mezzanine card, XMC, supports PCIe 3.0. Nothing is currently in the works but watch for new mezzanine standards to emerge as processor technology shifts to faster fabric solutions. Optical interconnections could also be in the “cards.”
Faster in the Future
New schemes for improving data movement are continuously being proposed for study, this is one area where there is no shortage of innovation. One protocol scheme in particular has started some spirited conversation among a small group of VITA members. Chord signaling, developed by Kandou Bus, is a multiwire version of NRZ signaling, CNRZ-5 encoding, which transmits five bits over six wires (“C” stands for chord signaling). Even more intriguing is the drive of the Optical Internetworking Forum (OIF) to go beyond 112 Gbps at the chip die level, where hopefully some of the innovation will lead to improvements that can be extended to VITA technology.
“Big data” systems are the driver for serial fabrics. VITA’s role is to be vigilant in tracking new advancements and then working with the user side of the VITA technology ecosystem to be sure connector technology is mapped to the appropriate form factors to meet the user’s needs with a mainstream, cost-effective solution.