By Ann R Thryft, Contributing Technical Editor -EDN
, 1/5/2008
At A Glance
- The difficulty of implementing 10-GbE (gigabit Ethernet) over
100m copper, which the 10GBaseT spec stipulated in June 2006, has
delayed the widespread deployment of 10-GbE. Major drivers for
10-GbE include server virtualization and 1-GbE link
aggregation.
- 10GBaseT's complex technology has resulted in expensive PHY
(physical)-interface chips that run too hot.
- Second-generation 10-GbE PHY silicon at the 65-nm-process node,
due in 2008 and 2009, will help cut PHY-chip power consumption to
approximately 5 to 6W, will improve design, and will lower costs
through greater integration.
- Manufacturers are developing direct-attachment copper-twinaxial
cables for use with smaller-footprint, lower-power SFP+ optical
hot-pluggable-transceiver modules, offering an alternative for
10-GbE over distances of 10 to 15m within data centers.
For several years, next-generation Ethernet capable of 10-Gbps
speeds has been on the brink of entering the mainstream. Some areas
of the network have for some time been using optical technology,
and demand is increasing for 10-GbE (gigabit-Ethernet) data rates
as traffic increases.
But vastly more complex technology than in previous generations of
Ethernet is necessary to run 10-GbE over 100m lengths of copper.
That requirement has resulted in expensive MAC
(media-access-controller) and switch silicon and costly and
inconvenient PHY (physical)-layer-interface chips, limiting the
speedy LAN to high-performance applications covering shorter
distances within data centers.
This picture is beginning to change, but widespread deployment
of 10-GbE is not likely for at least two years
(Figure 1).
In June 2006, the IEEE finalized the 10GBaseT P802.3an spec for
operation over 100m copper. Switch and controller chips that meet
the spec are now available, but, along with PHY chips, they are
expensive and consume too much power. These problems have been
especially severe with 10GBaseT PHY-layer chips.
Silicon power consumption must be less than 5W/port for use in
commercially available equipment in chips such as switches and NICs
(network-interface controllers). “That’s not the case with most of
these chips now,” says Alan Weckel, senior analyst for Dell’Oro
Group. “In contrast, optical pluggable [transceiver] modules
consume less than 1W.” Most 10GBaseT PHY chips currently consume 8
to 10W per port.
Although per-port prices have for some time been falling, lower
component prices alone don’t necessarily create demand. “It will
take time to ramp up volumes in the next couple of years,” says Jag
Bolaria, senior analyst for
The Linley Group.
Meanwhile, the cost of optical fiber is decreasing, “and the early
stages of 10GBaseT could be the next kicker in the road in getting
volumes up.”
The need for speed
Analysts expect a fairly rapid ramp for 10-GbE technology over
the next four to five years. Drivers include sheer bandwidth
demands because of server-performance increases and other factors,
such as virtualization (see
“Virtualization and 10-GbE”).
The demand for 10-GbE over copper, including 10GBaseT, comes from
the fact that copper costs less and is easier to install than
fiber, even though 10GBaseT requires unshielded Category 6A or
shielded Category 7 cable to meet the spec’s maximum distance of
100m. Although fiber will still find use for longer runs, the
shorter enterprise and data-center runs need less expensive,
10-Gbps links.
During the next five years, 10-GbE should replace 1-GbE in two
major applications that will drive significant port growth,
according to Dell’Oro Group. In wiring-closet-switch uplinks, most
10-GbE fixed ports will be uplinks on 24- and 48-port 1-GbE
switches, and the remaining 10-GbE ports will be 10-GbE-only
switches, those finding use at the top of the stack for
aggregation. The other major application area is in direct server
connections.
Currently, copper finds use mainly between switches and PCs and
between switches and servers. The uplinks from wiring-closet
switches to data centers that are currently optical links will
remain optical and will have significant potential for volume, says
Weckel.
Because enterprises are undergoing a major wiring-closet upgrade,
the market for switching is increasing, as well. In
switch-to-switch connections, the medium for 1-GbE is now optical
and will remain optical at 10-GbE. In direct-server connections,
the medium is copper, and it will remain copper at the higher
speed.
As the number of 1-GbE ports increases, aggregation of those
ports becomes a major reason to begin using 10-GbE technology.
Without it, the uplink bandwidth is less than the bandwidth of the
downstream ports, and blocking occurs. Tier 1 OEMs are selling
1-GbE gear for around $100/port, says Kamal Dalmia, vice president
of marketing for
Teranetics.
For 10GBaseT switches at their introduction, the price will be
approximately $500/port, a per-gigabit cost of roughly half that of
1-GbE equipment. Another main driver is the need to connect
high-performance computing-blade servers in data centers with
bandwidth commensurate with their processing speeds. Those servers
include eight or 16 processors, each of which is increasingly
likely to contain four or even eight cores, ramping up power
requirements and speed.
Aside from the combined power-consumption and technology issues
in implementing the technology, another major issue could hinder
the rate of 10-GbE deployments, says The Linley Group’s Bolaria. If
the cost of implementing one 10-GbE port is too high, it may make
more sense to aggregate links by combining two to four 1-GbE ports;
more than that number would be expensive and cumbersome.
More complexity, power
The first 10-GbE standard, IEEE 802.3ae, which originated in
2002, specified several PHY interfaces for optical-transmission
media. In 2004, the IEEE 802.3ak-2004 short-reach amendment allowed
10-GbE over 15m coaxial cabling using 10GbaseCX4 PHY-layer chips.
Compared with 1000BaseT, 10GBaseT data rates require highly complex
digital-signal processing in silicon to deal with echo cancellation
and crosstalk cancellation, as well as more sophisticated analog
circuits.
All of this processing to clean up the received signal must take
place on links that are 10 times faster than the 1000BaseT spec.
The 1000BaseT spec left a wide margin for compensation, but
10GBaseT does not, says Brad Booth, who acted as chairman of the
IEEE 10GBaseT task force and is chairman of the
Ethernet Alliance.
The 10GBaseT spec includes a subset for a short-reach PHY
interface over a maximum 30m of Category 6 four-wire UTP
(unshielded-twisted-pair) cable as a means of going to a
lower-power operation mode. When the spec’s developers wrote the
draft, the concern was that most devices for a 100m range would
consume 10 to 12W of power, and those levels of power consumption
limit 10-GbE deployment in switch applications, says Booth.
In a shorter-reach application with the 30m option, the PHY-layer
device could drop to lower power, which helps eliminate a lot of
noise along with many cancellation circuits. The spec also allows
the use of Category 6 cable for distances as long as 55m, because
70% of data centers’ reach is 55m or less. However, operation over
this distance doesn’t buy lower power consumption.
Additional PHY-silicon issues are the need for multiple ports to
pare down the cost per port, and the inclusion of multispeed ports
for backward compatibility with 1-Gbps and even 100-Mbps Ethernet.
An alternative method of connecting 10-GbE over copper may be
possible using a new form factor in hot-pluggable
optical-transceiver modules.
The SFP+ (small-form-factor-pluggable-plus) module has a smaller
footprint than the previous SFP form factor and consumes less
power, allowing greater module density on a line card and offering
lower per-port costs. Manufacturers are also developing
direct-attachment SFP+ copper-twinaxial cables for distances of
approximately 10 to 15m, which are adequate for connections within
a data center.
10-GbE-silicon issues
The process technology for most chips now implementing 10-GbE
over copper is 130 or 90 nm. The next generation may go to 65 nm,
which is the next-lowest-cost node at which volumes may increase,
says The Linley Group’s Bolaria. Some in the industry believe that
it will take a 45-nm process to reach the lower power consumption
required to drive volumes and are trying to integrate some of the
analog circuitry necessary for 10GBaseT at that process node. But
the high amount of analog circuitry could be a problem in such
smaller geometries, says the Ethernet Alliance’s Booth.
“One alternative may be multichip modules with analog front ends
running in one process technology and digital back ends,” he
says.
Definitions of “high volume” can vary. Huge volumes are on the
order of 10 million and 20 million ports, says Bolaria. In 2007,
volumes for 10-GbE-switch ports with 10W PHY interfaces were
approximately 640,000, and they contained 10W PHY-layer
chips.
“The next tier would be less than 5W, which will enable
[manufacturers to ship] a few million ports,” he says. “But, to get
to 20 million in port volume, PHY silicon will probably need to
consume less than 2W.” As ports per chip increase, chip volumes
will decline somewhat.
The power budget of a typical first-generation 10-GbE endpoint
NIC is approximately 25W, but designers usually want to stay closer
to 15W, says Blaine Kohl, vice president of
Tehuti Networks. You
can build a single-port-10-GbE NIC with a 7W PHY chip, but you can
tweak the power allocation even more in some designs. Manufacturers
will most likely build single- and dual-port adapters for endpoints
using this year’s generation of 10GBaseT PHY silicon.
However, switches need PHY chips closer to 3W. By 2010, 3 to 4W
10GBaseT PHY chips will probably be available. At that point, with
2 to 3W controllers, you may be able to integrate a controller chip
and a PHY chip in one package. Switches could then adopt a 10GBaseT
PHY chip, endpoints could adopt the 10GBaseT package, and
manufacturers could be shipping 10GBaseT equipment in volume.
After offering samples of 10-GbE-controller and 10GBaseT PHY
chips for a year and a half,
Solarflare Communications
has just introduced second-generation, 65-nm parts. The SFT9000
10GBaseT PHY chip consumes less than 6W and features multispeed
autonegotiation at speeds as low as 100 Mbps.
The SFC4500 10-GbE controller chip consumes 2.2W and features
virtualization acceleration. Solarflare expects next year to make
available a single-chip LAN-on-motherboard device that will
integrate 10GBaseT-PHY and 10-GbE-controller silicon, according to
Bruce Tolley, vice president of marketing.
Teranetics’ first-generation all-CMOS TN1010 10GBaseT multirate
PHY chip supports 1-GbE and 100-Mbps Ethernet. The company’s
next-generation 65-nm chip will be available this year, with 30 to
40% less power consumption than the current total of about 10W,
says CEO Matt Rhodes.
Fulcrum Microsystems’
single-chip, 24-port FM4000 10-GbE IP (Internet Protocol) Version
4/6 switch/router chips target use in data-center-switching
platforms in high-performance-computing, server-, and storage-host
interconnection, and data-center-aggregation applications. The
Layers 2/3/4 chips have full line-rate performance on all ports
with a total throughput of 360 million packets/sec.
Their 300-nsec latency provides a highly responsive network fabric
that exceeds the performance of specialty fabrics, such as
InfiniBand and Fibre Channel, and suits
high-performance-clustered-computing applications, says Bob Nunn,
Teranetics’ president and CEO.
Broadcom makes
single-chip, 10-GbE switch and controller silicon and optical
transceivers. The 65-nm process technology for these chips requires
a significant effort, especially for the switches, because they are
large, highly integrated chips with both analog and digital
components, says Eric Hayes, director of marketing for the
network-switching line of business. The 65-nm BCM56820 switch
chip’s power consumption is 1W per 10-GbE port—a drastic reduction
from the previous 130-nm switch chip’s 11W per port
(Figure 2).
“We’re seeing a clear requirement to go to 10-GbE with all of the
features and capabilities that were available at 1-GbE, such as
security, strong Layer 4 classification for QOS [quality of
service], and Layer 3 routing,” he says.
Tehuti Networks based its new SFP+-adapter-reference designs,
which include the single-port TN7587-S and dual-port TN7587-D NICs
for optical 10-GbE, on the company’s single-chip TN3016 10-GbE
single- or dual-port controllers. The reference designs include
AMCC’s (Applied Micro Circuits
Corp’s) 10-GbE SFP+ QT2025 PHY chip.
When you populate the NIC with two SFP+ optical modules, the
TN7587-D’s power dissipation is 15W. The single- and dual-port
controllers dissipate 6 and 7W, respectively.
What's next?
Many enterprises have just begun to consider 40- and 100-GbE for
aggregating 10-Gbps links in data centers.
“Even telephone companies are requesting 100-Gbit products,” says
The Linley Group’s Bolaria. “There’s demand for it now in the core
of the network.”
Because of video-on-demand and Web-based applications with millions
of simultaneous users, such as Facebook, Netflix, and YouTube, some
industry observers predict that the bandwidth-scaling needs of
carriers and ISPs (Internet-service providers) may bypass
40-GbE.
For the first time, however, no aggregation technology is in
place for the upcoming lower speed of Ethernet, which could hinder
the market, says Dell’Oro Group’s Weckel. For the lower speed to be
successful, manufacturers must be shipping products using the
higher speed, even if not in high volumes.
But with 10-GbE to the server, no 40- or 100-GbE line cards exist
to provide uplinks to data centers on the aggregation side. “The
higher-speed uplink technology must be present for there to be
truly widespread adoption,” he says.
The pressure will increase even more when servers begin to
appear with 10-GbE interfaces on the motherboard, which may occur
with the next generation of
Intel server CPUs. Ultimately,
10-GbE may go all the way to the desktop. For many, Ethernet has
become the fabric of choice in the network, and that situation is
also driving 10-GbE deployment.