Power Distribution Architectures White Paper

Power Distribution Architectures

Revision 2

by Waite D. Ave

Universal Networking Services

Power Distribution Architectures

Part 1 - Summary
Typically, power is distributed to IT loads at 400/230VAC. But in North America, however, power is
traditionally distributed at 208/120VAC, which results in deficiencies including greater cost, lower
eciency, and greater space consumed. Using 415/240VAC distribution enables higher density per rack
without adding extra circuit breakers as would be the case with 208/120VAC distribution.

Common voltages:

400/230VAC

415/240VAC

480/277VAC

208/120VAC

The designation of the two values

when written as 208/120 or
415/240 represents the line to line
voltage (first number) and line to
neutral voltage (second number). In
three-phase system with power
factor equal to 1, the value of the
line to neutral voltage is equal to the
line to line voltage divided by the
square root of 3 (i.e. 120 = 208 / 3).

Today, virtually all IT equipment is designed for worldwide compatibility and can operate on a voltage
range of 100 to 240 volts. This means it can operate on North American voltages of 208 and 120, the
Japanese voltages of 200 and 100, and on 230 volts in the rest of the world. Operating IT equipment at
its highest possible voltage of 240 volts is most ecient.
Known North American distribution system deficiencies:

Low distribution voltage (120volts) increases energy costit is a law of physics that for the same
amount of power, as voltage decreases, current increases. The more current that runs through a wire,
the higher the electrical losses and energy cost.

Low distribution voltage(120volts) that increases copper costthe lower the voltage, the thicker the
wire required to carry the higher current.

Oversized traditional PDUs that consume floor space and increase floor weightthe increased floor
space consumption and floor weight is mainly due to the heavy isolation transformers inside the PDUs
and the large cabinets that enclose them.

Oversized PDU isolation transformers that increase energy costsThe collective PDU capacity in many
data centers are 1.5 to 3 times the data center capacity which increases electrical transformer losses
and energy costs.

Multiple branch circuits per rack, with corresponding breakers and cabling - at a lower voltage more
branch circuits are required for a given rack density which increases cable clutter, complexity and cost.

The deficiencies explained above increase in magnitude as rack power densities increase. High density
server installations where racks can draw from 10 kW to 30 kW per rack place significant strains on the
traditional power distribution system.

Universal Networking Services

Power Distribution Architectures

Lets do some math..
An increase in circuit voltage while holding current constant increases the capacity of that circuit. The
power capacity for a three-phase branch circuit is calculated by using the following formula:

Power = Volts x Amps x 3

The volts in this formula refers to the line to neutral voltage. Therefore, when comparing the line to
neutral voltages of both distribution methods, the comparison should be between 120 V and 240 V and
NOT between 208 V and 240 V. For example, assume 20 amp circuits are provided to the load in either
case. The power capacity for the 120 V line to neutral distribution method is calculated as:

120V branch circuit power capacity = 20 amps x 120 V x 3 = 7.2 kW

However, the capacity for the 240 V line to neutral distribution method is calculated as:

240V branch circuit power capacity = 20 amps x 240 V x 3 = 14.4 kW

Given the same circuit current rating, the 240 V distribution method provides 100% more power
than the 120 V distribution method.

The National Electrical Code (NEC), NFPA 70 and NFPA 70E, as well as local electrical codes,
provide minimum standards and requirements in the area of wiring design and protection, wiring
methods and materials, as well as equipment for general use with the overall goal of providing safe
electrical distribution systems and equipment.

Part 2 - Types of Power Distribution

Panel Board Distribution - In panel board distribution, the main data center power is distributed to
multiple wall-mounted panelboards. Panelboards (typically rated from 1.5 kVA to 75 kVA) may come
assembled by the vendor or they may be assembled in the field by the electrical contractor. Individual
branch circuits or power cables are cut, terminated, and connected in the field by the contractor. The
cables are run either overhead in cable trays (also called ladder trays), or they are run under the raised
floor (sometimes in trays under the floor, and sometimes placed directly on the floor) to provide power
close to the IT rack enclosures. Wall mount panelboards are a very low cost power distribution strategy,

Universal Networking Services

Power Distribution Architectures

made up of parts that an electrician can acquire in days, not weeks, to implement quickly. This approach
is very much custom engineered to meet the unique needs of a particular data center.

Advantages

Lowest first cost, primarily driven by lower cost components

Accommodates unusual physical room constraints

Electrician has more flexibility in breaker and cable combinations, since they are not choosing from preconfigured assemblies

Parts can be obtained very fast, i.e. from a local electrical supply

Disadvantages

Increased risk of human error since installations are custom engineered and there is greater reliance on the
quality of work and skill of the electrician

If the plenum is used for distributing air, cabling restricts high volume air required by modern IT equipment
(over time) which impacts eectiveness of cooling distribution

Creates an environment that is not readily changeable

Cable tracing & removing cables can be dicult because of the placement of the power
cables under the floor or in trays

Usually used

For smaller capacity installations and when lowest first cost is a top priority

When IT changes are not likely or frequent

When density per IT rack is low (if power cables are distributed in air plenum)

Traditional Field Wired PDUs - In traditional power distribution systems, the main data center power is
distributed to multiple power distribution units (PDUs), typically rated from 50kVA to 500kVA, throughout
the IT space. Sometimes, PDUs without transformers are referred to as RPPs (Remote Power Panels).
PDUs are fed from centralized sub-feed breakers and are generally placed along the perimeter of the
space, throughout the room. Some have an IT rack form factor that line up in a row of racks providing
improved aesthetics for the space and bring the distribution closer to the load. Branch circuits are
distributed from the PDUs to the IT equipment. Each IT rack enclosure uses one or more branch circuits.

Universal Networking Services

Power Distribution Architectures

Advantages

Higher degree of monitoring options than panel board approach

Low first cost, primarily driven by lower PDU cost than factory-configured or modular
PDUs

Accommodates unusual physical room constraints and allows for more strategic placement of the
distribution when compared to wall-mounted panelboards.

Electrician has more flexibility in breaker and cable combinations, since they are not choosing from preconfigured assemblies

Disadvantages

Increased risk of human error since field-wired designs mean greater reliance on the quality of work and
skill of the electrician (i.e. wire terminations done in the field may result in loose connections and other
defects)

Limited warranty of whole power distribution system, since components like breakers and power cables are
installed in the field

If the plenum is used for distributing air, cabling restricts high volume air required by modern IT equipment
(over time) which impacts eectiveness of cooling distribution

Cable tracing & removing cables can be dicult because of the placement of the power cables under the
floor or in trays

Usually used

When low first cost is a priority compared to a factory-configured & modular distribution

When floor space is available but unique room constraints limit the use of factory-
configured or modular designs

When IT changes are not likely or frequent and growth of IT is done at the pod level

When the IT layout is unknown at the time of the PDU specification, since exact branch breaker ratings and
power cable lengths do not get specified until closer to installation

Traditional Factory-Configured PDUs - With factory-configured distribution, much of the field-wiring

work that is otherwise done on site is done in a controlled factory. The PDU is configured to the
customers requirements with factory-assembled branch breakers with power cables pre-cut and
terminated to the length and ampacities required by the IT racks. These cable /breaker assemblies are
installed onto the PDU at the factory, so that the only onsite work involves running the conduit from a
sub-feed breaker to the PDU input, and then routing the pre-connected cables to the appropriate IT

Universal Networking Services

Power Distribution Architectures

racks. By standardizing the power distribution solution, data center projects can eliminate costly and
time-consuming one-time engineering.

Advantages

Pre-fabricated breaker / power cable assemblies improve reliability by minimizing field work

Integrated design with integrated and pre-configured intelligence for better capacity and change
management as load requirements change

System-level warranty of whole power distribution system, since components are de- signed, tested, and
integrated in a factory

Accommodates unusual room constraints since PDUs can be rolled into any location on IT floor

Lower first cost than modular distribution approaches

Disadvantages

The IT room layout must be understood earlier in the planning cycle, since specific breakers and cable
lengths are procured with the system; better coordination between IT and facilities is typically necessary

Higher field cost to install new power cables and breakers as load requirements change compared to
modular distribution approaches

Cable tracing & removing cables can be dicult because of large volumes of cables overlayed in troughs

Takes up space on the IT room floor

Heavier shipping weight since it ships with all breakers and cables attached

Usually used

When a data center plans to scale their future IT at the pod level

When a data center requires portable equipment due to likelihood of future move

When space is not constrained and low first cost is a priority

Overhead/Underfloor Busway Distribution - In order to meet the modern IT requirements, alternative

power distribution approaches are appearing in data centers. These approaches are more flexible, more
manageable, more reliable, and more ecient. Specifically, they include the following attributes:

Integrated branch circuit power metering Capacity and redundancy are managed on every circuit

Flexible changeable power cords with tool-less expansion IT zones and associated power distribution
can be deployed over time, by anyone

Reduced footprint Less floor space required than with traditional PDUs

High eciency Transformer-less options and minimized copper use

Universal Networking Services

Power Distribution Architectures

Two modular distribution systems can achieve these desired characteristics:

Overhead / underfloor modular distribution using plug-in units powered by busway

overhead or underfloor to feed IT enclosures

Floor-mount modular distribution using branch circuit cables, distributed overhead in

cable troughs to the IT enclosures, that are pre-terminated with breaker modules that
plug into a finger-safe backplane of a modular PDU. In general, these approaches have
a higher first cost per watt than traditional distribution. However, from a lifetime cost
perspective, these approaches have lower TCO because of the ability to implement
changes faster, avoid stranded capacity (oversizing) with better capacity management,
improve eciency, and reduce maintenance expense.

Advantages

Zero footprint on IT room floor, making more space available for IT equipment

Improved cable management and tracing, with distribution cable from plug-in unit
located directly over or under IT racks

Inventory of cables can be reduced dramatically with fixed distance from busway to each IT rack

Tool-less pre-assembled plug in units improves the reliability by eliminating field wire cutting and
terminating branch circuits

System-level warranty of whole power distribution system, since components are de- signed, tested, and
integrated in factory

Broader range of capacities available in market to address greater capacity needs

Simpler adds, moves, and changes because cables are not stacked on top of each
other

Disadvantages

Ceiling height could constrain overhead implementation of this approach, as space is required above the
racks (approximately 25 inches or 0.6 m); raised floor used as air plenum constrains underfloor
implementation
Field installation of multiple bus sections and field-integrated management capabilities requires more time
and expense and can create unique problems
Can interfere with ducts and air containment systems when hung above IT enclosures Row placement
and length need to be thought through in advance for busway placement since it is intrusive to install or
move bus in a live IT environment
Oversizing of bus is more likely since it is commonly built out on day one

Usually used

When floor space in the IT room is constrained

In large facilities with an open floor plan with a very well defined IT row layout

When there is a high degree of confidence in final IT load requirement

When there is a high frequency of IT equipment turn-over requiring new circuits to be installed (i.e. lab
environment)

Floor-Mount Power Distribution- Instead of traditional circuit breaker panels with bolted wire
terminations, modular PDUs have a backplane into which pre-terminated circuit breaker modules are
installed. This arrangement eliminates on-site termination of wires. A new row of 24 IT enclosures, along
with all of the associated branch circuit wiring and rack outlet strips, can be installed in an hour, without
any wire cutting or terminations. Installation involves installing the sub-feed breakers, placing the
distribution units on the floor, running the wire from the sub-feed breakers to the units, inserting the preassembled circuits, and routing them to the racks. Power monitoring is included in each branch circuit
and automatically configures when plugged in. A large number of data center projects involve the
upgrade of an existing data center, with common projects being the addition of capacity or installation of

Universal Networking Services

Power Distribution Architectures

a high-density zone. The floor-mount modular distribution system is particularly well suited to these
types of retrofit projects, because installation is much less disruptive than installing a traditional PDU. As
the data center evolves, modular PDUs can operate alongside existing traditional PDUs. In smaller data
centers, the modular PDU may be directly integrated with the UPS system into a compact arrangement
that can be located in the IT room and integrated into an IT enclosure lineup. In this case, the need for a
separate UPS room is eliminated. In some cases there may be one or more zones within a data center
where only a small number of branch circuits are needed. This can occur when there is a cluster of very
high density racks, or when a small group of racks is isolated by room shape or other constraints. In
these cases, there are smaller versions of modular PDUs that directly mount into an IT rack, consuming
zero floor footprint

Advantages

Pre-fabricated backplane and branch circuit modules ensures a well integrated, reliable system

Quicker installation time due to roll-in form factor and pre-assembled modules which require minimal field
work

Integrated design with plug and play intelligence for better capacity and change management as load
requirements change

System-level warranty of whole power distribution system, since components are de- signed, tested, and
integrated in factory

Improved aesthetics, with IT rack form factor for lining up in a row of racks, closer to the load

Ease of scaling additional PDUs as new IT demand is identified

Accommodates unusual room constraints since PDUs can be rolled into any location on
IT floor

Disadvantages

Greater number of cables to inventory since distances from PDUs to IT racks vary

Cable tracing as well as adds and changes can be dicult, especially for high density
applications, due to volume and size of cables located in troughs

Takes up floor space in IT room

Usually used

When data center has uncertain growth plan and locations are not precisely defined in advance, requiring
flexibility to add/move distribution

Universal Networking Services

Power Distribution Architectures

When the room is constrained in shape or ceiling height or has obstructions

When IT personnel can re-configure the breakers and branch circuits without involvement of third parties
When speed of deployment is a high priority
When retrofitting an existing data center with additional capacity or installation of a high-density zone

Part 3 - Relative Cost of dierent distribution

The final power distribution (i.e. downstream of transformers to the IT racks) accounts for less than 5%
of the total capital cost of a data centers physical infrastructure. Although this represents a small
percentage of the total, its important to understand the dierences in system cost, installation cost, and
TCO for the five architectures in order to select the optimal approach for a particular need.

On a per watt basis, there is not a significant dierence in the installed cost; however, data centers arent
typically built upfront for a known final capacity, and when growth plans are factored in, the upfront cost
delta among the architectures is greater. Some approaches are highly scalable, which means significant
capital expense can be deferred to future years or avoided altogether. Others incur a significant expense
on day one, for an uncertain future capacity need. From an operating cost standpoint, cooling
ineciency can result when running power beneath a raised floor that is also being used as an air
plenum for distributing cool air to the IT equipment. These under-floor cables, as well as the floor
openings for the wire, interfere with the airflow resulting in an inecient cooling distribution system that
must be oversized to adequately cool the load.

About Universal Networking Services

Universal Networking Services is an industry-leading provider of power and cooling solutions for mission-critical IT
infrastructure. From planning to site commissioning, our engineered solutions are reliable, scalable and modular to
increase eciency and lower operational expenses. Our services include: design/build, product acquisition,
installation, management and maintenance services.

To learn more about Waite Ave please contact Waite at [email protected]

To learn more about Universal Networking Services visit www.criticalpowerandcooling.com

Universal Networking Services