Quantum leaps in server technology have enabled owners to increase power densities in mission-critical data centers in recent years — often in excess of 12kW per server rack or 375W per sq ft of floor space. At the same time, many owners of such facilities aim for high reliability and low power usage effectiveness (PUE), often well below the 1.92 average for data centers, according to the Environmental Protection Agency (EPA). Moreover, the EPA estimates that data centers use a significant amount of energy, accounting for 1.5% of total U.S. electricity consumption at a cost of $4.5 billion annually — an amount that’s expected to almost double over the next five years.
“Something’s gotta give,” one would think. In fact, it has never been more critical for engineers and contractors to design, install, commission, and maintain an electrical system that meets the owner’s mission for an efficient, high-reliability data center. Despite the challenging times, innovative use of strategies, such as using 415/240V and other medium-voltage power distribution systems, makes it possible to optimize reliability and efficiency. However, they must be applied at an appropriate site, integrated with an energy-efficient mechanical system, and vetted through a full life-cycle commissioning process.
Euro-distribution comes to the United States
For the most part, power is distributed to data centers in the United States at 480/277V and transformed down to 208/120V to feed the racks. In contrast, 415/240V is the European standard. All computer equipment is capable of operating at 240V. In fact, it operates more efficiently at this voltage. As a result, some of the leading U.S. companies are migrating to 415/240V distribution with a 415/240V uninterrupted power supply (UPS) operating at 60 Hz.
One drawback of this arrangement is that few vendors make UPS modules at this specification; therefore, the owner limits the field of bidders from the onset. However, the information technology (IT) load will function efficiently with no need for power distribution units (PDUs). In addition, the 3% loss associated with PDUs will also be eliminated. In the end, the overall building load will be lower, and — with an energy-efficient mechanical system — the PUE will be lower as well. The owner will also save money on construction costs, given the elimination of the PDUs from the power distribution system and the fact that, at 240V, the installation will use half the copper.
Pulling medium-voltage power inside
Allowing medium-voltage power to penetrate deeper into the facility is another underused method of gaining operating efficiency and reducing construction costs. Typically, the electric utility might supply power to the data center at 4,160V or 15kV, with the electric utility transformer reducing the voltage to 480/277V outside the building.
Some data centers are bringing these medium voltages inside to the main electric utility switchboard and beyond. UPS systems, generators, and chillers can all be purchased to operate at 4,160V. As a result, feeder sizes will be dramatically smaller than they would have been at 480V, saving considerable construction costs. The 4,160V power can be transformed to 480V and other lower distribution voltages just before entering the computer room.
The barrier to medium-voltage distribution at 15kV or 5,000V is a heightened concern over the potential for fatal arc-flash incidents. An arc flash poses risks to IT workers in data centers as they work with electrical panels in the PDUs that supply power to the racks. The arc flash risk must be addressed when discussing the pros and cons of this design option with owners.
For data centers requiring large capacity (above 10MW), owners can increase IT operating efficiency, lower PUE, and lower operating costs by taking electric utility service at a distribution/sub-transmission voltage of 35kV (or more) and transforming to 15kV or 5kV. Depending on the electric utility rate structure, the owner may save a lot of operating expense; however, not all electric utilities have the capability to provide customers with higher voltage service — and some require technical proof of need for such voltage classes.
Innovative use of DC UPS
Because of the loss of efficiency associated with transforming voltages, some engineers and owners have discussed the use of DC power distribution. Consider that in a typical electric utility system, the electric utility service will transform outside the building — twice with the UPS, again within the PDU, and yet again at the power supply to each server. DC power is theoretically attractive because there is a single transformation at the electric utility switchboard before feeding the data center. However, there are two major disadvantages on this front. First, most DC distribution is 48V, and the voltage drop over distance is severe at 48V. As a result, the cables are enormous — and so are the construction costs. Second, the voltage decay can be reduced if DC is run at 400V, but this approach increases the risk of fatal shocks to IT personnel, because 400V runs all the way to the racks. In comparison, a 4,160V distribution system terminates at the UPS substations.
Mechanical system: a keystone of PUE
At today’s typical power densities, data centers generate intense heat and require massive amounts of energy for their chilled-water cooling systems. For this reason, increasing the efficiency of the data center — in particular, lowering the PUE — has more to do with design of the mechanical system than it does with the electrical system.
The single biggest impact is made through the use of air- or water-side economizing, which enables owners and operators to turn off mechanical chillers several months of the year, depending on a facility’s location. However, the site must meet certain criteria to optimize the use of this energy-saving solution, such as a climate with moderate year-round temperatures and humidity levels and low air pollution.
Water-side economizing systems, which use fluid coolers outside the building to lower the temperature of recirculating water, also require an appropriate site to optimize energy savings. Although this technology is not as energy-efficient as air-side economizing, it can be used where low humidity and air pollution are barriers to air-side economizing.
It’s important to note two additional design issues associated with the mechanical system. First, it must be designed to the same level of redundancy as the electrical system to achieve the owner’s target level of reliability. Second, the mechanical system must be designed to reduce the most common source of harmonic distortion in the electrical system: variable frequency drives (VFD) on the pumps and — to a lesser extent — the chillers.
Mechanical systems are often designed with 6-pulse VFDs or specified by the mechanical contractor, because they are less costly than 24-pulse VFDs. However, this is not the place to save the owner money. The 24-pulse VFD generates a much smoother, cleaner wave, which is essential not only for reliable operation of the IT equipment, but also for meeting the standard for reflected distortion back into the power grid.
Full life-cycle commissioning
Independent third-party, full life-cycle commissioning is a significant contributor to the successful performance of the data center. This systematic process assures that building systems individually and collectively perform according to the design intent and the owner’s operational requirements.
In any project, especially a mission-critical data center, these goals are best achieved by beginning in the design phase and continuing through construction, acceptance, and the warranty period with actual verification of performance.
Design and engineering peer reviews conducted during the design phase of the project include the owner’s PUE goal, with a careful review of the design approach and calculations. This is the ideal time to identify design and equipment-specification issues that may adversely affect PUE — not at start-up, when the commissioning agent will give the owner the actual test data that confirm, or refute, the design PUE.
Testing challenges at medium voltage
While the commissioning process is the same whether or not the electrical system design is low voltage (480V), medium voltage (4,160V or 15kV), or a combination (480VAC with 400VDC or 600VDC UPS), there are greater challenges and costs associated with testing higher voltage and combination AC/DC systems.
When commissioning a data center, rented load banks rather than the actual server racks provide the load. Using load banks at 480V, the system is loaded to capacity, and the backs of the electrical gear are removed for infrared scanning and physical testing. The same procedure does not apply to medium-voltage gear, which is always tested with the equipment cubicles closed and panels in place. In addition, medium-voltage gear typically has inspection ports covered by Plexiglas to reduce the impact of an equipment failure and enhance safety; therefore, the commissioning agent must use different testing protocols, eliminating some tests altogether in the absence of an inspection port.
Commissioning costs increase with the need to rent an additional set of medium-voltage load banks for functional testing of medium-voltage systems, as well as low-voltage load banks for performing the integrated testing prior to full acceptance. It is not uncommon to rent load banks at 208/120V, 480/277V, 4160V or 15kV and 48V for DC systems, if these are part of the design — and to keep them on-site until the project is completed. At costs ranging from $1,500 to $30,000 per week per load bank, efficient scheduling is essential to managing commissioning costs on behalf of the owner.
The introduction of medium voltage into data centers has also required commissioning agents with the capability to provide operations and maintenance manuals and training of in-house staff and contractors to safely conduct maintenance and operations of medium-voltage equipment and switchgear. While reliability assurance testing is advisable in a standard 480V design following annual maintenance, it is essential before putting a medium-voltage system back into service.
Ultimately, full life-cycle commissioning benefits the owner as well as every member of the project team. The process provides a return on the owner’s investment with improved system performance, O&M, and energy efficiency over the life of the data center. The process benefits engineers and contractors by identifying correctable design and construction issues earlier — in some cases, before construction begins, contributing to meeting cost estimates and the construction schedule while reducing change-orders.
With the increasingly competitive environment for data centers, owners, engineers, and contractors can no longer afford to lean on old ways of thinking, old designs, and old specifications simply because they worked well in the past. The reality of increasing power density, increasing electricity consumption and associated costs, and increasing demand for high reliability and high efficiency have all created challenges that can only be met with creative thinking and approaches to design, installation, commissioning, and maintenance.
Gillick is president of Primary Integration Solutions, Inc., Charlotte, N.C. He can be reached at: [email protected].
