ASME Paper Documents Reliable Data Center Operation With Outside Air and Evaporative Cooling

One of the longest running and most debated topics regarding data center operation is whether or not you can reliably cool a modern data center using outside air alone or with supplemental evaporative cooling.  If you can successfully operate a data center without using any compressorized equipment there are obviously huge energy and money savings.

The two key factors that hold operators back from implementing an obvious saving strategy is fear of equipment failures due to temperature/humidity excursions and due to airborne contaminants.  While server manufacturers publish data in their specification sheets that clearly indicate that their equipment can tolerate a wide range of temperature and humidity there is not much information regarding the impact of particulates and other contaminants.  ASHRAE has recognized the robustness of modern IT equipment by expanding the recommended and allowable temperature and humidity ranges in their widely followed data center design guidelines.  Very little is said regarding air quality other than a recommendation to use at least a MERV 8 filter system.

Over the last 5 years the Mestex division of Mestek has hosted a National Science Foundation research site at their manufacturing facility in Dallas.  This site is part of an Industry/University Cooperative Research Center with principal research from the Mechanical and Aerospace Engineering Department at the University of Texas at Arlington.  A fully instrumented "data pod" has been operating using a commercially available indirect/direct evaporative cooling system from Mestex that can also operate in 100% fresh air mode.  In addition to the dozen sensors normally included with the Aztec brand IDEC system from Mestex the "pod" includes an array of 64 sensors located on the front and rear of the four server racks.  Data has been streamed from all sensors every 15 seconds for the last 4 years.  In addition to this detailed tracking of temperature and humidity conditions there have been a number of studies conducted using copper and silver coupons to evaluate the corrosion potential of operating using outside air and evaporative cooling.  Keep in mind that this application is in Dallas, Texas...a relatively hot/humid climate area.  In addition, because the "pod" is installed between two manufacturing buildings in an industrial zone near downtown Dallas the measured air quality around the "pod" is classified as G2, or moderately harmful to PCBs.

The June, 2017 Volume 139 edition of the ASME Journal of Electronic Packaging includes a paper presenting the results of the last 4 years of the research at this site.  The paper entitled "Qualitative Study of Cumulative Corrosion Damage of Information Technology Equipment in a Data Center Utilizing Air-Side Economizer Operating in Recommended and Expanded ASHRAE Envelope" provides a comprehensive look at the impact of operating a data center in a "real world" application.

The most interesting point presented in the summary section of the paper is that, in spite of the servers installed in this test site already being several years old, there has not been a single server failure in the entire four years of operation.  The ability to dramatically reduce the cost of operating a data center...without unfounded concerns about reliability...is finally being proven true.

Why Do We Design Thermos Bottles?

Over the last couple of months since my last posting I have been very busy managing our movement into new markets and grasping at new opportunities.  One of the benefits of taking the deep dive into these markets is getting to look at some of the details of product design and application to the specific problem to be solved. 

This has raised a question in my mind.

Why does the mission critical industry design "thermos bottles" and then fret over the cost of and methods of getting rid of the heat that all those servers generate? 



There is something that strikes me as illogical about creating buildings or modular data centers with super insulated walls and ceilings that are guaranteed to trap the heat that is dumped into the hot aisle (assuming they have aisle separation).  Then the mechanical system is tasked with rejecting all of the pent up energy without costing the owner a fortune.  Is it any wonder that data centers are one of the largest consumers of electrical energy in the world?

Centuries ago architects and designers figured out that it is more efficient to cool a space if you simply dump the heat out to the atmosphere.  Buildings used to be designed to take advantage of stratification and stack effect to cause the hot air generated in the space to rise and leave the building.  No need to cool the air back down to a reasonable temperature and put it back into the space so that you can heat it all up again.  Lofted ceilings and roof lines came into the design world for a reason. 

So, why is the data center different?  Frankly, I don't know.  Why not take the hot aisle air and vent it out to the atmosphere?  Sure, you have to replace that exhausted air with new air from the outside but unless the data center is located in Death Valley the odds are that the air being brought into the building is at a lower temperature than the air that would be recycled from the hot aisle of a data center designed to operate under the latest ASHRAE TC 9.9 guidelines for best practices. 

My best guess why we continue to do what is intuitively illogical is inertia.  "We have always done it that way".  I think it is time to rethink the old ways and come up with creative solutions in the design of data centers.

Indirect Evaporative Cooling Research Project Launched

Aztec ASC 3-D Model for CFD Research

ASME Paper Documents CFD Modeling of Aztec IDEC System

The 2013 ASME “International Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems”, aka ”InterPACK 2013” is just concluding in San Francisco.  As the conference title implies there are papers and presentations from all over the planet that are focused on research into improving electronics, computers, and data centers.

One of those papers presents results from an on-going research project that Mestex has started with the College of Engineering at the University of Texas at Arlington.  This research project will likely go on for a couple of years and this paper presents some of the first findings that are being used to establish a “baseline” for the rest of the research.

The paper is listed in the proceedings as “InterPACK2013-73302”.  The human-readable title is “CFD MODELING OF INDIRECT/DIRECT EVAPORATIVE COOLING UNIT FOR MODULAR DATA CENTER APPLICATIONS” and the paper covers exactly what the title suggests.  The IDEC product that the paper covers is the Aztec ASC-20 and the goal is to establish that the factory data that we present in our literature can be validated against a detailed CFD model of the product.

By modeling the Aztec ASC-20 components and creating the 3-D CFD model using factory dimensional drawings the researcher was able to confirm that the published factory data is accurate and the ASC-20 will perform as predicted based upon the operating parameters.  This important result can be used to further our research into optimizing the performance of evaporative cooling and fresh air cooling solutions for mission critical/data center applications.  A full scale modular data center mock up is being installed at the Mestex facility and additional documentation and validation of the performance will be conducted over the next several months.  The CFD baseline model will be used to simulate filter performance and airflow changes prior to making the physical changes to the research module.

The Aztec, and Alton DEC, evaporative cooling products have been used in the industrial and large commercial market since 1946.  Over those 66 years the products have been refined and optimized.  This research project will take the product lines to another level of thermal performance, water use optimization, and control software optimization with a specific emphasis on the needs of the mission critical market.

Dusting Off Your Data

CONTAMINANTS IN THE DATA CENTER

Time to get back on my soapbox again…this time it is about “contaminants” in data centers as an excuse to avoid using fresh air cooling or having outside air enter the white space.  The bottom line is that unless your data center is located in “an emerging country” then the odds of a contaminant-created hardware failure in anything like a short time frame are about the same as winning the lottery…assuming you take some pretty basic steps in the design.

Contaminant control, or more correctly, concern over contaminant control has been around for decades.  I remember doing some research over 25 years ago on the impact of ozone on telecommunications equipment.  Bell Labs, as it was known long ago, had performed some pretty interesting tests to document what could be a very real problem under the right circumstances.  The results of those tests indicated that, with the exception of certain locations, the air in the equipment room was worse than the air outside so it made more sense to flush the room with outside air than to avoid bringing outside air into the space.

Particle and gaseous contaminants CAN be a problem if ignored.  However, the extent of the problem and how quickly it manifests itself needs to be considered. 

Phenomenon like copper creep and circuit bridging do occur…but only when the conditions at the server are right to support those failure modes.  Two things generally need to be in place for the failure mode to even begin.  First, there needs to be a fine coating of dust particles on the circuit boards.  Second, the relative humidity at the board needs to be at the deliquescent RH…or the point where the dust starts to absorb moisture and become “wet”.  If the RH is too low then dust might affect localized temperatures on the board but the mechanism to cause bridging simply does not exist.  The converse is also true…no dust…then no mechanism even with a relatively high humidity level.

Dust can come from anywhere.  Every time someone enters the data center they bring in some amount of dust particles.  Every time a box is opened in the data center particles are created.  And, yes, every time outside air is brought into the data center it is possible that dust can enter.  In fact, a data center with no outside air is actually vulnerable to the worst kind of dust intrusion…uncontrolled infiltration through doors, cracks, pipe openings, or wind pressure.  Maintaining a positive pressure in the white space helps to prevent infiltration and keeps the worst dust (and gases) out of the data center. 

ASHRAE, through the TC 9.9 committee, has set a target for data center “cleanliness”.  It is ISO Class 8.  ASHRAE has also noted that ISO Class 8 conditions can be met with a MERV 8 filter…a common and inexpensive filter available at virtually any HVAC parts house.  If the air being filtered is coming from the outside then ASHRAE recommends a MERV 11 or MERV 13 filter.  These might not be quite as common as the MERV 8 but they are also readily available and can fit in a standard 2” filter rack.

The interesting side note about the ASHRAE recommendations is just how extremely conservative they are.  ASHRAE recommends no more than 15mg/m³ of “fine” particles…defined as particles less than 2.5mm in size.  However, IBM (who should know something about computers) has a limit of 150mg/m³ and a “fine” particle definition of particles less than 5mm in size.

Once again, owners are being led down a path to purchase cooling systems and equipment that fail to optimize their energy savings through an inflated fear of something that happens very rarely in the developed world and is easily controlled with proper filtration.  Products such as our Aztec ASC indirect evaporative cooling systems are designed with MERV 14 filters in mind and can actually accept MERV 16 filters…the highest MERV rating point…that is suitable for operating rooms and can remove all bacteria and most tobacco smoke.  This allows the Aztec system to optimize the use of fresh air cooling and use a more efficient heat transfer system than air-to-air heat exchanger systems…and still exceed the extremely conservative ASHRAE recommendations for particulate control.

How I Spent My Summer Vacation

It has been a while since I have posted anything to this blog...no, I was not on sabbatical on some desert island...I have been traveling around North America talking to consulting engineers, contractors, and data center owners and operators.  This posting just provides a few insights that I garnered over the last 2 months on the road.

First, the data center/mission critical market continues to occupy the minds and the design resources of many, many companies in the design community.  It is clear that this is a market segment that is vibrant and all indications are that it will continue to be for quite some time to come.  The latest issue of Datacenter Dynamics FOCUS indicated that the world is now consuming over 300 Tkwh annually to drive data centers, with the US consuming over 25 Tkwh alone.  The consumption in the US is projected to grow over 9% in 2013.  While this information points to a growing market it also points to the urgent need for improved operating efficiency in data centers.

Second, and related to the first item, is the lack of knowledge about new "best practices" in data center design.  I have talked to dozens of engineers, contractors, and data center people who are not aware of the latest design guidelines from ASHRAE.  In fairness, those guidelines were only officially announced a few weeks ago...but they have been rumored and discussed for about a year now.  I mentioned in one of my earlier posts that education of the design community is an important, and ongoing, task.  This has been reinforced to me over the last 2 months.

Third, for those engineers and contractors who understand and embrace the new standards, is the challenge of convincing the data center people to adopt those standards.  This is less of a problem at the top levels of the data center company than it is on the floor of the data center.  The IT equipment operators who live in "the white space" seem not to understand the allowable operating temperatures of the equipment that they manage every day.  I have heard many different reasons for their reluctance to adopt the new best practices but I think it comes down to fear.  Because of stringent SLAs the operators worry about losing any equipment for any period of time...even though there is mounting research that this fear is unfounded.

Fourth, I have heard of several cases where the local electric utility has started to put limits on the available service capacity for planned centers.  In the US we are so comfortable with the idea that our electric grid can provide unlimited power that we forget that is not true.  We have a fixed number of powerplants with only so much generating capacity.  With the tremendous growth of data centers, and data centers with 300 to 500 watt per square foot electrical demands, there is a limit to what a utility can do.  And timing is another element of the equation.  A data center can be built in a matter of months...a powerplant takes years.  So even when a utility sees the demand coming they cannot add capacity as quickly as the demand can be added.

So, these are a few observations from the last couple of months.  Of course there is more to the story and feel free to comment on this post with any questions you might have.  I will try to respond as quickly as possible.

How to Save Almost $100,000 Per Year In Your 1 Megawatt Data Center

Over the last few weeks while I have been traveling there have been some interesting bits of information released in the mission critical world.

For example, Dell introduced their 12^th generation PowerEdge servers.  This generation of servers is warranted to handle temperature excursions up to 45 degrees C, or 113 degrees F, for up to 90 hours per year.  One of Dell’s rationales behind marketing the server at those conditions was to allow fresh air cooling in virtually the entire continental US.  Other research by Dell has indicated that their servers can operate 87% of the year in Washington, DC using fresh air cooling alone.

The energy saving potential of raising the inlet temperatures that high can be enormous.  Instead of running chillers or compressors 8,760 hours a year they are only operating 1,138 hours per year. 

To put that into numbers is difficult but let’s try a little example.

If the PowerEdge server power consumption is 300 watts then the cooling system must remove 300 watts times 8,760 hours per year or 2,628 kwh of heat (8,961,480 btu).  That can either be accomplished using mechanical cooling or fresh air cooling or a combination of the two.

A pretty efficient HVAC system will remove about 4.5 watts of heat per watt of electrical energy used.  So to cool that PowerEdge server using mechanical cooling will require 2,628,000 watts of heat divided by 4.5, or 584 kwh of compressor power.

To cool that same server using fresh air for 87% of the year will only require 75.8 kwh of compressor power.  Of course, the fan energy stays the same in both cases but the compressor savings of 508.2 kwh PER SERVER can really start to add up.  At an aggressive electric rate of 4.5 cents/kwh that amounts to $22.87 PER SERVER PER YEAR.

At modest densities of, say, 40 servers per rack the savings amounts to $915 PER RACK PER YEAR.  Now consider how many racks are in the typical server room or data center.  If the data center has a server load of 1 megawatt then a density of forty, 300 watt, servers per rack will translate into 83 racks.  So the annual savings would be almost $76,000 in this example.

To make the savings even greater the same HVAC unit that provides the fresh air could also provide indirect evaporative cooling and completely eliminate the compressor-based cooling…adding another $3.50 PER SERVER PER YEAR of savings.  That would add another $11,620 PER YEAR in savings.

Preaching to the Choir

Electrical Power Meters Keep Spinning

I have had a busy few weeks traveling to meetings and visiting with owners, operators, engineers, and researchers.  This has given me an interesting perspective and awareness of an issue that our industry needs to address.  My awareness of this issue was increased by an editorial in Mission Critical Magazine that bemoaned the lack of progress in data center design due to secrecy regarding "best practices".

I came away from all of those meetings with the sense that there are many very smart people who know how to design more efficient solutions to energy use in mission critical applications.  "Best practices" can be described by experts from the largest server manufacturers, global data center developers/operators, and from academia.  The issue is that we are all sitting around a large table in a closed meeting room and sharing that knowledge with others who already have a pretty good idea what to do.  We are "preaching to the choir".

The result is that the vast majority of data centers, server rooms, and telecom facilities are operating in very inefficient ways.  While a Microsoft might be able to design a data center with a 1.2 PUE the rest of the world is struggling to reach a 2.0. 

This came out in a technical committee meeting at ASHRAE's mid-year meeting a few days ago.  A comment was made by a server cooling system manufacturer that he finds it very difficult to convince smaller users to adopt the latest operating standards that could save the user tens of thousand of dollars a year in energy costs.  This sentiment was echoed by several around the room and pointed to how difficult it has been to educate the broader public on the reliability of modern equipment in warmer rooms.

And when I say "broader public" I mean just that.  The mechanical design director for a global retail data center operator told me that he knows his equipment will run just fine at 78 or 80 degree F inlet temperatures but his customers have not gotten the message and demand a "cold" room.  It seems that until corporate IT managers and executives understand all of this we will continue to see skyrocketing energy use by data centers.  Even small server rooms could benefit from elevated temperatures if key elements of "best practices" were implemented.  So called "legacy" data centers might be difficult to retrofit but they can certainly be upgraded with the basic elements of "best practices"...if only the occupants understood what is possible.

The industry has a massive educational challenge if it is to stem the rising cost and consumption of energy.  And the education cannot come soon enough because the projections are that server power densities will continue to climb and data storage power densities will climb even faster.  Today we talk about 300 watt per square foot densities but systems are being designed already that push almost 10 times that density.  It may seem that we have an endless supply of power from the grid but there are only so many power plants around the world and building a new one takes a decade or longer...data power consumption grows at a much faster rate and will stress grids around the world eventually if we cannot educate the "broader public" more effectively.

Direct Evaporative Cooling Analysis for Two Diverse Climates

One of the common concerns expressed about the use of evaporative cooling for data centers, server rooms, telecom facilities, or other facilities housing heat producing electronics is the ability of evaporative cooling to achieve the target inlet conditions for the electronics.

These two psychrometric charts show the results of an actual analysis in two distinctly different climates.  The target server inlet temperatures were between 65 and 85 degrees F and between 20% and 80% RH.  The mechanical system criteria mandated that direct evaporative cooling be used.

The proposed mechanical system consisted of a direct evaporative cooling system with 12" cellulose media, a steam humidifier, DDC controls, and a hot aisle/outside air mixing section.  The controls would be configured to modulate the outside air dampers, hot aisle dampers, evap media water flow, and the humidifier to maintain the target conditions.

Server Inlet Temperatures from Evaporative Cooling System in Pacific Northwest US

Server Inlet Temperatures from Evaporative Cooling System in Southeastern US

As you can see from the charts the proposed system would easily achieve the desired results.  In fact, it was found that outside air cooling could achieve the targets during roughly 15% of the year, direct evaporative cooling could achieve the results during roughly 60% of the year, and remaining hours of the year when the air was too cold to properly operate the evaporative cooling without fear of freezing a combination of hot aisle and outside air supplemented with the humidifier would hit the target.  The only excursions of temperature over the maximum target of 85 degrees would occur for no more than 5 hours a year based on the NOAA TMY2 weather history.

While not every location would achieve these results the diversity of climate for these two studies imply that direct evaporative cooling, with creative use of mixing and controls, will work in many more climates than most people expect.  To further expand the capabilities of evaporative cooling an indirect evaporative cooling element could be provided to produce even broader temperature control.  Although it was not analyzed for this case we would expect that adding an indirect evaporative cooling element to these systems would have eliminated the few hours of excursion above the maximum dry bulb temperature.

As a means to dramatically reduce energy consumption for these data modules the evaporative cooling solutions such as those manufactured by the Aztec division of Mestek would prove to be extremely effective.  Since evaporative cooling systems are also relatively simple mechanical systems with no refrigerants maintenance of the systems does not require licensed refrigeration technicians and there are very few elements that could fail...increasing uptime and providing much faster recovery time should a repair be necessary.

Aztec Evaporative Cooling Solutions and the University of Texas at Arlington

On April 12, 2012 management and engineering representatives of Aztec, part of Mestex division of Mestek family of products, met with engineering representatives from the University of Texas at Arlington to define how resources will be jointly applied to researching and developing an advanced technology indirect evaporative cooling solution for data centers. Research already started at both organizations will be shared in order to more quickly advance the development of a viable solution to the high rate of energy and water consumption by data centers.

The joint project will be part of the NSF-I/UCRC program. This program, initiated by the National Science Foundation, is a collaboration between 5 major universities and a select group of industry contributors. The stated purpose of the program is to develop commercially viable solutions that will improve energy efficiency in data centers. Research projects range from chip level solutions all the way to complete, large scale, data center solutions. Aztec will be contributing special knowledge of evaporative cooling and outside air cooling solutions that has been developed over 40 years of product development and marketing.

Aztec Evaporative Cooling Solutions attends Data Center World Expo

Aztec Evaporative Cooling Solutions, a division of Mestek produced in the Mestex (Dallas) facility, was present at the recent Data Center World Expo in Las Vegas.  Aztec was showing an example of the ASC indirect evaporative cooling solution for data centers, server rooms, telecomm facilities, or IT product research labs.  The unit on display highlighted the system's integrated cooling tower technology, integrated DDC control system with multiple sensor options and BacNet or IP access, isolated direct drive plenum fan assembly designed for up to 200,000 hours of operation, and variable frequency drives for both cooling tower and supply fans. 
 
The show was attended by IT facility professionals from all over north America and Mexico.  Roughly 80 vendors were displaying an array of infrastructure products for data centers ranging from power distribution systems to cooling equipment.  Also visible in many booths were thermal and power simulation tools and DCIM software. 

The Aztec ASC system was the only factory assembled and tested evaporative cooling option for data centers that was on display.  Designed for long life, and with a history of extremely low failure rates, the Aztec ASC was considered to be a viable solution for many of the attendees.

Green Grid Updates Free Cooling Maps for Data Centers

The Green Grid has released White Paper #46 as an update to their "free cooling" maps for data center design and operation.  The research was edited by Emerson Network Power, Intel, and Schneider Electric. 

The reason for this update to the "free cooling" maps was the latest changes to the ASHRAE TC 9.9 operating/design guidelines for data centers.  For those who have not yet seen those new guidelines they allow a much larger operating range for data centers and server rooms that use some of the latest equipment from companies like Dell and HP.

For those of us who are "metric challenged" 40 degrees C = 104 degrees F and 35 degrees C = 95 degrees F.

When you consider that many data center operators still seem to want their rooms at 70 degrees or lower it is clear that these new criteria are a massive change in operation and design concepts.  It is also clear that adopting the newest guidelines can result in enormous energy savings.

The Green Grid paper includes a couple of maps to quickly illustrate how extensive the potential for "free cooling" has become under the latest operating/design guidelines. In these maps the darker the blue color the more hours that "free cooling" could be employed.  The darkest color blue indicates that all 8760 hours are suitable for "free cooling".  The maps also consider the coincident dewpoint temperatures as that metric is important also.

This first map is for ASHRAE Class A3 environments and shows that virtually all of North America could have their data centers cooled without using chillers or compressors.  The second map is for ASHRAE Class A2 environments and shows that roughly 80% of North America could still be cooled most of the year with no chillers or compressors.

The question for data center operators and designers who want to implement these new temperatures is what to do about those 500 or 1,000 hours when the outside air conditions are not quite right.

It is still quite possible to operate the center with no compressors or chillers if the designer will incorporate an evaporative cooling system such as the Aztec indirect evaporative cooling system or even the Alton direct evaporative cooling system.

Since evaporative cooling systems operate using 100% outside air all the time they make an excellent "hybrid" approach.  During the many hours of the year when "free cooling" will satisfy the conditions either type of evaporative cooling system will provide cool, filtered, outside air.  The Aztec indirect evaporative cooling system has the added advantage of allowing recirculation of hot aisle air during the very coldest months when "free cooling" could actually over-cool the data center.

During those few hours of the year, however, when it is simply too warm for "free cooling" to work, the Aztec or Alton systems can automatically initiate their evaporative cooling cycles and trim the outside air temperatures down to levels that fall well within the new ASHRAE guidelines...again, with no compressor or chiller energy required.  The air leaving the evaporative cooling system will usually be about 3 degrees F higher than the wet bulb temperature.  This chart should give you an idea of the potential air temperature that an evaporative cooling system can provide.

The Green Grid whitepaper is just the latest in a growing number of research papers and documents that point operators and designers in a direction that can save tens of thousands of dollars and kwh if they are willing to make the investment in the latest technologies from both the IT equipment manufacturers and the HVAC equipment manufacturers.

Recently there was an interesting article published in Mission Critical magazine that addressed cooling in data centers.  More specifically the article addressed the waste that is currently happening in many, many data centers by operating the center at too low a temperature.

ASHRAE TC 9.9, at the urging of IT equipment manufacturers, has been raising the recommended and allowable temperature and humidity ranges for all types of IT equipment.  There are now certain classes of equipment that have allowable operating temperatures of 113 degrees F and 80% RH...but we still see data center designs that call for 60 to 70 degree air entering the servers.  Even the most critical classes allow temperatures of 80 degrees F and 60% RH.

One of the reasons that is often presented for operating the center at such low temperatures is reliability.  There is now research that suggests that this is not a valid concern.

The article in Mission Critical magazine, authored by Mark Monroe, cites a few interesting bits of information. 

Using the Arrhenius model  for predicting MTBF (mean time between failure), raising the server inlet temperature from 77 degrees F to 104 degrees F reduced the MTBF from 15 years to 13 years...both probably well beyond the replacement cycle for the servers.  Given that prediction why run servers at even 80 degrees?

A second study by E. Pinherio, W.D. Weber, and L. A. Barroso’s (2007), “Failure Trends in a Large Disk Drive Population,” determined that there was no discernible relationship between disk drive failures and operating temperature.

Finally, Intel provided information to ASHRAE that allowed creation of a reliability factor calculation that was time and temperature based.  This "X-Factor" could be used to estimate changes in reliability from a baseline temperature of 68 degrees F.  The interesting thing that comes from this is that using an indirect evaporative cooling system, such as the Aztec ASC product line that can provide server inlet temperatures lower than 68ºF for the vast majority of the year, could actually increase reliability according to the algorithm.

The potential operating cost savings are huge.  Switching to the Aztec system that offers 100% outside air cooling most of the year, with supplemental evaporative cooling during the extreme highs, could save $67,000/1,000 kw of IT load for the average data center in the US...according to the information in the article.

Prineville Server Farm with "free cooling"

Data Center Dynamics is an international organization with a single mission of sharing best practices among data center designers and operators around the world.  The organization publishes a trade magazine called "Focus" and they have just released their January, 2012 edition.  This edition is a retrospective look at 2011.

One of the articles included comments from some of the industry's leading players in response to two questions:  "What was the most important data center development of 2011?" and "What single advancement will most positively impact the data center sector in 2012?"

Some of the responses were:

Bill Kosik; Principal data center energy technologist, HP Enterprise Business Technology Services:

"For the first time in 2011, many of our clients wanted to implement a design temperature of 75 degrees F for the inlet air to the IT equipment."  "When you couple increased supply air temperatures with ultra-efficient air-conditioning equipment (indirect evaporative cooling as an example), you start to see PUEs drop into the low 1.2s/upper 1.1s..."

Andrew Donoghue; Analyst, The 451 Group:

"ASHRAE released a white paper....redefined and reclassified new allowable ranges up to 113 degrees F.  Higher operating temperatures could mean that new facilities can be built without the need for expensive cooling technology, such as mechanical chillers."

Dileep Bhandarkar; distinguished engineer, Global Foundation Services, Microsoft:

"Broad recognition across the industry that free air cooling technology is now considered mainstream."

Jim Hearnden; Product technologist, data center power and cooling, Dell Services:

"Newer technology will permit higher server intake temperatures, which will be a great step forward in 2012."

The common thread through all of the comments is the drive to lower energy costs by raising server inlet temperatures.  Most of the more advanced companies are even going to the point of using 100% outside air with no tempering at all.  Aztec indirect evaporative cooling systems from Mestex, a division of Mestek, offer an alternative that filters and cools the air down to within 2 degrees of the wet bulb temperature (usually in the 70 to 80 degree range).  This allows the designer and operator to have acceptable server inlet temperatures and still have a very low PUE.  For installations that still need some degree of control over the air temperature and desire filtered, clean, air this might be the best solution.

Ten Reasons to Tone Down on Climate Control

Sometimes the best thing to do is to acknowledge when someone else does something right.  In this case Nicholas Greene posted an article on TechAxcess that sums up the ten reasons why data center designers and operators should adopt modern cooling design criteria for their centers.  Nicholas clearly articulates the reasons and I cannot improve on what he says.

I can only reinforce the message that there are alternative cooling methods that can provide cool...not cold...filtered, clean, air for data centers.  Aztec Indirect Evaporative Cooling Systems from the Mestex division of Mestek can send 65 to 80 degree air to the cold aisle and allow the designer to exhaust 100% of the heat from the hot aisle without using any refrigerants or compressors.  The result is cold aisle conditions that meet the latest ASHRAE TC 9.9 criteria and that address the issues that Nicholas covered in his article.

As Nicholas says...it is time for operators and designers outside of the "big names" to get on board and start implementing these energy saving technologies.

Saving Energy in Data Center Design (Video)

One of the most effective ways to improve the efficiency of a data center is by providing a cooling solution that uses a fraction of the energy of the conventional approach. Major players like Yahoo, Microsoft, Google, and Facebook have already adopted this technology for their data centers. I hope this 12 minute video provides some food for thought.

Green Data Center Design Conference Notes

Having just attended the Green Data Center Design Conference in San Francisco I came away with a few observations.

First, although this was a relatively sparsely attended event there were some "heavy hitters" present. Microsoft and eBay both had presentations and representatives in the house, as well as Digital Realty Trust, APC, and Intel. Other members of the audience seemed to include users and providers in the "middle tier" data center class.

It was this "middle tier" that received some attention from APC during their presentation. It was noted that this class of user with facilities in the 2,500 to 10,000 square foot range are not adopting "best practices" as demonstrated by the Microsoft's and Google's of the world. Some of this might be attributed to lower capital budgets but more likely to the slow spread of information about what can be accomplished. PUE numbers in the mid-2s to over 3 can be found in these facilities while large leading edge companies are operating in the mid- to low- 1s.

The term "PUE" itself came under some scrutiny as the right metric for measuring data center performance going forward. Part of this is the thought that other factors ought to be considered such as water usage (the "WUE") but the single biggest reason PUE is being questioned is the massive under-utilization of servers in virtually all data applications. Most of the experts speaking at this conference put utilization factors in the range of 10% to 15%. PUE is calculated based on total installed server capacity...not actual capacity in use. This not only results in PUE numbers that are grossly overstated but also in tremendous wasted capital dollars and far oversized infrastructures such as HVAC.

As for future trends in data centers and large server rooms the consensus also seems to be that "virtualization" will become more commonplace as a means to drive up utilization. "Cloud" services will have the same impact both for public clouds and private, corporate, clouds. This will result in reduced demand for more space for many companies as they begin to use what they have more effectively. Although one presenter suggested that this will cause a dip in construction demand the others seemed to believe that the rapid growth of data consumption will simply outstrip the rate of virtualization and still create demand.

In the cooling/HVAC arena the other trend that was presented by the major players was the rush to "chiller-less" design. Whether this was pure 100% outside air solutions or evaporative cooling solutions the key was to eliminate the chillers, DX equipment, and all of their ancillary equipment costs and complications. Data center specialists from eBay and Microsoft repeatedly pointed out that modern servers are designed to withstand operating temperatures approaching 230 degrees C. The ASHRAE TC 9.9 guidelines just introduced proposed server inlet temperatures of over 110 degrees F for some applications and the most stringent conditions were still around 81 degrees F. eBay commented that their failure rate of components in chiller-less designs is barely greater than in their legacy designs and was more often caused by vibration or software issues than temperature issues. Humidity control was also questioned and an engineering manager for Digital Realty Trust commented that he has seen no problems with relative humidity levels down around 20%. There seemed to be no upper limit that concerned the DRT engineers and they were strongly proposing evaporative ("adiabatic") cooling for their future data centers. Using a system such as the Aztec indirect evaporative cooling system would easily produce temperatures in the ASHRAE guideline range and do so without introducing more moisture into the data center environment.