Mestex Data Center Research Published

One of the initiatives within Mestex is our collaboration with the University of Texas at Arlington ("UTA") and the National Science Foundation ("NSF") Industry University Cooperative Research Centers ("I/UCRC").  Our area of particular interest and research support is related to reducing the energy consumption of data centers via the use of outside air and evaporative (adiabatic) cooling solutions.  Most of this research stays hidden behind closed doors but when the research results in findings that can significantly aid the mission critical industry those results are published.

Over the last several years, Mestex has hosted a small data pod at our facility in Dallas, Texas.  That data pod has been cooled by an "off the shelf" Mestex product with the only variations from the standard catalog item being the ability to add special filters (one of the research areas regarding particulates) and the control software designed by Mestex specifically for data centers.  We recently received the following email from Dr. Dereje Agonafer, Presidential Distinguished Professor at the University of Texas at Arlington.


Congratulations Dr. Shah and Dr. Awe and the Mestex team on getting this paper published in a journal.



We continue to be proud of our working relationship with Mestex – resulted in significant research implemented in product applications as well as archival publications. In 2016, our joint work was featured in: “Breakthroughs from NSF I/UCRCs appeared on the 6th edition of the Compendium, published in a printed book and online, and is intended for Congressional and White House staffers and visitors to the NSF, and for members of the general public to help them realize the impacts of research taking place within I/UCRCs. Our joint work with Mestex was featured in the book – reference below:



2016 Compendium, Successful Industry-Nominated Technological Breakthroughs for NSF I/UCRCs in “More Efficient Data Centers: Maximizing Airside Cooling,” p. 111-112, http://faculty.washington.edu/scottcs/NSF/2016/NSF-book-2016-Final.pdf

The research documented in this paper helps to show data center industry designers and operators the benefits of Airside Cooling for their centers.  Further research findings were just published in the ASME Journal of Electronic Packaging.  That research, a collaboration between Mestex, UTA, and IBM focused on the reliability of cooling data center electronic equipment with outside air in a somewhat dirty environment.  On going research between Mestex and UTA is now focusing on filter performance.

ASME Paper Documents Reliable Data Center Operation With Outside Air and Evaporative Cooling

One of the longest running and most debated topics regarding data center operation is whether or not you can reliably cool a modern data center using outside air alone or with supplemental evaporative cooling.  If you can successfully operate a data center without using any compressorized equipment there are obviously huge energy and money savings.

The two key factors that hold operators back from implementing an obvious saving strategy is fear of equipment failures due to temperature/humidity excursions and due to airborne contaminants.  While server manufacturers publish data in their specification sheets that clearly indicate that their equipment can tolerate a wide range of temperature and humidity there is not much information regarding the impact of particulates and other contaminants.  ASHRAE has recognized the robustness of modern IT equipment by expanding the recommended and allowable temperature and humidity ranges in their widely followed data center design guidelines.  Very little is said regarding air quality other than a recommendation to use at least a MERV 8 filter system.

Over the last 5 years the Mestex division of Mestek has hosted a National Science Foundation research site at their manufacturing facility in Dallas.  This site is part of an Industry/University Cooperative Research Center with principal research from the Mechanical and Aerospace Engineering Department at the University of Texas at Arlington.  A fully instrumented "data pod" has been operating using a commercially available indirect/direct evaporative cooling system from Mestex that can also operate in 100% fresh air mode.  In addition to the dozen sensors normally included with the Aztec brand IDEC system from Mestex the "pod" includes an array of 64 sensors located on the front and rear of the four server racks.  Data has been streamed from all sensors every 15 seconds for the last 4 years.  In addition to this detailed tracking of temperature and humidity conditions there have been a number of studies conducted using copper and silver coupons to evaluate the corrosion potential of operating using outside air and evaporative cooling.  Keep in mind that this application is in Dallas, Texas...a relatively hot/humid climate area.  In addition, because the "pod" is installed between two manufacturing buildings in an industrial zone near downtown Dallas the measured air quality around the "pod" is classified as G2, or moderately harmful to PCBs.

The June, 2017 Volume 139 edition of the ASME Journal of Electronic Packaging includes a paper presenting the results of the last 4 years of the research at this site.  The paper entitled "Qualitative Study of Cumulative Corrosion Damage of Information Technology Equipment in a Data Center Utilizing Air-Side Economizer Operating in Recommended and Expanded ASHRAE Envelope" provides a comprehensive look at the impact of operating a data center in a "real world" application.

The most interesting point presented in the summary section of the paper is that, in spite of the servers installed in this test site already being several years old, there has not been a single server failure in the entire four years of operation.  The ability to dramatically reduce the cost of operating a data center...without unfounded concerns about reliability...is finally being proven true.

Why Do We Design Thermos Bottles?

Over the last couple of months since my last posting I have been very busy managing our movement into new markets and grasping at new opportunities.  One of the benefits of taking the deep dive into these markets is getting to look at some of the details of product design and application to the specific problem to be solved. 

This has raised a question in my mind.

Why does the mission critical industry design "thermos bottles" and then fret over the cost of and methods of getting rid of the heat that all those servers generate? 



There is something that strikes me as illogical about creating buildings or modular data centers with super insulated walls and ceilings that are guaranteed to trap the heat that is dumped into the hot aisle (assuming they have aisle separation).  Then the mechanical system is tasked with rejecting all of the pent up energy without costing the owner a fortune.  Is it any wonder that data centers are one of the largest consumers of electrical energy in the world?

Centuries ago architects and designers figured out that it is more efficient to cool a space if you simply dump the heat out to the atmosphere.  Buildings used to be designed to take advantage of stratification and stack effect to cause the hot air generated in the space to rise and leave the building.  No need to cool the air back down to a reasonable temperature and put it back into the space so that you can heat it all up again.  Lofted ceilings and roof lines came into the design world for a reason. 

So, why is the data center different?  Frankly, I don't know.  Why not take the hot aisle air and vent it out to the atmosphere?  Sure, you have to replace that exhausted air with new air from the outside but unless the data center is located in Death Valley the odds are that the air being brought into the building is at a lower temperature than the air that would be recycled from the hot aisle of a data center designed to operate under the latest ASHRAE TC 9.9 guidelines for best practices. 

My best guess why we continue to do what is intuitively illogical is inertia.  "We have always done it that way".  I think it is time to rethink the old ways and come up with creative solutions in the design of data centers.

Mestex and the National Science Foundation Advisors Meet in Dallas

Although it has been far too long since I have posted to this blog due to my travel schedule I have some news to share.

Over the last few days (Oct. 1 & 2) we have been participating in the Industry Advisory Board ("IAB") meeting of the NSF-I/UCRC ES2 ("National Science Foundation-Industry/University Cooperative Research Centers Energy Smart Electronic Systems") research consortium.  That mouthful of letters represents a group of universities and companies whose expressed goal is to reduce the energy consumption of data centers by 20-35%.

The consortium is currently working on fourteen research projects and Mestex serves as an advisor ("mentor") on three of those projects.  Two of the projects that Mestex is mentoring cover research on evaporative and fresh air cooling of data centers and a second project on contaminants in data centers that use fresh air cooling.  As you might guess, the project on evaporative and fresh air cooling offers the greatest opportunity for the consortium to reach the stated goals.  In order to support that research, Mestex has installed a small data pod at it's facility in Dallas and is cooling that data pod with a commercially available Aztec ASC-5 unit.  The ASC-5 has built-in DDC controls that facilitate the use of multiple temperature and humidity sensors for control without any special modifications.  The controls also include a provision for pressure sensing control and that is also implemented in this case.

In addition to the data that is presented by the standard Aztec DDC controls there are additional thermocouples and sensors installed that are streaming data to researchers at the University of Texas at Arlington.

One of the most critical considerations that prevents many data center operators from reducing their energy consumption by huge amounts is the reluctance to introduce outside air to the facility.  The second Mestex project is focused on that research and we were fortunate to have the input of one of the world's experts on contamination control provide test coupons and laboratory analysis of the results.  Dr. Prabjit "PJ" Singh, of IBM, provides guidance and analysis to companies around the world and is a major source of information for the ASHRAE TC 9.9 committee on data center cooling.  Dr. Singh, Dr. Dereje Agonafer from the University of Texas at Arlington, and several members of the NSF IAB toured the Mestex facility at the conclusion of the meetings this week. 

Drs. Singh and Agonafer are shown here learning about the technology behind the patented "Digital High Turndown Burner" that was developed at Mestex.  Jim Jagers, Mestex Sales Manager,  conducted the tour and provided a "deep dive" into how this unique technology works before the group proceeded to the research data pod for additional discussions.

An "Open Access Project" Update

The Mestex "Open Access Project" continues to move forward so I thought I would provide a brief update on the current research activity and the plans for the next few months.

The installation at the Mestex facilities in Dallas has been brought up to the expected final configuration with a total of 120 servers, intelligent PDUs, and switches distributed over 4 cabinets.  We have separated the hot and cold aisles with a combination of a hard wall and flexible "curtains"...this has turned out to be one of the more important features of the installation.  The indirect/direct evaporative cooling system is fully functional although we have also found the need to increase the hot aisle exhaust pressure relief in order to reduce the "back pressure" in the hot aisle. 

In addition to the combination temperature and humidity sensors that are part of the standard Aztec control system, and used by the DDC control system to manage the operation of the Aztec unit, we have also installed 32, 10K thermistors.  These sensors are used to feed information to our data acquisition system that is running in the background collecting more granular detail about the system performance.  These sensors are located on the fronts and backs of the cabinets.

As I mentioned, we have spent some time resolving hot aisle/cold aisle separation issues.  Although the Aztec unit is monitoring cold aisle pressure and operating the supply fan to maintain a target positive pressure in the cold aisle we found that we still had hot aisle air migrating back into the cold aisle.  Over the last few days we have spent time filling small gaps and sealing around the cabinets more carefully and the results were immediately noticeable.  The cold aisle temperature was reduced by 5 to 6 degrees F. 

The other factor contributing to better separation was the reduction of the "back pressure" in the hot aisle.  We had addressed some of this earlier by removing the standard room exhaust grill and replacing it with a screen that had much greater free area.  While that made a measureable difference in server temperature rise we had simply moved the pressure issue from inside the data pod to the return air ductwork on the Aztec unit.  That has now been resolved by doubling the size of the pressure relief openings in the return ductwork.  Supply fan operation is now improved, server temperature rise is now on target, and supply fan motor power consumption has been reduced.  We monitor and report real time PUE for the pod and these changes have lowered the real time PUE to between 1.08 and 1.35, depending upon the system operating mode.

Now that we are beginning to see the kind of stable operation that we were anticipating we have started to plan the next phases of the research.

The Aztec unit is designed to operate in three modes, or some mixture of those modes, depending upon the sensor inputs.  The unit can operate in 100% fresh air cooling mode, in an indirect evaporative cooling mode, or in an indirect/direct evaporative cooling mode.  Each of those modes introduces characteristics that the data center industry wants to research. 

The next round of research will focus on two aspects of fresh air/evaporative cooling:

• We will be installing coupons in the space to collect data on contaminants and their potential impact on the circuits in the servers.  This project is projected to run for at least 1 month and support is being provided by IBM.
• Following the collection of this data (and possibly overlapping) we will be installing particle count measuring devices.  These devices will be installed upstream of the filters in the Aztec unit, downstream of the filters, within the cold aisle, and within the hot aisle.  The filter racks in the Aztec unit will allow us to evaluate filters of different MERV ratings and see how well they perform in a typical HVAC unit installation versus the controlled lab environment.

As you can tell, this site offers a unique opportunity for researchers to take their lab research findings and compare them to a real world application with real world equipment.  Mestex is pleased to be a part of this NSF sponsored research into data center cooling technologies.  We will be hosting a tour for the industry advisory board of the NSF-I/UCRC during their upcoming meeting at the University of Texas at Arlington.

NEWS RELEASE

New data center construction expected to boom as demand triples

 

Mestex Open Access Project helps data center operators plan for “build as you grow” expansion

 

DALLAS, April 28, 2014 – The digital revolution is sapping the power grid, but a new approach to data center construction may help reverse the trend of ever-increasing energy consumption for powering and cooling these facilities. To help data center operators better understand their options, Mestex, the industry leader in evaporative cooling systems, is providing a free tool to demonstrate how infrastructure can be better deployed to manage competing demands for more capacity and greater energy efficiency.

 “Data centers are the enablers of this digital revolution,” said Mike Kaler, president of Mestex. “The increase in global digital demand and cloud computing is exponential. As demand rises, data centers that house digital information consume more electricity, half of it being used to cool the facility. We wanted to help people see how energy is being consumed and ways for managing infrastructure and costs.”

The company believes intelligent technology combined with a flexible, scalable and energy-saving approach is the best way to “build as you grow.” Adding plug-and-play cooling units – such as Mestex’s own Aztec Evaporative Cooling Units – as capacity increases is the most economical strategy for data centers to manage expansion or new construction while reducing total cost of ownership. Aztec systems are proven to lower power usage by 70% when compared to traditional air conditioning; the system’s digital controls, when integrated with other building automation systems, can extend that savings even further.

To help data center operators get a realistic picture of how their own expansion might play out, the company recently launched the Mestex Open Access Project to provide information technologists, facility managers and financial executives the ability to evaluate energy-saving concepts in a real-world environment. 

“We’ve opened access to our equipment, controls and data, because we want to encourage energy savings and demonstrate to data center decision makers that there are smart, effective ways to increase efficiency and optimize operations,” Kaler said.

The web-based interface offers visibility into the physical plant and air conditioning system of an operating data center being tested as a part of a project spearheaded by the National Science Foundation. The “open access” gives anyone with Internet access an unembellished look at how a data center is operating, in real time, 24/7.

The Open Access Project harnesses the power of Mestex’s direct digital control (DCC) system, which comes standard on all of its HVAC products and can be easily integrated with other HVAC vendors’ products and building automation systems to create an intelligent network that controls cooling for optimal efficiency, performance and longevity, as well as provides web-based system monitoring and management.

 

Note:

Mestex President Mike Kaler will be hosting a presentation on mission-critical cooling systems on Wednesday, April 30, at 10:30 a.m. at AFCOM Data Center World at the Mirage Casino-Hotel, Las Vegas, Nevada. The company is exhibiting (booth #1227) at the conference April 28 – May 2.

Links:

Aztec Cooling Units for IT applications

Mestex Open Access Project Live View  (To access, Internet Explorer 10 or above is required. Enter “guest” as user name and password.)

 

# # #

 

Mestex (www.Mestex.com), a division of Mestek, Inc., is a group of HVAC manufacturers with a focus on air handling and a passion for innovation. Mestex is the only HVAC manufacturer offering industry-standard direct digital controls on virtually all of its products, which include Applied Air, Alton, Aztec, Koldwave, Temprite and LJ Wing HVAC systems.

 

Media Contact:

Christina Divigard

Divigard & Associates

413 341 6780 or Christina@Divigard.com

 

 

Indirect Evaporative Cooling Research Project Launched

Aztec ASC 3-D Model for CFD Research

ASME Paper Documents CFD Modeling of Aztec IDEC System

The 2013 ASME “International Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Microsystems”, aka ”InterPACK 2013” is just concluding in San Francisco.  As the conference title implies there are papers and presentations from all over the planet that are focused on research into improving electronics, computers, and data centers.

One of those papers presents results from an on-going research project that Mestex has started with the College of Engineering at the University of Texas at Arlington.  This research project will likely go on for a couple of years and this paper presents some of the first findings that are being used to establish a “baseline” for the rest of the research.

The paper is listed in the proceedings as “InterPACK2013-73302”.  The human-readable title is “CFD MODELING OF INDIRECT/DIRECT EVAPORATIVE COOLING UNIT FOR MODULAR DATA CENTER APPLICATIONS” and the paper covers exactly what the title suggests.  The IDEC product that the paper covers is the Aztec ASC-20 and the goal is to establish that the factory data that we present in our literature can be validated against a detailed CFD model of the product.

By modeling the Aztec ASC-20 components and creating the 3-D CFD model using factory dimensional drawings the researcher was able to confirm that the published factory data is accurate and the ASC-20 will perform as predicted based upon the operating parameters.  This important result can be used to further our research into optimizing the performance of evaporative cooling and fresh air cooling solutions for mission critical/data center applications.  A full scale modular data center mock up is being installed at the Mestex facility and additional documentation and validation of the performance will be conducted over the next several months.  The CFD baseline model will be used to simulate filter performance and airflow changes prior to making the physical changes to the research module.

The Aztec, and Alton DEC, evaporative cooling products have been used in the industrial and large commercial market since 1946.  Over those 66 years the products have been refined and optimized.  This research project will take the product lines to another level of thermal performance, water use optimization, and control software optimization with a specific emphasis on the needs of the mission critical market.

Dusting Off Your Data

CONTAMINANTS IN THE DATA CENTER

Time to get back on my soapbox again…this time it is about “contaminants” in data centers as an excuse to avoid using fresh air cooling or having outside air enter the white space.  The bottom line is that unless your data center is located in “an emerging country” then the odds of a contaminant-created hardware failure in anything like a short time frame are about the same as winning the lottery…assuming you take some pretty basic steps in the design.

Contaminant control, or more correctly, concern over contaminant control has been around for decades.  I remember doing some research over 25 years ago on the impact of ozone on telecommunications equipment.  Bell Labs, as it was known long ago, had performed some pretty interesting tests to document what could be a very real problem under the right circumstances.  The results of those tests indicated that, with the exception of certain locations, the air in the equipment room was worse than the air outside so it made more sense to flush the room with outside air than to avoid bringing outside air into the space.

Particle and gaseous contaminants CAN be a problem if ignored.  However, the extent of the problem and how quickly it manifests itself needs to be considered. 

Phenomenon like copper creep and circuit bridging do occur…but only when the conditions at the server are right to support those failure modes.  Two things generally need to be in place for the failure mode to even begin.  First, there needs to be a fine coating of dust particles on the circuit boards.  Second, the relative humidity at the board needs to be at the deliquescent RH…or the point where the dust starts to absorb moisture and become “wet”.  If the RH is too low then dust might affect localized temperatures on the board but the mechanism to cause bridging simply does not exist.  The converse is also true…no dust…then no mechanism even with a relatively high humidity level.

Dust can come from anywhere.  Every time someone enters the data center they bring in some amount of dust particles.  Every time a box is opened in the data center particles are created.  And, yes, every time outside air is brought into the data center it is possible that dust can enter.  In fact, a data center with no outside air is actually vulnerable to the worst kind of dust intrusion…uncontrolled infiltration through doors, cracks, pipe openings, or wind pressure.  Maintaining a positive pressure in the white space helps to prevent infiltration and keeps the worst dust (and gases) out of the data center. 

ASHRAE, through the TC 9.9 committee, has set a target for data center “cleanliness”.  It is ISO Class 8.  ASHRAE has also noted that ISO Class 8 conditions can be met with a MERV 8 filter…a common and inexpensive filter available at virtually any HVAC parts house.  If the air being filtered is coming from the outside then ASHRAE recommends a MERV 11 or MERV 13 filter.  These might not be quite as common as the MERV 8 but they are also readily available and can fit in a standard 2” filter rack.

The interesting side note about the ASHRAE recommendations is just how extremely conservative they are.  ASHRAE recommends no more than 15mg/m³ of “fine” particles…defined as particles less than 2.5mm in size.  However, IBM (who should know something about computers) has a limit of 150mg/m³ and a “fine” particle definition of particles less than 5mm in size.

Once again, owners are being led down a path to purchase cooling systems and equipment that fail to optimize their energy savings through an inflated fear of something that happens very rarely in the developed world and is easily controlled with proper filtration.  Products such as our Aztec ASC indirect evaporative cooling systems are designed with MERV 14 filters in mind and can actually accept MERV 16 filters…the highest MERV rating point…that is suitable for operating rooms and can remove all bacteria and most tobacco smoke.  This allows the Aztec system to optimize the use of fresh air cooling and use a more efficient heat transfer system than air-to-air heat exchanger systems…and still exceed the extremely conservative ASHRAE recommendations for particulate control.

Equilibrium

Equilibrium…we all try to achieve it in our lives.  An argument can go on forever if both sides maintain a high energy level and refuse to cool things down.  An argument can end when both sides take it down a notch and each reaches a happy place that they can both accept…a state of emotional equilibrium.

Odd as it may seem your air conditioning system is trying to do the same thing…reach a happy state of equilibrium…a balance between the high energy state and the low energy state.  Fortunately the system won’t get there under most circumstances because when the high and low energy states are equal then the unit stops providing cooling.

To simplify our thinking about this, substitute the word “temperature” for the word “energy”.  Now remember back to your days in physics class and remember that energy flows from a high state to a low state until the two states match and then the flow stops.  An air conditioning system takes advantage of that basic law of physics by absorbing the warm energy in a room and sending it to a lower energy place where the warmth is released and the cycle can start all over again.

One problem in this description of an air conditioning unit is that we usually don’t want the high energy released back into our rooms so we have to send that energy outside the room, or building, to get rid of it.  We do that by using refrigerants or water to transport the heat energy.  We also have to be sure that when we send it outside that it is at a higher energy level than the outdoors.  That is why we have compressors (and chillers which are just really big compressors) in our systems.  The compressors act as both a pump and as a device to actually add energy to the fluid that is pumped through the cooling coil in the room.  If you grab the side of a pipe entering a cooling coil it will feel relatively cold.  If you grab the pipe between the coil and the compressor it will feel a bit warmer.  If you grab the pipe on the leaving side of the compressor you might burn your hand.  The system has added enough energy to make sure that when the refrigerant or water reaches the outdoors it is at a higher energy state than the air outside the building.  That can be quite a challenge in a place like Phoenix or Dubai.

Most manufacturers realize that their equipment might be installed in those climates so they pick components in their systems that can operate under those circumstances.  But there are limits to what can be done.

The most widely available commercial cooling systems on the market are DX packaged units.  In order to satisfy the largest market (and sell the most equipment) these units are intended to be used for comfort cooling of people.  Since most people are “comfortable” when their office is around 75 degrees these units are designed around that operating point.  That is their happy point and that is the temperature of the air that is being returned to the cooling coil where heat energy can be absorbed into refrigerant or water and then sent outside to be removed.  The units will continue to operate at higher temperatures but remember that we need to be sure we send the heat outside at a higher level than the air outside. 

If you read the technical manuals for virtually every rooftop unit on the market you will see that, for a lot of really esoteric reasons, that rooftop unit is designed to operate at no more than 90 degrees returning to the cooling coil.  At that point the combination of components in the rooftop unit will be “maxed out” if the outside air temperature is in the 120 to 130 degree range. When the outside air temperature is 135 degrees on a roof in Phoenix then the high energy state and low energy state are so close together that almost no work is done and the temperature of the air coming out of the air conditioning unit starts to go up because the system is no longer rejecting much heat.  The system has quit working as intended and the situation usually spirals out of control as more heat builds up in the refrigerant or water.  Eventually so much heat has built up, and is compounded by the compressor, that the system shuts itself down to protect itself.

So, what does all this rambling have to do with anything?  Most of my blogs lately have been about mission critical/data center energy issues.  It is a big deal, and lots of folks are working on solutions, but economics sometimes trumps clear thinking or limits what can be achieved.

We are seeing more and more data centers specified with DX rooftop packaged units.  While these are normally high quality products they were originally designed to be at a happy place with, at most, 90 degree air being returned to the coil.  In a data center that is being designed to the latest ASHRAE standards the cold aisle can be anywhere between 80.6 and 113 degrees F.  When you allow a 20 degree F temperature rise across the servers before you return the air to the unit then I think you can see the problem.  The rooftop unit is being asked to operate well above its built in safety circuit limits.  Thus you end up with a self-limiting factor on how effective you can be in reducing the operating expense in the data center.  Even if you believe that your servers will be fine at 80.6 degrees F your HVAC unit probably will not be so fine.  And, to be honest, this same logic applies to CRAC units as they are nothing more than split DX systems.  So you have self-limited your options to cold aisle temperatures of no more than about 70 degrees F and your data center costs more to run than it could.

There is a class of rooftop unit that is better equipped to handle these situations and that class of equipment is commonly known as a DOAS, or Dedicated Outdoor Air System.  These systems, like our Applied Air FAP product, have been designed to expect Phoenix type temperatures across the cooling coil.  Returning hot aisle air at 105 or 110 degrees F is well within their “normal” operating ranges.  These systems are more expensive than a conventional rooftop packaged unit because of the components that are selected but they also provide the operating range that will allow the designer and operator to take advantage of the elevated temperatures that ASHRAE and the IT equipment people recommend for reducing data center operating expenses.

How I Spent My Summer Vacation

It has been a while since I have posted anything to this blog...no, I was not on sabbatical on some desert island...I have been traveling around North America talking to consulting engineers, contractors, and data center owners and operators.  This posting just provides a few insights that I garnered over the last 2 months on the road.

First, the data center/mission critical market continues to occupy the minds and the design resources of many, many companies in the design community.  It is clear that this is a market segment that is vibrant and all indications are that it will continue to be for quite some time to come.  The latest issue of Datacenter Dynamics FOCUS indicated that the world is now consuming over 300 Tkwh annually to drive data centers, with the US consuming over 25 Tkwh alone.  The consumption in the US is projected to grow over 9% in 2013.  While this information points to a growing market it also points to the urgent need for improved operating efficiency in data centers.

Second, and related to the first item, is the lack of knowledge about new "best practices" in data center design.  I have talked to dozens of engineers, contractors, and data center people who are not aware of the latest design guidelines from ASHRAE.  In fairness, those guidelines were only officially announced a few weeks ago...but they have been rumored and discussed for about a year now.  I mentioned in one of my earlier posts that education of the design community is an important, and ongoing, task.  This has been reinforced to me over the last 2 months.

Third, for those engineers and contractors who understand and embrace the new standards, is the challenge of convincing the data center people to adopt those standards.  This is less of a problem at the top levels of the data center company than it is on the floor of the data center.  The IT equipment operators who live in "the white space" seem not to understand the allowable operating temperatures of the equipment that they manage every day.  I have heard many different reasons for their reluctance to adopt the new best practices but I think it comes down to fear.  Because of stringent SLAs the operators worry about losing any equipment for any period of time...even though there is mounting research that this fear is unfounded.

Fourth, I have heard of several cases where the local electric utility has started to put limits on the available service capacity for planned centers.  In the US we are so comfortable with the idea that our electric grid can provide unlimited power that we forget that is not true.  We have a fixed number of powerplants with only so much generating capacity.  With the tremendous growth of data centers, and data centers with 300 to 500 watt per square foot electrical demands, there is a limit to what a utility can do.  And timing is another element of the equation.  A data center can be built in a matter of months...a powerplant takes years.  So even when a utility sees the demand coming they cannot add capacity as quickly as the demand can be added.

So, these are a few observations from the last couple of months.  Of course there is more to the story and feel free to comment on this post with any questions you might have.  I will try to respond as quickly as possible.

How to Save Almost $100,000 Per Year In Your 1 Megawatt Data Center

Over the last few weeks while I have been traveling there have been some interesting bits of information released in the mission critical world.

For example, Dell introduced their 12^th generation PowerEdge servers.  This generation of servers is warranted to handle temperature excursions up to 45 degrees C, or 113 degrees F, for up to 90 hours per year.  One of Dell’s rationales behind marketing the server at those conditions was to allow fresh air cooling in virtually the entire continental US.  Other research by Dell has indicated that their servers can operate 87% of the year in Washington, DC using fresh air cooling alone.

The energy saving potential of raising the inlet temperatures that high can be enormous.  Instead of running chillers or compressors 8,760 hours a year they are only operating 1,138 hours per year. 

To put that into numbers is difficult but let’s try a little example.

If the PowerEdge server power consumption is 300 watts then the cooling system must remove 300 watts times 8,760 hours per year or 2,628 kwh of heat (8,961,480 btu).  That can either be accomplished using mechanical cooling or fresh air cooling or a combination of the two.

A pretty efficient HVAC system will remove about 4.5 watts of heat per watt of electrical energy used.  So to cool that PowerEdge server using mechanical cooling will require 2,628,000 watts of heat divided by 4.5, or 584 kwh of compressor power.

To cool that same server using fresh air for 87% of the year will only require 75.8 kwh of compressor power.  Of course, the fan energy stays the same in both cases but the compressor savings of 508.2 kwh PER SERVER can really start to add up.  At an aggressive electric rate of 4.5 cents/kwh that amounts to $22.87 PER SERVER PER YEAR.

At modest densities of, say, 40 servers per rack the savings amounts to $915 PER RACK PER YEAR.  Now consider how many racks are in the typical server room or data center.  If the data center has a server load of 1 megawatt then a density of forty, 300 watt, servers per rack will translate into 83 racks.  So the annual savings would be almost $76,000 in this example.

To make the savings even greater the same HVAC unit that provides the fresh air could also provide indirect evaporative cooling and completely eliminate the compressor-based cooling…adding another $3.50 PER SERVER PER YEAR of savings.  That would add another $11,620 PER YEAR in savings.

Too Hot to Handle? A Simple Reminder

Well, this is embarrasing.  I have been in the HVAC industry for over 40 years now and have helped design and manufacture some of the more sophisticated products that have been introduced.  But, in spite of that I have to admit that I messed up.  And the lesson that I was reminded of can help you too if your residential, commercial, or mission critical system is struggling to keep up with the heat.

Over the past couple of weeks the temperature here in Texas has been over 100 degrees F every day...sometimes up around 105 to 110.  That is nothing unusual for Texas in the summer and not as bad as last year.  But I started to notice that my residential HVAC unit was no longer able to maintain my thermostat setpoint of 77 to 79 degrees F.  The system was consistently running 3 degrees behind and running non-stop...and was only installed a year ago.

Refrigerant leak?  Undersized?  Dog left the door open?

No...it was one of the most common problems in any HVAC system that is not running correctly...the condenser coil was coated with a fine film of dirt.  Let me repeat that...a FINE film of dirt.  Not clogged...not even very obvious at a quick glance...a FINE film.  In my case it was actually a fine film of dryer lint since the clothes dryer outlet was located behind the condensing unit...but the point is that had a service tech not looked at the coil with a flashlight I never would have noticed the dirt.  Running water over the coil from a garden house to wash off the film dropped the system head pressure and restored the system's ability to maintain the thermostat setpoint without running non-stop.

Many years ago Louisiana State University conducted some tests on residential HVAC systems to determine the impact of dirty condensing coils.  The results were eye-opening.  A fine film of dirt, similar to what I had on my system, would reduce system capacity by up to 20%.  If your home, business, or server room is too hot then imagine what giving it an extra 20% of capacity could do...and it would only cost you a bit of water and time to wash off the coils...with no service tech assistance required.

Preaching to the Choir

Electrical Power Meters Keep Spinning

I have had a busy few weeks traveling to meetings and visiting with owners, operators, engineers, and researchers.  This has given me an interesting perspective and awareness of an issue that our industry needs to address.  My awareness of this issue was increased by an editorial in Mission Critical Magazine that bemoaned the lack of progress in data center design due to secrecy regarding "best practices".

I came away from all of those meetings with the sense that there are many very smart people who know how to design more efficient solutions to energy use in mission critical applications.  "Best practices" can be described by experts from the largest server manufacturers, global data center developers/operators, and from academia.  The issue is that we are all sitting around a large table in a closed meeting room and sharing that knowledge with others who already have a pretty good idea what to do.  We are "preaching to the choir".

The result is that the vast majority of data centers, server rooms, and telecom facilities are operating in very inefficient ways.  While a Microsoft might be able to design a data center with a 1.2 PUE the rest of the world is struggling to reach a 2.0. 

This came out in a technical committee meeting at ASHRAE's mid-year meeting a few days ago.  A comment was made by a server cooling system manufacturer that he finds it very difficult to convince smaller users to adopt the latest operating standards that could save the user tens of thousand of dollars a year in energy costs.  This sentiment was echoed by several around the room and pointed to how difficult it has been to educate the broader public on the reliability of modern equipment in warmer rooms.

And when I say "broader public" I mean just that.  The mechanical design director for a global retail data center operator told me that he knows his equipment will run just fine at 78 or 80 degree F inlet temperatures but his customers have not gotten the message and demand a "cold" room.  It seems that until corporate IT managers and executives understand all of this we will continue to see skyrocketing energy use by data centers.  Even small server rooms could benefit from elevated temperatures if key elements of "best practices" were implemented.  So called "legacy" data centers might be difficult to retrofit but they can certainly be upgraded with the basic elements of "best practices"...if only the occupants understood what is possible.

The industry has a massive educational challenge if it is to stem the rising cost and consumption of energy.  And the education cannot come soon enough because the projections are that server power densities will continue to climb and data storage power densities will climb even faster.  Today we talk about 300 watt per square foot densities but systems are being designed already that push almost 10 times that density.  It may seem that we have an endless supply of power from the grid but there are only so many power plants around the world and building a new one takes a decade or longer...data power consumption grows at a much faster rate and will stress grids around the world eventually if we cannot educate the "broader public" more effectively.

Direct Evaporative Cooling Analysis for Two Diverse Climates

One of the common concerns expressed about the use of evaporative cooling for data centers, server rooms, telecom facilities, or other facilities housing heat producing electronics is the ability of evaporative cooling to achieve the target inlet conditions for the electronics.

These two psychrometric charts show the results of an actual analysis in two distinctly different climates.  The target server inlet temperatures were between 65 and 85 degrees F and between 20% and 80% RH.  The mechanical system criteria mandated that direct evaporative cooling be used.

The proposed mechanical system consisted of a direct evaporative cooling system with 12" cellulose media, a steam humidifier, DDC controls, and a hot aisle/outside air mixing section.  The controls would be configured to modulate the outside air dampers, hot aisle dampers, evap media water flow, and the humidifier to maintain the target conditions.

Server Inlet Temperatures from Evaporative Cooling System in Pacific Northwest US

Server Inlet Temperatures from Evaporative Cooling System in Southeastern US

As you can see from the charts the proposed system would easily achieve the desired results.  In fact, it was found that outside air cooling could achieve the targets during roughly 15% of the year, direct evaporative cooling could achieve the results during roughly 60% of the year, and remaining hours of the year when the air was too cold to properly operate the evaporative cooling without fear of freezing a combination of hot aisle and outside air supplemented with the humidifier would hit the target.  The only excursions of temperature over the maximum target of 85 degrees would occur for no more than 5 hours a year based on the NOAA TMY2 weather history.

While not every location would achieve these results the diversity of climate for these two studies imply that direct evaporative cooling, with creative use of mixing and controls, will work in many more climates than most people expect.  To further expand the capabilities of evaporative cooling an indirect evaporative cooling element could be provided to produce even broader temperature control.  Although it was not analyzed for this case we would expect that adding an indirect evaporative cooling element to these systems would have eliminated the few hours of excursion above the maximum dry bulb temperature.

As a means to dramatically reduce energy consumption for these data modules the evaporative cooling solutions such as those manufactured by the Aztec division of Mestek would prove to be extremely effective.  Since evaporative cooling systems are also relatively simple mechanical systems with no refrigerants maintenance of the systems does not require licensed refrigeration technicians and there are very few elements that could fail...increasing uptime and providing much faster recovery time should a repair be necessary.

"Make everything as simple as possible, but not simpler."

I have addressed this topic before but it bears discussing again.  I was reading an article in a high tech blog the other day and they repeated the oft quoted "rule" of good design from Albert Einstein..."make everything as simple as possible, but not simpler."...  A few months ago I also quoted an engineer who reminded me that a system is not "sustainable" if it is not "maintainable".

It seems that in spite of these two pieces of advice, and numerous studies that highlight efficiency degradation when equipment is not properly maintained, we continue to see elaborate custom cooling solutions when a simple "off the shelf" product will accomplish the same thing...and has a better chance of staying that way.

As an industry we bemoan the lack of qualified service technicians and then we turn around and send them to jobsites populated with unique, one of a kind, complicated HVAC solutions.  What are we thinking?

I will admit that there are some cases that are so difficult to solve that something special is truly needed.  Critical human medical care might apply.  Some very high tech product production might apply.  Production of pharmaceuticals might apply.  But most server rooms and data centers no longer seem to apply.  ASHRAE and the server manufacturers themselves have said that the old ways no longer apply.  IT equipment can stand much higher temperatures and humidities than previously thought and much broader swings of those measures than ever before.  So why design around complex custom equipment?

As a manufacturer we know, and can pretty accurately predict, how a standard piece of equipment will perform in any given situation.  As soon as we are asked to "change it just a little"...which normally actually means throwing out the original design and starting over...then all bets are off.  We can use the same standard of components that we would normally use with an expectation of similar performance but, in reality, we no longer know exactly what to expect.

And then there is the issue of compliance with the myriad of agency and code safety tests that all manufacturers must apply to their equipment.  Standard equipment is designed, tested, and certified to meet those standards...custom equipment is designed to the standards but is probably not tested and certified to the standards.

And finally we have the issue of maintainability.  Service technicians are trained to work on specific types of equipment.  Many types of standard equipment require licensed technicians for service.  Given the broad range of equipment types in the market today it would be extremely rare to find a service technician who could be proficient on all standard equipment....much less something he or she has never seen before.

The topic of "total cost of ownership" is starting to pop again in some publications.  It is reassuring to see that some people are starting to go back to considering something beyond the initial capital expense...but operating expenses consist of more than just energy costs...remember the cost of maintaining the mechanical system in the long run so that the money spent up front for an efficient solution does not go out the window a couple of years down the road.

Aztec Evaporative Cooling Solutions and the University of Texas at Arlington

On April 12, 2012 management and engineering representatives of Aztec, part of Mestex division of Mestek family of products, met with engineering representatives from the University of Texas at Arlington to define how resources will be jointly applied to researching and developing an advanced technology indirect evaporative cooling solution for data centers. Research already started at both organizations will be shared in order to more quickly advance the development of a viable solution to the high rate of energy and water consumption by data centers.

The joint project will be part of the NSF-I/UCRC program. This program, initiated by the National Science Foundation, is a collaboration between 5 major universities and a select group of industry contributors. The stated purpose of the program is to develop commercially viable solutions that will improve energy efficiency in data centers. Research projects range from chip level solutions all the way to complete, large scale, data center solutions. Aztec will be contributing special knowledge of evaporative cooling and outside air cooling solutions that has been developed over 40 years of product development and marketing.

Aztec Evaporative Cooling Solutions attends Data Center World Expo

Aztec Evaporative Cooling Solutions, a division of Mestek produced in the Mestex (Dallas) facility, was present at the recent Data Center World Expo in Las Vegas.  Aztec was showing an example of the ASC indirect evaporative cooling solution for data centers, server rooms, telecomm facilities, or IT product research labs.  The unit on display highlighted the system's integrated cooling tower technology, integrated DDC control system with multiple sensor options and BacNet or IP access, isolated direct drive plenum fan assembly designed for up to 200,000 hours of operation, and variable frequency drives for both cooling tower and supply fans. 
 
The show was attended by IT facility professionals from all over north America and Mexico.  Roughly 80 vendors were displaying an array of infrastructure products for data centers ranging from power distribution systems to cooling equipment.  Also visible in many booths were thermal and power simulation tools and DCIM software. 

The Aztec ASC system was the only factory assembled and tested evaporative cooling option for data centers that was on display.  Designed for long life, and with a history of extremely low failure rates, the Aztec ASC was considered to be a viable solution for many of the attendees.

Green Grid Updates Free Cooling Maps for Data Centers

The Green Grid has released White Paper #46 as an update to their "free cooling" maps for data center design and operation.  The research was edited by Emerson Network Power, Intel, and Schneider Electric. 

The reason for this update to the "free cooling" maps was the latest changes to the ASHRAE TC 9.9 operating/design guidelines for data centers.  For those who have not yet seen those new guidelines they allow a much larger operating range for data centers and server rooms that use some of the latest equipment from companies like Dell and HP.

For those of us who are "metric challenged" 40 degrees C = 104 degrees F and 35 degrees C = 95 degrees F.

When you consider that many data center operators still seem to want their rooms at 70 degrees or lower it is clear that these new criteria are a massive change in operation and design concepts.  It is also clear that adopting the newest guidelines can result in enormous energy savings.

The Green Grid paper includes a couple of maps to quickly illustrate how extensive the potential for "free cooling" has become under the latest operating/design guidelines. In these maps the darker the blue color the more hours that "free cooling" could be employed.  The darkest color blue indicates that all 8760 hours are suitable for "free cooling".  The maps also consider the coincident dewpoint temperatures as that metric is important also.

This first map is for ASHRAE Class A3 environments and shows that virtually all of North America could have their data centers cooled without using chillers or compressors.  The second map is for ASHRAE Class A2 environments and shows that roughly 80% of North America could still be cooled most of the year with no chillers or compressors.

The question for data center operators and designers who want to implement these new temperatures is what to do about those 500 or 1,000 hours when the outside air conditions are not quite right.

It is still quite possible to operate the center with no compressors or chillers if the designer will incorporate an evaporative cooling system such as the Aztec indirect evaporative cooling system or even the Alton direct evaporative cooling system.

Since evaporative cooling systems operate using 100% outside air all the time they make an excellent "hybrid" approach.  During the many hours of the year when "free cooling" will satisfy the conditions either type of evaporative cooling system will provide cool, filtered, outside air.  The Aztec indirect evaporative cooling system has the added advantage of allowing recirculation of hot aisle air during the very coldest months when "free cooling" could actually over-cool the data center.

During those few hours of the year, however, when it is simply too warm for "free cooling" to work, the Aztec or Alton systems can automatically initiate their evaporative cooling cycles and trim the outside air temperatures down to levels that fall well within the new ASHRAE guidelines...again, with no compressor or chiller energy required.  The air leaving the evaporative cooling system will usually be about 3 degrees F higher than the wet bulb temperature.  This chart should give you an idea of the potential air temperature that an evaporative cooling system can provide.

The Green Grid whitepaper is just the latest in a growing number of research papers and documents that point operators and designers in a direction that can save tens of thousands of dollars and kwh if they are willing to make the investment in the latest technologies from both the IT equipment manufacturers and the HVAC equipment manufacturers.

How We Used To Do It

I was recently reading an engineering magazine article (I know, I need to get a life) and came across a question that set me to thinking..."how did people stay cool before we had chillers?".  After all, in the grand scheme of life we have only had chillers and air conditioning systems for a very short time.  So what did people do before those things existed and what can we learn from that?

One of the first lessons from the past is that hot air rises.  Seems obvious doesn't it?  Believe it or not there is actually a company that is successfully convincing people that by making their air even hotter than everyone else they can do a better job of keeping people comfortable from 20 or 30 feet above them.  But that is a different story for another time.

Stack Effect

Because people realized that hot air rises, many early structures in very warm climates would be built with very high roof lines.  This would allow the hottest air to stay above the people and increase their comfort.  Many of those structures would also have vents or openings at the highest point of the roof so that the hot air could escape.  As that hot air left the structure it would be replaced by cooler outside air near the floor level.  A continuous circulation pattern would develop that kept the "cooling cycle" going.  The taller the structure, and the hotter the air, the faster this cycle would operate.  Today, we call that phenomenon "stack effect" and you see it in every tall building elevator shaft in the world.  You also see it in chimneys for residences.

After the invention of air conditioning though we seem to have forgotten one of the key elements of this natural cooling cycle...venting the hot air out of the building.  In most modern air conditioned buildings we keep the hottest air inside the building and just keep cooling it back down in a constant cycle that requires compressor or chiller energy.  In many cases the hot air inside the building is still cooler than the hot air outside the building so this might make sense during the hottest months of the year.  However, in the case of a data center or server room, the hot aisle air is usually much hotter than the air outside...but most data centers use cooling equipment that constantly tries to cool down that hot aisle air resulting in huge energy consumption.

Some systems also take advantage of the "stack effect" in a shorter building by recognizing that any heat source in the space will create it's own "mini stack effect".  Cooler air will be drawn towards the heat source and the hot air above the heat source can be exhausted.  This creates some natural circulation in the space and is one of the key principles behind "displacement ventilation".

Another lesson from the past is that evaporating water will make air cooler.  We actually use that very same principle in modern chiller systems that include a cooling tower.  The cooling tower is nothing more than a very large evaporative cooler.  In the old days people would use wet cloths or reeds in a window opening and when air entered the building through those wet items (probably accelerated by the building "stack effect") the entering air would get cooler and the people would be more comfortable.  Today there are many types and sizes of evaporative coolers available, such as those from Alton and Aztec divisions of Mestek, and they work even better than those primitive early methods.  But no compressor or chiller energy is required.

Of course there are building construction techniques that are also based on lessons from the past.  Positioning a building so that the smallest outside wall area is the one that sees the most sun will help keep the occupants cooler.  Using "thermal mass"...thick, heavy, walls...can also keep occupants cooler by storing cool night air energy in the wall and releasing it slowly during the hottest part of the day.  Again, we often build very light weight buildings today and try to compensate by adding insulation but nothing beats two feet of solid rock.  Some architects are working to revive this technique and research is continuing on using chemical treatments on walls and ceilings that allow them to store energy longer.  One case where creating a lot of "thermal mass" might not be such a good idea is in the data center world.  Depending upon how the hot aisle air is handled it might actually be a good idea to make the walls very thin so that the heat can escape to the outside through the walls.  Finally, the use of shades and window coverings is also a key lesson from the past.  Some companies, such as the American Warming division of Mestek, offer exterior solar shades that actually track the position of the sun and change angle in order to maximize the shading effect.

There are many other lessons from the past that could be discussed but the key is to stop and think about how we used to do things.  Sometimes adapting ideas from the past to ideas from today can result in the best overall solution.

Recently there was an interesting article published in Mission Critical magazine that addressed cooling in data centers.  More specifically the article addressed the waste that is currently happening in many, many data centers by operating the center at too low a temperature.

ASHRAE TC 9.9, at the urging of IT equipment manufacturers, has been raising the recommended and allowable temperature and humidity ranges for all types of IT equipment.  There are now certain classes of equipment that have allowable operating temperatures of 113 degrees F and 80% RH...but we still see data center designs that call for 60 to 70 degree air entering the servers.  Even the most critical classes allow temperatures of 80 degrees F and 60% RH.

One of the reasons that is often presented for operating the center at such low temperatures is reliability.  There is now research that suggests that this is not a valid concern.

The article in Mission Critical magazine, authored by Mark Monroe, cites a few interesting bits of information. 

Using the Arrhenius model  for predicting MTBF (mean time between failure), raising the server inlet temperature from 77 degrees F to 104 degrees F reduced the MTBF from 15 years to 13 years...both probably well beyond the replacement cycle for the servers.  Given that prediction why run servers at even 80 degrees?

A second study by E. Pinherio, W.D. Weber, and L. A. Barroso’s (2007), “Failure Trends in a Large Disk Drive Population,” determined that there was no discernible relationship between disk drive failures and operating temperature.

Finally, Intel provided information to ASHRAE that allowed creation of a reliability factor calculation that was time and temperature based.  This "X-Factor" could be used to estimate changes in reliability from a baseline temperature of 68 degrees F.  The interesting thing that comes from this is that using an indirect evaporative cooling system, such as the Aztec ASC product line that can provide server inlet temperatures lower than 68ºF for the vast majority of the year, could actually increase reliability according to the algorithm.

The potential operating cost savings are huge.  Switching to the Aztec system that offers 100% outside air cooling most of the year, with supplemental evaporative cooling during the extreme highs, could save $67,000/1,000 kw of IT load for the average data center in the US...according to the information in the article.