The Texas Experience

 Once again it has been many months since I have posted anything. Since then we have obviously been dealing with the pandemic as well as learning the impact of a weak cybersecurity policy. Although I could probably write a novel regarding the cybersecurity issue there are many more expert people on that topic.

Today I want to go back to the original purpose of this blog that was to capture and share some random thoughts related to the industry.

One of the latest events that started the wheels turning in my head again was the partial shutdown of the Texas electrical grid. Again there are experts in grid design and operation who can, and have, discussed this event in great technical detail. There has also been the, now expected, political finger pointing and partisan debates. But I wanted to share some of those random thoughts about what this "localized" event might teach us.

Many engineers, especially those involved in critical HVAC infrastructure, are already aware of the fragile nature of the electrical grid in many parts of the US. At Mestex we provide systems to many different types of applications where a power outage could have a costly impact. Preparing for those potential failures was usually someone else's problem. The electrical engineers and suppliers of standby power systems were intended to handle the "short" periods without electrical power. Although we researched ways to integrate backup operation into our equipment the results would have proven too expensive to be marketable. What Mestex has continued to focus on is applying systems that try to utilize "site resources" efficiently in order to reduce the demands on the backup systems.

Looking at a broader picture we have seen the global trend toward "electrification". The goal, it seems, is to reduce greenhouse gases and other atmospheric pollutants that contribute to climate change. While some are still skeptical about climate change it has reached the point of a consensus among scientists around the world. Many countries and large corporations are on board with taking steps to mitigate their impact. Electrification is intended to move the source of pollutants away from the "site" where power is used to the "source" where power is generated. In theory this would allow better control of contaminants at a single point instead of at hundreds or thousands of "site" points. This would also facilitate the use of alternative energy sources such as wind generators that would be difficult to implement at the "site" level. So we have a relative rush to requiring electrical vehicles, electric residential heating systems, and even electric commercial/industrial heating.

At the same time that electrification is moving forward in areas that the average person can see there is a convergence of our ever increasing digital life with our daily power consuming life. Data centers are being built and turned on almost daily around the world. To the average person this is great as it means they are always connected no matter where they go. This also helps the electrification effort by providing the opportunity for sophisticated remote traffic management, demand control power distribution, and "smart" home appliances. But the electrical power consumed by data centers is almost mind blowing to the average person. A single, moderately efficient, small data center can consume as much electrical power as five thousand homes. Clusters of large data centers (as is common due to scale and locale) can draw enough electrical power to support entire towns or even entire less populated countries.

When these data centers or data center clusters are inserted into an already fragile electrical grid they add a strain factor that was not anticipated when the power station was designed 20 or 30 years ago. Data centers can be designed, built, and activated in months versus power stations that require years to complete. It is inevitable that a mismatch of power supply and power demand will occur.

It seems to me that part of what the Texas experience showed us is, first, electric power is critical to basic life support facilities such as water and sanitation. My second thought is that as much thought and research should be put into the development of highly efficient, "clean", "site" energy systems as into the electrification idea. Off-loading the grid with effective "site" solutions could help with the balance of supply and demand on the grid. Many large companies have already taken steps with solar arrays over their parking lots, small-scale wind generators on site, or private co-generation plants. In most cases though these are extremely expensive solutions. Their implementations have been driven as much by corporate "green" initiatives as anything.

Companies should also not lose sight of current technologies that are still viable "site" solutions and counterbalances to grid overloads. Although Mestex has transformed itself over the last few years into generating more revenue from cooling solutions than from their traditional natural gas heating solutions most people still consider the company to be a gas heating company. In applications that require large amounts of outside air, or that simply move huge amounts of air that must be heated, a modern and efficient natural gas heating system is a much more "climate friendly" "site" solution than an equivalent electric heat "source" solution. Mestex can provide such systems based on their decades of manufacturing such systems and research into optimized digital control of such systems. 

 Engineers and companies can meet their goals of responsible environmental stewardship by keeping in mind the contribution of "site" solutions as they also work to meet the transition to greater electrification.

Cybersecurity

You have probably heard or read about "Internet of Things", or "IoT" as it is called.  The numbers of devices being connected to the Internet are staggering with some projections of over 26 billion devices connected by the end of this year.  Many of those devices are going to be HVAC products either via a connected Building Management System or as a "stand alone" device with remote monitoring and diagnostic capabilities.

AHRI recently sponsored a meeting to discuss the security implications of connected HVAC products. It has already been acknowledged that one of the major "hacks" in the last few years (Target stores) was made through the HVAC equipment.  One of the messages of the AHRI meeting was that HVAC equipment is becoming a key target for hackers (either domestic or foreign) due to the lack of rigorous "cybersecurity" protection.  In one study a building system was tested using four attack models and 54 "threat vectors" were discovered.

The need to increase HVAC cybersecurity mechanisms is obvious in the Target case but there are other scenarios that cause concern to the government and utilities.  Many products are now being connected to the electric grid for purposes of load management or to implement real time pricing strategies.  The fear is that lax security at the HVAC equipment level could allow a hacker to penetrate and disable parts, or all, of the electric grid through the same ports used for communication with the grid.  Hacking into a building system that is not isolated from the occupants' business network would obviously open the door to financial information, proprietary product information, and personnel information that could be extremely damaging to a business.  During the meeting it was noted that small businesses that have been hacked have a high probability of going through bankruptcy due to the cost of recovery.

But suppose the HVAC equipment is not connected to the Internet but only to the building management system or even as a stand alone piece of equipment?  Could such a system be hacked also?  I would suggest that, although more difficult, it is entirely possible.  Most modern HVAC equipment operates with a digital control system.  That controller will have a port used for diagnostics or software updates.  A "bad actor" with a laptop and a cable could gain total control of the unit and disrupt a business operation through temperature or ventilation control settings.  Interestingly, in the AHRI meeting, it was noted that the three most common attack pathways were WiFi, Bluetooth, and finally an Ethernet cable....so a physical connection as mentioned above is not even necessary.

The financial, legal, and reputational impact on an HVAC manufacturer whose equipment is used as the pathway for a hack can be substantial.  Unfortunately there are no current cybersecurity standards for HVAC equipment as there are for medical devices, vehicles, military applications, or financial institutions.  A key goal of the AHRI meeting was to identify which current standards might be adapted to the HVAC industry and what role AHRI would play in establishing an industry standard.  There was also discussion of whether or not this should lead to an industry certification process so that manufacturers certify their equipment and processes to serve as an affirmative defense in a case where their equipment was the doorway into a hack.

In the meantime, before an industry standard might be created, manufacturers are warned to establish their own cybersecurity policy...updated frequently...as a means to establish that they are following "best practices" with regard to cybersecurity.  There are a number of cybersecurity policies from NIST, ASHRAE, UL and others that could be modified or adapted by an individual company to create such a policy.  NIST-SP800-171 is one such document that includes a comprehensive check list of security steps that could be used as a model.

The bottom line is that no matter how an HVAC manufacturer chooses to respond to this growing concern some response is better than no response at all.

Disrupting Distruptive Technologies

I recently had a chance to investigate an interesting product concept that has the potential to significantly change the way some systems are cooled.  The term "disruptive" came to mind and that caused me to think about "disruptive technologies" and what it sometimes takes to bring them to fruition.

It seems that there are factors that stand in the way of rapid commercialization of products that have the potential to truly change an industry.

Fear....a technology or product that can disrupt an industry will tend to be resisted by the incumbents in that industry.  The greater their current investment in their current technology the harder they will push back against the new technology.  The fear of losing their market to the "upstart" will either lead to: trying to ignore it and hope it goes away; or actively looking for weaknesses and promoting heavily against it.  In the HVAC industry, and many others, the "upstart" is usually a small company without the staying power to eventually convince an incumbent company to take a chance on their technology.  Eventually, the "upstart" does actually go away from lack of resources to sustain the fight.

Liability...in order for a new product to see the light of day in the construction industry, in particular, it must be specified or recommended by consultants or owner influencers.  Consultants, especially, pay large premiums for professional "errors and omissions" insurance.  The definition of "errors" extends beyond simply making a mistake in a calculation to failure to use "industry standard" practices.  Since most disruptive technologies do not come to the market with vast installed bases that would begin to qualify them as an "industry standard" practice, consultants will shy away from specifying them.

Collateral consequences....sometimes the technology can be proven in laboratory or field trial cases successfully but widespread adoption requires the participation of other industries and their decision makers.  If the new technology requires an unrelated industry to make significant changes to how they do business, or design their products, the disruptive technology could end up stillborn due to the actions...or inactions...of companies outside of their own industry.

Finally...communication...by their nature disruptive technologies are technologies that others have not considered or visualized.  If the inventor of the new thing and the team behind it cannot clearly and demonstrably communicate the value of the product then it will either go nowhere or take an extremely long time to find acceptance.  In Mestex' participation as a sponsor of a National Science Foundation research consortium we see this almost every month.  Teams of extremely bright young engineers and scientists work diligently to come up with the "next big thing" but fail to communicate the value to companies that could commercialize the idea...if they only understood it clearly.

It is often said that the construction industry has been the slowest to adopt new techniques and is among the least efficient industries.  Perhaps the ideas to change that are already incubating or being tested...but fear, collateral consequences, and communication are standing in the way.

Quality

Over the years that I have been at Mestex I have marveled at how well we have been able to control our warranty expenses.  Having come from one of the large HVAC product manufacturers where it was not uncommon to have warranty expense at 3 to 4 percent of sales our average of less than 1 percent of sales is extraordinary.

Of course, this level of product quality does not happen by accident.  Selecting components that are designed for long life, using material gauges that are one grade heavier than most competitors, constructing many products with welded steel frameworks... all contribute to products that are designed to last.  But a great design can fall down at the production level so we have also implemented laser alignment systems, rotating component balancing systems, multi-point functional testing of every product that leaves the building, and compliance with all industry standards for safety.  We are confident that when a product leaves our building that it has been built to a high standard for longevity, service, and efficiency.

But there are standards outside of our normal industry standards that can improve our quality beyond even the product itself.  One of those internationally recognized standards is, of course, ISO 9001.  Reaching into all elements of our business and documenting how we do things in an effort to make all phases of the business better is an expected result of attaining an ISO certification.  Mestex is proud to announce that we have now received such a certification and we are now an ISO 9001-2008 certificate holder. (Certificate No.: TRC 00937 issued to Mestex, Dallas)

We don't intend to stop there however and we have already started work on attaining an ISO 14001 certification.  This process is targeted at our environmental practices and policies.  Although Mestex has taken major steps over the last few years to reduce our environmental impact we believe that there is more that we can do and that is our next target.

Kit Car or Ferrari?

Sticking with my motorsports analogies for a while...do you want to buy a "kit car" or a Ferrari even if the Ferrari costs more?

I am a big fan of Top Gear and watch most every episode at least once.  From time to time they will air an episode that features some sort of "kit" or one-off vehicle from a small manufacturer or garage.  Often times these vehicles have breathtaking performance and attractive costs when compared to a Ferrari.  But they usually end the broadcast with the conclusion that, great as the "kit" car might be, they would not buy one.

Time after time these "one-off" vehicles break during Top Gear's testing.  When that happens the crew has to wait for unique parts to be delivered or sort out the problem without the benefit of an owner's manual or factory service department to call.  Even in cases where the vehicle performs well they find that it cannot be licensed for the street or has no DMV certifications.  So they conclude that a buyer would be better off to spend a bit more and purchase a vehicle that is tested, certified, and supported by a company that is large enough to stand behind their products for the long haul.  Both solutions can provide exhilarating transportation but only one can be counted on to provide that transportation for as long as the owner keeps it.

The same question should be asked regarding HVAC equipment.  There are many small businesses that can purchase components, sub-contract the sheet metal, and hire temporary help to assemble units.  But the end user is basically getting the HVAC version of a "kit car".  The first cost might be attractive but certifications, IOMs, application and testing expertise, technical support, and supplier financial strength will all typically be important missing ingredients.

For some speculative buildings the "kit car" solution might be chosen because the developer knows he will soon pass the potential problems off to the new owner.  But for larger corporate owners and developer/owners is the "kit car" solution really the right answer?  Those end-users are investing in a building that will last years...they deserve an HVAC solution that matches that time frame from a company that will also be around to provide support for the duration.

Politics and the Building Industry

Climate Change Initiative

Coal Fired Power Plants in Danger

I am not sure how many folks listened to President Obama's speech this week regarding climate change initiatives.  I know that I was not one.  However, I have read the document that served as the background for the speech and there are some things in this document that folks in the building design community and mission critical world, in particular, should pay attention to.  Those things could have a significant impact on the types of systems that we can design and implement in the coming years.

The theme of the speech and the document is primarily reduction of carbon emissions and increases in "renewable" sources of energy.  There are some other things in the document that are focused on electric generation infrastructure.  However there is a potentially ominous element to that topic that is related to the overarching goal of reducing carbon emissions. 

By means of a "Presidential Memorandum" Mr. Obama has instructed the EPA to accelerate transitioning power plants to "clean" energy sources, i.e. anything but coal.  As we have seen in some other cases in the HVAC industry as soon as the EPA has a mandate of that sort they move quickly to implement regulations that may, or may not, be carefully thought out for the old "unintended consequences" issue.

In my opinion the danger is rapidly removing significant generating capacity from the grid at a pace that cannot be matched on the construction side.  Even though the document also outlines a directive to speed up permitting of power plants it is still a fact that building a multi-megawatt power plant can take years.  With coal being the primary energy source for roughly 40% of US power plants you can see how a too quick implementation of rules that curtail their use can lead to problems.  Many states already operate on the edge of rolling blackouts and brownouts each summer so shutting down or limiting coal fired plants could get ugly.

Exacerbating this problem is the rapid and continuing growth of the data center market.  When these things come on line they gobble up megawatts of generating capacity in a single site...and they can come on line in a matter of months, not years.  Even if they never reach full utilization the power companies must be prepared to provide that power.  ASHRAE and others have tried, somewhat in vain, to communicate that these centers can operate without the heavy energy use of compressors or chillers.  As long as the local electric utility still has generating capacity that can be allocated to the data center this is OK...although not a very "sustainable" approach if you believe in that concept.  But, if that same utility now has to shut down 10 or 20 percent of its generating capacity then there may simply not be enough power to allow the luxury of overly cold air in the data center.

The implications for other building types are similar, although not nearly as extreme.  Systems that optimize the use of outside air as their primary cooling source augmented by smaller compressor or chiller plants could become the basis of design.  Concepts such as chilled beams that utilize higher chilled water temperatures and minimal fan power might need to migrate to smaller buildings than you see them in today.  And building shells will need to make more extensive use of passive and active shading systems.

So, once again, the building industry is going to be impacted by external forces that may have the best of intentions but that will also require rethinking of how we design and operate those buildings.

Trusting The Weatherman

Designing to a Standard

It is an interesting fact that many projects are "over-designed".  This is nothing especially new but it seems that we are seeing more of it lately.  As an example we are currently working on a project that will be located north of Detroit but is being designed to operate at temperatures that exceed the ASHRAE 0.4% design criteria for Phoenix.  On the surface this seems to be overkill in the extreme.  The increase in capital costs for equipment that will probably never have to perform to that level could easily drive the project over budget.

The psychology behind making design decisions of that type basically indicates a lack of confidence.  The end-user chooses to ignore the ASHRAE climactic weather data and recommended design points because he or she lacks confidence in the data.  Personal experience of temperatures that exceed the published design conditions add to the lack of confidence in the recognized standard.  ASHRAE has tried to address this by also publishing the 10, 20, and 50 year maximum (or minimum) recorded temperatures.

This criteria is similar to the "100 year flood" criteria that civil and site planning engineers use.  Many of us have seen, or experienced, times when the "100 year flood" line has not only been crossed but crossed multiple times.  At a recent meeting that I chaired we had a presentation by a well regarded environmental and site planning engineer.  The presentation showed how the location of various coastal high water design lines have changed over the last few years...moving further inland and changing the flood insurance status of existing structures that were originally well outside of the potential flood area.

Could it be that the climate is actually changing as many people suggest?  Do we need to revisit our temperature design criteria more often?  The alternative is to ignore the standard and add an arbitrary "risk premium" to the design criteria...adding costs that might not be necessary.

When I was a young consulting engineer many years ago I was told to design to the ASHRAE design points.  One reason was that by using a recognized standard I could always fall back on that point as evidence that I had used proper engineering practices in my design.  The nature of mechanical equipment was such that most systems ended up over-sized anyway and could throttle their performance to meet the criteria.  If owners today add a "risk premium" to their design criteria...and then the mechanical equipment also ends up over-sized...then the capital costs and system capacities are doubly over stated.  As we move towards a market where building operating characteristics are posted by the front door, much like an automobile's gas mileage rating, the practice of arbitrarily over-sizing systems will put some owners at a disadvantage when it comes time to lease the space.

Too Hot to Handle? A Simple Reminder

Well, this is embarrasing.  I have been in the HVAC industry for over 40 years now and have helped design and manufacture some of the more sophisticated products that have been introduced.  But, in spite of that I have to admit that I messed up.  And the lesson that I was reminded of can help you too if your residential, commercial, or mission critical system is struggling to keep up with the heat.

Over the past couple of weeks the temperature here in Texas has been over 100 degrees F every day...sometimes up around 105 to 110.  That is nothing unusual for Texas in the summer and not as bad as last year.  But I started to notice that my residential HVAC unit was no longer able to maintain my thermostat setpoint of 77 to 79 degrees F.  The system was consistently running 3 degrees behind and running non-stop...and was only installed a year ago.

Refrigerant leak?  Undersized?  Dog left the door open?

No...it was one of the most common problems in any HVAC system that is not running correctly...the condenser coil was coated with a fine film of dirt.  Let me repeat that...a FINE film of dirt.  Not clogged...not even very obvious at a quick glance...a FINE film.  In my case it was actually a fine film of dryer lint since the clothes dryer outlet was located behind the condensing unit...but the point is that had a service tech not looked at the coil with a flashlight I never would have noticed the dirt.  Running water over the coil from a garden house to wash off the film dropped the system head pressure and restored the system's ability to maintain the thermostat setpoint without running non-stop.

Many years ago Louisiana State University conducted some tests on residential HVAC systems to determine the impact of dirty condensing coils.  The results were eye-opening.  A fine film of dirt, similar to what I had on my system, would reduce system capacity by up to 20%.  If your home, business, or server room is too hot then imagine what giving it an extra 20% of capacity could do...and it would only cost you a bit of water and time to wash off the coils...with no service tech assistance required.

Preaching to the Choir

Electrical Power Meters Keep Spinning

I have had a busy few weeks traveling to meetings and visiting with owners, operators, engineers, and researchers.  This has given me an interesting perspective and awareness of an issue that our industry needs to address.  My awareness of this issue was increased by an editorial in Mission Critical Magazine that bemoaned the lack of progress in data center design due to secrecy regarding "best practices".

I came away from all of those meetings with the sense that there are many very smart people who know how to design more efficient solutions to energy use in mission critical applications.  "Best practices" can be described by experts from the largest server manufacturers, global data center developers/operators, and from academia.  The issue is that we are all sitting around a large table in a closed meeting room and sharing that knowledge with others who already have a pretty good idea what to do.  We are "preaching to the choir".

The result is that the vast majority of data centers, server rooms, and telecom facilities are operating in very inefficient ways.  While a Microsoft might be able to design a data center with a 1.2 PUE the rest of the world is struggling to reach a 2.0. 

This came out in a technical committee meeting at ASHRAE's mid-year meeting a few days ago.  A comment was made by a server cooling system manufacturer that he finds it very difficult to convince smaller users to adopt the latest operating standards that could save the user tens of thousand of dollars a year in energy costs.  This sentiment was echoed by several around the room and pointed to how difficult it has been to educate the broader public on the reliability of modern equipment in warmer rooms.

And when I say "broader public" I mean just that.  The mechanical design director for a global retail data center operator told me that he knows his equipment will run just fine at 78 or 80 degree F inlet temperatures but his customers have not gotten the message and demand a "cold" room.  It seems that until corporate IT managers and executives understand all of this we will continue to see skyrocketing energy use by data centers.  Even small server rooms could benefit from elevated temperatures if key elements of "best practices" were implemented.  So called "legacy" data centers might be difficult to retrofit but they can certainly be upgraded with the basic elements of "best practices"...if only the occupants understood what is possible.

The industry has a massive educational challenge if it is to stem the rising cost and consumption of energy.  And the education cannot come soon enough because the projections are that server power densities will continue to climb and data storage power densities will climb even faster.  Today we talk about 300 watt per square foot densities but systems are being designed already that push almost 10 times that density.  It may seem that we have an endless supply of power from the grid but there are only so many power plants around the world and building a new one takes a decade or longer...data power consumption grows at a much faster rate and will stress grids around the world eventually if we cannot educate the "broader public" more effectively.

GreenBuild Toronto

This past week I was able to participate in the GreenBuild conference in Toronto.  This was my first opportunity to attend and I have come away with a few impressions.

First, I was surprised at the sheer size of the exhibit area.  The Metro Toronto exhibit hall is quite large and split into two sections spanning the CN Rail lines in downtown Toronto.  Both sections were completely full with exhibits for everything from flooring to roofing and everything in between.  All exhibitors promoted the "green" or "sustainable" aspects of their products...even if those aspects might not have been obvious on first or even second thought.  In addition to the three Mestek HVAC product booths the exhibit area included product displays from most of the major north American HVAC manufacturers.

The HVAC equipment companies, as well as many of the lighting and appliance companies, shared one common thread.  That is the emphasis on the "man-machine interface"...how the user interacts with the equipment for control or information.  Touch screen displays were everywhere providing access to virtual control points and providing occupant feedback on temperatures and, most importantly, energy utilization.  The degree of sophistication of the displays varied but the message was the same...in order to conserve energy people must have some sense of how much they are using.  It goes back to the old saying that "you can't manage what you can't measure".  The use of energy metrics have been proven to change occupant behavior and the HVAC industry is stepping up to help with that effort.

Mestex started our efforts in that regard several years ago with our Adaptaire DDC control system and we currently provide basic control and trending information via the Internet.  More developments are underway to further enhance the user feedback and help focus their attention on conservation of our energy resources.

Mestex has been providing this open protocol system for almost 10 years and has successfully integrated with virtually every other major control protocol on the market.  This has allowed building owners to gain visibility into the various Mestex brand products through their own building automation system in order to provide a richer set of points for monitoring and control.

Sustainability at Mestek Dallas

I am currently at the FMA Sustainability Conference in Florida and have been listening to 2 days of presentations about ways to be a more sustainable operation. Most of the audience would be described as executive level sustainability managers for Fortune 500 companies. After listening to these presentations, and giving one myself, it struck me that at the Mestek facility in Dallas we are actually pretty far down the road in implementing sustainable practices. For many, if not most, of the companies presenting or attending the conference the motivation was social responsibility with a by-product of reduced energy costs and improved financial performance. Our motivation was financial with the by-product being sustainability...but no matter what drives you the end result is the same. So, what has Dallas done to make us claim to be working in the direction of sustainability?

*All lighting was upgraded to high efficiency ballasts and high efficiency fixtures throughout the facility.
*Occupancy sensors are used to control lighting in areas that are not occupied at all times.
*The HVAC system is predominantly based on zero-GWP evaporative cooling. In areas that require mechanical cooling all of the systems have been converted to R410a from *R22 for reduced GHG impact.
*Our painting process is being changed from a solvent based system to a water based system to reduce VOCs.
*We have eliminated the need for one inefficient steam boiler in our coil testing area by changing our test procedure and by changing the cutting oil used in our production process to an oil that can be cleaned with water.
*We have eliminated the need for an auxiliary cooling system in our server room by enclosing the hot server exhaust stream and venting it outside the server room.
*All scrap metals are captured and sent to a recycling facility.
*Corrugated cardboard packaging is now being compressed to reduce the impact on the landfill.
*Each office has a paper products recycle bin and the material is consolidated weekly and sent to a recycling facility.
*80% of our customer training is now conducted by web-based conferencing...eliminating thousands of miles of air travel and the associated carbon footprint.
*All product literature is available in electronic form and only printed when necessary and in the amount required.
*IOM materials can be downloaded from the Internet by scanning a barcode tag on products shipped starting in late 2011...eliminating the need for multiple copies of thick manuals.
*Invoices, acknowledgments, and engineering submittals are all generated electronically...eliminating reams of paper consumption annually.
*E-mail has replaced fax as our preferred means of sharing printed documents.

All of these steps are items that have been covered during presentations here in Florida. In hindsight it turns out the Mestek Dallas is well along the road to sustainability...no matter the motivation.