Recently, a data center retro-commissioning project uncovered issues that could have led to catastrophic failure. A maintenance workaround was fixed and, by identifying and subsequently eliminating single points of failure in a strategic manner, the facility’s resiliency was significantly improved. Ultimately, disaster was avoided.
The importance of retro-commissioning — where commissioning (Cx) processes are applied to an existing system with the intent of improving the resiliency and efficiency of its operation — has never been greater. Data centers (DCs) increasingly impact our daily lives, and retro-commissioning helps ensure they operate successfully.
Applying the retro-commissioning process to DCs can be challenging. They are unique buildings that have complex mechanical and electrical systems with advanced monitoring and controls, critical uptime requirements, and strict maintenance procedures. DC’s also consume large amounts of electricity and, in many instances, water. These facilities often have stranded capacity due to information technology (IT) equipment updates that can be discovered through a review of the building’s mechanical and electrical systems. Increased resiliency, enhanced controls, a reduction in operation and maintenance costs, increased realized capacity, improved marketability, and potential financial incentives are all good reasons to explore retro-commissioning.
Retro-commissioning is particularly useful for DCs that have never been commissioned, operate with unjustifiable metrics, experience frequent downtime, are operating under new ownership or operators, or are undergoing infrastructure upgrades or developing expansion plans. Retro-commissioning also helps improve the marketability of facilities such as those repositioning to be edge DCs. In addition, retro-commissioning can help DCs that are pursuing certification programs such as UL 3223. Certification to UL 3223 helps mitigate risk for data center owners and operators by providing a set of criteria to increase end-user transparency, provider accountability, and proper data center documentation. Existing DCs are required to provide proof of successful Cx as part of the certification process. If the DCs have no record of any previous commissioning activities or have undergone significant modifications post-Cx, then retro-commissioning could help meet the intent of the certification program.
The complexity of DC infrastructure can make retro-commissioning difficult, especially if there is active IT equipment served by the mechanical and electrical systems. It is imperative that a systematic approach be taken when planning the retro-commissioning exercise. There are four phases for retro-commissioning: planning, investigating, implementing, and handoff. Here are the specific considerations for each phase as they apply to DCs.
Planning
During the planning phase, the Cx agent is responsible for interviewing the stakeholders to define the goals, objective, and scope of the project. DCs typically have a myriad of stakeholders, such as operations, IT, disaster recovery, engineering, procurement, real estate, security, finance, and sales. The stakeholders have varied objectives and appetite for risk and the Cx agent should drive the collaborative process. At this point it is worth revisiting the owner’s project requirements (OPR) that were established during the previous DC builds and determine if they have changed over time. If the owner’s requirements have changed drastically, a discussion is required to set expectations and evaluate the benefits of retro-commissioning as wholesale changes are not often viable with this exercise. In addition, the Cx agent should review the feasibility of retro-commissioning during this phase. For example, if the DC infrastructure is nearing the end of its useful life, or there are major existing flaws, retro-commissioning might not be the prudent approach.
FIGURE 1.
After the interviews, the Cx agent should work with the facility managers and owners to compile as-built or design MEP floor plans and controls drawings, test and balance (TAB) reports, and past utility data. Access to the Engineer of Record (EOR), BMS contractor, and TAB contractor can also be extremely useful. Reviewing these documents, consulting with past and current team members, and surveying the DC and its supporting spaces will allow the Cx agent to develop a retro-commissioning plan.
It is important to consider phasing the work by discipline, weather, availability of systems, or any topic meaningful to the success of the project. The Cx agent should determine the scope of the work and plan on quantifying the anticipated outcomes using metrics that identify the energy and water consumption, such as power usage effectiveness (PUE) and water usage effectiveness (WUE). The total energy use is another important metric. However, it includes the energy consumed by IT equipment, which is typically not affected by the retro-commissioning exercise. Therefore, the reduction in the energy use alone cannot capture the positive impact of retro-commissioning.
The Cx agent should discuss the planning at the end of this phase with the stakeholders and identify and weigh potential risks against the proposed scope’s financial impact such as return on investment (ROI) and life cycle cost analysis (LCCA).
After this discussion, the Cx agent should complete the retro-commissioning plan and identify the metrics that will help determine the overall success of the project.
Investigation
During the investigation phase, it is important to conduct an in-depth building investigation and ensure the operation of the existing systems serving the DC are understood. A review of the operating documents, such as method of procedures (MOP), standard operating procedures (SOP), and critical equipment work authorizations (CEWA), can be beneficial when developing the retro-commissioning plan and prior to working on any systems serving an active DC. MOPs, SOPs, and CEWAs should incorporate contingency and safety plans to mitigate risk and limit operator exposure to dangerous equipment or situations. A safety expert should establish safety protocols and oversee safety during operation of all equipment. The Cx agent should create a functional performance test (FPT) and/or integrated systems test (IST) scripts and review with the project stakeholders. Scheduling work is a very important aspect even though there is no perfect time for retro-commissioning activities in an active DC. The Cx agent should formalize a retro-commissioning plan and issue it to the project stakeholders for review.
Implementation
It is the Cx agent’s responsibility to implement and verify the planning and approved scope of work. During the implementation phase, it is extremely important to follow safety protocols established during the planning and investigation phases. It is also critical to continually identify risks and set expectations as any activity, no matter how well planned, presents risk to the operation of the critical spaces. Because of the risk associated with retro-commissioning, it is recommended to avoid wholesale changes and instead phase the implementation plan. A phased plan allows for greater flexibility and fallback options if unforeseen issues arise. For colocation data centers, it is imperative that the team be aware of existing service level agreements (SLAs) as there can be serious implications if the SLAs are violated during the retro-commissioning exercise. To close out the implementation phase, it is important to establish measures to review and trend the metrics (i.e. PUE, WUE, etc.), which were utilized to justify the scope of work. It is recommended that facility managers continue to collect and trend data after the handoff phase.
FIGURE 2. Retro-commissioning these CRAC units resulted in cooling capacity and efficiency gains.
Handoff
During the handoff phase, the Cx agent should develop and submit the final retro-commissioning report. The report should include the final versions of MOPs, SOPs, and CEWAs for dealing with critical situations and scenarios in the future. It is critical to document the changes made to the systems as well. At a minimum, the team should update mechanical and electrical single-line diagrams and provide a narrative of the changes to the sequence of operations and any set points. Once the report is complete, the Cx agent should hold a closeout meeting with the stakeholders and the building operators.
It’s only the beginning…
As the world depends more every day on the smooth operations of data centers, the importance of retro-commissioning grows. Its many benefits, such as improved reliability along with increased system efficiencies and system capacity, mean that retro-commissioning is a service that will be in high demand for decades to come.
ESD was recently tasked with retro-commissioning a DC that was to be leased to a new tenant. The mechanical systems were approximately 10 years old and comprised of three dry coolers (N+1 redundant), two glycol water pumps (N+1 redundant), 10 water-cooled computer room air conditioning (CRAC) units (N+2 redundant) with integral water-side economizer (WSE) coils serving the critical IT load, and two water-cooled CRAC units (N+1 redundant) with integral WSE coils serving the electrical room.
During the investigation phase, it was discovered that the DC was operating with legacy set points and the sequence of operations was limiting the ability to economize and reduce energy consumption. The supply air temperature to the servers was controlled to 54°F DB and the design return air temperature was 72° DB (18° DB airside delta). Optimizing the operating parameters to be consistent with the ASHRAE TC 9.9 recommended thermal envelope led to an increase in the realized cooling capacity of the mechanical system and a more efficient system.
