Model Predictive Control Enabling Flexible Operation of Data Centers

Contents

Abstract

Kurzfassung

Contents

List of Figures

List of Tables

1. Introduction and motivation

2. Outline of the thesis

3. Literature Review

4. Implementation background

5. Data Center Control Strategy Methodology
5.1. Coordination by Model Predictive Control
5.2. Real-time MPC
5.3. Prediction based on ARIMA Model
5.4. Optimal Control
5.4.1. Objective Function
5.4.2. Equality constraints
5.4.2.1. DC power consumption
5.4.2.2. Real-time regulation violation
5.4.2.3. Predicted regulation violation
5.4.2.4. Power consumption of Server Rack System
5.4.2.5. Power consumption of UPS system
5.4.2.6. Storage level deviation
5.4.2.7. Power consumption of CRAC
5.4.2.8. Targeted power level deviation of CRAC system
5.4.3. Inequality constraints
5.4.3.1. Server Rack System
5.4.3.2. UPS System
5.4.3.1. CRAC system
5.4.4. Key Performance Indicators of Simulation Result

6. Data Center Model
6.1. Workload Profile
6.2. UPS Capacity
6.3. CRAC System information

7. Case Study
7.1. Parameters
7.2. Simulation result
7.2.1. Scenario 1
7.2.1.1. Regulation violation
7.2.1.2. Workload queue
7.2.1.3. State of charge
7.2.1.4. Data center temperature
7.2.2. Scenario 2
7.2.2.1. Regulation violation
7.2.2.2. Workload queue
7.2.2.3. State of charge
7.2.2.4. Data center temperature

8. Conclusion and Outlook

9. Bibliography

Appendix A

Appendix B

Appendix C

Abstract

To rise to the challenge of the growing number of distributed Renewable Energy Sources (RES) for grid integration, Ancillary Service (AS) is increasingly crucial to maintaining the stability of power grid worldwide. In recent years, discussions about Data Centers (DCs) no longer limit to their energy efficiency. Considering the rising rigid demand from ICT customer and the high energy demand of DC, it is possible for DC to be one of Demand Response (DR) resources providing ASs in the smart grid. This thesis presents an online energy-aware scheduling algorithm based on Model Predictive Control (MPC), which realizes a proper adjustment of DC power demand, enabling the flexible operation of DC. The present work focuses on the identification and implementation of an MPC strategy which aims at a proper scheduling for DC which makes the total power consumption of DC flexible to track the reference signal in a DR context. It is demonstrated how the combination and interaction of the components under DC architecture can be utilized to achieve the realizable potential of operational flexibility for AS. Numerical simulation results have been carried out aimed at the later application in real pilot DCs. Furthermore, the capacity of resisting disturbance of this MPC approach has been discussed.

Keywords: Data Center, Model Predictive Control, Ancillary Service, Demand Response, Thermal Modeling, Deferrable Workload, UPS, ARIMA Prediction, Regulation Command, Load Following.

Kurzfassung

Um der wachsenden Zahl verteilter erneuerbarer Energiequellen (RES) zur Netzintegration gerecht zu werden, wird Ancillary Service (AS) sogar zur Aufrechterhaltung der Stabilität des Stromnetzes weltweit unentbehrlich. Themen des Rechenzentrums (DC) sind in den letzten Jahren nicht mehr nur im Hinblick auf den energieeffizienten Betrieb. In Anbetracht der steigenden starren Nachfrage von IKT- Kunden und des hohen Energiebedarfs von DC ist es möglich, dass DC eine der Demand Response (DR) -Ressourcen ist, die ASs im Smart Grid bereitstellen. Diese Masterarbeit stellt einen Online-Energiesparplanungsalgorithmus auf der Basis von Modellprädiktive Regelung (MPC) vor, der eine korrekte Anpassung des DC- Leistungsbedarfs ermöglicht und den flexiblen Betrieb von DC realisiert. In der vorliegenden Arbeit konzentrieren wir uns auf die Identifizierung und Implementierung einer Methodik, die auf eine rationelle Zeitplanung für DC abzielt und es dem Gesamtstromverbrauch von DC ermöglicht, das Referenzsignal in einem DR- Hintergrund zu verfolgen. Wir zeigen, wie die Kombination und Interaktion der Komponenten unter DC-Architektur genutzt werden kann, um das realisierbare Potenzial der betrieblichen Flexibilität für AS zu erreichen. Es wurden numerische Simulationsergebnisse durchgeführt, die auf eine spätere Anwendung in realen Pilot- DCs hinweisen. Darüber hinaus wurde die Energieeffizienz von DC auf der Grundlage dieser Methodik diskutiert.

Stichworte: Datenzentrum, Modellprädiktive Regelung, Zusatzservice, Demand Response, Thermische Modellierung, Abschaltbare Arbeitslast, USV, ARIMAVorhersage, Regelungsbefehl, Lastfolge.

List of Figures

Figure 1: Development of data center capacity worldwide[2]

Figure 2: The Future of the Smart Grid

Figure 3: A breakdown of energy consumption by different components of DC

Figure 4: Demand response enhancement utilizing UPS

Figure 5: Overview of data center[44]

Figure 6: Real-time MPC framework of data center

Figure 7: ARIMA (6, 0, 5) Model

Figure 8: ARIMA (3, 0, 3) Model

Figure 9: Workload queue

Figure 10: UPS of data center

Figure 11: Data center thermal model

Figure 12: Air circuit

Figure 13: Recommended server room air temperature

Figure 14: Power flow into data center

Figure 15: Workload ratio and power consumption curve

Figure 16: Required power profile within the day

Figure 17: Adjusted profile of upcoming workload

Figure 18: CRAC System

Figure 19: Test for the length of thermal system specified Time-Step

Figure 20: Regulation violation/ scenario 1

Figure 21: Power coordination/ scenario 1

Figure 22: Length of workload queue/ scenario 1

Figure 23: SOC/ scenario 1

Figure 24: Air temperature at outlet/ inlet monitoring point/ scenario 1

Figure 25: The aggregate air temperature of the hot aisle/ scenario 1

Figure 26: The aggregate temperature of server rack system/ scenario 1

Figure 27: Server room air temperature/ scenario 1

Figure 28: Regulation violation/ scenario 2

Figure 29: Power coordination/ scenario 2

Figure 30: Length of workload queue/ scenario 2

Figure 31: SOC/ scenario 2

Figure 32: Air temperature at outlet/ inlet monitoring point/ scenario 2

Figure 33: The aggregate air temperature of the hot aisle/ scenario 2

Figure 34: The aggregate temperature of server rack system/ scenario 2

Figure 35: Server room air temperature/ scenario 2

Figure 36: Regulation violation/ scenario 1 (Signal 2)

Figure 37: Power coordination/ scenario 1 (Signal 2)

Figure 38: Length of workload queue/ scenario 1 (Signal 2)

Figure 39: SOC/ scenario 1 (Signal 2)

Figure 40: Air temperature at outlet/ inlet monitoring point/ scenario 1 (Signal 2)

Figure 41: The aggregate air temperature of the hot aisle/ scenario 1 (Signal 2)

Figure 42: The aggregate temperature of server rack system/ scenario 1 (Signal 2)

Figure 43: Server room air temperature/ scenario 1 (Signal 2)

Figure 44: Regulation violation/ scenario 2 (Signal 2)

Figure 45: Power coordination/ scenario 2 (Signal 2)

Figure 46: Length of workload queue/ scenario 2 (Signal 2)

Figure 47: SOC/ scenario 2 (Signal 2)

Figure 48: Air temperature at outlet/ inlet monitoring point/ scenario 2 (Signal 2)

Figure 49: The aggregate air temperature of the hot aisle/ scenario 2 (Signal 2)

Figure 50: The aggregate temperature of server rack system/ scenario 2 (Signal 2)

Figure 51: Server room air temperature/ scenario 2 (Signal 2)

List of Tables

Table 1: Load Profile Characterization of 3 U.S. DCs

Table 2: AIC scores regulation signal 1

Table 3: AIC scores regulation signal 2

Table 4: ASHRAE thermal guidelines

Table 5: Model parameters

Table 6: Key Performance Indicators/ scenario 1

Table 7: Key Performance Indicators/ scenario 2

1. Introduction and motivation

In recent years a significant number of data centers (DCs) are newly built throughout the world to meet ever-increasing needs of the digitalized society. DCs are at the core of many enterprises, including the small and medium-size enterprises (SMEs). Enterprises need IT-Support to process a great amount of data to extract useful information such as trends and predictions, in order to keep their competitive edge[1].

illustration not visible in this excerpt

Figure 1: Development of data center capacity worldwide[2]

As people become increasingly reliant on online services, social media, and cloud services, DCs have become a centralized part of intensive computational data execution (cloud) and remote storage. According to the report of Yole Development in July 2015[2], power needs for DCs will be doubled from 2010 to 2020 (predicted value) worldwide, approaching 60 GW in 2020. This enormous amount of power consumption for DCs is even greater than the installed global solar PV capacity in 2015 (51GW)[3].

The proportion of power generated from Renewable Energy Sources (RES) has increased over the years. According to the investigation of Kurzweil[4], the Global capacity of the installed solar panel has increased about 100 times from 2001 to 2014. However, unpredictability and intermittency are traits of power generation from RES.

Focusing on the control of power generation, the volatility of RES makes the maintenance of grid stability even more difficult. “The integration of renewable energy into the power grid is fundamental for improving sustainability, but causes significant challenges for the management of the grid that has the potential to increase costs considerably.”[5].

On the other hand, large-scale and fast-reacting energy storage is currently still not cost-effective[5]. Hence, application of the current energy storage technology to compensate for the large volatility of RES is not a reasonable solution. A better alternative for this issue is to regulate the energy consumption on the demand-side. Energy-intensive consumers should be prioritized to participate in the DR program than small energy consumers for their vast amount of energy demand[6].

illustration not visible in this excerpt

Figure 2: The Future of the Smart Grid

As a giant energy consumer, DCs have a large potential to be an appropriate candidate providing AS as one of DR resources in the smart grid. In other words, imposing regulations to DCs to contribute to the flexibility of the power grid is a worthy option to investigate due to their high power consumption. Additionally, the rise of power needs caused by the vast amount of data execution and storage requirement forces DC to enlarge the capacity of hardware for faster response. This rapid growth of DC construction scale as well as quantity increases the procurement cost of hardware. On the other hand, the development of DC also leads to the growth of operational cost significantly. In many cases, power costs exceed the purchase cost[5]. And proper DR adoption in DCs brings considerable monetary benefit to DC, which reduces the operational costs to a great extent. Meanwhile, such operational actions induced by

DR program improves the grid stability. Therefore, making DC act as one DR resource creates a win-win situation for both DC- and grid operator.

Electricity market has series of criteria and measures for DR resources concerning pre- qualification and failure of following instructions etc. However, it leads to a penalty and even disqualification within electricity market in America, if DC fails to match the DR signal for certain time [7]. In this work, the regulation signal from PJM is applied for the simulation test. This MPC approach can adjust itself by changing its parameters for different power trade circumstances, e.g., change the size of several kinds of time steps for European electricity market. Under this DR background, a methodology is found which aims at making the total power demand of DC flexible in order to enable DC to track the DR reference signal. In this thesis, an online energy-aware scheduling algorithm based on Model Predictive Control (MPC) approach is leveraged to demonstrate how DC achieves its potential of operational flexibility for providing AS, considering given demand respond and workload task. On the basis of operational boundaries and capacity, the focus on combination and interaction of the main components under DC architecture by modeling the synthesized behavior of data center will be highlighted. Related simulation results with the help of this MPC physical approach are later carried out. Those results present a high consistency between the power amounts based on a regulation command from grid operator and the corresponding energy demand of DC. Furthermore, the simulation result based on this methodology has been evaluated at the end.

2. Outline of the thesis

The reminder of this thesis is organized as follows:

Section 3 gives an overview of the literature related to this work. Readers will take a look at DC for DR participation, identifying the feasibility and applicability of DR in DCs. Meanwhile, this section describes the state-of-the-art MPC application for industrial consumers and the specifics of the MPC approach for DCs.

Section 4 presents the implementation context, where the modeling language and the related toolboxes are chosen to provide the solution for this work.

In Section 5 the methodology based on MPC approach is introduced. The detailed concept for DC modeling throughout this work is under the circumstance of participation in real-time DR. With the help of ARIMA Model the prediction for a Regulation signal from grid operator is generated for proposed real-time MPC Coordination.

Section 6 informs the reader of the DC Model in detail of this work.

In Section7, the final simulation results are carried out on the basis of this DC Model. Simultaneously, the selection of thermal system specified Time-Step length is discussed.

Finally, in Section 8, the essential conclusions are drawn and an outlook for future application in real pilot DCs is provided.

3. Literature Review

The literature review related to this thesis is divided into three main categories:

(1) DCs provide Ancillary Service,
(2) Model Predictive Control utilized for industrial consumer,
(3) The state-of-the-art MPC application for DCs.

To the improvement of the sustainable energy supply, different targets about renewable energy utilization are set: according to[8], European Union made a confirmation of the EU Climate and Energy Package, which is also known as the “20- 20-20” targets for emissions of greenhouse gas, use of renewable energy and promotion of energy savings; In China, the renewable energy target is included in the 5-year plans; The U.S. Clean Energy and Security Act, approved by the House of Representatives in 2009, requires electricity suppliers providing over 4 million MWh to produce 20% of electricity from RES by 2020[9].

Before the boom of RES, academia and industry have made great efforts to focus on the control on the power generation side to maintain the grid stability for long periods of time[10] [11]. However, the ever-increasing power integration into the grid from RES lead to the end of this conventional power control approach, since the power generation from RES is intermittent and uncertain. The increasing amount of power output from RES imposes challenges on grid stability. To compensate for the volatility of RES large number of energy resources are required to enhance the balancing capability of the power grid. Regarding the traditional concept, the higher share of RES means the larger leverage on reserve power plants for maintenance of grid stability[7]. According to the historical data in[12], it is shown that the conventional reserve power plants are related to high operational cost.

Thus, it is necessary to search for an alternative for this problem. Moving from conventional power systems towards Smart Grids, DR is a great concept to handle this situation in cost-effective ways. In this situation, energy consumers play an active role in adjusting their energy demand which reduces their energy cost. Through these DR incentives, energy consumers provide the AS to maintain the grid stability and reduce power failures, e.g., compensating the volatility of RES, either the surplus or deficit[13] [14]. There are mainly three DR resources providing ASs in smart grid: commercial, residential and industrial loads[15], which has been presented in several studies.

Under residential circumstances, Zhou et al. [16], have discussed the possible way by using agent-based modeling and simulation (ABMS) techniques for different types of commercial buildings to offer DR. Hu et al. [15], presented an approach which integrates three data sets based on a survey with respect to (1. the residential energy consumption; 2. the time use and 3. the customers’ reactions to financial incentives in DR program) to assess residential DR in a stochastic model. Chen et al.[17], developed one real-time price-based DR management for residential appliances by applying stochastic optimization and robust optimization. However, residential loads has its large burden like unpredictability and intermittency to participate in DR program, in comparison with industrial loads. And the technologies based on the macroscopic adjustment and control of the residential loads are still immature.

There is a wide range of studies focusing on the DR in industrial areas, e.g., the cement plant, metal smelter, steel powder manufacturer and DC, etc. In[7], the study is formed around a cement plant model. Zhang et al. found an approach which utilized energy storage to solve the poor granularity of the power consumption variation among a group of cement crushers. This methodology based on MPC succeeded to track regulation signal from grid operator. Similar to this work, the MPC approach coordinate the different components to enable the plant to provide a combined flexibility. For such optimization problem, mathematical model is widely utilized. According to[18], Huang et al. overcame several limitations of steel powder manufacturer as industrial loads, such as raw and intermediate materials and complex manufacturing process. Their approach solves energy cost minimization problem of the steel powder plant by using real-time price (RTP) based DR and one related mathematical modeling.

This thesis focuses on DC as the promising DR resource. Besides its large load for the grid and the fast-growing energy demand of DC mentioned above in Section 2, DC is now a great candidate to provide AS to grid thanks to the large operational flexibility. A certain number of studies have been presented in this emerging research field.

Despite the fact that DC currently performs little DR for several reasons, e.g., the lack of proper DR programs for DC[19] [20] [21], the wide recognition of its potential as DR resources keeps the focus of researchers on this emerging practice. For example, Ghatikar et al.[19] and Urgaonkar et al.[20] studied the new market designs for DCs. In[19] the implementation based on CRAC units adjustment and IT workload migration to decrease violation of energy command is recognized.

The wide recognition starts from two basic points: DC is extremely flexible load from the perspective of power grid operator, since they are normally high-monitored and automated [5]. In addition, the large energy share of its main components creates the large potential of DC flexible operation. Therefore, analysis of the main components and their energy share is needed fundamentally. Data centers are facilities applied for housing data processing equipment and associated ancillary components. In order to ensure the operational safety for DC, it is necessary to set a piece of equipment like environmental controls (e.g., industrial cooling systems), redundant or backup power supplies, redundant data communications connections and various security devices [20] [22] [23].

According to the study[22], Dumitrescu et al. presented general statistics about the DC energy demand. It is shown that 40% of the total electrical power of DC is used by IT systems effectively, the air-conditioning system consumes 35 percent of the electricity to maintain the environmental condition of operation.

Ghatikar et al.[19], investigated the Load Profile Characterization of several U.S. DCs in 2012. The detailed energy consumption of 3 sample DCs are listed in the following table

Table 1: Load Profile Characterization of 3 U.S. DCs

illustration not visible in this excerpt

Based on several ongoing studies with respect to the power consumption of different DCs[24] [25] [26], a representative data sample is found: Uninterruptible Power Supply (UPS) -, Server- and Refrigeration Systems power demand may account for approximately 10%, 40% and 40% of the entire energy need of DC respectively.

illustration not visible in this excerpt

Figure 3: A breakdown of energy consumption by different components of DC

In most current DCs, servers and the refrigeration system contribute to the main part of total power consumption of DC, while Uninterruptible Power Supply shares a smaller portion. According to the study[27], Servers stay centralized among a group of IT components which comprises storage devices and communication equipment, etc. The total energy need of DC is generally categorized into two types: power usage by IT equipment and by non-IT devices[28]. The power demand of Server- and Refrigeration system is prominent for each part respectively.

Thus, this work is supposed to tap the potential of DC flexibility mainly from Serverand Refrigeration system.

Ghatikar et al.[19], reviewed the findings and identify the feasibility and adoption of DR in data centers by exploring practical methods to overcome risks and study variable DR strategies based on different DC components from the site infrastructure (e.g., lighting, cooling) to information technology equipment (e.g. , storage, servers). They validate the DR strategy based IT equipment in 3 different directions: the load shed, the load shifting and the load migration- the geographic shifting of computational workload among a group of DCs. And the result of a field test in the load shifting environment[19] shows that there is a great potential for power demand reduction when the server is working in idle mode which enables DC to shave peak power during the deficit-time of power generation from RES. In[29] Liu et al. developed algorithms for DC via workload shifting and local generation to avoid the coincident peak, which is also proved to decrease the energy expenditure effectively. In work[30], Safavi et al. analyzed the trade-off between the operational stability of servers in DCs and the related energy usage under consideration of the length of workload queue during dispatching the new arriving jobs. The final simulation result based on their proposed optimization algorithm also reflects the DR potential of DC when the control strategy takes the load shifting technology into account.

Regarding the provided AS by leveraging on the refrigeration system, several approaches are presented in literature, ranging from heating, ventilation, and airconditioning (HVAC) system inside the residential building to the one in DC.

In the residential area, Sun et al.[31], investigated the impact of a large population of HVAC units on providing DR and how the HVAC physical parameters and their distributions affect the aggregate response to the grid operator.

In the study of Beil et al.[32], the HVAC loads of commercial buildings are deemed as potential candidates for providing AS because of the temporal flexibility offered by their inherent thermal inertia and the significant energy consumption of buildings for commercial use.

However, the study [33] points out the limit of DR potential based on the HVAC system belonging to residential and commercial buildings for their capacity range. Meanwhile, they consider DC as Ancillary Service provider and propose a related power management methodology, which is supposed for scheduling for HVAC system and a backup generator to track the regulation signal.

Wei et al.[34] modeled the HVAC system for both IT- and non- IT room in DC and propose an MPC formulation to co-schedule the DC and correlated components including HVAC, in combination with the use of UPS system and the local solar energy generation.

illustration not visible in this excerpt

Figure 4: Demand response enhancement utilizing UPS

Apart from the IT equipment and the cooling system, this work is supposed to take advantage of UPS to facilitate the flexible performance of the DC, which is shown in Figure 4. There is a wide range of studies where UPS temporarily enhances the capability of DR subject to provide AS further[7] [19] [33] [34]. Although the UPS system capacity limits the long time-DR performance[19], UPS as a backup device is adopted as a secondary choice for DR participation and strengthens the operational flexibility of the entire DR subject evidently.

Yet, research so far has not considered the combined operation- scheduling of the IT equipment, refrigeration system and UPS system as the site infrastructure enabling DC to participate in DR program, in spite of nearly 90% of total DC power demand caused by these parts. On the other hand, the aforementioned mismatching regulation signal from grid operator for certain time leads to a penalty and even disqualification within electricity market. Therefore this study is intended to accelerate the applicability and adoption of DR in DCs, based on state of the art.

In this work, a coordination framework is proposed in which the current DR technologies are leveraged, ranging from the DC workload shifting to the refrigeration system adjustment. This control algorithm based on MPC aims at avoiding the operation-accident of DC in a DR environment.

Model Predictive Control, also referred to as Receding Horizon Control (RHC), is currently one of the most popular advanced control methods. The birth of MPC can be traced back to the 1970s. In 1980s Model Predictive Controller was widely used in chemical, refinery and other process industries[35].

In comparison with other traditional control methods, explicit prediction of future controlled plant behavior over a finite receding time horizon is the basic and inherent feature of MPC. MPC is a control algorithm that optimizes a sequence of manipulated variable adjustments over a prediction horizon by utilizing a process model to optimize forecasts of process behavior based on a linear or quadratic objective, which is subjected to equality or inequality constraints[36].

Instead of optimization on the global horizon, this control technology is based on rolling finite-horizon optimizations, i.e., the resulting optimal input trajectory is obtained from optimization at each sampling instant over a finite horizon. Among the optimal results gained by solving the online optimization problem bound by related prediction horizon, the first value from the results of the relevant optimization will be taken as the control action at each time step[37]. Further, the whole procedure is performed online repeatedly with one time step shifted forward.

MPC utilizing finite-horizon optimizations only obtain the suboptimal solution for the global solution. On the contrary, the optimal global solution can be only gained by optimizing the problem over a global prediction horizon ideally. Yet, the unneglectable negative impact of uncertainties, such as model-plant mismatch, time-varying behavior, and disturbances, prevents from obtaining the best global solution of the problem, which can be not included in the process model of the controller and must be taken into account in the real case.

Based on the characterization of MPC, the receding horizon optimization can incorporate the uncertainties effectively because of the frequent optimization at each time step under the consideration of the reference trajectory, so that the controller is able to give an appropriate corrective control command to the plant in time in order to drive the future output as close as possible to the target value [35]. This is the reason why MPC shares the wide acceptance and popularity in the industry and draws more and more attention from academia.

MPC has proven to be capable of dealing with multi-objective problems and handling hard constraints explicitly[37]. It is currently widely applied as advanced control methodology in industrial applications. For example, in the study[38], Kufoalor et al. present a range of approaches for both formulating and efficiently solving industrial MPC problems. Arce et al.[39] studied the air-supply control based on MPC aiming at the operational safety and the high performance of Fuel cells. An MPC strategy operates the spray dryer is presented in[40]. The industrial spray dryer for enriched milk powder production achieves significant energy savings based on MPC approach.

Regarding the specifics of MPC approach in DR environment, several approaches have been proposed in the literature. In [7] the control strategy for cement crusher equipped with energy storage for DR is presented by Zhang et al., which is based on MPC and shows little deviation against the regulation signal. The study [34] presents that MPC optimizes the co-scheduling of HVAC systems for both non-IT- and IT-room in DC, in order to enable DC to participate in DR program. However, the most studies for DCs, which provides an MPC approach coordinating the diverse components such as IT equipment, local power generation from RES, refrigeration system, etc., aim at the reduction of operational cost and energy savings [34] [41] [42] [43]. Few studies focus on a control strategy that achieves the full potential of components under DC architecture to obtain the operational flexibility DCs to the utmost extent. Enabling DC to avoid failing to consistently track the reference signal in a DR context stands in the middle of the stage and shows its unneglectable meaning for current DCs in the smart grid. So this work is intended to fill the gap in this research field.

4. Implementation background

As the primary requirement for the thesis is to model the DC components and design as well as simulate an MPC algorithm coordinating the central DC components. Here three MATLAB toolboxes are leveraged: YALMIP, MATLAB optimization toolbox and MATLAB econometrics toolbox, based on MATLAB 2015b.

YALMIP and MATLAB optimization toolbox can be used together to model and solve optimization problems typically occurring in this control algorithm.

YALMIP is a free open source and developed by Johan Löfberg from Automatic Control Laboratory, ETHZ. This toolbox is based on MATLAB language. And users have only three basic YALMIP commands to learn for modeling and solving the optimization problems. YALMIP is mostly used to test and approach numerical simulation under specific circumstances for simulation to obtain the necessary results.

YALMIP is initially intended to model SDPs and solve these by interfacing eternal solvers. In this work, MOSEK solver is applied for the linear programming (LP).

MATLAB econometrics toolbox provides functions for modeling economic data. The adftest, AIC-test and forecast function are utilized for time series in MATLAB econometrics toolbox to predict the DR regulation signal over a certain time horizon.

In next section, the introduction for the methodology in this work is presented.

5. Data Center Control Strategy Methodology

5.1. Coordination by Model Predictive Control

The basic thinking of MPC approach is to predict the future behavior of DC as the control plant over a finite time horizon. An optimal control input is obtained from a series of results by computing the related optimization problem while meeting the demand of grid operator and ensuring satisfaction of pre-set constraints as well as the objective function of the system. The control input is the first part of the optimal results and then applied to DC until the next sampling instant, at which the horizon is shifted one time step forward and the whole optimization is repeated again.

The forthcoming state of DC is forecasted by utilizing a process model belonging to the Model Predictive Controller at each instant t, over a prediction horizon (n). The equation of the control logic in this thesis is represented as follows:

illustration not visible in this excerpt

Where

X_t — The state of DC at time step t —Control command at time step t

U_t— weighting parameters of plant model of Controller

A, B𝐵— weighting parameters of plant model of Controller

X_t+i for i=1…n as the forecasted state of DC is dependent on the past control inputs as well as state outputs until instant t, and also on the upcoming control signals U_t+i for i=0…n-1. For the optimization, the n upcoming control signals need to be considered as the ones to be dispatched to the system and then calculated. Only the first calculated value as mentioned above from n results will be taken as the real control command at instant t. Each optimization of a given condition aims at maintaining the real power consumption of the DC as near as possible to DR power demand which is also regarded as the reference trajectory for MPC.

In this work, the variables in the optimization problem can be separated into state variables () and control variables ().

The vector of state variables [] includes:

- Workload Queue ()
- State of Charge of UPS system ()
- Storage Level Deviation ()
- the aggregate temperature of Server racks system ()
- the average temperature of the cold aisle/ air flow inlet temperature(௜)
- air flow outlet temperature ( )
- the average temperature of the hot aisle (ℎ)
- the average temperature of Server Room ()

The vector of control variables [] consists of:

- Power consumption of Server Rack System ()
- Power consumption of UPS system ()
- Power consumption of CRAC ()

The energy use this work focuses on is mainly divided into:

(1) Servers execute data for customers.
(2) CRAC maintains the temperature of server room in a determined range to ensure safe operation of servers.
(3) UPS avoids sudden operational interruption of DC and requests the demand of grid.

A typical schematic diagram of a DC is presented in Figure 5.

illustration not visible in this excerpt

Figure 5: Overview of data center[44]

The state variables Workload Queue (), State of Charge of UPS () with Storage Level Deviation () and the temperature of several parts belong to the categories Server Rack System, UPS system, and CRAC system, respectively. Sensors in DC are equipped for monitoring the state variables above.

This work is supposed to coordinate the components from the aforementioned categories by utilizing MPC approach, which includes the consideration of the interaction among the DC devices, e.g., the increased data execution of servers needs CRAC system to remove more generated heat, and also the sustainable flexible performance of DC, e.g., the energy level near to full capacity of UPS system may lead to a limited flexibility in next time step.

5.2. Real-time MPC

The aforementioned regulation signal in this work is gained from PJM. For real-time DR provision, the DC receives a single regulation signal every 5 minutes. Note that this MPC approach can adjust itself by changing its parameters for different power trade circumstances. Thus change of the corresponding parameters will be done for the later application in European electricity market. Moreover, this regulation signal instructs DC to adjust its power consumption in accordance with the command of targeted energy demand only for the next five minutes. Nevertheless, each optimization at instant t on the basis of MPC approach needs one reference trajectory over the related MPC time period.

Thus, prediction of regulation signals combined with one single real-time signal helps us to gain a certain number of DR signals constituting the reference trajectory for optimization.

The framework of real-time MPC coordination is illustrated in Figure 6.

illustration not visible in this excerpt

Figure 6: Real-time MPC framework of data center

On the basis of historical regulation signals, the autoregressive integrated moving average (ARIMA) predictor generates the forecasted signals for the next R steps at time step t. With the real-time regulation signal replacing the first step of regulation prediction, a group of signals is inputted into the Model Predictive Controller. Then the results will be obtained by optimizing over the total energy demand of DC over R time steps.

5.3. Prediction based on ARIMA Model

For the prediction of the regulation signal, the ARIMA Model is used to forecast the future command from grid operator. In this work, the ARIMA Model is trained with the help of Econometrics Toolbox in MATLAB.

ARIMA Model, a generalization of an autoregressive moving average (ARMA) model, is fitted to time series data for forecasting future development of the time series based on understanding the present data.

Different series of the regulation signal are gained from PJM data archive. PJM is the largest competitive wholesale electricity market in the world until the development of the European Integrated Energy Market in the 2000s. The regulation signal, published in[45], will be used in this thesis. And this single regulation signal is obtained every 5 minutes, which determines the aforementioned size of MPC time step simultaneously.

For the determination of ARIMA order (p, d, q), the value of d is obtained in terms of adftest results. The parameter d is denoted as the degree of differencing. Adftest aims at the stationary check of time series. If the original time series is unstable, differencing will be utilized for obtaining stationary time series according to adftest. Then the parameter d equals the number of times that the data have past values subtracted.

According to the Akaike information criterion (AIC) scores, the values of parameters p and q are gained, with which the ARIMA model has the best performance for prediction. Two series of regulation signals are applied for the case study in Section 7. Regulation signal 1 and 2 are obtained based on two uncorrelated dates in order to check the general applicability of the MPC approach.

The AIC test is stated by following equation:

illustration not visible in this excerpt

Where

AIC— A/C value

k— The number of prediction parameters in the model

L— The maximum logarithmic likelihood value for the model|

n— The number of observation steps

Starting from the historical regulation data of signal 1 and 2, the ARIMA (6, 0, 5) model and ARIMA (3, 0, 3) model are determined after the aforementioned model training test, respectively. The AIC score of both signals are listed in the following tables and the ARIMA order is gained on the basis of the lowest score value.

illustration not visible in this excerpt

Table 2: AIC scores regulation signal 1

illustration not visible in this excerpt

Table 3: AIC scores regulation signal 2

The ARIMA Models are described by following equations:

illustration not visible in this excerpt

Where

— The Regulation signal at time step t — The white noise at time step t — Moving-average parameters — Auto-regressive parameters

The Auto-regressive- and Moving-average parameters are trained and gained by the Econometrics Toolbox in Matlab, which includes the function of Adftest and AIC test.

After the determination of ARIMA parameters, the prediction frequency is set according to the time step size of the controller, every 5 minutes. Because of the characterization of ARIMA model, an accurate prediction cannot be ensured over a long time period. Thus the number of observation steps is decided according to the prediction accuracy. The Mean Square Error analysis of prediction is stated by following equation:

illustration not visible in this excerpt

Where

E_i— Value of estimated regulation signal

R_i— Value of real-time regulation signal

n— The number of observation steps

(೔−೔)[2]

(5)

Based on this analysis, the MSE value of two ARIMA models for 15 steps is presented in following figures. It is shown that ARIMA model within 12 steps provides the forecasted values with reasonable accuracy. Thus, each sub-optimization in this MPC approach predicts the signal behavior in next one hour. I.e., with the time step size of 5 minutes, the prediction in MPC approach under the real-time DR circumstances over 1 hour-time horizon is feasible, since prediction MSE is controlled below 10%.

illustration not visible in this excerpt

Figure 7: ARIMA (6, 0, 5) Model

illustration not visible in this excerpt

Figure 8: ARIMA (3, 0, 3) Model

Subject to the elapsed time of ARIMA order determination, the order will be set once and applied for the entire MPC time period. This decision is based on the correlation of signals within a few days[7]. Therefore, the ARIMA models won’t be re-trained and error on the prediction might occur during the simulation.

5.4. Optimal Control

In this work, the regulation baseline is set close to the average energy need of the Server Rack System and the CRAC system of DC, while the change of UPS system storage level cannot be estimated over one long time period. The average power required for Data center is related to the IT workload. And the regulation baseline is fixed with the regulation capacity, which are denoted as B in kW and R in kW, respectively. The corresponding calculation is described in Section 7 in detail. In addition, the future research will focus on how to calculate the optimal values of B and R reasonably.

The per unit value of the aforementioned regulation signal from PJM ranges from -1.0 to 1.0. The regulation command for DC, as the aimed power consumption, consists of the regulation signal, the regulation baseline B and the regulation capacity R.

The regulation command is stated by the following equation:

illustration not visible in this excerpt

Where

P_DR,t — The regulation command for DR provider at time step t (kW)

R— Regulation baseline (kW)

B— Regulation capacity (kW)

W_t—Regulation Signal for DC at time step t

The detailed regulation command P_DR,t will be explained in the following section 5.4.2.

The objective of the optimal control is to enable the flexible operation of DC, in order to provide high quality AS, while the mismatching regulation signal from grid operator is avoided to the greatest extent.

The formulations with respect to the optimization of MPC are presented as follows.

5.4.1. Objective Function

Three main objectives constitute the object function in this work:

1) Minimize real-time regulation violation ()
2) Minimize summation of predicted regulation violation (∑ ௜)
3) Minimize summation of storage level deviation (∑௜)

The real-time regulation violation, predicted regulation violation and storage level deviation are penalized with variable weighting factors, as in objective function for minimization:

illustration not visible in this excerpt

Where

— The real-time regulation violation

V_i— The predicted regulation violation

D_k— The storage level deviation

C_k— The targeted power level deviation of CRAC system

n— The length of receding finite-horizon of Model Predictive Controller

,, — Penalty parameters for regulation violation, storage level deviation and targeted level deviation of CRAC system respectively

With the penalty parameter , it is ensured that the control action of UPS and CRAC system is a secondary choice while there is no penalty for the flexible regulation for server rack system. The preference for regulation provision is indicated by the penalty parameter[46]. I.e. the flexibility provided by UPS is inferior to other equipment in DC, since it must be averted that only UPS system is charged or discharged to supply the operational flexibility to the grid. Otherwise, there can be little space for the energy storage contributing to the DR provision in the future time steps. Further, the temperature in DC will not fluctuate intensively, if the CRAC system power need is kept at the targeted level.

The impact of penalty parameters will be discussed in detail in section 7 for a case study.

5.4.2. Equality constraints

The objective function is minimized subject to the following equality constraints:

5.4.2.1. DC power consumption

illustration not visible in this excerpt

Where

P_DC,t— Total power consumption of entire DC at time step t (kW)

P_IT,t— Power consumption of Server Rack System at time step t (kW)

P_UPS,t— Power consumption of UPS system at time step t (kW)

P_CRAC,t— Power consumption of CRAC at time step t (kW)

As mentioned in Section 2, this thesis focuses on the combined operation- scheduling of the IT equipment, refrigeration system and UPS system enabling DC to participate in DR program, as these three parts lead to 90% of total DC power demand.

5.4.2.2. Real-time regulation violation

According to the real-time regulation signal obtained from grid operator, the real-time regulation violation is stated as:

illustration not visible in this excerpt

Where

— Real-time regulation violation at time step t (kW)

P_DC,t— Total power consumption of entire DC at time step t (kW) — Regulation baseline (kW)

[...]