Wind Turbine Yaw Offset Control based on Reinforcement Learning

TECHNICAL FIELD

Various embodiments of the invention relate to methods and devices for controlling yaw offsets of one or more wind tur- bines by utilizing reinforcement learning.

BACKGROUND

Wind turbines have been used for many years as a more envi- ronmental friendly energy source. More and more onshore and offshore wind turbine farms have been constructed all over the world. Currently, wind farms are operated in such a way that each wind turbine of a wind farm is operated at its in- dividual optimum operation point according to the Betz limit.

Such techniques of operating each wind turbine of a wind farm at its individual optimum operation point face certain re- strictions and drawbacks. The interaction between nearby wind turbines in a wind farm modifies the power output, compared to power outputs of isolated wind turbines. For example, along the wind direction, an upstream wind turbine generates a wake which influences a downstream wind turbine. Such ef- fects can decrease the overall performance (e.g., total power output) of the wind farm.

When a wind turbine extracts energy from the wind, the speed of wind is reduced in the area of the wake and the downstream wind turbines receive a reduced wind speed. In a large wind farm, where the wind turbines are arranged in rows and col- umns, the downstream wind turbines are usually operating in the area of the wake of the upstream wind turbines. This re- sults in reduced power output of downstream wind turbines.

The operation of each wind turbine of a wind farm at its op- timum level may not result in the maximum total/aggregate power output from the wind farm.

One of the most commonly used wake model to study the effects of the wake in a wind farm is the Park and Jensen model. Ac- cording to the Park and Jensen model, rotating the nacelle of a wind turbine (yawing) changes the direction of the wake be- hind the wind turbine and results in a yaw offset. The yaw offset can affect the power production of an individual wind turbine. Typically, the larger the yaw offset of a wind tur- bine, the less the power production of the wind turbine. The yaw offset is the angle offset between the axis of the rotor plane and the incoming wind direction, around a rotational axis perpendicular to the horizontal plane. If the wind tur- bine is operated in such a way that the rotor plane faces the wind perpendicularly, then the yaw offset is zero degree.

Wind turbines are typically operated in a way such that they face the wind perpendicular to the rotor plane. This is a state of zero yaw offset. Any rotation of the wind turbine from this position would change the yaw offset. Changing the yaw offset from its zero degree position deflects the wake behind the wind turbine and further influences the power out- put of a wind turbine.

Although influences of the wake of the upstream wind turbine upon the downstream wind turbine can be mitigated or even eliminated by setting the yaw offset of the upstream wind turbine, the power production of the upstream wind turbine will decrease simultaneously. Thus, it is challenging and difficult to achieve an optimal (maximal) total power output of a wind farm by adjusting yaw offsets of some or all wind turbines in a wind turbine farm.

SUMMARY

Therefore, there is a need for advanced techniques of con- trolling yaw offsets of wind turbines. This need is met by the features of the independent claims. The features of the dependent claims define embodiments.

A method of controlling a yaw offset of a first wind turbine (an upstream wind turbine) comprises receiving data indica- tive of a current state of the first wind turbine and of a current state of a second wind turbine (a downstream wind turbine) adjacent to the first wind turbine downstream along the wind direction, determining one or more controlling ac- tions associated with the yaw offset of the first wind tur- bine based on the current state of the first wind turbine, the current state of the second wind turbine, and a rein- forcement learning algorithm, and applying the determined one or more controlling actions to the first wind turbine.

A computer program or a computer-program product or a comput- er-readable storage medium include program code that can be executed by at least one processor. Executing the program code causes the at least one processor to perform a method of controlling a yaw offset of a first wind turbine (an upstream wind turbine). The method comprises receiving data indicative of a current state of the first wind turbine and of a current state of a second wind turbine (a downstream wind turbine) adjacent to the first wind turbine downstream along the wind direction. The method also includes determining one or more controlling actions associated with the yaw offset of the first wind turbine based on the current state of the first wind turbine, the current state of the second wind turbine, and a reinforcement learning algorithm, and applying the de- termined one or more controlling actions to the first wind turbine .

A device for controlling a yaw offset of a first wind turbine comprises one or more processors. The one or more processors are configured to receive data indicative of a current state of the first wind turbine and of a current state of a second wind turbine adjacent to the first wind turbine downstream, determine one or more controlling actions associated with the yaw offset of the first wind turbine based on the current state of the first wind turbine, the current state of the second wind turbine, and a reinforcement learning algorithm, and apply the determined one or more controlling actions to the first wind turbine.

A method of controlling yaw offsets of a plurality of wind turbines being arranged in sequence along a wind direction, comprises receiving data indicative of a respective current state for each wind turbine of the plurality of wind tur- bines, determining controlling actions associated with the yaw offsets of the plurality of wind turbines based on the current states of the wind turbines of the plurality of wind turbines and a reinforcement learning algorithm, and applying the determined controlling actions to the wind turbines of the plurality of wind turbines.

A computer program or a computer-program product or a comput- er-readable storage medium include program code that can be executed by at least one processor. Executing the program code causes the at least one processor to perform a method of controlling yaw offsets of a plurality of wind turbines being arranged in sequence along a wind direction. The method com- prises receiving data indicative of a respective current state for each wind turbine of the plurality of wind tur- bines, determining controlling actions associated with the yaw offsets of the plurality of wind turbines based on the current states of the wind turbines of the plurality of wind turbines and a reinforcement learning algorithm, and applying the determined controlling actions to the wind turbines of the plurality of wind turbines.

A device for controlling yaw offsets of a plurality of wind turbines being arranged in sequence along a wind direction comprises one or more processors. The one or more processors are configured to receive data indicative of a respective current state for each wind turbine of the plurality of wind turbines, determine controlling actions associated with the yaw offsets of the plurality of wind turbines based on the current states of the wind turbines of the plurality of wind turbines and a reinforcement learning algorithm, and apply the controlling actions to the wind turbines of the plurality of wind turbines.

It is to be understood that the features mentioned above and those yet to be explained below may be used not only in the respective combinations indicated, but also in other combina- tions or in isolation without departing from the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates a part of a wind farm which includes 4 rows and 4 columns of wind turbines according to various examples.

FIGs. 2 - 4 schematically illustrate three exemplary scenari- os of three successive time steps that indicate influence of yaw offsets of an upstream wind turbine on total power out- puts of the upstream wind turbine and of a downstream wind turbine.

FIG. 5 schematically illustrates an exemplary wind turbine.

FIG. 6 schematically illustrates aspects with respect to a reinforcement learning algorithm 700.

FIG. 7 is a backup diagram for a policy p.

FIG. 8 is an example state tree to schematically illustrate core ideas of reinforcement learning algorithms that are uti- lized in various embodiments of this application. FIG. 9 is a flow diagram of an example method for controlling a yaw offset of an upstream wind turbine of a wind turbine pair.

Fig. 10 is a flow diagram of an example method for control- ling yaw offsets of a plurality of wind turbines being ar- ranged in sequence along a wind direction.

FIG. 11 schematically illustrates aspects with respect to a device for controlling a yaw offset of an upstream wind tur- bine of a wind turbine pair.

FIG. 12 schematically illustrates aspects with respect to a device for controlling yaw offsets of a plurality of wind turbines being arranged in sequence along a wind direction.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following, embodiments of the invention will be de- scribed in detail with reference to the accompanying draw- ings. It is to be understood that the following description of embodiments is not to be taken in a limiting sense. The scope of the invention is not intended to be limited by the embodiments described hereinafter or by the drawings, which are taken to be illustrative only.

The drawings are to be regarded as being schematic represen- tations and elements illustrated in the drawings, which are not necessarily shown to scale. Rather, the various elements are represented such that their function and general purpose become apparent to a person skilled in the art. Any connec- tion or coupling between functional blocks, devices, compo- nents, or other physical or functional units shown in the drawings or described herein may also be implemented by an indirect connection or coupling. A coupling between compo- nents may also be established over a wireless connection. Functional blocks may be implemented in hardware, firmware, software, or a combination thereof. Various techniques described herein generally relate to con- trolling wind turbine yaw offsets. More specifically, yaw offsets of one or more wind turbines in a wind farm can be controlled by applying one or more controlling actions, e.g., when the one or more wind turbines are starting up. Thus, wakes generated by the one or more wind turbines can be steered in a controlled manner and influences of the wakes on downstream wind turbines can be reduced or mitigated. Accord- ingly, the wind farm can provide a higher aggregate power output or even maximum power output.

Considering a wind farm 200 including a plurality of rows and a plurality of columns of wind turbines, Fig. 1 schematically illustrates a part of the wind turbine farm 200. Wind is blowing with a direction indicated by arrows 205. The part of the wind turbine farm 200 includes 4 rows and 4 columns of wind turbines 201a - 201p. Each wind turbine of the wind tur- bine farm 200 generates a wake. For example, wind turbines 201e and 201f operate at a state of zero yaw offset, and gen- erate wakes 210 and 211, respectively. Consequently, wind turbines 201f and 201g are positioned in the wakes 210 and 211, respectively, and provide less individual power output. On the other hand, wind turbines 201i and 201j operate at a state of non-zero yaw offset, and generate wakes 212 and 213, respectively. The wakes 212 and 213 do not influence corre- sponding downstream wind turbines 201j and 201k. Thus, wind turbine 201k can provide a higher individual power output than that of wind turbine 201g. However, wind turbine 201i provides a smaller individual power output than that of wind turbine 201e because of the non-zero yaw offset.

Various techniques are based on the finding that it can be difficult to control the yaw offset of the upstream wind tur- bines 201e or 201i to have both upstream wind turbines 201e or 201i and downstream wind turbines 201f or 201j providing a higher power output. For example, FIGs. 2 - 4 schematically illustrate three exem- plary scenarios of three successive time steps that show an influence of yaw offsets 800 of an upstream wind turbine (e.g., the wind turbine 201i of the wind farm 200) on total power outputs of the upstream wind turbine and of a down- stream wind turbine (e.g., the wind turbine 201j of the wind farm 200).

In FIG. 2, the upstream wind turbine 201i has a state with a zero degree yaw offset and generates a wake 212 which direct- ly hits the downstream wind turbine 201j. The upstream wind turbine 201i can generate its own optimal power output of 1,000 kW. On the other hand, the downstream wind turbine 201j can only generate a power output of 300 kW due to influences of the wake 212. In total, the two wind turbines can generate a sum of power output of 1,300 kW.

After setting the yaw offset 800 of the upstream wind turbine 201i to +10 degrees, as shown in FIG. 3, the upstream wind turbine 201i gets into a new state with a +10 degrees yaw offset and generates a new wake 212 which does not hit the downstream wind turbine 201j. The upstream wind turbine 201i can generate a power output of 800 kW, which is less than its optimal power output due to the +10 degrees yaw offset. On the other hand, the downstream wind turbine 201j can generate a power output of 700 kW, because there is no influences of the wake 212 on itself. In total, the two wind turbines can generate a sum of power output of 1,500 kW, which is more than that of FIG. 2.

By further increasing the yaw offset 800 of the upstream wind turbine 201i to +15 degrees, as shown in FIG. 4, the upstream wind turbine 201i assumes a further state with a +15 degrees yaw offset and generates a further wake 212 which also does not hit the downstream wind turbine 201j. The upstream wind turbine 201i can generate a power output of 400 kW, which is less than its previous power output due to the increase of the yaw offset. On the other hand, the downstream wind tur- bine 201j can still generate a power output of 700 kW, be- cause the wake 212 still does not influence the downstream wind turbine 201j. In total, the two wind turbines can gener- ate a sum of power output of 1,100 kW, which is the smallest among the three successive time steps.

Various techniques described herein can be executed to set the yaw offset 800 of the upstream wind turbine 201i to an angle to obtain a state in which the upstream wind turbine 201i and the downstream wind turbine 201j provide a large power output.

FIG. 5 schematically illustrates an exemplary wind turbine 10 which may be any wind turbine in the wind farm 200, such as wind turbines 201a - 201p. The wind turbine 10 includes blades 11 attached to a rotor 12. The rotor 12 is connected to a nacelle 13, mounted on a tower 14. The nacelle 13 houses a generator, coupled with the rotor 12 rotating along with the blades 11. This connection can be via a shaft and gear. The yaw offset is an angle offset between the axis of the ro- tor 12 plane and the incoming wind direction, e.g., the wind direction 205 of FIG. 1.

Various embodiments of this invention provide methods and de- vices for determining one or more controlling actions associ- ated with yaw offsets of one or more wind turbines to maxim- ize total power output of a wind farm like the wind farm 200. The one or more controlling actions can be determined based on certain logic that relies on reinforcement learning.

Various techniques described herein generally relate to rein- forcement learning. Reinforcement learning generally de- scribes a machine-learning process associated with taking an appropriate action (here: how to map one or more environmen- tal parameters indicating states of a first (or upstream) wind turbine and of a second (or downstream) wind turbine ad- jacent to the first wind turbine to one or more controlling actions associated with a yaw offset of the first wind tur- bine) that maximizes a reward (here: various options exist, e.g., total power output of the first wind turbine and of the second wind turbine). Reinforcement learning is generally different from supervised learning: labeled training data is not required; rather, reinforcement learning enables to learn during operation by monitoring the reward.

Data indicative of a current state of the first wind turbine 201i and of a current state of the second wind turbine 201j adjacent to the first wind turbine 201i downstream along the wind direction are received and one or more controlling ac- tions associated with the yaw offset 800 of the first wind turbine 201i are determined using this information. This de- termination is, in particular, based on reinforcement learn- ing.

Various techniques are based on the finding that reference techniques - in which each wind turbine of a wind farm 200 is operated at its individual optimum operation point - face certain restrictions and drawbacks. For instance, the inter- action between nearby wind turbines in the wind turbine farm 200 modifies the power compared to their stand-alone values. Along the wind direction 205, an upstream wind turbine 201i generates a wake 212, which influences a downstream wind tur- bine 201j and further decreases the overall performance (e.g., total power production) of the wind farm 200. It is known that overall performance of the wind farm 200 is non- optimal if the waked turbine 201j is in full-wake of the up- stream wind turbine 201i.

These drawbacks are mitigated by the techniques described herein by using reinforcement learning. Various kinds and types of environmental parameters indicating states of wind turbines described herein can be used in connection with the reinforcement learning. Here, the learning process is perma- nently ongoing and challenges itself. An optimized control of the yaw offset of each wind turbine of the wind turbine farm 200 would increase the performance of each pair of upstream and downstream wind turbines (e.g., 201a&201b, 201b&201c, 201c&201d, 201e&201f, 201o&201p) by steering the wake

(e.g., 210, 211, 212, 213, etc.) away. The techniques de- scribed herein can help to reduce the performance loss of the whole wind farm 200 due to interference of the adjacent wind turbines .

After the one or more controlling actions associated with the yaw offset 800 of the first wind turbine 201i have been de- termined, the determined one or more controlling actions are applied to the first wind turbine 201i. Then, the same method or procedure for controlling yaw offsets 800 can be applied to another wind turbine 201j of an adjacent pair of wind tur- bine pair 201j&201k, e.g., until the yaw offsets 800 of all the wind turbines of the wind farm 200 have been appropriate- ly set by corresponding controlling actions.

FIG. 6 schematically illustrates aspects with respect to a reinforcement learning algorithm 700.

Generally speaking, reinforcement learning is a type of ma- chine learning and it is different from the other machine- learning techniques. Its inherent components are an agent module 703, an environment module 701, a policy module 702, and a reward module 704.

The reinforcement learning algorithm 700 includes an environ- mental module 701. The environmental module 701 is configured to receive data indicative of a current state of the first wind turbine 201i and data indicative of a current state of a second wind turbine 201j adjacent to the first wind turbine 201i downstream.

Examples of data indicating states of the first wind turbine 201i and of the second wind turbine 201j that can be used in the various examples described herein include a wind direc- tion, a wind speed, or a yaw offset. The examples of data al- so may comprise a location of the wake 212, a temperature, rotational speed of the blades 11, a location of a wind tur- bine including longitude and latitude, air pressure, precipi- tation, humidity, local time, a turbulence intensity, a site elevation, a vertical wind shear, or a horizontal wind shear etc.. The location of the wake 212 may be determined by the second wind turbine 201j. The techniques described herein are flexible in that a larger or smaller number of states (or pa- rameters) can be considered.

There are various options available of obtaining the data in- dicative of states of the first wind turbine and of the sec- ond wind turbine. For instance, the data indicative of states can be measured by one or more sensors. Alternatively or ad- ditionally, the data indicative of states can also be simu- lated or generated by one or more computer programs or input by experts. For example, the simulated data indicative of states can be one or more outputs of a function in response to one or more inputs of measurement data obtained from one or more sensors. For instance, experts or engineers can uti- lize weather models to simulate extreme weathers and obtain simulated environmental parameters, such as wind speed, tur- bulence intensity, vertical wind shear, and horizontal wind shear etc.. The function can be a linear function or a non- linear function represented by a neural network.

The reinforcement learning algorithm 700 also includes a pol- icy module 702 that comprises one or more policies. The poli- cy module 702 can determine a policy associated with a map- ping from states of the first wind turbine and of the second wind turbine - obtained from the environmental module 701 - to one or more controlling actions associated with the yaw offset of the first wind turbine to be taken when in those states. The policy may be a simple function or a lookup table that stores all or a part of possible state-action pairs for some examples, whereas it may involve extensive computation such as search process in other examples. In addition, the policy can also be a non-linear function realized by a neural network executing a supervised learning algorithm. The reinforcement learning algorithm 700 also includes an agent module 703. The agent module 703 is configured to apply the one or more controlling actions, based on the input of the policy module 702. The agent module 703 cooperates with the policy module 702, and also with a reward module 704.

The reinforcement learning algorithm 700 also includes the reward module 704. The reward module 704 can determine a re- ward value in a short-term temporal context after performing each determined action - i.e., without any changes in the states. The reward module 704 also can determine a long-term reward value. The objective of the agent module 703 is to op- timize the reward value. The reward value thus defines what are the "good" and "bad" controlling actions to be performed in a given state. The reward value may be used to alter the policy of the policy module 702; if a controlling action de- termined based on the policy is followed by low reward, then the policy may be changed to select some other controlling actions in that situation in the future. In general, the re- ward value may be a value of a stochastic function of the states of the first wind turbine and of the second wind tur- bine and the controlling actions taken.

Next, the function of the reinforcement learning algorithm 700 will be explained.

The agent module 703 interacts with the states of the first wind turbine and of the second wind turbine obtained by the environmental module 701. See action A _t . In this process, the agent module 703 cooperates with the reward module 704 to ob- tain a reward value for determining the appropriate control- ling action. Then the agent module 703 may determine the next controlling action based on previous reward value and in this way define the next controlling action, which will then re- sult in new reward values, and so on. So these terms, agent, environment, state, action (or controlling action), and re- ward, are the most important ones to understand it. In FIG. 6, the subscript t refers to the current timestep and t + 1 to the next timestep. The agent module 703 is in a cur- rent state S _t associated with a current reward value R _t . The agent module 703 then applies the determined controlling ac- tion A _t , and thus also may change the environment (or state). As a result a reward value R _{t+ 1} is gained, that can be either positive or negative, and the state changes, S _t+1 . This is il- lustrated by the feedback branch 710.

This idea of reinforcement learning is different from other Machine Learning techniques. In supervised learning, a target variable is known upfront and an algorithm can learn the pat- tern between dependent and independent variables. In unsuper- vised learning, no target variable is known. In Reinforcement Learning, new knowledge is "learned" with time. In this re- gard reinforcement Learning is a mixture of unsupervised and supervised learning. It comes closest to "human" learning.

Next, a practical example of using reinforcement learning for control of a yaw offset 800 of a first wind turbine is ex- plained. The data indicative of states of the first wind tur- bine and of the second wind turbine are monitored by the en- vironmental module 701. The states of the first wind turbine and of the second wind turbine correspond to a pair of up- stream and downstream wind turbines in a wind turbine farm 200. The environmental module 701 communicates with the first wind turbine's controller and the second wind turbine's con- troller which may have access to all data indicative of states of the first wind turbine and of the second wind tur- bine and provide the data to the environmental module 701.

The agent module 703 can apply controlling actions based on a reference reward value or a predicted reward value of a pre- defined value function obtained from the reward module 704, the current states S _t of the first wind turbine and of the second wind turbine, the measured reward value R _t , and the policy of the policy module 702. The possible actions that the first wind turbine can perform based on the states of the first wind turbine and of the sec- ond wind turbine and rewards, comprise: setting the yaw off- set 800 of the first wind turbine within a range of -45° and +45°.

The states S _t is defined by some or all parameters that have an impact on controlling yaw offset of the first wind tur- bine. Among these are: Measured wind speed (by nacelle ane- mometer); Location of the wake; Turbulence intensity; Site elevation; Outdoor temperature; Vertical or horizontal wind shear (if any of these are available from more complex meas- urement devices or control algorithms).

The reward value is defined as the total produced power out- put of the first wind turbine and of the second wind turbine.

An iterative learning process is performed, in which the first (upstream) wind turbine and the second (downstream) wind turbine for each timestep measure their states, respec- tively, and an action (setting the yaw offset of the first wind turbine within a range of -45° and +45°) is determined for the first wind turbine. Then, a reward is obtained (total power output of the first wind turbine and of the second wind turbine after setting the yaw offset of the first wind tur- bine) and the first wind turbine and the second wind turbine get a new state, respectively, which are defined by their condition measurements, respectively.

The learning process is continuous, because the algorithm runs repeatedly for the complete lifetime of the turbines - this is in contrast to a one-time parameter definition up- front.

With this setup, the algorithm 700 learns to adapt to its am- bient conditions and can achieve an optimal total power out- put of the first wind turbine and of the second wind turbine by setting the yaw offset of the first wind turbine. The learning process goes on, so the first wind turbine and the second wind turbine learn to adapt to new states.

Next, details with respect to the policy of the policy module 702 are explained in connection with FIG. 7.

Fig. 7 is a backup diagram for a policy p, by which a look- ahead from a state s to its possible successor states S ^. is illustrated. Each open circle represents a state and each solid circle represents a state-action pair (s,a). Starting from state s, the root node at the top, the agent module 703 could take any of some set of actions - three are shown in the diagram - based on its policy p. From each of these, the environment module 701 could respond with one of several next state, S ^. (two are shown in the figure), along with a reward, r, depending on its dynamics given by a function p.

The value function specifies what is good in a long run. Roughly speaking, the value of a future state is the predict- ed total amount of reward that the agent module 703 can accu- mulated over the future, starting from the current state, by selecting a chain of actions successively. Whereas the reward determines the immediate desirability of states, the value of value function indicates the long-term desirability of the chain of states after taking into account the states that are likely to follow and the rewards available in those states. For example, a state might always yield a low reward value, but still have a high value of the value function, because it is regularly followed by other states that yield high reward values.

The model for representing the environment considered by the policy module 702 is something which mimics behaviors of the environment, or more generally, which allows inferences to be made about how the environment will behave. For instance, the model might infer the next state and next reward given a state and action. In various embodiments, solving a reinforcement learning task means, roughly, finding a policy that achieves a high reward value, e.g., immediately for fixed states or over a long run for varying states. An optimal policy is defined to be better than or equal to any other policies if its immediate reward value or the value of the value function is greater than or equal to that of any other policies for all states.

FIG. 8 is an example state tree to schematically illustrate aspects of reinforcement learning algorithms that are uti- lized in various embodiments of this application. This state tree includes a plurality of nodes (s _t , s _t+1 , ......, s _t+k) , and a plurality of edges (a _t , a _t+1 , ......, a _t+k-1 ).S _t indicates a current state at time or step t. s _t+1 , s _t+2 , s _t+k indicate three future states at step or time t+ 1, t+ 2, and t+ k, respectively. a _t , a _t+1 , and a _t+k-1 indicate actions taken at state s _t , s _t+1 , and s _t+k-1 , respectively.

Each state of s _t , s _t+1 , and s _t+k may be a member of a sub- set or complete set of an state space S = {S ₁ ,S ₂ ,...,S _n-1 ,S _n }, in which n may be finite or infinite. Each member of the state space S represents a state which has been or would be encoun- tered by the agent module 703. Each action of a _t , a _t+1 , and a _t+k-1 may be selected from a subset or complete set of an action space A = {a ₁ ,a ₂ ,···,a _m }, in which m may be finite or in- finite. Each member of the action space represents an action that has been or would be performed by the agent module 703.

There are various options available for determining the one or more controlling actions. Two options that can be applied are (i) action-reward methods and (ii) action-value methods.

First (i) action-reward methods will be explained.

Each action may be associated with a reference reward value given that that respective action is taken at a respective state. For example, r _t is the reference reward value given that a _t is taken at s _t , and r _t may be determined by a function of a _t and s _t , e.g., r _t = ƒ(s _t ,a _t ), which may comprise a linear function or a neural network. Thus, different combinations of (s _t ,a _t) result in different reference reward values and dif- ferent actions taken at s _t correspond to different reference reward values. For instance, r _t = ƒ(s ₃ ,a ₈ ), if a ₈ is selected as the action a _t at s _t = s ₃ .

For instance, the neural network for determining the refer- ence reward value could be trained based on training data available based on historical site measurements of the state. The neural network could be a convolutional neural network.

At the current state s _t , in some cases, one action, e.g., a _t , may be determined based on a predefined set of actions, e.g., {a ₁ ,a ₃ ,a ₈ ,···a _m-1 }, which is a subset of the action space A, by taking the respective reference reward value, e.g., r _t , into account. For example, the action a _t may be determined as the one with the maximum reference reward value in the predefined set of actions. Alternatively or additionally, the action a _t may be determined based on a predefined set of actions, e.g., {a ₁ ,a ₃ ,a ₈ ,···a _m-1 }, using an arbitrary selection (i.e., using a random contribution).

In some other cases, at the current state s _t , more actions, e.g., a _t and a _t+1 , may be determined based on more predefined sets of actions, e.g., {a ₁ ,a ₃ ,a ₈ ,···a _m-1 } and {a ₂ ,a _B ,a ₁₆ ,...a _m-10 }, respectively, by taking the respective reference reward val- ues, e.g., r _t and r _t+1 , into account. For example, both a _t and a _t+1 may be determined as the one with the maximum reference reward value in the predefined set of actions, respectively. Alternatively or additionally, the actions a _t and a _t+1 may be determined based on predefined set of actions, e.g., {a ₁ ,a ₃ ,a ₈ ,···a _m-1 } and {a ₂ ,a ₅ ,a ₁₆ ,···a _m-10 }, using an arbitrary selec- tion (i.e., using a random contribution). In addition, the more predefined set of actions may be the same or different.

After applying the determined one or more actions, the rein- forcement learning algorithms will receive measurement data associated with a current reward value, which is a real re- ward value (i.e., not a predefined reference reward value). For instance, this measurement could be a power measurement of a power output of the first wind turbine's generator and of the second wind turbine's generator. For example, after applying a _t or a _t and a _t+1 , the reinforcement learning algo- rithms will receive the current reward value at time (or step) t+ 1 or t+ 2, respectively. Then, the reinforcement learning algorithms will perform a comparison between the reference reward value and the current reward value. After- wards, the reinforcement learning algorithms are trained based on the comparison. As will be appreciated, the rein- forcement learning algorithm can thus be trained without re- quiring predefined labeled training data (as would be the case, e.g., for supervised learning). Rather, during opera- tion of the wind turbines, the training can be implemented using the comparison between the reference reward value (an- ticipated in the model) and the actual reward value indicated by the measurement data.

Next, (ii) action-value methods will be explained for deter- mining the appropriate controlling action.

In some embodiments, at the current state s _t , the reinforce- ment learning algorithms may predict a reward value for a fu- ture point in time, e.g., time t+ 1 or t+ k, for k > 1, based on a predefined value function of the reinforcement learning algorithms. The value function may comprise a linear function or a neural network. Thus, a predicted reward value is ob- tained.

For instance, the neural network could be trained based on training data available based on historical site measurements of the states. The neural network could be a convolutional neural network.

Then, one action a _t or a chain of actions from a _t to a _t+k-1 are determined based on the corresponding set of actions, respec- tively, by taking the predicted reward value into account. For example, the action a _t or the chain of actions from a _t to a _t+k-1 may be determined as the one or the chain with the max- imum predicted reward value.

After applying the determined one action a _t or chain of ac- tions from a _t to a _t+k-1, the reinforcement learning algorithms will receive a measurement data associated with a current ag- gregate reward value. For example, after applying a _t or the chain of actions from a _t to a _t+k-1, the reinforcement learning algorithms will receive the current reward value at time (or step) t+ 1 or t+k, respectively. Then, the reinforcement learning algorithms will perform a comparison between the predicted reward value and the current aggregate reward val- ue. Afterwards, the reinforcement learning algorithms are trained based on the comparison.

In various embodiments, the reinforcement learning algorithms may be pre-trained by using prior information or knowledge. The reinforcement learning algorithms may have a model to mimic behaviors of the environment.

By using reinforcement learning, appropriate controlling ac- tions can be taken for the operational control of a yaw off- set of an upstream wind turbine, without requiring exemplary supervision or complete models of the environment. Reinforce- ment learning algorithms can also make use of as many envi- ronmental parameters indicating states of wind turbines as possible to precisely select actions. In addition, the learn- ing process of the reinforcement learning algorithms contin- ues during normal operation of the wind turbines, therefore, the agent module 703 can learn to adapt to new states. Thus, optimal short-term or long-term goals can be achieved by ap- plying the reinforcement learning algorithms.

Referring to FIG. 1 again, the yaw offset 800 of each wind turbine of the wind turbine farm 200 can be controlled by utilizing the above-mentioned reinforcement learning algo- rithms. In some cases, all the wind turbines in the wind farm 200 are divided into pairs including two adjacent wind turbines, along the wind direction, such as wind turbine pairs 201a&201b, 201b&201c, and 201c&201d in Fig. 1. Then, the yaw offset of the upstream wind turbine of each pair is con- trolled based on current states of both wind turbines of the pair and a reinforcement learning algorithm 700. Finally, yaw offsets of all the wind turbines of the wind turbine farm 200 are optimized by performing a method based on reinforcement learning (e.g., reinforcement learning algorithm 700) itera- tively.

Fig. 9 is a flow diagram of an example method 400 for con- trolling the yaw offset of the upstream wind turbine of each wind turbine pair of the wind turbine farm 200. For conven- ience, the method will be explained as being performed by a system of one or more computers located in one or more loca- tions. For example, a reinforcement learning system running one or more reinforcement learning algorithms 700 mentioned above, can perform the method 400. Other devices including a control circuitry could implement the method of FIG. 9. The method may be executed by a processor upon loading program code, e.g., provided by a computer-readable storage medium, a computer program or a computer program product.

The method of FIG. 9 can employ one or more reinforcement learning algorithms 700 as described above in connection with FIGs. 1 - 8.

At optional step 401, the reinforcement learning algorithm 700 is pre-trained by using tuples of the following data: wind directions, wind speeds, yaw offsets, and sums of power outputs, wherein all the data are obtained from any two wind turbines that are adjacent to each other along the wind di- rections or expert knowledge, before implementing the rein- forcement learning algorithm to control a yaw offset of a first wind turbine (an upstream wind turbine of a wind tur- bine pair).

For example, by referring to Fig. 1, a general reinforcement learning algorithm for all or part of the wind turbines of the wind turbine farm can be pre-trained by using the data obtained from any wind turbine pair of the wind turbine farm, such as, 201a&201b, 201b&201c, 201c&201d, 201e&201f, ......,

201o&201p. Alternatively or additionally, a specific rein- forcement learning algorithm for each wind turbine of the wind turbine farm 200 can also be pre-trained by using the data obtained from any wind turbine pair of the wind farm, such as, 201a&201b, 201b&201c, 201c&201d, 201e&201f, ......,

201o&201p .

Then, for the following steps the (pre-trained) reinforcement learning algorithm is applied.

At optional step 481, it is checked whether a further pair of wind turbines remains to be subject to control. At the ini- tial iteration 489 of step 481, this will be the case.

Accordingly, step 482 is executed; here, a current pair of wind turbines is selected from all wind turbine pairs that remain to be subject to control. This is the active pair of wind turbines; and these wind turbines are denoted first wind turbine and second wind turbine. The second wind turbine is downstream of the first wind turbine in the wind direction.

At step 402, data indicative of a current state of the first wind turbine and of a current state of a second wind turbine of the active pair is received.

The current states of the first wind turbine and of the sec- ond wind turbine specify a value of at least one of the fol- lowing parameters: a wind direction, a wind speed, or the yaw offset. The current states also may specify a location of a wake generated by the first wind turbine. The one or more controlling actions comprise setting the yaw offset within a range of -45° and +45°.

At step 404, one or more controlling actions associated with the yaw offset of the first wind turbine based on the current state of the first wind turbine, the current state of the second wind turbine, and a reinforcement learning algorithm, e.g., the reinforcement learning algorithm 700 of FIG. 6, are determined .

The reinforcement learning algorithm may utilize any one of the above-mentioned action-reward methods or the action-value methods. In addition, the reinforcement learning algorithm also may utilize other existing reinforcement learning algo- rithms, such as, Q-Learning, Deep Q-Learning, Double Deep Q- Learning, Dynamic Programming (DP), Temporal-Difference (TD) Learning, Monte Carlo Methods, Policy Gradient Methods, On- policy Prediction with Approximation, etc.

When determining the one or more controlling actions associ- ated with the yaw offset of the first wind turbine, it is possible to take into account one or more reward values. For example, reference reward values or predicted reward values of a predefined value function can be used. Reward values can, in particular, include the sum of power outputs of both the first wind turbine and the second wind turbine.

In some cases, the one or more controlling actions are deter- mined based on a predefined set of controlling actions (e.g., by an appropriate selection process from the predefined set), wherein each of the controlling actions in the predefined set of controlling actions is associated with a reference reward value given that that respective controlling action is deter- mined at the current states of the first wind turbine and of the second wind turbine.

A respective reference reward for each controlling action in the predefined set of controlling actions may be respectively generated by processing the current states of the first wind turbine and of the second wind turbine together with the cor- responding controlling action using an action reward neural network. The action reward neural network may be trained by supervised learning using labeled data obtained from experi- ences. Alternatively, a database which stores arrays or ta- bles representing mappings from (current states of the first wind turbine and of the second wind turbine, action) pairs to reference rewards may be searched.

In some implementations, the action reward neural network is configured to receive as input current states of the first wind turbine and of the second wind turbine together with a controlling action and to process the current states of the first wind turbine and of the second wind turbine together with the controlling action to generate a respective refer- ence reward for the action.

The one or more controlling actions are determined based on the predefined set of controlling actions taking the respec- tive reference reward value into account. For example, the one or more controlling actions may be determined as one or more controlling actions with the maximum reference reward or maximum sum of reference rewards. Sometimes, to balance ex- ploration and exploitation of the reinforcement learning al- gorithm, the one or more controlling actions are determined based on the predefined set of controlling actions using an arbitrary selection. For instance, it would be possible to randomly or at least partly randomly select (arbitrary selec- tion) one or more random controlling action from the prede- fined set of controlling actions with a probability ϵ (0< ϵ <

1) and select one or more controlling actions with the maxi- mum reference reward or maximum sum of reference reward with a probability 1 — ϵ . It is preferred to set 0< ϵ<0.1.

In some other cases, a reward value for a future point in time may be predicted, after multiple iterations of said ap- plying the determined one or more controlling actions based on a predefined value function of the reinforcement learning algorithm, wherein the one or more controlling actions are determined based on the predefined set of controlling actions taking the predicted reward value into account.

The value function is used to predict a long-term reward, such as, several hours, a whole day, a week and so on. On this timescale, typically the environmental conditions change. On the other hand, the above-mentioned reference re- ward is used to estimate a short-term reward, such as, one minute or several minutes where environmental conditions are considered to be rather stable.

For example, it would be possible to predict a respective re- ward value for a future point in time for each possible chain of controlling actions selected from the predefined set of controlling actions, respectively, by processing the current states of the first wind turbine and of the second wind tur- bine of the current pair, together with the corresponding chain of controlling actions using an action value neural network. The action value neural network may be trained by supervised learning using labeled data obtained from experi- ences. Alternatively, the value function may be a predefined linear function.

In some implementations, the action value neural network is configured to receive as input current states of the first wind turbine and of the second wind turbine together with a chain of controlling actions and to process the current states of the first wind turbine and of the second wind tur- bine together with the chain of controlling actions to gener- ate a respective reward value for a future point for the pos- sible chains of controlling actions.

After generating reward values of all or part of possible chains of controlling actions starting from the current state and ending at the future point, it is possible to determine the chain of controlling actions with the highest predicted reward value as the one or more controlling actions to be performed .

At step 406, the determined one or more controlling actions are applied to the first wind turbine. After the determina- tion of the one or more controlling actions by the reinforce- ment learning algorithm, it is possible to apply the one or more controlling actions to control the yaw offset of the first wind turbine.

In some cases, for the action-reward methods, upon applying the determined one or more controlling actions, measurement data associated with a current reward value is received, and a comparison between the reference reward value and the cur- rent reward value is performed, and the reinforcement learn- ing algorithm is trained based on the comparison.

In some other cases, for the action-value methods, upon the multiple iterations of said applying the determined one or more controlling actions, it is possible to receive a meas- urement associated with a current aggregate reward value across the multiple iterations, perform a comparison between the predicted reward value and the current aggregate reward value, and train the reinforcement learning algorithm based on the comparison.

In addition, either the reference reward or the predefined value function comprises a sum of power outputs of both the first wind turbine and the second wind turbine or other pa- rameters for assessing the performance of both the wind tur- bines.

Next, a further iteration 489 of step 481 is performed: In various embodiments, a general reinforcement learning algo- rithm based on the method 400 may be executed for controlling yaw offsets of all the wind turbines in the wind turbine farm 200. This can be achieved by toggling through the various pairs of adjacent wind turbines. There are various options available for implementing the se- lection logic at step 482. For instance, after finishing con- trolling the yaw offset of the first wind turbine of an ini- tial active pair, the next pair selected at the next itera- tion 489 of step 482 may include the second wind turbine of the initial active pair as an upstream wind turbine of a fur- ther downstream wind turbine. This means: the second wind turbine of the initial active pair of a first iteration 489 is the first wind turbine of the subsequent active pair of a subsequent, second iteration 489. Such scenario corresponds to moving through pairs of the wind turbines of the wind farm along the wind direction.

The method 400 for controlling the yaw offset of a wind tur- bine can learn by the reinforcement learning algorithms from direct interaction with the environment, without requiring exemplary supervision or complete models of the environment. The reinforcement learning algorithms can also make use of various environmental parameters to precisely select control- ling actions of the yaw offset of the wind turbine. In addi- tion, the learning process of the reinforcement learning al- gorithms continues during normal operation of the wind tur- bine, therefore, the reinforcement learning algorithms can learn to adapt to new states. Thus, optimal short-term or long-term goals, such as, a sum of power outputs of both the first wind turbine and the second wind turbine, can be achieved by applying the reinforcement learning algorithms to control the wind turbine.

In addition, the method 400 is easy to implement and does not need high-performance computing resources or devices, because the method 400 takes into account pairs of wind turbines when determining the controlling actions associated with the yaw offsets.

While method 400 has been explained in connection with a sce- nario in which at step 482 pairs of wind turbines are select- ed, it would generally be possible to select, at step 482 subgroups of all wind turbines of the wind farm that include more than two wind turbines, e.g., three or four wind tur- bines. The wind turbines of the subgroup can be arranged in sequence along the wind direction. Here, the controlling ac- tion of the yaw offset of a first wind turbine - upstream along the wind direction - can be determined depending on the current states of the remaining wind turbines in the sub- group.

As will be appreciated from the above, method 400 can be la- beled as a bottom-up approach for setting the yaw offset.

This is because there are multiple iterations 489, each iter- ation 489 being associated with a respective wind turbine. In other examples, a top-down approach would be possible where the reinforcement learning algorithm is used to concurrently set the yaw offsets of multiple wind turbines. Such a scenar- io is explained in connection with FIG. 10.

Fig. 10 is a flow diagram of an example method 500 for con- trolling yaw offsets of a plurality of wind turbines. The plurality of wind turbines are arranged in sequence along a wind direction 205. The plurality of wind turbines can form a subgroup of all wind turbines of a wind farm. For conven- ience, the method 500 will be explained as being performed by a system of one or more computers located in one or more lo- cations. For example, a reinforcement learning system running one or more reinforcement learning algorithms disclosed in this application, can perform the method 500. Other devices including a control circuitry could implement the method of FIG. 10. The method may be executed by a processor upon load- ing program code, e.g., provided by a computer-readable stor- age medium, a computer program or a computer program product.

The method of FIG. 10 can employ one or more reinforcement learning algorithms 700 as described above in connection with the preceding FIGs 1 - 8. At optional step 501, the reinforcement learning algorithm is pre-trained by using tuples of the following data: wind di- rections, wind speeds, yaw offsets, and sums of power out- puts, wherein all the data are obtained from any plurality of wind turbines that are arranged in sequence along the wind directions or expert knowledge. The data may comprise loca- tions of wakes generated by each wind turbine of the plurali- ty of wind turbines.

For example, by referring to Fig. 1, a general reinforcement learning algorithm for all or part of the wind turbines of the wind turbine farm 200 can be trained by using the data obtained from any wind turbine sequence of the wind turbine farm 200, such as, wind turbine sequences starting with 201a, 201e, 201i, and 201m, respectively. Alternatively or addi- tionally, a specific reinforcement learning algorithm for each wind turbine sequence can also be trained by using the data obtained from any wind turbine sequence of the wind tur- bine farm 200.

At step 502, for each wind turbine of the plurality of wind turbines, data indicative of a respective current state is received. The plurality of wind turbines may be arranged in sequence of along a wind direction 205, of the wind farm 200. The respective current state of each wind turbine of the plu- rality of wind turbines specifies one or more of the follow- ing: a wind direction, a wind speed, or the yaw offset. The controlling actions comprise setting the yaw offset within a range of -45° and +45°. The respective current state may specify locations of wakes generated by each wind turbine of the plurality of wind turbines as well.

At step 504, controlling actions associated with the yaw off- sets of the plurality of wind turbines based on the current states of the wind turbines of the plurality of wind turbines and a reinforcement learning algorithm, e.g., the reinforce- ment learning algorithm 700 of FIG. 5, are determined. This occurs in a combined manner. This means that the reinforce- ment learning algorithm is trained to determine, in one run, controlling actions associated with multiple yaw offsets for all wind turbines of the plurality of wind turbines. This may be labeled a top-down approach.

The reinforcement learning algorithm may utilize any one of the above-mentioned action-reward methods or the action-value methods. In addition, the reinforcement learning algorithm also may utilize other exist reinforcement learning algo- rithms, such as, Q-Learning, Deep Q-Learning, Double Deep Q- Learning, Dynamic Programming (DP), Temporal-Difference (TD) Learning, Monte Carlo Methods, Policy Gradient Methods, On- policy Prediction with Approximation, etc.

When determining the controlling actions associated with the yaw offsets of the plurality of wind turbines, it is possible to take into account one or more reward values. For example, reference reward values or predicted reward values of a pre- defined value function can be used. Reward values can, in particular, comprise a sum of power outputs of all the wind turbines of the plurality of wind turbines.

In some cases, the controlling actions are determined based on a predefined set of controlling actions (e.g., by an ap- propriate selection process) from the predefined set, wherein each of the controlling actions in the predefined set of con- trolling actions is associated with a reference reward value given that that respective controlling action is determined at the current state.

An associated reference reward for each controlling action in the predefined set of controlling actions may be respectively generated by processing the current states of the wind tur- bines of the plurality of wind turbines together with the corresponding controlling actions of all the wind turbines of the plurality of wind turbines using an action reward neural network. The action reward neural network may be trained by supervised learning using labeled data obtained from experi- ences. Alternatively, a database which stores arrays or ta- bles representing mappings from (current states of the wind turbines of the plurality of wind turbines, actions) pairs to reference rewards may be searched.

In some implementations, the action reward neural network is configured to receive as input the current states of the wind turbines of the plurality of wind turbines together with cor- responding controlling actions of all wind turbines of the plurality of wind turbines and to process the current states of the wind turbines of the plurality of wind turbines to- gether with corresponding controlling actions of all wind turbines of the plurality of wind turbines to generate a re- spective reference reward for the actions.

The controlling actions are determined based on the prede- fined set of controlling actions taking the respective refer- ence reward value into account. For example, the controlling actions may be determined by selecting the controlling ac- tions with the maximum reference reward or maximum sum of reference rewards. Sometimes, to balance exploration and ex- ploitation of the reinforcement learning algorithm, the con- trolling actions are determined based on the predefined set of controlling actions using an arbitrary selection. For in- stance, it would be possible to randomly or at least partly randomly select (arbitrary selection) controlling actions from the predefined set of controlling actions with a proba- bility ϵ (0< ϵ < 1) and select controlling actions with the maximum reference reward or maximum sum of reference reward with a probability 1 — ϵ . It is preferred to set 0< ϵ<0.1.

In some other cases, a reward value for a future point in time may be predicted based on a predefined value function of the reinforcement learning algorithm, wherein the controlling actions are determined based on a predefined set of control- ling actions taking the predicted reward value into account. The value function is used to predict a long-term reward, such as, several hours, a whole day, a week and so on. On this timescale, typically the environmental conditions change. On the other hand, the above-mentioned reference re- ward is used to estimate a short-term reward, such as, one minute or several minutes where environmental conditions are considered to be rather stable. The value function may com- prise a linear function or a neural network

For example, it would be possible to predict a respective re- ward value for a future point in time for each possible chain of controlling actions selected from the predefined set of controlling actions, respectively, by processing the current states of the wind turbines of the plurality of wind turbines together with the corresponding controlling actions of all the wind turbines of the plurality of wind turbines using an action value neural network. The action value neural network may be trained by supervised learning using labeled data ob- tained from experiences. Alternatively, the value function may be a predefined linear function.

In some implementations, the action value neural network is configured to receive as input current states of the wind turbines of the plurality of wind turbines together with the corresponding controlling actions of all the wind turbines of the plurality of wind turbines and to process the current states of the wind turbines of the plurality of wind turbines together with the corresponding controlling actions of all the wind turbines of the plurality of wind turbines to gener- ate a respective reward value for a future point for the chain of controlling actions.

After generating reward values for a future point of all or part of possible chains of controlling actions starting from the current state and ending at the future point, it is pos- sible to determine the chain of controlling actions with the highest predicted reward value as the controlling actions as- sociated with the yaw offsets of the plurality of wind tur- bines to be performed.

At step 506, the determined controlling actions are applied to the wind turbines of the plurality of wind turbines, re- spectively. After the determination of the controlling ac- tions associated with the yaw offsets of the plurality of wind turbines by the reinforcement learning algorithm, it is possible to apply the determined controlling actions to con- trol the yaw offsets of the plurality of wind turbines.

In some cases, for the action-reward methods, upon applying the determined controlling actions, measurement data associ- ated with a current reward value is received, and a compari- son between the reference reward value and the current reward value is performed, and the reinforcement learning algorithm is trained based on the comparison.

In some other cases, for the action-value methods, upon ap- plying the determined controlling actions, it is possible to receive a measurement associated with a current aggregate re- ward value, perform a comparison between the predicted reward value and the current aggregate reward value, and train the reinforcement learning algorithm based on the comparison.

In addition, either the reference reward or the predefined value function comprises a sum of power outputs of all the wind turbines of the plurality of wind turbines, or other pa- rameters for assessing the performance of all the wind tur- bines.

In various embodiments, a general reinforcement learning al- gorithm based on the method 500 may be executed for control- ling yaw offsets of all the wind turbines in a wind turbine farm, such as the wind turbine farm 200. Alternatively, each sequence of the wind turbines of a wind farm may have its own reinforcement learning algorithm for controlling yaw offsets of all the wind turbines of the respective sequence based on the method 500.

The method 500 for controlling yaw offsets of all wind tur- bines arranged in a sequence along the wind direction can learn by the reinforcement learning algorithms from direct interaction with the environment, without requiring exemplary supervision or complete models of the environment. The rein- forcement learning algorithms can also make use of as many environmental parameters indicating states of wind turbines as possible to precisely select controlling actions of the yaw offsets of all the wind turbines. In addition, the learn- ing process of the reinforcement learning algorithms contin- ues during normal operation of the wind turbine, therefore, the reinforcement learning algorithms can learn to adapt to new states. Thus, optimal short-term or long-term goals, such as, power output of the whole wind turbine farm, can be achieved by applying the reinforcement learning algorithms to control yaw offsets of all the wind turbines of each se- quence, simultaneously.

In various embodiments described herein, a prediction of fu- ture states of one or more wind turbines in a wind turbine farm can be performed by using a states prediction model. The states prediction model may be a neural network.

The states prediction neural network is configured to receive as input a current weather information and information of me- chanical structure of the one or more wind turbines to gener- ate future states of the one or more wind turbines including predictions of at least one of the following parameters: a wind speed, a turbulence intensity, a site elevation, a tem- perature, a vertical wind shear, a location of the generated wake or a horizontal wind shear. The current weather infor- mation comprises various images and/or data captured by weather satellites or radars, data measured by equipment or devices at weather stations. Also, a current weather information and information of me- chanical structure of the one or more wind turbines may be received and future states of the one or more wind turbines may be generated by the state's prediction neural network. Then, it is possible to determine one or more controlling ac- tions of the one or more wind turbines based on not only the current state of the one or more wind turbines but also the future states of the one or more wind turbines and a rein- forcement learning algorithm. Both the short-term reward and the long-term reward are predicted by further taking the fu- ture environmental states into account.

For example, for the method 400 of FIG. 9, the one or more controlling actions associated with the yaw offset of the first wind turbine may be determined based on the current state of the first wind turbine, the current state of the second wind turbine, the predicted future states of the first wind turbine and of the second wind turbine, and a reinforce- ment learning algorithm.

For the method 500 of FIG. 10, the controlling actions asso- ciated with the yaw offsets of the plurality of wind turbines may be determined based on the current states of the wind turbines of the plurality of wind turbines, the predicted fu- ture states of the wind turbines of the plurality of wind turbines, and a reinforcement learning algorithm.

By further taking predictions of future states into account, both the method 400 and the method 500 for controlling yaw offsets of one or more wind turbines can utilize more infor- mation of the environment (such as, weather information) to determine more precise and reliable controlling actions and further obtain more power output of a wind farm.

In addition, before executing methods 400 or 500, there may be a step or procedure to determine whether a wake generated by an upstream wind turbine will influence a downstream wind turbine or not, for each wind turbine of a wind turbine farm. The determination can be performed by some algorithm taking a wind direction and a distance between the upstream wind tur- bine and the downstream wind turbine into account. If there are influences, the methods 400 or 500 will be executed.

FIG. 11 schematically illustrates aspects with respect to a device 800. The device 800 includes one or more processors 810 and a memory 820. The one or more processors 810 can load program code from the memory 820 and execute the loaded pro- gram code. The program code could define the reinforcement learning algorithm 700 and the modules 701-704 (cf. FIG. 6). Then, the one or more processors 810 can perform various techniques as described herein to control a yaw offset of an upstream wind turbine of a wind turbine pair of a wind farm (e.g., the method 400). The one or more processors 810 may comprise CPUs, GPUs, TPUs, or any other neural network pro- cessors.

Specifically, the one or more processors 810 are configured to receive data indicative of a current state of the first wind turbine and a current state of a second wind turbine ad- jacent to the first wind turbine downstream, determine one or more controlling actions associated with the yaw offset of the first wind turbine based on the current state of the first wind turbine, the current state of the second wind tur- bine, and a reinforcement learning algorithm 700, and apply the determined one or more controlling actions to the first wind turbine.

Both the first and second wind turbines can thus include a communication interface 81, one or more sensors 82, and a control unit 83, respectively.

The device 800 includes a communication interface 830. For example, the device 800 may receive data indicative of a cur- rent state of the first wind turbine and of a current state of a second wind turbine adjacent to the first wind turbine downstream, via the communication interface 830 from sensors 82 associated with both the first and second wind turbines, respectively. It would be also possible that the device 800 receives a current weather information from a database 840 that belongs to a meteorological center. In addition, the de- vice 800 may send, to the controlling element 83 of the first wind turbine, controlling signals including the one or more controlling actions associated with the yaw offset of the first wind turbine, via the communication interface 830.

The device 800 may include a user interface 850 for users to interact with the device 800. For example, the user can input data or information into the device 800 via the user inter- face 850, such as, tuples of the following data: wind direc- tions, wind speeds, yaw offsets, controlling actions, and sums of power outputs, wherein all the data are obtained from any two wind turbines that are adjacent to each other along the wind directions or expert knowledge, as well as infor- mation of mechanical structures of both the first and second wind turbines.

The device 800 may be used to execute the program codes of the above-mentioned reward neural network, value neural net- work, and states prediction neural network, respectively.

FIG. 12 schematically illustrates aspects with respect to a device 900 that is similar to the device 800. The device 900 includes one or more processors 910 and a memory 920. The one or more processors 910 can load program code from the memory 920 and execute the loaded program code. The program code could define the reinforcement learning algorithm 700 and the modules 701-704 (cf. FIG. 6). Then, the one or more proces- sors 910 also can perform various techniques as described herein to control yaw offsets of a plurality of wind turbines being arranged in sequence along a wind direction 205 in a wind turbine farm 200(e.g., the method 500). The one or more processors 910 may comprise CPUs, GPUs, TPUs, or any other neural network processors. Specifically, the one or more processors 910 are configured to receive data indicative of a respective current state for each wind turbine of the plurality of wind turbines,deter- mine controlling actions associated with the yaw offsets of the plurality of wind turbines based on the current states of the wind turbines of the plurality of wind turbines and a re- inforcement learning algorithm 700, and apply the determined controlling actions to the wind turbines of the plurality of wind turbines.

Each wind turbine of the plurality of wind turbines can thus include a communication interface 81, one or more sensors 82, and a control unit 83, respectively.

The device 900 includes a communication interface 930. For example, the device 900 may receive data indicative of a re- spective current state for each wind turbine of the plurality of wind turbines, via the communication interface 930 from sensors 82 associated with each wind turbine of the plurality of wind turbines, respectively. It would be also possible that the device 900 receives a current weather information from a database 840 that belongs to a meteorological center. In addition, the device 900 may send, to the controlling ele- ment 83 of each wind turbine of the plurality of wind tur- bines, corresponding controlling signals including corre- sponding controlling actions associated with the yaw offset of the corresponding wind turbine, respectively, via the com- munication interface 930.

The device 900 may include a user interface 940 for users to interact with the device 900. For example, the user can input data or information into the device 900 via the user inter- face 940, such as, tuples of the following data: wind direc- tions, wind speeds, yaw offsets, and sums of power outputs, wherein all the data are obtained from any plurality of wind turbines that are arranged in sequence along the wind direc- tions or expert knowledge, as well as information of mechani- cal structures of each wind turbine of the plurality of wind turbines.

The device 900 may be used to execute the program codes of the above-mentioned reward neural network, value neural net- work, and states prediction neural network, respectively.

Summarizing, techniques based on reinforcement learning, of controlling a yaw offset of an upstream wind turbine and of controlling yaw offsets of a plurality of wind turbines being arranged in sequence along a wind direction, in a wind farm, have been described. This is used to make efficient use of various environmental information, including but not limited to the wind speed, to control the yaw offsets of an upstream wind turbine, or of a plurality of wind turbines being ar- ranged in sequence along a wind direction, thereby achieving an optimal power output of a whole wind farm.