TECHNICAL FIELD

[oooi] The present disclosure relates to a method of controlling an energy supply system comprising a central power grid and at least one local power grid configured to supply at least one entity with energy, and a device performing the method.

BACKGROUND

[0002] Microgrids act as decentralized local sources of energy for individual households, communities and institutions. The microgrids can operate together with a central grid (also called macro-grid) or in an island mode where the microgrids operate fully autonomously. Microgrids with renewable energy sources are becoming increasingly important from a global climate change perspective and increasing adoption rate of micro-grids in individual households, communities or institutions is envisaged. “Multi-Agent Reinforcement Learning for Microgrids” by Dimeas and Hatziargyriou, IEEE Xplore, discusses energy management in microgrids using machine learning.

[0003] Seasonally (winter/ summer) as well as daily (day/ night) changing environmental conditions also create a varying demand for electricity usage. In other words, the weather conditions and individual household preferences are dynamic, which must be considered when modelling a microgrid system. Sources of renewable energy like solar panels (as these panels are dependent on the light from the sun rather than the heat) and wind turbines can continue to work efficiently even during peak winters and even the power transmission loss decreases because of drop in resistance in the power transmitting cables, which should be taken into account when modelling a microgrid system.

[0004] Due to varying climatic conditions and demand of total energy, the microgrid system must be controlled accordingly. However, manual switching from the central grid to the microgrid or vice versa is possible when dealing with a smaller number of control parameters, but when there are numerous complex control parameters like e.g. fulfilling of demand-supply, dynamic pricing, reducing carbon footprint, etc, that the system must be capable of handling, manual intervention is not feasible and a more intelligent control mechanism must be provided.

SUMMARY

[0005] One objective is to solve, or at least mitigate, this problem in the art and thus to provide an improved method of controlling an energy supply system comprising a central power grid and one or more local power grids.

[0006] This objective is attained in a first aspect by a method of controlling an energy supply system comprising a central power grid and at least one local power grid configured to supply at least one entity with energy. The method comprises acquiring information indicating, for a past time period, amount of energy having been supplied to the entity by the central power grid and the at least one local power grid, energy supply capacity of the central power grid and the at least one local power grid, and information indicating conditions affecting the energy supply and the energy capacity during said past time period and determining, based on the acquired information, expected energy delivery capacity of the central power grid and the at least one local power grid and expected energy demand of the entity during an upcoming time period. The method further comprises controlling selection of amount of energy supplied from the central power grid and from the at least one local power grid according to an energy criteria to be satisfied such that the expected energy demand of the entity is met during the upcoming time period.

[0007] This objective is attained in a second aspect by a device configured to control an energy supply system comprising a central power grid and at least one local power grid configured to supply at least one entity with energy. The device comprises a processing unit and a memory, which memory contains instructions executable by the processing unit, whereby the device is operative to acquire information indicating, for a past time period, amount of energy having been supplied to the entity by the central power grid and the at least one local power grid, energy supply capacity of the central power grid and the at least one local power grid, and information indicating conditions affecting the energy supply and the energy capacity during said past time period and to determine, based on the acquired information, expected energy delivery capacity of the central power grid and the at least one local power grid and expected energy demand of the entity during an upcoming time period. Further, the device is operative to control selection of amount of energy supplied from the central power grid and from the at least one local power grid according to an energy criteria to be satisfied such that the expected energy demand of the entity is met during the upcoming time period.

[0008] Advantageously, a device referred to as a microgrid controller is configured to control, based on historical demand and ability of a grid to deliver energy, energy supply to an entity such as a house from a central grid and a local grid comprising renewable energy sources, according to a preferred energy criteria such that an upcoming energy demand of the house is satisfied.

[0009] Thus, the microgrid controller is configured to control selection of energy sources where energy subscribers have access to energy from both a central grid and from a local micro-grid generating power from one or more renewable energy sources. Together with the advent of smart central grids with dynamic pricing schemes to manage supply and demand, it will be important for a subscriber to decide which energy source to use at any given point in time and subscribers should also be able to switch between energy sources intelligently. Such energy subscriber - i.e. an energy subscriber which not only consumes energy from the central grid but further produces energy via a local grid - is commonly referred to as a prosumer.

[0010] This switching will in embodiments take into account parameters such as weather conditions, load on central grid, time of day, time of week, time of year, occurring blackouts and/ or central grid maintenance, humidity, wind, surplus energy fed back to the central grid from prosumers, etc.

[0011] In an embodiment, the microgrid controller controls feedback of energy produced by the at least one local power grid to the central power grid. Hence, in case the energy subscriber is a prosumer which also is capable of feeding back locally produced (renewable) energy to the central grid, such capacity is taken into account. As is understood, energy fed back to the central grid maybe consumed also by energy subscribers which do not have access to a local grid. Further, not all energy systems allow a prosumer to feedback locally produced energy to the central grid.

[0012] In an embodiment, the at least one local power grid comprises one or more renewable energy sources supplying the entity with energy. [0013] In an embodiment, the one or more renewable energy sources comprises one or more of solar energy sources, wind energy sources, geothermal energy sources and hydropower energy sources.

[0014] In an embodiment, the at least one local power grid comprises one or more energy storage devices capable of storing energy produced by the renewable energy sources.

[0015] In an embodiment, the determining of expected energy delivery capacity further comprises taking into account expected weather conditions and/ or planned power grid maintenance during the upcoming time period.

[0016] The microgrid controller may control supply of energy to hundreds or even thousands of entities. In an embodiment, a machine learning (ML) framework will be applied at the microgrid controller in order for the controller to be able to handle the resulting great amount of data being generated and the many parameters being evaluated for controlling selection of energy sources. In an embodiment, the ML framework is implemented by applying reinforcement learning (RL).

[0017] In a third aspect, a computer program is provided comprising computerexecutable instructions for causing the device of the second aspect to perform the method of the first aspect when the computer-executable instructions are executed on a processing unit included in the device.

[0018] In a fourth aspect, a computer program product is provided comprising a computer readable medium, the computer readable medium having the computer program according to the third aspect embodied thereon.

[0019] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, in which:

[0021] Figure 1 illustrates a prior art microgrid system in which embodiments may be implemented;

[0022] Figure 2 illustrates prosumers/ consumers connected to an energy supply system;

[0023] Figure 3 illustrates a system where a microgrid controller according to an embodiment controls supply of energy to a group of households;

[0024] Figure 4 illustrates a flowchart of a method performed by a microgrid controller of controlling an energy supply system comprising a central power grid and three local power grids configured to supply a respective household with energy;

[0025] Figure 5 illustrates a reinforcement learning (RL) process deployed for creating and training an RL agent according to an embodiment;

[0026] Figure 6 illustrates interaction between a microgrid controller of an embodiment and an entity during an RL process;

[0027] Figure 7 illustrates deployment of a trained RL agent at the microgrid controller for controlling a respective entity according to an embodiment; and

[0028] Figure 8 illustrates a microgrid controller according to an embodiment.

DETAILED DESCRIPTION

[0029] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown.

[0030] These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0031] Figure 1 illustrates a prior art microgrid system in which embodiments may be implemented, where an entity such as a house 10 is equipped with roof- mounted solar panels n supplying household equipment such as a fridge 12 and a washing machine 13 and potentially also charging equipment for an electric vehicle 14 with energy via an energy conversion system 15. Further shown is a wind turbine 16 which maybe used for the same purpose.

[0032] While a central grid 17 supplies the house 10 with energy, it may also be possible that some or all of the energy produced by the solar panels 11 and the wind turbine 16 are fed back to the central grid 17. For instance, if the household does not consume all produced wind and solar energy, the house owner may sell any surplus energy to other consumers by supplying the surplus energy to the central grid 17 for consumption at another place. Alternatively, any surplus energy may be stored at an appropriate energy storage for subsequent consumption at the house 10.

[0033] Further, it may be the case that the house owner remotely may control the system using a smart phone 19.

[0034] A consumer of energy which also produces energy as illustrated in Figure 1 is commonly referred to as a prosumer.

[0035] Figure 2 illustrates prosumers 20 which consume as well as produces energy while consumers 21 only consumes energy. The prosumers 20 may thus feedback any locally produced energy to the central power grid 22 for consumption by the consumers 21, if allowed for technical and/ or regulatory reasons. The central power grid 22 may in its turn receive power from a conventional power plant 23, a renewable energy plant 24 and/ or energy storage 25.

[0036] As described, managing a microgrid system becomes complex with an ever increasing number of control parameters to consider. Therefore, an intelligent mechanism for controlling supply of energy to one or more entities is proposed based on energy demand of the entities.

[0037] As is understood, the entity being supplied with energy from the central power grid and a local power grid may be a household (or a group households on a particular premise), a radio site accommodating radio base stations and other equipment, a factory, a public building, etc.

[0038] Figure 3 illustrates a system where a microgrid controller 30 according to an embodiment controls supply of energy to a group of households 31a, 31b, 31c each being connected to the central power grid 40 and a local power grid referred to as a microgrid. In this exemplifying embodiment, the microgrid of each household 31a, 31b, 31c is comprises a solar energy source 32a, 32b, 32c and a wind energy source 33a, 33b, 33c, respectively. Oher sources of renewable energy such as geothermal energy from heat inside the earth or hydropower from flowing water may be envisaged. As is understood, the microgrid controller 30 may in practice control energy supply to hundreds or even thousands of entities.

[0039] Further, each household 31a, 31b, 31c may comprise a local energy storage 34a, 34b, 34c, where surplus renewable energy is stored for later use.

[0040] Figure 4 illustrates a flowchart of a method performed by the microgrid controller 30 of controlling the energy supply system comprising the central power grid 40 and the three local power grids (equipped with solar 32a, 32b, 32c and wind 33a, 33b, 33c energy sources) configured to supply the respective house 31a, 31b, 31c with energy.

[0041] The microgrid controller 30 is thus typically capable of communicating with each individual entity 31a, 32a, 33a and further a central power grid operator 41 in order to be able to control selection of energy source and to acquire information controlling the selection.

[0042] Further, the microgrid controller 30 may communicate with other microgrid controllers which are configured to control selection of energy sources for other entities (which is not shown in Figure 3) ; a group of microgrid controllers may interoperate to cover a great geographical area comprising a huge number of different entities.

[0043] In the following exemplifying embodiment, control of energy supply to the first house 31a will be described in detail. As is understood, the same process may simultaneously be performed for the second house 31b and the third house 31c

[0044] In order to determine the upcoming energy demand of the first house 31a, the microgrid controller 30 acquires information in a first step S101 indicating, for a past time period: a) amount of energy having been supplied to the house 31a by the central power grid 40 and the local power grid 32a, 33a; b) energy supply capacity of the central power grid 40 and the local power grid 32a, 33a; and c) information indicating conditions affecting the energy supply and the energy capacity during the past time period.

[0045] As is understood, the historical data acquired may extend over a long time period, such as weeks, months or even years. The historical data may be stored locally at each entity and/ or at a central storage held by a central power grid operator.

[0046] Based on the acquired historical data, the microgrid controller 30 determines in step S102, for an upcoming time period: d) expected energy delivery capacity of the central power grid 40 and the local power grid 32a, 33a; and e) expected energy demand of the house 31a.

[0047] Step S102 may be referred to as a data mining and energy usage analysis step.

[0048] A number of parameters may be analysed in this step. Further, when determining energy demand of the first house 31a, it may further be necessary to take into account energy demand of the second house 31b and the third house 31c, as these demands may affect central grid load and thus capacity.

[0049] For instance, the following parameters may be analysed in step S102 for an upcoming time period, such as the next coming hour:

D _g : total central grid energy demand,

S _g total energy supply capacity of central grid,

D _r = total renewable energy demand,

S _r = total renewable energy supply capacity of all the renewable energy sources,

N _p = number of prosumers connected to the microgrid controller,

N _c = number of consumers connected to the microgrid controller,

E _g = average central grid energy consumption by each entity, and

E _r = average renewable energy consumption by each entity. [0050] In an example, the microgrid controller 30 determines in step S102 - based on the historical data acquired in step S101 - that for the next upcoming hour, say 14:00-15:00 on a Wednesday for a given weather condition (“sunny, light wind”), the first house 31a usually has an energy demand of 2 kWh as indicated by the energy that in fact has been delivered to the entity historically between 14:00 and 15:00 on Wednesdays under sunny conditions.

[0051] As is understood, the microgrid controller 40 would typically estimate upcoming weather by attaining a weather forecast from an appropriate provider and any length of the upcoming time period may be envisaged such as 1 hour, 2 hours, 6 hours, etc.

[0052] However, the information indicating conditions historically affecting the energy capacity that was acquired in step S101 indicates that between 14:00 and 15:00 on Wednesdays, the load on the central grid is high and the central grid thus only has capacity to deliver 1 kWh (this may be the case regardless of weather conditions).

[0053] Hence, in order to meet the estimated energy demand of the first house 31a (which amounts to 2 kWh during the upcoming hour), the local power grid 32a, 33a must be capable of producing 1 kWh.

[0054] Further, the information indicating conditions historically affecting the energy capacity that was acquired in step S101 indicates that between 14:00 and 15:00 on Wednesdays, the solar panels 32a has an expected energy supply capacity of, say, 1.4 kWh in case of sunny weather, while the wind turbine 33a has an expected energy supply capacity of 0.4 kWh in case of light wind.

[0055] Based on these estimations, the microgrid controller 30 controls in step S103 selection of amount of energy supplied from the central power grid 40 and from the local power grid 32a, 33a according to an energy criteria to be satisfied such that the expected energy demand of the first house 31a is met.

[0056] In this exemplifying embodiment, the energy criteria “use as much renewable energy as possible” is evaluated, and the microgrid controller 30 thus selects the solar panels 32a and the wind turbine 33a to deliver in full, i.e. 1.4 + 0.4 = 1.8 kWh, while controlling the central grid 40 to deliver only 0.2 kWh, thereby satisfying the 2-kWH demand of the house 31a. [0057] In another example, the upcoming weather conditions are instead “cloudy, strong wind”, and the microgrid controller 40 determines in step S102 (from the historical data acquired in step S101) that under these particular conditions, the solar panels 32a instead has an expected energy supply capacity of 0.2 kWh, while the wind turbine 33a now has an expected energy supply capacity of 2.1 kWh.

[0058] The microgrid controller 30 thus controls the energy supply in step S103 such that 2 kWh is delivered to the house 31a from the local grid, for instance 0.2 kWh from the solar panels 32a and 1.8 kWh from the wind turbine 33a, while no energy is delivered to the house 31a from the central grid 40. In addition, 0.3 kWh of surplus energy produced by the wind turbine 33a maybe fed back to the central grid 40 and thus delivered to another entity if required.

[0059] For instance, the microgrid controller 30 may conclude that the second house 31b indeed requires a further 0.2 kWh for its energy demand to be met, in which case the microgrid controller 30 may inform a central power grid operator that at least part of the energy fed back to the central grid 40 from the first house 31a should be delivered to the second house 31b.

[0060] In yet another example, the information acquired in step S101 indicates that on Wednesdays between 14:00 and 15:00, the central power grid 40 typically suffers from blackouts in this particular area and thus commonly cannot deliver energy at all. In such a scenario, the microgrid controller 40 may control - even if in step S102 the energy demand is still determined to be 2 kWh - the solar panels 32a and the wind turbine 33a to deliver all energy to the house 31a even if such supply is less than 2 kWh. In such a scenario, the microgrid controller 30 may thus conclude that no energy will be delivered from the central power grid 40 during the next upcoming hour.

[0061] Further, in such an example, the microgrid controller 30 may conclude that e.g. in the preceding hour, i.e. 13:00-1400, no surplus renewable energy should be fed back to the central power grid 40 but instead stored in a local energy storage for later use since there is a risk that the upcoming energy demand of the house 31 between 14:00 and 15:00 otherwise cannot be met. Such storage may thus facilitate meeting the upcoming energy demand. [0062] Advantageously, the microgrid controller 30 according to embodiments is configured to control, based on historical demand and ability to deliver energy, energy supply to the house 31a from the central grid 40 and the local grid 32a, 33a according to a preferred energy criteria such that an upcoming energy demand of the house 31a is satisfied.

[0063] Thus, the microgrid controller 30 is configured to control selection of energy sources where energy subscribers, i.e. prosumers, have access to energy from both a central grid and from a local micro-grid generating power from one or more renewable energy sources. Together with the advent of smart central grids with dynamic pricing schemes to manage supply and demand, it will be important for a subscriber to decide which energy source to use at any given point in time and subscribers should also be able to switch between energy sources intelligently.

[0064] This switching will take into account parameters such as weather conditions, load on central grid, time of day, time of week, time of year, occurring blackouts and/or central grid maintenance, humidity, wind, surplus energy fed back to the central grid from prosumers, etc.

[0065] The switching may further take into account various indices such as

Customer Average Interruption Duration Index (CAIDI), System Average Interruption Duration Index (SAIDI), Momentary Average Interruption Frequency Index (MAIFI), System Average Interruption Frequency Index (SAIFI), etc.

[0066] There are numerous advantages associated with the embodiments described herein, as will be listed in the following.

[0067] 1. Stabilizing energy demand-supply.

[0068] 2. Prosumers will have flexibility of choosing energy sources, and possibly also feeding back any locally produced surplus energy to the central grid.

[0069] 3. Optimized resource allocation may be achieved within the community served by the microgrid controller (s).

[0070] 4. Peak hour energy consumption may be mitigated by controlling supply and demand of electricity in an area.

[0071] 5. Central grid in many developed European countries (e.g. Sweden) is using hydropower and nuclear, which provides for a very low carbon footprint. In many grids, coal and fossil fuel is used, which causes high carbon emission. Sometimes the grid can have mix of high and low carbon. By adopting the solution set forth herein, carbon footprint can be reduced wherever required.

[0072] 6. Energy source-switching is enabled in smart home solutions.

[0073] 7. The selection between central grid and local grid reduces level of peak hour-usage and eventually provides for a healthier energy transmission network and reduces maintenance.

[0074] 8. The occurrence of power failures are reduced since the energy subscribers need not rely solely on the central power grid.

[0075] 9. The need for a high-availability central grid is produced.

[0076] 10. Sustainability is increased.

[0077] As previously mentioned, the microgrid controller 30 may control supply of energy to hundreds or even thousands of entities. In an embodiment, a machine learning (ML) framework will be applied at the microgrid controller 30 in order for the controller to be able to handle the resulting great amount of data being generated and the many parameters being evaluated for controlling selection of energy sources.

[0078] Reinforcement Learning (RL) is a flexible framework that applies data driven optimization to control problems. It is based on an agent’s interaction with an environment, where the RL agent visits states of the environment, executes actions, and receives rewards for such actions. In embodiments of the invention, so-called meta RL may be applied to handle the continuously changing environment.

[0079] Applying RL as ML framework at the microgrid controller 30 may in an embodiment imply that each individual entity 31a, 32a, 33a is view upon as its own environment and for each environment/entity, an RL agent is trained using the previously described historical data and evaluating upcoming energy demands in order to be able to make informed decisions on the selection of energy sources to satisfy the upcoming energy demands.

[0080] The goal of RL is to find a policy that optimizes a value function. The value function is a mapping between the different states and the expected rewards, also known as the expected return. An optimal policy selects actions that lead to a state with the highest expected return. [0081] Figure 5 illustrates an RL process 200 deployed for creation and training of an RL agent 202 to subsequently be used for controlling selection of energy sources at the house 31a represented by the environment 201. As is understood, the RL process 200 is performed for each respective house 31a, 31b, 31c. In other worlds, an RL agent is trained for each environment, i.e. each house 31a, 31b, 31c. One common algorithm applied by RL agents is the so-called Deep Q-Network (DQN) RL algorithm.

[0082] Reference is further made to Figure 6 showing interaction between the microgrid controller 40 and the first house 31a during the RL process 200.

[0083] The first house 31a is thus represented by the environment 201, to which the microgrid controller 40 sends input data in step S201, which hence practically constitutes a training signal for the RL agent 202 (via the response of the environment 201). The input data may indicate e.g. current source of energy, weather conditions, time of day, etc. The house 31a will return a reward in step S202 indicating how well the environment 201 responds to the input data. As is understood, in a training phase, a large amount of (sets of) input data may be supplied to the environment 201 in order to appropriately train the RL agent 202.

[0084] The environment 201 (i.e. the first house 31a) thus returns a reward for each set of input data being supplied to the RL process 200 and the agent 202 at the microgrid controller 40 will accordingly take appropriate action in terms of controlling selection of amount of energy supplied from the central power grid 40 and from the energy sources 32a, 33a of the local power grid at the house 31a according to an energy criteria to be satisfied such that the expected energy demand of the entity 31a is met, as previously discussed with reference to step S103.

[0085] These actions will thus be fed back in step S203 from the agent 202 located at the microgrid controller 40 to the environment 201 (i.e. the house 31a) for each reward received. Such actions may include one or more or of select central grid or local grid, feedback surplus renewable energy to central grid, store renewable energy in local storage, consume energy from local storage, etc.

[0086] To conclude, the RL agent 202 is trained based on the input data supplied to the environment 201, and actions are taken by the RL agent 202 in the environment 201 to arrive at an optimal policy regarding energy source selection to be performed for said environment 201. As is understood, the house 31a may ignore the instruction of energy source usage alerted by the microgrid controller 40 or simply overrule the instruction. Such behaviour would be observed by the agent 202.

[0087] Ultimately, the trained RL agent 200 maybe used at the microgrid controller 40 to find an optimal regime for controlling selection of energy sources at the entities 31a, 31b, 31c. Once an appropriately trained RL agent 202 has been attained, instructions maybe provided by the microgrid controller 40 to the trained RL agent 202 to ensure that a desired result is arrived at before the instructions are provided the first house 31a. In line with the previously described example, the microgrid controller 40 may use the trained RL agent to determine power control strategy for the upcoming hour 14:00-15:00 on a Wednesday at a certain forecasted weather condition.

[0088] The actual environment 201 is determined by factors such as the weather conditions, number of people residing in the first house 31a and their consumption patterns. Thus, the environment 201 in which the RL agent 202 is trained maybe continuously and slowly changing, hence the RL agent 202 may need to continuously change its policy and be further trained.

[0089] Further, in a multi-agent setting, the policy of an agent depends on the policy of other agents. While three entities 31a, 31b, 31c are illustrated in Figure 3, the microgrid controller 40 may in practice control hundreds or even thousands of entities. One approach to deal with such multi-agent setting is to implement a so- called “forget” approach, wherein the RL agent 202 continuously receives recent observations of other agents and updates its policy.

[0090] Again with reference to Figures 5 and 6, rewards sent in step S202 from the first house 31a to the microgrid controller 40 is an indication of how well the actions of the agent 202 supplied to the first house 31a in step S203 complies with the RL policy to be attained given the input data of step S201.

[0091] Illustrated below is an example of a reward function that could be used to calculate total reward: reward = f X ₁ ,X ₂ ,X ₃ ...X _n ; W _1: W ₂ . W _m ), where X are values of input variables and W are weight values which are configurable parameters. [0092] The above function provides the flexibility to calculate total reward dynamically based on multiple factors. The simplest form of a reward function can be linear in all the features which takes in weightage parameters along with respective parameters and returns total calculated reward. This function can be changed according to the need that may vary from subscriber to subscriber.

[0093] An example reward for a single subscriber may be defined as follows:

Xi= Energy consumption in kWh,

X2= Emission of carbon footprint,

X3= Power outages in minutes,

X4= Voltage stability,

X5= Amount of energy fed back to the central grid in kWh (if applicable; for a pure consumer this is zero),

X6= Reactive power in kWh,

Wi...We= Weights associated with each of the input variables or factors X1...X6

[0094] This is equivalent to:

[0095] R = i Wi * X where n = total number of factors included in the reward function.

[0096] Each of the X _L has different scales, which can create instability during the optimization process (reward maximization process). To avoid this instability, it is possible to normalize each X _L using their respective mean (/z _£ ) and standard deviation (0 ) using the normalization formula X _L = (X _t - [i ) I J _L .

[0097] As previously described, the idea is to maximize the reward in order to arrive at a trained RL agent 202 complying with the set RL policy of the RL process 200.

[0098] The reward generated by each entity in this example utilizes weights W for all factors X of the all the factors-based on user requirement. For instance:

- if the user prioritizes cost factor, higher weightage is attached to the cost incurred per unit of electricity consumption in kWh, - if carbon footprint minimization is the priority as per the local government policy, then higher weights are associated with that factor,

- once the agent takes any actions which results in outage then such actions will receive high negative reward,

- every action taken by the agent must meet the load requirements at the subscriber end which will ensure stability in terms of grid load/voltage, and

- rewards may also take in consideration the incentives of a subscriber of selling back the excess renewable energy to the central grid.

[0099] The states of the environment 201 visited by the RL agent 202 is a result of the input data being supplied by the microgrid control 40 to the first house 31a in step S201.

[00100] As a simple example, consider the following input data is supplied to the environment 201 of the RL process 200:

1) Current source of energy,

2) Time of day,

3) Day of week,

4) Current load of the central grid,

5) Weather, a) Temperature, b) Humidity, c) Wind speed,

6) Main outage prediction,

7) Predicted load, and

8) Own power available.

[00101] Given these, an example state as perceived by the RL agent 202 may have the following appearance:

Current source of energy: “Main grid”,

Time of day: “Morning”, Day of week: “Sunday”,

Current load of the central grid: “High”,’

Weather

Temperature: “Low”,

Humidity: “Medium”,

Wind speed: “High”,

Main outage prediction: “Grid failure”,

Predicted load: “High”, and

Own power available: “Yes”.

[00102] Figure 7 illustrates deployment of the trained RL agent 202 at the microgrid controller 40 for controlling the respective entity 31a, 31b, 31c.

[00103] Thus, after having trained the RL agent 202 utilizing the RL process 200 for each entity 31a, 31, 31c as previously described, the microgrid controller 40 will utilize he trained RL agent 202 to control the selection of energy at the respective entity 31a, 31, 31c.

[00104] In step S301, the microgrid controller 30 supplies to the trained RL agent 202 with information indicating an upcoming time period during which the houses 31a, 31b, 31c are to be controlled. Further in step S301, expected weather conditions for the upcoming period and other useful information such as planned maintenance of the central grid or any local grid may be included with the information.

[00105] Now, as described during the training of the RL agent 202 historical data was supplied to RL agent indicating:

- amount of energy having been supplied to the house 31a by the central power grid 40 and the local power grid 32a, 33a;

- energy supply capacity of the central power grid 40 and the local power grid 32a, 33a; and

- information indicating conditions affecting the energy supply and the energy capacity during the past time period, e.g. weather conditions, time of day, grid maintenance, etc. [00106] As a result, the trained RL agent 202 computes in step S302 - given the indicated time period of S301 and optionally expected weather conditions and /or planned maintenance:

- expected energy delivery capacity of the central power grid 40 and the local power grid 32a, 33a;

- expected energy demand of the house 31a; and further optionally

- energy demand of the second house 31b and the third house 31c, and

- ability of any of the houses 31a, 31b, 31c to feedback renewable energy to the central grid 40.

[00107] Finally in step S303, the microgrid controller 30 controls - based on the computations of the trained RL agent 202 - selection of amount of energy supplied from the central power grid 40 and from the sources 32a, 33a of the local power grid according to an energy criteria to be satisfied such that the expected energy demand of the first house 31a is met for the upcoming time period.

[00108] As is understood, the microgrid controller 30 controls the energy being supplied to the second house 31b and the third house 31c accordingly using the trained RL agent for each of the second house 31b and the third house 31c such that the expected energy demand of those two houses are met as well.

[00109] Figure 8 illustrates a device 30 referred to as a microgrid controller configured to controlling an energy supply system comprising a central power grid and one or more local power grids configured to supply one or more entities with energy, where the steps of the method performed by the microgrid controller 30 in practice are performed by a processing unit 111 embodied in the form of one or more microprocessors arranged to execute a computer program 112 downloaded to a storage medium 113 associated with the microprocessor, such as a Random Access Memory (RAM), a Flash memory or a hard disk drive. The processing unit 111 is arranged to cause the microgrid controller 30 to carry out the method according to embodiments when the appropriate computer program 112 comprising computerexecutable instructions is downloaded to the storage medium 113 and executed by the processing unit 111. The storage medium 113 may also be a computer program product comprising the computer program 112. Alternatively, the computer program 112 maybe transferred to the storage medium 113 by means of a suitable computer program product, such as a Digital Versatile Disc (DVD) or a memory stick. As a further alternative, the computer program 112 may be downloaded to the storage medium 113 over a network. The processing unit 111 may alternatively be embodied in the form of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), etc. The microgrid controller 30 further comprises a communication interface 114 (wired or wireless) over which it is configured to transmit and receive data.

[00110] The device 30 of Figure 8 maybe provided as a standalone device or as a part of at least one further device. For example, the device 30 maybe provided in a node of a core network, or in an appropriate device of a radio access network (RAN), such as in a radio base station in communication with entities to which energy is supplied, in an internet server, etc. Alternatively, functionality of the device 30 may be distributed between at least two devices, or nodes. These at least two nodes, or devices, may either be part of the same network part (such as the core network) or maybe spread between at least two such network parts.

[00111] Thus, a first portion of the instructions performed by the device 30 may be executed in a first device, and a second portion of the of the instructions performed by the device 30 maybe executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the device 30 maybe executed.

[00112] Hence, the method according to the herein disclosed embodiments are suitable to be performed by a device 30 residing in a cloud computational environment. Therefore, although a single processing circuitry 111 is illustrated in Figure 8, the processing circuitry 111 may be distributed among a plurality of devices, or nodes. The same applies to the computer program 112. Embodiments maybe entirely implemented in a virtualized environment.

[00113] The aspects of the present disclosure have mainly been described above with reference to a few embodiments and examples thereof. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims. [00114] Thus, while various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.