DEPLOYMENT AND UPDATE OF MACHINE LEARNING MODELS IN A RADIO ACCESS NETWORK

TECHNICAL FIELD

[001] Disclosed are embodiments related to the deployment and updating of machine learning (ML) models in a radio access network (RAN).

BACKGROUND

[002] Overall Architecture of NG-RAN

[003] FIG. 1 illustrates the current 5G RAN (a. k.a., the Next Generation RAN (NG-RAN)) architecture.

The NG-RAN architecture is described in 3GPP Technical Specification (TS) 38.401 v17.0.0. The NG-RAN consists of a set of base stations (denoted "gNBs”) connected to a 5G core network (5GC) through an NG interface. The gNBs can be interconnected through an Xn interface. A gNB may consist of a gNB central unit (gNB-CU) and one or more gNB distributed units (gNB-DU(s)). A gNB-CU and a gNB-DU are connected via an F1 interface. One gNB-DU is connected to only one gNB-CU. The NG, Xn and F1 interfaces are logical interfaces. A gNB-CU may comprise a gNB-CU control plane (CP) function (gNB-CU-CP) and a gNB-CU user plane (UP) function (gNB-CU-UP).

[004] Ongoing 3GPP discussion

[005] The 3GPP Technical Report (TR) 37.817 V17.0.0 (“TR 37.817”) has been produced as outcome of the Study Item (SI) “Enhancement for Data Collection for NR and EN-DC” defined in 3GPP Technical Document (Tdoc) No. RP-201620.

[006] The study item aimed to study the functional framework for RAN intelligence enabled by further enhancement of data collection through use cases, examples, etc., and identify the potential standardization impacts on current NG-RAN nodes and interfaces.

[007] TR 37.817 identifies the following high-level principles that should be applied for Al-enabled RAN intelligence:

The detailed AI/ML algorithms and models for use cases are implementation specific and out of RAN3 scope.

The study focuses on AI/ML functionality and corresponding types of inputs/outputs.

The input/output and the location of the Model Training and Model Inference function should be studied case by case. The study focuses on the analysis of data needed at the Model Training function from Data Collection, while the aspects of how the Model Training function uses inputs to train a model are out of RAN3 scope.

The study focuses on the analysis of data needed at the Model Inference function from Data Collection, while the aspects of how the Model Inference function uses inputs to derive outputs are out of RAN3 scope.

Where AI/ML functionality resides within the current RAN architecture, depends on deployment and on the specific use cases.

The Model Training and Model Inference functions should be able to request, if needed, specific information to be used to train or execute the AI/ML algorithm and to avoid reception of unnecessary information. The nature of such information depends on the use case and on the AI/ML algorithm.

The Model Inference function should signal the outputs of the model only to nodes that have explicitly requested them (e.g. via subscription), or nodes that take actions based on the output from Model Inference.

An AI/ML model used in a Model Inference function has to be initially trained, validated and tested by the Model Training function before deployment.

NG-RAN SA is prioritized; EN-DC and MR-DC are down-prioritized, but not precluded from Rel.18.

Functional framework and high-level procedures defined in this TR should not prevent from “thinking beyond” them during normative phase if a use case requires so.

User data privacy and anonymisation should be respected during AI/ML operation.

[008] Radio Access Network (RAN) Intelligence

[009] FIG. 2 illustrates the Functional Framework for RAN Intelligence. As shown in FIG. 2, the framework includes the following functions: 1) a data collection function; 2) a model training function; 3) a model inference function; and 4) an actor function, or Actor.

[0010] The data collection function provides training data (e.g., a set of training data samples - i.e., one or more training data samples) to the model training function. Training data is data that is used by the model training function to train a model (e.g., a neural network or other model). In Machine Learning (ML) (a.k.a., “Artificial Intelligence (Al)”) parlance, a model (e.g. a neural network) is defined as a functional approximation, whose parameters (e.g., neural network weights) are optimized to approximates a mathematical function, whose input-output behavior is characterized by a data set (i.e., the training set). In many Reinforcement Learning (RL) systems, the function approximated by the model is the Q-function, which assigns a value to a state-action pair. In turn, the Q-function (hence the ML model) determine the behavior (or policy) of the RL agent. The data collection function also provides inference data to the model inference function, which uses the inference data to produce an output (a.k.a., an inference)

[0011] ML model specific data preparation (e.g., data pre-processing and cleaning, formatting, and transformation) may also be carried out in the data collection function. Examples of interference and training data may include measurements from user equipments (UEs) or different network entities, feedback from the Actor, and output from the model inference function.

[0012] The model training function performs the ML model training, validation, and testing which may generate model performance metrics as part of the model testing procedure. The model training function is also responsible for data preparation (e.g. data pre-processing and cleaning, formatting, and transformation) based on training data delivered by a data collection function, if required. The model training function deploys a trained, validated and tested model (e.g., a model that parameterizes or approximates at least one of a policy function, a value function and a Q-function in a deep reinforcement learning environment) to the model inference function or delivers an updated model to the model inference function.

[0013] The model inference function provides model inference output (e.g. predictions or decisions). The model inference function may provide model performance feedback to the model training function when applicable. The model inference function is also responsible for data preparation (e.g. data pre-processing and cleaning, formatting, and transformation) based on inference data delivered by a data collection function, if required. The model inference function may provide model performance feedback information to the model training function, which uses this feedback information for monitoring the performance of the model.

[0014] The actor is a function that receives the output from the model inference function and triggers or performs corresponding actions. The Actor may trigger actions directed to other entities or to itself. The actions may generate feedback information, provided to the data collection function, that may be needed to derive training or inference data.

[0015] Three use cases have been identified for RAN Intelligence: 1) Network Energy Saving, 2) Load Balancing, and 3) Mobility Optimization and the potential standard impacts. These are described in TR 37.817.

[0016] For Network energy saving use case, TR 37.817 states:

The following solutions can be considered for supporting AI/ML-based network energy saving:

1. AI/ML Model Training is located in the 0AM and AI/ML Model Inference is located in the gNB [5G base station],

2. AI/ML Model Training and AI/ML Model Inference are both located in the gNB. Note: gNB is also allowed to continue model training based on model trained in the 0AM. In case of CU-DU split architecture, the following solutions are possible:

3. AI/ML Model Training is located in the 0AM and AI/ML Model Inference is located in the gNB-CU.

4. AI/ML Model Training and Model Inference are both located in the gNB-CU.

[0017] For Mobility Optimization use case, TR 37.817 states:

Considering the locations of AI/ML Model Training and AI/ML Model Inference for mobility solution, the following two options are considered:

1 . The AI/ML Model Training function is deployed in 0AM, while the model inference function resides within the RAN node.

2. Both the AI/ML Model Training function and the AI/ML Model Inference function reside within the RAN node.

Furthermore, for CU-DU split scenario, following option is possible:

3. AI/ML Model Training is located in CU-CP or 0AM and AI/ML Model Inference function is located in CU-CP. Note: gNB is also allowed to continue model training based on model trained in the 0 AM.

[0018] For Load Balancing use case, TR 37.817 states:

The following solutions can be considered for supporting AI/ML-based load balancing:

1. AI/ML Model Training is located in the 0AM and AI/ML Model Inference is located in the gNB.

2. AI/ML Model Training and AI/ML Model Inference are both located in the gNB.

In case of CU-DU split architecture, the following solutions are possible:

3. AI/ML Model Training is located in the 0AM and AI/ML Model Inference is located in the gNB-CU.

4. AI/ML Model Training and Model Inference are both located in the gNB-CU.

Note: gNB is also allowed to continue model training based on model trained in the 0AM.

Other possible locations of the AI/ML Model Inference are FFS.

[0019] 3GPP Technical Docment (Tdoc) R3-215244 proposes to introduce a model management function in the Functional Framework for RAN Intelligence, as shown in FIG. 3. Tdoc R3-215244 states: Model deployment/update should be decided by model management instead of model training. The model management may also host a model repository. The model deployment/update should be performed by model management.

Model performance monitoring is a key function to assist and control model inference. The model performance feedback from model inference should be first sent to model management. If the performance is not ideal, the model management may decide to fallback to traditional algorithm or change/update the model.

The model training should be also controlled by model management.

The model management function may be taken by either 0AM or CU or other network entities depending on the use cases. Clearly defining a model management function is useful for future signalling design and analysis.

Proposal 1 : Introduce a model management function into AI/ML framework [as shown in FIG. 3],

Model management function supports following roles: I) Requesting model training and receiving the model training result; II) Model deployment/updates for inference, ill) Model performance monitoring, including receiving performance feedback from model inference and taking necessary action, e.g. keep the model, fallback to traditional algorithm, change or update the model, iv) Model storage.

[0020] Training architectures for Reinforcement Learning (RL)

[0021] The main objective of model training is to produce a model (e.g., neural network that parameterizes or approximates at least one of a policy function, a value function and a Q -function) that can generalize to conditions and situations not directly experienced in the training data (i.e., a model that performs well when used with inference data that differs from the training data used in the training process). This process is also known as a training process.

[0022] Recent advances in the field of reinforcement learning (RL) have focused on techniques that could improve the quality of learning, learning efficiency (i.e., how much information can be extracted from given training data) and learning speed. Many of these techniques rely on advanced training architectures that exploit parallel and distributed collection of training data which may comprise a set of training data samples, such as, for example, experience samples (or "experiences” for short), combined with either a centralized or distributed training process. In one scenario, each “rollout worker” (i.e., a function that combines the functionality of the model inference function and the Actor function) receives a model update from a Model Training function. The rollout worker (e.g., an RL agent) uses the received model to interact with an external environment by selecting actions and applying the actions to the environment. In return, the rollout worker can collect experience samples that can be used for further training and improving the model. Typically, an experience sample is a tuple that comprises: i) an observation (e.g., state vector) for time step t (denoted St), ii) an action (At) selected based on St, ill) an observation for time step t+1 (denoted St+1), and iv) a reward value Rt based on St and St+1 . Some techniques provide a shared storage memory, also known as “replay buffer" or “experience buffer,” in which the rollout workers store the experience samples (e.g., at each time step, the rollout worker generates and stores an experience in the replay buffer). The Model Trainer function can then filter experiences from the replay buffer to train/update the model (e.g., a new set of weights of a neural network), which is then provided to the distributed rollout workers.

[0023] Parallel and distributed experience sample collection allows the evaluation of multiple versions of a model in parallel and to quickly produce a new model. It also allows for improved diversity in the collected information, as different rollout workers can be tasked to test the model against different versions of the environment. This allows improved quality of the collected experiences, which in turns enables: producing a model that better generalizes against conditions (e.g., events) unseen during the training process, improving the speed of learning because updates of the model can be provided more frequently due to the high throughput of the training data generation, and improving learning efficiency (i.e., the improved data diversity provided by parallel and distributed rollout workers enables production of a better model for a given amount of experience samples compared to the case where a single rollout worker is used). Using these techniques in a RAN could achieve a performance that otherwise would not be possible to achieve.

SUMMARY

[0024] Certain challenges presently exist. For instance, one problem with existing technology is that the specifications clearly state that the ML model is implementation specific. This implies that a specific ML model (a.k.a., “Al model” or “AI/ML model”) cannot be transferred to a RAN node in a way that its details (or semantics) are visible over an open interface (such as a 3GPP interface, or an O-RAN interface). In other words, the details of an ML model and potentially some - if not all - of the properties of the ML model are not to be revealed.

[0025] This issue can be addressed in a proprietary way, e.g., by extending an open interface with private Information Elements (lEs) that can only be understood (encoded/decoded) by a network node which is aware of these proprietary extensions. A typical way to represent a piece of information as proprietary over open standardized interfaces is to encode the associated IE as an OCTET STRING. OCTET STRING over 3GPP interfaces is seen as transparent payload (similar to an executable software package) and can be used for carrying ML models and associated information not to be revealed. However, the issue remains when different network nodes (e.g., different RAN nodes) of a communication network are provided by different organizations/vendors that do not have a shared understanding of the proprietary extensions, and the ML model (and its associated properties/details) has to be carried over lEs defined in an open interface.

[0026] Even though it is possible to use OCTET STRING encoding to avoid that an ML model design is decodable over open interfaces, this approach creates some problems.

[0027] One problem is present when multiple models are signaled to different nodes in the network. In this case, it is not possible to identify the difference in the nodes' behavior due to the difference in the models' performance. Namely, given that all models are un-decodable, it is not possible to associate a given behavior to a specific model. Similarly, it is not possible to associate a change in performance with a specific model. This implies that it cannot be understood which, of the multiple models, is producing a positive performance impact and which is producing a negative performance impact. This greatly limits the possibility to use ML models effectively. For example, it is not possible to establish a causal relationship between reported measurement data and the originating configuration.

[0028] Another problem similar to the one described above is that, according to specifications, if a network node hosting an ML model inference function decides to update such model, there are no means to let other nodes understand that the ML model has changed. Namely, even in a simplified scenario where in a network a single ML model has been deployed, different nodes in the network would not understand that the performance impact of the model has changed due to the updating of the model. This, again, implies that it is not possible to derive a cause-effect rule by which the model outputs can be used with predictable performance impacts.

[0029] Another problem with the above is that, even if the network node can decode the ML model and use it as a software package (i ,e. , a set of one or more software programs or applications), the ML model might be an obfuscated “black box” which is only accessible through an Application Program Interface (API). This is particularly desirable for developers of ML models, which might want to protect the added value of their products as well as prevent sensitive information leakage from the datasets used to train the ML models. However, given that the API needs to be known by the network node in order to use the ML model, the network node's software needs to be updated each time a new ML model functionality that changes the API is developed.

[0030] Accordingly, in one aspect there is provided a method for deploying and/or updating a model. The method includes sending or receiving a message comprising one or more of the following: i) opaque information comprising a software package implementing the model to be deployed or updated or ii) non-opaque information comprising property information indicating one or more properties of the model.

[0031] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of a network node causes the network node to perform any of the methods disclosed herein In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium In another aspect there is provided a network node that is configured to perform the methods disclosed herein. The network node may include memory and processing circuitry coupled to the memory.

[0032] An advantage of the embodiments disclosed herein is that they avoid explicitly revealing properties of the ML model that should not be disclosed, while exposing details of the model that are needed to make the interaction amongst nodes participating in the procedures associated to the model (e.g., nodes that provide inputs or feedback information) work. For example, non-opaque information such as outputs generated by the model, allow the node hosting the model to know that such outputs might need to be signaled over open interfaces. Similarly, information about the inputs needed by the model allows the node hosting the model to request over open interfaces the input information needed to run the model. Furthermore, the embodiments allow the deployment and update of ML models as “black boxes” while providing the necessary information (e.g., API structure, inputs/outputs) to use them successfully.

BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0034] FIG. 1 illustrates the current 5G RAN (a.k.a., the Next Generation RAN (NG-RAN)) architecture

[0035] FIG. 2 illustrates a Functional Framework for RAN Intelligence.

[0036] FIG. 3 illustrates the introduction of a model management function in the Functional Framework for RAN Intelligence.

[0037] FIG. 4 is a message flow diagram according to an embodiment

[0038] FIG. 5 is a message flow diagram according to an embodiment.

[0039] FIG. 6 is a message flow diagram according to an embodiment.

[0040] FIG. 7 is a message flow diagram according to an embodiment.

[0041] FIG. 8 is a message flow diagram according to an embodiment.

[0042] FIG. 9 is a flowchart illustrating a process according to an embodiment.

[0043] FIG. 10 is a block diagram of a network node according to an embodiment.

DETAILED DESCRIPTION

[0044] Terminology [0045] Transparent information is data of a data type whose data structure is defined in an interface. Specified information is information that is visible, i.e. , it is specified, over the signaling interface through which the information travels, i.e., an information whose semantic meaning is known (or specified) for the interface carrying the information. A node terminating the interface and receiving such information is able to decode it simply by means of implementing the interface standardized design. Transparent information is synonymous with “specified information,” “visible information,'' “non-opaque information," and “information in non-opaque format ''

[0046] Opaque information is data of an opaque data type (e.g., OCTET STRING), where an opaque data type is a data type whose data structure is not defined in the interface. That is, opaque information is information travelling across an interface whose meaning is not openly specified. This type of information may also be referred to as information not visible to the interface. Any node terminating the interface and receiving such type of information over the interface is able to decode the information only if it knows, by proprietary means, how such decoding should take place. An example of implementation of opaque information in a 3GPP interface (e.g., NGAP, or XnAP, or F1AP) is an Information Element (IE) encoded as an OCTET STRING. The IE may be used to convey content (e.g., an XML file, a serialized structured data (e.g., with Google Protocol Buffer), a string of bits) whose semantic meaning is not specified for the interface. Opaque information may be obtained after one or more security techniques (such as data encryptions, data scrambling, signatures) are applied to an original (plain-text) information (e.g., the original ML model to be deployed). Opaque information is synonymous with “unspecified information,” “information in opaque format," “nonvisible information,” or “obscure information."

[0047] As used herein a “network node” can be a node in a radio access network (RAN), an 0AM, a Core Network node, an 0AM, an SMO, a Network Management System (NMS), a Non-Real Time RAN Intelligent Controller (Non-RT RIC), a Real-Time RAN Intelligent Controller (RT-RIC), a gNB, eNB, en-gNB, ng-eNB, gNB-CU, gNB-CU-CP, gNB-CU-UP, eNB-CU, eNB-CU-CP, eNB-CU-UP, lAB-node, lAB-donor DU, lAB-donor-CU, IAB-DU, IAB-MT, O-CU, O-CU-CP, O-CU-UP, O-DU, O-RU, O-eNB, a UE. References to network nodes herein should be understood such that a network node may be a physical node or a function or logical entity of any kind, e.g. a software entity implemented in a data center or a cloud, e.g. using one or more virtual machines, and two network nodes may well be implemented as logical software entities in the same data center or cloud.

[0048] As used herein a “RAN node” is a node or entity in a radio access network (RAN), such as a gNB, eNB, en-gNB, ng-eNB, gNB-CU, gNB-CU-CP, gNB-CU-UP, eNB-CU, eNB-CU-CP, eNB-CU-UP, lAB-node, lAB- donor DU, lAB-donor-CU, IAB-DU, IAB-MT, O-CU, O-CU-CP, O-CU-UP, O-DU, O-RU, or O-eNB

[0049] A function, or entity of a RAN node is to be intended as one of the functional entities comprised in a RAN node. A RAN node in split deployment comprises different functions or entities. For example, a gNB comprising a gNB-CU, one or more gNB-DU, one or more gNB-CU-CP. [0050] The terms model training, model optimizing, model optimization, model updating are herein used interchangeably with the same meaning unless explicitly specified otherwise.

[0051] The terms model changing, modify or similar are herein used interchangeably with the same meaning unless explicitly specified otherwise. In particular, they refer to the fact that the type, structure, parameters, connectivity of a model may have changed compared to a previous format/configuration of the model.

[0052] The terms Al model, ML model, AI/ML model, and AIML model are herein used interchangeably with the term model.

[0053] Data collection refers to a process of collecting data for the purpose of model training, data analytics, and/or inference.

[0054] The embodiments disclosed herein are independent with respect to specific AI/ML model types or learning problems/setting (e.g. supervised learning, unsupervised learning, reinforcement learning, hybrid learning, centralized learning, federated learning, distributed learning, ...). Non limiting examples of AI/ML algorithms may include supervised learning algorithms, deep learning algorithms, reinforcement learning type of algorithms (such as DQN, A2C, A3C, etc.), contextual multi-armed bandit algorithms, autoregression algorithms, etc., or combinations thereof. Such algorithms may exploit functional approximation models, hereafter referred to as AI/ML models, such as neural networks (e.g. feedforward neural networks, deep neural networks, recurrent neural networks, convolutional neural networks, etc ). Examples of reinforcement learning algorithms may include deep reinforcement learning (such as deep Q-network (DQN), proximal policy optimization (PPO), double Q- learning), actor-critic algorithms (such as Advantage actor-critic algorithms, e g. A2C or A3C, actor-critic with experience replay, etc), policy gradient algorithms, off-policy learning algorithms, etc.

[0055] The network nodes described herein can be directly connected (i.e., with a signaling connection between them) or indirectly connected (i.e., with a signaling connection traversing intermediate network nodes between them, relaying the signaling information exchanged between them). Additionally, as used herein transmitting a message to an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e , one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message from a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node).

[0056] Introduction [0057] This disclosure describes solutions to deploy and/or update ML models in a RAN node (e.g., gNB, gNB-DU, gNB-CU-CP), enabling the possibility of differentiating operations concerning ML models signaled as opaque information (e.g., as OCTET STRING) or as transparent information.

[0058] In a possible realization of the methods described herein, the RAN node where the ML models are deployed or updated is mapped to the ML model inference function within the Functional Framework for RAN Intelligence defined in 3GPP TR 37.817.

[0059] The embodiments disclosed herein describes a process for deploying and/or updating ML models in network nodes. In one embodiment, the process includes sending to a network node (e.g. RAN node) a first message related to the deployment or updating of an ML model, wherein the first message includes opaque information for the ML model. The opaque information may comprise a software package (i.e. , one or more software programs or applications) which includes the ML model. In some embodiments, the first message also includes transparent information for the ML model. The transparent information may comprises parameters or characteristics of the ML model needed to use it properly. In another embodiment, the process includes sending a second message to the network node, wherein the second message comprises the transparent information, but not the opaque information.

[0060] In one example, when an ML model is deployed or updated, a first network node (e.g., an 0AM node or a CN node or a RAN node) sends to a second network node the ML model itself and additional information associated to the ML model. The ML model is sent in an unspecified format (i.e., is sent as opaque information), where the whole of the model or at least parts of it are not specified. Namely, decoding of part or all of the model is only possible to an entity that is aware of how the model was encoded in the first place.

[0061] Additional information associated with the ML model can be: 1) sent as opaque information (e.g., OCTET STRING); 2) sent as transparent information; or 3) not sent at all (e.g., hardcoded, e.g. detailed in a technical specification, or hardcoded in a RAN node, or configured by other systems). Such additional information associated with the ML model could be signaled either together with the model (e.g., as part of the same message or procedure) or separately (e.g., with a separate message or procedure).

[0062] In another example, the ML model and additional information associated to the ML model could both be signaled as specified information.

[0063] The additional information associated to the ML model concerns characteristics (or properties) of the ML model, examples of which are described herein.

[0064] Embodiments for network nodes [0065] In a communication network, the following operations are executed to deploy or update an ML model in a RAN node (e.g. , a gNB, a gNB-CU or a gNB-DU).

[0066] (Operation 1) Determining that an ML model is to be deployed or is to be updated for a first network node (e.g., a RAN node). The decisions concerning the deployment and/or the update can be taken by the same network node or by different network nodes (e.g., a second network node determining that deployment is needed, and a third network node determining that update is needed).

[0067] (Operation 2) Determining/obtaining a mapping of a first set of one or more properties of the ML model to the ML model itself and/or to a second set of one or more properties of the ML model. The mapping/obtaining can be determined by only one network node (e.g., a second network node, that can be an Element Manager, or a third network node, that can be a Network Manager, or a RAN node), or more than one network node. The determination can be performed for the case of new deployment as well as for the case of an update.

[0068] In one embodiment, a mapping/determining is determined from a Model ID to the ML model to be deployed/updated, so that a specific Model ID value is used to indicate a specific ML model that is deployed/updated. In this example, the Model ID may uniquely identify the model within an PLMN or, in general, within a configured or preconfigured area scope, such as a Tracking Area, or the cells served by a RAN node. In one example, every time the model is updated, the Model ID is also updated, still maintaining uniqueness within the preconfigured area. In another example, every time the model is updated, the Model ID is kept unchanged, still maintaining uniqueness within the preconfigured area.

[0069] In another embodiment, a mapping is determined from a Model ID to a combination of "Model vendor” and “Model version”. The overall identifier may reflect the same uniqueness properties described above.

[0070] In another embodiment, a mapping is determined from the combination of a Model ID and a model version to the ML model to be deployed/updated, so that the combination of the specific Model ID and the model version is used to indicate a specific ML model that is deployed/updated. In this example, it is assumed that the model version is changed every time the model is updated, while the Model ID might stay fixed.

[0071] In some embodiments, multiple ML models may be deployed and updated at a given network node. Within such multitude of ML models deployed at the network node, more than one may apply to the same use case, namely more than one model may work with input information within a common superset (e.g., use case specific), produce outputs within a common superset (e.g., use case specific) and need feedback information within a common superset (e.g., use case specific). However, each of such models will be identified by different Model IDs. This will allow the network node hosting the deployed/updated models to label procedures and information relative to any of such specific models with the appropriate Model ID, so to let neighboring nodes receiving the information to understand that the information has been produced by means of using the model with the associated Model ID.

[0072] (Operation 3) Sending to a RAN node one or more messages to deploy or to update the ML model, the message(s) comprising one or more of the following: i) unspecified information for an ML model to be deployed or updated (e.g., the code constituting the model) or II) specified information for the ML model to be deployed or updated.

[0073] (Operation 4) Sending to a RAN node one or more messages comprising specified information comprising a mapping of a first set of one or more properties of the ML model to: a) the ML model itself and/or b) a second set of one or more properties of the ML model.

[0074] The first sending and the second sending can occur in the same message or in distinct messages. The first sending and the second sending can be executed by the same network node (e.g., second network node) or by different nodes (e.g., the second network node - for example an Element Manager - executing the first sending, and the third network node - for example a Network Manager - executing the second sending).

[0075] Unspecified information of an ML model can comprise: i) an ML model and/or an ML algorithm, ii) properties of an ML model not to be openly revealed over the signaling interface (e.g., a second set of one or more properties of the ML model), ill) additional support software packages and/or libraries required to execute the ML model of ML algorithm at the RAN node, iv) security certificates and/or other tools for validating/authenticating ML models.

[0076] Specified information of an ML model can comprise one or more or a combination of: i) one or more properties of an ML model; II) a first set of one or more properties of an ML model mapped to a second set of one or more properties of the ML model, wherein the second set of properties is not included (in one example, a Model ID is mapped to a certain Model Vendor and a certain use case, and neither the Model Vendor, nor the use case are explicitly part of the non-opaque information, but can be derived from the Model ID); ill) a first set of one or more properties of an ML model encoding a second set of one or more properties of the ML model, where the encoding can be partially masked (in one example, a Model ID is encoded in a way that contains the Model Vendor and the Model Version; the Model Vendor and the Model Version can be visible or at least in part visible over the signaling interface via the Model ID (e.g., the least "X” significant bits of the Model Version are masked (e.g., all bits set to "1”))). The non-opaque information described above may be signaled as opaque information, depending on implementation choices.

[0077] In one example, the non-opaque information comprises a Model ID, mapped (associated) to the use case for which the ML model is to be used. A RAN node receiving the opaque information (comprising the ML model), the non-opaque information comprising the Model ID, and capable to map the Model ID to the corresponding use case, can determine how to use the ML model. A RAN node receiving the same information but not capable to map the Model ID to the corresponding use case, can derive that the ML model is not usable.

[0078] In another example, the non-opaque information can comprise the use case (or a list of use cases) associated to the ML model. The receiver would be able to understand that, e.g., the ML model being deployed is for "Network Energy Saving” or for "Load Balancing Optimization”. This information may be already sufficient to determine the set of inputs, outputs, and feedback information relevant to the model, assuming that such information have been specified on a per use case basis.

[0079] In one example of deployment/update of an ML model, the RAN node may use the non-opaque information for determining an action to discard the opaque information containing the ML model. As an example, if the non-opaque Model ID contains a part (e.g., a number of bits) indicating a specific vendor and if the receiving node belongs to a different vendor, the information can be used to discard the opaque information consisting of the ML model.

[0080] In one example of deployment/update of an ML model, the RAN node may use the non-opaque information for determining an action to use I retrain I further modify the ML model comprised in the opaque information. The non-opaque information used could be the same of the previous example, e.g., a vendor index within the Model ID.

[0081] Considering the above, different variants are possible, based on:

[0082] a) whether there are properties of the ML model comprised in the unspecified (opaque) information of an ML model (and which properties they are);

[0083] b) whether there are properties of the ML model comprised in the specified (non-opaque) information of an ML model (and which properties they are);

[0084] c) whether there are properties of the ML model not signaled at all (e.g., hardcoded or configured), and which properties they are;

[0085] d) whether or not there is a mapping between a first set of one or more properties of the ML model, comprised in the non-opaque information of the ML model, and a second set of (other) one or more properties of the ML model, the second set either comprised in the opaque information or not signaled at all; and/or

[0086] e) how the mapping is realized between the first set of one or more properties of the ML model comprised in the non-opaque information of the ML model and the second set of (other) properties of the ML model.

[0087] Some non-limiting examples for encoding specified (non-opaque) information of an ML model are listed below. For simplicity, only one property (the Model ID) is used as non-opaque information: [0088] a) one property realizes an encoding of other property(ies); or example, the Model ID is a bit string of (X+Y) bits, where the first part of X most significant bits is an ML Model vendor identifier, the subsequent part of Y bits is the ML Model type identifier;

[0089] b) part of the encoding can be masked; for example, the Model ID is a bit string of (X+Y) bits, where the first X most significant bits correspond to the ML Model vendor identifier and the subsequent Y bits correspond to the ML Model type, while the last Z bits of the Model Type are all “1”;

[0090] c) a set of bits, octets, digits, where one part of the set corresponds to a first property, and a second part corresponds to a second property (potentially, only a portion of a property is visible);

[0091] d) a bitmap, where bits in different positions are associated to different operations (e.g., Lifecycle management operation) of the ML model; for example, if the bit at position “X” has value “1” it indicates that the operation in question is an update of an ML model; if the bit at position “Y” has value “1" it indicates that the ML model can be re-trained at RAN (otherwise, re-training at RAN is not allowed); if the bit at position “Z" has value “1” it indicates that the RAN node can upload a re-trained ML model to 0AM; and/or

[0092] e) a bitmap, where bits at different positions are associated to different use cases supported by the ML model; for example, if the bit at position “X” has value “1” it indicates that the ML model is for “Network Energy Saving” use case, if the bit at position “Y” has value “1” it indicates that the ML model is for “Load Balancing Optimization" use case.

[0093] The content of the unspecified (opaque) information and the specified (non-opaque) information can vary between the case of ML Model deployment and the case of ML Model update. This is because, apart from the part concerning the ML model itself, which is expected to change between “deployment” and “update”, also some properties of the ML model can change, e.g., the Model Version or the Model type, or the mapping between nonopaque information and opaque information can change. If, for simplicity we consider the Model ID as the means to provide the mapping, it is possible that the value of the Model ID used for model deployment and the value of the Model ID used for model update are different, or the value of the Model ID used for model deployment and the value of the Model ID used for model update are the same, but the mapping is different.

[0094] During an update, the second network node (or the third network node) can send to the RAN node where the ML model was deployed, the new mapping, and the network node (second or third network node) can also send an indication to indicate that the (new) model replaces the previous one.

[0095] In yet another example, the Model ID may represent a model deployed by the node hosting the model training function, while the Model Version may represent the version of the model initially deployed, where version number is increased every time the model is updated. [0096] The node that performs a model update (e.g. , model retraining) may generate a version number for the model identified by the Model ID. The node responsible for the update may signal the model ID and new version number to other nodes involved in the procedures concerning the ML model (e.g., node hosting the Model Training Function, nodes receiving model outputs). This guarantees that all nodes involved in the ML processes are aware that the model has been updated and that new outputs and new performance impacts may be experienced.

[0097] In yet another embodiment, the node hosting the model training function may deploy to the node hosting the model inference function a model with a new model ID. However, in the same signaling, the node hosting the model training function may include information stating that the new model is a replacement for an already deployed model with a different model ID. The receiving node would therefore deduce that the previously deployed model needs to be dismissed and replaced by the newly deployed model. Operations will resume using the newly deployed model with its associated model ID.

[0098] Property Information for an ML model

[0099] The property information for an ML model can be one or more or a combination of the following:

[00100] a) Identifiers of the ML Model, such as a Model ID, a model index, an ML Model Reference that uniquely identifies an ML model and obtained combining a Model ID together with one or more other network identities, such as a PLMN ID, a Job ID, a Trace ID, a transport network address (a model identifier, e.g. a Model ID can be realized in various ways, e.g., encoded as an INTEGER, an OCTET STRING, a BIT STRING, an ENUMERATED, a STRING, . .);

[00101] b) ML Model version, subversion;

[00102] c) ML Model vendor;

[00103] d) A Uniform Resource Locator (URL);

[00104] e) A Uniform Resource Identifier (URI);

[00105] f) An ML model name (e.g., in human readable format);

[00106] g) Type information indicating type of model, e.g., linear regression, feedforward neural network, convolutional neural network, recurrent neural network, graph neural network, autoencoder, transformer, etc.;

[00107] h) Time stamps, e.g., the production time of the ML model;

[00108] i) Information indicating a validity time (e g., expiry date) of the ML model;

[00109] j) Information indicating that the operation performed for the ML model is the deployment of a (new) model; [00110] k) Information indicating that the operation performed for the ML model is the update of a (previous) model;

[00111] I) Information indicating (e.g., retrainability flags) whether the ML model is re-trainable by external systems, e.g., by other RAN nodes or another network node (e.g. implementing a Real-Time RIG), in case the training function resides at the RAN, or by the RAN in case the training function resides at the 0AM;

[00112] m) If the ML model is retrained by an external system, information indicating of whether the retrained model should be signaled back to the node/system hosting the training function and that deployed the model in the first place;

[00113] n) Information indicating concerning how re-training of the ML model can be achieved. For example, a RAN node is allowed to retrain, of 0AM is allowed to retrain, or a RAN node can trigger 0AM to retrain, or a RAN node can trigger another RAN node to retrain, or automatic/ re-training at the ML model inference host (e g., a RAN node) is supported, or re-training triggered or enabled by ML model inference host (e.g., a RAN node) is supported (the latter two cases are explained below);

[00114] o) Information indicating whether the network node/system performing retraining, e.g., RAN node (or the function in the RAN node), is allowed or requested or configured to upload a re-trained model (e.g., to 0AM or SM0);

[00115] p) Information indicating whether use of the ML model has any impact on user consent. Namely, whether any of the information concerning the model, such as input data, output data, feedback data etc. are subject to consent from the user whose UE is involved in generating such data;

[00116] q) Information indicating security aspects related to the ML model, such as whether the outputs are encrypted according to a specific encryption mechanism, and/or whether anonymization of data that can be associated to a UE is or is to be carried out, and/or certificates;

[00117] r) Information indicating whether ML model can be revoked by the node/system that receives it, namely if the receiving node can stop using the model;

[00118] s) Information indicating whether ML model can be revoked by the node/system that deployed it, namely if the node that deployed it can command to stop using the model;

[00119] t) Information indicating which Lifecycle Management operations (e.g., validating, testing, monitoring, ...) a network node/system, e.g., a RAN node (or the function in a RAN node) is allowed or not allowed or requested or configured to perform for the ML model; [00120] u) Information indicating whether the RAN node where the model is deployed/updated is allowed or requested or configured to transfer/share a re-trained model;

[00121] v) Information indicating use case(s) (a.k.a. use case indicators) for which the ML model can be used, e.g. , scenarios or radio network functions for which the ML model has been trained or can be further retrained. Non-limiting examples of use case can be: Network Energy Saving, Power Saving, Load Balancing Optimization, Mobility Optimization, Link Adaptation, QoS Optimization, QoE Optimization, Coverage and Capacity Optimization, MIMO Layer Optimization, CSI Prediction, Beam management, Positioning, Channel Coding, Reference Signal Enhancements, Interference Optimization,

[00122] w) Information indicating an ML algorithm type (e.g., supervised learning, unsupervised learning, reinforcement learning);

[00123] x) Information indicating a type of ML training (e.g. hybrid learning, centralized learning, federated learning, distributed learning);

[00124] y) ML Model signature / checksum, e.g., derived from the model parameters, enabling a model integrity check;

[00125] z) Information indicating services or service categories for which the ML model can be used (e.g., Mobile Broadband, Extended Reality, Time Sensitive Communication, Industrial loT, Satellite communications, V2X, Gaming, ...);

[00126] aa) Information indicating inputs of the ML model inference function, in different variants, such as: a set of inputs the ML model needs, recommended inputs, minimum inputs for a workable ML model, minimum inputs required to achieve a certain level of performance, minimum set of inputs needed to produce a minimum set of outputs, mandatory inputs, optional inputs,;

[00127] bb) Information indicating inputs that are mandatory or optional to make the ML model workable, or workable with some level of performance (namely, out of the list of inputs specified to be needed by the model for best performance, an indication of the inputs that must be available for the model to function with a specific performance, e g., accuracy) Note: The minimum set of inputs comprises inputs that are necessary to produce the minimum set of outputs (with a certain minimum performance, e.g., accuracy). In case the performance is provided as part of the 0AM to RAN signaling, it may be specified per output for the case that only the minimum set of inputs is available/used and/or for the case that the superset of inputs is avail able/used. If not all inputs defined by the superset can be made available, or if it's too costly to acquire all inputs defined by the superset, the RAN node may need to run the ML model in "basic mode” using only the minimum set of inputs. Similarly, in case one or more computational requirements are provided as part of the 0AM to RAN signaling, they may be specified for the case that only the minimum set of inputs is avail able/used and/or for the case that the superset of inputs is available/used.

Note that missing values may need to be replaced appropriate (placeholder) values as explained below;

[00128] cc) Information indicating a superset of inputs the ML model may need. Namely, this is not the list of exact inputs needed by the model, but a bigger (more comprehensive) list. Once the RAN node decodes the model, the RAN will be aware of the exact inputs required;

[00129] dd) An indication/instruction of how to deal with missing inputs, i.e., whether to provide a substitute/replacement value for a missing input to the ML model inference function, e.g., the mean/median or another statistic of the missing input, or a marker/flag value, e.g., NULL, or -1 E308, or not provide any value for the missing input. In the latter two cases, where the ML model inference host must not provide a substitute value to the ML model inference function, the ML model can be expected to replace missing inputs by itself or be able to cope with the missing input in another way;

[00130] ee) Information indicating outputs generated by the ML model inference function, in different variants, such as: a set of output the ML model will generate, whether some outputs are mandatorily or optionally produced by the ML model (namely, out of the list of outputs specified to be produced by the model, an indication of the inputs that must or may be produced by the model), recommended outputs, minimum outputs, mandatory outputs, optional outputs, output generated when only a minimum set of input is provided, set of output available when the superset of input is available, recommended/suggested actions,;

[00131] ff) Information indicating a superset of outputs the ML model may generate. Namely, this is not the list of exact outputs generated by the model, but a bigger (more comprehensive) list. Once the RAN decodes the model, the RAN will be aware of the exact outputs generated;

[00132] gg) An indication that “basic mode" operation (or alike) is used when only using a minimum set of inputs;

[00133] hh) Information indicating feedback information relevant and useful to retrieve ML model performance analysis;

[00134] ii) Information indicating whether some feedback information is required, mandatory or optional, to assess/monitor the ML model performance and/or to update/improve the ML model. Namely, out of a list of feedback information specified to be of relevance for the model, an indication of the feedback that must be available to enable ML model performance monitoring and/or improvement. In case the ML model is executed by a RAN node and updated or deployed by a second or a third network node, the required feedback information should be provided to the second or third network node, or to a network node hosting a data collection functionality for training data collection; [00135] jj) Information indicating a ype of feedback associated to output values or actions produced by the

ML model;

[00136] kk) Indication concerning memory requirements, required computational power, required support programs and/or libraries, and other requirements for the platform that needs to be used for running the ML model. Such requirements (individually or cumulatively) may be indicated in absolute values, or as a score, or as index, or a weight;

[00137] II) An index indicating the computational requirements of the model, where for example, the index could range from 0 to 100 and where 0 means no computational requirements, while 100 means maximum computational requirements. Computational requirements may apply to processing power requirements or memory requirements or similar requirements/metrics;

[00138] mm) Indication of requirements (e.g., computational requirements) when a minimum set of input is available/used and/or when the superset of inputs is available/used;

[00139] nn) An indication of performance of the ML model provided to the first RAN node, such as accuracy and/or uncertainty/confidence (interval) with which model output predictions are generated. Possibly the accuracy and/or uncertainty/confidence interval may be provided per range of input values;

[00140] oo) An indication of accuracy and/or uncertainty/confidence when a minimum set of input is used. As stated above, the accuracy and/or uncertainty/confidence interval may be provided per range of input values;

[00141] pp) Information indicating uncertainties of the model output, i.e., if the model produces a measure of uncertainty (e.g., a confidence interval) along with its outputs;

[00142] qq) Information indicating uncertainties of the model input, i.e., if the model accepts/expects a measure of uncertainty (e.g., a confidence interval) along with its inputs and, if so, what type of uncertainty metric;

[00143] rr) Information indicating the format with which one or more model output is produced, such as integer, floating-point number/value, Boolean, etc. In addition, the output format may include the number of bytes or bits used to represent the output;

[00144] ss) Information indicating the API protocol or architecture used to communicate with the ML model, e.g., representational state transfer (REST) architecture, remote procedural call (RPC) protocol, simple object access protocol (SOAP);

[00145] tt) Information indicating the API language used to communicate with the ML model, e g., JavaScript Object Notation (JSON), Extensible Markup Language (XML), HyperText Markup Language (HTML), plain text; [00146] uu) Information indicating the way the data is structured in the API messages, e.g., the API schema;

[00147] vv) An indication or a configuration of how training data should be collected for the AIML model. This may comprise one or more information related to. (the format to be used for training data samples, such as the type of information that should be aggregated to construct a training data sample. This may include the state (or input) used by the model inference function, the model inference output, an action determined using the model inference output (e.g., a change in one or more a network or user parameter value), one or more measurement associated to the model inference output (e.g., measurements of information predicted by the model inference, which could be carried out by one or more network node, one or mode UEs or combinations thereof));

[00148] ww) An indication or a configuration associated to an exploration strategy for the AIML model or algorithm used for controlling one or more network or user parameters;

[00149] xx) An indication of the data formatting required to execute the AIML model, i.e , required by the model inference function. Examples may include one or more information to scale/de-scale, normalize/de-normalize input or output data of the ML model prior to or after executing the model inference function.

[00150] As described above, in some embodiments, the ML model is received as an unspecified payload (e.g., as a software package) comprising the ML model and that it is not specified how to decode the ML model or the unspecified payload as such Hence, the node receiving the ML model may not be able to decode the ML model, but it can interact with the ML model (e.g., it could interact with the ML model via specific Application Program Interfaces (APIs) provided for this purpose, e.g., inference, re-training, etc ). Moreover, the node also receives information about the one or more APIs to interact with the ML model. The procedure to perform inference and re-training may be part of the procedures for such APIs. The node will be instructed on how to carry out such procedures. The signaling of information concerning the APIs may be part of the signaling used to deploy the ML model.

[00151] Re-training performed automatically by ML Software Package (not triggered by the ML model execution/inference host)

[00152] As noted above, a node receiving the ML model may also receive an indication (in form of properties of the ML model) that the ML model is re-trained automatically by the ML software package. If this is the case, in one embodiment, the node receiving a trained ML model understands that the model re-training is triggered and carried out automatically by the ML software package. Moreover, the ML model execution/inference host cannot influence the re-training and it is unaware of the details. The re-training is fully controlled by the ML software package. The ML model is seen as a black box, which re-trains/updates itself. This also means that re-training cannot be triggered by the inference host, but is triggered by the ML software package, or it may be done, e.g., whenever new training data is available. [00153] Automatic re-training further implies that the inference data, which is assumed to be provided (by the inference host to the software package) as inputs via a specific API to receive outputs, comprises the training data. In one example, the inference data comprises historical measurements of a metric (e.g., quantity) that is to be predicted by the ML model for a time in the future. In this case, the inference data inherently comprises the ground truth of outputs (e.g., predictions) generated by the ML model at an earlier point. The ML software package can use that data to re-train/update the ML model. In fact, the inference host can understand that there has been an update of the ML model by means of a new Model ID (or version number). Note that if the inference data does not inherently comprise new training data, it may do so, e.g., based on instruction from the node providing (e.g., sending) the ML model or by definition in specifications.

[00154] Re-training triggered or enabled by ML model execution/! nference host

[00155] In many cases, inference data does not contain training data because training data includes information that is not needed to perform inference (e.g., ground truth, reward signals, etc). In those cases, the node receiving the ML model may receive an indication (again, in form of properties of the ML model) that the ML model re-training must be triggered or enabled by the ML model execution/i nference host by means of providing the required new training data through a separate re-training API, if needed. As mentioned above, the signaling must indicate what data is needed for the (re-)training (the same applies for inference).

[00156] The re-training is triggered or at least enabled by the execution/i nference host by providing (e.g., feeding) new training data, e.g., via the intended API, so the execution/inference host can decide if and when to retrain the ML model and can even influence the re-training (to a certain extent), because it can decide what training data to feed into the ML software package. In one example, it can decide to feed more training samples from input (or output) ranges where the ML model performance is worse or is of particular importance for the (overall) performance at or in the vicinity of the local node. The inference host can trigger re-training based on instruction (e.g., configuration) from the node providing the ML model or even another node. It may trigger re-training, e.g., based on observed ML model performance degradation or system performance degradation when monitoring certain performance metrics, e.g., system KPI.

[00157] Upon receiving new training data, e.g., via a specific API, the ML software package re- trains/updates the ML model internally, i.e., the details of the ML model remain hidden from the inference host. Similar to the above, the inference host can understand that the update of the ML model has been completed when receiving a new Model ID (or version). The main difference between the automatic re-training case and this case is that, in this case, action from the inference host is needed to enable the re-training.

[00158] Example Embodiments [00159] FIG. 4 illustrates a use case in which a RAN node 401 receives from a second network node 402 an optional configuration message m450 that comprises configuration information that maps a Model ID to particular model properties. For example, message m450 includes the Model ID and property information indicating the particular model properties. For instance, the message may include an IE that comprises the Model ID and the property information and indicates that the property information is associated with the Model ID. After receiving message m450 RAN node 401 stores in a storage unit the Model ID and the property information such that the Model ID is associated with (e.g., linked to) the property information so that the RAN node can later retrieve from the storage unit the property information using the Model ID as the search query.

[00160] As shown in FIG. 4 RAN node 401 receives from a third network 403 node a deployment message m452 that comprises i) opaque information that comprises a software package (SP) implementing an ML model and ii) transparent information that comprises a Model ID (which in this example is the same Model ID as was included in configuration message m450). In some embodiments, the message may also include i) a use case indicator that provides indication of the use case for which the ML model is used (e g., “Network Energy Saving” or “Load Balancing Optimization”, or “Mobility Optimization”) and/or ii) additional property information (e.g., information indicating one or more inputs, one or more outputs, one or more feedbacks).

[00161] In some embodiments, after receiving deployment message m452, RAN node 401 uses the Model ID from deployment message m452 to retrieve from its storage unit the property information associated with the Model ID, if any. RAN node 401 then uses the retrieved property information (hereafter “model properties"), if any, and the model properties included in deployment message m452, if any, to execute the software package and/or provide feedback information pertaining to the ML model.

[00162] Additionally, in some embodiments in which deployment message m452 comprises the use case indicator, after receiving deployment message m452, RAN node 401 uses the use case indicator to obtain model properties that are associated with the use case indicator, which model properties are for the ML model implemented by the software package.

[00163] In one embodiment, the use case indicator is associated with (e.g., points to): a) the full list of inputs/outputs/feedback information the ML model needs; b) a standardized set of outputs that the ML model is expected to produce (the set of inputs may be left to implementation); c) a set of standardized inputs (the outputs may be left for implementation); and/or d) a set of standardized outputs conditionally present, i.e., expected only if a set of standardized input is provided. In one specific example, a 3GPP standard can indicate a list of inputs, a list of outputs, and/or a list of feedbacks for a given use case indicator. For example, for a use case indicator that indicates the Energy Saving use case: [00164] i) the inputs to the ML model include: a) UE location information (e.g., coordinates, serving cell ID, moving velocity) interpreted by gNB implementation when available; b) UE measurement report(s) (e.g., UE RSRP, RSRQ, SINR measurement, etc.), including cell level and beam level UE measurements; input from neighbouring RAN nodes (e.g., Current/Predicted energy efficiency, Current/Predicted resource status, and/or Current energy state (e g., active, high, low, inactive));

[00165] ii) the feedback information for the ML model include: a) Resource status of neighbouring NG-RAN nodes; b) energy efficiency; c) UE performance affected by the energy saving action (e.g., handed-over UEs), including bitrate, packet loss, latency; and/or d) system KPIs (e.g., throughput, delay, RLF of current and neighbouring NG-RAN node).

[00166] In a variant of the example above, the 3GPP standard the mapping can be expressed indicating that different sets of input I output I feedback can be used for one or more Use cases, e.g.: a first set of input maps to Use Case “A” and "B”; a second set of input maps to Use Case "C”; ...; an Nth set of input maps to Use Case "N”; a first set of output maps to Use Case “A” and “B”; a second set of input maps to Use Case “C”; a first set of feedback maps to Use Case ‘‘A” and “B”; and/or a second set of feedback maps to Use Case “C”.

[00167] After obtaining the model properties for the indicated use case based on the use case indicator, RAN node 401 uses the model properties for the given use case plus the model properties from configuration message m450, if any, plus the model properties from deployment message m452, if any, to execute the software package and/or provide feedback. For example, in some embodiments, for the signaled use case (e.g., “Network Energy Saving”), the RAN node is able to determine a set of standardized inputs, outputs, and feedback information. Whether the ML model uses a reduced set of such inputs, outputs, and feedback (or additional input I output I feedback) is up to implementation. What is relevant for the example is that standardized inputs/outputs/feedback are known without signaling them explicitly.

[00168] FIG. 4 further shows that second network node 402 may send to RAN node 401 a second configuration message m454 that comprises configuration information that maps the Model ID to particular model properties. Upon receiving this second configuration message, RAN node 401 may either i) store in the storage unit the model properties from configuration message m454 so that these particular model properties and the particular property information included in configuration message m450 are both associated with the Model ID or ii) replace in the storage unit the particular property information from configuration message m450 with the particular property information from configuration message m454. In this way, second network node 402 can reconfigure RAN node 401.

[00169] After second network node 402 sends the second configuration message m454, third network node 403 may send a second deployment message m456 that comprises i) opaque information that comprises an updated software package implementing an updated version of the ML model and ii) transparent information that comprises a Model ID (which in this example is the same Model ID as was included in configuration messages m450 and m454).

[00170] In some embodiments, after receiving second deployment message m456, RAN node 401 uses the Model ID from deployment message m456 to retrieve from its storage unit the particular property information that are associated with the Model ID and then uses the retrieved model properties (as well as model properties included in second deployment message m456, if any, and/or model properties derived from the use case indicator, if any, to execute the updated software package and/or provide feedback regarding the updated ML model.

[00171] While FIG. 4 shows that second network node 402 is separate and distinct from third network node 403, in some embodiments they are the same network node.

[00172] FIG. 5 illustrates a use case in which a ML model is deployed as a software payload using a centralized resolving function (CRF) 506 (a.k.a., ML deployment resolver). As shown in FIG. 5, third network node 403 transmits to RAN node 401 a deployment message m550 comprising opaque information comprising a software package that implements an ML model (deployment message m550 may be identical to deployment message m452 or m456). The deployment message m550 includes a Model ID or includes sufficient information to enable RAN node 401 to derive a model ID (e.g. RAN node 401 may be able to obtain or derive the Model ID from the software package included in deployment message m550).

[00173] After receiving deployment message m550, RAN node 401 extracts the software package included therein and associated information (e.g., Model ID), if any, that is transferred together with the software package. If the Model ID cannot be extracted from the message, then RAN Node derives a Model ID.

[00174] In this example, model properties (e.g., a use case indicator) that is/are needed to execute the ML model and/or provide feedback is/are not included in deployment message m550. Accordingly, RAN node 401 uses the extracted/derived Model ID to obtain the necessary model properties by transmitting to CRF 506 a model property request message m552 that comprises the Model ID. CRF 506, in response to receiving the model property request message m552 uses the Model ID included in the message to locate and retrieve a set of one or more model properties associated with the Model ID and transmits to RAN node 401 a response message m554 responsive to request message m552, where the response message m554 comprises the set of model properties (e.g., use case indicator, model input information, model output information, and/or model feedback information). After receiving message m554, RAN node execute the software package using the received model properties.

[00175] While FIG. 5 shows that CRF 506 is separate and distinct from third network node 403, in some embodiments they are the same network node. Similarly, while FIG. 5 shows that CRF 506 is separate and distinct from RAN node 401, in some embodiments they are the same network node [00176] FIG. 6 illustrates an example in which an ML model is requested by RAN node 401 (or other network node) after receiving a message comprising information pertaining to the ML model (e.g., the Model ID for the model). For example, as shown in FIG. 6, RAN node receives message m650 and message m650 comprises information pertaining to the ML model (e.g., non-opaque information concerning the ML model). Examples of such non-opaque information can be: the ML model's Model ID, a combination of Model ID and Model version, a combination of Model ID and Model vendor, a Model vendor, an ML-based use case, an indication that use of ML in RAN is of interest, an indication that an ML model (without any specific indication of a model identifier) is available for the network node to be used.

[00177] After receiving message m650, RAN node 401 determines to request a new or updated Model corresponding to the non-opaque information (e.g. the ML Model ID). Accordingly, RAN node 401 transmit a request message (a.k.a., query message) m652 to another network node 602 (e.g. the third network node or an ML deployment resolver) with at least part of the received non-opaque information (e.g. the ML Model ID).

[00178] Network node 602, in response to receiving query message m652 uses information included in message m652 to obtain a software package that implements the ML model and then transmits to RAN node 401 a deployment message m654 that comprises opaque information comprising the software package. Message m654 may further comprise ML model properties (in opaque format or non-opaque format).

[00179] FIG. 7 illustrates a successful ML model update example. In the figures below it is assumed that the node hosting the training function is an OAM 702, while the node hosting the inference function, which carries out the retraining, is RAN node 401 . As shown in FIG. 7, RAN node 401 sends to OAM 702 a model update message m750 to inform the OAM that the ML model signaled as an undefined payload (e.g., as an OCTET STRING) has been retrained. Model update message m750 may also include one or more of the following:

- A Model ID and potentially model version, uniquely identifying the ML model and model version before retraining

Some form of indication of the range or some statistics of the training data used, for example:

- Ranges of training data values used, such as value ranges of predicted metrics and actual measurements for the same metrics at the time the prediction was assumed to be valid,

- Nodes/systems providing the training data used, e.g., training data collected from a selected number of cells or a selected group of UEs with specific characteristics.

Optionally performance of the updated model in terms of, for example:

- one or more performance measurements for the model outputs, e.g , accuracy

- key performance indicators associated to actions that are influenced by the model outputs

A new Model ID identifying the model after re-training (if applicable) For example, if the RAN re-training changed the ML model and a new Model ID is/was required to be assigned to it. For example, the ML 1 software package may have updated the ML model internally, assigned a new Model ID and informed the execution/! nference host about the new Model ID.

- A new model version identifying the model after re-training. Similar to the above, if the RAN re-training caused a change of the model version, e.g., if the ML software package has updated the ML model and informed the execution/inference host about the new model version (number).

- An indication of whether the re-trained model is active or not, namely whether the model is being currently used to influence actions and decisions within the system or not.

The new ML model as an undefined payload (e.g., as an OCTET STRING).

[00180] If message m750 includes the performance information mentioned above, then the 0AM assesses this information and responds to NG-RAN that it can use the re-trained model (i.e. , the 0AM 702 transmits to RAN node 401 a model update response message m752 comprising information indicating that RAN node 401 can use the re-trained model).

[00181] If message m750 does not include the performance information mentioned above, then the OAM performs its own evaluation of the updated model, for example, by means of collecting system KPIs that may reveal how well the model in use is performing, and based on that responds to NG-RAN that it can use the retrained model.

[00182] In some cases, the model performance can only be assessed for models that are active (e.g., in use in the network). If a model is inactive, the OAM may not receive any model performance indication/metric, however, the OAM may still be in charge of deciding whether the new model should be kept or discarded.

[00183] After receiving model update response message m752, RAN node 401 provides model performance information to OAM after running the updated model by transmitting to the OAM model performance information message m754 comprising the model performance information.

[00184] FIG. 8 illustrates an unsuccessful ML model update example. As shown in FIG. 8, RAN node 401 sends to OAM 702 the model update message m750 as described above.

[00185] If message m750 includes the performance information mentioned above, then the OAM assesses this information and responds to RAN node 401 that it rejects the retrained model with an appropriate cause (i.e., the OAM 702 transmits to RAN node 401 a model update reject message m852 comprising information indicating that the retrained model is rejected and, optionally, information indicating a reason why it was rejected).

[00186] If message m750 does not include the performance information mentioned above, then the OAM performs its own evaluation of the updated model and based on that responds to RAN node 401 that it rejects the retrained model with an appropriate cause. [00187] FIG. 9 is a flow chart illustrating a process 900, according to an embodiment, for deploying and/or updating a model. Process 900 may begin in step s902. Step s902 comprises sending or receiving a message comprising one or more of the following: opaque information comprising a software package implementing the model to be deployed or updated, or non-opaque information comprising property information indicating one or more properties of the model. For example, the message sent/received in steps s902 may be one of: message m450, message m452, message m454, message m456, message m550, message m552, message m554, message m650, message m652, message m654. Hence, process 900 may be performed by network node 401 , network node 402, network node 403, network node 506 (a.k.a., CRF).

[00188] In some embodiments, the message comprises both the opaque information and non-opaque information.

[00189] In some embodiments, the message comprises the opaque information but not the non-opaque information.

[00190] In some embodiments, the message comprises the non-opaque information but not the opaque information. In some embodiments, the property information comprises a model identifier (ID) for the model or information from which the network node can derive the model ID.

[00191] In some embodiments, the property information further comprises additional property information, the process comprises receiving the message, and the process further comprises storing the model ID and the additional property information such that the model ID can be used to locate and retrieve the additional property information. In some embodiments the process further comprises, after storing the model ID and the additional property information, receiving a deployment message (e.g., message m452, m456, m550) comprising opaque information comprising a software package implementing the model to be deployed or updated; using the model ID to retrieve the additional property information; and using the addition property information to run the software package. In some embodiments, the method further comprises, before using the model ID to retrieve the additional property information, obtaining the model ID from the deployment message or deriving the model ID using information included in the deployment message.

[00192] In some embodiments, the process comprises receiving the message containing the non-opaque information (e.g., the message is received by network node 401), and the process further includes transmitting to another network node (e.g., a CRF 506, network node 402, network node 403) a request message comprising the model ID; and receiving from the another network node a deployment message that is responsive to the request message and comprises the opaque information comprising a software package implementing the model to be deployed or updated. [00193] In some embodiments the process comprises receiving the message containing the opaque information (e.g., the message is received by network node 401) and the process further includes obtaining a model ID from the message or deriving the model ID using information included in the message; and using the model ID to obtain property information for the model. In some embodiments using the model ID to obtain property information for the model comprises: transmitting to another network node (e.g., network node 402, network node 403, CRF 506) a request message comprising the model ID; and receiving a response message transmitted by the another network node, the response message being responsive to the request message and comprising property information for the model.

[00194] FIG. 10 is a block diagram of network node 1000, according to some embodiments. Network node 1000 can be used to implement any of the network node described herein, such as, for example, RAN node 401 , second network node 402, third network node 403, CRF 506. As shown in FIG. 10, network node 1000 may comprise: processing circuitry (PC) 1002, which may include one or more processors (P) 1055 (e.g., one or more general purpose microprocessors and/or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e. , network node 1000 may be a distributed computing apparatus); at least one network interface 1048 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1045 and a receiver (Rx) 1047 for enabling network node 1000 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 1048 is connected (physically or wirelessly) (e.g., network interface 1048 may be coupled to an antenna arrangement comprising one or more antennas for enabling network node 1000 to wirelessly transmit/receive data); and a storage unit (a.k.a., "data storage system") 1008, which may include one or more non-volatile storage devices and/or one or more volatile storage devices. In embodiments where PC 1002 includes a programmable processor, a computer readable storage medium (CRSM) 1042 may be provided. CRSM 1042 may store a computer program (CP) 1043 comprising computer readable instructions (CRI) 1044. CRSM 1042 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CR1 1044 of computer program 1043 is configured such that when executed by PC 1002, the CRI causes network node 1000 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, network node 1000 may be configured to perform steps described herein without the need for code. That is, for example, PC 1002 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software.

[00195] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[00196] Additionally, as used herein, transmitting a message to a device encompasses transmitting the message directly to the device or transmitting the message indirectly to the device (i.e., one or more nodes are used to relay the message from the source to the device). Likewise, as used herein, receiving a message from a device encompasses receiving the message directly from the device or indirectly from the device (i.e., one or more nodes are used to relay the message from the device to the receiving node).

[00197] Also, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[00198] Abbreviations

[00199] 3GPP 3rd Generation Partnership Project

[00200] 5G 5th Generation

[00201] 5GC 5G Core network

[00202] 5GS 5th Generation System

[00203] AMP Access and Mobility Management F

[00204] ASN.1 Abstract Syntax Notation One

[00205] AT Attention

[00206] AR Augmented Reality

[00207] AS Access Stratum

[00208] CGI Cell Global Identity

[00209] CN Core Network

[00210] CP Control Plane

[00211] CU Central Unit

[00212] CU-CP Central Unit Control Plane

[00213] CU-UP Central Unit User Plane [00214] DU Distributed Unit

[00215] DASH Dynamic Adaptive Streaming over HTTP

[00216] DC Dual Connectivity

[00217] DL Downlink

[00218] DNS Domain Name System

[00219] DU Distributed Unit

[00220] E-CGI E-UTRAN CGI

[00221] eNB Evolved Node B / E-UTRAN Node B

[00222] en-gNB A gNB acting as a secondary node in an EN-DC scenario (i.e. in a DC scenario with an eNB as the master node and a gNB as the secondary node.

[00223] EN E-UTRAN-NR

[00224] EPC Evolved Packet Core

[00225] EPS Evolved Packet System

[00226] E-UTRA Evolved UTRA

[00227] E-UTRAN/EUTRAN Evolved UTRAN

[00228] gNB Radio base station in NR

[00229] HSS Home Subscriber Server

[00230] HTTP Hypertext Transfer Protocol

[00231] I AB Integrated Access and Backhaul

[00232] ID Identifier/ldentity

[00233] IE Information Element

[00234] LTE Long Term Evolution

[00235] MAC Medium Access Control

[00236] MCC Mobile Country Code

[00237] MCE Measurement Collection Entity / Measurement Collector Entity [00238] MDT Minimization of Drive Tests

[00239] MME Mobility Management Entity

[00240] MNC Mobile Network Code

[00241] MTSI Multimedia Telephony Service for IMS

[00242] N3IWF Non-3GPP Interworking Function

[00243] NG Next Generation

[00244] NG The interface between an NG-RAN and a 5GC.

[00245] NGAP NG Application Protocol

[00246] NG-RAN NG Radio Access Network

[00247] NID Network identifier

[00248] NR New Radio

[00249] NWDAF Network Data Analytics Function

[00250] O&M Operation and Maintenance

[00251] QAM Operation and Maintenance

[00252] PDCP Packet Data Convergence Protocol

[00253] PDU Protocol Data Unit

[00254] PLMN Public Land Mobile Network

[00255] QMC QoE Measurement Collection

[00256] QoE Quality of Experience

[00257] RAN Radio Access Network

[00258] RAT Radio Access Technology

[00259] RLC Radio Link Control

[00260] RNC Radio Network Controller

[00261] RRC Radio Resource Control

[00262] RVQoE RAN Visible QoE [00263] S1 The interface between the RAN and the CN in LTE.

[00264] S1AP S1 Application Protocol

[00265] S-NSSAI Single Network Slice Selection Assistance Information

[00266] SMO Service Management and Orchestration

[00267] SRB Signaling Radio Bearer

[00268] TA Tracking Area

[00269] TCE Trace Collection Entity / Trace Collector Entity

[00270] TNGF Trusted Non-3GPP Gateway Function

[00271] TWIF Trusted WLAN Interworking Function

[00272] UDM Unified Data Management

[00273] UE User Equipment

[00274] UMTS Universal Mobile Telecommunication System

[00275] URI Uniform Resource Identifier

[00276] URL Uniform Resource Locator Uniform Resource Locator

[00277] UTRA Universal Terrestrial Radio Access

[00278] UTRAN Universal Terrestrial Radio Access Network

[00279] WLAN Wireless Local Area Network

[00280] Xn The interface between two gNBs in NR.

[00281] XnAP Xn Application Protocol