CROSS-REFERENCE TO RELATED PATENT APPLICATION(S)

This application claims the benefit of U.S. Provisional Application No. 63/393,572, filed July 29, 2022, the disclosure of which is incorporated herein by reference as if set forth in full.

TECHNICAL FIELD

This disclosure generally relates to systems and methods for wireless communications and, more particularly, to artificial intelligence (Al) and machine learning (ML) entity testing.

BACKGROUND

Al and ML entities require precise testing to guarantee optimal performance. There is a need for a system that streamlines testing requests, initiation, preparation, and comprehensive reporting, serving as a reliable framework for the evaluation of Al and ML entities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustrative schematic diagram for artificial intelligence (AI)/machine learning (ML) entity testing and control, in accordance with one or more example embodiments of the present disclosure.

FIG. 2 depicts an illustrative schematic diagram for an intelligent evaluator, in accordance with one or more example embodiments of the present disclosure.

FIG. 3 illustrates a flow diagram of a process for an illustrative intelligent evaluator system, in accordance with one or more example embodiments of the present disclosure.

FIG. 4 illustrates an example network architecture, in accordance with one or more example embodiments of the present disclosure.

FIG. 5 schematically illustrates a wireless network, in accordance with one or more example embodiments of the present disclosure.

FIG. 6 illustrates components of a computing device, in accordance with one or more example embodiments of the present disclosure. DETAILED DESCRIPTION

The following description and the drawings sufficiently illustrate specific embodiments to enable those skilled in the art to practice them. Other embodiments may incorporate structural, logical, electrical, process, algorithm, and other changes. Portions and features of some embodiments may be included in, or substituted for, those of other embodiments. Embodiments set forth in the claims encompass all available equivalents of those claims.

After an AI/ML Entity is trained, validation is done to ensure the training process is completed successfully. However, even when validation is conducted successfully during development, it is necessary to test and check if the AI/ML Entity is working correctly under certain runtime contexts or using certain testing data sets. Testing may involve interaction with third parties (besides the developer of the AI/ML training function), e.g., the operators may use the AI/ML training function or third-party systems/functions that may rely on the results computed by the AI/ML entity for testing.

After completing the AI/ML entity training, and when the performance of the trained AI/ML entity meets the expectations on both training and validation data, the AI/ML entity is made available to the consumer(s) via the AI/ML training report. Before applying the AI/ML entity to the target inference function or AI/ML-enabled function, the AI/ML training MnS producer may need to allow the consumer to evaluate the performance of the AI/ML entity via the AI/ML testing process using the consumer’s provided testing data. The testing data have the same pattern as the input part of the training data.

For these reasons, the provision of AI/ML testing and its control need to be standardized to enable multi-vendor interaction among the different systems. If the testing performance is not acceptable or does not meet the pre-defined requirements, the consumer may request the AI/ML training producer to re-train the AI/ML entity with specific training data and/or performance requirements.

Example embodiments of the present disclosure relate to systems, methods, and devices for solution for AI/ML entity testing for 5G system (5GS). Note that in this disclosure the term “AI/ML” can be used interchangeably with “ML”. The AI/ML entity is either an ML model or an entity containing an ML model and its related metadata.

In one or more embodiments, an intelligent evaluator system may be conceptualized as an entity that resembles a machine learning model. For instance, akin to the structure and functionality of various machine learning models, this entity encompasses an intricate mathematical formula that represents the core model. However, the entity may also encapsulate additional components that enable effective management and operation of the model. Therefore, while the main aspect of the entity mirrors a machine learning model, it also incorporates other elements that provide the system with an extra layer of manageability.

In one or more embodiments, an intelligent evaluator system may operate through multiple phases in order to make Al and machine learning truly functional. For example, these phases might encompass a training phase, a validation phase, a testing phase, an emulation phase, a deployment phase, and an inference phase. These phases may allow the system to run the model in an emulated real environment, gradually deploy the entity to the target skeleton, and then activate the model to conduct inferences in that setup. Further, the performance of the model is consistently monitored across all these phases, which collectively form a comprehensive operational workflow. The goal is to make every operational phase controllable, manageable, monitorable, and evaluable, ensuring accountability of the system.

In one or more embodiments, an intelligent evaluator system may primarily focus on managing the testing phase of the workflow. This phase involves feeding testing data to the model, running the model's algorithm to generate inference outputs, and computing the performance score based on the testing data and results. Two significant use cases may be considered here. First, consumers might request to test a particular model using their own testing data. The system then evaluates the model and either returns a computed performance score or provides the raw inference output, enabling consumers to calculate the performance themselves. The second use case might involve the system's producer initiating the testing independently, possibly based on specific criteria, and subsequently sharing the test results with the consumers.

In one or more embodiments, an intelligent evaluator system may incorporate specific control measures, particularly when Al is applied in industries like telecom where small errors can lead to significant problems. Historically, every new feature, update, or upgrade had to be manually tested by operators before deployment, which is a tedious and time-consuming task. However, with Al, the entire workflow, from training to inference, can be automated, making it more frequent and efficient. Even so, the system ensures that this automated process remains under the control of the operators or consumers, allowing for necessary interventions at any phase if required.

In one or more embodiments, an intelligent evaluator system may employ Information Object Classes (IOCS) to model the information or data exchanged between the producer and consumer. This method allows for handling information in a generic manner, with specific modeling for different use cases. For instance, there might be a machine learning testing request IOC, representing a testing request sent by the consumer, and a network policy IOC, representing a policy used to control producer-initiated testing. Additionally, there may be a process IOC that can be controlled by the consumer and a report IOC used by the producer to communicate testing results to the consumer. These IOCS are interconnected, with the process IOC associated with a request or a policy, and the report IOC linked with the process.

In one or more embodiments, an intelligent evaluator system may include a sendee producer supported by one or more processors. This system may receive a request from a service consumer to create a first managed objected instance (MOI) for ML entity testing. The device may respond to the consumer to indicate whether the MOI creation request is accepted and may notify the consumer about the creation of the first MOI. The device may prepare for the AI/ML entity testing and create a second MOI for the AI/ML entity testing process, notifying the consumer in the process. The AI/ML entity testing may then be started by the device, and the consumer may be notified about the progress of the AI/ML testing process by modifying the second MOI. After completion of the AI/ML entity testing, the device may create a third MOI for the AI/ML entity testing report and notify the consumer.

In this context, the first MOI may represent the AI/ML entity testing request or the AI/ML entity testing policy. The first MOI may contain information such as the identifier of the AI/ML entity to be tested, testing environment requirements, e.g., expected runtime context, and testing data. The first MOI may also contain information such as the identifier or inference type of the AI/ML entity to be tested, testing environment requirements, e.g., expected runtime context, and testing triggers, i.e., the conditions that would trigger the testing of an AI/ML entity.

The second MOI may contain information like the identifier of the AI/ML entity being tested, the associated AI/ML entity testing request, the associated AI/ML entity testing policy, testing progress, testing environment, e.g., the testing runtime context, and control of the process, like cancel, suspend and resume.

The third MOI may contain information like the identifier of the tested AI/ML entity, the associated AI/ML entity testing request, the associated AI/ML entity testing process, the testing result indicating the success or failure, and containing the inference output for each testing data example for a successful case, and the failure reason for the failed case.

In some embodiments, the service producer may also receive a request from a sendee consumer to modify the second MOI to control the AI/ML testing process, respond to the consumer to indicate whether the MOI modification request is accepted, notify the consumer about the modification of the second MOI, and control the AI/ML testing process accordingly per the modification of the MOI. Modification of the second MOI may be to change the attribute for canceling, suspending, resuming, or terminating the AI/ML testing process. Creation of any MOI may be notified to the consumer by notification, and modification of the second MOI may be notified to the consumer by notification.

The above descriptions are for purposes of illustration and are not meant to be limiting. Numerous other examples, configurations, processes, algorithms, etc., may exist, some of which are described in greater detail below. Example embodiments will now be described with reference to the accompanying figures.

FIG. 1 depicts an illustrative schematic diagram for artificial intelligence AI/ML entity testing and control, in accordance with one or more example embodiments of the present disclosure.

In one or more embodiments, an intelligent evaluator system may facilitate consumer- requested AI/ML entity testing.

After receiving an AI/ML training report about a trained AI/ML entity from the AI/ML Training management service (MnS) producer, the consumer may request the testing MnS producer to test the AI/ML entity before applying it to the target inference function. In the AI/ML testing request, the consumer provides the testing data which have the same pattern as the input part of the training data.

Any AI/ML entity needs to be tested with specific inputs and features that are applicable to the use case and the applicable deployment environment.

The AI/ML testing MnS producer performs the AI/ML testing using the consumer’s provided testing data. The AI/ML testing is to conduct inference on the tested AI/ML entity using the testing data as the inference inputs and produce the inference output for each testing dataset example. The AI/ML testing MnS producer may be the same as or different from the AI/ML Training MnS producer. After completing the AI/ML testing, the AI/ML testing MnS producer provides the testing report indicating the success or failure of the AI/ML testing to the consumer. For successful AI/ML testing, the testing report contains the testing results, e.g., the inference output for each testing dataset example. The AI/ML testing MnS producer needs to have the capabilities and provide the services needed to enable the consumer to request testing and receive results on the testing of a specific AI/ML entity or of an application or function that contains an AI/ML entity. To achieve the desired outcomes, any AI/ML entity needs to be tested with the appropriate testing data, which can reflect the current status of the network where the AI/ML entity is expected to be deployed. Correspondingly, the AI/ML testing MnS producer needs to support the required management services to test the AI/ML entities. In one or more embodiments, an intelligent evaluator system may facilitate control of AI/ML entity testing.

Given a testing capability as provided by a given AI/ML testing MnS producer, a consumer (e.g., an operator) may wish to control and manage that testing process capability. For example, the operator may wish to define policies on how frequently testing for a given AI/ML entity may be executed. Correspondingly, the 3GPP management system needs to provide the capability to allow the AI/ML entity testing to be configured.

The following are some of the potential requirements that are needed for AI/ML testing.

REQ-AI/MLTEST-1 The AI/ML testing MnS producer should have a capability for an authorized consumer to request the testing of a specific AI/ML entity.

REQ-AI/MLTEST-2 The AI/ML testing MnS producer should have a capability to create a testing process instance per the testing request for an authorized consumer.

REQ-AI/MLTEST-3 The AI/ML testing MnS producer should have a capability to report to an authorized consumer the results of a specific instance of AI/ML testing process with the result of a successful AI/ML entity testing containing the inference output for each testing data example.

REQ-AI/MLTEST-4 The AI/ML testing MnS producer should have a capability for an authorized consumer (e.g. the operator) to configure or modify an instance of AI/ML testing process.

REQ-AI/MLTEST-5 The AI/ML testing MnS producer should have a capability to test a specific AI/ML entity using specific data specified by the consumer or using data at a location address specified by the consumer or using data with specific characteristics defined by the consumer.

REQ-AI/MLTEST-6 The AI/ML testing MnS producer should have a capability to test a specific AI/ML entity for a specified expected runtime context as may be stated by the consumer.

REQ-AI/MLTEST-8 The AI/ML testing MnS producer should support a capability for an authorized consumer to define the reporting characteristics related to a specific instance of AI/ML testing request.

REQ-AI/MLTEST-9 The AI/ML testing MnS producer should support a capability for an authorized consumer to manage the AI/ML testing request, including suspending, resuming, canceling the request, or adjusting the desired runtime context of the testing.

FIG. 2 depicts an illustrative schematic diagram for an intelligent evaluator, in accordance with one or more example embodiments of the present disclosure. Referring to FIG. 2, there is shown an example of AI/ML entity testing related NRMs.

In one or more embodiments, an intelligent evaluator system may facilitate solutions for AI/ML entity testing. Should be noted that the name of the attributes or information elements used here in this disclosure may be named differently.

In one or more embodiments, an intelligent evaluator system may provide a network resource model (NRM) based solution. This solution uses the instances of following IOCS for interaction between AI/ML testing MnS producer and consumer to support the AI/ML entity testing.

1) The IOC representing the AI/ML entity testing request, for example, is named as MLTe stingRequest. This IOC is created by the AI/ML entity testing MnS consumer on the producer, and it contains the following attributes: identifier of the AI/ML entity to be tested;

- testing environment requirements, e.g., expected runtime context;

- testing data.

2) The IOC representing the AI/ML entity testing policy, for example, named as MLTe sting Pol icy. This IOC is created by the AI/ML entity testing MnS consumer on the producer to control the testing initiated by the producer, and it contains the following attributes : identifier or inference type of the AI/ML entity to be tested;

- testing environment requirements, e.g., expected runtime context;

- testing triggers, i.e., the conditions that would trigger the testing of an AI/ML entity.

3) The IOC representing the AI/ML entity testing process, for example, named as MLTe stingProces s . This IOC is created by the AI/ML entity testing MnS producer and reported to the consumer, and it contains the following attributes: identifier of the AI/ML entity being tested;

- the associated AI/ML entity testing request;

- the associated AI/ML entity testing policy;

- testing progress;- testing environment, e.g., the testing runtime context;

- control of the process, like cancel, suspend and resume.

4) The IOC representing the AI/ML testing report, for example, named as MLTe stingReport. This IOC is created by the AI/ML testing MnS producer and reported to the consumer, and it contains the following attributes:

- identifier of the tested AI/ML entity;

- the associated AI/ML entity testing request; - the associated AI/ML entity testing process;

- testing result indicating the success or failure, and containing the inference output for each testing data example for the successful case, and the failure reason for the failed case.

The examples of IOCS and their relations with the IOCS are depicted in FIG. 2.

It is understood that the above descriptions are for the purposes of illustration and are not meant to be limiting.

In some embodiments, the electronic device(s), network(s), system(s), chip(s) or component(s), or portions or implementations thereof, of FIGs. 4-6, or some other figure herein, may be configured to perform one or more processes, techniques, or methods as described herein or portions thereof. One such process is depicted in FIG. 3.

For example, the process may include, at 302 decode a request from a service consumer for a creation of a first managed object instance (MOI) specifically intended for machine learning (ML) entity testing.

The process further includes, at 304, encoding a response to the service consumer, wherein the response comprises an indication of whether the request for MOI creation is accepted.

The process further includes, at 306, initiating the ML entity testing upon acceptance of the request.

The process further includes, at 308, creating a second MOI for the ML entity testing report upon completion of the ML entity testing.

The process further includes, at 310, encoding a notification to the service consumer regarding the creation of this second MOL

For one or more embodiments, at least one of the components set forth in one or more of the preceding figures may be configured to perform one or more operations, techniques, processes, and/or methods as set forth in the example section below. For example, the baseband circuitry as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below. For another example, circuitry associated with a UE, base station, network element, etc. as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below in the example section.

It is understood that the above descriptions are for purposes of illustration and are not meant to be limiting.

FIGs. 4-6 illustrate various systems, devices, and components that may implement aspects of disclosed embodiments. FIG. 4 illustrates an example network architecture 400 according to various embodiments. The network 400 may operate in a manner consistent with 3GPP technical specifications for LTE or 5G/NR systems. However, the example embodiments are not limited in this regard and the described embodiments may apply to other networks that benefit from the principles described herein, such as future 3GPP systems, or the like.

The network 400 includes a UE 402, which is any mobile or non-mobile computing device designed to communicate with a RAN 404 via an over-the-air connection. The UE 402 is communicatively coupled with the RAN 404 by a Uu interface, which may be applicable to both LTE and NR systems. Examples of the UE 402 include, but are not limited to, a smartphone, tablet computer, wearable computer, desktop computer, laptop computer, in- vehicle infotainment system, in-car entertainment system, instrument cluster, head-up display (HUD) device, onboard diagnostic device, dashtop mobile equipment, mobile data terminal, electronic engine management system, electron! c/engine control unit, electronic/ engine control module, embedded system, sensor, microcontroller, control module, engine management system, networked appliance, machine-type communication device, machine-to-machine (M2M), device-to-device (D2D), machme-type communication (MTC) device, Internet of Things (loT) device, and/or the like. The network 400 may include a plurality of UEs 402 coupled directly with one another via a D2D, ProSe, PC5, and/or sidelink (SL) interface. These UEs 402 may be M2M/D2D/MTC/IoT devices and/or vehicular systems that communicate using physical sidelink channels such as, but not limited to, PSBCH, PSDCH, PSSCH, PSCCH, PSFCH, etc. The UE 402 may perform blind decoding attempts of SL channels/links according to the various embodiments herein.

In some embodiments, the UE 402 may additionally communicate with an AP 406 via an over-the-air (OTA) connection. The AP 406 manages a WLAN connection, which may serve to offload some/all network traffic from the RAN 404. The connection between the UE 402 and the AP 406 may be consistent with any IEEE 802. 11 protocol. Additionally, the UE 402, RAN 404, and AP 406 may utilize cellular- WLAN aggregation/integration (e.g., LWA/LWIP). Cellular- WLAN aggregation may involve the UE 402 being configured by the RAN 404 to utilize both cellular radio resources and WLAN resources.

The RAN 404 includes one or more access network nodes (ANs) 408. The ANs 408 terminate air-interface(s) for the UE 402 by providing access stratum protocols including RRC, PDCP, RLC, MAC, and PHY/L1 protocols. In this manner, the AN 408 enables data/voice connectivity between CN 420 and the UE 402. The ANs 408 may be a macrocell base station or a low power base station for providing femtocells, picocells or other like cells having smaller coverage areas, smaller user capacity, or higher bandwidth compared to macrocells; or some combination thereof. In these implementations, an AN 408 be referred to as a BS, gNB, RAN node, eNB, ng-eNB, NodeB, RSU, TRxP, etc.

One example implementation is a “CU/DU split” architecture where the ANs 408 are embodied as a gNB-Central Unit (CU) that is communicatively coupled with one or more gNB- Distributed Units (DUs), where each DU may be communicatively coupled with one or more Radio Units (RUs) (also referred to as RRHs, RRUs, or the like) (see e g., 3GPP TS 38.401 v 16.1.0 (2020-03)). In some implementations, the one or more RUs may be individual RSUs. In some implementations, the CU/DU split may include an ng-eNB-CU and one or more ng- eNB-DUs instead of, or in addition to, the gNB-CU and gNB-DUs, respectively. The ANs 408 employed as the CU may be implemented in a discrete device or as one or more software entities running on server computers as part of, for example, a virtual network including a virtual Base Band Unit (BBU) or BBU pool, cloud RAN (CRAN), Radio Equipment Controller (REC), Radio Cloud Center (RCC), centralized RAN (C-RAN), virtualized RAN (vRAN), and/or the like (although these terms may refer to different implementation concepts). Any other ty pe of architectures, arrangements, and/or configurations can be used.

The plurality of ANs may be coupled with one another via an X2 interface (if the RAN 404 is an LTE RAN or Evolved Universal Terrestrial Radio Access Network (E-UTRAN) 410) or an Xn interface (if the RAN 404 is a NG-RAN 414). The X2/Xn interfaces, which may be separated into control/user plane interfaces in some embodiments, may allow the ANs to communicate information related to handovers, data/context transfers, mobility, load management, interference coordination, etc.

The ANs of the RAN 404 may each manage one or more cells, cell groups, component earners, etc. to provide the UE 402 with an air interface for network access. The UE 402 may be simultaneously connected with a plurality of cells provided by the same or different ANs 408 of the RAN 404. For example, the UE 402 and RAN 404 may use carrier aggregation to allow the UE 402 to connect with a plurality of component carriers, each corresponding to a Pcell or Scell. In dual connectivity scenarios, a first AN 408 may be a master node that provides an MCG and a second AN 408 may be secondary node that provides an SCG. The first/second ANs 408 may be any combination of eNB, gNB, ng-eNB, etc.

The RAN 404 may provide the air interface over a licensed spectrum or an unlicensed spectrum. To operate in the unlicensed spectrum, the nodes may use LAA, eLAA, and/or feLAA mechanisms based on CA technology with PCells/Scells. Prior to accessing the unlicensed spectrum, the nodes may perform medium/carrier-sensing operations based on, for example, a listen-before-talk (LBT) protocol.

In V2X scenarios the UE 402 or AN 408 may be or act as a roadside unit (RSU), which may refer to any transportation infrastructure entity used for V2X communications. An RSU may be implemented in or by a suitable AN or a stationary (or relatively stationary) UE. An RSU implemented in or by: a UE may be referred to as a “UE-type RSU”; an eNB may be referred to as an “eNB-type RSU”; a gNB may be referred to as a “gNB-type RSU”; and the like. In one example, an RSU is a computing device coupled with radio frequency circuitry located on a roadside that provides connectivity support to passing vehicle UEs. The RSU may also include internal data storage circuitry to store intersection map geometry, traffic statistics, media, as well as applications/software to sense and control ongoing vehicular and pedestrian traffic. The RSU may provide very low latency communications required for high speed events, such as crash avoidance, traffic warnings, and the like. Additionally or alternatively, the RSU may provide other cellular/WLAN communications services. The components of the RSU may be packaged in a weatherproof enclosure suitable for outdoor installation, and may include a network interface controller to provide a wired connection (e.g., Ethernet) to a traffic signal controller or a backhaul network.

In some embodiments, the RAN 404 may be an E-UTRAN 410 with one or more eNBs 412. The an E-UTRAN 410 provides an LTE air interface (Uu) with the following characteristics: SCS of 15 kHz; CP-OFDM waveform for DL and SC-FDMA waveform for UL; turbo codes for data and TBCC for control; etc. The LTE air interface may rely on CSI- RS for CSI acquisition and beam management; PDSCH/PDCCH DMRS for PDSCH/PDCCH demodulation; and CRS for cell search and initial acquisition, channel quality measurements, and channel estimation for coherent demodulation/detection at the UE. The LTE air interface may operating on sub-6 GHz bands.

In some embodiments, the RAN 404 may be an next generation (NG)-RAN 414 with one or more gNB 416 and/or on or more ng-eNB 418. The gNB 416 connects with 5G-enabled UEs 402 using a 5G NR interface. The gNB 416 connects with a 5GC 440 through an NG interface, which includes an N2 interface or an N3 interface. The ng-eNB 418 also connects with the 5GC 440 through an NG interface, but may connect with a UE 402 via the Uu interface. The gNB 416 and the ng-eNB 418 may connect with each other over an Xn interface.

In some embodiments, the NG interface may be split into two parts, an NG user plane (NG-U) interface, which carries traffic data between the nodes of the NG-RAN 414 and a UPF 448 (e.g., N3 interface), and an NG control plane (NG-C) interface, which is a signaling interface between the nodes of the NG-RAN 414 and an AMF 444 (e.g., N2 interface).

The NG-RAN 414 may provide a 5G-NR air interface (which may also be referred to as a Uu interface) with the following characteristics: variable SCS; CP-OFDM for DL, CP- OFDM and DFT-s-OFDM for UL; polar, repetition, simplex, and Reed-Muller codes for control and LDPC for data. The 5G-NR air interface may rely on CSI-RS, PDSCH/PDCCH DMRS similar to the LTE air interface. The 5G-NR air interface may not use a CRS, but may use PBCH DMRS for PBCH demodulation; PTRS for phase tracking for PDSCH; and tracking reference signal for time tracking. The 5G-NR air interface may operating on FR1 bands that include sub-6 GHz bands or FR2 bands that include bands from 24.25 GHz to 52.6 GHz. The 5G-NR air interface may include an SSB that is an area of a downlink resource grid that includes PSS/SSS/PBCH.

The 5G-NR air interface may utilize BWPs for various purposes. For example, BWP can be used for dynamic adaptation of the SCS. For example, the UE 402 can be configured with multiple BWPs where each BWP configuration has a different SCS. When a BWP change is indicated to the UE 402, the SCS of the transmission is changed as well. Another use case example of BWP is related to power saving. In particular, multiple BWPs can be configured for the UE 402 with different amount of frequency resources (e.g., PRBs) to support data transmission under different traffic loading scenarios. A BWP containing a smaller number of PRBs can be used for data transmission with small traffic load while allowing power saving at the UE 402 and in some cases at the gNB 416. A BWP containing a larger number of PRBs can be used for scenarios with higher traffic load.

The RAN 404 is communicatively coupled to CN 420 that includes network elements and/or network functions (NFs) to provide various functions to support data and telecommunications services to customers/subscribers (e.g., UE 402). The components of the CN 420 may be implemented in one physical node or separate physical nodes. In some embodiments, NFV may be utilized to virtualize any or all of the functions provided by the network elements of the CN 420 onto physical compute/storage resources in servers, switches, etc. A logical instantiation of the CN 420 may be referred to as a network slice, and a logical instantiation of a portion of the CN 420 may be referred to as a network sub-slice.

The CN 420 may be an LTE CN 422 (also referred to as an Evolved Packet Core (EPC) 422). The EPC 422 may include MME 424, SGW 426, SGSN 428, HSS 430, PGW 432, and PCRF 434 coupled with one another over interfaces (or “reference points”) as shown. The NFs in the EPC 422 are briefly introduced as follows. The MME 424 implements mobility management functions to track a current location of the UE 402 to facilitate paging, bearer activation/ deactivation, handovers, gateway selection, authentication, etc.

The SGW 426 terminates an SI interface toward the RAN 410 and routes data packets between the RAN 410 and the EPC 422. The SGW 426 may be a local mobility anchor point for inter-RAN node handovers and also may provide an anchor for inter-3 GPP mobility. Other responsibilities may include lawful intercept, charging, and some policy enforcement.

The SGSN 428 tracks a location of the UE 402 and performs security functions and access control. The SGSN 428 also performs inter-EPC node signaling for mobility between different RAT networks; PDN and S-GW selection as specified by MME 424; MME 424 selection for handovers; etc. The S3 reference point between the MME 424 and the SGSN 428 enable user and bearer information exchange for inter-3GPP access network mobility in idle/active states.

The HSS 430 includes a database for network users, including subscription-related information to support the network entities’ handling of communication sessions. The HSS 430 can provide support for routmg/roammg, authentication, authorization, naming/addressmg resolution, location dependencies, etc. An S6a reference point between the HSS 430 and the MME 424 may enable transfer of subscription and authentication data for authenticating/authorizing user access to the EPC 420.

The PGW 432 may terminate an SGi interface toward a data network (DN) 436 that may include an application (app)Zcontent server 438. The PGW 432 routes data packets between the EPC 422 and the data network 436. The PGW 432 is communicatively coupled with the SGW 426 by an S5 reference point to facilitate user plane tunneling and tunnel management. The PGW 432 may further include a node for policy enforcement and charging data collection (e.g., PCEF). Additionally, the SGi reference point may communicatively couple the PGW 432 with the same or different data network 436. The PGW 432 may be communicatively coupled with a PCRF 434 via a Gx reference point.

The PCRF 434 is the policy and charging control element of the EPC 422. The PCRF 434 is communicatively coupled to the app/content server 438 to determine appropriate QoS and charging parameters for service flows. The PCRF 432 also provisions associated rules into a PCEF (via Gx reference point) with appropriate TFT and QCI.

The CN 420 may be a 5GC 440 including an AUSF 442, AMF 444, SMF 446, UPF 448, NSSF 450, NEF 452, NRF 454, PCF 456, UDM 458, and AF 460 coupled with one another over various interfaces as shown. The NFs in the 5GC 440 are briefly introduced as follows.

The AUSF 442 stores data for authentication of UE 402 and handle authentication- related functionality. The AUSF 442 may facilitate a common authentication framework for various access types..

The AMF 444 allows other functions of the 5GC 440 to communicate with the UE 402 and the RAN 404 and to subscribe to notifications about mobility events with respect to the UE 402. The AMF 444 is also responsible for registration management (e g., for registering UE 402), connection management, reachability management, mobility management, lawful interception of AMF-related events, and access authentication and authorization. The AMF 444 provides transport for SM messages between the UE 402 and the SMF 446, and acts as a transparent pro3 for routing SM messages. AMF 444 also provides transport for SMS messages between UE 402 and an SMSF. AMF 444 interacts with the AUSF 442 and the UE 402 to perform various security anchor and context management functions. Furthermore, AMF 444 is a termination point of a RAN-CP interface, which includes the N2 reference point between the RAN 404 and the AMF 444. The AMF 444 is also a termination point of NAS (Nl) signaling, and performs NAS ciphering and integrity protection.

AMF 444 also supports NAS signaling with the UE 402 over an N3IWF interface. The N3IWF provides access to untrusted entities. N3IWF may be a termination point for the N2 interface between the (R)AN 404 and the AMF 444 for the control plane, and may be a termination point for the N3 reference point between the (R)AN 414 and the 448 for the user plane. As such, the AMF 444 handles N2 signalling from the SMF 446 and the AMF 444 for PDU sessions and QoS, encapsulate/de-encapsulate packets for IPSec and N3 tunnelling, marks N3 user-plane packets in the uplink, and enforces QoS corresponding to N3 packet marking taking into account QoS requirements associated with such marking received overN2. N3IWF may also relay UL and DL control-plane NAS signalling between the UE 402 and AMF 444 via an Nl reference point between the UE 402and the AMF 444, and relay uplink and downlink user-plane packets between the UE 402 and UPF 448. The N3IWF also provides mechanisms for IPsec tunnel establishment with the UE 402. The AMF 444 may exhibit an Namf servicebased interface, and may be a termination point for an N14 reference point betw een two AMFs 444 and an N17 reference point between the AMF 444 and a 5G-EIR (not shown by FIG. 4).

The SMF 446 is responsible for SM (e.g., session establishment, tunnel management between UPF 448 and AN 408); UE IP address allocation and management (including optional authorization); selection and control of UP function; configuring traffic steering at UPF 448 to route traffic to proper destination: termination of interfaces toward policy control functions; controlling part of policy enforcement, charging, and QoS; lawful intercept (for SM events and interface to LI system); termination of SM parts of NAS messages; downlink data notification; initiating AN specific SM information, sent via AMF 444 over N2 to AN 408; and determining SSC mode of a session. SM refers to management of a PDU session, and a PDU session or “session” refers to a PDU connectivity service that provides or enables the exchange of PDUs between the UE 402 and the DN 436.

The UPF 448 acts as an anchor point for intra-RAT and inter-RAT mobility, an external PDU session point of interconnect to data network 436, and a branching point to support multihomed PDU session. The UPF 448 also performs packet routing and forwarding, packet inspection, enforces user plane part of policy rules, lawfully intercept packets (UP collection), performs traffic usage reporting, perform QoS handling for a user plane (e.g., packet filtering, gating, UE/DE rate enforcement), performs uplink traffic verification (e.g., SDF-to-QoS flow mapping), transport level packet marking in the uplink and downlink, and performs downlink packet buffering and downlink data notification triggering. UPF 448 may include an uplink classifier to support routing traffic flows to a data network.

The NSSF 450 selects a set of network slice instances serving the UE 402. The NSSF 450 also determines allowed NSSAI and the mapping to the subscribed S-NSSAIs, if needed. The NSSF 450 also determines an AMF set to be used to serve the UE 402, or a list of candidate AMFs 444 based on a suitable configuration and possibly by querying the NRF 454. The selection of a set of network slice instances for the UE 402 may be triggered by the AMF 444 with which the UE 402 is registered by interacting with the NSSF 450; this may lead to a change of AMF 444. The NSSF 450 interacts with the AMF 444 via an N22 reference point; and may communicate with another NSSF in a visited network via an N31 reference point (not shown).

The NEF 452 securely exposes services and capabilities provided by 3GPP NFs for third party, internal exposure/re-exposure, AFs 460, edge computing or fog computing systems (e.g., edge compute node, etc. In such embodiments, the NEF 452 may authenticate, authorize, or throttle the AFs. NEF 452 may also translate information exchanged with the AF 460 and information exchanged with internal network functions. For example, the NEF 452 may translate between an AF-Service-Identifier and an internal 5GC information. NEF 452 may also receive information from other NFs based on exposed capabilities of other NFs. This information may be stored at the NEF 452 as structured data, or at a data storage NF using standardized interfaces. The stored information can then be re-exposed by the NEF 452 to other NFs and AFs, or used for other purposes such as analytics.

The NRF 454 supports service discovery functions, receives NF discovery requests from NF instances, and provides information of the discovered NF instances to the requesting NF instances. NRF 454 also maintains information of available NF instances and their supported services. The NRF 454 also supports service discovery functions, wherein the NRF 454 receives NF Discovery Request from NF instance or an SCP (not shown), and provides information of the discovered NF instances to the NF instance or SCP.

The PCF 456 provides policy rules to control plane functions to enforce them, and may also support unified policy framework to govern network behavior. The PCF 456 may also implement a front end to access subscription information relevant for policy decisions in a UDR of the UDM 458. In addition to communicating with functions over reference points as shown, the PCF 456 exhibit an Npcf service-based interface.

The UDM 458 handles subscription-related information to support the network entities’ handling of communication sessions, and stores subscription data of UE 402. For example, subscription data may be communicated via an N8 reference point between the UDM 458 and the AMF 444. The UDM 458 may include two parts, an application front end and a UDR. The UDR may store subscription data and policy data for the UDM 458 and the PCF 456, and/or structured data for exposure and application data (including PFDs for application detection, application request information for multiple UEs 402) for the NEF 452. The Nudr servicebased interface may be exhibited by the UDR 221 to allow the UDM 458, PCF 456, and NEF 452 to access a particular set of the stored data, as well as to read, update (e.g., add, modify), delete, and subscribe to notification of relevant data changes in the UDR. The UDM may include a UDM-FE, which is in charge of processing credentials, location management, subscription management and so on. Several different front ends may serve the same user in different transactions. The UDM-FE accesses subscription information stored in the UDR and performs authentication credential processing, user identification handling, access authorization, registration/mobility management, and subscription management. In addition to communicating with other NFs over reference points as shown, the UDM 458 may exhibit the Nudm service-based interface.

AF 460 provides application influence on traffic routing, provide access to NEF 452, and interact with the policy framework for policy control. The AF 460 may influence UPF 448 (re)selection and traffic routing. Based on operator deployment, when AF 460 is considered to be a trusted entity, the network operator may permit AF 460 to interact directly with relevant NFs. Additionally, the AF 460 may be used for edge computing implementations,

The 5GC 440 may enable edge computing by selecting operator/3rd party services to be geographically close to a point that the UE 402 is attached to the network. This may reduce latency and load on the network. In edge computing implementations, the 5GC 440 may select a UPF 448 close to the UE 402 and execute traffic steering from the UPF 448 to DN 436 via theN6 interface. This may be based on the UE subscription data, UE location, and information provided by the AF 460, which allows the AF 460 to influence UPF (re)selection and traffic routing.

The data network (DN) 436 may represent various network operator services, Internet access, or third party services that may be provided by one or more servers including, for example, application (app)Zcontent server 438. The DN 436 may be an operator external public, a private PDN, or an intra-operator packet data network, for example, for provision of IMS services. In this embodiment, the app server 438 can be coupled to an IMS via an S-CSCF or the I-CSCF. In some implementations, the DN 436 may represent one or more local area DNs (LADNs), which are DNs 436 (or DN names (DNNs)) that is/are accessible by a UE 402 in one or more specific areas. Outside of these specific areas, the UE 402 is not able to access the LADN/DN 436.

Additionally or alternatively, the DN 436 may be an Edge DN 436, which is a (local) Data Network that supports the architecture for enabling edge applications. In these embodiments, the app server 438 may represent the physical hardware sy stems/ devices providing app server functionality and/or the application software resident in the cloud or at an edge compute node that performs server function(s). In some embodiments, the app/content server 438 provides an edge hosting environment that provides support required for Edge Application Server's execution.

In some embodiments, the 5GS can use one or more edge compute nodes to provide an interface and offload processing of wireless communication traffic. In these embodiments, the edge compute nodes may be included in, or co-located with one or more RAN410, 414. For example, the edge compute nodes can provide a connection between the RAN 414 and UPF 448 in the 5GC 440. The edge compute nodes can use one or more NFV instances instantiated on virtualization infrastructure within the edge compute nodes to process wireless connections to and from the RAN 414 and UPF 448.

The interfaces of the 5GC 440 include reference points and service-based itnterfaces. The reference points include: N 1 (between the UE 402 and the AMF 444), N2 (between RAN 414 and AMF 444), N3 (between RAN 414 and UPF 448), N4 (between the SMF 446 and UPF 448), N5 (between PCF 456 and AF 460), N6 (between UPF 448 and DN 436), N7 (between SMF 446 and PCF 456), N8 (between UDM 458 and AMF 444), N9 (between two UPFs 448), N10 (between the UDM 458 and the SMF 446), Ni l (between the AMF 444 and the SMF 446), N12 (between AUSF 442 and AMF 444), N13 (between AUSF 442 and UDM 458), N14 (between two AMFs 444; not shown), N15 (between PCF 456 and AMF 444 in case of a nonroaming scenario, or between the PCF 456 in a visited network and AMF 444 in case of a roaming scenario), N16 (between two SMFs 446; not shown), and N22 (between AMF 444 and NSSF 450). Other reference point representations not shown in FIG. 4 can also be used. The service-based representation of FIG. 4 represents NFs within the control plane that enable other authorized NFs to access their services. The service-based interfaces (SBIs) include: Namf (SBI exhibited by AMF 444), Nsmf (SBI exhibited by SMF 446), Nnef (SBI exhibited by NEF 452), Npcf (SBI exhibited by PCF 456), Nudm (SBI exhibited by the UDM 458), Naf (SBI exhibited by AF 460), Nnrf (SBI exhibited by NRF 454), Nnssf (SBI exhibited by NSSF 450), Nausf (SBI exhibited by AUSF 442). Other service-based interfaces (e.g., Nudr, N5g-eir, and Nudsf) not shown in FIG. 4 can also be used. In some embodiments, the NEF 452 can provide an interface to edge compute nodes 436x, which can be used to process wireless connections with the RAN 414. In some implementations, the system 400 may include an SMSF, which is responsible for SMS subscription checking and verification, and relaying SM messages to/from the UE 402 to/from other entities, such as an SMS-GMSC/IWMSC/SMS- router. The SMS may also interact with AMF 444 and UDM 458 for a notification procedure that the UE 402 is available for SMS transfer (e.g., set a UE not reachable flag, and notifying UDM 458 when UE 402 is available for SMS).

The 5GS may also include an SCP (or individual instances of the SCP) that supports indirect communication (see e.g., 3GPP TS 23.501 section 7.1.1); delegated discovery (see e.g., 3GPP TS 23.501 section 7.1.1); message forwarding and routing to destination NF/NF service(s), communication security (e.g., authorization of the NF Service Consumer to access the NF Service Producer API) (see e.g., 3GPP TS 33.501), load balancing, monitoring, overload control, etc.; and discovery and selection functionality for UDM(s), AUSF(s), UDR(s), PCF(s) with access to subscription data stored in the UDR based on UE's SUPI, SUCI or GPSI (see e.g., 3GPP TS 23.501 section 6.3). Load balancing, monitoring, overload control functionality provided by the SCP may be implementation specific. The SCP may be deployed in a distributed manner. More than one SCP can be present in the communication path between various NF Services. The SCP, although not an NF instance, can also be deployed distributed, redundant, and scalable.

FIG. 5 schematically illustrates a wireless network 500 in accordance with various embodiments. The wireless network 500 may include a UE 502 in wireless communication with an AN 504. The UE 502 and AN 504 may be similar to, and substantially interchangeable with, like-named components described with respect to FIG. 4.

The UE 502 may be communicatively coupled with the AN 504 via connection 506. The connection 506 is illustrated as an air interface to enable communicative coupling, and can be consistent with cellular communications protocols such as an LTE protocol or a 5G NR protocol operating at mmWave or sub-6GHz frequencies.

The UE 502 may include a host platform 508 coupled with a modem platform 510. The host platform 508 may include application processing circuitry 512, which may be coupled with protocol processing circuitry 514 of the modem platform 510. The application processing circuitry 512 may run various applications for the UE 502 that source/sink application data. The application processing circuitry 512 may further implement one or more layer operations to transrmt/receive application data to/from a data network. These layer operations may include transport (for example UDP) and Internet (for example, IP) operations

The protocol processing circuitry 514 may implement one or more of layer operations to facilitate transmission or reception of data over the connection 506. The layer operations implemented by the protocol processing circuitry 514 may include, for example, MAC, RLC, PDCP, RRC and NAS operations.

The modem platform 510 may further include digital baseband circuitry 516 that may implement one or more layer operations that are “below” layer operations performed by the protocol processing circuitry 514 in a network protocol stack. These operations may include, for example, PHY operations including one or more of HARQ acknowledgement (ACK) functions, scrambling/descrambling, encoding/decoding, layer mapping/de-mapping, modulation symbol mapping, received symbol/bit metric determination, multi-antenna port precoding/decoding, which may include one or more of space-time, space-frequency or spatial coding, reference signal generation/detection, preamble sequence generation and/or decoding, synchronization sequence generation/detection, control channel signal blind decoding, and other related functions.

The modem platform 510 may further include transmit circuitry 518, receive circuitry 520, RF circuitry 522, and RF front end (RFFE) 524, which may include or connect to one or more antenna panels 526. Briefly, the transmit circuitry 518 may include a digital-to-analog converter, mixer, intermediate frequency (IF) components, etc. ; the receive circuitry 520 may include an analog-to-digital converter, mixer, IF components, etc.; the RF circuitry 522 may include a low-noise amplifier, a power amplifier, power tracking components, etc.; RFFE 524 may include filters (for example, surface/bulk acoustic wave filters), switches, antenna tuners, beamforming components (for example, phase-array antenna components), etc. The selection and arrangement of the components of the transmit circuitry 518, receive circuitry 520, RF circuitry 522, RFFE 524, and antenna panels 526 (referred generically as “transmit/receive components”) may be specific to details of a specific implementation such as, for example, whether communication is TDM or FDM, in mmWave or sub-6 gHz frequencies, etc. In some embodiments, the transmit/receive components may be arranged in multiple parallel transmit/receive chains, may be disposed in the same or different chips/modules, etc.

In some embodiments, the protocol processing circuitry 514 may include one or more instances of control circuitry (not shown) to provide control functions for the transmit/receive components.

A UE 502 reception may be established by and via the antenna panels 526, RFFE 524, RF circuitry 522, receive circuitry 520, digital baseband circuitry 516, and protocol processing circuitry 514. In some embodiments, the antenna panels 526 may receive a transmission from the AN 504 by receive-beamforming signals received by a plurality of antennas/antenna elements of the one or more antenna panels 526.

A UE 502 transmission may be established by and via the protocol processing circuitry 514, digital baseband circuitry 516, transmit circuitry' 518, RF circuitry 522, RFFE 524, and antenna panels 526. In some embodiments, the transmit components of the UE 504 may apply a spatial filter to the data to be transmitted to form a transmit beam emitted by the antenna elements of the antenna panels 526.

Similar to the UE 502, the AN 504 may include a host platform 528 coupled with a modem platform 530. The host platform 528 may include application processing circuitry 532 coupled with protocol processing circuitry 534 of the modem platform 530. The modem platform may further include digital baseband circuitry 536, transmit circuitry 538, receive circuitry 540, RF circuitry 542, RFFE circuitry 544, and antenna panels 546. The components of the AN 504 may be similar to and substantially interchangeable with like-named components of the UE 502. In addition to performing data transmission/reception as described above, the components of the AN 508 may perform various logical functions that include, for example, RNC functions such as radio bearer management, uplink and downlink dynamic radio resource management, and data packet scheduling. FIG. 6 illustrates components of a computing device 600 according to some example embodiments, able to read instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically, FIG. 6 shows a diagrammatic representation of hardware resources 601 including one or more processors (or processor cores) 610, one or more memory /storage devices 620, and one or more communication resources 630, each of which may be communicatively coupled via a bus 640 or other interface circuitry. For embodiments where node virtualization (e g., NFV) is utilized, a hypervisor 602 may be executed to provide an execution environment for one or more network slices/sub-slices to utilize the hardware resources 601.

The processors 610 include, for example, processor 612 and processor 614. The processors 610 include circuitry such as, but not limited to one or more processor cores and one or more of cache memory, low drop-out voltage regulators (LDOs), interrupt controllers, serial interfaces such as SPI, I2C or universal programmable serial interface circuit, real time clock (RTC), timer-counters including interval and watchdog timers, general purpose I/O, memory card controllers such as secure digital/multi-media card (SD/MMC) or similar, interfaces, mobile industry processor interface (MIPI) interfaces and Joint Test Access Group (JTAG) test access ports. The processors 610 may be, for example, a central processing unit (CPU), reduced instruction set computing (RISC) processors, Acorn RISC Machine (ARM) processors, complex instruction set computing (CISC) processors, graphics processing units (GPUs), one or more Digital Signal Processors (DSPs) such as a baseband processor, Application-Specific Integrated Circuits (ASICs), an Field-Programmable Gate Array (FPGA), a radio-frequency integrated circuit (RFIC), one or more microprocessors or controllers, another processor (including those discussed herein), or any suitable combination thereof. In some implementations, the processor circuitry 610 may include one or more hardware accelerators, which may be microprocessors, programmable processing devices (e.g., FPGA, complex programmable logic devices (CPLDs), etc.), or the like.

The memory /storage devices 620 may include main memory, disk storage, or any suitable combination thereof. The memory /storage devices 620 may include, but are not limited to, any type of volatile, non-volatile, or semi-volatile memory such as random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), Flash memory, solid-state storage, phase change RAM (PRAM), resistive memory such as magnetoresistive random access memory (MRAM), etc., and may incorporate three-dimensional (3D) cross-point (XPOINT) memories from Intel® and Micron®. The memory/storage devices 620 may also comprise persistent storage devices, which may be temporal and/or persistent storage of any type, including, but not limited to, nonvolatile memory, optical, magnetic, and/or solid state mass storage, and so forth.

The communication resources 630 may include interconnection or network interface controllers, components, or other suitable devices to communicate with one or more peripheral devices 604 or one or more databases 606 or other network elements via a network 608. For example, the communication resources 630 may include wired communication components (e.g., for coupling via USB, Ethernet, Ethernet, Ethernet over GRE Tunnels, Ethernet over Multiprotocol Label Switching (MPLS), Ethernet over USB, Controller Area Network (CAN), Local Interconnect Network (LIN), DeviceNet, ControlNet, Data Highway-i-, PROFIBUS, or PROFINET, among many others), cellular communication components, NFC components, Bluetooth® (or Bluetooth® Low Energy) components, WiFi® components, and other communication components. Network connectivity may be provided to/from the computing device 600 via the communication resources 630 using a physical connection, which may be electrical (e.g., a “copper interconnect”) or optical. The physical connection also includes suitable input connectors (e.g., ports, receptacles, sockets, etc.) and output connectors (e.g., plugs, pins, etc.). The communication resources 630 may include one or more dedicated processors and/or FPGAs to communicate using one or more of the aforementioned network interface protocols.

Instructions 650 may comprise software, a program, an application, an applet, an app, or other executable code for causing at least any of the processors 610 to perform any one or more of the methodologies discussed herein. The instructions 650 may reside, completely or partially, within at least one of the processors 610 (e.g., within the processor’s cache memory), the memory/storage devices 620, or any suitable combination thereof. Furthermore, any portion of the instructions 650 may be transferred to the hardware resources 601 from any combination of the peripheral devices 604 or the databases 606. Accordingly, the memory of processors 610, the memory/storage devices 620, the peripheral devices 604, and the databases 606 are examples of computer-readable and machine-readable media.

For one or more embodiments, at least one of the components set forth in one or more of the preceding figures may be configured to perform one or more operations, techniques, processes, and/or methods as set forth in the example section below. For example, the baseband circuitry as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below. For another example, circuitry associated with a UE, base station, network element, etc. as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below in the example section.

Additional examples of the presently described embodiments include the following, non-limiting implementations. Each of the following non-limiting examples may stand on its own or may be combined in any permutation or combination with any one or more of the other examples provided below or throughout the present disclosure.

For one or more embodiments, at least one of the components set forth in one or more of the preceding figures may be configured to perform one or more operations, techniques, processes, and/or methods as set forth in the example section below. For example, the baseband circuitry as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below. For another example, circuitry associated with a UE, base station, network element, etc. as described above in connection with one or more of the preceding figures may be configured to operate in accordance with one or more of the examples set forth below.

The following examples pertain to further embodiments.

Example 1 may include an apparatus comprising decode a request from a sendee consumer for a creation of a first managed object instance (MOI) specifically intended for machine learning (ML) entity testing; encode a response to the service consumer, wherein the response comprises an indication of whether the request for MOI creation may be accepted; initiate the ML entity testing upon acceptance of the request; create a second MOI for the ML entity testing report upon completion of the ML entity testing; and encode a notification to the service consumer regarding the creation of this second MOI.

Example 2 may include the apparatus of example 1 and/or some other example herein, wherein the processing circuitry may be further configured to indicate acceptance or rejection of the testing request to the service consumer.

Example 3 may include the apparatus of example 1 and/or some other example herein, wherein the processing circuitry may be further configured to create a third managed object instances (MOIs) for different stages of the ML entity testing.

Example 4 may include the apparatus of example 1 and/or some other example herein, wherein the first MOI represents the ML entity testing request including an identifier of the ML entity to be tested, and testing environment requirements.

Example 5 may include the apparatus of example 3 and/or some other example herein, wherein the third MOI represents the ML entity testing process, including the identifier of the ML entity being tested, the associated ML entity testing request, testing progress, and control of the process.

Example 6 may include the apparatus of example 1 and/or some other example herein, wherein the second MOI represents the ML entity testing report, including an identifier of the ML entity that has been tested, the associated ML entity testing request, the associated ML entity testing process, and/or the testing result.

Example 7 may include the apparatus of example 1 and/or some other example herein, wherein the processing circuitry may be further configured to manage the ML entity testing process based on a modification of MOIs.

Example 8 may include the apparatus of example 7 and/or some other example herein, wherein the modification of the third MOI may be to change an attribute for canceling, suspending, resuming, or terminating the ML testing process.

Example 9 may include the apparatus of example 1 and/or some other example herein, wherein the processing circuitry may be further configured to notify the consumer about the creation or modification of any MOI.

Example 10 may include a computer-readable medium storing computer-executable instructions which when executed by one or more processors result in performing operations comprising: decode a request from a service consumer for a creation of a first managed object instance (MOI) specifically intended for machine learning (ML) entity testing; encode a response to the service consumer, wherein the response comprises an indication of whether the request for MOI creation may be accepted; initiate the ML entity testing upon acceptance of the request; create a second MOI for the ML entity testing report upon completion of the ML entity testing; and encode a notification to the service consumer regarding the creation of this second MOL

Example 11 may include the computer-readable medium of example 10 and/or some other example herein, wherein the operations further comprise indicate acceptance or rejection of the testing request to the service consumer.

Example 12 may include the computer-readable medium of example 10 and/or some other example herein, wherein the operations further comprise create a third managed object instances (MOIs) for different stages of the ML entity testing.

Example 13 may include the computer-readable medium of example 10 and/or some other example herein, wherein the first MOI represents the ML entity testing request including an identifier of the ML entity to be tested, and testing environment requirements. Example 14 may include the computer-readable medium of example 12 and/or some other example herein, wherein the third MOI represents the ML entity testing process, including the identifier of the ML entity being tested, the associated ML entity testing request, testing progress, and control of the process.

Example 15 may include the computer-readable medium of example 10 and/or some other example herein, wherein the second MOI represents the ML entity testing report, including an identifier of the ML entity that has been tested, the associated ML entity testing request, the associated ML entity testing process, and/or the testing result.

Example 16 may include the computer-readable medium of example 10 and/or some other example herein, wherein the operations further comprise manage the ML entity testing process based on a modification of MOIs.

Example 17 may include the computer-readable medium of example 16 and/or some other example herein, wherein the modification of the third MOI may be to change an attribute for canceling, suspending, resuming, or terminating the ML testing process.

Example 18 may include the computer-readable medium of example 10 and/or some other example herein, wherein the operations further comprise notily the consumer about the creation or modification of any MOI.

Example 19 may include a method comprising: decode a request from a service consumer for a creation of a first managed object instance (MOI) specifically intended for machine learning (ML) entity testing; encode a response to the service consumer, wherein the response comprises an indication of whether the request for MOI creation may be accepted; initiate the ML entity' testing upon acceptance of the request; create a second MOI for the ML entity testing report upon completion of the ML entity testing; and encode a notification to the service consumer regarding the creation of this second MOL

Example 20 may include the method of example 19 and/or some other example herein, further comprising indicate acceptance or rejection of the testing request to the sendee consumer.

Example 21 may include the method of example 19 and/or some other example herein, further comprising create a third managed object instances (MOIs) for different stages of the ML entity testing.

Example 22 may include the method of example 19 and/or some other example herein, wherein the first MOI represents the ML entity testing request including an identifier of the ML entity to be tested, and testing environment requirements. Example 23 may include the method of example 21 and/or some other example herein, wherein the third MOI represents the ML entity testing process, including the identifier of the ML entity being tested, the associated ML entity testing request, testing progress, and control of the process.

Example 24 may include the method of example 19 and/or some other example herein, wherein the second MOI represents the ML entity testing report, including an identifier of the ML entity that has been tested, the associated ML entity testing request, the associated ML entity testing process, and/or the testing result

Example 25 may include the method of example 19 and/or some other example herein, further comprising manage the ML entity testing process based on a modification of MOIs.

Example 26 may include the method of example 25 and/or some other example herein, wherein the modification of the third MOI may be to change an attribute for canceling, suspending, resuming, or terminating the ML testing process.

Example 27 may include the method of example 19 and/or some other example herein, further comprising notily the consumer about the creation or modification of any MOL

Example 28 may include an apparatus comprising means for: decode a request from a service consumer for a creation of a first managed object instance (MOI) specifically intended for machine learning (ML) entity testing; encode a response to the service consumer, wherein the response comprises an indication of whether the request for MOI creation may be accepted; initiate the ML entity testing upon acceptance of the request; create a second MOI for the ML entity testing report upon completion of the ML entity testing; and encode a notification to the service consumer regarding the creation of this second MOL

Example 29 may include the apparatus of example 28 and/or some other example herein, further comprising indicate acceptance or rejection of the testing request to the service consumer.

Example 30 may include the apparatus of example 28 and/or some other example herein, further comprising create a third managed object instances (MOIs) for different stages of the ML entity testing.

Example 31 may include the apparatus of example 28 and/or some other example herein, wherein the first MOI represents the ML entity testing request including an identifier of the ML entity to be tested, and testing environment requirements.

Example 32 may include the apparatus of example 30 and/or some other example herein, wherein the third MOI represents the ML entity testing process, including the identifier of the ML entity being tested, the associated ML entity testing request, testing progress, and control of the process.

Example 33 may include the apparatus of example 28 and/or some other example herein, wherein the second MOI represents the ML entity testing report, including an identifier of the ML entity that has been tested, the associated ML entity testing request, the associated ML entity testing process, and/or the testing result.

Example 34 may include the apparatus of example 28 and/or some other example herein, further comprising manage the ML entity testing process based on a modification of MOIs.

Example 35 may include the apparatus of example 34 and/or some other example herein, wherein the modification of the third MOI may be to change an attribute for canceling, suspending, resuming, or terminating the ML testing process.

Example 36 may include the apparatus of example 28 and/or some other example herein, further comprising notify the consumer about the creation or modification of any MOL Example 37 may include an apparatus comprising means for performing any of the methods of examples 1-36.

Example 38 may include a network node comprising a communication interface and processing circuitry connected thereto and configured to perform the methods of examples 1- 36.

Example 39 may include an apparatus comprising means to perform one or more elements of a method described in or related to any of examples 1 -36, or any other method or process described herein.

Example 40 may include one or more non-transitory computer-readable media comprising instructions to cause an electronic device, upon execution of the instructions by one or more processors of the electronic device, to perform one or more elements of a method described in or related to any of examples 1-36, or any other method or process described herein.

Example 41 may include an apparatus comprising logic, modules, or circuitry to perform one or more elements of a method described in or related to any of examples 1-36, or any other method or process described herein.

Example 42 may include a method, technique, or process as described in or related to any of examples 1-36, or portions or parts thereof.

Example 43 may include an apparatus comprising: one or more processors and one or more computer-readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method, techniques, or process as described in or related to any of examples 1-36, or portions thereof.

Example 44 may include a signal as described in or related to any of examples 1-36, or portions or parts thereof.

Example 45 may include a datagram, packet, frame, segment, protocol data unit (PDU), or message as described in or related to any of examples 1-36, or portions or parts thereof, or otherwise described in the present disclosure.

Example 46 may include a signal encoded with data as described in or related to any of examples 1-36, or portions or parts thereof, or otherwise described in the present disclosure.

Example 47 may include a signal encoded with a datagram, packet, frame, segment, protocol data unit (PDU), or message as described in or related to any of examples 1-36, or portions or parts thereof, or otherwise described in the present disclosure.

Example 48 may include an electromagnetic signal carrying computer-readable instructions, wherein execution of the computer-readable instructions by one or more processors is to cause the one or more processors to perform the method, techniques, or process as described in or related to any of examples 1-36, or portions thereof.

Example 49 may include a computer program comprising instructions, wherein execution of the program by a processing element is to cause the processing element to carry out the method, techniques, or process as described in or related to any of examples 1-36, or portions thereof.

Example 50 may include a signal in a wireless network as shown and described herein.

Example 51 may include a method of communicating in a wireless network as shown and described herein.

Example 52 may include a sy stem for providing wireless communication as shown and described herein.

Example 53 may include a device for providing wireless communication as shown and described herein.

An example implementation is an edge computing system, including respective edge processing devices and nodes to invoke or perform the operations of the examples above, or other subject matter described herein. Another example implementation is a client endpoint node, operable to invoke or perform the operations of the examples above, or other subject matter described herein. Another example implementation is an aggregation node, network hub node, gateway node, or core data processing node, within or coupled to an edge computing system, operable to invoke or perform the operations of the examples above, or other subject mater described herein. Another example implementation is an access point, base station, road-side unit, street-side unit, or on-premise unit, within or coupled to an edge computing system, operable to invoke or perform the operations of the examples above, or other subject mater described herein. Another example implementation is an edge provisioning node, service orchestration node, application orchestration node, or multi-tenant management node, within or coupled to an edge computing system, operable to invoke or perform the operations of the examples above, or other subject mater described herein. Another example implementation is an edge node operating an edge provisioning service, application or service orchestration service, virtual machine deployment, container deployment, function deployment, and compute management, within or coupled to an edge computing system, operable to invoke or perform the operations of the examples above, or other subject mater described herein. Another example implementation is an edge computing system operable as an edge mesh, as an edge mesh with side car loading, or with mesh-to-mesh communications, operable to invoke or perform the operations of the examples above, or other subject mater described herein. Another example implementation is an edge computing system including aspects of network functions, acceleration functions, acceleration hardware, storage hardware, or computation hardware resources, operable to invoke or perform the use cases discussed herein, with use of the examples above, or other subject mater described herein. Another example implementation is an edge computing system adapted for supporting client mobility, vehicle-to-vehicle (V2V), vehicle-to-every thing (V2X), or vehicle-to-infrastructure (V2I) scenarios, and optionally operating according to ETST MEC specifications, operable to invoke or perform the use cases discussed herein, with use of the examples above, or other subject mater described herein. Another example implementation is an edge computing system adapted for mobile wireless communications, including configurations according to an 3GPP 4G/LTE or 5G network capabilities, operable to invoke or perform the use cases discussed herein, with use of the examples above, or other subject mater described herein. Another example implementation is a computing system adapted for network communications, including configurations according to an O-RAN capabilities, operable to invoke or perform the use cases discussed herein, with use of the examples above, or other subject mater described herein.

Any of the above-described examples may be combined with any other example (or combination of examples), unless explicitly stated otherwise. The foregoing description of one or more implementations provides illustration and description, but is not intended to be exhaustive or to limit the scope of embodiments to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments.

TERMINOLOGY

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an” and “the” are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specific the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operation, elements, components, and/or groups thereof.

For the purposes of the present disclosure, the phrase “A and/or B” means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, and/or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The description may use the phrases “in an embodiment,” or “In some embodiments,” which may each refer to one or more of the same or different embodiments. Furthermore, the terms “comprising,” “including,” “having,” and the like, as used with respect to embodiments of the present disclosure, are synonymous.

The terms “coupled,” “communicatively coupled,” along with derivatives thereof are used herein. The term “coupled” may mean two or more elements are in direct physical or electrical contact with one another, may mean that two or more elements indirectly contact each other but still cooperate or interact with each other, and/or may mean that one or more other elements are coupled or connected between the elements that are said to be coupled with each other. The term “directly coupled” may mean that two or more elements are in direct contact with one another. The term “communicatively coupled” may mean that two or more elements may be in contact with one another by a means of communication including through a wire or other interconnect connection, through a wireless communication channel or ink, and/or the like.

The term “circuitry” as used herein refers to, is part of, or includes hardware components such as an electronic circuit, a logic circuit, a processor (shared, dedicated, or group) and/or memory (shared, dedicated, or group), an Application Specific Integrated Circuit (ASIC), a field-programmable device (FPD) (e.g., a field-programmable gate array (FPGA), a programmable logic device (PLD), a complex PLD (CPLD), a high-capacity PLD (HCPLD), a structured ASIC, or a programmable SoC), digital signal processors (DSPs), etc., that are configured to provide the described functionality. In some embodiments, the circuitry may execute one or more software or firmware programs to provide at least some of the described functionality. The term “circuitry” may also refer to a combination of one or more hardware elements (or a combination of circuits used in an electrical or electronic system) with the program code used to carry out the functionality of that program code. In these embodiments, the combination of hardware elements and program code may be referred to as a particular type of circuitry.

The term “processor circuitry” as used herein refers to, is part of, or includes circuitry capable of sequentially and automatically carrying out a sequence of arithmetic or logical operations, or recording, storing, and/or transferring digital data. Processing circuitry may include one or more processing cores to execute instructions and one or more memory structures to store program and data information. The term “processor circuitry” may refer to one or more application processors, one or more baseband processors, a physical central processing unit (CPU), a single-core processor, a dual-core processor, a triple-core processor, a quad-core processor, and/or any other device capable of executing or otherwise operating computer-executable instructions, such as program code, software modules, and/or functional processes. Processing circuitry may include more hardware accelerators, which may be microprocessors, programmable processing devices, or the like. The one or more hardware accelerators may include, for example, computer vision (CV) and/or deep learning (DL) accelerators. The terms “application circuitry” and/or “baseband circuitry” may be considered synonymous to, and may be referred to as, “processor circuitry.”

The term “memory” and/or “memory circuitry” as used herein refers to one or more hardware devices for storing data, including RAM, MRAM, PRAM, DRAM, and/or SDRAM, core memory, ROM, magnetic disk storage mediums, optical storage mediums, flash memory devices or other machine readable mediums for storing data. The term “computer-readable medium” may include, but is not limited to, mcmoy, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instructions or data.

The term “interface circuitry” as used herein refers to, is part of, or includes circuitry that enables the exchange of information between two or more components or devices. The term “interface circuitry” may refer to one or more hardware interfaces, for example, buses, I/O interfaces, peripheral component interfaces, network interface cards, and/or the like. The term “user equipment” or “UE” as used herein refers to a device with radio communication capabilities and may describe a remote user of network resources in a communications network. The term “user equipment” or “UE” may be considered synonymous to, and may be referred to as, client, mobile, mobile device, mobile terminal, user terminal, mobile unit, mobile station, mobile user, subscriber, user, remote station, access agent, user agent, receiver, radio equipment, reconfigurable radio equipment, reconfigurable mobile device, etc. Furthermore, the term “user equipment” or “UE” may include any type of wireless/wired device or any computing device including a wireless communications interface.

The term “network element” as used herein refers to physical or virtualized equipment and/or infrastructure used to provide wired or wireless communication network services. The term “network element” may be considered synonymous to and/or referred to as a networked computer, networking hardware, network equipment, network node, router, switch, hub, bridge, radio network controller, RAN device, RAN node, gateway, server, virtualized VNF, NFVI, and/or the like.

The term “computer system” as used herein refers to any type interconnected electronic devices, computer devices, or components thereof. Additionally, the term “computer system” and/or “system” may refer to various components of a computer that are communicatively coupled with one another. Furthermore, the term “computer system” and/or “system” may refer to multiple computer devices and/or multiple computing systems that are communicatively coupled with one another and configured to share computing and/or networking resources.

The term “appliance,” “computer appliance,” or the like, as used herein refers to a computer device or computer system with program code (e.g., software or firmware) that is specifically designed to provide a specific computing resource. A “virtual appliance” is a virtual machine image to be implemented by a hypervisor-equipped device that virtualizes or emulates a computer appliance or otherwise is dedicated to provide a specific computing resource. The term “element” refers to a unit that is indivisible at a given level of abstraction and has a clearly defined boundary, wherein an element may be any type of entity including, for example, one or more devices, systems, controllers, network elements, modules, etc., or combinations thereof. The term “device” refers to a physical entity embedded inside, or attached to, another physical entity in its vicinity, with capabilities to convey digital information from or to that physical entity. The term “entity” refers to a distinct component of an architecture or device, or information transferred as a payload. The term “controller” refers to an element or entity that has the capability to affect a physical entity, such as by changing its state or causing the physical entity to move.

The term “cloud computing” or “cloud” refers to a paradigm for enabling network access to a scalable and elastic pool of shareable computing resources with self-service provisioning and administration on-demand and without active management by users. Cloud computing provides cloud computing services (or cloud services), which are one or more capabilities offered via cloud computing that are invoked using a defined interface (e.g., an API or the like). The term “computing resource” or simply “resource” refers to any physical or virtual component, or usage of such components, of limited availability within a computer system or network. Examples of computing resources include usage/access to, for a period of time, servers, processor(s), storage equipment, memory devices, memory areas, networks, electrical power, input/output (peripheral) devices, mechanical devices, network connections (e.g., channels/links, ports, network sockets, etc ), operating systems, virtual machines (VMs), software/applications, computer files, and/or the like. A “hardware resource” may refer to compute, storage, and/or network resources provided by physical hardware element(s). A “virtualized resource” may refer to compute, storage, and/or network resources provided by virtualization infrastructure to an application, device, system, etc. The term “network resource” or “communication resource” may refer to resources that are accessible by computer devices/sy stems via a communications network. The term “system resources” may refer to any kind of shared entities to provide services, and may include computing and/or network resources. System resources may be considered as a set of coherent functions, network data objects or services, accessible through a server where such system resources reside on a single host or multiple hosts and are clearly identifiable. As used herein, the term “cloud service provider” (or CSP) indicates an organization which operates typically large-scale “cloud” resources comprised of centralized, regional, and edge data centers (e.g., as used in the context of the public cloud). In other examples, a CSP may also be referred to as a Cloud Service Operator (CSO). References to “cloud computing” generally refer to computing resources and services offered by a CSP or a CSO, at remote locations with at least some increased latency, distance, or constraints relative to edge computing.

As used herein, the term “data center” refers to a purpose-designed structure that is intended to house multiple high-performance compute and data storage nodes such that a large amount of compute, data storage and network resources are present at a single location. This often entails specialized rack and enclosure systems, suitable heating, cooling, ventilation, security, fire suppression, and power delivery systems. The term may also refer to a compute and data storage node in some contexts. A data center may vary in scale between a centralized or cloud data center (e.g., largest), regional data center, and edge data center (e.g., smallest).

As used herein, the term “edge computing” refers to the implementation, coordination, and use of computing and resources at locations closer to the “edge” or collection of “edges” of a network. Deploying computing resources at the network’s edge may reduce application and network latency, reduce network backhaul traffic and associated energy' consumption, improve service capabilities, improve compliance with security or data privacy requirements (especially as compared to conventional cloud computing), and improve total cost of ownership). As used herein, the term “edge compute node” refers to a real-world, logical, or virtualized implementation of a compute-capable element in the form of a device, gateway, bridge, system or subsystem, component, whether operating in a server, client, endpoint, or peer mode, and whether located at an “edge” of an network or at a connected location further within the network. References to a “node” used herein are generally interchangeable with a “device”, “component”, and “sub-system”; however, references to an “edge computing system” or “edge computing network” generally refer to a distributed architecture, organization, or collection of multiple nodes and devices, and which is organized to accomplish or offer some aspect of services or resources in an edge computing setting.

Additionally or alternatively, the term “Edge Computing” refers to a concept, as described in [6], that enables operator and 3rd party services to be hosted close to the UE’s access point of attachment, to achieve an efficient service delivery through the reduced end-to- end latency and load on the transport network. As used herein, the term “Edge Computing Service Provider” refers to a mobile network operator or a 3rd party service provider offering Edge Computing service. As used herein, the term “Edge Data Network” refers to a local Data Network (DN) that supports the architecture for enabling edge applications. As used herein, the term “Edge Hosting Environment” refers to an environment providing support required for Edge Application Server's execution. As used herein, the term “Application Server” refers to application software resident in the cloud performing the server function.

The term “Internet of Things” or “loT” refers to a system of interrelated computing devices, mechanical and digital machines capable of transferring data with little or no human interaction, and may involve technologies such as real-time analytics, machine learning and/or Al, embedded systems, wireless sensor networks, control systems, automation (e.g., smarthome, smart building and/or smart city technologies), and the like. loT devices are usually low-power devices without heavy compute or storage capabilities. “Edge loT devices” may be any kind of loT devices deployed at a network’s edge. As used herein, the term “cluster” refers to a set or grouping of entities as part of an edge computing system (or systems), in the form of physical entities (e.g., different computing systems, networks or network groups), logical entities (e.g., applications, functions, security constructs, containers), and the like. In some locations, a “cluster” is also referred to as a “group” or a “domain”. The membership of cluster may be modified or affected based on conditions or functions, including from dynamic or property -based membership, from network or system management scenarios, or from various example techniques discussed below which may add, modify, or remove an entity in a cluster. Clusters may also include or be associated with multiple layers, levels, or properties, including variations in security features and results based on such layers, levels, or properties.

The term “application” may refer to a complete and deploy able package, environment to achieve a certain function in an operational environment. The term “AI/ML application” or the like may be an application that contains some AI/ML models and application-level descriptions. The term “machine learning” or “ML” refers to the use of computer systems implementing algorithms and/or statistical models to perform specific task(s) without using explicit instructions, but instead relying on patterns and inferences. ML algorithms build or estimate mathematical model(s) (referred to as “ML models” or the like) based on sample data (referred to as “training data,” “model training information,” or the like) in order to make predictions or decisions without being explicitly programmed to perform such tasks. Generally, an ML algorithm is a computer program that leams from experience with respect to some task and some performance measure, and an ML model may be any object or data structure created after an ML algorithm is trained with one or more training datasets. After training, an ML model may be used to make predictions on new datasets. Although the term “ML algorithm” refers to different concepts than the term “ML model,” these terms as discussed herein may be used interchangeably for the purposes of the present disclosure.

The term “machine learning model,” “ML model,” or the like may also refer to ML methods and concepts used by an ML-assisted solution. An “ML-assisted solution” is a solution that addresses a specific use case using ML algorithms during operation. ML models include supervised learning (e.g., linear regression, k-nearest neighbor (KNN), decision tree algorithms, support machine vectors, Bayesian algorithm, ensemble algorithms, etc.) unsupervised learning (e.g., K-means clustering, principle component analysis (PCA), etc.), reinforcement learning (e.g., Q-leaming, multi-armed bandit learning, deep RL, etc.), neural networks, and the like. Depending on the implementation a specific ML model could have many sub-models as components and the ML model may train all sub-models together. Separately trained ML models can also be chained together in an ML pipeline during inference. An “ML pipeline” is a set of functionalities, functions, or functional entities specific for an ML-assisted solution; an ML pipeline may include one or several data sources in a data pipeline, a model training pipeline, a model evaluation pipeline, and an actor. The “actor” is an entity that hosts an ML assisted solution using the output of the ML model inference). The term “ML training host” refers to an entity, such as a network function, that hosts the training of the model. The term “ML inference host” refers to an entity, such as a network function, that hosts model during inference mode (which includes both the model execution as well as any online learning if applicable). The ML-host informs the actor about the output of the ML algorithm, and the actor takes a decision for an action (an “action” is performed by an actor as a result of the output of an ML assisted solution). The term “model inference information” refers to information used as an input to the ML model for determining inference(s); the data used to train an ML model and the data used to determine inferences may overlap, however, “training data” and “inference data” refer to different concepts.

The terms “instantiate,” “instantiation,” and the like as used herein refers to the creation of an instance. An “instance” also refers to a concrete occurrence of an object, which may occur, for example, during execution of program code. The term “information element” refers to a structural element containing one or more fields. The term “field” refers to individual contents of an information element, or a data element that contains content. As used herein, a “database object”, “data structure”, or the like may refer to any representation of information that is in the form of an object, attribute- value pair (AVP), key-value pair (KVP), tuple, etc., and may include variables, data structures, functions, methods, classes, database records, database fields, database entities, associations between data and/or database entities (also referred to as a “relation”), blocks and links between blocks in block chain implementations, and/or the like.

An “information object,” as used herein, refers to a collection of structured data and/or any representation of information, and may include, for example electronic documents (or “documents”), database objects, data structures, files, audio data, video data, raw data, archive files, application packages, and/or any other like representation of information. The terms “electronic document” or “document,” may refer to a data structure, computer file, or resource used to record data, and includes various file ty pes and/or data formats such as word processing documents, spreadsheets, slide presentations, multimedia items, webpage and/or source code documents, and/or the like. As examples, the information objects may include markup and/or source code documents such as HTML, XML, JSON, Apex®, CSS, JSP, MessagePack™, Apache® Thrift™, ASN.l, Google® Protocol Buffers (protobuf), or some other document(s)/format(s) such as those discussed herein. An information object may have both a logical and a physical structure. Physically, an information object comprises one or more units called entities. An entity is a unit of storage that contains content and is identified by a name. An entity may refer to other entities to cause their inclusion in the information object. An information object begins in a document entity, which is also referred to as a root element (or "root"). Logically, an information object comprises one or more declarations, elements, comments, character references, and processing instructions, all of which are indicated in the information object (e.g., using markup).

The term “data item” as used herein refers to an atomic state of a particular object with at least one specific property at a certain point in time. Such an object is usually identified by an object name or object identifier, and properties of such an object are usually defined as database objects (e.g., fields, records, etc.), object instances, or data elements (e.g., mark-up language elements/tags, etc.). Additionally or alternatively, the term “data item” as used herein may refer to data elements and/or content items, although these terms may refer to difference concepts. The term “data element” or “element” as used herein refers to a unit that is indivisible at a given level of abstraction and has a clearly defined boundary. A data element is a logical component of an information object (e.g., electronic document) that may begin with a start tag (e.g., “<element>”) and end with amatching end tag (e.g., “</element>”), or only has an empty element tag (e.g., “<element />”). Any characters between the start tag and end tag, if any, are the element’s content (referred to herein as “content items” or the like).

The content of an entity may include one or more content items, each of which has an associated datatype representation. A content item may include, for example, attribute values, character values, URIs, qualified names (qnames), parameters, and the like. A qname is a fully qualified name of an element, attribute, or identifier in an information object. A qname associates a URI of a namespace with a local name of an element, attribute, or identifier in that namespace. To make this association, the qname assigns a prefix to the local name that corresponds to its namespace. The qname comprises a URI of the namespace, the prefix, and the local name. Namespaces are used to provide uniquely named elements and attributes in information objects. Content items may include text content (e g., “<element>content item</element>”), attributes (e.g., “<element attribute="attributeValue">”), and other elements referred to as “child elements” (e.g., “<elementl><element2>content item</element2></elementl>”). An “attribute” may refer to a markup construct including a name-value pair that exists within a start tag or empty element tag. Attributes contain data related to its element and/or control the element’s behavior.

The term “channel” as used herein refers to any transmission medium, either tangible or intangible, which is used to communicate data or a data stream. The term “channel” may be synonymous with and/or equivalent to “communications channel,” “data communications channel,” “transmission channel,” “data transmission channel,” “access channel,” “data access channel,” “link,” “data link,” “carrier,” “radiofrequency carrier,” and/or any other like term denoting a pathway or medium through which data is communicated. Additionally, the term “link” as used herein refers to a connection between two devices through a RAT for the purpose of transmitting and receiving information. As used herein, the term “radio technology” refers to technology for wireless transmission and/or reception of electromagnetic radiation for information transfer. The term “radio access technology” or “RAT” refers to the technology used for the underlying physical connection to a radio based communication network. As used herein, the term “communication protocol” (either wired or wireless) refers to a set of standardized rules or instructions implemented by a communication device and/or system to communicate with other devices and/or systems, including instructions for packetizing/depacketizing data, modulating/demodulating signals, implementation of protocols stacks, and/or the like.

As used herein, the term “radio technology” refers to technology for wireless transmission and/or reception of electromagnetic radiation for information transfer. The term “radio access technology” or “RAT” refers to the technology used for the underlying physical connection to a radio based communication network. As used herein, the term “communication protocol” (either wired or wireless) refers to a set of standardized rules or instructions implemented by a communication device and/or system to communicate with other devices and/or systems, including instructions for packetizing/depacketizing data, modulating/demodulating signals, implementation of protocols stacks, and/or the like. Examples of wireless communications protocols may be used in various embodiments include a Global System for Mobile Communications (GSM) radio communication technology, a General Packet Radio Service (GPRS) radio communication technology, an Enhanced Data Rates for GSM Evolution (EDGE) radio communication technology, and/or a Third Generation Partnership Project (3 GPP) radio communication technology including, for example, 3 GPP Fifth Generation (5G) or New Radio (NR), Universal Mobile Telecommunications System (UMTS), Freedom of Multimedia Access (FOMA), Long Term Evolution (LTE), LTE- Advanced (LTE Advanced), LTE Extra, LTE-A Pro, cdmaOne (2G), Code Division Multiple Access 2000 (CDMA 2000), Cellular Digital Packet Data (CDPD), Mobitex, Circuit Switched Data (CSD), High-Speed CSD (HSCSD), Universal Mobile Telecommunications System (UMTS), Wideband Code Division Multiple Access (W-CDM), High Speed Packet Access (HSPA), HSPA Plus (HSPA+), Time Division-Code Division Multiple Access (TD-CDMA), Time Division-Synchronous Code Division Multiple Access (TD-SCDMA), LTE LAA, MuLTEfire, UMTS Terrestrial Radio Access (UTRA), Evolved UTRA (E-UTRA), Evolution- Data Optimized or Evolution-Data Only (EV-DO), Advanced Mobile Phone System (AMPS), Digital AMPS (D-AMPS), Total Access Communication System/Extended Total Access Communication System (TACS/ETACS), Push-to-talk (PTT), Mobile Telephone System (MTS), Improved Mobile Telephone System (IMTS), Advanced Mobile Telephone System (AMTS), Cellular Digital Packet Data (CDPD), DataTAC, Integrated Digital Enhanced Network (iDEN), Personal Digital Cellular (PDC), Personal Handy-phone System (PHS), Wideband Integrated Digital Enhanced Network (WiDEN), iBurst, Unlicensed Mobile Access (UMA), also referred to as also referred to as 3GPP Generic Access Network, or GAN standard), Bluetooth®, Bluetooth Low Energy (BLE), IEEE 802.15.4 based protocols (e.g., IPv6 over Low power Wireless Personal Area Networks (6L0WPAN), WirelessHART, M1W1, Thread, 802.11a, etc.) WiFi-direct, ANT/ANT+, ZigBee, Z-Wave, 3GPP device-to-device (D2D) or Proximity Services (ProSe), Universal Plug and Play (UPnP), Low-Power Wide- Area-Network (LPWAN), Long Range Wide Area Network (LoRA) or LoRaWAN™ developed by Semtech and the LoRa Alliance, Sigfox, Wireless Gigabit Alliance (WiGig) standard, Worldwide Interoperability for Microwave Access (WiMAX), mmWave standards in general (e g., wireless systems operating at 10-300 GHz and above such as WiGig, IEEE 802. Had, IEEE 802. Hay, etc.), V2X communication technologies (including 3GPP C-V2X), Dedicated Short Range Communications (DSRC) communication systems such as Intelligent- Transport-Systems (ITS) including the European ITS-G5, ITS-G5B, ITS-G5C, etc. In addition to the standards listed above, any number of satellite uplink technologies may be used for purposes of the present disclosure including, for example, radios compliant with standards issued by the International Telecommunication Union (ITU), or the European Telecommunications Standards Institute (ETSI), among others. The examples provided herein are thus understood as being applicable to various other communication technologies, both existing and not yet formulated.

The term “access network” refers to any network, using any combination of radio technologies, RATs, and/or communication protocols, used to connect user devices and service providers. In the context of WLANs, an “access network” is an IEEE 802 local area network (LAN) or metropolitan area network (MAN) between terminals and access routers connecting to provider services. The term “access router” refers to router that terminates a medium access control (MAC) senice from terminals and forwards user traffic to information servers according to Internet Protocol (IP) addresses.

The term “SMTC” refers to an SSB-based measurement timing configuration configured by SSB-MeasurementTimingConfiguration. The term “SSB” refers to a synchronization signal/Physical Broadcast Channel (SS/PBCH) block, which includes a Primary Syncrhonization Signal (PSS), a Secondary Syncrhonization Signal (SSS), and a PBCH. The term “a “Primary Cell” refers to the MCG cell, operating on the primary frequency, in which the UE either performs the initial connection establishment procedure or initiates the connection re-establishment procedure. The term “Primary SCG Cell” refers to the SCG cell in which the UE performs random access when performing the Reconfiguration with Sync procedure for DC operation. The term “Secondary Cell” refers to a cell providing additional radio resources on top of a Special Cell for a UE configured with CA. The term “Secondary Cell Group” refers to the subset of serving cells comprising the PSCell and zero or more secondary cells for a UE configured with DC. The term “Serving Cell” refers to the primary cell for a UE in RRC_CONNECTED not configured with CA/DC there is only one serving cell comprising of the primary cell. The term “serving cell” or “serving cells” refers to the set of cells comprising the Special Cell(s) and all secondary' cells for a UE in RRC_CONNECTED configured with CA. The term “Special Cell” refers to the PCell of the MCG or the PSCell of the SCG for DC operation; otherwise, the term “Special Cell” refers to the Pcell.

The term “Al policy” refers to a type of declarative policies expressed using formal statements that enable the non-RT RIC function in the SMO to guide the near-RT RIC function, and hence the RAN, towards better fulfilment of the RAN intent.

The term “Al Enrichment information” refers to information utilized by near-RT RIC that is collected or derived at SMO/non-RT RIC either from non-network data sources or from network functions themselves.

The term “Al -Policy Based Traffic Steering Process Mode” refers to an operational mode in which the Near-RT RIC is configured through Al Policy to use Traffic Steering Actions to ensure a more specific notion of network performance (for example, applying to smaller groups of E2 Nodes and UEs in the RAN) than that which it ensures in the Background Traffic Steering.

The term “Background Traffic Steering Processing Mode” refers to an operational mode in which the Near-RT RIC is configured through 01 to use Traffic Steering Actions to ensure a general background network performance which applies broadly across E2 Nodes and UEs in the RAN.

The term “Baseline RAN Behavior” refers to the default RAN behavior as configured at the E2 Nodes by SMO

The term “E2” refers to an interface connecting the Near-RT RIC and one or more O- CU-CPs, one or more O-CU-UPs, one or more O-DUs, and one or more O-eNBs.

The term “E2 Node” refers to a logical node terminating E2 interface. In this version of the specification, ORAN nodes terminating E2 interface are: for NR access: O-CU-CP, O- CU-UP, O-DU or any combination; and for E-UTRA access: O-eNB.

The term “Intents”, in the context of O-RAN systems/implementations, refers to declarative policy to steer or guide the behavior of RAN functions, allowing the RAN function to calculate the optimal result to achieve stated objective.

The term “O-RAN non-real-time RAN Intelligent Controller” or “non-RT RIC” refers to a logical function that enables non-real-time control and optimization of RAN elements and resources, AI/ML workflow including model training and updates, and policy-based guidance of apphcations/features in Near-RT RIC.

The term “Near-RT RIC” or “O-RAN near-real-time RAN Intelligent Controller” refers to a logical function that enables near-real-time control and optimization of RAN elements and resources via fine-grained (e.g., UE basis, Cell basis) data collection and actions over E2 interface.

The term “O-RAN Central Unit” or “O-CU” refers to a logical node hosting RRC, SDAP and PDCP protocols.

The term “O-RAN Central Unit - Control Plane” or “O-CU-CP” refers to a logical node hosting the RRC and the control plane part of the PDCP protocol.

The term “O-RAN Central Unit - User Plane” or “O-CU-UP” refers to a logical node hosting the user plane part of the PDCP protocol and the SDAP protocol

The term “O-RAN Distributed Unit” or “O-DU” refers to a logical node hosting RLC/MAC/High-PHY layers based on a lower layer functional split.

The term “O-RAN eNB” or “O-eNB” refers to an eNB or ng-eNB that supports E2 interface.

The term “O-RAN Radio Unit” or “O-RU” refers to a logical node hosting Low-PHY layer and RF processing based on a lower layer functional split. This is similar to 3GPP’s “TRP” or “RRH” but more specific in including the Low-PHY layer (FFT/iFFT, PRACH extraction). The term “01” refers to an interface between orchestration & management entities (Orchestration/NMS) and 0-RAN managed elements, for operation and management, by which FCAPS management, Software management, File management and other similar functions shall be achieved.

The term “RAN UE Group” refers to an aggregations of UEs whose grouping is set in the E2 nodes through E2 procedures also based on the scope of Al policies. These groups can then be the target of E2 CONTROL or POLICY messages.

The term “Traffic Steering Action” refers to the use of a mechanism to alter RAN behavior. Such actions include E2 procedures such as CONTROL and POLICY.

The term “Traffic Steering Inner Loop” refers to the part of the Traffic Steering processing, triggered by the arrival of periodic TS related KPM (Key Performance Measurement) from E2 Node, which includes UE grouping, setting additional data collection from the RAN, as well as selection and execution of one or more optimization actions to enforce Traffic Steering policies.

The term “Traffic Steering Outer Loop” refers to the part of the Traffic Steering processing, triggered by the near-RT RIC setting up or updating Traffic Steering aware resource optimization procedure based on information from Al Policy setup or update, Al Enrichment Information (El) and/or outcome of Near-RT RIC evaluation, which includes the initial configuration (preconditions) and injection of related Al policies, Triggering conditions for TS changes.

The term “Traffic Steering Processing Mode” refers to an operational mode in which either the RAN or the Near-RT RIC is configured to ensure a particular network performance. This performance includes such aspects as cell load and throughput, and can apply differently to different E2 nodes and UEs. Throughout this process, Traffic Steering Actions are used to fulfill the requirements of this configuration.

The term “Traffic Steering Target” refers to the intended performance result that is desired from the network, which is configured to Near-RT RIC over 01.

Furthermore, any of the disclosed embodiments and example implementations can be embodied in the form of various types of hardware, software, firmware, middleware, or combinations thereof, including in the form of control logic, and using such hardware or software in a modular or integrated manner. Additionally, any of the software components or functions described herein can be implemented as software, program code, script, instructions, etc., operable to be executed by processor circuitry. These components, functions, programs, etc., can be developed using any suitable computer language such as, for example, Python, PyTorch, NumPy, Ruby, Ruby on Rails, Scala, Smalltalk, Java™, C++, C#, “C”, Kotlin, Swift, Rust, Go (or “Golang”), EMCAScript, JavaScript, TypeScript, Jscript, ActionScript, Server- Side JavaScript (SSJS), PHP, Pearl, Lua, Torch/Lua with Just-In Time compiler (LuaJIT), Accelerated Mobile Pages Script (AMPscript), VBScript, JavaServer Pages (JSP), Active Server Pages (ASP), Node.js, ASP.NET, JAMscript, Hypertext Markup Language (HTML), extensible HTML (XHTML), Extensible Markup Language (XML), XML User Interface Language (XUL), Scalable Vector Graphics (SVG), RESTful API Modeling Language (RAML), wiki markup or Wikitext, Wireless Markup Language (WML), Java Script Object Notion (JSON), Apache® MessagePack™, Cascading Stylesheets (CSS), extensible sty lesheet language (XSL), Mustache template language, Handlebars template language, Guide Template Language (GTL), Apache® Thrift, Abstract Syntax Notation One (ASN. 1), Google® Protocol Buffers (protobuf), Bitcoin Script, EVM® bytecode, Solidity™, Vyper (Python derived), Bamboo, Lisp Like Language (LLL), Simplicity provided by Blockstream™, Rholang, Michelson, Counterfactual, Plasma, Plutus, Sophia, Salesforce® Apex®, and/or any other programming language or development tools including proprietary programming languages and/or development tools. The software code can be stored as a computer- or processorexecutable instructions or commands on a physical non-transitory computer-readable medium. Examples of suitable media include RAM, ROM, magnetic media such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk), flash memory, and the like, or any combination of such storage or transmission devices.

ABBREVIATIONS

Unless used differently herein, terms, definitions, and abbreviations may be consistent with terms, definitions, and abbreviations defined in 3GPP TR 21.905 vl6.0.0 (2019-06). For the purposes of the present document, the following abbreviations may apply to the examples and embodiments discussed herein.

Table 1 Abbreviations:

The foregoing description provides illustration and description of various example embodiments, but is not intended to be exhaustive or to limit the scope of embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments. Where specific details are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that the disclosure can be practiced without, or with variation of, these specific details. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.