CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit and priority to U.S. Provisional Patent Application No. 63/363,592, filed on April 26, 2022, entitled “SPLIT-COMPUTE ARCHITECTURE”, the disclosure of which is incorporated by reference herein in its entirety. [0002] This application also incorporates by reference herein the disclosures to related copending applications, PCT Application No. PCT/US2023/019563, filed April 24, 2023, PCT Application No. PCT/US2023/019832, filed April 25, 2023, “SPLIT-COMPUTE ARCHITECTURE”, filed April 26, 2023 (Attorney Docket No. 0120-497W01), “PERIPHERAL DEVICES IN A SPLIT-COMPUTE ARCHITECTURE”, filed April 26, 2023 (Attorney Docket No. 0120-498W01), “MULTIPLE APPLICATION RUNTIMES IN A SPLIT-COMPUTE ARCHITECTURE”, filed April 26, 2023 (Attorney Docket No. 0120- 505WO1), and “MACHINE LEARNING PROCESSING OFFLOAD IN A SPLIT-COMPUTE ARCHITECTURE”, filed April 26, 2023 (Attorney Docket No. 0120-509W01).

FIELD

[0003] Implementations relate to a wearable device processing architecture.

BACKGROUND

[0004] Some devices (e.g., wearable devices) can have advanced display capabilities. These devices can be challenged to fit sufficiently capable electronics into a small form factor. These issues become increasingly challenging in applications, such as accessibility, where a device might be expected to be worn for a full day.

[0005] Existing commercially available systems cannot support continuous usage scenarios. For example, wearable devices (e.g., smart glasses, smart watches, head mounted displays, and the like) are intended for intermittent engagement and are built around phone-class System-on- Chips (SoCs). These devices can provide only a few hours of battery life with a display on. In addition, thermal comfort can be an issue due to the small volume head mounted displays.

SUMMARY

[0006] Example implementations include a wearable device and a companion device using a split-compute architecture. To conserve wearable device resources when processing data using a machine learning model, the companion device can process layers of the machine learning model that otherwise would be performed by the wearable device. The processing of the data can be offloaded from the wearable device to the companion device if communicating data uses fewer resources than processing the data using a layer of the machine learning model and/or if the companion device can improve the processing of the machine learning model.

[0007] In a general aspect, a device, a system, a non-transitory computer-readable medium (having stored thereon computer executable program code which can be executed on a computer system), and/or a method can perform a process with a method including communicatively coupling a wearable device with a companion device, initiating, by the wearable device, a computing process that includes a machine learning model including a plurality of layers, prior to processing data using a layer of the plurality of layers, determining, by the wearable device, whether to process the data using the wearable device or the companion device, in response to determining the wearable device is to process the data, processing, by the wearable device, the data using the layer, and in response to determining the companion device is to process the data, communicating, by the wearable device to the companion device, the data, information associated with the machine learning model, and information associated with the layer.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Example implementations will become more fully understood from the detailed description given herein below and the accompanying drawings, wherein like elements are represented by like reference numerals, which are given by way of illustration only and thus are not limiting of the example implementations.

[0009] FIG. 1 illustrates a block diagram of a high-level split-compute architecture according to an example implementation.

[0010] FIG. 2 illustrates a block diagram of the high-level split-compute architecture with a shared runtime environment according to an example implementation.

[0011] FIG. 3 illustrates a block diagram of a wearable device split-compute architecture according to an example implementation.

[0012] FIG. 4 illustrates a block diagram of a high-level split-compute architecture according to an example implementation.

[0013] FIG. 5 illustrates a block diagram of activity elements in a wearable device application according to an example implementation.

[0014] FIG. 6 illustrates a block diagram of a wearable device application in a wearable device runtime environment according to an example implementation. [0015] FIG. 7 is a block diagram illustrating a machine learning model according to an example implementation.

[0016] FIG. 8 illustrates a block diagram of a system using a split-compute architecture according to an example implementation.

[0017] FIG. 9 is a block diagram of a machine learning model according to an example implementation.

[0018] FIG. 10 is a block diagram of a method of operating a wearable device according to an example implementation.

[0019] FIG. 11 is a block diagram of a method of operating a companion device according to an example implementation.

[0020] It should be noted that these Figures are intended to illustrate the general characteristics of methods, and/or structures utilized in certain example implementations and to supplement the written description provided below. These drawings are not, however, to scale and may not precisely reflect the precise structural or performance characteristics of any given implementation and should not be interpreted as defining or limiting the range of values or properties encompassed by example implementations. For example, the positioning of modules and/or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in the various drawings is intended to indicate the presence of a similar or identical element or feature.

DETAILED DESCRIPTION

[0021] Wearable devices can provide only a few hours of battery life with a display on. For example, computationally expensive operations such as image rendering, distortion correction, location services, and/or the like for wearable display optics may not always be possible to execute efficiently on a low-power embedded system implemented in the wearable device. Therefore, display architectures for thin-client wearable devices (e.g., smart glasses), however, can have the opportunity to reduce device onboard power and thermal footprint, by offloading computation-intensive operations (e.g., graphics operations) to a companion device (e.g., a mobile device, a smartphone, a server, and/or the like). In some implementations, reducing device onboard power and thermal footprint can include offloading the processing of data from the wearable device to the companion device if communicating data uses fewer resources than processing the data using a machine learning model or a portion of the machine learning model. In some implementations, performance improvements can include offloading the processing of data from the wearable device to the companion device if performance can be improved by using a machine learning model or a portion of the machine learning model on the companion device.

[0022] Some wearable devices can have implementation constraints. For example, a smart glasses implementation constraint can include (1) smart glasses should amplify key services through wearable computing. This can include supporting technologies such as AR and visual perception. For example, a smart glasses implementation constraint can include (2) smart glasses should last a full day of use on a single charge. For example, a smart glasses implementation constraint can include (3) smart glasses should look and feel like real glasses. Wearable devices can include augmented reality (AR) and virtual reality (VR) devices. Wearable devices can include smart glasses, head worn devices, and/or head mount devices. The wearable devices, head worn devices, and/or head mount devices can be AR/VR devices. Fully stand-alone wearable devices (e.g., smart glasses) solutions with mobile SoCs that have the capability to support the desired features may not meet the power and industrial design constraints listed above. On-device compute solutions that meet constraints (1), (2) and (3) may be difficult to achieve with current technologies. Current technology is limited in that standalone solutions with mobile system on a chip (SoC) technologies that meet the functionality requirements won’t meet the power and industrial design constraints listed above.

[0023] A split-compute architecture can be used to solve the problems associated with existing wearable devices technology implementations meeting the aforementioned constraints. A splitcompute architecture can be an architecture that moves an application runtime environment to a remote compute endpoint, such as a smartphone, a server, the cloud, a desktop computer, and the like, hereinafter often referred to as a companion device. In some implementations, display content can be streamed from the companion device back to the wearable device. Continuing the smart glasses example, the majority of the compute and rendering does not happen on the smart glasses, therefore the split-compute architecture can allow leveraging low-power processor and/or low-power microcontroller MCU based systems. In some implementations, the split-compute architecture combined with a wearable device including an MCU can allow minimizing power usage, meeting constraints (1) and (2) and (3). With new innovation in codecs and networking, it is possible to sustain the required networking bandwidth in a low power manner. In some implementations, wearable devices can communicate with a companion device over a well-defined protocol. This architecture can be platform independent.

[0024] FIG. 1 illustrates a block diagram of a high-level split-compute architecture according to an example implementation. As shown in FIG. 1, the split-compute architecture can include a wearable device 105 and a companion device 110. In an example implementation, the wearable device 105 can be smart glasses, an augmented reality/virtual reality headset, a head mounted display (HMD), a smart watch, a smart ring, and/or the like. As an example, FIG. 1 shows the wearable device 105 as smart glasses 135. In an example implementation, the companion device 110 can be another wearable device, a mobile device, a smart phone, a tablet, a server, and a device including a processor and an operating system. As an example, FIG. 1 shows the companion device 110 as smart phone 140.

[0025] The wearable device 105 and the companion device 110 can be communicatively coupled. For example, the wearable device 105 and the companion device 110 can be communicatively coupled wired or wirelessly. In other words, the wearable device 105 and the companion device 110 can be endpoints of a two-way, wired and/or wireless, communication link. As an example, the wearable device 105 and the companion device 110 can communicate an audio stream and/or video stream over communications line 115. As an example, the wearable device 105 and the companion device 110 can communicate data (e.g., IMU, camera, input, and/or the like) over communications line 120. As an example, the wearable device 105 and the companion device 110 can communicate a control stream over communications line 125. Communications line 115, communications line 120, and/or communications line 125 can be referred to collectively as communications line 130. In some implementations, the communications line 130 can be bi-directional.

[0026] In some implementations, the companion device 110 can include a runtime environment the wearable device 105 can connect to. In some implementations, the wearable device 105 can stream data such as IMU and camera imagery into the runtime environment. In some implementations, the runtime environment of the companion device 110 can be configured to perform the tracking, perception and/or application rendering and/or deliver the output back to the wearable device 105 through a graphics application programming interface (API), such as encoded video or rendering commands. In some implementations, the runtime environment can be a code or software application and/or container executing on the companion device 110. In some implementations, the runtime environment can be software operating on the companion device 110 at the same time as the wearable device 105 and/or during a time when the wearable device 105 and the companion device 110 are communicatively coupled. In some implementations, input is captured from the wearable device 105 and injected into (e.g., communicated to) the runtime environment of the companion device 110.

[0027] As an example, absent the companion device 110, a computing process would entirely be executed on the wearable device 105. The computing process can be any computing process associated with the function of the wearable device. For example, the computing process can be a computing process configured to display content (e.g., as an image or video) on a display of the wearable device 105. The computing process can be any computing process including at least one task (or a plurality of tasks). The at least one task can be computer instructions (e.g., code) stored on a memory of the wearable device 105 that are executed by a processor of the wearable device 105. The at least one task can be computer instructions (e.g., code) stored on a memory of the companion device 110 that are executed by a processor of the companion device 110.

[0028] The at least one task can be computer instructions (e.g., code) configured to generate a result. The result can be an intermediate result or output of the computing process. The result can be a completed or final result or output of the computing process. The at least one task can be instructions executed by the processor of the wearable device 105. The at least one task can be instructions executed by the processor of the companion device 110. In some implementations, a task(s) can be implemented as a service. A service can be, for example, a machine-to-machine interaction over a network. A service can be implemented as a background operation. For example, a service can perform network transactions, play audio, perform I/O, interact with a content provider, and/or the like from the background.

[0029] By including the companion device 110 (e.g., the runtime environment of the companion device 110), device onboard power usage and thermal footprint can be reduced by offloading one or more tasks of the plurality of tasks to the companion device 110. In an example implementation, the plurality of tasks can be fragmented. Fragmenting tasks can include methodically and/or randomly assigning plurality of tasks between the wearable device 105 and the companion device 110. For example, methodically assigning plurality of tasks between the wearable device 105 and the companion device 110 can be based on resource usage. For example, if the amount of resources used to cause a task (or tasks) to be executed (noting that executing a task includes performing computer operations) on the companion device 110 is greater than executing the task (or tasks) on the wearable device 105, then the task (or tasks) can be executed on the wearable device 105.

[0030] For example, a task (or tasks) can include and/or use computer data (e.g., a high- resolution image captured by a camera of the wearable device 105). If communicating the computer data uses more resources (e.g., battery resources of the wearable device 105) than a task including processing the data by the wearable device 105, the wearable device 105 could be assigned to complete the task. Otherwise, the companion device 110 should be assigned to complete the task. In this example implementation, both the wearable device 105 and the companion device 110 are capable of performing the task (or tasks). In some implementations, two or more companion devices may be used to complete the process including the plurality of tasks.

[0031] In an example implementation, the task can be associated with a machine learning model. For example, the machine learning model can include a plurality of layers. Each of the plurality of layers can include at least one neuron. For example, the machine learning model can include a plurality of convolution layers. Each of the plurality of convolution layers can include at least one convolution. In an example implementation, the wearable device 105 and the companion device 110 each include a same machine learning model and are capable of processing the machine learning model. Therefore, in an example implementation, if communicating data that is to be processed by a layer, neuron, convolution layer, convolution, and/or the like uses more resources of the wearable device 105 than processing the data on the wearable device 105, the wearable device 105 can process the data. However, if communicating data that is to be processed by a layer, neuron, convolution layer, convolution, and/or the like uses fewer resources of the wearable device 105 than processing the data on the wearable device 105, the wearable device 105 can communicate the data to the companion device 110 and the companion device can use the machine learning model to process the data. Moving the processing of the data by the machine learning model from the wearable device 105 to the companion device 110 can conserve resources of the wearable device 105.

[0032] FIG. 2 illustrates a block diagram of the high-level split-compute architecture with a shared runtime environment according to an example implementation. As shown in FIG. 2, the wearable device 105 is communicatively coupled to two or more companion devices 110-1, 110-2, ... 110-n via communications lines 130-1, 130-2, ... 130-n respectively. In some implementations, the wearable device 105 could roam between various companion devices 110- 1, 110-2, ... 110-n, selecting the companion devices 110-1, 110-2, ... 110-n that provides the best experience at that point in time. The wearable device 105 can connect to multiple companion devices 110-1, 110-2, ... 110-n runtimes simultaneously. Therefore, the wearable device 105 can connect to multiple runtime environments simultaneously.

[0033] For example, the runtime environment associated with companion device 110-1 can be configured to project content onto a display of the wearable device 105 while the runtime environment associated with companion device 110-2 can be configured to access the data sources (e.g., sensors) from the wearable device 105, a database server, the internet, and/or the like for processing. In addition, or alternatively, the companion devices 110-1, 110-2, ... 110-n can be communicatively coupled (e.g., wired or wirelessly) to share resources and/or data. For example, companion device 110-1 and companion device 110-2 can be communicatively coupled enabling the runtime environment associated with companion device 110-1 to receive data from the runtime environment associated with companion device 110-2 for use when the runtime environment associated with companion device 110-2 generates images (or frames) to project content onto the display of the wearable device 105.

[0034] For example, companion device 110-1 and companion device 110-2 can be communicatively coupled (e.g., via communications line 205) enabling the runtime environment associated with companion device 110-2 to share processing resources with the runtime environment associated with companion device 110-1. For example, the runtime environment associated with companion device 110-1 may be instructed to generate content (e.g., an image, a frame, a map, and/or the like) that is more efficiently generated by the runtime environment associated with companion device 110-2. For example, companion device 110-2 may include a map database and map (e.g., image) generator. The runtime environment associated with companion device 110-1 may be instructed to generate content that includes a map. In this example, the runtime environment associated with companion device 110-1 can request a map from the runtime environment associated with companion device 110-2.

[0035] Continuing the example above, should a determination be made that the companion device 110 is to perform the task(s), the companion device 110 (or the wearable device 105) can determine that two or more companion devices 110 can perform the task(s). For example, the task(s) can include generating content (e.g., an image) to be displayed on the wearable device 105. In this example, the companion device 110-1 can be configured to generate the content. However, data used to generate the content may be generated by the companion device 110-2 which is then communicated (e.g., via communications line 205) from the companion device 110-2 to the companion device 110-1. For example, the companion device 110-2 can be a wearable smart watch configured to sense a user heart rate. Data representing the heart rate can be communicated from the companion device 110-2 to the companion device 110-1. Then, the companion device 110-1 can generate content using the data representing the heart rate and the content can be communicated to the wearable device 105 for display on a display of the wearable device 105.

[0036] FIG. 3 illustrates a block diagram of a wearable device split-compute architecture according to an example implementation. As shown in FIG. 3, the wearable device splitcompute architecture can include a hardware abstraction layer (HAL) 310 block. The HAL 310 can be a layer of software configured to interface between an operating system (e.g., RTOS 340) and a hardware device at a general or abstract level rather than at a hardware level. In some implementations, using abstraction layers can make the split-compute architecture platform independent. The HAL 310 can be called from the operating system kernel. In some implementations, the HAL 310 can be a virtual HAL. The virtual HAL can minimize interpretation latency based on the similarity in architectures of a guest and a host platform. Virtualization technique helps map the virtual resources to physical resources and use the native hardware for computations in the virtual HAL.

[0037] Accordingly, as an example, the HAL 310 can include a connectivity 315 block, a codec 320 block, a graphics processing unit (GPU) 325 block, and a display 330 block each configured to interface with a corresponding hardware device. For example, the connectivity 315 can be configured to interface between the operating system (e.g., RTOS 340) and Bluetooth hardware, WIFI hardware, ultra-wideband (UWB) hardware, 5G hardware, and/or the like. Therefore, the wearable device split-compute architecture can be configured to utilize any connectivity hardware designed into the wearable device 105.

[0038] For example, the codec 320 can be configured to interface between the operating system (e.g., RTOS 340) and an encoder and/or a decoder (e.g., a hardware-based encoder and/or decoder). The codec 320 standard can be, for example, H.265, H.264, MPEG, VP9, machine learned, and/or the like. The codec 320 can be an image, video, and/or audio codec. Therefore, the wearable device split-compute architecture can be configured to utilize any codec software and/or hardware designed into the wearable device 105. For example, the GPU 325 can be configured to interface between the operating system (e.g., RTOS 340) and GPU hardware (e.g., a GPU ASIC). Therefore, the wearable device split-compute architecture can be configured to utilize any GPU (e.g., to render an image on a display system) designed into the wearable device 105. For example, the display 330 can be configured to interface between the operating system (e.g., RTOS 340) and display hardware (e.g., a wearable device display). Therefore, the wearable device split-compute architecture can be configured to utilize any display (e.g., display driver system) designed into the wearable device 105.

[0039] As shown in FIG. 3, the wearable device split-compute architecture can include an operating system (OS) abstraction layer (OSAL) 335. In some implementations, using abstraction layers can make the split-compute architecture platform independent. The OSAL 335 can be configured to provide an interface to common system functions offered by the OS of the wearable device 105. These OSAL 335 can simplify development and porting software (e.g., applications) to multiple OS and hardware platforms. In some implementations, the OSAL 335 can operate as (or similar to) an application programming interface (API). In some implementations, the OSAL 335 can be platform dependent. [0040] The OSAL 335 can include a real-time operating system (RTOS) 340 block. The RTOS 340 can be configured to process, for example, multi -threaded applications to meet real-time deadlines. For example, the RTOS 340 can be configured to process a plurality of tasks each (or a grouping of tasks) having a maximum completion time. Although an RTOS is illustrated, any OS can be used. For example, a high-level operating system (HLOS) can be used. In a splitcompute system, using tasks enables distribution of these tasks (or groups of tasks) between computing devices. For example, a content display operation for the wearable device 105 can be divided between the wearable device 105 and the companion device 110 (e.g., the runtime environment associated with companion device 110). In other words, the wearable device 105 (and/or the companion device 110) can be configured to (e.g., using the wearable device splitcompute architecture) cause the companion device 110 to perform a portion (a task or a grouping of tasks) of a process (e.g., content display).

[0041] As shown in FIG. 3, the wearable device split-compute architecture can include a peripheral drivers 345 block. The peripheral drivers 345 can be configured to interface between the OS and a peripheral device. For example, the wearable device 105 can include a plurality of peripheral devices including, for example, a camera(s), a microphone(s), a speaker(s), an input(s), an inertial measurement unit(s) (IMU), and/or the like. Therefore, the peripheral drivers 345 can be configured to interface between the RTOS 340 and the peripheral device(s) of the wearable device 105. For example, the peripheral drivers 345 can sense and communicate data, as peripheral data, (e.g., IMU data) to the application module 630 via the device client 305 and the core 420 using the communications line 120. In other words, peripheral data can be the data that is collected by and/or processed by (e.g., compressed, packaged, parsed, filtered, denoised, and/or the like) the peripheral device via the peripheral drivers 345 and packaged for communication with and use by the wearable device 105 and/or the companion device 110.

[0042] As shown in FIG. 3, the wearable device split-compute architecture can include a device client 305. The device client 305 can be configured to control communication between the wearable device 105 and the companion device 110. For example, device client 305 can be configured to generate, initialize, and control the communications line 130 and the communications over the communications line 130. The communications line 130 can operate as, for example, a socket (e.g., a network socket, a TCP/IP network socket, and the like). A socket can be one endpoint of a two-way communication link between computer code (e.g., applications, programs, software systems, and/or the like) running on two computing devices. In some implementations, bi-directional handshaking can be used so that both the client and the host can be configured to indicate what capabilities each support. The socket mechanism can be configured to provide inter-process communication (IPC) by establishing named communication contact points between two endpoints and/or between two endpoints and an intermediate device (e.g., an access point (AP)). A socket can be configured to provide a bidirectional first-in first-out (FIFO) communication channel. A socket connecting to the network is created at each end of the communication. As an example, each socket can have an address (or memory location). The address (or memory location) can be, for example, an IP address and a port number. Accordingly, the device client 305 can be configured to write to and read from a socket associated with the companion device 110 (or the runtime environment associated with companion device 110). In some implementations, the device client 305 can be platform independent.

[0043] In some implementations, a software development kit (SDK) can be associated with the wearable device 105 and the companion device 110. The SDK can be used when developing applications for the wearable device 105 and/or the companion device 110. The SDK can enable the implementation of the split-compute architecture. Therefore, any wearable device 105 and/or companion device 110 (regardless of the hardware and/or software platform) that includes the split-compute architecture can use an application that was developed using the SDK. Therefore, an application does not have to be developed and ported to each hardware and/or software platform that may be used as a wearable device 105 and/or the companion device 110. The SDK can be included (or have elements that can be included) with the application when the application is installed on the wearable device 105 and/or the companion device 110.

[0044] FIG. 4 illustrates a high-level split-compute architecture according to an example implementation. As shown in FIG. 4, a system can include the wearable device 105 and the companion device 110. The split-compute architecture for the system can include a device client 305 block associated with the wearable device 105, an application 410 block, an SDK 415 block, and a core 420 block associated with the companion device 110. As mentioned above, the device client 305 can be configured to control communication between the wearable device 105 and the companion device 110. The core 420 can be configured to control communication between the companion device 110 and the wearable device 105. Accordingly, the core 420 can be configured to, at least, generate, initialize, and control the communications line 130 and the communications over the communications line 130. The communications line 130 can operate as, for example, a socket (as described above).

[0045] In some implementations, the application 410 can include the file format for applications used on the OS that holds the application logic (e.g., the Android package kit (APK)). In some implementations, wearable device applications can link with a stub (e.g., not a complete) version of the SDK to support compile and testing services. In some implementations, at runtime, the application 410 can load the wearable device SDK 415 directly from a wearable device runtime environment. In some implementations, the SDK 415 can provide developers the application programming interface (API) used to build wearable device applications. In some implementations, an API versioning scheme allows introduction of new APIs while maintaining backwards compatibility. In some implementations, the wearable device runtime environment can be a collection of core services responsible for maintaining the wearable device execution environment. In some implementations, wearable device applications may not interact with the core services directly. In some implementations, one or more interaction may go through the SDK. In some implementations, the device client 305 can be a thin client running on the wearable device 105 hardware.

[0046] As an example, the application 410 can be configured to generate (or help generate) content for display on the wearable device 105. For example, the application can be configured to process a task(s). For example, the application can generate the content (e.g., as an image) and communicate the content to the core 420 via the SDK 415. Communicating the content to the core 420 via the SDK 415 can be one of the features that allows the application 410 to be developed for any hardware and/or software platform. For example, the SDK 415 can be configured to communicate with the application 410 when the application 410 is developed. The SDK 415 can also be configured to communicate with the core 420 associated with a plurality of hardware and/or software platforms. After receiving the content, the core 420 can communicate the content to the wearable device 105 via the device client 305 using, for example, a pre-established socket.

[0047] As mentioned above, in some implementations, the wearable device 105 can be configured to connect to more than one companion device 110 at a given time. In some implementations, a different companion device 110 can be configured to provide different services (e.g., using an application 410). In some implementations, with low-latency, high- bandwidth 5G connections becoming mainstream, the companion device 110 can be configured to operate in the cloud (e.g., connecting through 5G standards).

[0048] FIG. 5 illustrates a block diagram of activity elements in a wearable device application according to an example implementation. As shown in FIG. 5, the application 410 can include a wearable activity 505 block, a wearable activity service 510 block, and a wearable activity host 515 block, and the core 420 can include a core services 520 block. [0049] In some example implementations, wearable device application design can resemble an activity model. Referring to FIG. 3, the RTOS 340 can be configured to process, a task, a plurality of tasks each (or a grouping of tasks) having a maximum completion time. Therefore, each activity can be a task executed in a parallel process and having a time (or amount of time) to be completed by.

[0050] In an example implementation, activities can be executed in a service context. In some implementations, running in a service context can allow the application 410 to run concurrently with companion device 110 applications. In some implementations, the application can continue running and rendering when a companion device 110 display is off.

[0051] In some implementations, one or more application 410 can include a wearable activity service 510. The wearable activity service 510 can be configured to expose the wearable activity 505 so that the wearable activity 505 can be instantiated by the SDK 415. Therefore, the wearable activity 505 can be instantiated at a later point in time. From a developer’s point of view, the wearable activity service 510 can be boilerplate code that does not directly relate to application 410 logic. The wearable activity 505 can be the code related to application 410 logic. In some implementations, wearable activity 505 can be managed by an activity manager and behave similarly to a standard OS activity counterpart.

[0052] In some implementations, the service binding can be used by the wearable device runtime environment to start and manage the life cycle of the application 410. Once the activity manager binds to the service as part of a launch flow, the SDK 415 can instantiate and attach a wearable activity host 515 as, for example, a class that can be responsible for general activity state control. For example, during initialization the wearable activity host 515 can request a surface from a window manager. This surface is then used as a backing store for a virtual display that’s used to render the contents of the application 410.

[0053] FIG. 6 illustrates a block diagram of a wearable device application in a wearable device runtime environment according to an example implementation. As shown in FIG. 6, a companion device OS 605 can simultaneously process two or more application environments. For example, the companion device OS 605 can include an application module 610. The application module 610 can be associated with standard OS application activity. In addition, the companion device OS 605 can include an application module 630 operating in association with a wearable runtime environment 625. The wearable runtime environment 625 can be associated with the wearable device 105.

[0054] An activity(s) 620, 640 can be a single, focused task that the application can perform. For example, some applications can include a user interface (UI). Therefore, the activity(s) 620, 640 can be configured to create a window to place the UI. The window can be a full -screen window, a floating window, embedded into other windows, a hidden window, and/or the like. The different types of windows can be associated with different activity(s) 620, 640. The activity(s) 620, 640 can be configured for any task, a window is just one example.

[0055] The activity manager 615, 635 can be configured to communicate information about, and interact with the activities 620, 640. The activity manager 615, 635 can be further configured to communicate information about, and interact with tasks, threads, services and other processes.

[0056] In an example implementation, the wearable runtime environment 625 can be a virtual runtime environment. The virtual runtime environment can be configured to operate in the background of a computing device. Therefore, the wearable runtime environment 625 can be a virtual runtime environment associated with the wearable device 105 and configured to operate as a background process on the companion device 110. In other words, the wearable runtime environment 625 can operate without a user interface shown on a display of the companion device 110. For example, the wearable runtime environment 625 can operate in a hidden window of the companion device 110. Therefore, a user of the companion device 110 may have no visual or I/O control of an application using the wearable runtime environment 625 if the wearable runtime environment 625 is a virtual runtime environment.

[0057] In an alternative, or additional implementation, the application 410 and/or the application module 630 can be a virtual process. The virtual process can be configured to operate in the background of a computing device. Therefore, the application 410 and/or the application module 630 can be a virtual process associated with the wearable device 105 and configured to operate as a background process on the companion device 110. In other words, the application 410 and/or the application module 630 can operate without a user interface shown on a display of the companion device 110. For example, the application 410 and/or the application module 630 can operate in a hidden window of the companion device 110. Therefore, a user of the companion device 110 may have no visual or VO control of the application 410 and/or the application module 630 if the process is a virtual runtime process.

[0058] FIG. 7 is a block diagram illustrating a machine learning model according to an example implementation. Example implementations can include a machine learning model 705. As shown in FIG. 7, the machine learning model 705 can include a plurality of layers in a convolutional neural network with no sparsity constraints. The layered neural network can include three (s) layers 715, 720, 725. Each layer 715, 720, 725 can be formed of a plurality of neurons 710. In this implementation, no sparsity constraints have been applied. Therefore, all neurons 710 in each layer 715, 720, 725 are networked to all neurons 710 in any neighboring layers 715, 720, 725. If sparsity constraints were applied, all neurons 710 in each layer 715, 720, 725 would not be networked to all neurons 710 in any neighboring layers 715, 720, 725.

[0059] In an example implementation, each layer 715, 720, 725 can be processes on the wearable device 105 and/or the companion device 110. For example, layer 715 can be processes on the wearable device 105 and layers 720 and 725 can be processes on the companion device 110. The wearable device 105 can be configured to determine whether or not layer 715, 720, 725 can be processes on the wearable device 105 or the companion device 110. In an example implementation, each neuron 710 can be processes on the wearable device 105 and/or the companion device 110. The wearable device 105 can be configured to determine whether or not neurons 710 can be processes on the wearable device 105 or the companion device 110.

[0060] In an example implementation, the wearable device 105 can be configured to determine whether or not layer 715, 720, 725 and/or neurons 710 can be processed on the wearable device 105 or the companion device 110 based on a comparison of the resources used to communicate the data for processing to the companion device 110 and the resources used to process the data (e.g., process a layer and/or neuron) on the wearable device 105. For example, if communicating the data uses more resources than processing the data, the wearable device 105 can process the data. For example, if communicating the data uses fewer resources than processing the data, the wearable device 105 can communicate the data for processing by the companion device 110.

[0061] FIG. 8 illustrates a block diagram of a system using a split-compute architecture according to an example implementation. As shown in FIG. 8, the system includes the wearable device 105 and the companion device 110. The wearable device 105 can include the device client 305, a machine learning model 805, and a machine learning processing manager 815. The companion device 110 can include the core 420 and the wearable runtime environment 625. The wearable runtime environment 625 can include a machine learning model 810, and a machine learning processing manager 820.

[0062] In an example implementation, the machine learning model 805 and the machine learning model 810 are configured to generate the same output based on an input. In other words, the machine learning model 805 and the machine learning model 810 are a same machine learning model trained using the same training data. In an example implementation, the machine learning model 805 is a trained machine learning model that has been copied to generate the machine learning model 810. In order to simplify the description of FIG. 8, the machine learning model 805 and the machine learning model 810 can have the structure of the machine learning model 705. [0063] Initially, the machine learning model 805 can be triggered with input data (e.g., an image captured using a camera of the wearable device 105). Prior to the wearable device 105 processing the input data using the machine learning model 805, the machine learning processing manager 815 can determine whether or not the layer 715 of the machine learning model 805 should be used to process the input data. For example, if communicating the input data to the companion device 110 uses more resources than processing the input data, the wearable device 105 can process the input data using layer 715 of the machine learning model 805. However, if communicating the input data uses fewer resources than processing the input data, the wearable device 105 can communicate the input data for processing by the companion device 110 using layer 715 of the machine learning model 810.

[0064] In this example, the machine learning processing manager 815 can determine the layer 715 of the machine learning model 805 is to be used to process the input data. After processing the input data using the layer 715 of the machine learning model 805 and before processing the resultant data using the layer 720 of the machine learning model 805, the machine learning processing manager 815 can determine whether or not the layer 720 of the machine learning model 805 should be used to process the resultant data.

[0065] In this example, the machine learning processing manager 815 can determine the layer 720 of the machine learning model 805 is not to be used to process the input data. For example, communicating the resultant data can use fewer resources than processing the resultant data. Therefore, the wearable device 105 can communicate the resultant data for processing by the companion device 110 using layer 720 of the machine learning model 810. For example, the machine learning processing manager 815 can communicate the resultant data and information about the machine learning model 805 (e.g., an identification of the machine learning model) and a processing status of the data (e.g., the next layer to be used and a mapping of the data to (each of) the plurality of neurons 710). The communication can be directed to the machine learning processing manager 820 via the device client 305 and the core 420.

[0066] The machine learning processing manager 820 can be configured to direct or input the data (e.g., using the map the data to each of the plurality of neurons 710) to layer 720 (e.g., the neurons 710 layer 720) of the machine learning model 810 and cause the data to be processed by layer 720 of the machine learning model 810 and then by layer 725 of the machine learning model 810. The output of the layer 725 of the machine learning model 810 can be the output of the machine learning model 810 which should be equivalent to the output of the machine learning model 805 had the wearable device 105 completed the process. [0067] Further, a similar determination by the machine learning processing manager 815 of whether or not to process data by a neuron 710 of the machine learning model 805 can be made. In this scenario, the data associated with an input of a neuron 710 of the machine learning model 805 can be communicated together with information about the neuron 710. Moving the processing of the data by the machine learning model from the wearable device 105 to the companion device 110 can conserve resources of the wearable device 105.

[0068] FIG. 9 is a block diagram illustrating a machine learning model (e.g., an auto encoder, an encoder-decoder network, gaze detection, head pose, object recognition, and/or the like) according to an example implementation. The machine learning model of FIG. 9 can represent a machine learning model processed on the wearable device 105 and/or the companion device 110. The machine learning model of FIG. 9 can include a first machine learning model 905 and a second machine learning model 910. In an example implementation, the first machine learning model 905 can be configured to be processed on the wearable device 105 and the second machine learning model 910 can be configured to be processed on the wearable device 105. In an example implementation, the first machine learning model 905 can be configured to be processed on the companion device 110 and the second machine learning model 910 can be configured to be processed on the companion device 110. In an example implementation, the first machine learning model 905 can be configured to be processed on the wearable device 105 and/or the companion device 110 and the second machine learning model 910 can be configured to be processed on the wearable device 105 and/or the companion device 110. In an example implementation, a portion of the first machine learning model 905 can be configured to be processed on the wearable device 105 and/or the companion device 110 and a portion of the second machine learning model 910 can be configured to be processed on the wearable device 105 and/or the companion device 110. Processing of first machine learning model 905 and the second machine learning model 910 can be any combination of the aforementioned example implementations.

[0069] A machine learning model can include at least one convolution layer(s) or convolution(s). For example, as shown in FIG. 9, the machine learning model 905 can include four (4) convolution layers each including three (3) convolutions where a first convolution layer 955-1 includes convolutions 915-1, 915-2, 915-3, a second convolution layer 955-2 includes convolutions 920-1, 920-2, 920-3, a third convolution layer 955-3 includes convolutions 925- 1, 925-2, 925-3, and a fourth convolution layer 955-4 includes convolutions 930-1, 930-2, 930- 3. Further, as shown in FIG. 9, the machine learning model 910 can include four (4) convolution layers each including three (3) convolutions, where a first convolution layer 960-1 includes convolutions 935-1, 935-2, 935-3, a second convolution layer 960-2 includes convolutions 940- 1, 940-2, 940-3, a third convolution layer 960-3 includes convolutions 945-1, 945-2, 945-3, and a fourth convolution layer 960-4 includes convolutions 950-1, 950-2, 950-3. The machine learning model can include skip connection(s) (e.g., the output of convolution 915-3 can be communicated as input to convolution 950-1) and other computations needed to realize the machine learning model outputs.

[0070] A convolution layer (e.g., convolution layer 955-1, 955-2, 955-3, 955-4, 960-1, 960-2, 960-3, and/or 960-4) or convolution can be configured to extract features from data 5 (e.g., an image) as input data. Features can be based on color, frequency domain, edge detectors, and/or the like. A convolution can have a filter (sometimes called a kernel) and a stride. For example, a filter can be a 1x1 filter (or Ixlxn for a transformation to n output channels, a 1x1 filter is sometimes called a pointwise convolution) with a stride of 1 which results in an output of a cell generated based on a combination (e.g., addition, subtraction, multiplication, and/or the like) of the features of the cells of each channel at a position of the MxM grid. In other words, a feature map having more than one depth or channel is combined into a feature map having a single depth or channel. A filter can be a 3x3 filter with a stride of 1 which results in an output with fewer cells in/for each channel of the MxM grid or feature map. The output can have the same depth or number of channels (e.g., a 3x3xn filter, where n = depth or number of channels, sometimes called a depthwise filter) or a reduced depth or number of channels (e.g., a 3x3xk filter, where k<depth or number of channels). Each channel, depth, or feature map can have an associated filter. Each associated filter can be configured to emphasize different aspects of a channel. In other words, different features can be extracted from each channel based on the filter (this is sometimes called a depthwise separable filter). Other filters are within the scope of this disclosure.

[0071] Another type of convolution can be a combination of two or more convolutions. For example, a convolution can be a depthwise and pointwise separable convolution. This can include, for example, a convolution in two steps. The first step can be a depthwise convolution (e.g., a 3x3 convolution). The second step can be a pointwise convolution (e.g., a 1x1 convolution). The depthwise and pointwise convolution can be a separable convolution in that a different filter (e.g., filters to extract different features) can be used for each channel or each depth of a feature map. In an example implementation, the pointwise convolution can transform the feature map to include c channels based on the filter. For example, an 8x8x3 feature map can be transformed to an 8x8x256 feature map based on the filter. In some implementations more than one filter can be used to transform the feature map to an MxMxc feature map. [0072] A convolution can be linear. A linear convolution describes the output, in terms of the input, as being linear time-invariant (LTI). Convolutions can also include a rectified linear unit (ReLU). A ReLU is an activation function that rectifies the LTI output of a convolution and limits the rectified output to a maximum. A ReLU can be used to accelerate convergence (e.g., more efficient computation).

[0073] In an example implementation, a combination of depthwise convolutions and depthwise and pointwise separable convolutions can be used. Each of the convolutions can be configurable (e.g., configurable feature, stride and/or depth). For example, the machine learning model 905 can include convolutions 915-1, 915-2, 915-3, 920-1, 920-2, 920-3, 925-1, 925-2, 925-3, 930- 1, 930-2, and 930-3 that can transform the data 5 into a code representation (sometimes called a feature map). The machine learning model 910 can include convolutions 935-1, 935-2, 935- 3, 940-1, 940-2, 940-3, 945-1, 945-2, 945-3, 950-1, 950-2, and 950-3 that can incrementally transform the code representation into the data 10 as output data. Data 10 can output the trained output result. For example, data 10 can represent a reconstructed image in an encoder-decoder model, a gaze direction, head pose, an object identification, and/or the like.

[0074] In an example implementation, the first machine learning model 905 can be included on and configured to be processed by the wearable device 105. Further, the first machine learning model 905 and the second machine learning model 910 can be included on and configured to be processed by the companion device 110. In an example implementation, the machine learning model 805 can be triggered with data 5 (e.g., an image captured using a camera of the wearable device 105). Prior to the wearable device 105 processing the data 5 using the first convolution layer 955-1, the machine learning processing manager 815 can determine whether or not the first convolution layer 955-1 should be used to process the data 5. For example, if communicating the data 5 to the companion device 110 uses more resources than processing the data 5 using the first convolution layer 955-1, the wearable device 105 can process the data 5 using the first convolution layer 955-1. However, if communicating the data 5 uses fewer resources than processing the data 5, the wearable device 105 can communicate the data 5 for processing by the companion device 110 using first convolution layer 955-1 on the companion device 110.

[0075] As an example, determining a quantity of resources used by the wearable device 105 to communicate data can be based on a size of the data (e.g., a number of bits, bytes, megabytes, etc.) and resources used to process (e.g., format, compress, packetize and/or the like) a unit (e.g., a bit, byte, megabyte, etc.) of data. The resources used to process a unit of data can be predetermined (e.g., previously measured) for the wearable device 105 and stored on the wearable device 105. As an example, determining a quantity of resources used by the wearable device 105 to process data by a convolution layer (or a convolution) can be based on a size of the data (e.g., a number of bits, bytes, megabytes, etc.) and resources used to process a unit (e.g., a bit, byte, megabyte, etc.) of data by the convolution layer (or a convolution). The resources used to process a unit of data can be predetermined (e.g., previously measured) for the wearable device 105 and stored on the wearable device 105. The resources used to process a unit of data can be previously measured historical data of the wearable device 105.

[0076] In an example implementation, the machine learning processing manager 815 can communicate the data and information about the machine learning model 905 (e.g., an identification of the machine learning model) and a processing status of the data (e.g., the next convolution layer to be used and a mapping of the data to the convolution or elements of the convolution). The communication can be directed to the machine learning processing manager 820 via the device client 305 and the core 420.

[0077] The comparison of resources used to communicate data to resources used to process data can be repeated for each convolution layer 955-1, 955-2, 955-3, 955-4 and/or each convolution 915-1, 915-2, 915-3, 920-1, 920-2, 920-3, 925-1, 925-2, 925-3, 930-1, 930-2, and 930-3. Moving the processing of the data by the convolution layer (or convolution) from the wearable device 105 to the companion device 110 can conserve resources of the wearable device 105.

[0078] Example 1. FIG. 10 is a block diagram of a method of operating a wearable device according to an example implementation. As shown in FIG. 10, in step SI 005 communicatively coupling a wearable device with a companion device. In step S1010 initiating, by the wearable device, a computing process that includes a machine learning model including a plurality of layers. In step S1015 prior to processing data using a layer of the plurality of layers, determining, by the wearable device, whether to process the data using the wearable device or the companion device. In step SI 020 in response to determining the wearable device is to process the data, processing, by the wearable device, the data using the layer. In step SI 025 in response to determining the companion device is to process the data, communicating, by the wearable device to the companion device, the data, information associated with the machine learning model, and information associated with the layer. A computing process can be initiated by a user of the wearable device providing corresponding user input (e.g., a gesture) to the wearable device. In some implementations, the wearable device can initiate the computing process based on another computing process, based on a spatial position of the wearable device, and/or the like. Causing a device to perform a task can include sending an instruction configured to initiate processing and/or trigger processing of a task by the device and/or another device(s). The information associated with the layer can include the machine learning model, an identification of the machine learning model, the layer identification (e.g., number), a mapping of data, a location of data, and/or the like). A map or mapping can link data to a neuron. The mapping can be a table, a database, a data structure, a linked list, and/or the like.

[0079] Example 2. The method of Example 1, wherein the initiating of the computing process can be triggered by receiving the data.

[0080] Example 3. The method of Example 1, wherein the wearable device can include the machine learning model, and the companion device includes the machine learning model.

[0081] Example 4. The method of Example 1, wherein the machine learning model can be a trained machine learning model.

[0082] Example 5. The method of Example 1, wherein the determining of whether to process the data using the wearable device or the companion device can include determining a first resource usage based on communicating the data from the wearable device to the companion device and determining a second resource usage based on processing the data using the layer on the wearable device.

[0083] Example 6. The method of Example 5, wherein the determining that the wearable device is to process the data can include determining the first resource usage is greater than the second resource usage.

[0084] Example 7. The method of Example 5, wherein the determining that the companion device is to process the data can include determining the second resource usage is greater than the first resource usage.

[0085] Example 8. The method of Example 1, wherein the wearable device can be smart glasses.

[0086] Example 9. The method of Example 1, wherein the companion device can be at least one of another wearable device, a mobile device, a smart phone, a tablet, a server, and a device including a processor and an operating system.

[0087] Example 10. FIG. 11 is a block diagram of a method of operating a companion device according to an example implementation. As shown in FIG. 11, in step SI 105 communicatively coupling a wearable device with a companion device. In step SI 110 receiving, by the companion device from the wearable device, data, information associated with a machine learning model, and information associated with a layer of the machine learning model. In step 1115 mapping, by the companion device, the data to the layer of the machine learning model based on the information associated with a layer of the machine learning model. Mapping can include, for example, using a table, a database, a linked list, a data structure, and/or the like to link, associate, and/or input data to respective neurons, convolutions, and the like of the layer. In step SI 120 processing, by the companion device, the data using the layer of the machine learning model to generate a feature map. In step SI 125 processing, by the companion device, the feature map using at least one additional layer of the machine learning model to generate an output of the machine learning model.

[0088] Example 11. The method of Example 10, wherein the wearable device can include the machine learning model, and the companion device includes the machine learning model.

[0089] Example 12. The method of Example 10, wherein the machine learning model can be a trained machine learning model.

[0090] Example 13. The method of Example 10, wherein the layer can include a plurality of plurality of neurons, and the mapping of the data includes inputting a portion of the data to a corresponding neuron of the plurality of neurons based on the information associated with a layer of the machine learning model.

[0091] Example 14. The method of Example 10, wherein the wearable device can be smart glasses.

[0092] Example 15. The method of Example 10, wherein the companion device can be at least one of another wearable device, a mobile device, a smart phone, a tablet, a server, and a device including a processor and an operating system.

[0093] Example 16. A method can include any combination of one or more of Example 1 to Example 15.

[0094] Example 17. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to perform the method of any of Examples 1-16.

[0095] Example 18. An apparatus comprising means for performing the method of any of Examples 1-16.

[0096] Example 19. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform the method of any of Examples 1-16.

[0097] Example implementations can include a non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to perform any of the methods described above. Example implementations can include an apparatus including means for performing any of the methods described above. Example implementations can include an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform any of the methods described above.

[0098] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0099] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine- readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

[00100] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (a LED (light-emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor/screen) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[00101] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[00102] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[00103] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the specification.

[00104] In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[00105] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different implementations described.

[00106] While example implementations may include various modifications and alternative forms, implementations thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit example implementations to the particular forms disclosed, but on the contrary, example implementations are to cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements throughout the description of the figures. [00107] Some of the above example implementations are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of operations may be re-arranged. The processes may be terminated when their operations are completed, but may also have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, subprograms, etc.

[00108] Methods discussed above, some of which are illustrated by the flow charts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. A processor(s) may perform the necessary tasks.

[00109] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein.

[00110] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example implementations. As used herein, the term and/or includes any and all combinations of one or more of the associated listed items.

[00111] It will be understood that when an element is referred to as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., between versus directly between, adjacent versus directly adjacent, etc.). [00112] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of example implementations. As used herein, the singular forms a, an and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms comprises, comprising, includes and/or including, when used herein, specify the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.

[00113] It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

[00114] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example implementations belong. It will be further understood that terms, e.g., those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[00115] Portions of the above example implementations and corresponding detailed description are presented in terms of software, or algorithms and symbolic representations of operation on data bits within a computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others of ordinary skill in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consi stent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[00116] In the above illustrative implementations, reference to acts and symbolic representations of operations (e.g., in the form of flowcharts) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be described and/or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits, field programmable gate arrays (FPGAs) computers or the like.

[00117] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, or as is apparent from the discussion, terms such as processing or computing or calculating or determining of displaying or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical, electronic quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[00118] Note also that the software implemented aspects of the example implementations are typically encoded on some form of non-transitory program storage medium or implemented over some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or a hard drive) or optical (e.g., a compact disk read only memory, or CD ROM), and may be read only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known to the art. The example implementations are not limited by these aspects of any given implementation.

[00119] Lastly, it should also be noted that whilst the accompanying claims set out particular combinations of features described herein, the scope of the present disclosure is not limited to the particular combinations hereafter claimed, but instead extends to encompass any combination of features or implementations herein disclosed irrespective of whether or not that particular combination has been specifically enumerated in the accompanying claims at this time.