TECHNICAL FIELD

[1] The present disclosure relates generally to a hardware accelerator extension to transfer learning in order to extending/finishing training to the edge. The various systems, methods, and apparatuses described herein may be applied to, for example, developing highly customized neural networks at internet of things (IoT) and other edge devices.

BACKGROUND

[2] Classification is a machine learning task that predicts a class for a given set of data points. In recent years, there has been a significant accuracy improvement in

classification tasks such as image and speech recognition. New approaches use artificial deep neural networks and achieve super human accuracies on large public datasets such as ImageNet. This development is fueled by the availability of extremely large datasets and datacenter/cloud-based computing power for training. The learning on mostly consumer centric domains (e.g., based on freely shared photos) is now being transferred to other domains, such as the industrial and healthcare domains. Applications include deep reinforcement learning based optimization of large power generating gas turbine emissions, vision based robotic control and grasping of non-standard objects, visual quality control of manufactured goods, etc.

[3] A challenge that industrial applications of deep neural networks face is that there typically is not as much data available as in other domains in which these algorithms have been successfully deployed. For example, one cannot record thousands of representations of one part with a particular defect to detect it in the future automatically. In addition, many industrial applications require high data security and not all vendors are comfortable to share or even store this data in the cloud. Moreover, there are constraints regarding latencies, common real-time requirements, bandwidth and the requirement to run the facility offline without data connection.

[4] These considerations motivate a current trend from cloud applications to edge computing applications. The term“edge computing” refers to computing that is performed on device nodes that are located close to, or on, a machine as opposed to a centralized cloud computing environment. While the edge approach addresses some of the challenges discussed above, edge computing also brings the problem of limiting available computing performance. Complex deep neural networks cannot be run in real-time applications on low power CPUs, resulting in a trend towards AI specific hardware accelerators. However, data centers used to train large deep neural network in the cloud still have significantly more performance than edge AI accelerators that have to run on a few watts to allow passive cooling and thus high reliability without moving parts. For example, AlphaGO Lee was trained on 48TPUs in a datacenter for several months to achieve maximum performance as described in Silver, D. et al Mastering the game of Go with deep neural networks and tree search. Nature 529, 484—489 (2016).

[5] Flexible solutions that allow training on the edge are CPU or GPU based.

Unfortunately, these solutions are either computationally limited in comparison to AI specific hardware accelerators or require significantly more power, which limits their use in the industrial edge environment and other operating environments where edge devices are used.

SUMMARY

[6] Embodiments of the present invention address and overcome one or more of the above shortcomings and drawbacks, by providing methods, systems, and apparatuses related to a hardware accelerator extension to transfer learning by extending/fmishing training to the edge.

[7] According to some embodiments, a computer-implemented method for training a neural network on a hardware accelerator of an edge device includes dividing a trained neural network into a domain independent portion and a domain dependent portion. The domain independent portion of the neural network is deployed onto a dedicated neural network processing unit of the hardware accelerator of the edge device, and the domain dependent portion of the neural network is deployed onto one or more additional processors of the hardware accelerator of the edge device. The domain dependent portion on the additional processors of the hardware accelerator is retrained using data collected at the edge device.

[8] According to other embodiments, an edge device is connected to a remote computer system over a network. The edge device has a hardware accelerator that comprises one or more communication buses, a dedicated neural network processing unit, and one or more processors. The dedicated neural network processing unit executes a domain independent portion of a trained neural network. The processors execute a domain dependent portion of the trained neural network connected to the domain independent portion over the communication buses. The domain dependent portion is re-trained by the processors using data collected at the edge device.

[9] According to other embodiments, a system for training a neural network comprises a computer and an edge device. The computer divides a neural network into a domain independent portion and a domain dependent portion, wherein at least the domain independent portion is pre-trained. The edge device receives the domain independent portion and domain dependent portion from the computer via a network. The edge device deploys the domain independent portion of the neural network onto a dedicated neural network processing unit of the hardware accelerator of the edge device. The domain dependent portion of the neural network is deployed onto one or more additional processors of the hardware accelerator of the edge device. The edge device trains the domain dependent portion on the additional processors of the hardware accelerator using data collected at the edge device. Following the deployments and the retraining, the edge device uses the domain independent portion and the domain dependent portion to perform one or more tasks.

[10] As a variation to the system set forth above, in some embodiments, the edge device only receives the domain independent portion from the computer via a network. This domain independent portion of the neural network is deployed onto a dedicated neural network processing unit of the hardware accelerator of the edge device. Then the edge device generates a new version of the domain dependent portion of the neural network onto one or more additional processors of the hardware accelerator of the edge device. The edge device trains the new version of the domain dependent portion on the additional processors of the hardware accelerator using data collected at the edge device. Following the deployments and the training, the edge device uses the new version of the domain independent portion and the domain dependent portion to perform one or more tasks.

[11] Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments that proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[12] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:

[13] FIG. 1 A shows an example edge device architecture configured in accordance with some embodiments of the present invention;

[14] FIG. IB shows an alternative edge device architecture configured in accordance with some embodiments of the present invention; and

[15] FIG. 2 illustrates a computer-implemented method for training a neural network on a hardware accelerator of an edge device, according to some embodiments.

DETAILED DESCRIPTION

[16] The following disclosure describes the present invention according to several embodiments directed at methods, systems, and apparatuses related to hardware accelerator extension to transfer learning by extending/fmishing training to the edge.

[17] In typical neural network deployments, maximally large data sets are recorded at the edge, and uploaded to the cloud to retrain the neural network for optimal performance.

The network is then downloaded and applied at the edge. This process is repeated whenever a new class needs to be added to the network. The reason for this approach is that neural network hardware accelerators are designed to maximize their inference performance. Fast, iterative changing of neural network weights, required for training, are not considered in the architecture, as the hardware is computationally not powerful enough for full-blown training and will not be exposed to millions of examples from large databases at the edge. An alternative is to collect the data and do the training at a local server for data privacy concerns. Thereafter, the updated network is uploaded back on the edge hardware for future inferences. Neither of these approaches allows training with minimal data, inside the edge device (without interfacing other systems), and while allowing continuous operation of the device inference (i.e., without stopping the factory processing line). [18] With the techniques described herein a neural network is divided into a fixed and a flexible part. The fixed part is pre-trained, for example, with millions of examples in the cloud and used as feature extraction layers. The fixed part can then be deployed on the inference-optimized parts of the hardware accelerators. The flexible part of the neural network is used to train new classes and adapt current classifiers on the fly. This flexible part can then be deployed in the flexible parts of the hardware accelerator such as Streaming Hybrid Architecture Vector Engine (SHAVE) processors, GPU, or CPUs. By performing neural network training/adaptation at the edge, on low power hardware we can enable more flexible applications including continuous learning based approaches. This will make the update and refinement process of neural networks significantly more efficient than for conventional solutions.

[19] FIG. 1 A shows an example edge device architecture configured in accordance with some embodiments of the present invention. This Edge Device 105 is connected to a Remote Computer 110 over a Network 115. The main component of the Edge Device 105 in this example is Artificial Intelligence (AI) Hardware Accelerator 120. The Hardware Accelerator 120 includes a Dedicated Neural Network Processing Unit 125 comprising one or more processing units optimized for executing neural networks at high speed and low power. The Hardware Accelerator 120 further includes one or more Additional Processors 130.

These Additional Processors 130 may include, for example, SHAVE processors, Graphical Processing Units (GPUs), or even central processing units (CPUs). A single type of processor may be employed, or combinations of different types of processors (e.g., SHAVE processors and CPUs) may be used with the present invention. A Communication Bus 135 connects the Dedicated Neural Network Processing Unit 125 and the Additional Processors 130. Non-limiting examples of hardware accelerators that may be used with the present invention include Intel's Myriad™ X VPU. In the Myriad™ X VPU, the Dedicated Neural Network Processing Unit 125 is embodied in the neural compute engine, while the Additional Processors 130 are embodied by Myriad’s SHAVE cores.

[20] Continuing with reference to FIG. 1, at the Remote Computer 110 a Trained Neural Network 150 is divided into a Domain Independent Portion 140 and a Domain Dependent Portion 145. This division may be made, for example, by the original designer of the Trained Neural Network 150 or the Trained Neural Network 150 can be analyzed (e.g., using visualization tools known in the art) to ascertain the functional characteristics different portions of the Trained Neural Network 150.

[21] The Domain Independent Portion 140 is the portion of the Trained Neural Network 150 which performs functions that can be used across multiple domains (e.g., feature extraction), while the Domain Dependent Portion 145 performs functions that are more specific to a particular domain or application (e.g., classification). The Hardware Accelerator 120 executes the Domain Independent Portion 140 and a Domain Dependent Portion 145 as a complete neural network using the Communication Bus 135 as necessary to transfer data between the two Portions 140, 145. Techniques for transferring data between different hardware accelerator components are generally known in the art and, in general, any such technique may be employed to facilitate the data transfer described above.

[22] Each Portion 140, 145 comprises one or more layers. As is generally understood in the art, the term“layers” refers to a collection of nodes operating together at a specific depth within a neural network and different network architectures such as convolutional layers or polling layers. A neural network can be divided into different layers based on its underlying architecture. Research on deep neural networks has shown that the front layers in a network resemble Gabor filters or color blobs. Similar functions have been used in classical image processing approached for preprocessing and feature extraction. These features are then combined in the later layers of the network to distinguish between classes or perform other functions. The domain specificity of different layers of a neural network is analyzed in: J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages 3320- 3328. Curran Associates, Inc., Dec. 2014. It has been shown that by using this approach, it is possible to generate significantly higher recognition accuracies with small training sets than when using the limited data for training. For example, Matlab published a comparison where they trained a network with 5000 images per class from scratch and received 75% accuracy and then used a transfer learning approach where they only retrained the last 3 layers of AlexNet with 500 images per class and achieved 85% accuracy. See“Deep Learning with MATLAB: Transfer Learning with Neural Networks in MATLAB,” available at https:.4''www.mathworks.com/'videos/'transfer-iearmng~with-n eural-networks-in-matlab-

14920071 _. 75092 J vim] (last accessed January 9, 2019). [23] For example, a convolutional neural network may be designed with the following layers: an input layer; a series of convolutional and pooling layers, and one or more fully connected layers. The input layer receives the raw pixel values of the image. The convolutional layer comprises a set of filters (or kernels) that activate when a specific feature is detected in the input data. For example, the convolutional layers may perform edge detection and/or curve detection. The pooling layer reduces the spatial size of the outputs of the convolutional layer. In some CNNs, a pooling layer is inserted periodically between successive convolutional layers. After the series of convolutional and max pooling layers, the high-level reasoning of the neural network is performed via one or more fully connected layers. For example, the fully connected layer may perform image processing tasks such as object detection, object segmentation, image classification, or localization. Techniques for implementing these image processing techniques are generally known in the art, and thus the techniques are not described in detail herein.

[24] With the transfer learning technique described herein, the CNN can be trained with a large dataset (e.g., the ImageNet dataset). The input layer and the series of

convolutional and pooling layers are used as a domain independent feature extractor. The fully connected layers of the trained CNN are replaced with domain dependent fully connected layers on the Edge Device 105. Effectively, this can be understood as retaining the complete CNN, where only the weights of the fully connected layers are allowed to change when performing the gradient descent/optimization procedure.

[25] In some embodiments, the division between the Domain Independent Portion 140 and a Domain Dependent Portion 145 of the Trained Neural Network 150 may be made on the basis of the hardware capabilities of the Hardware Accelerator 120. For example, as noted above, the Domain Dependent Portion 145 can just comprise the fully connected layers of the Trained Neural Network 150 that perform classification (or other tasks) on extracted feature data. Depending on the task involved, it may be desirable to also include some of the other layers (e.g., one or more convolutional and pooling layers) in the Domain Dependent Portion 145 to allow these layers to be retrained based on data that is local to the edge device. In order to assess how to divide the Trained Neural Network 150, the Remote Computer 110 may query the Edge Device 105 for its hardware characteristics and then divide the Trained Neural Network 150 accordingly. These hardware characteristics may include generally any information that would be indicative of the processing capabilities of the Edge Device 105. This information may include, without limitation, processing speed, available RAM, and bus speed of the Hardware Accelerator 120.

[26] FIG. IB shows a variation on the example of FIG. 1 A. Here, rather than dividing the Trained Neural Network 150 on the Remote Computer 110 and send the two Portions to 140, 145 to the Edge Device, the Remote Computer 110 simply sends the complete Trained Neural Network 150. Then, the Edge Device 105 divides the Trained Neural Network 150 into the Domain Independent Portion 140 and the Domain Dependent Portion 145. In some embodiments, the Trained Neural Network 150 may be delivered to the Edge Device 105 with instructions specifying the hardware characteristics required to include certain layers in the Domain Dependent Portion 145. In this way, the Edge Device 105 can perform the division locally based on an evaluation of its hardware characteristics.

[27] FIG. 2 illustrates a computer-implemented method 200 for training a neural network on a hardware accelerator of an edge device, according to some embodiments. This method 200 may be used to allow customization of a pre-trained neural network at an edge device. For example, in one embodiment, the neural network is trained using a first set of data that is not specific to a factory operating environment. In general this data can take any form and will vary based on the application. Non-limiting examples of the training data include generic image and/or audio data. A domain independent version is created and can be retrained at the edge device using a second set of data that is specific to the factory operating environment. It should be noted that“domain independent” does not necessarily imply that the original neural network is entirely generic. For example, a neural network can be trained based on library of sensor measurements collected from a group of factories. A portion of this neural network can then be re-trained at an edge device of a particular factory to refine the neural network with locally collected sensor measurements.

[28] Starting at step 205, a trained neural network is divided into a domain independent portion and a domain dependent portion. In some embodiments, the trained neural network is divided at a remote computer system connected to the edge device over a network. For example, in some embodiments, the neural network portions are transferred as one or more Extensible Markup Language (XML) files describing the architecture of the network and weights for each node. In other embodiments, the neural network is transferred as one or more executable applications compiled for the architecture of the edge computer. In other embodiments, the trained neural network is divided at the edge device using the additional processors. The number of layers of the trained neural network included in the domain dependent portion may be selected, for example, based on hardware characteristics of the edge device.

[29] Continuing with reference to FIG. 2, at step 210, the domain independent portion of the neural network is deployed onto a dedicated neural network processing unit of the hardware accelerator of the edge device. The hardware accelerator may include in silicon optimized neural network specific operations such as convolutions or matrix multiplication or extremely fast memory-compute connectivity optimized for common neural network design execution. The domain dependent portion of the neural network is deployed at step 215 onto one or more additional processors of the hardware accelerator of the edge device. The method of deployment can vary depending on how the neural network was developed and the architecture of the edge device. However, in general any deployment technique known in the art may be used.

[30] The domain dependent portion is re-trained at step 220 on the additional processors of the hardware accelerator using data collected at the edge device. For example, in one embodiment, the edge device is a component with in a factory setting that acts as a controller. The edge device receives measurements, makes predictions based on those measurements, and then generates one or more outputs that control factory devices. In this case, the measurement data gathered during operations can be used to re-train the domain dependent portion of the neural network. Note that this method of re-training not only allows device-level customization of the neural network, but also training with higher data fidelity than may be available at remote computing system. For example, sensor measurements may be gathered at 0.01 second intervals and 10 seconds worth of data may be required to make a prediction. If the data needs to be transferred to a remote computer, storage and bandwidth requirements may require the data to be sampled on a per-second basis, thus requiring the training to be performed based on 10 measurements. However, at the edge device, every measurement can be used for training, thus potentially increasing the accuracy of overall model. Finally, once the neural network is re-trained, the edge device uses it to perform one or more tasks at step 225. For example, to continue with the examples set forth above, the edge device may classify image data and generate an output that is used to control a factory device. [31] In some embodiments, the remote computer system is not transferred at the domain dependent portion to the edge device; instead, the domain dependent portion is fully generated at the edge based on edge data. However, providing a pre-trained domain dependent portion with the same class outputs should help the conversion speed of the algorithms. This assumes that the difference between the domains is not very large and one can find a convex path to the optimum classifier in the new domain. The training can also try this approach and if no sufficient classifier can be adapted; then one can reinitiate the fully connected layers in the domain dependent network randomly and start training this part from scratch.

[32] To further illustrate the applicability of the technique described in FIG. 2, imagine the use case where the output of an industrial process needs to be classified in good products and faulty products based on a camera image. It would be useful to defme/specify the detected fault so that the next steps like a rework can be automated. As certain faults are very rare, there is no data available on these fault classes during the training phase. Using the approach applied in conventional systems one would need to stop production to ensure that no faults are missed, add the new data to the training and upload the new model before activating the production line again. This is a very costly and sometimes infeasible approach as there can be long ramp-up times of the machines until they produce at desired quality levels. This can be solved by using a Progressive Neural Network learning approach illustrated in: A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.04671, 2016.

[33] Similar to transfer learning, this approach borrows from previously found classifiers to minimize necessary parameter optimizers and take advantage of the larger dataset that was available to train the feature extraction parts of the network. The advantage of this approach is that previously learned classifiers are frozen and not affected by the training of a new classifier. This enables that the classification can continue during training without loss of accuracy. Again, the network can be split into a fixed, high performance optimized part and a flexible adaptive part. The training can be outsourced in the flexible layers of the hardware accelerator to enable operation in current edge hardware. [34] The training technique described above enables a large number of new

applications through training at the edge. As noted above, one advantage of the disclosed techniques is that data does not need to be recorded and moved to a powerful machine for training resulting in a more efficient process. Given this minimal effort, the user will be able to adapt the training more frequently for finer adjustments. The user can give corrective feedback during operation for continuous learning at the edge. The performance of the classifier that uses large pre-trained networks from millions of examples as a basis and adapts to the use case by a small set of training data is significantly higher than a network that is trained from the few examples from scratch. Re-training and adding of classes can be done during online operation. This may result in reducing costly shutdowns of monitored production lines. In addition, training components on the edge generates a logical separation layer, enabling users to isolate parts of their processes from pre-training which may happen externally, such as on a cloud platform.

[35] The embodiments of the present disclosure may be implemented with any combination of hardware and software. For example, aside from hardware accelerator architectures presented above with respect to FIGS. 1 A and IB, one can use the hardware accelerator as a co-processor, computing all flexible (i.e., domain dependent) parts in the main CPU and outsourcing the fixed (i.e., domain independent) part to the accelerator. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, computer- readable, non-transitory media. The media may have embodied therein computer readable program code for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[36] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

[37] An executable application, as used herein, comprises code or machine readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters.

[38] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[39] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and/or combinations thereof. No claim element herein is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase“means for.”