Title:
METHOD FOR TRAINING NEURAL NETWORK, AND RELATED DEVICE
Document Type and Number:
WIPO Patent Application WO/2021/238734
Kind Code:
A1
Abstract:
A method for training a neural network, and a related device. In the method, after completing forward calculation of one piece of micro-batch data, an accelerator immediately performs reverse calculation on a forward calculation result of the micro-batch data. When the accelerator starts the reverse calculation, release of feature values, which are generated during the forward calculation, of the micro-batch data can be started until the reverse calculation of the micro-batch data is completed, and at this moment, the feature values, which are generated during the forward calculation, of the micro-batch data are completely released. Thereafter, the accelerator can perform forward calculation and reverse calculation on the next piece of micro-batch data until reverse calculation of all pieces of micro-batch data is completed. Therefore, during the whole calculation process, the accelerator does not need to store feature values, which are generated during the forward calculation, of all pieces of micro-batch data, and as a result, the peak value of the amount of memory occupation of the accelerator can be kept at a lower value, and the training efficiency of a neural network can be improved.
More Like This:
| WO/2017/083744 | NEURAL RANDOM ACCESS MACHINE |
| WO/2022/231823 | AUTOMATED FAULT DETECTION IN MICROFABRICATION |
| WO/2021/209789 | MODIFYING STRUCTURE OF ARTIFICIAL NEURAL NETWORKS BY COLLOCATING PARAMETERS |
Inventors:
CHEN XIANPING (CN)
QIN YONG (CN)
QIN YONG (CN)
Application Number:
PCT/CN2021/094579
Publication Date:
December 02, 2021
Filing Date:
May 19, 2021
Export Citation:
Assignee:
HUAWEI TECH CO LTD (CN)
International Classes:
G06N3/04
Foreign References:
| CN107506695A | 2017-12-22 | |||
| CN110795228A | 2020-02-14 |
Other References:
YANPING HUANG, CHENG YOULONG, BAPNA ANKUR, FIRAT ORHAN, CHEN MIA XU, CHEN DEHAO, LEE HYOUKJOONG, NGIAM JIQUAN, LE QUOC V, WU YONGH: "GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism", CORR (ARXIV), CORNELL UNIVERSITY LIBRARY, vol. 1811.06965, no. v5, pages 1 - 11, XP055730504
XU WENCHAO; PANG YUXIN; YANG YANQIN; LIU YANBO: "Human Activity Recognition Based On Convolutional Neural Network", 2018 24TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), IEEE, 20 August 2018 (2018-08-20), pages 165 - 170, XP033454144, DOI: 10.1109/ICPR.2018.8545435
XU WENCHAO; PANG YUXIN; YANG YANQIN; LIU YANBO: "Human Activity Recognition Based On Convolutional Neural Network", 2018 24TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), IEEE, 20 August 2018 (2018-08-20), pages 165 - 170, XP033454144, DOI: 10.1109/ICPR.2018.8545435
Attorney, Agent or Firm:
SHENPAT INTELLECTUAL PROPERTY AGENCY (CN)
Download PDF:
Previous Patent: DISPLAY DEVICE AND IMAGE RECOGNITION RESULT DISPLAY METHOD
Next Patent: VIDEO TRANSMISSION METHOD AND DEVICE
Next Patent: VIDEO TRANSMISSION METHOD AND DEVICE