Title:
MIXUP DATA AUGMENTATION FOR KNOWLEDGE DISTILLATION FRAMEWORK
Document Type and Number:
WIPO Patent Application WO/2022/121515
Kind Code:
A1
Abstract:
A method of training a student neural network is provided. The method includes feeding a data set including a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The method further includes combining the two corresponding input vectors to form a synthesized data vector, and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the synthesized data vector into the student neural network, using the masked soft label vector to determine an error for modifying weights of the student neural network, and modifying the weights of the student neural network.
Inventors:
FUKUDA TAKASHI (JP)
Application Number:
PCT/CN2021/124782
Publication Date:
June 16, 2022
Filing Date:
October 19, 2021
Export Citation:
Assignee:
IBM (US)
IBM CHINA CO LTD (CN)
IBM CHINA CO LTD (CN)
International Classes:
G10L17/02; G06N3/08
Foreign References:
| CN111640425A | 2020-09-08 | |||
| CN111341341A | 2020-06-26 | |||
| CN109637546A | 2019-04-16 | |||
| JP2019159654A | 2019-09-19 | |||
| US20190130896A1 | 2019-05-02 |
Other References:
HONGYI ZHANG, CISSE MOUSTAPHA, DAUPHIN YANN N, LOPEZ-PAZ DAVID: "mixup: Beyond Empirical Risk Minimization", 27 April 2018 (2018-04-27), XP055716970, Retrieved from the Internet [retrieved on 20200721]
WANG DONGDONG; LI YANDONG; WANG LIQIANG; GONG BOQING: "Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation From a Blackbox Model", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 13 June 2020 (2020-06-13), pages 1495 - 1504, XP033805184, DOI: 10.1109/CVPR42600.2020.00157
WANG DONGDONG; LI YANDONG; WANG LIQIANG; GONG BOQING: "Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation From a Blackbox Model", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 13 June 2020 (2020-06-13), pages 1495 - 1504, XP033805184, DOI: 10.1109/CVPR42600.2020.00157
Attorney, Agent or Firm:
KING & WOOD MALLESONS (CN)
Download PDF:
Previous Patent: STEAMER ASSEMBLY AND COOKING APPLIANCE
Next Patent: BIOMARKER SELECTION AND MODELING FOR TARGETED MICROBIOMIC TESTING
Next Patent: BIOMARKER SELECTION AND MODELING FOR TARGETED MICROBIOMIC TESTING