Title:
BERT MODEL FINE-TUNING METHOD AND APPARATUS BASED ON CONVOLUTIONAL NEURAL NETWORK
Document Type and Number:
WIPO Patent Application WO/2022/116441
Kind Code:
A1
Abstract:
Disclosed are a BERT model fine-tuning method and apparatus based on a convolutional neural network. The method comprises: constructing a first BERT model, a hidden layer of which is a transformer block network, and a second BERT model, a hidden layer of which is a convolutional neural network, wherein the number of layers of the hidden layer of the first BERT model is equal to the number of layers of the hidden layer of the second BERT model; training the first BERT model according to a first text set, and performing knowledge distillation on the second BERT model on the basis of the trained first BERT model, so as to obtain a knowledge distillation loss and a distribution loss of the second BERT model; inputting a second text set into the second BERT model, so as to obtain a cross entropy loss of the second BERT model; and updating a network parameter of the second BERT model according to the knowledge distillation loss and the cross entropy loss. The present application is based on neural network technology. By means of the method, a BERT model, a hidden layer of which is a convolutional neural network, is fine-tuned, and the number of parameters in the fine-tuned BERT model is also significantly reduced, thereby greatly improving the calculation speed of the model, and ensuring the accuracy of text classification of the model.
Inventors:
CHEN HAO (CN)
QIAO YIXUAN (CN)
GAO PENG (CN)
QIAO YIXUAN (CN)
GAO PENG (CN)
Application Number:
PCT/CN2021/083933
Publication Date:
June 09, 2022
Filing Date:
March 30, 2021
Export Citation:
Assignee:
PING AN TECH SHENZHEN CO LTD (CN)
International Classes:
G06N3/08; G06N3/04
Foreign References:
| CN112529153A | 2021-03-19 | |||
| CN111611377A | 2020-09-01 | |||
| US20180268292A1 | 2018-09-20 | |||
| CN111291836A | 2020-06-16 | |||
| CN112016674A | 2020-12-01 |
Other References:
HONG LI, CHI XING, HUA YAN, LU MA, QUN LIAO, AN ZHI, LIANG LIU, CHAO LI, SEN YUN, CHAO YONG, ETC: "Exploration and Practice of Named Entity Recognition (NER) Technology in Meituan Search", EXPLORATION AND PRACTICE OF NER TECHNOLOGY IN MEITUAN SEARCH- MEITUAN TECHNICAL TEAM, 23 July 2020 (2020-07-23), pages 1 - 16, XP055936148, Retrieved from the Internet [retrieved on 20220628]
Attorney, Agent or Firm:
SHENZHEN TALENT PATENT SERVICE (CN)
Download PDF: