Title:
VIDEO QUESTION-ANSWER METHOD, DEVICE AND SYSTEM, AND STORAGE MEDIUM
Document Type and Number:
WIPO Patent Application WO/2024/046038
Kind Code:
A1
Abstract:
A video question-answer method, device and system, and a storage medium. The method comprises: extracting a video feature vector for an inputted video, and extracting a text feature vector for a question text and a candidate answer text; splicing the video feature vector and the text feature vector to obtain a spliced feature vector, inputting the spliced feature vector into a first pre-training model, and the first pre-training model learning cross-modal information between the video feature vector and the text feature vector by means of a self-attention mechanism to obtain a second spliced feature vector; dividing the second spliced feature vector into a second video feature vector and a second text feature vector, inputting the second video feature vector and the second text feature vector into a modal fusion model, the modal fusion model processing the second video feature vector and the second text feature vector by means of a mutual attention mechanism to obtain a video expression and a text expression, and respectively pooling and fusing the video expression and the text expression to obtain a fused feature vector; predicting a correct candidate answer according to the fusion feature vector.
Inventors:
WANG BINGQIAN (CN)
Application Number:
PCT/CN2023/111455
Publication Date:
March 07, 2024
Filing Date:
August 07, 2023
Export Citation:
Assignee:
BOE TECHNOLOGY GROUP CO LTD (CN)
International Classes:
G06F16/332
Foreign References:
| CN115391511A | 2022-11-25 | |||
| CN113807222A | 2021-12-17 | |||
| CN114925703A | 2022-08-19 | |||
| US20220164548A1 | 2022-05-26 |
Other References:
MA ZHIYANG, ZHENG WENFENG, CHEN XIAOBING, YIN LIRONG: "Joint embedding VQA model based on dynamic word vector", PEERJ COMPUTER SCIENCE, vol. 7, 3 March 2021 (2021-03-03), pages e353, XP093142989, ISSN: 2376-5992, DOI: 10.7717/peerj-cs.353
Attorney, Agent or Firm:
AFD CHINA INTELLECTUAL PROPERTY LAW OFFICE (CN)
Download PDF: