Title:
SEQUENCE PROCESSING METHOD AND APPARATUS
Document Type and Number:
WIPO Patent Application WO/2021/238289
Kind Code:
A1
Abstract:
A sequence processing method and apparatus, which relate to the field of artificial intelligence, and in particular to the field of sequence data processing. The method comprises: receiving an input sequence (S410); performing self-attention computation on a first element in the input sequence by using elements included in M windows, so as to obtain a representation of the first element, wherein each window includes one element or a plurality of successive elements in the input sequence, different windows are spaced by at least one element, at least one window in the M windows does not include the first element, and M is an integer greater than or equal to 1 (S420); and on the basis of the representation of the first element, obtaining an output sequence corresponding to the input sequence (S430). With regard to elements in a sequence, using elements in one or more windows instead of all the elements in the sequence to perform self-attention computation can reduce the amount of self-attention computation, wherein at least one window can skip a first element, and the position of the window is not fixed, such that the limitation on a self-attention dependence range can be reduced.
More Like This:
Inventors:
HUANG WENYONG (CN)
YEUNG YU TING (CN)
CHEN XIAO (CN)
YEUNG YU TING (CN)
CHEN XIAO (CN)
Application Number:
PCT/CN2021/073868
Publication Date:
December 02, 2021
Filing Date:
January 27, 2021
Export Citation:
Assignee:
HUAWEI TECH CO LTD (CN)
International Classes:
G06F40/289
Domestic Patent References:
| WO2019240900A1 | 2019-12-19 |
Foreign References:
| CN111783446A | 2020-10-16 | |||
| CN109919188A | 2019-06-21 | |||
| CN110162625A | 2019-08-23 | |||
| CN110096711A | 2019-08-06 | |||
| CN202010454695A | 2020-05-26 |
Attorney, Agent or Firm:
LONGSUN LEAD IP LTD. (CN)
Download PDF: