Title:
APPARATUS AND METHOD FOR BATCH REBALANCE IN DISTRIBUTED DATA PARALLEL DNN TRAINING
Document Type and Number:
WIPO Patent Application WO/2023/065076
Kind Code:
A1
Abstract:
Provided herein are apparatus and methods for batch rebalance in distributed data parallel DNN training. An apparatus includes interface circuitry; and processor circuitry coupled with the interface circuitry, wherein the processor circuitry is to: obtain sorted samples of a mini batch via the interface circuitry, wherein the sorted samples are in an ascend or descend order based on a volume of each of the samples; and assign the sorted samples to each of a plurality of local batches one by one in an order from a first local batch to a last local batch of the plurality of local batches and then from the last local batch to the first local batch until all of the sorted samples are assigned. Other embodiments may also be disclosed and claimed.
Inventors:
MA GUOKAI (CN)
GONG JIONG (CN)
LIU HONGZHEN (CN)
GONG JIONG (CN)
LIU HONGZHEN (CN)
Application Number:
PCT/CN2021/124478
Publication Date:
April 27, 2023
Filing Date:
October 18, 2021
Export Citation:
Assignee:
INTEL CORP (US)
MA GUOKAI (CN)
GONG JIONG (CN)
LIU HONGZHEN (CN)
MA GUOKAI (CN)
GONG JIONG (CN)
LIU HONGZHEN (CN)
International Classes:
G06F9/50; G06N20/00
Foreign References:
| US20180356803A1 | 2018-12-13 | |||
| CN112732444A | 2021-04-30 | |||
| US20170039485A1 | 2017-02-09 | |||
| US20200089541A1 | 2020-03-19 | |||
| US20210133505A1 | 2021-05-06 | |||
| US20190332422A1 | 2019-10-31 |
Attorney, Agent or Firm:
BEIJING EAST IP LTD. (CN)
Download PDF: