Title:
DISTRIBUTED PROCESSING SYSTEM AND DISTRIBUTED PROCESSING METHOD
Document Type and Number:
WIPO Patent Application WO/2020/095678
Kind Code:
A1
Abstract:
The present invention performs distributed processing that is effective when applied to deep learning. This distributed processing system comprises a plurality of low-order aggregation networks 1[m] (m = 1 to M) and a high-order aggregation network 2. Each of the low-order aggregation networks 1[m] includes a plurality of distributed processing nodes 3 arranged in ring form. Each of the distributed processing nodes 3 generates distributed data per neural network weight of the node. Each of the low-order aggregation networks 1[m] aggregates, for each low-order aggregation network, the distributed data generated by each of the distributed processing nodes 3. The high-order aggregation network 2 generates aggregate data in which the aggregation results of each of the low-order aggregation networks 1[m] are further aggregated and distributes the generated data to each of the low-order aggregation networks 1[m]. Each of the low-order aggregation networks 1[m] distributes the distributed aggregate data to each of the distributed processing nodes 3 that belongs to the same low-order aggregation network. Each of the distributed processing nodes 3 updates the neural network weight on the basis of the distributed aggregate data.
Inventors:
KAWAI KENJI (JP)
KATO JUNICHI (JP)
NGO HUYCU (JP)
ARIKAWA YUKI (JP)
ITO TSUYOSHI (JP)
TANAKA KENJI (JP)
SAKAMOTO TAKESHI (JP)
KATO JUNICHI (JP)
NGO HUYCU (JP)
ARIKAWA YUKI (JP)
ITO TSUYOSHI (JP)
TANAKA KENJI (JP)
SAKAMOTO TAKESHI (JP)
Application Number:
PCT/JP2019/041482
Publication Date:
May 14, 2020
Filing Date:
October 23, 2019
Export Citation:
Assignee:
NIPPON TELEGRAPH & TELEPHONE (JP)
International Classes:
G06N3/08; G06F9/50
Other References:
JIA, XIANYAN ET AL.: "Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes", ARXIV, 30 July 2018 (2018-07-30), XP081252221, Retrieved from the Internet [retrieved on 20191212]
SERGEEV, ALEXANDER ET AL.: "Horovod: fast and easy distributed deep learning in TensorFlow", ARXIV, 21 February 2018 (2018-02-21), XP081215801, Retrieved from the Internet [retrieved on 20191212]
SERGEEV, ALEXANDER ET AL.: "Horovod: fast and easy distributed deep learning in TensorFlow", ARXIV, 21 February 2018 (2018-02-21), XP081215801, Retrieved from the Internet
Attorney, Agent or Firm:
YAMAKAWA, Shigeki et al. (JP)
Download PDF:
Previous Patent: ACCESS CONTROL METHOD, ACCESS CONTROL DEVICE, AND DATA PROCESSING DEVICE
Next Patent: PROBE PIN AND INSPECTION JIG
Next Patent: PROBE PIN AND INSPECTION JIG