Title:
DATA PROCESSING METHOD IN DISTRIBUTED SYSTEM, AND RELATED SYSTEM
Document Type and Number:
WIPO Patent Application WO/2023/040348
Kind Code:
A1
Abstract:
Provided in the present application is a data processing method in a distributed system. The method comprises: a manager splitting, into a map task and a reduce task, a data processing request to be processed; a first worker executing the map task, and writing, into a shared storage area, intermediate data obtained by executing the map task, wherein the intermediate data is divided into a plurality of partitions according to the number of second workers, and each partition belongs to one second worker; an aggregator storing, in a centralized manner, the intermediate data in the shared storage area that belongs to the same partition; and the second worker acquiring the intermediate data of the corresponding partition, so as to execute the reduce task. Therefore, the efficiency of pulling intermediate data is improved, the transmission size is increased, the number of instances of transmission is reduced, and the network performance is fully exerted.
Inventors:
XU HUA (CN)
BAO XIAOMING (CN)
LIU CHUN (CN)
ZHOU MINJUN (CN)
CHEN JIAHUAI (CN)
BAO XIAOMING (CN)
LIU CHUN (CN)
ZHOU MINJUN (CN)
CHEN JIAHUAI (CN)
Application Number:
PCT/CN2022/095934
Publication Date:
March 23, 2023
Filing Date:
May 30, 2022
Export Citation:
Assignee:
HUAWEI TECH CO LTD (CN)
International Classes:
G06F3/06
Domestic Patent References:
| WO2017113278A1 | 2017-07-06 |
Foreign References:
| CN114237510A | 2022-03-25 | |||
| CN103327128A | 2013-09-25 | |||
| CN111857538A | 2020-10-30 | |||
| JP2006091966A | 2006-04-06 | |||
| US8510538B1 | 2013-08-13 | |||
| CN202111076633A | 2021-09-14 |
Attorney, Agent or Firm:
SHENPAT INTELLECTUAL PROPERTY AGENCY (CN)
Download PDF: