Title:
DATA PROCESSING METHOD AND SYSTEM
Document Type and Number:
WIPO Patent Application WO/2021/258831
Kind Code:
A1
Abstract:
A data processing method and system. The data processing method is applied to the data processing system, and the data processing system comprises a computing node and a storage node. The data processing method comprises the following steps: a computing node acquiring metadata of a file to be read (S810); then, according to the starting position of each slice in said file, calling multiple threads, and concurrently reading data of each slice (S820); and finally, according to the order of the starting positions of the slices in said file, storing the data of each shard slice in a memory space (S830). By means of the method, when a computing node reads a file to be read, a memory space capable of accommodating said file can be applied for in one step according to metadata of said file, and said file can be concurrently read to improve the efficiency of data reading, thereby improving the efficiency of processing the whole AI or big data task.
Inventors:
ZHU QI (CN)
CUI BAOLONG (CN)
WANG JUNJIE (CN)
CUI BAOLONG (CN)
WANG JUNJIE (CN)
Application Number:
PCT/CN2021/088588
Publication Date:
December 30, 2021
Filing Date:
April 21, 2021
Export Citation:
Assignee:
HUAWEI TECH CO LTD (CN)
International Classes:
G06F9/50
Foreign References:
| CN109710572A | 2019-05-03 | |||
| CN106202152A | 2016-12-07 | |||
| US20180137172A1 | 2018-05-17 |
Download PDF: