To provide a cargo handling system capable of starting cargo handling operation at cargo handling scheduled time irrespective of a cargo handling position.

The cargo handling system 1 includes: a plurality of automatic guided vehicle 4 for conveying a container; a gantry crane 5 for handling a container between the automatic guided vehicle 4 and a container ship; and a controller 6. The controller 6 includes: a cargo handling plan setting section 13 which sets target cargo handling time of the cargo handling operation, from information on the container loading position in the container ship, and which calculates transfer start time from the automatic guided vehicle 4 to the gantry crane 5 based on the target cargo handling time; and a cargo handling plan correction section 14 which obtains an actual cargo handling time that is actually taken in cargo handling, and which recalculates the transfer start scheduled time in cargo handling operation to be performed after that, according to a difference between the target cargo handling time and the actual cargo handling time.