To reduce the time needed for the whole image processing including restoration processing on EDoF images which have their depth of field extended and other correction processing.

A RAW image whose depth of field has been extended via a lens part which extends the depth of field by modulating phases and an image pickup element is acquired, and then the position (x', y') of a pre-processing image (position taking distortion correction into account) is obtained based on the position (x, y) of a post-processing pixel of interest (pixel to be processed) (step S04). Subsequently, a restoration filter corresponding to the real space of the position (x', y') is created based on a restoration filter stored in storage for each divided region of a screen (step S03). The restoration filter thus created and an N×N matrix image extracted from the RAW image based on the position (x', y') are subjected to deconvolution processing, whereby the pixel values of pixels having extended depth of field are restored (step S06).