To provide a mechanism for achieving high-accuracy focus detection, even when a change occurs in the movement and blur state of an object, during focus detection operation.

A focus adjusting device includes setting means for two-dimensionally setting a plurality of range-finding areas 504 in an X direction and in a Y-direction with respect to an imaging element 5, generation means for generating a focus evaluation value for evaluating a focusing state by using the high-frequency component of the image signal of the imaging element 5 corresponding to the plurality of range-finding areas arranged in parallel in the X direction, and focus adjusting means for adjusting the focusing state, so that an AF evaluation value being the total sum of the focus evaluation values generated by the generation means becomes maximum. At least one boundary of two boundaries in the X direction of a first range-finding area A[1,1] included in the plurality of range-finding areas arranged in parallel in the Y direction is set within the area in the X direction of a second range-finding area A[2,1].