To provide a sheet conveying device, an image reading device, a method of controlling the sheet conveying device, and a sheet conveying system, that can satisfactorily convey a sheet irrespective of the thickness of a sheet by using a plurality of conveyance passages, and to provide an image reading system.

The sheet conveying device includes: a U-turn path performing the conveyance by reversing the front and back of a sheet from a sheet feeding part; a sheet conveyance passage (and straight path) performing straight conveyance without reversing the front and back of a sheet from the sheet feeding part; a conveyance passage switching means for performing the switching to either one of the U-turn path and the straight path; a thick sheet detection means for detecting a thick sheet having a thickness equal to or more than a predetermined threshold; and a control part controlling the sheet conveyance from the sheet feeding part. The control part stops the conveyance of the thick sheet when the thick sheet detection means detects the thick sheet in a switched state to the U-turn path by the conveyance passage switching means, or continues the conveyance of the thick sheet by performing the switching from the U-turn path to the straight path by the conveyance passage switching means.