To provide a security management system and a security management method for particularizing a user of a mobile phone to authenticate a person without the need for a watchman and a server for storing image data in advance and comparing the image data stored in advance with image data photographed by the mobile phone.

An image input section 111 provided with the mobile phone 100 receives an image of a face of a user and produces image data of the face of the user. When an input data processing section 131-1 compares image data of faces of permittees being registered people which an input data authentication data storage section 132-1 stores in advance with image data of the face of the user and determines that the user is the same as a registered person, the input data processing section 131-1 requests the user to enter a password, and transmits the combination between the password entered by the user and a telephone number of the mobile phone 100 to the security management apparatus. When the received combination between the password and the telephone number is coincident with a combination between the password and the telephone number stored in advance, the security management apparatus opens / closes a door entered from / exited to a particular place.