To prevent any illegal access to a protection target due to the use of an electronic card or an electronic key illegally acquired by a person other than an owner.

This authentication server for limiting an operation to a protection target by communicating with a portable terminal having a biological information input part which inputs biological information is provided with: a receiving part which receives biological information input by a biological information input part of the portable terminal and the location information of the portable terminal from the portable terminal by communication equipment; a biological information authentication part for identifying the biological information of a predetermined user by a processor based on the biological information received by the receiving part; a location authenticating part for authenticating that the portable terminal is positioned at a predetermined location by the processor based on the location information of the portable terminal received by the receiving part; and an operation permitting part for permitting the operation to the protection target by the processor when authentication succeeds by the biological information authentication part and the position authentication part.