To provide a technique for readily detecting and classifying defects at a high speed.

This defect classifying device acquires the coordinate values of a typical point of respective defect elements, by specifying a plurality of defect elements from among image data of an inspection area on a substrate. A defect element pair determining part 53 performs Delaunay triangulation with the typical point of the respective defect elements as base point, and determines a defect element pair constituted of the defect elements of both end points of the respective sides that constitute a plurality of triangles. A determining part 52 acquires a plurality of defect element pairs constituted of the deflect elements derived from the same defect. A cluster-forming part 56 has a binary search tree structure and forms a plurality of clusters which are clusters of the defect elements, respectively derived from the same defect. A characteristic quantity is calculated, by extracting a defect image including the defect elements included in the respective clusters, and the defect corresponding to the respective clusters is classified. Thus, the defect is detected easily and classified properly, at a high speed.