To make compatible both of a high-performance classification function and a high-throughput image collection function in a defect observation device for automatically collecting and classifying defective images existing on a sample such as a semiconductor wafer.

An image processor (defect classification device section) has a function for estimating the conforming state (reference image) of a part where defects are present by using a defective image, and a function for determining fatality by the defects, an irregular state or the like by using the estimation result, thus making compatible both of a high-throughput image collection sequence without acquiring reference images and highly precise defect classification.