FIELD OF THE INVENTION

[0001 ] The present invention generally relates to methods and systems for testing digital imaging devices. In the context of the present invention, an imag- ing device refers to a device capable of capturing light emitted by or reflected from an object and creating an image on the basis of that light. A digital imaging device refers to an imaging device capable of creating an image in a digital format. An imaging device may be a dedicated imaging device, such as a still or video camera, imaging microscope or telescope, or the imaging device may be integrated to any of a wide variety of multi-purpose electronic devices, such as smartphones, palmtop, laptop or desktop computers, surveillance devices, etc., to name just a few of the multitude of different imaging devices.

BACKGROUND OF THE INVENTION

[0002] US published patent application no 2009/303324 A1 by Stewart Green- hill et al. concerns an automatic testing system for surveillance camera installations. The automatic testing system of Greenhill et al. comprises comprising an input port to receive test or "probe" images from at least one camera in the installation; a computer memory to store a reference image from the at least one camera; a computer processor to compare a probe image with a reference image from the same camera and produce an output when maintenance is required for that camera. The comparison involves extracting salient features from both the probe and reference images; calculating matching factors between the salient features extracted from both images; and computing a decision about whether maintenance is required from the matching factors.

[0003] The automatic testing system of Greenhill et al. leaves several questions unanswered, however. In view of the fact that the Greenhill system is directed to testing surveillance camera installations, as opposed to the surveillance cameras at the point of time they leave the manufacturing plant, a rapid throughput of testing is not of primary concern because the time to test a cam- era is supposedly much shorter than the time required to transfer the testing system to the next site. Accordingly, one of the open questions in the Greenhill et al. system is optimization of testing speed. Another open question is determining which features are implemented in which camera. For example, it is pointless to test autofocus speed in a simple web camera that lacks autofocus function. DISCLOSURE OF THE INVENTION

[0004] An object of the present invention is thus to provide a method, an apparatus and a computer program product so as to answer the open questions identified above.

[0005] The object of the invention is achieved by aspects of the inventions as defined in the attached independent claims. The dependent claims and the following detailed description and drawings relate to specific embodiments which solve additional problems and/or provide additional benefits.

[0006] According to the invention, the testing system is configured to receive the following information in respect of each of the imaging devices: an identifier of the imaging device; flow control instructions regarding testing of the imaging device; and images captured by the imaging device. In other words, the imaging device provides flow control instructions to the testing system. A benefit of this feature is that when new types or versions of imaging devices are tested, the testing system need not be updated simultaneously. This is because the flow control instructions for testing a new imaging device can be programmed into the imaging device itself.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the following the invention will be described in greater detail by means of specific embodiments with reference to the attached drawings, in which:

Figure 1 shows a block diagram of a testing system according to an embodiment of the invention;

Figure 2 shows an isolated test environment according to an embodiment of the invention; and

Figures 3 and 4 show exemplary test charts.

DETAILED DESCRIPTION OF THE SPECIFIC EMBODIMENTS

[0008] The following description of specific embodiments uses the term "camera" to refer to any imaging device, examples of which were listen in the introductory portion of this description. Figure 1 shows a block diagram of a testing system and figure 2. shows a one version of isolated test environment according to an embodiment of the invention. The three major sections shown in Figure 1 are a controlled isolated testing environment 1 10, environment controller 150 and an analysis system 170. In the controlled testing environment 1 10, reference numeral 100 denotes the camera (imaging device) to be tested. Reference numeral 120 denotes scene-altering means for dynamically changing the reference content of the scene (eg a visual display unit systems or motorized test chart changer), that is under direct control of the environment controller 150, and under indirect control of the camera 100 to be tested. Reference numeral 131 denotes the projection of the camera 100 to be tested. Reference numeral 130 denotes boundary lines of the image plane of the camera projection 131 that fits within a scene controlled by scene-altering means 120. Figures 3 and 4 show illustrative examples of test charts 200, 300 that may be shown by the scene-altering means. Reference numeral 121 denotes stereo speakers which plays sounds in case of testing video recording functionality of camera 100. reference numeral 122 denotes adjustable lighting of the isolated testing environments 1 10. Speakers 121 and lighting 122 are under direct control of the environment controller 150, and under indirect control of the camera 100 to be tested. Reference numeral 123 from Figure 2 denotes an adjustable stand for camera 100 under test, such as a robot arm with six degrees of freedom, whose primary purpose is to subject one camera 100 at a time to a testing position. The camera 100 has an image plane of the camera projection 130, denoted by reference 131 defined by the focal length of its lens and the size of light-sensitive sensor. When the camera 100 is placed in the testing position, the image plane 131 of the camera projection 130 comprises a dynamically alterable scene with one or more test targets.

[0009] Within the environment controller 150, reference numeral 160 denotes an environment control server whose primary purpose is to control isolated testing environment 1 10 based on service requesting of the imaging system 100 under test. Reference numeral 162 denotes a database in which the controller server stores test data obtained from the camera 100 to be tested. Database 162 contain reference data, such as reference images, also in case when use visual display unit as dynamic reference content changer. The database 162 may naturally be physically integrated in the control server 160, and if desired, the verification data and testing data may be divided into separate databases and/or to separate computers. Reference numeral 164 denotes a communication medium via which the camera 100 under test communicates with the verification server. For instance, the communication medium 164 may be any of commonly available wired or wireless communication media, such as Bluetooth, infrared, wireless local-area network (WLAN), universal serial bus (USB), to name just a few of the wide variety of ubiquitous examples. Reference numeral 166 denotes set of communication mediums via which the control server 160 controls the reference content changer 120, stereo speakers 121 , adjustable lighting 122 and adjustable imaging device stand 123.

[0010] According to an optional but advantageous feature, the control server 160 is configured to request the adjustable stand 123, such as a robot arm with multiple degrees of freedom, to take one camera at a time from a platform, subject the camera for testing and replace it on the platform from which it took the camera, or on a different platform. The platform or platforms, which is/are not shown separately, may be manually or automatically movable device racks or a conveyor belts,

[0011] Within the analysis system 170, reference numeral 180 denotes a verification server whose primary purpose is to verify correct operation of the imaging system 100 under test. The verification server is also connected to the database 162 where verification server stores reference data, such as reference images, and verified test data obtained from the camera 100 to be tested. Reference numeral 184 denotes a communication medium via which the environment control server 160 communicates with the verification server.

[0012] For instance, the communication medium 184 may be any of commonly available wired or wireless communication media. In a cloud-based implementation, the verification server 180 can serve multiple testing environments simultaneously. Alternatively, the verification server may be part of the environment control server 160, in which case it only serves one testing environment at a time.

[0013] In the testing system of the present embodiment, the environment control server 160 acts as a controller for receiving flow control instructions from the camera 100 under test and for instructing the scene-altering means, such as the display 120, to vary the currently displayed test targets, on the basis of the flow control instructions received from the camera. In the present embodiment the controller also receives an identifier, such as serial number, of each camera under test. The serial number or other identifier of the camera may be used to associate test results obtained from a specific camera with the identity of the camera.

[0014] The testing system of the present embodiment further comprises an analysis apparatus for receiving one or more images (actually, data sets representative of images) from the camera 100 under test, and for performing one or more analyses on the basis of the images received from the camera 100. For the purposes of the present disclosure, an image received from the camera is a shorthand notation for a data set representative of the respective image. In other words, the analysis system analyses not the image (light) cast on the camera's sensor during an exposure but the data set that the camera creates on the basis of that exposure.

[0015] The testing system, and particularly the verification server 180, may perform various forms of analyses on the images captured by the camera 100 under test. A non-exhaustive list of analyses comprises:

- Autofocus speed and accuracy. As to autofocus speed, the camera may send an autofocus start signal to the analysis system at the point of time when it starts to acquire focus. After acquiring optimal focus, or what the camera under test considers optimal focus, it sends an autofocus end signal. The analysis system then records the time interval between the autofocus start and end times. Autofocus accuracy may be determined by configuring the camera to focus on a predetermined focusing target, next to which is another target that extends to distances both less and more distant from the camera than the focus target is. If the point of best focus is obtained from the target that is equidistant from the camera with the focusing target, the camera's autofocus works optimally.

- Exposure: For instance, the camera may create images of a stepwise or step-less graduated grayscale target, and if the images exhibit continuous gradation of tone across the graduated grayscale targets, the camera's auto-exposure operation is functional.

- Colour accuracy can be determined by analysing images of calibrated colour patches captured by the camera under various lighting conditions. The red, green and blue values of samples of a few pixels across should match corresponding values obtained by a known reference camera.

- Focus tracking capability of the camera may be measured by moving a focusing target across the camera's field of view (or a part of it).

- Temporal noise.

- Auto white balance. - Dynamic range.

- Signal-to-noise ratio.

[0016] All of the above tests are performed under control of the camera 100 under test, in such a manner that the camera outputs flow control instructions acted on by the environment control server 160 and verification server 180, while the images are being received and stored into the database 162. Naturally, it is possible to store the images captured by the camera 100 under test and analyse the images after the capturing of images is complete. Or, the verification server 180 may analyse the captured images in real time and store only the some key parameters, or in a simple case, only a pass/fail flag, in respect of each camera to be tested.

[0017] In more advanced implementations, the testing system comprises an audio source, such as a computer-controlled sound generator, and stereo speaker systems 121 for testing the audio-capturing features in video recording. In addition to images, the camera 100 may capture the audio or video and send data representative of the captured audio or video to the testing system. The testing system is preferably installed in a controlled isolated environment, which is at least reasonably free from external disturbances. In particular, the testing system should be free of sporadically changing lighting conditions, vibrations or external sounds.

[0018] Those skilled in the art will realize that the embodiment described above is by no means the only possible one. Rather the implementation can be varied in various ways.

[0019] Various use cases of the testing system shown in Figure 1 will be described next. For instance, autofocus capabilities of cameras may be tested via the following procedure. Before operational testing of the camera 100, the camera 100 under test requests calibration service to calibrate testing environment 1 10 based on setting of the camera 100 under test from the environment control server 160. The calibration process adjustments to testing environment 1 10 such that the reference image (test target) fully covers the image plane 131 of the camera projection 130 and the illumination conditions in the test environment responding to the real world. During calibration, the environment control server 160 presents a calibration target, such as the one shown in Figure 3, on the display 120. The Camera 100 under test captures a test image and transfers it, via the communication mediums 164 and 184 to the verification server 180 for analysis. The verification server 180 calculates the current size of the image plane 131 and the position and orientation of the reference data 200 in the test image. Based on this information the environment control server 160 adjusts the camera 100 under test into a correct location such that definition of the calibration process is fulfilled. For instance, the environment control server 160 may be configured to adjust the mutual positions of the camera 100 and image plane 131 such that the test charts occupy a predetermined percentage of the image area or a predetermined number of pixels, regardless of the magnification of the lens, which in turn depends on the focal length and cell size.

[0020] After calibration, the camera 100 under test requests the environment control server 160 to present a reference test target for actual testing. For instance, in the case of autofocus testing, the test target may be similar to the one shown in Figure 4, wherein five portions of the test target comprise vertical black-and-white lines. The environment control server 160 adjusts the device stand 123 in correct position and orientation based on results of the calibration process. The camera 100 under test captures one or more test images and transfers them, via the communication medium 164 and 184, to the verification server 180 for analysis. The verification server 180 may then count the number of detected lines (eg by detecting the five areas that comprise the vertical black-and-white lines) and by measuring the contrast (eg by measuring the ratio of luminosity values between the white lines (luminosity peaks) and black lines (maximum troughs). An abnormally low number of detected lines, such as lines with an abnormally low contrast, indicates malfunction in autofocus abilities of the camera.

[0021] In some implementations, the verification server 180 may report results of the analysis back to the camera 100 under test.

[0022] It will be apparent to a person skilled in the art that, as the technology advances, the inventive concept can be implemented in various ways. The invention and its embodiments are not limited to the examples described above but may vary within the scope of the claims.