The present invention relates to a refrigerator wherein the sharpness of the image captured by at least one camera placed in the refrigerator is detected and the blurry images are not processed and presented to the user.

The accessibility of the consumption status and the stock management of the food stored in refrigerators by the user remotely is a necessity in today’s advanced technology. When the users are away from the refrigerator, it is important for them to monitor which products are in the refrigerator, which products are running low or which products are close to their expiration dates without opening the refrigerator door. To this end, the image of the shelves in the refrigerator is captured by imaging devices such as cameras, which is placed at least onto the door and/or at least onto the body, and is provided to the user via a mobile application, a mobile device or a display screen on the refrigerator. Providing the user with high quality images obtained by means of the imaging device is critical to enable the user to monitor the food items in the refrigerator properly and manage said information such as food consumption.

The image from the camera, for example the image of the door shelves, may be blurred when taken while the door is being opened and closed quickly. Similarly, if the door is moving quickly while the image of the shelves in the body is being taken by the camera placed on the door, the image of the body shelves taken may be blurred.

In the refrigerators with camera, at least one camera is disposed on the door and the body. The cameras are activated when the door is opened and when the door reaches a certain angle while being closed, the cameras are triggered by the control unit to capture images. Captured images are sent to the cloud and the control unit in order to be processed and provided to the end user. However, since some of the captured images are not useful, they are not needed to be sent to the cloud or the control unit, and then the end user.

Another factor which causes the images captured in the refrigerators with camera is vapor. Since the cameras are located in a cold environment in the refrigerator, condensation forms on the camera lens. The resulting condensation prevents clear images from being taken by the camera. There is a need to identify the situations which cause the captured images to be blurry, examples of which are given above, and prevent such images from being sent to the cloud/the control unit and the end user. Thus, a decrease in the energy consumption of the refrigerator, savings in cloud costs and an increase in user satisfaction will be provided.

In the state of the art refrigerators with camera, a film coating is provided on the lens in front of the camera. Said coating moistens the lens, defrosting the condensation and preventing a blurry image from being captured by the camera. However, over time, said anti-fog film loses its effectiveness and cannot dissolve the condensation formed in front of the camera. Therefore, determining whether the effectiveness of the film continues is important in terms of customer satisfaction and capturing a clear image with the camera. It is difficult to keep track of when the anti-fog films need to be replaced. Therefore, it is important to determine that the film will expire in the near future and to provide the necessary service support to the user before the film expires completely.

In the state of the art Japanese Patent Document No. JP6615361B2, a refrigerator with camera is disclosed, wherein the condensation is removed by mounting a heater to the camera lens and the energy to be supplied to the heater is related to the opening / closing of the door.

The aim of the present invention is the realization of a refrigerator with camera wherein unusable, blurry images are detected, unusable images are not shown to the user and the user is warned if such unwanted situations occur.

Another aim of the present invention is the realization of a refrigerator wherein the condensation on the camera is monitored and the anti-fog film on the camera has expired or is about to expire.

In the refrigerator with camera of the present invention, the images of the interior of the refrigerator are captured by means of the cameras placed on the body and the door. Whether the images are usable or not can be determined by means of image processing techniques. This process can be performed in the product or in a cloud environment. Thus, the unusable images coming from the product can be identified and the user can be warned. Furthermore, if this process can be performed in the product, the images are not sent to the cloud and energy saving is provided.

Each photograph, each image consists of thousands of two-dimensional pixels. The camera sensor converts the image captured by the camera to pixel values. Therefore, pixels are numerical values in any data processing unit such as a computer or processor. The numerical value of the pixels, which is a two-dimensional matrix, is converted into a numerical value according to data such as the color in the pixel and the brightness value of this color. For example, the numeric value of a darkest, non- bright black-weighted pixel is zero or very close to zero. A pixel matrix with a predominantly grayish color but slightly brighter colors will have a value greater than zero but as low as 10-30. The brightest, lightest colors have a value above 200 up to 255. When the image captured by a camera is taken incorrectly, for example when the front of the camera is closed, etc., the color distributions in the pixels will be very limited and it can be determined that the image is erroneous. However, if the image from the camera covers, for example, an entire body shelf but is not sharp, there will still be pixels with different values in this image. Therefore, it is difficult to calculate sharpness based on pixel values alone.

Whether the image captured by the camera is sharp or not can be calculated according to the frequencies of the pixels, that is, the sharpness of the transition between the pixels. For example, if a sharp image of a milk carton and a dark pot standing side by side on a shelf is captured by the camera, the transition between the pixel values of the two images will be sharp. So the image will have high frequency. Since the pixel value of a gray pot is 30-40 and the pixel values of a white milk can is above 200, the transitions between them have a high frequency.

Said frequency value can be calculated by methods such as Eaplace transform and Fast Fourier transform. With these methods, an amplitude graph which changes depending on the pixel values of the image is drawn by the control unit. Said amplitude value shows the frequency which is the transition between pixels. The control unit contains a frequency graph which changes according to the many pixels of clear pictures. Said reference values predetermined by the manufacturer enable the calculation of the sharpness of the image captured by the camera.

The transitions between the pixel matrices of blurry images are not sharp. Even though the color distribution in the pixels is different from each other, since the image is blurred, the brightness values are low and the transition of the pixel values, that is, the frequency, is low. Therefore, the control unit may decide that a low frequency image is not sharp.

In an embodiment of the present invention, an anti-fog film is coated in front of the camera lens. The film at least partially covers the camera lens. The cool air circulating in the body forms condensation on the camera lens. The film is activated at certain intervals or is operated continuously, thus ensuring that the condensation is removed.

In said embodiment, when the anti-fog film expires, the condensation on the camera causes images to be blurred. Therefore, after a while, the user should be warned that the film is no longer functional.

The refrigerator of the present invention comprises a control unit which detects blurry images by using the frequency fluctuation depending on the change in the pixel values of the image taken by the camera, and which calculates in advance the expiry of the anti-fog film.

The blurriness of the image captured by the camera may not be caused by only condensation. The reason may be the quick opening/closing of the door or the image quality of the camera being adversely affected by a foodstuff placed in front of the camera. In such cases, the user must be prevented from unnecessarily replacing the anti-fog film.

Therefore, the control unit contains an average economic life of the anti-fog film which is precalculated by the manufacturer. When a blurry image is captured by the camera, the control unit compares the precalculated average economic life of the film with the usage time until that moment before warning the user about the economic life of the film. If the two durations are close to each other, it is concluded that the economic life of the film has expired.

By means of the present invention, a refrigerator with camera is realized, wherein the sharpness scores of the images captured by the camera are calculated and the blurry images are not presented to the user.

The model embodiments related to the refrigerator realized in order to attain the aim of the present invention are shown in the attached figure, where:

Figure 1 - is the perspective view of the refrigerator in an embodiment of the present invention.

The elements illustrated in the figures are numbered as follows:

1. Refrigerator

2. Body

3. Door

4. Camera

5. Control unit

The refrigerator (1) comprises a body (2) into which the food items to be cooled are placed; a door (4) which provides access into the body (2) and which is in connection with the body (2); at least one shelf attached onto the body (2) and/or the inner panel of the door (3); at least one camera (4) placed in a position on the body (2) so as to capture the image of the shelves on the door (3) and/or in a position on the door (3) so as to capture the shelves in the body (2); and a control unit (5).

The refrigerator (1) of the present invention comprises the control unit (5) which detects whether the image captured by the camera (4) is unblurred or not by comparing the frequency changes related to the pixel values of the capture image with the frequency changes related to the unblurred images predetermined by the manufacturer.

In the refrigerator (1) of the present invention, the control unit (5) contains the graphs of frequency values predetermined by the manufacturer, defining the sharpness of the transition between pixels in terms of pixel values. There are three frequency transitions in said variation graphs. Said frequencies, which are low, mid and high frequency transitions, determine whether the image captured by the camera is sharp or not. The mid and high frequencies are the frequencies that are only achieved in the transitions between pixels in sharp images. Since the transition between individual pixels in blurred images is unclear, it is smoother and has lower frequencies. The transition between said values of pixel matrices ranging from 0 to 255 can be calculated by mathematical modeling known as Fast Fourier or Laplace transform. A matrix obtained as a result of said mathematical models is processed by the control unit on the image captured by the camera and a frequency distribution graph is created. Said frequency distribution graph is compared with the high frequency, low frequency and mid frequency graphs predetermined by the manufacturer and the sharpness score of the image is calculated. Thus, blurry images are detected.

In an embodiment of the present invention, the refrigerator (1) comprises the control unit (5) wherein images with mid and/or high frequency values are considered sharp and images with low frequency values are considered blurry according to a graph which classifies as low, mid or high the frequency values representing the sharpness of the transition between the pixels in terms of the pixel values of the image captured by the camera (4).

In an embodiment of the present invention, the refrigerator (1) comprises an anti-fog film which at least partially covers the front of the camera (4) and which prevents the formation of condensation on the front of the camera (4).

In an embodiment of the present invention, the refrigerator (1) comprises the control unit (5) which decides that the anti-fog film is about to expire by detecting the condensation on the front of the camera (4) according to whether the image is unblurred or not depending on the frequency distribution related to the pixel values in the image captured by the camera (4).

In an embodiment of the present invention, the refrigerator (1) comprises a control unit (5) which warns the user by audio and/or written message when the anti-fog film has expires or is about to expire. In an embodiment of the present invention, the refrigerator (1) comprises a control unit (5) which warns the user to close the door (3) more quickly or more slowly or that there is an obstacle in front of the camera (4) upon deciding that the image captured by the camera (4) is not unblurred. By means of the present invention, a refrigerator (1) with camera is realized, wherein the energy consumption is efficiently managed, unnecessary images are not stored and only sharp images are presented to the user.