COVER

The present invention relates to a refrigerator with camera and a protective cover which covers the lens at least partially and to the evaluation of the images taken in the refrigerator before being sent to the end user.

The accessibility of the consumption status and the stock management of the food stored in refrigerators by the user remotely is a necessity in today’s advanced technology. When the users are away from the refrigerator, it is important for them to monitor which products are in the refrigerator, which products are running low or which products are close to their expiration dates without opening the refrigerator door. To this end, the image of the shelves in the refrigerator is captured by imaging devices such as cameras, which is placed at least onto the door and at least onto the body, and is provided to the user via a mobile application, a mobile device or a display screen on the refrigerator. Providing the user with high quality images obtained by means of the imaging device is critical to enable the user to monitor the food items in the refrigerator properly and manage said information such as food consumption.

In the refrigerators with camera, at least one camera is disposed on the door and the body. The cameras are activated when the door is opened and when the door reaches a certain angle while being closed, the cameras are triggered by the control unit to capture images. Captured images are sent to the cloud and the control unit in order to be processed and provided to the end user.

In the refrigerators with camera, a protective cover is provided, which is can be opened/closed and which covers the camera in order to provide isolation to protect the camera against low temperatures in the cooling volume or in order to prevent images from being captured as per user preference. When images are to be captured by means of the camera, said protective cover is opened by being triggered by a control unit or opened manually by the user and then, the camera is enabled to capture images. However, if the cover cannot be opened for some reason or if there is a problem with the mechanism and the control unit does not detect this problem, images cannot be captured by the camera since the cover is in the closed position. Therefore, since this type of images are not useful, they are not needed to be sent to the cloud or the control unit, and then the end user.

For example, when the cover is black or gray, if the camera is triggered when the cover is in the closed position and images are captured, the captured images are predominantly blackish/grayish, dark and unclear images and the food items cannot be seen.

Such images are not required to be sent to the cloud/control unit or viewed by the end user. Sending such a dark image to the cloud or the control unit and the end user increases the energy consumption of the product and the cloud cost unnecessarily. Furthermore, when the refrigerator is not provided with a sensor or a mechanism detecting whether the camera cover is closed or open, the end user can be mislead to think that the images of the interior of the refrigerator are being captured. This reduces consumer satisfaction.

In the state of the art Japanese Patent Application No. JP2020003114A, a refrigerator with camera is disclosed, wherein the camera captures images depending on the opening and closing of the door and the images are sent to the server. Said refrigerator also stores the date/time information of the images. A refrigerator is disclosed, wherein based on this information the images are not presented to the user when the time between two images is shorter than a certain value.

In the state of the art Japanese Patent Document No. JP2018156275A, a refrigerator with camera is disclosed, wherein it is checked if a predetermined object is in the image or not, and if the object is in the frame, the image is presented to the user.

The aim of the present invention is the realization of a refrigerator with camera wherein the images captured when the cover in front of the camera is closed are identified and said images are not sent to the user and an error message is generated.

In the refrigerator with camera of the present invention, the images of the interior of the refrigerator are captured by means of the cameras placed on the body and the door. Whether the images are usable or not can be determined by means of image processing techniques. This process can be performed in the product or in a cloud environment. Thus, the unusable images coming from the product can be identified and the user can be warned. Furthermore, if this process can be performed in the product, the images are not sent to the cloud and energy saving is provided.

Every image such photograph, painting, etc. is made up of small two-dimensional pixels. Each pixel is represented by whole numbers between 0 and 255 without a unit, which represent color and light intensity. In color images there are three color channels, namely, red, green and blue color channels. All other colors are obtained by mixing these three primary colors in different ratios. In black and white or grayscale images, there is only one color channel and colors are represented by different shades of gray between black and white.

The control unit contains information about the pixel values predetermined by the manufacturer based on the colors and light intensities in images. For example, the pixel value of the darkest color black is zero. The pixel value of the lightest color white is 255. The colors in this range has a value based on their brightness and colors. For example, there is a certain number (for example, 400-500 pixels) of pixels which create the image of a red cola can and among the pixels which create this image, the pixels which represent the red parts of the cola can have values between 70 and 160 and the pixels which represent the white parts corresponding to the cola brand logo have values between 220 and 255.

The images captured when the camera cover is closed are in the color of the cover. For example, if the cover is black, than the image captured would be black. In a completely black image, the pixel values of all three color channels are close to zero based on the darkness of the black. If the image is grayish, the pixel values can be close to zero but greater than zero. Therefore, if the image captured by the camera is completely black, the average value of the pixel values would be close to zero. Thus, the average pixel values of the entire image is calculated and compared with a limit value by the control unit and if the calculated value is lower than the limit value predetermined by the manufacturer, it is determined that the camera cover is closed. The grayscale version of the image can be processed instead of the color version. In this case, similarly, the average value of the pixels would be close to zero.

However, if the color of the cover is not black, the pixel values would be close to the value which correspond to the color of the cover. Thus, the control unit cannot decide if the cover is closed or not by determining that the pixel values are zero or close to zero by comparing the pixel values with the limit value predetermined by the manufacturer. Since the image would be color when the cover is colored, the captured image would have a pixel value greater than zero. However, in a refrigerator shelf image captured when the cover is not closed, the pixel distribution of the colors is wider since most of the products can be seen. Even if the captured image of the colored cover has a pixel value greater than zero, it would be much lower than the pixel value distribution of a normal image of a shelf the camera is expected to capture. Therefore, it can be determined that a colored cover is closed by checking if the entire image is the same color or in similar colors, in other words, by checking the pixel distribution.

In an embodiment of the present invention, the graph used in determination is a color histogram graph. The color histogram graph is the graph of the pixel values and the number of pixels in an image (pixel values on the x-axis and number of pixels on the y-axis). Various pixel values are expected in a normal image wherein the refrigerator shelf and several food items thereon can be seen. Therefore, the graph of pixel values versus number of pixels would be scattered throughout the graph plane and the standard deviation would be relatively high. For example, if the camera captured an image when a colored camera cover is closed, the pixel value and number of pixels distribution in the captured image would be relatively narrow and the standard deviation would be low. Thus, the control unit can detect if the cover is closed or not by comparing said pixel value vs. number of pixels graph with the values predetermined by the manufacturer using the normal images captured when the camera cover is not closed. The clear image predetermined by the manufacturer is at least one picture of the food items on the shelf of the body or the door taken when the camera cover is not closed, wherein the food items in the image can be seen clearly by the user.

The histogram graph can also be applied to black- gray covers. In this case, since the pixel value on the graph would be close to zero, the cover can be detected to be open by means of a similar comparison.

The model embodiments related to the refrigerator realized in order to attain the aim of the present invention are shown in the attached figure, where:

Figure 1 - is the perspective view of the refrigerator in an embodiment of the present invention.

The elements illustrated in the figures are numbered as follows:

1. Refrigerator

2. Body

3. Door

4. Camera

5. Control unit

The refrigerator (1) comprises a body (2) into which the food items to be cooled are placed; a door (4) which provides access into the body (2) and which is in connection with the body (2); at least one shelf attached onto the body (2) and/or the inner panel of the door (3); at least one camera (4) placed in a position on the body (2) so as to capture the image of the shelves on the door (3) and/or in a position on the door (3) so as to capture the shelves in the body (2); and a control unit (5).

The refrigerator (1) of the present invention comprises a cover which can be opened / closed and which covers at least one camera (4), and the control unit (5) which determines that the camera (4) captures images when the cover is closed by comparing the pixel values of the image with the pixel values of an unblurred image predetermined by the manufacturer. In the refrigerator (1) of the present invention, the pixel values which correspond to the color and brightness values are stored in the control unit (5) memory. Said pixel values are generated by the control unit (5) for each image captured by the camera (4). The image is divided into pixels which form a two-dimensional matrix and the value of each pixel is tagged by the control unit (5). The pixel values tagged by the control unit (5) are compared with the distribution of the pixel values stored in the memory. For example, in an image which has 10,000 pixels, if the pixel distribution is in a wide range between 40 and 230, it is determined that there are several colors in the captured picture and the camera (4) cover is open. If the image tagged by the control unit (5) consists of for example, 1000 pixels having a distribution of 30-50 pixel values, it is determined that the camera (4) captures the image of the cover.

In an embodiment of the present invention, the refrigerator (1) comprises the cover which is black or gray, covering the camera (4) at least partially, and the control unit (5) which evaluates the pixel values of the image captured by the camera (4) and determines that the cover is closed if the pixel value of the image is close to zero.

In an embodiment of the present invention, the refrigerator (1) comprises the cover which is colored, covering the camera (4) at least partially, and the control unit (5) which evaluates the histogram graph (pixel value vs. number of pixels graph) which is the distribution of pixel values of the captured image and compares the standard deviation value with a value predetermined by the manufacturer so as to detect if the cover is closed or not.

In an embodiment of the present invention, the refrigerator (1) comprises the control unit (5) which chooses pixels from at least two corners of the captured image and compares the pixel values with each other, and determines that the cover is closed if the pixel values are close to each other. In this embodiment, if the pixel values taken from at least two end points are close to each other, it is determined that the image is captured when the cover is closed, without the need for using the pixel value vs. number of pixels distribution.

In an embodiment of the present invention, the refrigerator (1) comprises the control unit (5) which does not process and deletes the image if the image is captured when the camera (4) is covered by the cover. Thus, savings in energy is provided by avoiding the processing of unnecessary volumes of data.

In an embodiment of the present invention, the refrigerator (1) comprises the control unit (5) which informs the user that the image is captured when the camera (4) is covered by the cover with an audio or text message delivered by means of an imaging unit such as a mobile device.

By means of the present invention, a refrigerator with camera (1) is realized, wherein the camera (4) lens is protected against the vapor formed on the camera (4) as a result of the cool air or the camera (4) can be covered when the user does not prefer the images to be captured and the unusable images caused by the cover which remains closed during said protection are deleted without being presented to the user.