OF A PREDEFINED SPACE

Field

The present disclosure relates to entropy rating. Examples relate to an apparatus, a system, and a method for rating an entropy of a predefined space.

Background

In the context of the present disclosure, entropy can be thought of as a measure of disorder or messiness that a predefined space exhibits. The predefined space may be a room of a hotel, for instance. Cleaning rooms in a hotel and an allocation of cleaning personnel to the rooms is usually determined according to a fixed workflow. Since there is no objective measure of messiness, such as an entropy score, of the rooms, working time needed for each room cannot be estimated properly. A level of messiness or a type of cleaning steps needed in a room are not taken into account in planning a sequence of cleaning shifts. Moreover, if cleaning is performed partially by cleaning robots, it proves difficult to coordinate their deployment with out knowing in which rooms their specialized skills are needed.

Hence, there may be a demand for improved entropy or messiness rating.

Summary

The demand may be satisfied by the subject matter of the appended claims.

An example relates to an apparatus for rating an entropy of a predefined space. The apparatus comprises processing circuitry configured to acquire at least one digital image of the prede fined space from an image sensor. The processing circuitry is further configured to divide the digital image into one or more image segments corresponding to one or more space segments of the predefined space. The processing circuitry is further configured to correlate the image segments with corresponding reference image segments. The reference image segments relate to a predefined entropy score of the respective space segments. The processing circuitry is further configured to determine an entropy score for the image segments based on the corre lation.

Another example relates to a system for rating an entropy of a predefined space. The system comprises an image sensor configured to capture a digital image of the predefined space. The system comprises an apparatus as described herein.

Another example relates to a method for rating an entropy of a predefined space. The method comprises dividing a digital image into one or more image segments corresponding to one or more space segments of the predefined space. The method further comprises correlating the image segments with corresponding reference image segments, wherein the reference image segments relate to a predefined entropy score of the respective space segments. The method further comprises determining an entropy score for the image segments based on the correla tion.

Brief description of the Figures

Some examples of apparatuses and/or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

Fig. 1 illustrates an example of an apparatus for rating an entropy of a predefined space;

Fig. 2 illustrates a process chart of exemplary steps of image processing;

Fig. 3 illustrates an exemplary digital image of the predefined space;

Figs. 4a-c illustrate examples of an output of the apparatus;

Fig. 5 illustrates an example of a system for rating an entropy of a predefined space;

Fig. 6 illustrates an example of a method for rating an entropy of a predefined space. Detailed Description

Some examples are now described in more detail with reference to the enclosed figures. How ever, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications of the features as well as equivalents and alternatives to the features. Furthermore, the terminology used herein to describe certain ex amples should not be restrictive of further possible examples.

Throughout the description of the figures same or similar reference numerals refer to same or similar elements and/or features, which may be identical or implemented in a modified form while providing the same or a similar function. The thickness of lines, layers and/or areas in the figures may also be exaggerated for clarification.

When two elements A and B are combined using an “or”, this is to be understood as disclosing all possible combinations, i.e., only A, only B as well as A and B, unless expressly defined otherwise in the individual case. As an alternative wording for the same combinations, “at least one of A and B” or “A and/or B” may be used. This applies equivalently to combinations of more than two elements.

If a singular form, such as “a”, “an” and “the” is used and the use of only a single element is not defined as mandatory either explicitly or implicitly, further examples may also use several elements to implement the same function. If a function is described below as implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity. It is further understood that the terms "include", “in cluding”, “comprise” and/or “comprising”, when used, describe the presence of the specified features, integers, steps, operations, processes, elements, components and/or a group thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and/or a group thereof.

Fig. 1 illustrates an example of an apparatus 100 for rating an entropy of a predefined space. The apparatus 100 comprises processing circuitry 110 configured to acquire at least one dig ital image 120 of the predefined space from an image sensor 130. For example, the processing circuitry 110 may be a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which or all of which may be shared, a digital signal processor (DSP) hardware, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processing circuitry 110 may optionally be coupled to, e.g., read only memory (ROM) for storing software, random access memory (RAM) and/or non-volatile memory.

The apparatus 100 may be, e.g., part of a computer system. The computer system may include multiple constituent computer systems which may, for example, be handheld devices, appli ances, laptop computers, desktop computers, mainframes, distributed computer systems, dat acenters, or wearables.

The predefined space may relate to a space with certain dimensions, e.g., one or more rooms in a hotel, a hospital, an office, a private home, or an outdoor space. The entropy may be considered an objective measure for a disorder, messiness, or untidiness the predefined space exhibits. For instance, the higher the disorder of the predefined space, the higher may be the entropy.

The image sensor 130 may be thought of a device comprising a photo-sensitive area config ured to generate an electric signal based on an incident light from the predefined space. The digital image 120 may relate to, e.g., a bitmapped or vector image representing at least one view of the predefined space. The digital image 120 may be inferred from the electric signal of the image sensor 130. The digital image 120 may comprise a plurality of pixels arranged in an array. Each of the pixels may represent a discrete quantity of a property (such as inten sity, color) of the incident light at a respective point of the view of the predefined space.

The processing circuitry 110 may be coupled with a hardware and/or software interface to the image sensor 130 for acquiring the at least one digital image 120. It is to be noted that the processing circuitry 110 and the image sensor 130 may be integrated into a shared device. In other examples, the processing circuitry 110 and the image sensor 130 may belong to different devices. For the latter, a communication module may be necessary to transmit machine-read- able data representing the digital image 120 from the image sensor 130 (or a respective device, e.g., a camera system) to the processing circuitry 110 (or a respective device, e.g., a computing system). The apparatus 100 is further explained with reference to Fig. 2 which illustrates a process chart of exemplary steps of image processing 200 which may be performed by the processing circuitry 110.

The digital image 120 may be taken as an input to a processing circuitry (not shown), such as processing circuitry 110. As a first step 210, the digital image 120 is divided into several image segments, such as image segments 215-1, 215-2, 215-3, corresponding to space seg ments of the predefined space. The image segments 215-1, 215-2, 215-3 may relate to differ ent segments of the view, e.g., a bird’s eye view, of the predefined space. Each of the image segments 215-1, 215-2, 215-3 may represent a respective space segment.

The digital image 120 is shown as a rectangle only by way of illustration. The rectangle may rather indicate a shape of the captured view of the predefined space. In other examples, the digital image 120 (or the captured view) may also have a different shape than the one shown in Fig. 2. The digital image 120 may not have a physical structure as it may be thought of as machine-readable data. However, it may be useful to visualize it, e.g., as two-dimensional structure, as the digital image 120 may comprise, for example, datapoints (pixels) associated to coordinates in a two-dimensional coordinate system. In case the digital image 120 com prises three-dimensional spatial information about the predefined space, the two-dimensional representation shown in Fig. 2 may illustrate a two-dimensional projection of the three-di mensional spatial information.

In Fig. 2, six similar-sized image segments, among others the image segments 215-1, 215-2, 215-3, are shown as rectangles by way of illustration. In other examples, the digital image 120 may be divided into more or less image segments. The image segments may be of any shape and may differ in shape or size. The image segments or a subset thereof may partly overlap other image segments. Moreover, the digital image 130 may, alternatively, be only partly divided into image segments. In some examples, the digital image 120 may be divided into only one image segment, i.e., the digital image 120 may equal the image segment.

As a second step 220, the processing circuitry selects the image segment 215-1. The selection of image segment 215-1 is meant as an example. In other examples, other image segments, e.g., 215-2, 215-3, of the digital image 120 may be similarly selected and processed with the steps explained below. Optionally, more than one of the image segments may be selected and processed in parallel.

As a third step 230, the processing circuitry compares the image segment 215-1 with a refer ence image segment 235. The reference image segment 235 may be stored on a data storage coupled with the processing circuitry. The reference image segment 235 and the image seg ment 215-1 may relate to the same space segment of the predefined space. The reference image segment 235 may be thought of a priorly captured or otherwise generated digital image depicting said space segment. For example, an image sensor, such as the image sensor 130, may have priorly captured a digital image of the predefined space, preferably with a camera angle corresponding to a camera angle of the image sensor 130 when the digital image 120 was captured. Thus, the image segment 215-1 and the reference image segment 235 may de pict a similar/same view of the predefined space. The priorly captured digital image may have been digitally processed, e.g., noise may be removed, before being used for providing the reference image segment 235.

The data storage may also store a specific entropy score assigned to the reference image seg ment 235. The specific entropy score may be predefined, e.g., by a human operator or a pre defined mapping function associating a specific state of the predefined space with the specific entropy score. For example, assuming the predefined space is a hotel room, the hotel room may be tidied as envisaged by a hotel management. A resulting state of the hotel room may be rated as clean and may be assigned a value, the specific entropy score, describing a high level of cleanliness (compared to an untidied state of the hotel room). Alternatively, the spe cific entropy score may indicate a particularly messy state of the hotel room or a certain level, e.g., a medium level, of messiness/cleanliness.

Assuming the digital image 120 and a reference digital image (from which the reference im age segment 235 may be inferred) are taken from the predefined space at different times of the day or with different light sources illuminating the predefined space, problems may arise when comparing the image segment 215-1 with the reference image segment 235 caused by different light intensities, for instance. Thus, it may be necessary that the processing circuitry adapts lighting conditions of the image segment 215-1 in accordance with reference lighting conditions of the reference image segment 235, i.e., the processing circuitry may apply a digital filter on the digital image 120 to align brightness and/or contrast of the image segment 215- land the reference image segment 235.

Alternatively, these problems may be circumvented by ensuring that the lighting conditions when the digital image 120 and the reference digital image are taken, are similar, e.g., in case of the predefined space being a hotel room, doors and windows of the hotel room may be closed and certain light sources at certain positions in the hotel room may be turned on. Be sides, an image sensor, such as the image sensor 130, may take a white balance with a certain white spot provided in the predefined space before capturing the digital image 120.

The processing circuitry may use any algorithm or method for comparing the image segment 215-1 with the reference image segment 235. For example, the processing circuitry may com pare values of light intensity or color of light indicated by corresponding pixels of the image segment 215-1 and the reference image segment 235. The processing circuitry may use, e.g., a statistical method for correlating the image segment 215-1 with the reference image segment 235. The processing circuitry may compare the image segment 215-1 with the reference image segment 235 for determining a distance between data of the image segment 215-1 and data of the reference image segment 235. For example, the processing circuitry may determine a shared information distance, a Kullback-Leibler divergence, or a cross-correlation between information bits of the image segment 215-1 and information bits of the reference image seg ment 235.

The processing circuitry may, for instance, apply a permutation test (e.g., a permutation t-test) to the image segment 215-1 based on the reference image segment 235. The permutation test may be thought of a test for statistical significance of data included in the image segment 215- 1

In other examples, the processing circuitry may determine an earth mover’s distance between the two datasets, the image segment 215-1 and the reference image segment 235. The earth mover’s distance may be thought of as a measure of distance between a distribution of values in the image segment 215-1 and a distribution of values of the reference image segment 235. Informally, if the distributions are interpreted as two different ways of piling up a certain amount of earth (dirt) over a certain area, an earth mover’s distance of said distributions may be thought of a minimum cost for turning one pile into the other, assuming the minimum cost is an amount of dirt which is required to be moved for turning the one pile into the other multiplied by a distance by which the dirt needs to be moved. The processing circuitry may determine the earth mover’s distance based on a distribution of color information and/or in tensity information over the image segment 215-1 and over the reference image segment 235. The distribution of intensity information may, e.g., indicate a texture of surfaces in the corre sponding space segment.

In other examples, the processing circuitry may correlate the image segment 215-1 with sev eral reference image segments representing the space segment corresponding to the image segment 215-1.

In some examples, the processing circuitry may compare the image segment 215-1 with the reference image segment 235 based on a pretrained machine learning model. The pretrained machine learning model may be trained based on reference digital images of the predefined space. For training the machine learning model, a respective predefined entropy score may be allocated to the reference digital images or reference image segments and fed into the machine learning model as desired output.

The machine-learning model is a data structure and/or set of rules representing a statistical model that the processing circuitry may use to perform the above tasks without using explicit instructions, instead relying on models and inference. The data structure and/or set of rules represents learned knowledge (e.g., based on training performed by a machine-learning algo rithm). For example, in machine-learning, instead of a rule-based transformation of data, a transformation of data may be used, that is inferred from an analysis of historical and/or train ing data. In the proposed technique, the content of reference image segments is analyzed using the machine-learning model (i.e., a data structure and/or set of rules representing the model).

The machine-learning model is trained by a machine-learning algorithm. The term “machine learning algorithm” denotes a set of instructions that are used to create, train or use a machine learning model. For the machine-learning model to analyze the content of the reference image segments, the machine-learning model may be trained using training and/or historical (refer ence) image segments as input and training content information (e.g., labels indicating a re spective entropy score) as output. By training the machine-learning model with a large set of training image segments and associated training content information (e.g., labels or annotations), the machine-learning model “learns” to recognize the content of the image seg ments, so, the content of image segments, such as the image segment 215-1, that are not in cluded in the training data can be recognized using the machine-learning model. By training the machine-learning model using training image segments and a desired output, the machine learning model “learns” a transformation between the image segments and the output, which can be used to provide an output based on non-training digital images provided to the ma chine-learning model.

The machine-learning model may be trained using training input data (e.g., training image segments). For example, the machine-learning model may be trained using a training method called “supervised learning”. In supervised learning, the machine-learning model is trained using a plurality of training samples, wherein each sample may comprise a plurality of input data values, and a plurality of desired output values, i.e., each training sample is associated with a desired output value. By specifying both training samples and desired output values, the machine-learning model “learns” which output value to provide based on an input sample that is similar to the samples provided during the training. For example, a training sample may comprise a training digital image as input data and one or more labels as desired output data. The labels indicate the entropy score of the corresponding space segment.

Apart from supervised learning, semi-supervised learning may be used. In semi -supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm or a similarity learning algorithm). Classification algorithms may be used as the desired outputs of the trained machine-learning model are restricted to a limited set of values (categorical variables), i.e., the input is classified to one of the limited set of values (type of exercise, execution quality). Similarity learning algorithms are similar to classification algorithms but are based on learning from examples using a similarity function that measures how similar or related two objects are.

Apart from supervised or semi-supervised learning, unsupervised learning may be used to train the machine-learning model. In unsupervised learning, (only) input data are supplied, and an unsupervised learning algorithm is used to find structure in the input data such as training and/or historical digital images (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data comprising a plurality of input values into subsets (clusters) so that input values within the same cluster are similar according to one or more (predefined) similarity criteria, while being dissimilar to input val ues that are included in other clusters.

Reinforcement learning is a third group of machine-learning algorithms. In other words, re inforcement learning may be used to train the machine-learning model. In reinforcement learning, one or more software actors (called “software agents”) are trained to take actions in an environment. Based on the taken actions, a reward is calculated. Reinforcement learning is based on training the one or more software agents to choose the actions such that the cu mulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards).

Furthermore, additional techniques may be applied to some of the machine-learning algo rithms. For example, feature learning may be used. In other words, the machine-learning model may at least partially be trained using feature learning, and/or the machine-learning algorithm may comprise a feature learning component. Feature learning algorithms, which may be called representation learning algorithms, may preserve the information in their input but also transform it in a way that makes it useful, often as a pre-processing step before per forming classification or predictions. Feature learning may be based on principal components analysis or cluster analysis, for example.

In some examples, anomaly detection (i.e., outlier detection) may be used, which is aimed at providing an identification of input values that raise suspicions by differing significantly from the maj ority of input or training data. In other words, the machine-learning model may at least partially be trained using anomaly detection, and/or the machine-learning algorithm may com prise an anomaly detection component.

In some examples, the machine-learning algorithm may use a decision tree as a predictive model. In other words, the machine-learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input image segments) may be repre sented by the branches of the decision tree, and an output value corresponding to the item may be represented by the leaves of the decision tree. Decision trees support discrete values and continuous values as output values. If discrete values are used, the decision tree may be denoted a classification tree, if continuous values are used, the decision tree may be denoted a regression tree.

Association rules are a further technique that may be used in machine-learning algorithms. In other words, the machine-learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. The machine-learning algorithm may identify and/or utilize one or more relational rules that represent the knowledge that is derived from the data. The rules may, e.g., be used to store, manipulate, or apply the knowledge.

For example, the machine-learning model may be an Artificial Neural Network (ANN). ANNs are systems that are inspired by biological neural networks, such as can be found in a retina or a brain. ANNs comprise a plurality of interconnected nodes and a plurality of con nections, so-called edges, between the nodes. There are usually three types of nodes, input nodes that receive input values (e.g., values of the image segment 215-1), hidden nodes that are (only) connected to other nodes, and output nodes that provide output values (e.g., the entropy score 245). Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g., of the sum of its inputs). The inputs of a node may be used in the function based on a “weight” of the edge or of the node that provides the input. The weight of nodes and/or of edges may be adjusted in the learning process. In other words, the training of an ANN may comprise adjusting the weights of the nodes and/or edges of the ANN, i.e., to achieve a desired output for a given input.

Alternatively, the machine-learning model may be a support vector machine, a random forest model or a gradient boosting model. Support vector machines (i.e., support vector networks) are supervised learning models with associated learning algorithms that may be used to ana lyze data (e.g., in classification or regression analysis). Support vector machines may be trained by providing an input with a plurality of training input values (e.g., reference image segments) that belong to one of several categories (e.g., high entropy score, medium entropy score, low entropy score, etc.). The support vector machine may be trained to assign a new input value to one of the several categories. Alternatively, the machine-learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine-learning model may be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the process of natural selection. In some examples, the machine-learning model may be a combination of the above examples.

Referring back to Fig. 2, as a fourth step 240, the processing circuitry determines an entropy score 245 for the image segment 215-1 based on the correlation between the image segment 215-1 and the reference image segment 235. By way of illustration, the entropy score is illus trated as dot pattern. The processing circuitry may infer the entropy score 245 from the cor relation, e.g., by means of averaging or integrating differences (distances) between the data of the image segment 215-1 and the reference image segment 235. The entropy score 245 may be a real number indicating messiness, a disorder, or an untidiness of the space segment cor responding to the image segment 215-1. Consequently, the entropy score 245 may provide an objective measure for rating a cleanliness of the space segment.

Optionally, the processing circuitry may perform a fifth step 250 which is generating a heatmap 255 for the predefined space. Assuming the processing circuitry has determined, for each of the image segments, a respective entropy score, the heatmap 255 may allocate the respective entropy score to the image segment. The heatmap 255 may be thought of a color coding of the different entropy scores. This may be useful for graphical representation of the entropy scores.

Fig. 3 illustrates an exemplary digital image 300 of a predefined space. The digital image 300 represents a bird’s eye view of a room. The digital image 300 shows some objects in the room which may be relevant for rating the cleanliness of the room: a window (frame) 310, a desk 320, a door (frame) 330, a carpet 340, a bed 350 with two pillows 352-1, 352-2, and a blanket 354, a glass 360 lying on the carpet 340 with a liquid pouring out of the glass 360, a towel 370, a trash bin 380, and dirt 390 on the carpet 340.

Figs. 4a-c illustrate examples of an output 400 of the apparatus 100 when inputting the digital image 300 of Fig. 3 into the apparatus 100. The output 400 may be considered data processed for graphical output, e.g., via a graphical user interface, or data intended for further pro cessing, e.g., for generating cleaning instructions for a cleaning robot or for estimating a cleaning time. The apparatus 100 may be coupled to a display which may show the output 400 for inspection of a human operator. Additionally or alternatively, the apparatus 100 may comprise an interface for sending the output 400 to a user device, i.e., a computer system of a human operator, or a cleaning robot.

In Fig. 4a, the output 400 is a heatmap 410. The heatmap 410 shows a grid of tiles, e.g., 415-

1, 415-2, 415-3, representing image segments of the digital image 300. The tiles 415-1, 415-

2, 415-3 are arranged in the heatmap 410 according to a position of respective space segments of the predefined space, i.e., the heatmap 410 reproduces a spatial distribution of the space segments.

The tiles 415-1, 415-2, 415-3 are colored according to a color code mapping a value range of an entropy score to a specific color or color shade. Darker color shades refer to a lower level of cleanliness. For instance, the tile 415-1 refers to a clean segment of the carpet 340, so, the apparatus 100 has assigned a bright color to the tile 415-1. The tile 415-2 refers to an image segment of image 300 showing the unmade bed 350, which is interpreted as less clean than tile 415-1 by the apparatus 100, thus, the tile 415-2 has a darker color than tile 415-1. The tile 415-3 refers to an image segment of image 300 showing the glass 360 lying on the carpet 340 which is interpreted as less clean than tile 415-2 by the apparatus 100, thus, it has a darker color than tile 415-2.

Similarly, other tiles of the heatmap 410 are colored according to an interpretation of the cleanliness by the apparatus 100, i.e., the apparatus 100 rates a respective entropy score of the tiles based on image analytics applied to image segments of the digital image 300 and assigns a color to the tiles according to the color coding of the heatmap 410. Consequently, tiles rep resenting space segments with the dirt 390 on the floor, the full trash bin 380, and the towel 370 on the floor are colored in different dark color shades.

In other examples, the color coding may be different than the one shown in Fig. 4a. For in stance, the color coding may comprise only two colors, one for a clean state of a respective space segment and one for a not clean state of a respective space segment. An assignment of a value range of the entropy score to the colors may be determined in any suitable manner. For instance, in other examples, darker colors may indicate a clean state and brighter colors may indicate a less clean state of a respective space segment. A scale of the entropy score may be defined in any suitable manner, e.g., a high value of the entropy score may indicate a high level of messiness or, alternatively, a high level of cleanliness. A sensitivity of the appa ratus 100 may be adjusted in any suitable manner, e.g., depending on the application. For instance, for a hotel with high cleaning standards, it may be necessary to adjust the apparatus 100 to rate smaller differences between an actual state and an “ideal” (reference) state of a hotel as messy, whereas in a private home, smaller differences may be tolerated and, therefore, still rated as clean by the apparatus 100.

In Fig. 4b, the output 400 is the heatmap 410 with additional context information 420-1, 420- 2, 420-3, 420-4. The context information 420-1, 420-2, 420-3, 420-4 labels parts of the heatmap 410 by marking a region in the heatmap 410 and classify a type of objects in respec tive space segments of the room. The context information 420-1, 420-2, 420-3, 420-4 labels a region of the heatmap 410 as “pillow, blanket”, “towel on floor”, “dust on carpet”, and “unknown object”, respectively.

The apparatus 100 may generate the additional context information, e.g., only for image seg ments with an entropy score exceeding a predefined threshold. This may be advantageous since computing power may be saved for image segments with an entropy score lower than the predefined threshold. The predefined threshold may be defined in settings of the apparatus 100 and adjusted according to a specific application.

The apparatus 100 generates the context information 420-1, 420-2, 420-3, 420-4 based on any suitable method for object detection and classification. For instance, an input to the apparatus 100 may be a function mapping the image segments to objects in respective space segments. As a location of objects in the space segments may be known for a reference state of the space segments, the mapping may be determined based on the known location of the objects.

Alternatively, the apparatus 100 may detect or classify the objects based on a pretrained ma chine learning model, e.g., a convolutional deep neural network. The pretrained machine learning model may use as input the image segments of the digital image 300 and may output the context information 420-1, 420-2, 420-3, 420-4. The machine learning model may be trained using reference image segments which are already labelled. For instance, a set of clas ses, e.g., towel, trash bin, desk, may be defined and multiple digital images of objects of a respective class may be provided and used as input to the to the machine learning model. Optionally, an existing visual network may be used as machine learning model. An output of the last layer of the visual network may be used as features for classification tasks during training (transfer learning) to accelerate a training process of the machine learning model . The apparatus 100 may also use a machine learning model to classify a material of surfaces shown in the image segments, e.g., wood, textile, plastic, metallic, paper. The apparatus 100 may also count a number of classified objects. Classified objects found in image segments rated as not clean may be defined as objects to be modified, i.e., as objects that need to be cleaned or removed. Additionally, some classified objects may be defined as dangerous, e.g., for an op eration of cleaning robots. In other words, some objects in the predefined space, such as bro ken glass, may possibly break parts of the cleaning robot if the cleaning robot operates in the predefined space.

If an object could not be classified, it may be defined as unknown object, e.g., in the case of the “unknown object” indicated by the context information 420-4. Then, it may be necessary to make sure the unknown object is safe for an operation of cleaning robots. The apparatus 100 may, therefore, comprise an interface to a user device of a human operator and send a labelling request to the user device via the interface. The labelling request may comprise in formation about the unknown object. The user device may eventually reply to the labelling request and send a label back to the apparatus 100 indicating if the unknown object is danger ous or safe. Alternatively, the apparatus 100 may use a pretrained machine learning model to label the unknown object as dangerous or safe.

Labels found for unknown objects may be stored and used to improve an accuracy of the object classification. A machine learning model used to classify the objects may be newly or further trained by adding the labels to a training process for the machine learning model.

In some examples, the apparatus 100 may comprise an interface for exchanging data with a cleaning robot. If an object in the predefined space is classified (labelled) as dangerous, the apparatus 100 may send a blocking instruction via the interface to the cleaning robot. The blocking instruction may cause the cleaning robot to be blocked for an operation in the pre defined space.

In other examples, the apparatus 100 may consider the class, the amount (number), the mate rial (surfaces), or the location of the classified objects for determining the entropy score and update the heatmap 410 accordingly. For instance, liquids that are detected on a carpet may be rated with a higher level of messiness than items on a floor. In other words, the entropy score of image segments may additionally indicate an estimated (time) effort for cleaning corresponding space segments.

Optionally, the apparatus 100 may comprise an interface for exchanging data with an air qual ity sensor installed in the predefined space. The air quality sensor may send data indicating a level of air pollution to the apparatus 100. The apparatus 100 may then consider the level of air pollution for rating the entropy of the predefined space, i.e., for determining the entropy score of the image segments. For example, the entropy score of all image segments may be increased/decreased by a certain value inferred from the level of air pollution.

The output 400 of the apparatus 100 shown in Fig. 4c is a modified version of the digital image 300. Areas of the digital image 300, such as areas 430-1, 430-2, 430-3, 430-4, 430-5, representing objects that are defined as objects to be modified are colored according to a color scheme 440. The color scheme 440 defines three different categories “dangerous”, for all objects labelled as dangerous, “human work zone”, for all objects that shall be cleaned by a human, and “robot work zone”, for all objects that shall be cleaned by a cleaning robot. The area 430-1 is assigned to category “dangerous” since the area 430-1 refers to the (possibly broken) glass 360 which may destroy a cleaning robot driving over the glass 360. The areas 430-2, 430-3, 430-4 are assigned to the category “robot work zone”. The areas 430-2, 430-3, 430-4 refer to the full trash bin 380, the towel 370, the dirt 390 on the floor, respectively. The area 430-5 refers to the unmade bed 350 and is assigned to the category “human work zone”.

Additionally, the apparatus 100 may determine cleaning instructions based on a class, a ma terial, an amount, or a location of the classified objects. For example, the apparatus 100 may load a logical component from a registry. The logical component may map a condition, i.e., certain combination of classes, materials, amounts, and locations of the classified objects, to certain cleaning instructions and work zones. Work zones may be understood as subjects or entities (such as human, cleaning robot, robot hoover, pick-up robot) that shall perform the cleaning instruction. An exemplary logical component is described in table 1. The apparatus 100 may optionally send the cleaning instructions to a user device or a cleaning robot.

Table 1

In other examples, the apparatus 100 may additionally estimate a cleaning time needed to execute the cleaning instructions. The apparatus 100 may infer the cleaning time from the entropy score of the image segments or from a class, a material, an amount, a location of the objects that need to be cleaned. The apparatus 100 may apply a pretrained machine learning model to estimate the cleaning time. The machine learning model may be trained based on historical data indicating a cleaning time needed for priorly performed cleaning steps. For instance, the cleaning time may be tracked by a clock function integrated into a cleaning robot. The tracked cleaning time may be stored in a logfile of the cleaning robot and send to the apparatus 100.

Referring back to Fig. 4c, the output 400 also includes context information 450. The context information 450 indicates a total entropy score which may be an average or aggregated en tropy score of all image segments. The context information 450 further indicates a cleaning time estimated for human cleaning personnel. Furthermore, the context information 450 indi cates if a detected object is labelled as dangerous.

Fig. 5 illustrates an example of a system 500 for rating an entropy of a predefined space. The system 500 includes an image sensor, such as image sensor 130, configured to capture a dig- ital image, such as the digital image 120, of the predefined space. The system 500 further includes an apparatus for rating the entropy of the predefined space, such as the apparatus 100. Optionally, the image sensor 130 may be integrated into a drone 510. The drone 510 is an unmanned aircraft which is, e.g., either controlled via a ground-based controller or operates at least partially autonomous. If the apparatus 100 and the drone 510 are not integrated into one device, the drone 510 may require a wireless communication module to transmit the dig ital image 120 from the image sensor 130 to the apparatus 100.

The system 500 may optionally include an air quality sensor 520 for sensing a level of air pollution and/or a level of oxygen in the predefined space. If the air quality sensor 520 and the apparatus 100 are not integrated into the one device, the air quality sensor 520 may require an interface for sending the level of air pollution and/or the level of oxygen to the apparatus 100. Optionally, the system 500 may include a sensor (not shown) for rating a sterility of surfaces in the predefined space, e.g., based on hyperspectral imaging.

The system 500 may optionally include a cleaning robot 530 for cleaning the predefined space. The apparatus 100 may send cleaning instructions to the cleaning robot 530. The clean ing robot 530 may clean the predefined space in accordance with the cleaning instructions.

The system 500 may be used for estimating a cleaning effort and a type of needed cleaning steps in a hotel. For instance, the drone 510 may autonomously fly into hotel rooms that need to be cleaned. The drone 510 may include the image sensor 130. The drone 510 may take images of the hotel rooms from certain positions and with certain camera angles, e.g., the drone 510 may be temporarily held by a magnet on a ceiling of the hotel rooms to be stabilized for taking the images and to be fixed in a defined position in each hotel room, thus, the images show a similar view of the hotel room each time the drone 510 takes images. The apparatus 100 may then process the images and rate the entropy of the hotel rooms. Additionally, the apparatus 100 may classify objects to be modified in the predefined space based on the im ages. The apparatus 100 may determine a cleaning time for cleaning the hotel rooms, i.e., the apparatus 100 estimates a time needed by human cleaning personnel or cleaning robots to restore reference conditions of the hotel rooms. The apparatus 100 may determine cleaning instructions indicating cleaning steps required for cleaning the hotel rooms. The apparatus 100 may generate a workflow for human cleaning personnel and cleaning robots and assign them the cleaning instructions for a certain work shift.

For instance, if the hotel was fully booked on one day and, the day after, a specific number of rooms may be needed for new guests within a given time window. A drone may be used to take pictures of the rooms. Based on the pictures, a “messiness detector”, such as apparatus 100, may determine an entropy score to indicate which rooms could be cleaned with less effort (lower entropy score). The messiness detector may coordinate housekeeping persons accord ingly (provide cleaning instructions). The messiness detector may also consider difficult cleaning conditions, e.g., cleaning a carpet from red wine may be estimated as more time consuming than cleaning a bathroom. Cleaning steps which can be done by a robot or drone may be assessed independently of cleaning steps which need to be done by a human. The messiness detector may calculate an entropy score for the room and provide a heatmap mark ing regions of the room which are to be cleaned to get a lower entropy score. The messiness detector may also be used to coordinate parallel workflows between machines and humans such that they do not interfere. However, if there are dangerous objects detected in a room, the messiness detector may block a cleaning robot from entering the room until the danger is removed.

Apparatuses, systems, or methods disclosed herein may be used for work coordination be tween human cleaning personnel and cleaning robots. This may be advantageous since it is based on objective measures for a level of messiness in the rooms. Consequently, the work coordination may lead to time efficient and safe cleaning processes. The objective measure may also enable a better quality control in a hotel. An operation time of cleaning robots may be optimized based on the objective measures and, thus, the system 500 may help saving energy costs.

Apparatuses, systems, or methods disclosed herein may be used for educational purposes. For instance, a gaming application may use the system 500 for rating a cleanliness of a child’s room and provide incentives (points) to the child for cleaning the room. The system 500 may determine in which parts of the room the entropy is high and which actions may decrease the entropy most efficiently (and give most points). In other examples, cleaning personnel may be educated based on cleaning instructions of the system 500. This may especially be advan tageous for cleaning personnel in hotels since the cleaning personnel may need to find a bal ance between needed cleaning level and privacy of hotel guests (i.e., objects of the hotel guests shall be touched as little as possible). The cleaning personnel may be guided by clean ing instructions of the system 500. The cleaning instructions may be visualized by using vir tual reality or augmented reality applications. Entropy rating as described herein may also be used for smart cleaning, e.g., a cleaning robot may detect automatically when an entropy of a room is high (a level of dust on the floor or in the air is high) and start cleaning when a predefined threshold is exceeded.

In other examples, the apparatus 100 may also be used for detecting mental health problems of elderly people since an increase of an average entropy in a private home of the elderly people may be taken as indicator for dementia, for example.

The skilled person having benefit from the present disclosure will appreciate that the entropy rating techniques disclosed herein may also be useful for determining a measure of disorder in other systems. For example, the disclosed concept may be useful for classifying a measure of disorder in a surrounding of a vehicle in autonomous driving applications. The disclosed concept may also be useful for structure analysis of microscopic images, e.g., in medical ap plications.

Fig. 6 illustrates an example of a method 600 for rating an entropy of a predefined space. The method 600 comprises dividing 610 a digital image, such as digital image 120, into one or more image segments corresponding to one or more space segments of the predefined space. The method 600 further comprises correlating 620 the image segments with corresponding reference image segments, wherein the reference image segments relate to a predefined en tropy score of the respective space segments. The method 600 further comprises determining 630 an entropy score for the image segments based on the correlation.

The method 600 may provide an objective measure, the entropy score, for a level of messiness in a room or other predefined space. The objective measure may allow an efficient coordina tion of cleaning personnel in hotels or hospitals, for instance.

More details and aspects of the method 600 are explained in connection with the proposed technique or one or more examples described above. The method 600 may comprise one or more additional optional features corresponding to one or more aspects of the proposed tech nique, or one or more examples described above.

Note that the present technology can also be configured as described below: (1) Apparatus for rating an entropy of a predefined space, comprising: processing circuitry configured to: acquire at least one digital image of the predefined space from an image sensor; divide the digital image into one or more image segments corresponding to one or more space segments of the predefined space; correlate the image segments with corresponding reference image segments, wherein the reference image segments relate to a predefined entropy score of the respective space segments; and determine an entropy score for the image segments based on the correlation.

(2) Apparatus of (1), wherein the processing circuitry is further configured to filter a brightness and/or contrast of the image segments to adapt lighting conditions of the image segments to reference lighting conditions of the reference image segments.

(3) Apparatus of (1) or (2), wherein the entropy score indicates a tidiness of the predefined space.

(4) Apparatus of any one of (1) to (3), wherein the processing circuitry is configured to determine the entropy score for the image segments based on an earth mover’s distance between the image seg ments and the corresponding reference image segments.

(5) Apparatus of any one of (1) to (4), wherein the processing circuitry is configured to determine the entropy score for the image segments based on a pretrained machine learning model, and wherein the pretrained machine learning model is trained based on the reference image segments and the respective predefined entropy score.

(6) Apparatus of any one of (1) to (5), wherein the processing circuit is further configured to generate a heatmap for the predefined space indicating respective entropy scores of the image segments.

(7) Apparatus of any one of (1) to (6), wherein the processing circuitry is further configured to, if the entropy score of at least one of the image segments exceeds a predefined threshold, detect and/or clas sify objects to be modified in the predefined space based on the digital image.

(8) Apparatus of (7), wherein the processing circuitry is configured to detect and/or classify the objects to be modified by determining at least one of a class, a material, an amount, a location of the objects to be modified based on a pretrained machine learning model applied to the digital image.

(9) Apparatus of (7), wherein the processing circuitry is configured to detect and/or classify the objects to be modified by: loading, from a data storage, a predefined map assigning at least one of the image segments to at least one predefined object in a respective space segment; and if the entropy score of the at least one of the image segments exceeds a prede fined threshold, define a respective predefined object as one of the objects to be modified.

(10) Apparatus of (6) and any one of (7) to (9), wherein the processing circuitry is further configured to update the heatmap for the predefined space by labelling the heatmap with at least one of a class, a mate rial, an amount of the objects to be modified and/or by marking a location of the objects to be modified in the heatmap.

(11) Apparatus of any one of (7) to (10), further comprising: interface circuitry configured to: if the object to be modified is classified as unknown, send a labelling request to a user device; and receive a label indicating if the unknown object is dangerous or safe from the user device.

(12) Apparatus of any one of (7) to (10), wherein the processing circuit is further con figured to: if the object to be modified is classified as unknown, apply a pretrained classifier to the respective image segment to label the unknown object as dangerous or safe.

(13) Apparatus of (11) or (12), further comprising: interface circuitry configured to, if the unknown object is labelled as dangerous, send a blocking instruction to a cleaning robot for blocking the cleaning robot to operate in the predefined space.

(14) Apparatus of any one of (1) to (13), wherein the processing circuitry is further configured to: load a predefined set of cleaning instructions from a data storage; and select a cleaning instruction from the predefined set of cleaning instructions based on at least one of a class, a material, an amount, a location of objects to be modified in the predefined space.

(15) Apparatus of (14), further comprising: interface circuitry configured to transmit the selected cleaning instruction or por tions thereof to a cleaning robot and/or a user device.

(16) Apparatus of any one of (1) to (15), wherein the processing circuitry is further configured to estimate an amount of dust in the predefined space based on the image segments.

(17) Apparatus of any one of (1) to (16), wherein the processing circuitry is further configured to estimate a cleaning time based on the entropy score of the image segments and/or a class, a material, an amount, a location of objects to be modified in the predefined space.

(18) Apparatus of any one of (1) to (17), wherein the processing circuitry is configured to estimate a cleaning time by using a pretrained machine learning model, wherein the pretrained machine learning model is trained based on a logfile of a cleaning robot and/or a user device, wherein the logfile comprises a prior needed cleaning time.

(19) Apparatus of any one of (1) to (18), wherein the processing circuit is further configured to: acquire a level of air pollution in the predefined space from an air quality sensor; and determine the entropy score based on the level of air pollution.

(20) System for rating an entropy of a predefined space, comprising: an image sensor configured to capture a digital image of the predefined space; and an apparatus of any one of (1) to (19).

(21) System of (20), further comprising: a drone comprising the image sensor; and a wireless communication module configured to transmit the digital image from the image sensor to the apparatus.

(22) System of (20) or (21), further comprising: an air quality sensor configured to capture a level of air pollution in the predefined space; and interface circuitry configured to send the level of air pollution to the apparatus.

(23) System of any one of (20) to (22), further comprising: a cleaning robot configured to clean the predefined space according to a cleaning instruction of the apparatus.

(24) Method for rating an entropy of a predefined space, comprising: dividing a digital image into one or more image segments corresponding to one or more space segments of the predefined space; correlating the image segments with corresponding reference image segments, wherein the reference image segments relate to a predefined entropy score of the respective space segments; and determining an entropy score for the image segments based on the correlation.

(25) A program having a program code for performing the method of (24), when the program is executed on a processor or a programmable hardware.

The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

Examples may further be or relate to a (computer) program including a program code to exe cute one or more of the above methods when the program is executed on a computer, proces sor, or other programmable hardware component. Thus, steps, operations, or processes of different ones of the methods described above may also be executed by programmed comput ers, processors, or other programmable hardware components. Examples may also cover pro gram storage devices, such as digital data storage media, which are machine-, processor- or computer-readable and encode and/or contain machine-executable, processor-executable or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLAs), (field) programmable gate arrays ((F)PGAs), graphics processor units (GPU), application-specific integrated circuits (ASICs), integrated circuits (ICs) or system-on-a-chip (SoCs) systems programmed to execute the steps of the methods described above.

It is further understood that the disclosure of several steps, processes, operations, or functions disclosed in the description or claims shall not be construed to imply that these operations are necessarily dependent on the order described, unless explicitly stated in the individual case or necessary for technical reasons. Therefore, the previous description does not limit the execu tion of several steps or functions to a certain order. Furthermore, in further examples, a single step, function, process, or operation may include and/or be broken up into several sub -steps, -functions, -processes or -operations.

The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Further more, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.