Method and System for Finding Objects

The present invention is related to a method of and a system for finding objects, in particular a spatial memorisation method and a spatial memorisation system (SMS) for finding tools within a workspace.

In industrial environments many scenarios exist, where sever al employees work in shifts at the very same workplace, i.e., one role (e.g. production line mechanic) is staffed by sever al people in turn (e.g. Alice, Bob, Charly) . This applies to very different workplaces, from work at a machine tool to managing a process line in the metal production and even work at counters/desks. The problem is that while the role is act ing continuously the people involved are changing, hence, things that a person has done or rather the locations objects needed for the work (tools, items, parts, etc.) have been placed at are not known to subsequent people acting in the same role. In other words the role does not have a short time memory. The effect is similar to certain mental diseases where someone can't remember things that happened a short while ago.

It is general practice to use standardised locations for com monly used tools, e.g. tool drawers with predetermined loca tions for tools (spanners, hammers, screw drivers, bottle openers, etc.) . However, the problem is the high effort of always putting all used objects back to the respective stand ardised locations. Further, this approach is not very practi cal for non-standard items. Also handovers talks once the next person takes over the role are general practice. Here the problem is also the high effort and loss of time. Fur ther, only a limited scope of items can be addressed in the handover talks.

Augmented reality (AR) devices (e.g. HoloLens device) can provide to a user of the AR device information about his en vironment like indicating objects or structures (e.g. a route through a building) . For example AR glasses (e.g. Google Glass) are known that can indicate additional information in the real environment for a user wearing the AR glasses by projecting the information into the field of view of the user at a specific location.

It is an object of the present invention to solve or at least alleviate the problems of the state of the art as above indi cated and to enable easy finding of needed objects.

Therefore the present invention provides a method of finding objects according to independent claim 1 and a corresponding system for finding objects according to the further independ ent claim. Embodiments and refinements of the present inven tion are subject of the corresponding dependent claims.

According to a first aspect of the present invention a method of finding objects, in particular a spatial memorisation method, comprises the steps of:

Tracking the locations of known objects in a predefined space by an optical tracking system based on specific appear ances and/or specific features of the known objects. Corre sponding locations of virtual objects, which are digital twins of the known objects, are memorised in a virtual space, which comprises a digital twin of the predefined space. Indicating the location of at least one known object in the predefined space to a user by an Augmented Reality (AR) device based on the corresponding memorised location of the respective at least one virtual object.

According to a second aspect of the present invention a sys tem for finding objects, in particular a spatial memorisation system (SMS) , is arranged and configured to execute the meth od according to the first aspect of the present invention and comprises an optical tracking system and an AR device. The optical tracking system is arranged and configured to track the locations of known objects in a predefined space based on specific appearances and/or specific features of the known objects. Corresponding locations of virtual objects, which are digital twins of the known objects, are memorised in a virtual space, which is a digital twin of the predefined space. The AR device is communicatively connected with the optical tracking system. The AR device is arranged and con figured to indicate the location of at least one known object in the predefined space to a user wearing the AR device based on the corresponding memorised location of the respective at least one virtual object.

With the method of and system for finding objects information about the locations of the known objects (e.g. necessary tools) is provided. The known objects each have a unique tag (e.g. a specific name like "hammer", "spanner", "wrench" etc. or identifier (ID) like "Tl", "T2", ..., "PI", "P2", ...) . A us er, in particular a user within a specific role (e.g. a tech nician, mechanic, etc.), bears the AR device. The optical tracking device keeps track of objects, in particular of ob jects relevant to an operational domain of the user. The optical tracking system or rather its components may be stationary and/or wearable. The optical tracking system rec ognises the known objects within a field of view of the opti cal tracking system, in particular of an optical device (e.g. camera) of the optical tracking system. The location of the known objects in the predefined space is transferred into the virtual space, where the respective virtual objects are memo rized at the corresponding location in the virtual space. Thereby, the position and/or orientation of the user in the predefined space may be actively determined (e.g. user is recognised in the predefined space for example by GPS, trian gulation or another position determination method) or pas sively derived (e.g. based on the field of view of the images recorded by the optical tracking device, the posi

tion/orientation of the user is derived) .

For example, when the optical tracking device recognises a first known object at a first location in the predefined space (e.g. a wrench on a desk) then the respective first virtual object (corresponding to the first known object) is memorised at a corresponding first location in the virtual space (e.g. the "virtual wrench" is memorised on the "virtual desk") . If the first known object is used by the user and then put at a second location (e.g. on a drawer) in the pre defined space, then the first known object is recognised by the optical tracking system at the second location (e.g.

wrench on the drawer) and the respective first virtual object is memorised at a corresponding second location in the virtu al space (e.g. the "virtual wrench" is memorised on the "vir tual drawer") . In other words, a digital twin of a predefined space like a workplace and the known objects therein like tools (aka its inventory) are set up over time.

The AR device may be a device for optically or acoustically indicating the location of at least one known object to the user. The AR device may be AR glasses like the Google Glass. The location of at least one of the known objects is indicat ed to the user by the AR device in order to find the at least one known object. For example, the last location of the first object or rather the last virtual location of the first vir tual object memorised is the second (virtual) location. Thus, the second (real) location is indicated to the user as the location of the first known object (e.g. the wrench is indi cated on the drawer) and the user can easily find the first known object in the predefined space (e.g. find the wrench in the workplace on the drawer) .

Further, an overview of all known objects being tracked with in a certain range in the predefined space may be provided. This overview may be provided either upon request of the user or automatically, when a new user starts to use the system for finding objects.

The AR device supports the retrieval of the known objects based on the information of the optical tracking device at a later point in time (typically for the next person executing the role) . Further, once the role of a mechanic is taken over by another person, an overview of all tracked known objects within certain range may be provided. Also more than one user can track and find (getting indicat ed) known objects in the same predefined space at the same time .

More specifically, when a person starting his/her shift as mechanic for example of a production line, the employee puts on the AR device and optionally the optical device of or the whole optical tracking system (or rather the SMS) . Then the person (user of the system for finding objects / SMS) starts working. While the user does his work, the optical tracking system is scanning the environment for known things (it is trained to recognize). Recognized known objects (specific ap pearance / features thereof) and the location they were last seen at in the predefined space (e.g. workplace) are then memorised (stored in a database like a server) . Thereby, the corresponding locations of the respective virtual items are stored in the virtual space, or, from a wider point of view, a relevant inventory of the workplace of the mechanic is tracked and indicated to the user. When a known object is not in its normal place, the system for finding objects / SMS can be asked (e.g. by voice) where it was last seen.

Thus, the method of and system for finding objects provide an external memory for the user (e.g. a forgetful person) and in particular for a role in question (e.g. the role of a mechan ic) . For example, in the case of a mechanic, the locations of typical tools of-the-trade (e.g. spanners, hammers, wrenches etc.) in the vicinity of the mechanic are automatically rec ognised and kept track of as well as later indicated to the current user in the role of the mechanic.

Also visually impaired persons or persons with Alzheimer's disease can easily find "lost" / "invisible" / "forgotten" things. Also people in a warehouse can easily find the place of needed things / inventory.

In an industrial scenario time can be saved by not searching, but finding things (known objects) immediately for example by just asking where to find them. Thus, time for employees is saved and costs for employers are reduced.

According to a refinement of the present invention the loca tion of at least one object is indicated in response to a re quest for finding the at least one object.

Only upon the request to find one ore more of the known ob jects in the predefined space the locations of the requested one or more known objects is indicated to the user based on the corresponding memorized locations of the respective vir tual objects in the virtual space such that he can easily find them in the (real) predefined space. For example, upon request to find a hammer and a wrench (known objects) in the workplace (predefined space) the memorized locations of the "virtual hammer" and the "virtual wrench" in the "virtual workplace" is used to indicate the locations of the (real) hammer and the (real) wrench in the (real) workplace to the user .

By indication the locations of known objects only upon re quest only known objects actually relevant to the user are indicated. Thus, the user is provided only with currently relevant information about the locations of (currently rele vant) known objects such that he can find the relevant known objects more easily. According to a further refinement of the present invention the request for finding the at least one object is a request for finding a predefined group of objects.

Multiple known objects can be defined as one group of ob jects. For example a set of tools needed for a certain task (e.g. changing a certain machine part of a production line like a mask of a printing machine) can be defined as group of tools for this task (e.g. "tools for changing the mask") . Thus, with only one request all relevant known objects can be found .

The finding of currently relevant known objects is simplified and sped up by requesting to find the respective predefined group of known objects.

According to a further refinement of the present invention the request for finding the at least one object is issued by the user. Alternatively or additionally the request for find ing the at least one object is automatically generated based on a predefined schedule. Alternatively or additionally the request for finding the at least one object is automatically derived from an automatically recognised situation and/or state of the predefined space.

The user may request to find one or more known objects or a predefined group of objects. Thereto the user may issue the request via a Human Interface Device (HID) like a keyboard, a mouse, a touchscreen, a microphone and the like.

By direct indication of the requested known object, the user can find a needed known object in the predefined space very fast . The request to find one or more known objects or a predefined group of objects may also be automatically issued based on the predefined schedule. For example at the beginning of a shift at a production line certain machine parts might always have to be changed. Based on this predefined schedule, the locations of the tools of the predefined group of tools need ed to change the respective machine parts at the beginning of the shift are automatically indicated to the user.

Thus, the user can immediately find the currently relevant known objects right at the time they are needed according to the predefined schedule.

Further, upon a semantic understanding of the situation and/or rather state of the predefined space (e.g. by recog nizing typical work procedures: "for disassembling this ma chine, spanner A is needed and wrench X) the locations of the relevant known objects are indicated to the user (e.g. span ner A and wrench X are indicated) .

Thereto, an Artificial Intelligence (AI) system like a Neu ronal Network (NN) trained to recognise the current situation and/or state of the predefined space can be used to issue the correct request for finding the respective known objects.

With the automatic recognition of the situation / state of the predefined space only the locations of currently relevant known objects are indicated to the user without having to is sue a request "manually" at any time.

According to a refinement of the present invention the method further comprises the initial steps of: Defining the predefined space, wherein borders of the predefined space are defined.

Generating the digital twin of the predefined space, wherein the borders of the predefined space and structures in the predefined space are identified in the three dimensional, 3D, domain and stored as the digital twin of the predefined space .

The predefined space may be a confined space like a workspace (e.g. of a mechanic for a production line), a certain room or building, a specific area (e.g. a premise of a company) etc. The predefined space is defined by defining its borders in the 3D domain. For example, a workspace is defined as space of 2m [Meter] length by 3m width by 2m height. Further, the location of the predefined space in a global coordinate sys tem may be defined. For example, the predefined space is lo cated at 10m in x direction, 2m in y direction and 0m in z direction .

After the predefined space has been defined, its borders and the structures inside are identified. The borders may be ac tual or virtual walls and stored as virtual borders of the digital twin of the predefined space. The structures inside the predefined space (e.g. desk, drawer, machine, etc.) are also stored as virtual structures in the digital twin. The digital twin thus resembles the predefined space as virtual space with virtual borders according to the defined and iden tified borders and with virtual structures inside according to the identified and stored structures inside the predefined space . The predefined space and the corresponding virtual space / digital twin can be any space in the real world. Thus, any known object anywhere in the real world can be found with the method of and system for finding objects after the predefined space and corresponding virtual space / digital twin have been defined and generated.

According to a refinement of the present invention the method further comprises the steps of:

Learning an object to be tracked as known object, where in the user presents the object to be tracked to the optical tracking system for memorising the specific appearance and/or specific features of the object to be tracked and provides a specific tag for the object to be tracked.

Generating the digital twin of the learned object to be tracked, wherein the recognised specific appearance and/or specific features of the object to be tracked are stored.

An object which should be found in the predefined space by indicating it to the user must me made a known object. Such object may be learned as new / additional known object in that the user presents the respective object to the optical tracking system, in particular to the optical device (e.g. camera) of the optical tracking system. The optical tracking system recognises the specific appearance and/or specific features of the presented object (e.g. the form, size, etc. of a new tool) . The object may be presented in different di rections to the optical tracking system for recognising the specific appearance and/or specific features from different angles such that its 3D form can be determined. The specific appearance and/or specific features of the object are memo- rised and thus the object becomes a learned object. The user also provides a tag (e.g. "tool X") for the learned object such that it can be uniquely identified.

Based on the memorised specific appearance and/or features of the learned object the corresponding digital twin is generat ed by storing the respective specific appearance and/or fea tures for the digital twin / virtual object of the learned object and the provided tag.

After the predefined space has been defined and its digital twin / virtual space has been generated a "basic set" of ob jects (e.g. standard tools like hammer, wrench X, wrench Y wrench Z, spanner A, spanner B etc.) can be predefined by presenting the objects of the "basic set" one after another to the optical tracking system and providing respective tags. Alternatively a predefined "basic set" of objects may be cop ied to the optical tracking system without having to present each single object of the "basic set" to the optical tracking system.

During operation new objects that should be tracked and then indicated can be learned. For example in case of items that are not domain-specific (e.g. a suitcase with special equip ment, an unusual spare part, etc.) the user may tag the newly recognised item (after having presented it to the optical tracking system) and add it to the scope of tracked items as new known object.

By using an AI system like a NN trained to recognize and tell the name / tag of any item brought into the field of view of the optical tracking system, the addition of new objects to be tracked as known objects can be significantly sped up. As any object may be added as known object, the user can find any object he is searching once this object has been learned as known object. Thus finding objects for example in a work place is significantly sped up even if a new user searches a known object, as all relevant objects are tracked and can be indicated .

According to a refinement of the present invention the prede fined space is a workspace of the user.

The workspace may be a workspace of a mechanic at a produc tion line and the like.

With the method of and system for finding objects, the loca tion of objects like tools, parts and the like at an work place is always "known" such that if a new user works at the work place he/she can find all known objects immediately as the locations of these objects are indicated to the new user (upon (automatic) request) .

According to a refinement of the present invention the known objects comprise at least one tool.

Especially in a working environment, where tools are used and not put back to a specific place the search for the tools may be tedious and time consuming. By tracking and indicating (upon (automatic) request) the location of the tools to any user working at the predefined space (workspace) no time for searching tools is wasted any more.

According to a refinement of the present invention the opti cal tracking system comprises an optical device, a recogni tion unit and a storage. The optical device is preferably a camera and more preferably a stereo camera. The optical de vice arranged and configured to generate images of the prede fined space. The recognition unit is arranged and configured to recognise specific appearances and/or specific features of objects in the generated images and to recognise known ob jects based on comparison with the specific appearances and/or specific features of the virtual objects. The recogni tion device is further arranged to provide the location of the recognised known objects in the predefined space by providing the corresponding location of the respective virtu al objects in the virtual space. The storage is arranged and configured to store the virtual objects and the virtual space as well as the provided locations of the virtual objects in the virtual space.

The optical device may be stationary (e.g. surveillance cam era (s) ) or wearable (e.g. bodycam) . The optical device may be a camera or a stereo camera. The optical device may comprise more than one optical device (e.g. several stationary cameras and several wearable cameras) . The optical device (s) gener ate (s) images of the predefined space in a certain field of view of the optical device (s) . The generated images are pro vided to the recognition unit.

The recognition unit can be implemented by computer (e.g. smartphone, PC, laptop, tablet, server, cloud system etc.). The recognition unit recognises the specific appearance and/or features of objects in the images or rather field of view of the optical device. When the recognised specific ap pearance and/or specific feature of an object are equal to one of those stored in one of the digital twins / virtual ob jects the corresponding known object is identified by the recognition unit. In case the recognised specific appearance and/or features are not equal to one of those stored in the digital twins /virtual objects, then the object is not a known object. Such object not yet known may be learned as new known object as described above (providing a tag for the new learned object and storing the recognised specific appearance and/or specific features in the new digital twin / virtual object resembling the new learned object) .

The recognition unit may comprise an AI system like a NN as described above, which is trained for automatically recognis ing objects (e.g. tools, parts, etc.) brought into the field of view of the optical device.

The storage may be a database implemented by a computer

(Smartphone, laptop, tablet, PC, server, cloud system etc.). The computer implementing the storage may be the same as the computer implementing the recognition unit. In the storage the digital twins of the known objects / the virtual objects with the memorised specific appearances and/or specific fea tures are stored. Further, the digital twin of the predefined space / the virtual space with the identified borders and structures is stored.

Further, the system for finding objects may comprise a local ization unit (e.g. GPS unit) for determining the position and/or orientation of the user as described above.

The system for finding objects or the SMS, respectively, may for example comprise a wearable computer (i.e. a smartphone) implementing the recognition unit and the storage and an at tached wearable camera (i.e. a bodycam) . The camera / bodycam can be worn on the shoulder or as glasses (i.e. Google Glass) or the like. With the optical recognition system objects in the predeter mined space and within the field of view of the optical de vice can be identified as known objects and the location of the known objects can be tracked and later indicated to the user (upon (automatic) request) .

The optical tracking system with the camera device, the recognition unit and the storage as well as optionally with the localisation unit provides for fast and reliable recogni tion and tracking of the known objects as well as for easy learning of new objects to be tracked as new known objects.

According to a refinement of the present invention the system for finding objects further comprises an input device. The input device is preferably a microphone device worn by the user and more preferably a microphone integrated into the AR device. The input device is communicatively connected to the optical tracking system and arranged and configured to input the request of the user to the optical tracking system.

The input device (e.g. keyboard, mouse, touchscreen, or any other HID) or rather the (integrated) microphone worn by the user (e.g. on the head as headset) records the requests of the user (in natural language) like "where is wrench X?" or "where are the tools needed for disassembling the pump?" and the like. Thereby, the requests may have a predefined pattern like "where is [tag of the item to be indicated]?".

As described above, an AI system like a NN trained for recog nising the requests of the user (in natural language) can be utilised such that the requests can have any pattern. For example a headset is used to communicate with the system for finding objects / the SMS by voice. The recognition unit of the optical tracking system is connected to a server or a cloud wirelessly, where the request is identified and the re spective known object is indicated to the user.

The input device and especially the microphone enables fast and easy input of requests by the user. Further, in case of the microphone the user can issue a request without stopping his/her current activity.

According to a further refinement of the present invention the recognition unit and/or the storage of the optical track ing system is comprised by a stationary computer system or by a cloud system.

For fast communication between the recognition unit and the storage storing the digital twins / virtual objects with the memorised specific appearances / specific features of the known objects the storage may be implemented by the same com puter as the recognition unit.

Thus, the tracking and indicating of the known objects in the predefined space can be executed in short time or rather real time such that the user does not have to stop and wait for the indication of the location (s) of the (requested) known obj ect ( s ) .

Alternatively, the recognition unit and/ or the storage may be implemented by a cloud system such that the storage is a highly available repository for several independent systems for finding objects and/or the recognition unit can be shared by several independent systems for finding objects. For example, as all memorisations of virtual objects and their locations may be stored in the cloud, this will work continuously for all employees of a workplace connected to the system for finding objects / SMS.

The implementation in the cloud system thus provides for high availability of the system for finding objects.

According to a further refinement of the present invention the recognition unit and/or the optical device and/or the storage of the optical tracking system and/or the input de vice is comprised by a portable computer system, preferably a smartphone .

With several components of the system for finding objects / SMS comprised in one portable device / smartphone a compact and easy to use system is provided.

According to a further refinement of the present invention the optical device and optionally the recognition unit and/or the storage of the optical tracking system are integrated in to the AR device.

With several or all components of the system for finding ob jects / SMS comprised in the AR device a very compact and very easy to use system is provided.

According to a further refinement of the present invention the recognition unit of the optical tracking system comprises a Neuronal Network (NN) . The NN is trained for recognising the specific appearances and/or specific features of objects in the predefined space in the images provided by the optical device of the optical tracking system. As described above, the recognition unit comprises the NN trained for automatically recognising the specific appearanc es / specific features of the objects in the field of view of the optical device. Thus, standard objects like standard tools can be automatically identified and tracked. Further, the class of an object can be automatically detected and thus learning of new objects to be tracked as new known objects can be simplified. Additionally, the recognition of already known objects can be improved with the additional information provided by the NN.

According to a refinement of the present invention the AR de vice comprises a speaker and/or AR glasses. The speaker is preferably a wearable speaker (e.g. headset) . The speaker is arranged and configured to acoustically indicate the location of the at least one known object. The AR glasses are arranged and configured to optically indicate the location of the at least one known object by projecting the location of the at least one object into a field of view of the user based on the corresponding location of the respective virtual object in the virtual space.

The speaker indicates the ( (automatically) requested) loca tion of the at least one known object in natural language to the user. For example, the location of a requested tool (e.g. hammer) is indicated with the phrase "The hammer is on the desk" .

Additionally, the location of an object may be indicated by "hot" or "cold" depending on the current location and/or ori entation of the user in the predefined space in order to nav- igate to places where the requested objects are located that can't be seen from the current position of the user.

The AR glasses project the ((automatically requested) objects into the field of view of the user. Thereto the memorised corresponding location of the respective virtual object in the virtual space is transformed into the location (e.g. 3D coordinates) in the real predefined space. This location in the real predefined space is optically indicated to the user with the AR glasses, for example, with arrows projected into the field of view of the user.

Additionally, the user can be navigated to locations current ly not visible in his field of view by pointers like blinking arrows projected at the border of his field of view.

With the acoustical or visual indication of the ( (automati cally) requested) location of at least one known object, the user does not have to stop his current work for searching said object but immediately knows the current location of it.

The present invention is subsequently described in more de tail by means of distinct embodiments. These embodiments only confer better understanding of the present invention and are not to be construed limiting for the scope of the present in vention. The all above and subsequently described features may be combined in any meaningful way, except a combination is explicitly described as impossible. In the following same or analogous elements are labelled with the same reference signs . Fig. 1 shows a schematic flow chart of the method of find ing objects according to the first aspect of the present in vention .

Fig. 2 shows a schematic view of the system for finding objects according to the second aspect of the present inven tion .

Fig. 3 shows a schematic view of a stationary camera.

Fig. 4 shows a schematic view of a bodycam.

Fig. 5 shows a schematic view of an AR device.

In Fig. 1 the method of finding objects is schematically de picted. The method comprises the steps of defining 1 the pre defined space, generating 2 the digital twin of the prede fined space, learning 3 an object to be tracked, generating 4 the digital twin of the learned object, tracking 5 the loca tions of known objects and indicating 6 the location of at least one known object.

Initially the steps 1 and 2 are executed. First, in the step of defining 1 a predefined space is defined. For example a workspace of a mechanic at a production line is defined as the predefined space. Then in the step of generating the dig ital twin of the predefined space the borders of and struc tures inside the predefined space are identified. For example a drawer, a desk and a machine of the production line are identified besides the borders. The identified borders and structures are stored as digital twin of the predefined space (i.e. a virtual space) . In the step of learning 3 an object to be tracked and the subsequent step of generating 4 the digital twin of the learned object, first a new object like a tool, item, part etc. that should be tracked is presented to an optical track ing system. The optical tracking system recognises the spe cific appearance and/or specific features of the new object and memorises them. Further a specific tag (e.g. "hammer", "wrench X", "spanner A" etc.) for the new object is provided. In the subsequent step of generating the digital twin of the learned object is generated. Thereby, the memorised specific appearance and/or specific features are stored together with the respective specific tag in the digital twin of the learned or rather new known object (i.e. virtual object) .

During normal operation the known objects in the predefined space are recognised by the optical tracking system in the step of tracking 5. Thereby, the recognised specific appear ances and/or specific features of objects in the predefined space are compared with the specific appearances and/or spe cific features of the stored virtual objects / digital twins of the known objects. If the specific appearance and/or spe cific features of an object are equal to those of a stored virtual object, the respective known object is recognised.

The location in the predefined space of the recognised known object is stored as corresponding location of the virtual ob ject in the virtual space.

If an object in the predefined space is not recognised as known object and the user wants to track this new object, learning the new object is selected (N: yes) and the steps 3 and 4 are executed for the respective new object. If the user does not want to track the not recognised object (N: no) then this object is disregarded and the step of tracking 5 contin ues .

During the continuous tracking 5 the user may request (R: yes) to find a specific known object. This can be done by a request in natural language (e.g. "Where is the wrench X?") . An Artificial Intelligence (AI) system like a Neuronal Net work (NN) trained for recognising requests may be used to recognise the request. Alternatively or additionally, the re quest may be issued automatically (R: yes) either based on a schedule and/or on automatic recognition of a situation and/or state of the predefined space. When such (automatic) request is issued, the location of the requested object is indicated to the user. For example, the mould of a machine has to be changed at a specific time point and the needed tools are automatically indicated. Also an AI system / NN can be used to automatically recognise the situation / state of the predefined space. For example, the production line is stopped and the user opens a panel and the AI system / NN de duces the kind of work that has to be done and indicates the corresponding tools needed for this work.

If no request for indicating is issued (R: no) the tracking 5 is continued.

In the step of indicating 6 after a request has been issued (R: yes) the requested location of at least one object is in dicated to the user with an Augmented Reality (AR) device e.g. acoustically or visually by projecting the location of the object into the field of view of the user.

After the step of indicating 6 is completed or while the step of indicating 6 is executed, the continuous tracking 5 is continued . In Fig. 2 the system for finding objects is schematically de picted. The system comprises an optical tracking system 10 and an AR device 20 as well as optionally a Human Interface Device 14.

The tracking system 10 is arranged and configured to execute the method of Fig. 1 and comprises an optical device 11 like a camera or stereo camera, a recognition unit 12, a storage 13 and optionally a localisation unit 15.

The optical device 11 generates images of the predefined space. The generated images are provided to the recognition unit 12.

The recognition unit 12 recognises specific appearances and/or specific features of objects in the generated images. If the specific appearance and/or specific features of one object are equal to those of a stored virtual object, the corresponding known object is recognised and the location is stored as corresponding location of the respective virtual object in the virtual space. If the specific appearance and/or features are not equal to those of any stored virtual object, no known object is recognised and this object may be learned as new known object as described above.

Upon request of the user or upon automatic request as de scribed above, the location of one ore more known objects may be optically or acoustically indicated to the user via the AR device 20. Thereto, the stored location of the respective virtual object is used to indicate the location of the real known object in the real predefined space. The HID 14 can be a keyboard, a mouse, a touchscreen, a mi crophone and the like. Via the HID 14 the user can issue re quests for finding known objects to the optical tracking sys tem 10.

The localisation unit 15 may be a GPS system for localising the position and or orientation of the user in the predefined space. Alternatively, the localisation unit may use the imag es of the optical device 11 for deriving (e.g. based on the field of view and/or the perspective) the position and/or orientation of the user in the predefined space. With the in formation about the position/orientation of the user in the predefined space the ( (automatically) requested) location of the at least one known object can be indicated to the user.

In Fig. 3 a stationary camera 11 as optical device of the op tical tracking system 10 is schematically depicted. The opti cal device may comprise more than one stationary camera 11. The stationary camera (s) 11 may monitor the predefined space and generate the images used for tracking the known objects.

In Fig. 4 a wearable bodycam 11 as optical device of the op tical tracking system 10 is schematically depicted. The bod ycam is worn by the user and for example attached on a shoul der of the user. The bodycam 11 has nearly the same field of view as the user. Thus, the ((automatically) requested) loca tion of the at least one known object can be very precisely indicated to the user.

The optical tracking system 10 may comprise one or more sta tionary cameras 11 monitoring the predefined space and/or one or more wearable bodycams 11 worn by one user (e.g. stereo camera) or several users at the same time (more than one user uses the system for finding objects in the same predefined area at the same time) .

In Fig. 5 AR glasses 20 as AR devices are schematically de picted. The AR glasses 20 comprise the optical device (e.g. camera) 11 and the HID (e.g. microphone) 14 of the optical tracking device. The optical device 11 has the same field of view as the user wearing the AR glasses 20. Two or more opti cal devices 11 may be integrated into the AR glasses 20 ena bling triangulation for better localisation of objects in the predefined space. The user can issue requests for finding known objects via the HID 14 in natural language.

The recognition unit 12 and storage 13 may be implemented in a smartphone (not depicted) connected to the AR glasses 20 and carried by the user or in a local server or decentralised cloud system (not depicted) connected to the AR glasses 20.

The localisation unit 15 may be integrated into the AR glass es 20 or be implemented by the smartphone or the local serv- er/cloud system.

While the user wears the AR glasses 20, the camera (s) 11 gen erate pictures of the predefined space. The images are pro vided to the recognition unit 12 comprised by the smartphone or implemented on the local server / cloud system. The recog nition unit 12, which may comprise an AI system / NN trained for recognising the specific appearance / specific features of objects, recognises the specific appearance / specific features of objects in the provided images and determines if the objects are known objects by comparing the specific ap pearances and/or specific features with those of the stored virtual objects. The user may issue a request for finding at least one known object via the HID / microphone 14 of the AR glasses. The request may be issued in natural language by the user and an AI system / NN trained for recognising requests in natural language may derive the request for finding at least one known object processable by the recognition unit 12. Alternatively or additionally, the request for finding the at least one object may be automatically issued based on a schedule or by automatic recognition of the situation / state of the predefined space via an AI system / NN trained for recognising the latter. The user wearing the AR glasses 20 gets indicated (upon (automatic) request) the location (s) of known object (s) in the predefined area by projecting this/these location (s) into the field of view of the user via the AR glasses based on the stored corresponding location (s) of the respective virtual object (s) in the virtual space.

List of reference signs

1 defining the predefined space

2 generating the digital twin of the predefined space 3 learning an object to be tracked

4 generating the digital twin of the learned object

5 tracking the locations of known objects

6 indicating 6 the location of at least one known object

10 optical tracking system

11 optical device

12 recognition unit

13 storage

14 Human Interface Device (HID)

15 localisation unit

20 Augmented Reality (AR) device