Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
APPARATUS FOR THE SEMANTIC-BASED OPTIMIZATION OF PRODUCTION FACILITIES WITH EXPLAINABILITY
Document Type and Number:
WIPO Patent Application WO/2020/182706
Kind Code:
A1
Abstract:
The invention relates to a system (1) for feedback-improved automatic solving of production facility related tasks, comprising: an input interface (2) being adapted to receive production facility related data; a semantic data enhancement module (3) being adapted to generate semantically enhanced data based on the production facility related data; a semantic-based reasoning module (4) being adapted to automatically provide an explainable artificial intelligence model, using the semantically enhanced data, wherein the artificial intelligence model relates to a predetermined production facility related task; a user interaction interface (5) being adapted to output explanation data regarding the explainable artificial intelligence model to a user (9); and a feedback module (6) adapted to receive feedback from the user (9) in response to the outputted explanation data and adapted to adjust the semantic-based reasoning module (4) and/or the user interaction interface (5) based on the received feedback.

Inventors:
ZILLNER SONJA (DE)
Application Number:
PCT/EP2020/056152
Publication Date:
September 17, 2020
Filing Date:
March 09, 2020
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
SIEMENS AG (DE)
International Classes:
G06N3/00; G06N3/04; G06N3/08; G06N5/02; G06N5/04; G06F3/0484; G06N5/00
Other References:
GUREVYCH IRYNA ET AL: "Interactive Data Analytics for the Humanities", 10 October 2018, INTERNATIONAL CONFERENCE ON COMPUTER ANALYSIS OF IMAGES AND PATTERNS. CAIP 2017: COMPUTER ANALYSIS OF IMAGES AND PATTERNS; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER, BERLIN, HEIDELBERG, PAGE(S) 527 - 549, ISBN: 978-3-642-17318-9, XP047489447
CHOO JAEGUL ET AL: "Visual Analytics for Explainable Deep Learning", IEEE COMPUTER GRAPHICS AND APPLICATIONS, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 38, no. 4, 1 July 2018 (2018-07-01), pages 84 - 92, XP011686421, ISSN: 0272-1716, [retrieved on 20180703], DOI: 10.1109/MCG.2018.042731661
STUMPF S ET AL: "Interacting meaningfully with machine learning systems: Three experiments", INTERNATIONAL JOURNAL OF HUMAN-COMPUTER STUDIES, ELSEVIER, AMSTERDAM, NL, vol. 67, no. 8, 1 August 2009 (2009-08-01), pages 639 - 662, XP026170314, ISSN: 1071-5819, [retrieved on 20090409], DOI: 10.1016/J.IJHCS.2009.03.004
ANUPMA YADAVS.C. JAYSWAL: "Modelling of flexible manufacturing system: a review", INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH, vol. 56, no. 7, 2018, pages 2464 - 2487
SHARP ET AL.: "Journal of Manufacturing Systems", vol. 48, March 2018, ELSEVIER, article "A survey of the advancing use and development of machine learning in smart manufacturing"
ZHOU ET AL.: "Journal of Advanced Engineering Informatics", vol. 29, 2015, ELSEVIER, article "A research on intelligent fault diagnosis of wind turbines based on ontology and FMECA", pages: 115 - 125
PISTOFIDIS ET AL.: "Modeling the semantics of failure Context as a means to offer context-adaptive maintenance applications", PROCEEDINGS OF THE SECOND EUROPEAN CONFERENCE OF THE PROGNOSTICS AND HEALTH MANAGEMENT SOCIETY, 2014
BARZ ET AL.: "Proceedings of 8th International Conference on Industrial Applications of Ho-lonic and Multi-Agent Systems (HoloMAS2017", 2017, article "Human-in-the-Loop Control Processes in Industry 4.0 Factories"
PANFILENKO ET AL.: "BPMN for Knowledge Acquisition and Anomaly Handling in CPS for Smart Factories", PROCEEDINGS OF THE IEEE INTERNATIONAL CONFERENCE ON EMERGING TECHNOLOGY & FACTORY AUTOMATION, BERLIN, GERMANY, ETFA, 2016
ZILLNER ET AL.: "Towards intelligent manufacturing, semantic modelling for the steel industry", PROCEEDINGS OF THE 17TH IFAC SYMPOSIUM ON CONTROL, OPTIMIZATION AND AUTOMATION IN MINING, MINERAL AND METAL PROCESSING, VIENNA, AUSTRIA, 2016
ABELE ET AL.: "An ontology-based approach for decentralized monitoring and diagnostics", PROCEEDING OF THE 12TH IEEE INTERNATIONAL CONFERENCE ON INDUSTRIAL INFORMATICS, July 2014 (2014-07-01)
ZILLNER ET AL.: "A semantic modelling approach for the steel production domain", PROCEEDINGS OF THE 1ST EUROPEAN STEEL TECHNOLOGY & APPLICATION DAYS (ESTAD, April 2014 (2014-04-01)
"Broad Agency Announcment, Explainable Artificial Intelligence", DARPA REPORT, DARPA-BAA-16-53, August 2016 (2016-08-01)
GUREVYCH ET AL.: "International Conference on Computer Analysis of Images and Patterns", October 2018, SPRINGER, article "Interactive Data Analytics for the Humanities", pages: 527 - 549
Download PDF:
Claims:
Patent Claims

1. A system (1) for feedback-improved automatic solving of production facility related tasks, comprising: an input interface (2) being adapted to receive production facility related data; a semantic data enhancement module (3) being adapted to gen erate semantically enhanced data based on the production fa cility related data; a semantic-based reasoning module (4) being adapted to auto matically provide an explainable artificial intelligence mod el, using the semantically enhanced data, wherein the artifi cial intelligence model relates to a predetermined production facility related task; a user interaction interface (5) being adapted to output ex planation data regarding the explainable artificial intelli gence model to a user (9), and to generate behavioral data describing how the user (9) navigates the explanation data; and a feedback module (6) adapted to receive feedback from the user (9) in response to the outputted explanation data and adapted to adjust the user interaction interface (5) based on the received feedback, wherein the feedback module (6) is adapted to use the behavioral data as input to adjust the us er interaction interface (5) .

2. The system (1) according to claim 1, wherein the feedback module (6) is adapted to adjust the semantic-based reasoning module (4) based on the received feedback.

3. The system (1) according to claim 1 or 2, wherein the se mantic data enhancement module (3) comprises a semantic map per (33) being adapted to automatically map data structures of the production facility related data in a foundational se mantic model, and to generate the semantically enhanced data using the foundational semantic model.

4. The system (1) according to any of claims 1 to 3, wherein the semantic data enhancement module (3) is adapted to gener ate the semantically enhanced data by semantically labeling the production facility related data using knowledge graphs and/or domain ontologies and/or context models.

5. The system (1) according to any of the preceding claims, wherein the semantic-based reasoning module (4) is adapted to provide the explainable artificial intelligence model based on at least one of inductive logical programming, reasoning, and semantic matching.

6. The system (1) according to any of the preceding claims, wherein the explainable artificial intelligence model pro vides an advice to a user (9) how to perform the predeter mined production facility related task, wherein the explana tion data to be outputted by the user interaction interface (5) comprises the advice how to perform the predetermined production facility related task.

7. The system (1) according to any of the preceding claims, wherein the explainable artificial intelligence model pro vides control information for controlling a machine (8n) of the production facility and provides information explaining the control information to a user (9), wherein the explana tion data to be outputted by the user interaction interface (5) comprises the information explaining the control infor mation .

8. The system (1) according to claim 7, further comprising a control module (7) being adapted to control a production fa cility machine (8n) based on the control information.

9. The system (1) according to any of the preceding claims, wherein the user interaction interface (5) is adapted to in teract with the user (9), allowing the user (9) to navigate the explanation data, wherein the explanation data encom passes information about how the explainable intelligence model was generated.

10. The system (1) according to any of the preceding claims, wherein the user interaction interface (5) is adapted to pre sent the explanation data to the user (9) based on a mental model of the user (9), wherein the feedback module (6) is adapted to assess the impact or relevance of the explanation data in accordance with the mental model based on the feed back .

11. The system (1) according to claim 10, wherein the seman tic-based reasoning module (4) is adapted to adjust the ex plainable artificial intelligence model by reinforcement learning, based on the feedback received by the feedback mod ule (6) from the user (9) .

12. The system (1) according to any of the preceding claims, wherein the feedback received by the feedback module (6) from the user (9) comprises indicators describing at least one of: a user (9) satisfaction, a degree of correspondence of the explanation data and the mental model, a trust building val ue, and added value of the system (1) in solving the prede termined production facility related task.

13. A computer-implemented method for feedback-improved mod elling of production facility related tasks, comprising the steps : receiving (SI) production facility related data; generating ( S2 ) semantically enhanced data based on the pro duction facility related data; providing (S3) an explainable artificial intelligence model, using the semantically enhanced data, wherein the artificial intelligence model relates to a predetermined production fa cility related task; outputting ( S4 ) explanation data regarding the explainable artificial intelligence model to a user (9), and generating behavioral data describing how the user (9) navigates the ex planation data; receiving ( S5 ) feedback from the user (9) in response to the outputted explanation data; and adapting ( S6 ) the user interaction interface (5) based on the received feedback, wherein the behavioral data is used as in put to adjust the user interaction interface (5) .

14. A computer program product (P) comprising executable pro gram code (PC) configured to, when executed by a computing device, perform the method according to claim 13.

15. A non-transitory, computer-readable storage medium (M) comprising executable program code (MC) configured to, when executed by a computing device, perform the method according to claim 13.

Description:
Description

Apparatus for the Semantic-based optimization of production facilities with explainability

The present invention relates to a system for feedback- improved automatic solving of production facility related tasks, to a computer implemented method for feedback-improved modelling of production facility related tasks and to a com puter program product and to a non-transitory, computer read able storage medium related to the computer-implemented meth od .

Traditionally, manufacturing facilities rely on hardcoded, non-flexible production or manufacturing routines and control profiles. For example, the trajectory of the machinery is de termined by the logic of manually programmed control scripts which deterministically define the precise movement and tim ing of the machine.

The lack of adaption to changing conditions may be overcome by implementing Artificial Intelligence (AI) techniques such as machine learning or deep learning. An overview is given in Anupma Yadav & S.C. Jayswal, "Modelling of flexible manufac turing system: a review", International Journal of Production Research, 56:7, 2464-2487, 2018.

The extent to which AI technologies are already used for ad vancing manufacturing systems is given in Sharp et al . , "A survey of the advancing use and development of machine learn ing in smart manufacturing", Elsevier, Journal of Manufactur ing Systems, 48, March 2018.

In order to improve the use of available data, semantic-based applications have been suggested, in particular in the fol lowing references: • Zhou et al, "A research on intelligent fault diagno sis of wind turbines based on ontology and EMECA", Elsevier, Journal of Advanced Engineering Informat ics, 29, page 115-125, 2015;

• Pistofidis et al . , "Modeling the semantics of failure Context as a means to offer context-adaptive mainte nance applications", Proceedings of the Second Euro pean conference of the prognostics and health manage ment society, Nantes, France, 2014;

• Barz et al . , "Human-in-the-Loop Control Processes in Industry 4.0 Factories", Proceedings of 8 th Interna tional Conference on Industrial Applications of Ho- lonic and Multi-Agent Systems (HoloMAS2017 ) , Lyon, France, 2017;

• Panfilenko et al . , "BPMN for Knowledge Acquisition and Anomaly Handling in CPS for Smart Factories", Proceedings of the IEEE International Conference on Emerging Technology & Factory Automation, Berlin, Germany, ETFA 2016;

• Zillner et al . , „Towards intelligent manufacturing, semantic modelling for the steel industry", Proceed ings of the 17th IFAC Symposium on Control, Optimiza tion and Automation in Mining, Mineral and Metal Pro cessing, Vienna, Austria, 2016;

• Abele et al . , "An ontology-based approach for decen tralized monitoring and diagnostics", Proceeding of the 12th IEEE International Conference on Industrial Informatics, INDIN 2014, Porto Alegre, RS, Brazil, July 2014;

• Zillner et al . , "A semantic modelling approach for the steel production domain", Proceedings of the 1st European Steel Technology & Application Days (ESTAD) , Paris, France, April 2014;

• Broad Agency Announcment, Explainable Artificial In telligence, DARPA report, DARPA-BAA-16-53, August 2016; and

• Gurevych et al . , "Interactive Data Analytics for the Humanities", International Conference on Computer Analysis of Images and Patterns, Springer, p. 527- 549, October 2018.

Many AI applications are based on black-box machine learning algorithms, such as deep learning neural networks, support vector machines or reinforcement learning. In general, no or only little explanation is provided how the results or recom mendations are derived.

Because of a lack of understanding how the new AI-based con trol mechanisms operate and derive their results, it is very difficult for expert users to interact with the control loops of the machines. One consequence is that expert users may hesitate to trust the outcomes of the AI techniques, another consequence being that it is difficult to include the ex perts' knowledge in improving the AI techniques.

It is therefore an object of the present invention to provide a system and a method for feedback-improved automatic solving of production facility related tasks which allow for seamless cooperation and interaction between (expert) users and AI- based machines.

This object is solved by the subject matter of the independ ent claims. Advantageous embodiments are set out in the de pendent claims.

According to a first aspect, a system for feedback improved automatic solving of production facility related tasks is provided. The system comprises an input interface which is adapted to receive production facility related data. The sys tem further comprises a semantic data enhancement module which is adapted to generate semantically enhanced data based on the production facility related data. A semantic-based reasoning module is adapted to provide an explainable artifi cial intelligence model, wherein the semantic-based reasoning module is adapted to use the semantically enhanced data. The artificial intelligence model relates to a predetermined pro- duction facility related task and is adapted to solve the production facility related task. The system comprises a user interaction interface which is adapted to output explanation data regarding the explainable artificial intelligence model to a user. Further, the system comprises a feedback module which is adapted to receive feedback from the user in re sponse to the outputted explanation data. The feedback module is adapted to adjust the semantic-based reasoning module and/or the user interaction interface based on the received feedback .

According to a second aspect, the invention provides a method for feedback-improved modelling of production facility relat ed tasks. The method comprises receiving production facility related data by an input interface. The method further com prises generating, by a computing device (e.g. by a semantic data enhancement module) , semantically enhanced data based on the production facility related data. An explainable artifi cial intelligence model is provided by a computing device (e.g. by a semantic-based reasoning module) using the seman tically enhanced data. The artificial intelligence model re lates to a predetermined production facility related task and may provide a solution the production facility related task. Explanation data regarding the explainable artificial intel ligence model is outputted to a user by a user interaction interface. Feedback is received by a feedback module from the user in response to the outputted explanation data. The se mantic-based reasoning module and/or the user interaction in terface is adapted by the feedback module based on the re ceived feedback.

The invention can provide a semantic-based mechanism for the optimized operation of production facilities which provides explanations for each optimization option, establishing the basis for the informed interaction and decision-making of an expert user while continuously learning from interaction with the user and feedback from the user. The expert user can fo- cus on ambitious and creative tasks instead of repetitive tasks which are taken over by the system.

The invention provides explanation data to the user which can help the user to understand how the explainable artificial intelligence model or certain results derived using the ex plainable artificial intelligence model are obtained. An ex plainable artificial intelligence model differs from black box approaches by providing additional explanations which are presented to the user to understand the underlying function ing of the artificial intelligence model. The explainable ar tificial intelligence model may be trained to solve optimiza tion problems. The explainable artificial intelligence model may implement any known method, examples being layer-wise relevance propagation, counterfactual methods, local inter pretable model-agnostic explanations, generalized additive models or rationalization.

The better insight of the user can help to improve the arti ficial intelligence model using the expert knowledge by providing feedback. The feedback can be used to automatically improve the artificial intelligence model by reinforcement learning. Alternatively or additionally, the feedback can be used to automatically adjust the explanation data presented to the user. A synergetic effect of adjusting both the seman tic-based reasoning module and the user interaction interface is that by steadily improving the machine-human interaction by presenting more suitable information to the user, the feedback of the user is improved which can in turn help to improve the semantic-based reasoning module.

Explanation data derived from an explainable artificial in telligence model based on semantically enhanced data can help the user to understand the solutions of the predetermined production facility related task obtained based on the arti ficial intelligence model even without detailed understanding of the underlying AI concepts. Often, AI applications are only implemented on level 0 (pro cess), level 1 (sense) or level 2 (monitor) of the conceptual topology of manufacturing IT systems. Using semantic technol ogies, giving the user explanations relating to the underly ing AI processes, and giving the user the opportunity to pro vide feedback can help to use AI techniques also on level 3 (manufacturing operations management) and level 4 (business planning and logistics) .

In some advantageous embodiments of the system, the produc tion facility related data received by the input interface can comprise data from external devices or databases, includ ing sensor data obtained by certain machines of the produc tion facility or configuration data with regard to the ma chines of the production facility. The data may be trans ferred to the system via the input interface. Additionally or alternatively, at least some of the data may be stored on ex ternal devices or databases in order to save storage space and the system can access the data via the input interface.

In some advantageous embodiments of the system, the semantic data enhancement module comprises a semantic mapper which is adapted to automatically map data structures of the produc tion facility related data in a foundational semantic model. The semantic mapper is further adapted to generate the seman tically enhanced data using the foundational semantic model. The semantically enhanced or semantically annotated data is used as input for reasoning by the semantic-based reasoning module to solve optimization problems. The semantic mapper may be adapted to enhance unstructured production facility related data by adding semantic labels. The semantic mapper ensures that semantic representations of the production fa cility related data are available.

In some advantageous embodiments of the system, the semantic data enhancement module comprises a storage device adapted to store the generated semantically enhanced data. In some advantageous embodiments of the system, the semantic data enhancement module is adapted to generate the semanti cally enhanced data by semantically labeling the production facility related data using knowledge graphs and/or domain ontologies and/or context models. The knowledge graphs, do main ontologies or context models may provide the correspond ing labels for annotating the production facility related da ta and may provide the required background knowledge for en hancing the production facility related data by process, con text and situational meta-labels. Knowledge graphs may be considered as representing a collection of interlinked de scriptions of entities, the entities being real-world ob jects, events, situations or abstract concepts. Domain ontol ogies are modelling domain specific definitions of terms. Context models define the way context data is structured and maintained. The context is described in a formal way and sim plifies the description of greater structures.

In some advantageous embodiments of the system, knowledge graphs may be task or skill models adapted to describe tasks or skills of a machine of the production facility, wherein the tasks or skills may comprise which equipment, products or tasks can be accomplished, material that can be worked on, or processes, tasks or activities needed to manufacture a prod uct, such as cutting, moving or pressing. A generic knowledge model, such as a foundational model, may determine how the machine skills and production tasks are formally described.

In addition, the semantic data enhancement module may com prise instantiations of the task or skill models, i.e. seman tically enhanced data from data that has been semantically mapped to the skill of the machine and the production task.

In some advantageous embodiments of the system, the semantic data enhancement module comprises a storage device adapted to store information regarding at least part of the knowledge graphs and/or domain ontologies and/or context models used by the semantic data enhancement module to generate the semanti cally enhanced data. In some advantageous embodiments of the system, the semantic- based reasoning module is adapted to provide the explainable artificial intelligence model based on at least one of induc tive logical programming, (automated) reasoning and semantic matching. (Inductive) logical programming is based on formal logic and deals with sets of sentences and logical forms which express facts and rules about a problem domain. Pro grams are separated into logical components and control com ponents. Inductive logical programming can be used to derive new hypotheses based on logical programming. Automated rea soning allows computers to automatically reason, i.e. to de duce certain results or conclusions. Automated reasoning can make knowledge captured in knowledge graphs explicit. Seman tic matching allows to identify nodes in two structures (such as classifications or ontologies) which semantically corre spond to each other. Semantic matching can align constraints of different knowledge graphs as well as constraint or goal driven planning to align workflows or processes with the (physical) constraints specified in the related knowledge models such as a machine skill model.

In some advantageous embodiments of the system, the explaina ble artificial intelligence model provides an advice to a us er how to perform the predetermined production facility re lated task. The explanation data to be outputted by the user interaction interface comprises the advice how to perform the predetermined production facility related task. The advice may comprise production optimization actions, choices of cer tain options or recommendations.

In some advantageous embodiments of the system, the explaina ble artificial intelligence model provides control infor mation for controlling a machine of the production facility and provides information explaining the control information to a user. The explanation data to be outputted by the user interaction interface comprises the information explaining the control information. In some advantageous embodiments, the system further compris es a control module which is adapted to control a production facility machine based on the control information. For exam ple, parameters of the production facility machine may be ad justed based on the control information. The complete manu facturing process can be controlled based at least partially on the control information obtained by the semantic-based reasoning module.

In some advantageous embodiments of the system, the user in teraction interface is adapted to provide the explanation da ta using natural language. Therefore, human decision-making process is augmented by providing explanations on the right level of understanding while continuously learning from us ers' interaction and feedback.

According to the invention, the user interaction interface is adapted to generate behavioral data describing how the user navigates the explanation data, wherein the explanation data encompasses information about how the explainable intelli gence model was generated. The feedback module is adapted to use the behavioral data to improve or adjust the user inter action interface. For example, the user interaction interface may be adapted to analyze which questions the user asks to get further information regarding the explainable artificial intelligence model.

In some advantageous embodiments of the system, the user in teraction interface is adapted to present the explanation da ta to the user based on a mental model of the user. The feed back module is adapted to assess the impact or relevance of the explanation data in accordance with the mental model based on the feedback. The mental model reflects the problem and solution model of the underlying problem space. It is modelling the underlying decision process of the user (e.g. based on experience-based or discipline-based know how) and ensures that humans and machines can cooperate in a synerget- ic way. The mental model ensures that only relevant infor mation is presented to the user. The mental model therefore influences which information is regarded to be relevant for the user to take certain decisions or to understand the solu tion to the predetermined production facility related task obtained based on the artificial intelligence model. By as sessing the impact or relevance of the explanation data, im provement can be achieved as to which information is to be presented to the user. The improvement may consist in omit ting unnecessary information which the user does not need to understand or implement the solution. The improvement may ad ditionally or alternatively consist in providing additional data which may be helpful for the user to understand the so lution of the production facility related task. Therefore, depending on the feedback of the user, more or less infor mation can be presented in the explanation data outputted by the user interaction interface to the user.

In some advantageous embodiments of the system, the semantic- based reasoning module is adapted to adjust the explainable artificial intelligence model by reinforcement learning, based on the feedback received by the feedback module from the user. Parameters of the explainable artificial intelli gence model may be changed based on the feedback received by the feedback module from the user. The feedback module may also compute an error vector comparing the solution computed by the semantic-based reasoning module based on the explaina ble artificial intelligence model which solves the predeter mined production facility related task with actual data de scribing the actual implementation in the production facility process. The data may be obtained from the user or may be au tomatically received by the input interface.

In some advantageous embodiments of the system, the feedback received by the feedback module from the user comprises indi cators describing at least one of the following: a user sat isfaction, a degree of correspondence of the explanation data and the mental model, a trust building value, and added value of the system in solving the predetermined production facili ty related task.

The invention also provides a computer program comprising ex ecutable program code configured to, when executed (e.g. by a computing device) , perform the method according to the second aspect of the invention.

The invention also provides a non-transitory computer- readable data storage medium comprising executable program code configured to, when executed (e.g. by a computing de vice) , perform the method according to the third aspect of the invention.

The invention also provides a data stream comprising (or con figured to generate) executable program code configured to, when executed (e.g. by a computing device), perform the meth od according to the second aspect of the invention.

The computing device as well as some or all components of the system may comprise hardware and software components. The hardware components may comprise at least one of microcon trollers, central processing units (CPU) , memories and stor age devices.

Brief description of the drawings

The invention will be explained in greater detail with refer ence to exemplary embodiments depicted in the drawings as ap pended .

The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the invention . Other embodiments of the present invention and many of the intended advantages of the present invention will be readily appreciated as they become better understood by reference to the following detailed description. Like reference numerals designate corresponding similar parts. It should be under stood that method steps are numbered for easier reference but that said numbering does not necessarily imply steps being performed in that order unless explicitly or implicitly de scribed otherwise. In particular, steps may also be performed in a different order than indicated by their numbering. Some steps may be performed simultaneously or in an overlapping manner .

Fig. 1 schematically shows a block diagram illustrating a system for feedback-improved automatic solving of production facility related tasks according to an embodiment of the invention;

Fig . 2 schematically illustrates a flowchart of a method for feedback-improved modelling of production facil ity related tasks according to an embodiment of the invention;

Fig . 3 schematically illustrates a block diagram illustrat ing a computer program product according to an em bodiment of the invention; and

Fig. 4 schematically illustrates a block diagram illustrat ing a non-transitory, computer-readable storage me dium according to an embodiment of the invention.

Detailed description of the invention

Figure 1 schematically illustrates a block diagram of a sys tem 1 for feedback-improved automatic solving of production facility related tasks. A production facility may comprise any building or area where goods are produced or manufac tured, examples being oil, gas, beer bottles or vehicles.

This may include any facility which is used in the production or manufacturing of an asset or product of a certain kind, including the processing that results in changing the condi- tion of assets or products. A production facility can be con ceptualized as the sum of all equipment that is used for pro ducing particular assets or products.

The production facility related task may generally refer to any optimization problem for determining decisions or choos ing options related to the production facilities. The produc tion facility related task may comprise maintenance predic tions such as forecasting the outage of an engine or another important part of a production line. The production facility related task may also comprise the optimization of control parameters ensuring the improved operation of the production facility. Examples are reduction of energy consummation or reduction of carbon dioxide footage. As another example, the production facility related task may relate to providing an optimal sequence of activities required for manufacturing a certain product.

The system 1 comprises an input interface 2, which may com prise any kind of wireless or wired connection which receives data or exchanges data with external data sources 8. The ex ternal data sources 8 may comprise databases providing pro duction facility data 81 which comprises operational data, facility data, documents, descriptions or configuration data related to machines or devices of the production facility.

The external data sources 8 may also comprise production fa cility machines or devices 8n. For example, the machines may comprise sensors for generating sensor data and the sensor data may be transmitted as facility related data to the input interface 2. The input interface 2 may be connected with any number n of individual data sources. The input interface 2 may also be adapted to access data directly via a connection without storing the data in the system 1 itself. The produc tion facility related data may also comprise information re garding semantic models.

The system 1 further comprises a semantic data enhancement module 3 which is connected to the input interface 2. Based on semantic models received by the input interface 2 or stored in internal storing devices of the system 1, the se mantic data enhancement module 3 generates semantically en hanced data based on the production facility related data.

The semantic data enhancement module 3 comprises a first data storage component 31 adapted to store domain ontologies and/or context models. At least some of the information re garding the domain ontologies and/or context models may be received via the input interface 2. The semantic data en hancement module 3 further comprises a second data storage component 32 adapted to store the semantically enhanced data generated by the semantic data enhancement module 3. Produc tion facility related data received from the input interface 2 may be stored in the second data storage component 32 and may be enhanced by adding semantic labels. A data storage de vice such as any conventional memory for storing data may comprise the first data storage component 31 and the second data storage component 32.

The semantic data enhancement module 3 comprises a semantic mapper 33 which is adapted to automatically map data struc tures of the production facility related data in a founda tional semantic model in order to generate semantically en hanced data using the foundational semantic model, domain on tologies and context models. For example, sensor data ob tained from a production facility machine 8n may be labeled as corresponding to a certain type of sensor data from a cer tain production facility machine 8n. The semantic mapper 33 may be adapted to generate semantically enhanced data using knowledge graphs and/or domain ontologies and/or context mod els .

The system 1 further comprises a semantic-based reasoning module 4 which is connected to the semantic data enhancement module 3. The semantic-based reasoning module 4 can access the semantically enhanced data stored in the second data storage component 32 and uses the semantically enhanced data to provide an explainable artificial intelligence model. The explainable artificial intelligence model may be consid ered to comprise an artificial intelligence algorithm togeth er with explanations regarding the algorithm. The explainable artificial intelligence model may be adapted to solve a spe cific task in the production facility and may give advice or recommendation how to solve the task. The expendable artifi cial intelligence model may additionally or alternatively be adapted to produce new improved control information or poli cies allowing machines of the production facility to accom plish specific tasks in improved manner (related to some pre defined optimization task) , together with explanation which is presented to the user.

The system 1 further comprises a user interaction interface 5 which is adapted to output explanation data regarding the ex plainable artificial intelligence model to a user 9. The user interaction interface 5 may comprise a display and an input device such as a keyboard or touch screen which helps the us er 9 to input information or select options presented to the user 9.

The user interaction interface 5 is based on an interface al lowing the presentation of understandable explanations. The design of the interface and user dialogue considers the cog nitive tasks of the user 9 as well as the mental model of the user interaction. The user interaction interface 5 may take the production process, machine skills and production steps into account. The user interaction interface 5 presents sim ple hypotheses in natural language to the user based on the explainable artificial intelligence model and using the men tal model. The user 9 can interact with the user interaction interface 5 by asking further questions or by exploring or navigating the explanation data. The user interaction inter face 5 therefore helps the user 9 to understand the underly ing way of functioning of the explainable artificial intelli gence model without requiring specific AI programming compe tences. The user 9 will understand the rationale why the ex- plainable artificial intelligence model is or is not recom mending and/or performing certain actions. The user interac tion interface 5 may be configured to provide the user expla nation data regarding the underlying assumptions of the ex plainable artificial intelligence model as well as explana tion data regarding when the explainable artificial intelli gence model is expected to succeed or fail. The user 9 can determine at an early stage if the results of the explainable artificial intelligence model are trustable based on false or insufficient assumptions.

The system 1 further comprises a feedback module 6 which is adapted to receive feedback from the user 9 in response to the outputted explanation data. The feedback module 6 allows to incorporate human feedback to improve the overall perfor mance of the explainable artificial intelligence model. The feedback from the user 9 is related to the specific produc tion facility related task solved by the explainable artifi cial intelligence model.

The feedback module 6 may comprise a user dialogue which al lows the user to provide feedback to certain predefined ques tions. The questions may comprise if the user 9 is satisfied with the explanation data, i.e. if the explanation data pro vided by the user interaction interface 5 was sufficiently clear and useful. The feedback module 6 may also ask the user 9 for feedback regarding the degree to which the user inter action interface 5 reflects the mental model of the user 9. For example, the user 9 may be asked if the proposed actions or recommendations fitted to the decision rationale of the user 9 and if the user 9 can build or reuse insights gained by the explanation data provided by the user interaction in terface 5. The user 9 may give feedback if the user 9 was able to interact with the system 1 to get the information needed. In particular, the user 9 may describe if the provid ed information was sufficient or if additional information was lacking. The user 9 may also describe that some of the information was not needed for taking a decision. The feed- back module 6 may also check if the user 9 trusts the system 1. The user 9 can provide information if the explanation data was sufficient and convincing such that the user 9 could rely on the results of the explainable artificial intelligence model. The user 9 may also provide feedback if the system 1 provides added value, i.e. if the system 1 helps to accom plish the predetermined production facility related task in better quality or with higher performance.

The feedback module 6 can be adapted to use the received feedback to improve the systems in two ways, namely by im proving the underlying AI models used by the semantic-based reasoning module 4, and by improving the interaction of the user 9 with the system 1 by adjusting the user interaction interface 5.

With regard to the first aspect, the feedback module 6 may adjust the semantic-based reasoning module 4. The feedback module 6 may be adapted to continuously improve the semantic- based reasoning module 4 by feedback loops, i.e. reinforce ment learning assessing how well the intended production fa cility related task has been addressed by the semantic-based reasoning module 4. The reinforcement learning algorithms may be implemented by the feedback module 6 and the semantic- based reasoning module 4. The feedback module 6 may give the user 9 an opportunity to share observations regarding the im pact of past actions of the system 1 in discrete time steps. At each time step, the user 9 will input his observations and feedback into the feedback module. The feedback module 6 ana lyzes the degree to which the intended goal related to the production facility related task has been accomplished and adjusts the explainable artificial intelligence model. For example, an error vector may be generated and taken to train a neural network of the explainable artificial intelligence model. By continuously assessing the performance of the se mantic-based reasoning module 4 in accordance to a specific predetermined production facility related task, the system 1 can be continuously improved. With regard to the second aspect, the feedback module 6 may adjust the user interaction interface 5 based on the feed back. For example, the feedback module 6 may assess the be havioral data generated by the user interaction interface 5. The feedback module 6 may determine based on the feedback if the explanation data was helpful to the user 9. The relevance of the explanation data is analyzed in accordance with the mental model and the explanation data outputted to the user 9 is adjusted based on the analysis.

Optionally, the system 1 further comprises a control module 7 which is adapted to control a production facility machine 8n based on the control information. The complete algorithm con trolling the machine 8n may be determined based on the con trol information. It may also be possible to only adjust cer tain parameters of the production facility machine 8n based on the control information.

Figure 2 shows a flowchart describing a computer-implemented method for feedback-improved modelling of production facility related tasks. The method can be performed using a system 1 described above.

In a first method step SI, production facility related data is received by an input interface 2. The data may be stored locally or may also be accessed via a wireless or wired con nection. For example, the data may be at least partially stored on an external server. Data sources from the produc tion facility may be connected to a semantic data storage component .

In a second method step S2, semantically enhanced data is generated based on the production facility related data. A semantic mapper 33 may automatically map data structures of the production facility related data in a foundational seman tic model. The semantic mapper 33 may generate the semanti cally enhanced data using the foundational semantic model. For generating the semantically enhanced data, knowledge graphs, domain ontologies and/or context models may be used. Production facility related data is represented in a semantic manner, i.e. by annotating data assets and items with seman tic labels. The semantic representation of all related data sources from the production facility is obtained.

In a third method step S3, an explainable artificial intelli gence model is provided by a semantic-based reasoning module 4, using the semantically enhanced data. The artificial in telligence model relates to a predetermined production facil ity related task, e.g. production planning. The production facility related task may be obtained from a user 9. The pro duction facility related task may also be transmitted from external devices via the input interface 2. The user 9 may select the production facility related task together with as sociated parameters, for instance a set of products which are to be produced. In this way, the user 9 is asking the system 1 to compute the most efficient sequence of activities accom plished by the production facility for producing the selected items .

The artificial intelligence model may be adapted to solve a certain optimization problem related to the predetermined production facility related task. The explainable artificial intelligence model may provide advice to a user 9 how to per form the predetermined production facility related task. The explainable artificial intelligence model may also provide control information for controlling a machine 8n of the pro duction facility and may provide information explaining the control information to a user 9. The explainable artificial intelligence model may be obtained based on logical program ming, reasoning and/or semantic matching.

In a fourth method steps S4, explanation data regarding the explainable artificial intelligence model is outputted to a user 9 by a user interaction interface 5. The explanation da ta may comprise the advice how to perform the predetermined production facility related task. The explanation data may also comprise the information explaining the control infor mation for controlling a machine 8n of the production facili ty. The explanation data may be generated based on a mental model of the user 9. The user 9 may interact with the user interaction interface 5, allowing the user 9 to navigate the explanation data, wherein the explanation data encompasses information about how the explainable intelligence model was generated. The user 9 may ask additional questions. In re sponse to the questions, the user interaction interface 5 may generate new explanation data based on the mental model of the user 9 and on the explainable artificial intelligence model. The user interaction interface 5 may allow the user 9 to easily navigate (zoom in and zoom out) the knowledge model or explanation data that has been constructed during the rea soning process with regard to the explainable artificial in telligence model. The user interaction interface 5 may gener ate behavioral data describing how exactly the user 9 navi gates the explanation data. In particular, the user interac tion interface 5 may analyze additional questions posed by the user 9.

In a fifth method step S5, feedback is received from the user 9 from a feedback module 6 in response to the outputted ex planation data. The feedback may comprise information or in dicators describing the satisfaction of the user 9, the cor respondence of the explanation data and a mental model used for providing the explanation data, a trust building value, and added value of the system 1 in solving the predetermined production facility related task. The feedback module 6 may receive as input the behavioral data describing how the user 9 is exploring the reasoning result or knowledge model. The feedback module 6 may further receive the results of the as sessment of the explanation collected after each interaction from the user 9. The feedback module 6 may further receive feedback from the user 9 about the performance of elements of the reasoning results related to the selected task that needs to be performed. The feedback is transformed into information or parameters that can be used to improve the overall reason ing components.

In a sixth method step S6, the semantic-based reasoning mod ule 6 is adapted based on the received feedback. Additionally or alternatively, the user interaction interface 5 is adjust ed based on the received feedback. Adapting the user interac tion interface 5 based on received feedback may be performed based on behavioral data generated by the user interaction interface 5. Further, the impact or relevance of the explana tion data may be assessed in accordance with the mental model based on the feedback in order to adjust the explanation data presented to the user 9. The explainable artificial intelli gence model may be adjusted by reinforcement learning, based on the feedback received by the feedback module 6 from the user 9.

After the sixth method step S6, new production facility re lated data may be received, SI. Alternatively, the explaina ble artificial intelligence model may be updated, S3, or the explanation data may be updated, S4.

A solution for the predetermined production facility related task obtained by the semantic-based reasoning module 4 may be used to automatically control external devices, such as a production facility machine 8n.

Data relating to the explainable artificial intelligence mod el, or certain components of the explainable artificial in telligence model, in particular explanation data regarding or extracted from the explainable artificial intelligence model can be stored on data storage components part of the system 1 or on external data storage components. In addition or alter natively, such data can be transmitted to external servers or devices for further analysis. In particular, an external con troller device can be adapted to control devices or machines 8n of the production facility based on the transmitted data. It should be understood that all advantageous options, vari ance in modifications described herein and the foregoing with respect to embodiments of the system according to the first aspect may be equally applied to embodiments of the method according to the second aspect, and vice versa.

In the foregoing detailed description, various features are grouped together in one or more examples for the purpose of streamlining the disclosure. It is to be understood that the above description is intended to be illustrative, and not re strictive. It is intended to cover alternatives, modifica tions and equivalents. Many other examples will be apparent to one skilled in the art upon reviewing the above specifica tion .

Figure 3 schematically illustrates a block diagram illustrat ing a computer program product P comprising an executable program code PC. The executable program code PC is configured to perform, when executed (e.g. by a computing device), the method according to the second aspect.

Figure 4 schematically illustrates a block diagram illustrat ing a non-transitory, computer-readable storage medium M com prising executable program code MC configured to, when exe cuted (e.g. by a computing device), perform the method ac cording to the second aspect.