FIELD OF THE INVENTION The present invention generally relates to process control systems. More particularly the present

invention relates to a method, user interface control arrangement and computer program product for

controlling a stationary user interface in an

industrial process control system as well as to such a process control system.

BACKGROUND A process control system normally comprises a number of process control devices involved in the control of the process .

The process control system does then typically comprise control computers as well as operator stations or monitoring computers via which operators can monitor the process. The operator stations are then normally stationary and provided in a control room or a control station in the plant.

At an operator station in a process control system it is common to have a graphical process control view displaying the process or a part of the process, where the process and its devices may be represented as graphical blocks interconnected in a logical structure.

It is also known to have video cameras installed at locations around the process control devices. The cameras capture video images that are streamed to the control room or control station for being observed via the monitoring computers. The operators can then view the video streams on the screens of their monitoring computers. However, in this case the video is typically displayed beside the graphical blocks. The majority of the display area is typically occupied by the graphical process control view. In some processes the overview is much better in the video stream. This means that in the above-mentioned situation, it is possible that an operator fails to notice important changes of the process because the video is dominated by the process view. It would thus be of interest to get a better overview of the process where a live video stream dominates the user interface display. However, it may still be necessary to see the process view. It is known to overlay process control graphics on live video in a process control environment. This is

described in US 2013/032125. However, the overlaying is in this document performed in a mobile device.

Furthermore it seems as the display of the process control graphics relies on the mobile device being brought out to the location where the video is captured in order to obtain knowledge of the process graphics that are to be combined with the live video. The teachings of this document cannot therefore easily be transferred to an environment where the user interface is stationary. There is thus a need for allowing a better overview of a process control system at a stationary user

interface . The present invention is provided for solving one or more of the above described problems.

SUMMARY OF THE INVENTION The present invention addresses the problem of

improving process overview of a process at a stationary user interface.

This object is according to a first aspect of the invention solved through a method for controlling a stationary user interface in an industrial process control system, the method being performed by a user interface control arrangement and comprising the steps of:

obtaining a first live video stream from a video camera monitoring the process at a first location,

obtaining a process control view for the first

location,

overlaying the process control view on the first live video stream, and

displaying the first live video stream with the

overlaid process control view on a display of the user interface . This object is according to a second aspect of the invention solved through a interface control

arrangement for controlling a stationary user interface in an industrial process control system, the user interface control arrangement comprising:

a video obtaining unit configured to obtain a first live video stream from a video camera monitoring the process at a first location,

a control graphics unit configured to obtain a process control view for the first location, and

a user interface control unit configured to overlay the process control view on the first live video stream and display the first live video stream with the overlaid process control view on a display of the user

interface .

This object is according to a third aspect of the invention also solved through an industrial process control system for controlling an industrial process and comprising a user interface control arrangement according to the second aspect. This object is according to a fourth aspect of the invention solved through a computer program product for controlling a stationary user interface in an

industrial process control system, the computer program product being provided on a data carrier comprising computer program code configured to cause a user interface control arrangement to, when the computer program code is loaded into at least one device

providing the user interface control arrangement, obtain a first live video stream from a video camera monitoring the process at a first location,

obtain a process control view for the first location, overlay the process control view on the first live video stream, and display the first live video stream with the overlaid process control view on a display of the user

interface . The present invention has a number of advantages. A video stream which in some types of processes best shows the changes in the process forms the foundation of an operator view on a display. This allows the operator to react quickly to important changes that occur in the process. At the same time the relevant process control view is also present and observed. This has the advantage of the operator not having to divert his or her attention from the live video stream in order to look at the process control view, which enhances the security and reliability of control operations .

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will in the following be

described with reference being made to the accompanying drawings, where

Fig. 1 schematically shows an industrial plant with a process control system operating an industrial process and comprising a number of video cameras and a user interface device acting as a user interface control arrangement,

Fig. 2 schematically shows a block schematic of the user interface device,

Fig. 3 shows a first and a second location in the premises of the industrial plant, where there are a number of process control devices, objects being produced in the process, and video cameras at the two locations ,

Fig. 4 schematically shows a live video stream captured by a first of the video cameras,

Fig. 5 schematically shows process graphics associated with the first location,

Fig. 6 schematically shows the process graphics overlaid on the live video stream,

Fig. 7 schematically shows an object with and without a visible activity indicator,

Fig. 8 shows a flow chart of a number of method steps being performed in a method of controlling a user interface, and

Fig. 9 schematically shows a data carrier with computer program code, in the form of a CD-ROM disc, for

performing the steps of the method.

DETAILED DESCRIPTION OF THE INVENTION In the following, a detailed description of preferred embodiments of a method, user interface control arrangement and a computer program product for

controlling a stationary user interface in a process control system will be given.

Fig. 1 schematically shows an industrial plant where an industrial process control system 10 is provided. The process control system 10 is a computerized process control system for controlling an industrial process. The process can be any type of industrial process, such as electrical power generation, transmission and distribution processes as well as water purification and distribution processes, oil and gas production and distribution processes, petrochemical, chemical, pharmaceutical and food processes, and pulp and paper production processes. These are just some examples of processes where the system can be applied. There exist countless other industrial processes. The processes may also be other types of industrial processes such as the manufacturing of goods. A process may be monitored through one or more process monitoring computers, which communicate with a server handling monitoring and control of the process.

In fig. 1 the process control system 10 therefore includes a number of process monitoring computers 12, 14 and 16. These computers may here also be considered to form operator terminals or work stations and are connected to a first data bus Bl . These operator terminals are also user interfaces to the operators of the process control system. They are furthermore stationary. One of the process monitoring computers 16 is also a user interface device acting as a user interface control arrangement. It should be realized that the user interface control arrangement is not necessarily limited to implementation in a user

interface. It may for instance be implemented through a server connected to the first bus Bl, which server controls the display of data via a user interface of a process monitoring computer.

To the first bus Bl there is also connected a first and a second video camera 32 and 34. The video cameras 32 and 34 are provided at a first and a second location in the facilities of the plant where the process is being run. These locations differ from the locations of the user interfaces. The user interfaces are stationary. Some of these user interfaces are provided in a control room, while others may be provided at a dedicated control station in the plant. The control room and control stations, and therefore also the user

interfaces, are thus separated from the first and second locations.

There is furthermore a second data bus B2 and between the first and second data busses Bl and B2 there are connected a server 18 providing control and protection of the process and a database 20 where data relating to control and protection of the process is stored. Such data relating to control and protection may here comprise process data such as measurements and control commands, while data relating to protection may

comprise alarm and event data as well as data on which alarms and events can be generated, such as

measurements made in the process. The database 20 may also comprise graphics related to the process such as graphical symbols representing the process control devices, logical process connection schemes and face plates of process control devices and physical

properties of the process control devices.

To the second data bus B2 there is also connected a number of further devices 24, 26, 28 and 30. These further devices 24, 26, 28 and 30 are field devices, which are devices that are interfaces to the process being controlled. A field device is typically an interface via which measurements of the process are being made and to which control commands are given. Because of this the field devices are furthermore process control devices. There is thus a first process control device 24, a second process control device 26, a third process control device 28 and a fourth process control device 30.

Fig. 2 shows a block schematic of the user interface device 16. The user interface device 16 comprises an internal bus 38 and to this internal bus there is connected a communication interface 40 for connection to the first data bus Bl, a processor 42, a program memory 44 as well as a display 36. The communication interface 40 may be an Ethernet communication

interface . In the program memory 44 there is provided software code which when being run by the processor 42 forms a video obtaining unit 46, a control graphics unit 48, a user interface control unit 50, an image recognition unit 52 and an activity determining unit 54.

Fig. 3 schematically shows two adjoining locations of the industrial plant. There is a first location LI and a second location L2. At the first location LI, the first, second and third process control devices 24, 26 and 28 are placed. At the first location LI also the first video camera 32 is placed or provided for

monitoring the process at this first location LI. In this example the first video camera 32 is fixed, i.e. it has a fixed position at the first location LI and is placed so that when having a first orientation it is able to cover the first and second process control devices 24 and 26, while if it has a second orientation it will be able to cover the third process control device 28. At the first location LI there is also a first object 01 being provided by the first process control device 24. At the second location L2, the second video camera 34 is located and set to cover the fourth process control device 30. A second object 02 is seen as being supplied from the second process control device 26 to the fourth process control device 30.

Furthermore, during the running of the process the object 01 being produced by the first process control device 24 may be moved from the first to the second process control device 26 for some further processing. It may thereafter be moved to the second location L2 for being handled by the fourth process control device 30.

A first embodiment of the invention will now be

described with reference also being made to fig. 4 - 8, where fig. 4 schematically shows a live video stream captured by the first video camera, fig. 5

schematically shows process graphics associated with the first location LI, fig. 6 schematically shows the process graphic overlaid on the live video stream, fig. 7 schematically shows the object with and without a visible activity indicator and fig. 8 shows a flow chart of a number of method steps being performed in a method of controlling a user interface.

In the exemplifying process there is a production of objects 01 and 02. The process is furthermore monitored by an operator via the interface device 16. In this monitoring the operator may need to act on alarms and trends to perform various operations in the process. Process operators in the industry need different kind of information to operate their process area in an efficient and safe way, such as the process area at the first location LI. This is mostly solved by using operator interface features of a control system or Human Machine Interface (HMI) panels connected to

Programmable Logic Controllers (PLCs) . Such interface features are here defined as being process control views. A process control view typically provides information of the process, sometimes in the form of interconnected graphical symbols representing the process control devices, where the interconnection may follow the process flow. In addition to this process control values and trends may be displayed.

However, in certain processes and manufacturing, it is many times very important to also have a real time video picture of the process in order to find out about the status of the process. This may for instance be needed if there is a lack of measurement methods in the part of the process being monitored. Such a video stream is often displayed beside the process control view . If the control system information of the process control view is of less importance compared to the video pictures, the above described normal Human

Machine Interface (HMI) solution is not efficient.

Video images may for instance be important to have when the operator does not have a visual contact with a machine in the process control system, which is the case if the operator terminal is stationary and separated from the location where the production takes place .

Fig. 3 shows an example of where this problem may be at hand. In the example of fig. 3, the first process control device 24 produces a product, which is the object 01, that is then transported to the second process control device 26 for being further processed. The object 01 may have properties that are not easily measured in the system, such as in the first process control device 24. However, these properties may be clearly visible after an optical inspection. For this reason the first video camera 32 is placed at the first location LI so that the object 01 can be easily

observed as it is transported between the first and second process control devices 24 and 26. The first video camera 32 thereby captures video images of the process and in this example video images of the object 01, which video images are conveyed as a first live video stream VS1 to the video obtaining unit 46 of the interface device 16 via the bus Bl .

Fig. 4 schematically shows an image captured by the first camera 32 depicting the first object 01 being transported on a transport belt between the first and second process control devices 24 and 26. The video obtaining unit 46 thus obtains or receives the first live video stream VS1, step 58. Furthermore, the control graphics unit 48 obtains a process control view PCV, step 60. This process control view PCV is a process control view for the first location LI and comprises graphical symbols related to the process control devices of the first location LI that can be seen by the first video camera 32. The graphical symbols may comprise a graphical symbol representing a physical property associated with a process control device or with the process at the process control device, a graphical symbol representing the process control device itself, a graphical symbol representing process control data trends of a process control device and a possible control command. Fig. 5 shows a few such symbols together with process control data. On the left hand side of fig. 5, there is shown next a symbol of a property in relation to the first process control device 24 together with a value 5.0 of this property, which value is a process control data value, a symbol representing the first process control device and a symbol representing a first actuating unit AU1 also related to the first process control device 24. On the right side of the figure there is shown a symbol representing trends of the second process control device 26, which symbol is a face plate and therefore also comprises such trend data, a symbol representing the second process control device 26 itself as well as a symbol representing a second actuating unit AU2 associated with the second process control device 26. The graphical symbols may for instance comprise icons obtained from a graphical symbol library in the database 20, while the process control data may be obtained from the server 18, which in turn may have collected it from the process control devices themselves. The actuating units AU1 and AU2 are process actuators that may be actuated or maneuvered by the operator in order to perform control activities in the process. In some variations the first video camera 32 has a fixed view, i.e. the area it covers and the part of the process control system that can be seen by it at the first location LI is fixed. It may for instance have the first orientation. Furthermore, the graphical symbols used may also be stored locally in the

interface device 16 and need not be fetched from the database 20. However, process control data will be needed to be fetched from the server 18.

After the video stream VS1 and the process control view PCV have been obtained these are then forwarded from the video obtaining unit 46 and the control graphics unit 48 to the user interface control unit 50. This unit 50 then overlays the process control view PCV on the first live video stream VS1, step 62, and then displays the video stream VS1 with overlaid process control view PCV on the display 36 of the user

interface device 16, step 64. The combination of video stream and process control view is schematically shown in fig. 6.

In this way an intelligent process control graphic view overlaid on a live video stream is provided as operator interface. The overlay furthermore comprises the information normally presented through control system HMIs, like alarms, process data, trends etc.

The display 36 may be a touch screen, in which case it forms a user interface where it is possible for the operator to directly actuate the first and second actuating units AUl and AU2 via the display 36 and the touch of a finger. Alternatively a separate user input unit such as a keyboard, mouse or trackball may be used .

The video stream VS1 that best shows the changes in the process thus forms the foundation of the operator view on the display 36, which allows the operator to react quickly on important changes that occur in the process. At the same time the relevant process control view is also present and observed. Furthermore the operator can directly perform operations on the combination of video stream and process control view. This has the advantage of the operator not having to divert his or her

attention from the live video stream in order to look at the process control view or to perform control activities, which enhances the security and reliability of control operations.

Furthermore, in this first embodiment, it is possible to make some further investigations of the video stream in order to enhance the control further. This is done through the use of the image recognition unit 52. The image recognition unit 52 may be provided with

augmented reality (AR) functionality. A 3D map of natural features at the location LI can be built using a variety of feature extraction methods such as corner or edge detection both with 2D RGB data and 3D RGBD (Red, Green, Blue, Depth) data. Using this sparse map it is also possible to determine if an object being processed that leaves the first process control device 24 can be detected in the live video stream VS1 and more particularly the shape of such an obj ect . Therefore the user interface control unit 50 may interrogate the image recognition unit 52 about if an object being processed is detected leaving the first process control device 24. If the answer is negative, step 66, then the user interface process control unit 50 proceeds and investigates if there is any input via an actuating unit, such as AU1 or AU2. However, if the image recognition unit 52 did detect an object being processed in the first video stream VS1, step 66, the shape of the object is then determined in the same unit 52, step 68. The image recognition unit 52 may thus detect the shape, for instance the 3D shape, of the object using image recognition, for instance in the form of the above-described AR functions. The image recognition unit 52 then forwards data about the shape to the activity determining unit 54, which data may define a 3D image of the shape. The activity determining unit 54 then determines an activity to be performed in the process based on the determined shape, step 70. The shape of the object may for instance be compared with a reference shape or ideal shape and an activity may be determined based on the difference. The activity may be a change in

relation to a process control device, for instance an activity upstream in the process flow from where the object is detected, such as a change in a setting to be made to the first process control device 24. The activity may also be an activity downstream in the process flow such as in the second process control device 26. The activity determining unit 54 may in such cases provide an automatic control command and supply it to the server 18 for making the adjustment of the process control device in question. The activity may also or alternatively be an activity to be performed on the object 01. In this case it is also possible that the activity determining unit 54 provides a visual activity indicator VAI, step 72, i.e. a visual

indication of the type of activity that is to be performed on the object 01. This visual activity indicator may then be linked to the identified object. The activity may thus be a unique activity that is to be performed on the object, the shape of which was identified. The indicator may thus be an indication of an activity that is individual for each object being detected or an activity that is dedicated to the identified object.

The indicator VAI may then be sent to the user

interface control unit 50 which makes the indicator VAI follow the object 01 in the video stream VS1. It may thus be overlaid and displayed on the object 01 in the first live video stream VS1 in order to allow the activity to be performed, step 74. The indicator VAI may thus be made to follow the object 01 as it moves in the video stream VS1. The visual activity indicator VAI may more particularly be an indicator that is linked to the geometry of the object, such as an indicator indicating areas of the object where material is to be removed or added. An object 01 with and without an indicator VAI is shown in fig. 7. In this figure there is an indication that the object is to be thinner at the middle. The visual activity indicator VAI may indicate the same type of activity to be performed on the different objects, where the degree with which the activity is to be performed on the different objects depends on their individual shapes.

Such an indication VAI may then be useful in assisting the operator in the performing of the activity at the second process control device 26, and the operator may then better define how much of the activity and where on the object the activity is to be performed. As an example the visual activity indicator VAI may be a cutting pattern, indicating where the object is to be cut, a general position indicator, a borderline etc.

The user interface control unit 50 then investigates if there is a user input in the form of a control command via a user input unit such as via an actuating unit AU1 and AU2 of the display 36. This investigation may be carried out in both cases, with or without object identification and accompanying activity indicator. If there is no control command, step 76, then the user interface control unit 50 returns and investigates if an object is detected, while if a control command is detected, step 76, which may be done with either the first or the second actuator unit AU1 and AU2 on the display 36, a corresponding control command is sent to a control device in the system, such as the server 18, for allowing this device to control the process based on the control command, step 78. Thereafter the user interface control unit 50 returns and investigates if an object is detected.

The use of visual activity indicator gives the operator further assistance in the control of the process without having to use other means for determining the control activity. Such other means may also be hard to implement in a process.

The above described first embodiment may be varied in a multitude of ways. It is for instance possible that the visual activity indicator VAI is not concerned with activities to be performed by the second process control device 26. It may be related to activities that are to be performed on the first object 01 in relation to the fourth process control device 30 at the second location L2, i.e. at a location that is not covered by the first video camera 32 but by the second video camera 34. In this case the second video camera 34 captures video images of the second location L2 that are delivered to a user interface, which may be

provided via the display 36 of the user interface device 16, but may also be via a display of any of the other user terminals 12 and 14. The interface control unit 50 may therefore export the visual activity indicator VAI for display on the object when present in the second live video stream obtained at the second location L2. It is thus possible to make the visible activity indicator VAI follow the object as it travels or moves to another location, in this example the second location L2, covered by another video camera than the one for which the object was identified.

Another variation that is possible concerns the first video camera 32. The first video camera 32 may be able to change the area that it covers. This change may involve rotating the camera. The video camera 32 may thus have an adjustable angle of rotation. It may additionally or instead be able to zoom. It may therefore also have an adjustable viewing angle, where a zooming out involves a decrease of the viewing angle and a zooming in involves an increase of the viewing angle. It is in both these cases possible that the graphical objects shown in the process control view PCV are dependent on the process control devices actually being covered by the first video camera 32. This means that the control graphics unit 48 may receive

information of the rotational and/or viewing angle covered by the first video camera 32 in order to determine the area covered by the first video camera 32 and thereafter determine a process control view that corresponds to the covered area. The determining of process control view may involve determining which process control devices are covered by the camera 32 and then only include graphical objects and process control data related to process control devices covered by the camera 32 in the process control view. The process control devices in question may easily be determined since the positions of both the camera and process control devices are known. An angle of rotation and a viewing angle may then directly be used in order to immediately locate the process control devices that can be seen and for which graphical objects are to be displayed.

The way this is done may be exemplified in relation to the third process control device 28. As can be seen in fig. 3, the first video camera 32 may initially have the first orientation where it covers the first and second process control devices 24 and 26. The first video camera 32 may then be set to rotate and/or zoom so that it has the second orientation where it covers the third process control device 28. In this case it will no longer be able to cover the first and second process control devices 24 and 26. In this case the process control view provided by the control graphics unit 48 will only comprise graphical symbols and process control data associated with the third process control device 28 and not with the first and second process control devices 24 and 26. There are a number of further variations that are possible to make. For instance, the graphical symbols were above described as being provided in the database. However, it should be realized that they may be placed in the interface device instead. This may be the case both when having a fixed orientation and when being able to change orientation of the first video camera.

It should furthermore be realized that the detection of the object and consequently also the provision of the visual activity indicator may be omitted from the first embodiment .

The functionality of the interface device may

furthermore both be provided in the form of one or more processors together with computer program memory including computer program code for performing their functions. As an alternative they may be provided in the form of an Application Specific Integrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA) . This computer program code may also be provided on one or more data carriers which perform the functionality of the present invention when the program code thereon is being loaded into an interface device. One such data carrier 80 with computer program code 82, in the form of a CD ROM disc, is schematically shown in fig. 9. Such computer program may as an alternative be provided on another server and downloaded therefrom into the interface device.

The invention can be varied in many ways. The interface arrangement, which may comprise the above-mentioned interface device or a server which interacts with the display of a user interface, may for instance comprise one or more of the video cameras. It can therefore be seen that the present invention is only to be limited by the following claims.