The invention relates to a method for safe train remote control, wherein an automatic train comprises

- an on-board ADS (= automatic driving system) unit, including a camera system monitoring a track section ahead of the automatic train, and - an automatic train mobile com router for forwarding and receiving data via a mobile communication network, wherein an ATO (= automatic train operation) trackside rail traffic operation and management centre comprises

- a train remote control console for remote control of the automatic train, includ- ing a display showing an image of the track section ahead of the train to a re- mote driver at the train remote control console, and in particular including a moving map display for indicating a train position of the automated train, and

- a management centre mobile com router for forwarding and receiving data via the mobile communication network. Such a method is known from EP 3 220 613 Al.

Trains are an important means of transport, for both passengers and freight, in a fast, efficient and ecofriendly way. Usually, a train driver on board, typically in the locomotive at the front of the train, watches the track section ahead of the train through a window and drives the train using local (on-board) driver's cabin.

However, efforts have been taken to automate driving of a train. In grade of au tomation (=GoA) level 4, generally no train driver is on board any more. Howev- er, there may be situations remaining where a human interaction is necessary or desired. Such situations may include emergency situations or technical defects in the automatic train control system, or more generally situations wherein the au- tomatic train control system recognizes a situation it cannot handle safely in au- tomatic mode. In order to end the train's journey safely, a human driver has to take over control, at least temporarily.

For this purpose, a train driver may be brought to the train who enters the driv- er's cab and takes over train control on board. However, this may be cumber- some and may cause significant delays. More convenient is a remote control of the train by a human train driver at an automatic train operation (=ATO) track- side rail traffic operation and management centre (also called simply control cen- tre). The train driver at the control center ("remote driver") requires basically all the information that an on board train driver would have, and requires access to basically all driving functions that an on board train driver would have.

An important information the remote train driver at the trackside control centre requires is a view of the track section ahead, in particular for identifying railway signs and railway signals as well as possible objects ahead of the train, including obstacles on the track.

EP 3 220 613 Al suggests that a train driver in a trackside control centre tempo- rarily takes over control of a driverless train, wherein the trackside control centre is connected to a camera of the train monitoring the track section ahead of the train. Communication is done via a packet based communication network.

Transmitting image information between the automatic train or its on-board au- tomatic driving system unit and the trackside control centre or its train remote control console where the remote driver sits requires significant bandwidth in a mobile communication network. In general, however, the bandwidth is often ra- ther limited. In order to save bandwidth, it is possibly to apply a low pixel resolution of image data and to apply image compression. However, if the pixel resolution is low, safety relevant details about the track section ahead of the train may become not recognizable. Image compression, too, may blur image details if applied too extensively. As an example, if the switching state of a railway signal cannot be identified reliably any more by the remote driver, the train safety may be imme- diately endangered by collisions. Further, the time resolution of image data may be chosen low in order to safe bandwidth, but this may lead to jerky images. In general, delays in the transfer of image data can endanger the train, too. EP 1 597 130 Bl, also published as WO 2004/074068 Al, describes a remote control for an unmanned locomotive over a track layout in a shunting yard. A portable communications device is used for commanding a desired destination for the locomotive and controlling movement of the locomotive. EP 2 765 053 Bl describes a rail train diagnostics system, wherein a wireless in- terface on board of a train transmits diagnostic data from an on-board control unit to a ground-based server. The diagnostics data is transmitted in a plurality of channels, including a live channel and a backfill channel. WO 2006/028318 Al describes an apparatus for wireless remote control of a magnetic levitation train, applying a bi-directional packet-based communication. Alstom recently announced metro train operation with passengers at GoA level 3, and during shunting with GoA level 4, with no staff on board but the possibility for remote train control, compare https ://www.alstom. com/press-releases-news/2020/5/world-first-automatic- train-operation-regional-passenger-trains-be as of 11.05.2021.

JP 002019001203 A describes a railway operation support system, wherein video signals of a camera of a train are transmitted using a video signal compression unit via a wireless network to a video display in a trackside control chamber. In a variant, an object detection part is connected to the output of a camera on the train, and if the object detection part finds an object to be gazed, then a video signal control part maintains a high visibility without reducing the resolution or time frame rate of the neighborhood area including the object. In a further vari- ant, the video compression unit imparts position information of the train as metadata of the video data to the control chamber. A storage on the train and a storage at the control chamber contain past video data of high resolution of the neighborhood area. The transmitted video signal contains only a difference be- tween the video data from the camera and from the on train storage, and at the control chamber, the transmitted difference is added to the video data from the storage at the control center, using the position information.

DE 697 31 009 T2 discloses an automatic obstacle identification, in particular for a railway anti-collision system, wherein a camera is automatically aligned. The automatic alignment can use a GPS position information.

DE 10 2012 215 544 A1 describes a method for monitoring a railway track, wherein a railway vehicle is equipped with a recording unit, and the recording is stored on the railway vehicle or at a central location. Recorded images may un- dergo an image analysis, may be provided with a time information or position information, and may be compared with image data recorded before. The rail- way vehicle may communicate with the central location via a wireless network. Object of the invention

It is the object of the present invention to provide a method for remote control of an automatic train, wherein video images of the track section ahead of the train, including enough details of the track section ahead for a save remote train con- trol, can be provided at the ATO trackside rail traffic operation and management centre, which requires only a low bandwidth consumption of a mobile communi- cation network.

Short description of the invention

This object is achieved, in accordance with the invention, by a method as intro- duced in the beginning, characterized in that the ATO trackside rail traffic operation and management centre and the au- tomatic train each comprise a map data base containing common reference ob- jects, with the reference objects representing typical types of trackside objects each, wherein for each reference object at least an object ID and a high resolu- tion object appearance information representative for the corresponding typical type of trackside object is stored, in particular wherein the high resolution object appearance information is vectorised, that the camera system generates high resolution image frames, to which image frame numbers are allocated, and which are processed in at least two processing lines, wherein in a first processing line,

- the high resolution image frames are reduced in resolution, compressed and encoded at the on-board ADS unit into low resolution image frames, and

- the low resolution image frames and their image frame numbers are transmit- ted via a first data channel from the on-board ADS unit to the train remote con- trol console via the mobile communication network, wherein in a second processing line,

- the high resolution image frames are subjected to a pattern recognition algo- rithm, which identifies objects in the high resolution image frames, allocates them to reference objects stored in the map data base of the on-board ADS unit, and determines the corresponding object ID and relative object image insertion properties of each identified object, including at least an image frame number, an insertion vector point and scaling/rotation parameters, and

- determined object IDs and image insertion properties are transmitted via a second data channel from the on-board ADS unit to the train remote control console via the mobile communication network, that at the ATO trackside rail traffic operation and management centre, the ob- ject IDs and image insertion properties of the second data channel are translated into high resolution object appearances according to the corresponding stored reference objects in the map data base of the ATO trackside rail traffic operation and management centre, and that these high resolution object appearances are seamlessly overlaid onto the low resolution image frames with matching image frame number received via the first data channel, resulting into the image of the track section ahead of the train shown to the remote driver at the ATO trackside rail traffic operation and management centre.

According to the present invention, the video images of the track section ahead of the train that are shown at the train remote control console to the remote driver is based on image data originating from (at least) two processing lines and respective data channels.

In order to capture all safety relevant information, the camera system of the au- tomatic train generates a video signal comprising high resolution image frames. However, these high resolution image frames are not transmitted via the mobile communication network as such, since this would require a large bandwidth.

In a first processing line, the high (pixel) resolution image frames are reduced in resolution in each image direction, typically by resampling the image with known techniques such as bilinear or bicubic interpolation, in this way generating low (pixel) resolution image frames. In addition the images are compressed by known techniques such as run length encoding or discrete cosine transformations or other video encoders. These low resolution image data frames are transmitted via a first data channel from the automatic train (or its on-board ADS unit) to the ATO trackside rail traffic operation and management centre (also called simply "control centre") via the mobile communication network. This requires only few bandwidth. The low resolution image frames are sufficient for giving the remote driver a general impression of the track section ahead, in particular for general orientation. For example, trackside vegetation such as grass or trees or trackside buildings do not need a high resolution in order to be recognizable, and they are not important for train safety. In the first data channel, known video image com- pression techniques (such as i-frames/p-frames/b-frames prediction) may be applied in order to further reduce the required bandwidth.

In a second processing line, the high (pixel) resolution image frames undergo a pattern recognition. The pattern recognition is based on a comparison with refer- ence objects stored in a map data base of the automatic train. These reference objects represent different types of objects that can typically be found ahead of a train, including all kinds trackside objects which might be relevant for safe train operation, in particular railway signals (including their switching state), railway signs, tracks, track switches (including their switching states) and bars of railway crossings. The reference objects preferably also include typical obstacles that may appear on a track, such as train waggons and locomotives, fallen trees, broke down vehicles, people (in particular track workers and walkers) and large animals (such as cows). For each reference object, at least an object ID (typical- ly an alphanumerical code) and a representative high resolution appearance in- formation is stored. The high resolution appearance information may be vector- ised, comprising a plurality of anchor points, lines, polylines, polygons or curves connecting the anchor points, and colour information for the lines, curves and/or surfaces enclosed by the lines and curves. If a reference object sufficiently matches with a structure contained in a high resolution image frame, then an object is identified in the high resolution image frame. An identified object is al- located the object ID of the matching reference object and relative object image insertion properties, including the image frame number in which it was identified, an insertion vector point ("center of gravity" of the identified object), and scal- ing/rotation parameters. The scaling/rotation parameters typically include a scal- ing value (the size of the identified object, typically relative to a stored standard size of the reference object), a first rotation value (rotation with respect to a first axis, e.g. a vertical axis) and a second rotation value (rotation with respect to a second rotation axis, e.g. a horizontal axis). The object IDs and relative object image insertion properties of identified objects are transmitted via a second data channel from the on board ADS unit to the control centre via the mobile commu- nication network. Transmitting this information requires only few bandwidth, too.

From the second data channel, the (safety relevant) identified objects can be reconstructed into high (pixel) resolution image data at the control centre. For this purpose, the control centre also comprises a map data base with reference objects, corresponding to the reference objects stored in the map data base of the automatic train. In other words, the map data bases of the automatic train and the control centre contain basically the same information on reference ob- jects. Based on the high resolution object appearance information contained in the map data base of the control centre, the high resolution object appearance for each identified object can be reconstructed for each image frame, based on the transmitted respective object ID, insertion vector point and scaling/rotation parameters, and the image frame number.

The reconstructed high resolution object appearances, based in the second data channel information, can then be overlaid into the low resolution image frame, received via the first data channel, with the corresponding image frame number. The resulting overlay image frame then contains high resolution image infor- mation where there are safety relevant identified objects, and low resolution im- age information where there is only landscape irrelevant for train safety. In this way, for example railway signals including their switching state which are recon- structed via the second data channel and shown in high resolution, can easily be recognized by the remote driver, while at the same time, the remote driver also gets a coarse impression of the landscape at low resolution.

It should be noted that the low resolution image frames may be interpolated to the high resolution of the high resolution object appearance at the control centre for simplifying the overlay procedure, in accordance with the invention, but this does not increase the information depth received via the first data channel.

The map data base of the ATO trackside rail traffic operation and management centre and the map data base of the automatic train are regularly synchronized, such that they both contain basically the same reference objects ("common ref- erence objects"). Note that in the most simple embodiment, the map data base does not require georeferencing information, although georeferencing infor- mation is useful for plausibility checks (see further below).

Further note that for remote control of the automatic train, the ATO trackside rail traffic operation and management centre and the automatic train are connected via the management centre mobile com router and the automatic train mobile com router via the mobile communication network, which may be a railway spe- cific mobile network or a public mobile network, or a combination of networks, for example GSM-R or 5G, 4G or LTE. Communication includes, besides image information, e.g. also driving commands and sensor readouts.

Preferred variants of the invention

A highly preferred variant of the inventive method is characterized in that in a third processing line,

- the high resolution image frames are subjected to a vectorization algorithm, which allocates vector elements, such as points, polylines or polygons, to the high resolution image frames, and determines corresponding vector properties of each allocated vector element, including an image frame number, and

- determined vector properties for each allocated vector element are transmitted via a third data channel from the on-board ADS unit to the train remote control console via the mobile communication network, that at the ATO trackside rail traffic operation and management centre, the vec- tor properties of the allocated vector elements of the third data channel are translated into vector element appearances, and that the vector element appearances originating from the third data channel are used for plausibility checks of and/or are included in the overlay of the high resolution object appearances originating from the second data channel and the low resolution image frames received via the first data channel with matching image frame number.

In this variant, high (pixel) resolution image frames undergo a vectorization al- gorithm at the on-board ADS unit in a third processing line. In other words, structures found in the high resolution image frames are abstracted. Note that this abstraction does not require defined reference objects such as railway sig- nals or railway signs, but may be based on general geometry, above all poly- gons. Accordingly, the map data base is not required for this third processing line. The abstraction results in the allocation of vector elements, in particular points, polylines and polygons. The polygons may contain a colour information, in particular for the polylines and for surfaces enclosed in polygons. The infor- mation necessary for reconstructing an allocated vector element is/are the corre- sponding vector properties; these include the image frame number of the frame in which the allocation was done, and information of the location and/or orienta- tion and/or size and/or colour of points and/or lines and/or surfaces included in the vector element. Typically, the locations of all points included in a vector ele- ment are provided, and lines are defined by their end points, and surfaces are defined by their bordering lines, and for each line and surface a colour infor- mation is provided.

The determined vector properties are transmitted from the on-board ADS unit to the control centre via the mobile communication network via a third data chan- nel, which requires only few bandwidth. At the control centre, the vector element appearances can be reconstructed based on the transmitted vector properties. The vector element appearances can be included into the image (i.e. the overlay image frames) of the track section ahead of the train shown to the remote driv- er, if desired. For example, an average of the reconstructed vector element ap- pearance data and the reconstructed high resolution object appearance data can be overlaid on the low resolution image frames of the first channel. Note that due to its vector type nature, the vector element appearance is intrinsically of high resolution, although of course it is still the result of an abstraction of the original (high pixel resolution) camera video data. Image details may remain visi- ble/recognizable in the vector element appearances that do not remain visi- ble/recognizable in the low resolution image frames. It is also possible to use the reconstructed vector element appearances for a plausibility check of the overlay image frames resulting from the first and second data channel information. If there is no sufficient match of the vector element appearance (based on the third channel data) and the overlay image (based on the first and second data channel data) or tits source data, the remote driver can be warned that no reliably image information is currently available. In both cases, a higher reliability or trustwor- thiness of the (regular) overlay image frames or of the train safety may be achieved.

An alternative, also preferred variant is characterized in that in a third processing line,

- the high resolution image frames are subjected to a vectorization algorithm, which allocates vector elements, such as points, polylines or polygons, to the high resolution image frames, and determines corresponding vector properties of each allocated vector element, including an image frame number, and

- determined vector properties for each allocated vector element are transmitted also via the second data channel from the on-board ADS unit to the train remote control console via the mobile communication network, that at the ATO trackside rail traffic operation and management centre, the vec- tor properties of the allocated vector elements of the second data channel of the third processing line are translated into vector element appearances, and that the vector element appearances originating from the second data chan- nel of the third processing line are used for plausibility checks of and/or are in- cluded in the overlay of the high resolution object appearances originating from the second data channel of the second pro-cessing line and the low resolution image frames received via the first data channel with matching image frame number. This variant corresponds to the previous variant, with the only differ- ence that the determined vector properties are sent to the control centre or its train remote console also via the second data channel, instead of a third data channel. In this way, the third channel can be done without; instead the second data channel transmits some more information, but still, only few bandwidth is required by the second data channel.

Another preferred variant is characterized in that the relative object image insertion properties of each identified object also include a georeference, that the map data base of the ATO trackside rail traffic operation and manage- ment centre and/or of the automatic train further contains a register of known objects defined by a georeference and an attributed reference object in each case, and that the ATO trackside rail traffic operation and management centre and/or the on-board ADS unit check the allocation of the identified objects to the refer- ence objects for plausibility based on a comparison of the georeferences and al- located reference objects of the identified objects on the one hand, and the georeferences and attributed reference objects of the known objects on the other hand.

In this variant, the map data base at the automatic train and/or the control cen- tre also contains georeferences (geographical locations) of known objects and their object type (attributed reference object, typically represented by the object ID); the map data base may also contain some further information on the known object such as its orientation in space or its size or a unique name, if desired. Objects identified in the second processing line further get attributed their georeference, i.e. their geographical location where they were found. The geo- graphical location of an identified object can in general be calculated from the (usually well known by GPS) geographical location of the automatic train at the time of the camera recording, preferably corrected by an estimated distance of the object from the train for better accuracy.

An identified object is plausible if it sufficiently matches with a known object in a map database; a match typically can be confirmed if the georeferences of the identified object and of the known object differ less than a threshold (e.g. less than 3 m, depending on the accuracy of the train and object localization) and if the object IDs of the identified object and the known object are identical. Note that in case of objects that may have different switching states (such as track switches or railway signals), all possible switching states are preferably included in the map data base as separate known objects with the same georeference, for simple verification of matching. Alternatively, only one known object with the object ID belonging to a particular switching state of the object may be stored for the corresponding georeference, and an identified object with allocated object ID belonging to any one of the possible switching states of the object is consid- ered as compatible for approving a matching.

If for an identified object a matching known object has been found, and the known object has attributed a unique name (such as a signal number), the unique name of the known object can be displayed to the remote driver. If no matching for identified objects can be found in the register of known objects, then the remote driver can be warned that no reliable overlay image data is available. Note that typically a matching of identified objects and known objects is only verified (or required to avoid a warning message) for particular types of identified objects, usually those of particular relevance for railway safety, e.g. railway signals. By checking the plausibility of the allocation of identified objects, the reliability of the second processing line or the image data based on the sec- ond data channel or the train safety may be improved.

In another preferred variant, the ATO trackside rail traffic operation and man- agement centre checks the allocation of the identified objects to the reference objects for plausibility based on a correlation of the high resolution object ap- pearances and corresponding object appearances in the low resolution image frame for matching frame numbers. If there is no minimum conformance of the high resolution object appearances and the corresponding structures in the low resolution image frame, the object allocation is questionable, and the remote driver can be warned that no reliably overlay image is available at the moment. With this variant, the reliability/trustworthiness of the overlay image frames or the train safety may be improved.

Further preferred is a variant which is characterized in that the on-board ADS unit and the ATO trackside rail traffic operation and man- agement centre each comprise at least one UTC clock, preferably two UTC clocks, for providing UTC time, that the communication between the automatic train mobile com router and the management centre mobile com router via the mobile communication network is packet-based, wherein each data packet is provided with a time stamp in a header section of the packet, indicating the UTC time of the packet being sent, that a handshake protocol flow notes UTC times as follows:

Tis UTC time when packet was sent, T _1C confirmed UTC time when packet was received, T _2S UTC time when acknowledgement was sent, and T _2C UTC time when acknowledgement was received, that a round trip delay ΔT _RoundTrip is determined, with ΔT _RoundTrip =T _1C -T1 _S +T _2C -T _2S , that an average round trip delay ΔT _RoundTrip _ _AVG is determined over a plurality of past packets, and that a warning message is delivered if ΔT _RoundTrip -ΔT _{RoundTrip_AVG} >T _Threshold ^RT , with T _Threshold ^RT : Threshold value for round trip delay, in particular with T _Threshold ^RT = K _conf *σ(ΔT _RoundTrip ), with K _con f: confidence estimation value; s: standard variance. If the data necessary for train control, in particular the image data belonging to the track section ahead of the train, cannot be de- livered in real time synchronization, then the remote driver may not be able to react in a timely manner in order to ensure train safety. Applying the above vari- ant based on a round trip delay, the delays in information or command transfer can be determined reliably. In case of uncommon or inacceptable delays, the remote driver can put the automatic train into a less dangerous state, for exam- ple by reducing the train speed or even stopping the train. The warning message is typically delivered in the display of the train remote control console, e.g. by greying out the displayed image of the track section ahead, in particular if pack- ets belonging to the first data channel or the second data channel are concerned. If the threshold value for round trip delay is compared to the standard deviation of past round trip delays, common delays can be distinguished from uncommon delays more easily. Alternatively, applying a fixed threshold for round trip delay ensures a desired absolute safety level.

A preferred further development of this variant is characterized in that a clock difference ΔT _clocksync is determined, with ΔT _clocksync = 0.5 * (T _1C -T _1S +T _2S -T _2C ) , that an average clock difference ΔT _{clocksync_AVG} is determined over a plurality of past packets, and that the warning message is also delivered if ΔT _clocksync - ΔT _{clocksync_AVG} >T _Threshold ^CD ith T _Threshold ^CD Threshold value for clock differ ^¬ ence. If the UTC clocks on the automatic train and in the control centre are not properly synchronized, this can obscure and aggravate the identification of de- lays of data transmission. By checking the clock difference, asynchronous UTC clocks can be identified, what improves the train safety. Further within the scope of the present invention is a system for safe train re- mote control, comprising a) for arrangement on an automated train:

- an on-board ADS (= automatic driving system) unit, including a camera system for monitoring a track section ahead of the automatic train, and

- an automatic train mobile com router for forwarding and receiving data via a mobile communication network, and b) for arrangement in an ATO (= automatic train operation) trackside rail traffic operation and management centre:

- a train remote control console for remote control of the automatic train, includ- ing a display for showing an image of the track section ahead of the train to a remote driver at the train remote control console, and in particular including a moving map display for indicating a train position of the automated train (ATR)and

- a management centre mobile com router for forwarding and receiving data via the mobile communication network, characterized in that the system is adapted for performing an inventive method as described above. The inventive system allows a safe remote control of an au- tomated train via a mobile communication network, requiring only few band- width. More specifically, an image or visual information of the track section ahead can be provided at the display of the train remote control console, con- taining both high resolution details on safety relevant objects ahead (such as railway signals and railway signs) and a low resolution overview on further envi- ronment ahead (such as trackside vegetation and buildings).

Further advantages can be extracted from the description and the enclosed drawing. The features mentioned above and below can be used in accordance with the invention either individually or collectively in any combination. The em- bodiments mentioned are not to be understood as exhaustive enumeration but rather have exemplary character for the description of the invention. Drawing

The invention is shown in the drawing. Fig. 1 shows a schematic overview of a train remote control setup, for the invention;

Fig. 2 illustrates schematically ATO / On board trackside remote logical functions, for the invention;

Fig. 3 illustrates schematically an embodiment example of a typical hard- ware implementation, for the invention;

Fig. 4 illustrates schematically an example of train authentication, for the invention;

Fig. 5 illustrates schematically an example of channel processing, for the invention; Fig. 6 illustrates schematically an embodiment example of a train remote control display;

Fig. 7 illustrates schematically a preferred variant of the inventive method of safe train remote control, in a first part at the automatic train;

Fig. 8 illustrates schematically the variant of Fig. 7, in a second part at the control centre;

Fig. 9 illustrates schematically a data transmission of packet in a preferred variant for the invention;

Fig. 10 illustrates schematically a preferred variant of delay verification of packet transmission for the invention. Overview

The invention relates to apparatus and methods for Safe Train Remote Control, in particular based on time sychronized video streaming. More specifically, the invention relates to apparatus and methods of safe train remote control of a semi or full automated train coping with limited bandwidth of a mobile communication network.

In case the automatic trains have to be supervised and controlled temporarily in specific situations, the automatic train shall allow for recovery situations to be taken over from a remote control station. The inventive remote control system will cope with public mobile networks, even when some temporal bandwidth limitations apply and still work safe and with high availability. In additon the remote train driving is seen as introduction of automatic train capabilities GoA4.

The invention includes in particular a method for a remote control device for an automatic train that has the following characteristics:

- The video images are streamed from the train server via mobile data link such that the video I-frames are time tagged and geo-referenced in the addi- tional header bytes of the streaming protocol.

- Dynamic mobile bandwidth limitation response such that on at least two chan- nels connecting the automatic train with the remote unit, and one channel is us- ing reduced spatial resolution and compression, and the other channel is using a vectorization and symbolization of objects, areas and points of interest, and both channels are united by seamless overlay such that a realistic impression is given to the operator.

- A safe and robust time synchronization method between remote device and automatic train by measuring message send and confirm times such that any information freeze, including but not limited to video streaming or control pack- ets, are reported to the operator within a time to alert.

A safety relevant remote connection monitoring is given that monitor the rele- vant air gap parameters and assures integrity by online tests. The invention in particular provides a safe and user friendly remote display and control unit.

Intended Use and Context

The subject of invention is part of the rail traffic operation infrastructure that al- lows automatic train driving up to grade of automation 4, which is defined as un- attended train operation including all functions of train operation are automa- tized.

The invention focuses mainly on the functionality of remote control and driving.

In essence the remote train control capability is intended to be used by an agent of the rail traffic operation and management centre for example in case the au- tomatic train raises an alert e.g. w a situation is recognized that the train cannot handle in automatic mode. In addition control in case of emergencies or degrad- ed functional train systems is also subject to human intervention by the remote driving capability. Further examples are a routine automatic train supervision check or start of mission preparation check. The train remote control capability is intended to be used to get a train safely to the end of journey, instead of a driver entering the train and continuing the journey manually. Additional example the intended use is that a train can be started remotely for a dedicated operation sequence, such as a start of mission after passengers have boarded the train un- til the predefined operation sequence is done, such as stop at the next station.

A typical automatic train drive control system including the necessary infrastruc- ture is summarized as follows in Fig. 1: The ATO trackside rail traffic operation and management center includes a train remote control console. This console allows to control a train remotely via a mobile communication network. In the train, as inherent function of the automatic driving system, the remote control capability is applied. The train remote control function is partitioned in two com- ponents, one on the train on-board unit, one in the ATO trackside rail traffic op- eration and management centre. The summary of high level functions for the train remote control is given below: a) Log-In and Authentication for take-over remote train control or release remote control; b) Remote access to train driving functionality; c) Supervise actual train status as well as train diagnostic interaction capability; d) Safe, reliable and real-time communication, radio links and networking man- agement between train and remote control console.

On-board Remote Train Control

The train remote control function is partitioned in two components, one on the train on-board unit, one in the ATO trackside rail traffic operation and manage- ment centre. Fig. 2 shows the two main components, with typical functional en- tities.

On the on-board automatic driving system the following components are typically included:

O-A) On-Board Assignment & Train Automatic Driving Controller This on board function receives the journey assignment from the ATO trackside, triggers the execution and ensures that the operations defined in the assignment are correctly scheduled and accomplished. The component also sends assign- ment execution reports to the trackside supervision. In the remote control mode, the automatic driving system is disabled and the remote control access is ena- bled.

O-B) On-Board Manoeuver Controller:

Component has the subfunctions Time Table Speed Optimizer, which establishes the optimum speed to achieve the platform stopping or passing points on time under constraint of optimal energy usage. Safe Profile Speed Optimizer estab- lishes the maximum speed the train can run without interfering with the ETCS speed limits. Automatic Stopping Point Controller establishes the speed profile to stop the train accurately at the Stopping Points. The calculation of the final speed curve considering the timing points, signaling, max speed, obstacles, environ- mental conditions is consolidated.

O-C) On-Board Movement Protection Controller This component is responsible for protect the train against non-authorized movement, against danger points and overspeeding. This component also en- sures: a safe opening authorisation and closure of the doors and a safe coupling or uncoupling of train consist. O-D) On-Board Incident Prevention Controller: The subfunction monitors and evaluates the journey travel situations according to the rail rule set in particular for signaling and sign actions. The subfunction identifies overall status and then initiates the required alert measures. O-E) Perception Controller accumulates identification and properties of Lateral Signal Monitoring, Obstacle Monitoring, Collision Monitoring, Platform Monitoring and Coupling Monitoring.

O-F) On-Board Train Interface Controller: generates the ATO Output commands including ATO Traction / Brake Control, and various action commands including but not limited to emergency brake, door control, catenary control, front/end light switch, coupling control, passenger carriage control. In addition the sub- function reads the status of the train and the train messages. The function also provides interface to legacy train control systems such as NTC, ETCS.

O-G) On-Board Train Maintenance Controller: diagnostic interaction, collects the detailed status information of all subcomponents, provides juridical records and supervises actual train status. On request the various information records are given to the On-Board Reference Data Management & Server.

O-H) On-Board Reference Data Server & Remote Controller is processing and managing the data flow between on-board and trackside as well as for on-board functionalities. The function includes also configuration data and map data man- agement. In addition login, communication setup and authentication as well as encryption are supported for the on-board side. The remote command handling and data exchange is accomplished. O-I) On-Board Positioning and Speed/Distance Measurement. The subfunction provides train position, direction as well as speed including safe confidence inter- vals.

Trackside Remote Train Control

The remote console allows accessing the train remotely.

T-A) Remote Mission Manager provides the input controls for the remote train driving capability including to select, start, abort a train journey or execute semi- automatic driving sequence. In addition the component translates the input ac- tions into commands. The functionality supervises the execution of the mission.

In addition it provides Log-In and Authentication for take-over remote train con- trol or release remote control. The Mission Manager allows to enter and handle the various configuration parameters, such that the applicable subset is synchro- nized between trackside the on-board train, including but not limited to the track map data base.

T-B) Remote Train Control Manager provides low level access to train driving control bypassing the automatic driving system. The barking force and the trac- tion force input are directly accessible via controls such as a throttle lever. In addition commands for Start-Speed to End-Speed ramp, Cruising Speed, Mini- mum Coasting Entry Speed, Maximum Coasting Speed are given. Control buttons for motor start/stop, horn, wiper, light, air-conditioning, catenary control, front/end light switch, door switch, or emergency stop etc. are available as soft- controls or lever switches. In addition the component translates the input actions into commands; depending on the train model this functionality is configurable to adapt to the TCS- interface. T-C) Remote Diagnostic Manager requests actual train status as well as train di- agnostic information, stores a history and displays the information in the appro- priate menus. Logging and monitoring of operating command actions and train state processes. Integrate and analyse information of other on-board sources such as obstacle detection subsystems, lateral signal recognition, passenger in- formation subsystems.

T-D) Remote Train Location Map Manager displays display train position infor- mation with full trackside visual information within the context of a track map situation.

T-E) Remote Streaming Control Manager provides starting, synchronization and image joining functionality for the different transmission channels.

T-F) Communication Client and Exchange Protocol Manager

The subfunction includes protocol protection of a safe a secure protocol sent via

2 channels simultaneously and received with valid cryptographic protection such as but not limited to message digest hash protection (MDx) and or cyclic redun- dancy check protection and repetition request/handshake and sequence number- ing and confirmed sequence number, time stamp and confirmed time stamp. The message validity check will include

Check of cryptographic protection safety code Check of authentication of the message Check of message type Check of sequence number Check of timeliness

The redundant train computer units (e.g. three units) will exchange the control messages such that each computer has a tuple of at least 6 messages in case of

3 units and 2 channels. A voting algorithm on each computer will compare the messages and vote, in case of dissimilarity, which message failed, if any. The failure result is also exchanged by the redundant train computers, and any failed message is voted out. Data Channel and Exchange Protocol between ATO-On-board and ATO-Trackside

For example it is a risk, if a remote driver is acting on the control elements, with a drivers cab video picture that is stale and does not represent the current train drivers front view scene. Same holds true for stale states of display elements, which may cause a wrong reaction of the train driver. Therefore safe streaming is mandatory for the application of remote train control.

One key feature of the remote console ATO trackside is the monitoring protocol data exchange on the basis of streaming. The Fig. 3 summarizes, where stream- ing data is exchanged from various optical systems that monitor signaling, ob- stacles, collision events, environment, platform, on board and passenger situa- tions up to rolling stock and coupling manoeuvers. Safe operation can be only achieved by time synchronized, low latency, and reli- able information streaming over communication networks. The redundancy of multiple physical channels allows comparing the information of one channel ver- sus the other channel. However sending the same information across dual chan- nels as well as channel switching from hot to standby on networks such as GSM- R, 3G to 5G public mobile are well known techniques.

Because the video streaming information of all sensors requires a high band- width, which may not fit into a single channel, the information of the redundant sensors is apportioned to the channels such that one sensor is associated with a set of channels and the other redundant sensor is associated with the other set of channels. Streaming refers to purpose of timeliness of information delivery rather than storing of the information. The remote control includes a communica- tion protocol such that provides confidentiality, message authentication, and re- play protection as well as real- time capabilities for the video stream and the control messages. Because the ATO trackside remote train driver console needs to communicate with various types of automatic trains a standardized protocol will be used, typically based on SRTP. It is assumed that the communication is packet based. A multimedia session is defined as a collection of sessions that may include video session, data session, control command session. The payload data stream of any multimedia session is packetized and encrypted. The process of encrypting a packet consists of generating the key stream segment corresponding to the packet, and then bitwise exclusive-ORing that key stream segment onto the data payload of the packet to produce the Encrypted Portion of the packet. Replay Protection is achieved by receiver maintaining a replay list, which contains the indices of all of the packets which have been received and authenticated.

The method includes that one video picture is sent in one or more packets. The packets exchanged between the ATO-On-Board and the ATO-Trackside includes a header containing the following information, besides typical data such as packet number; packet length also a timestamp identifying the time at which the packet is sent. Besides the time the packet was sent, the integrity can be enhanced, by End-to-End time synchronisation with time lag control of video stream between ATO on board camera and ATO trackside remote control console display. The video images are streamed such that the video I-frames are time tagged and georeferenced in the additional header bytes of the streaming protocol. Addition- al frame synchronization in the form of the extension of the one-byte header is added to the first packet, which allows up to 15 additional bytes. The header ex- tension byte containing an ID and a length is used. These following bytes are used to code the UTC time, when the data is measured. The UTC time is coded as a counter in milliseconds. The static identifier of the I-frame adds to the ro- bustness of the video decoding, because the I-frame is a key frame on which all other P and B type frames refers to. These bytes are also used for georeferenc- ing the video-image. For example 7 bytes UTC time and 8 bytes georeferencing including x and y coordinate of the pixel (0,0) left upper corner. A scale factor is used for the georeferencing in WGS84 latitude and longitude coordinates. Lat _SCaied = f _s ^* Latp _acket · The video camera properties relevant for image processing are fixed configuration parameters, known to ATO-trackside as well as ATO on board in- cluding but not limited to pixel scale factors for width and height, ATO on-board camera mounting position, rotation and focal length. Safe data connection with dual streams for low rate control data with secure pro- tocol. For video data compression depending on train speed the number of key- frames (I frames) is variable. Each video frame is enhanced with a pattern that correlates time coding from a global time source and frame number. On the re- mote display each video frame number is decoded and compared to the global time source such that the difference of time of transmission delay relative to the video captures time and the ATO trackside system time. With such a method the video delay tolerance can be monitored.

The time coding by the header extension as well as the time of packet sending can be compared to the real time clock of the ATO-Trackside computer, where the real-time clock is also synchronized on UTC time. The time difference of the video packet header time stamp and the real time clock is compared to a thresh- old as is the time stamps of the video picture take. Given the threshold is ex- ceeded from both, sending time stamp and camera time stamp an alert indica- tion is raised. This double real-time check on the packet level is much more effi- cient and safe, than indication of an empty streaming buffer.

Physical Laver of the Train Remote Control

Various sensors are connected to the On-board ADS computer including camera, positioning sensors, etc. The physical embodiment of the communication be- tween ATO-On-board and ATO trackside remote train control network is based on mobile com routers. The router forwards data across shared or public networks via any type of network channel or any combination of channels such as but not limited to GSM-R, Mobile Telecom 5G or 4G or LTE or W1_AN.

The bandwidth and the round trip delay are online measured in all channels, and the choice of routing is dynamically adjusted according to bandwidth. In particu- lar the major part of the bandwidth is consumed by video information. Therefore a balance is necessary, between optimal choosing the channels and reducing the bandwidth for data transfer between train ATO On board and ATO trackside re- mote control console such that nor integrity, nor operation is compromised.

Takeover or release remote train control

As given in Patent EP 3 220 613 Al, if an ATO on-board ADS initializes the com- munication in the packet based communication network, the ADS will publish its IP address, the symbolic mission name (train running number may be a part of this mission name) and the train ATO-On-Board hardware ID. Same procedure is used by the ATO-Trackside remote console, publishing fixed IP for each remote driver place including a pair of destination transport addresses. It is assumed that the IP communication setup can be established by known IP addresses or, given a DNS service is available, also via name resolution. The takeover by the remote control will typically include the selection of a specific train ID or mission name as well as entering various authorisation data in the train remote control console of the ATO rail traffic operation and management centre including but not limited to user credentials, mission authorisation and train authorisation.

For the packet based protocol, one preferred embodiment is to extend a public standardized protocol such as Real Time Streaming Protocol (RTSP) in order to enhance the integrity of the data transfer. However the method can also be ap- plied to a proprietary protocol. For the ATO-On-Board and ATO-Trackside session setup the keys need to be exchanged. The protocol typically includes a master key, from which session keys are identified. Additional authentication shall be successful, before the data encryption on the sender side and reversely on the reception side does take place. The RTSP exchanges are encrypted to protect keys exchange itself. If the media server detects that encryption is activated (based RTSP parameters in the URL), it sends the key management information and then the encrypted stream. Fig. 4 illustrates an example of train authentica- tion.

Switch and Mobile Air Gap Router Remote air gap information shall be available for diagnosis the connection be- tween ATO-on-board and ATO trackside such as SSID modes, IP connections, addresses, operator state, connection type (LTE, 5G), channels, International Mobile Equipment Identity, International Mobile Subscriber Identity, Cell ID, ac- tive protocols and services, Authentications Status, etc. An event log will allow checking for relevant connection or configuration events.

Time synchronisation between the ATO trackside remote control console and the ATO-on-board train is key constraint for a reliable and safe control loop. Time lags introduced in the control loop by insufficient transfer bandwidth or insuffi- cient time synchronisation between remote station and train may cause control instabilities or safety risks. A safe time base is established on the ATO-on-board with at least two independent computers that have a stable UTC clock, as pre- ferred embodiment, this could be GNSS receivers or computers connected to GNSS receivers that have time synchronization access better than 100 microsec- onds. In the data exchange between ATO-on-board and ATO-trackside remote control console, preferably for control messages, where an order- acknowledgement protocol is mandatory, the round trip time is measured to make sure that the round trip delay is smaller than a certain limit that depends on the control loop filter time constant. If the round trip time limit or the clock sync limit is exceeded an alert is given, because safety may be compromised.

The packet round trip time is continuously determined the data by 2 independent primary stratum UTC synchronized train computers that communicate via at least 2 data channels of mobile communication network and are received from at least 2 independent remote control console computers. For each time epoch at least 2 ^L 4 round trip delays can be determined.

The on-wire protocol header addition uses 4 time stamps that are embedded in the general protocol flow of order/request sent by one unit and re- sponse/acknowledge by the other unit, the round trip time is determined by both train ATO on-board component as well as ATO trackside remote control console. Each message carriers the time stamp and the confirmed time stamp in the mes- sage header. The time stamp Ts is a four byte value, which holds the sender's time stamp of the message. The confirmed time stamp T _c is a four byte value, which reflects the partner's time stamp of its last (youngest) valid message.

Tis is clock time, when packet was sent, Tic is confirmed clock time, when re- quest/order packet was received, T _2S is clock time, when acknowledgement was sent and T ₂ c, when ACK was received and confirmed.

The four clock time measurements are used to round trip delay ΔT _RoundTrip =T _1C -T _1S +T _2C -T _2S

The systematic part of the round trip delay is removed with a moving window average. The threshold can be determined by confidence estimation. An alert is raised, if the round trip threshold is exceeded: ΔT _RoundTrip - ΔT _{RoundTrip_AVG} > T _Threshold ^RT = Kconf Sigma(ΔT _RoundTrip )

The clock difference of the trackside and the on-board is determined as: ΔT _clocksync = 0.5*( T _1C -T _1S +T _2S -T _2C )

An alert is raised, if the sync threshold is exceeded: ΔT _clocksync - ΔT _{clocksync_AVG} > T _Threshold ^CD

Besides round trip delay, further statistical properties of the data channels shall be monitored, such as but not limited to bandwidth-usage (RX rate, TX rate), bit errors, reference signal received power, receive strength signal indica- tor/reference signal received quality, signal to interference plus noise ratio. The monitoring of these indicators is done on a statistical basis, with a sliding window limiting the length of the sample set, which is reset on each new mobile cell takeover. The central moments such as mean, variance and skewness are tracked and test statistic such as for a hypothesis test can determine any failure or degradation of the remote wireless connection. The integrity is represented by a confidence interval of a Gaussian probability density distribution, that is based on the statistical properties of variance an mean moment as well as a confidence inflation coefficient KFF, is linked to the Gaussian distribution and is computed using the Quantile QN function of the associated integrity risk.

Given the air gap remote channel characteristics are in the tolerance the channel is set healthy and can be used. A counter checks, if any channels are empty for a certain time and raises an alert, given the time limit is exceeded.

Video Image Processing

The Data Reference Server & Remote Controller function is processing and man- aging the data flow between on-board and trackside as well as for on-board func- tionalities. The data flow includes the overall data management including stream- ing video data, audio, status or command data. This function takes care of in- formation flow distribution designated to the trackside entities including the re- mote train control console.

Safe video processing and compressing constitutes one key function of the Data Reference Server & Remote Controller. Precursor for streaming media data is the data compression by reducing the resolution and color depth. The image pixel stream of the camera is then compressed by state of the art methods to remove redundancy on the image level such as run length encoding or discrete cosine transformations. In the next step the image is compressed by a codec-encoder that covers the timely inter-image redundancy. Long term prediction coded in the P- and B-frames compresses changes with reference to the intra frames (I- frames).

For further reduction of the bandwidth the inventive method is proposed: Mobile bandwidth limitation response such that on at least two channels connecting the automatic train with the remote control console can cope with bandwidth limita- tions such that one channel (named channel 1 here) is using reduced resolution and the other channel (named channel 2 here) is using a vectorization and sym- bolization of objects, areas and points of interest. Both channels carrying dissimi- lar information are united by seamless overlay such that a realistic impression is given to the operator (see Fig. 5).

The train on-board map as well as the remote control console map inherits iden- tical layers, where operational relevant points of interest (signals, signs, catena- ry-pylons, etc.) are geocoded. In addition the method includes map data base part, where besides object vector features, a typical object appearance image is stored. In channel 2 only the object ID and the relative object image insertion properties need to be transferred such as frame number, insertion vector point, georeferencing and scaling/rotation factors.

After reception of the object ID in the ATO-Remote control console the typical appearance object image is correlated to the expected insertion area of the video image, where geocoding and insertion vector of the object is guiding the ex- pected insertion location. This high resolution object appearance image from the map data base is overlaid into the low resolution video image such that disad- vantage of reduced video image quality is compensated by overlaying the object typical appearance pixel areas. This is important for example to detect the signal light status, where a high resolution is necessary. For vegetation image a low resolution is still acceptable without compromising safety or operational deci- sions. In addition the vectorized symbol from the map database is overlaid on top to display the relevant items highly prominent.

In the ATO trackside remote train control component, the overlay can be pro- cessed by resampling the low resolution video image to high resolution and then joining the overlay object appearance images. The image merging operation in- cludes options such as multiplying the overlay object appearance image intensity values with the video intensity values, adding both intensity values, averaging the values of the images or the max contrast joining. In one embodiment, for dark areas of the video image, the pixel choice is based on the minimum of the video image and the object appearance image and for bright areas; the maxi- mum intensity of the object appearance image is taken for the overlay.

In order to identify known objects video segmentation preprocessing on the train video images has to be prepared. The classification of different objects in the im- age is amongst the steps in the processing chain. The features of the image ob- jects are computed, and then compared those using detectors /feature classifi- ers. Feature extraction reduces the information to be transferred by extracting regions, edges, and corners. Successful object identifications are marked with a bounding box and an object ID. In the example Fig. 5 and Fig. 6 of the remote camera front view, the track objects are overlaid, a train on the opposite track is identified and a person ahead of the front track is identified.

As integrity of the graphical elements a validity code (cyclic redundancy check or comparable) is added at the generation source, which is including the time in- formation and in case of video images the frame number and the ID number, which is corresponding to the common map data base between the ATO/track- side and ATO on-board, insertion vector, scaling/rotation factor. On the remote display the pattern is decoded such that integrity is validated as well as the time difference is compared to the real-time clock.

The ATO on-board and ATO remote control map data base has at least two main parts, vector data and reference symbol icons. In the vector part: The vector map part includes rail infrastructure areas such as restriction areas, stopping ar- eas, level crossings, tunnels, bridges, platforms, vegetation, etc. The data base information for those infrastructure areas include at least an ID, a bounding box (width, height, length), track under infrastructure, line name, associated proper- ties and a link to a pixel image, that represents a typical area or the specific area including object area reference point (centre of gravity CoG), object pixel height/width. The area geometry is represented by a vector element such as point, ploy line or polygon area. The geometry type and the related number of geometry coordinates (e.g. latitude, longitude, altitude or segment, abscissa, displacement) are stored in the data base. A polygon is defined as a connected and closed sequence of four or more points, a ploy line is defined as a line se- quence of 2 or more points. In addition the data base map stores track assets, such as but not limited to signals, signs, fouling mark, pylons, balises. For the balises, besides the properties mentioned above, an extended linkage infor- mation is stored, that includes the vector between balise and track segment origin as well as BG identifier NID_BG; position of the balise in the balise group; arc length from segment start.

In addition the data bases stores the track geometry as a series of points, or higher geometrical abstractions such as but not limited to polylines, curves, clo- thoides, splines, etc. Due to the fact that the track location needs higher geo- metrical precision, the properties may include in the preferred embodiment for example point ID, associated segment ID, line ID, point heading, point slope, point cant. Depending on the chosen geometry type representation, the proper- ties may include center point and radius, or start point endpoint, etc. The topolo- gy of the track is represented by edges, and edge connections that represent track elements such as switches, buffers or junctions.

The second part of the map data base contains reference symbol icons with dou- ble layer, one layer as vector layer and one layer as pixel layer, both represent- ing the same object. Those symbol icons may be stored specific for the country or region or journey. In addition the data base contains other configuration in- formation such as but not limited to lever arms, system properties, IP-addresses and so on.

Plausibility checks are part of the checks, including if the map data base object type and location matches the location and identified type of the video image recorded object. In addition correlation of the object reference image and the low resolution pixel image are a means to check, if the object identification was cor- rect. Integrity is achieved by calculating the correlation between the icon pixel image and the low resolution video image on the remote side, over all pixel width and height. Given the correlation peak is inside the object bounding box, the process was matching, otherwise alert is given. In addition the reference image can be reduced to the object shape by image processing techniques (e.g. sobel operator) and matched to the vector shape of the object. All those checks are aimed to detect failures. Given all three data sets match (low resolution, data base reference image and vector image) or at least two out of three match, the transmission content is validated.

Remote Identification of Train Positioning Status

In case the train remote control function is used, the operator needs to know where the train actually is on the track network as well as where the train did identify its position. In case of a system failure, those two positions do not al- ways match. The remote control functions provide visual indication of the train front view as well as a moving map view. The remote operator can identify, if discrepancies appear for significant landmarks. The train position is graphically depicted on the moving map as a symbol. Further on the moving map are includ- ed relevant objects with associated labels such as balises, track segment board- ers, signals, signs, and landmarks. In addition segment ID and driven distance from segment ID from the calculated train position is also displayed in order to allow a crosscheck for the remote operator if this is the place where the train is expected to be according to the journey profile.

The ATO train positioning is typically referenced to track segments origins in- stead of balise group as per ETCS. Hence the driven distance of the train is given with respect to the segment profile identifier, segment profile version and seg- ment profile travelling direction. In one embodiment each track segment identity is defined by Identity of the SP's country or region (NID_C (k)) as well as Identi- ty of the requested Segment Profile (NID_SP (k)) along with a status ("Valid": SP requested "Invalid": SP not found in ATO-Trackside database) and version num- ber. Each segment is characterized by a Length of the segment of railway cov- ered by the segment profile L_SP (k) as well as altitude as reference at the be- ginning of the SP and a gradient G_New_Gradient (k), defined as value of the new gradient at the beginning of the Segment Profile. Other embodiments may use only a subset of the properties or additional properties. In order to cross- check this information to achieve a certain level of safety, the segment profile location needs to be also defined by a geographical position for example in lati- tude, longitude and altitude.

Positions inside the ATO-On-Board are typically defined relative to the start of Segment Profiles. The journey profile lists the sequence of track segments that the train is supposed to travel for a given route. When using ETCS the train posi- tion is referring to balise groups and driven distance. Hence to convert the train position from balise group reference to track segment reference, the segment profile needs to embody balises information including but not limited to BG iden- tifier NID_BG; position of the balise in the balise group; balise location as arc length from segment start. In order to crosscheck this information the balise lo- cation needs to be defined by a geographical position for example in latitude, longitude and altitude besides the referencing to the track segment start. The problem is that the segment profile is typically not achieving the safety level needed for the required operation, except extreme effort in the map generation process is invested. To ease this constraint, the counterchecking of balise loca- tion distance relative to track segment origin against difference of geodetical co- ordinates of the track origin and balise location allows to detect map failures and inconsistencies. If for example the balise position on track does not match within a certain error threshold to its geodetical position. Due to the fact that there are small errors between the segment length or curve radiuses these error may ac- cumulate and these error propagations are hard to detect, when under a certain level.

The stored balise position can be counterchecked to the measured balise posi- tion.

Given the segment is already present in the ATO-On-Board map database with the correct version, the information will be taken from this on-board map. Oth- erwise the track segment is requested by Segment Profile Request Packet. Checking to detect segment and journey profiles consistency errors includes con- sistency checks such as passing point versus journey/segment profile, ascending order of passing points, consecutiveness and completeness of segment profiles.

Remote Identification of the Train Control and Management Status

The status and output results of automatic driving controller shall be available on display in a human user interface with the output data as well as the error indi- cations. The remote control console issues an autodiagnostic request and the on board component will report with an autodiagnostic result. The list of rule ap- plicability identification and control output to the various subsystem, including but not limited to task assignment and journey&schedule management and as- signment execution supervision shall be displayed in the autodiagnostic function. In addition the train diagnostic capability allows to display the train on-board configuration. The version and status of key components and database configu- ration items can be checked.

The autodiagnositic status and output results of the on-board automated driving system, collected by the train maintenance controller, shall be available on dis- play in a human user interface with the output data as well as the error indica- tions. The supervision of journey assignment execution shall report an assign- ment forecast to the remote console assignment scheduler. A listing report of the executed Passing Point/Timing Point shall be available for diagnosis. In particular the relevant data such as journey profile indicating the protocol status, with ATO packet errors, timestamps and header data as well as assigned journey with passed and upcoming time points (arrival time, departure time, dwell time, sta- tion, door open side), temporary constrains (speed restrictions, low adhesion). In read access mode, the remote train console can check into current parameters and history to functional inputs such as

- ATO-TMS telegrams including time table, journey profile data, departure au- thorization;

- ETCS telegrams including executed and cleared movement authorities, speed profiles, position reports etc.; - Train motion and position outputs including time table speed profile, supervised speed envelope profile, automatic train stopping profile.

In addition the remote driver shall have the control to override mission data with for example train rescheduling or re-routing. The assignment task update and the assignment execution forecast shall be part of the remote mission manager function.

The autodiagnositic status and output results of the manoeuver controller shall be available on display in a human user interface with the output data as well as the error indications. The listing of manoeuver summary as well as the summary output list of TCMS commands shall be available for display.

The autodiagnositic status and output results of the movement protection controller, which validates the incoming Train Infrastructure data before provid- ing it to Speed Profile Optimizer, shall be available for display in a human user interface with the output data as well as the error indications. A summary of the track data is given including the actual and next track segment profiles as well as relevant rail infrastructure data, including but not limited to gradient, curves, powerless sections, static speed restrictions, and balise identifications. The dis- play option allows browsing into the journey profile including the objects proper- ties. In addition it is also capable to report map object properties; for example balise properties, composed of balise identity, location, number of balises in group, etc. Relevant specific information is contained in the segment profile de- tails data items such as platforms, tunnel, level crossing, and balises. So the re- mote driver can identify issues related to the track input data, where the system may be in a fault condition and the remote driver needs to solve it, before switching back to automatic mode. The remote driver shall have a means to clear errors or alerts as well as to manually override movement protection controller output data.

The autodiagnositic status and output results of the perception controller shall be available on display in a human user interface with the output data as well as the error indications. In particular the results of the identified objects from rail infrastructure such as but not limited to switches, stations, tunnels, bridges, ca- tenary pylons, signs and signals. This perception controller results, as per obsta- cle detection and classification, environmental monitoring and lateral signal de- tection shall be included in the autodiagnostic. The probability of detections as well as the attributes of classification shall be available in the menus such that the remote driver can identify misclassifications or failures. In read access mode, the remote train console can check into current parameters and history to all functional inputs to and action outputs of the automatic driving functionality such as

- Pattern recognition outputs including signal, sign, obstacle detection, platform and environment;

- On-board situation recognition outputs.

In addition the clearance of the railway loading gauge and any violation of un- known objects in this area shall be available in the remote GUI display by identi- fication, position and attributes. In addition the remote driver shall have control to override results or clear errors or alerts.

Train Diagnostic Interaction Capability

The remote train console allows for train diagnostic, that provides the relevant train component states. In addition it allows access to the train state and diag- nostic data base that covers failures of train components (e.g. traction unit, doors, and brake system); the remote console shall access a diagnosis function that detects failures associated to the automatic driving function. The diagnosis display message shall include at least the following information:

- ID: Diagnosis ID, which is unique within a maintenance interval;

- Time stamp of occurrence;

- Type classification: Information, Warning, and Failure;

- LRU: Component/function for which the diagnosis is associated; - Diagnosis content, which may include failure description or error sting.

Although the train state and diagnosis data base is manufacturer dependent, typ- ically it contains items such as but not limited to status data of:

- Catenary status;

- Catenary voltage;

- On-board power;

- Traction power and status;

- Break pressure;

- Break reservoir pressure;

- Break status of various break systems;

- Oil pressure;

- RPM motor;

- Wheel speed;

- Temperature of motor, gear box, cooling system;

- Door status;

- Climate comfort;

- Coupling status;

- Rolling stock status.

Remote Train Driving Capability

The remote train driving apparatus includes in the ATO trackside rail traffic oper- ation and management center a computer graphical user interface, which is ca- pable to display information as well as to receive human input action via for ex- ample a pointing device, keyboard or a touch screen device. In addition the criti- cal driver devices are separately equipped, such as but not limited to trac- tion/break lever, alert button, or operators vigilance button. The remote driving apparatus is also equipped with audible devices e.g. headphone & microphone to allow to communicate to train or to passengers. Safety is achieved by providing online and real-time all train states including states of the standard train driver DMI, additional states, which a train driver needs to collect by maintenance investigation, as well as track related infor- mation. Full operational information, adequate to the information a driver is ex- periencing in the locomotive driver's cab, is a vital part for safe operations. The look and feel of the remote train workplace similar to the driver's cab allows a seamless and intuitive work flow.

The Graphical User Interface presents all necessary information to the remote train driver. One preferred embodiment of the human machine interface func- tionality can include menus or display elements/modes as listed: a) Dialog elements for user authentication and log-in or exit; b) Dialog elements for start/stop a train journey or execute legs of a train jour- ney; c) Display elements for train status supervision with status warning in case of critical situation; d) Display elements of the train subsystem states, such as but not limited to train speed, break pressure, catenary voltage, state of motion control, etc.; e) Display element for the wireless remote connection properties; f) Display elements for the train mode and ATP (e.g. ETCS full supervision or PZB); g) Configurable display elements for specific purposes dependent on train model; h) Control elements for the train control, such as but not limited to train power- up/down, train acceleration/driving lever, braking lever, catenary control, front/end light switch, door switch, etc.; i) Control element for emergency stop and for horn; j) Configurable control elements that are represented by a specific device (but- ton, lever, switch); k) Configurable control elements that are part of a graphic user interface such as touch screen;

L) Control elements for ETCS or STM specific purposes; m) Elements for the train protection check or for cab signaling system (inductive or radio based); n) Video screen of the driver's cab view (optional on selection); o) Track map with train position and position data; p) Video screen with train driver cab instrument view; q) Menus from other sources such as obstacle detection subsystems, lateral sig- nal recognition, passenger information subsystems; r) Control element for operators vigilance or alertness.

An embodiment example of the remote driver display given in Fig. 6 includes the look-and-feel of an extended ETCS display with the first level groups: control el- ements, ETCS display elements, train state elements, moving map display, front cam display and driver cab view. Other display elements from the list above (a to r) can be arranged in a second dialog level or shown upon menu selection. For critical elements an optical and audible redundancy is provided, when the infor- mation needs operator action or is not up-to-date. E.g. an alert sound or a wait- ing/inactivity icon with a greyed-out menu is used if the information freezes or gets stale.

Given the information is not up-to-date by excess of a time threshold, a waiting pattern is shown to warn the operator. One preferred embodiment of the waiting icon or pattern may be for example as progress/loading-bar or as a greyed out image to show intuitively stale information. This waiting pattern is used in addi- tion to the time stamp in the displays. During initialization the display elements are run through a visual check, where all states are displayed in sequence or a check pattern is put on the screens. The train remote control console provides a functionality to execute or manage actions such as emergency stop, power cut, passenger call or live incident handling.

Fig. 7 illustrates a preferred variant of the inventive method for safe train re- mote control, with respect to the part taking place at an automatic train ATR. A camera system CAM installed at the front of the automatic train ATR monitors the track section ahead of the train (or its front locomotive/wagon) continuously, and generates corresponding image frames HRI. These image frames HRI have a high resolution in order to resolve all details necessary for a safe remote train control. The high resolution image frames HRI are then processed in three pro- cessing lines PL2, PL2 and PL3 here.

In the first processing line PL1, the high resolution image frames HRI are con- verted into low image frames LRI. Typically, the amount of pixels is reduced by a factor of 4 or more in each image direction, thus reducing the number of pixels by a factor of 16 or more. Further compression and encoding techniques apply. The low resolution image frames LRI are transmitted via a packet-based mobile communications network MCN via a first data channel CHI to an automatic train operation (=ATO) trackside rail traffic operation and management center, also called simply control centre (see Fig. 8). Due to the low resolution, only few bandwidth is required for the image transmission on the mobile communication network MCN. In the second processing line PL2, the high resolution image frames HRI undergo a pattern recognition. If patterns/structures recognized in a high resolution im- age frame HRI sufficiently match reference objects stored in a map data base MDB of the automatic train ATR, objects are identified. For each identified object, the object ID of the corresponding reference object in the map data base as well as relative object insertion properties are determined. In the example shown, a railway signal, an intermittent train control and a track section has been identi- fied as objects. Note that the reference objects represent typical trackside ob- jects, which may be found in large numbers along tracks. For each reference ob- ject, a (high resolution) appearance information is stored, in order to allow a comparison with the structures found in the high resolution image frames. This appearance information is typically a 3D vectorized information. The relative ob- ject insertion properties indicate the position of the identified object in the high resolution image frame HRI as well as its size and rotation. In the variant shown here, the map data base MDB also contains a register of known objects, wherein each known object is defined by an object ID of a corre- sponding reference object (indicating the type of known object) and a georefer- ence where the known object can be found (i.e. indication the geographical loca- tion of the known object), and here also a unique name for the known object. In the example, the railway signal identified as object also matches a known object of the register with the unique name "Signal 0158", and the intermittent train control identified as object also matches a known object of the register with the unique name "PZB 0739". Note that the track identified as object does not match with a known object here.

All identified objects are put into a list LIO, which contains the object IDs and the relative object insertion properties and the frame number of the (high resolution) image frame in which the objects were found. In the example shown, the list LIO also contains the information about the allocation to known objects where appli- cable. The list LIO is sent via the mobile communication network MCN via a sec- ond data channel CH2 to the control center (see Fig. 8). This requires only few bandwidth as compared to image frame transmission.

In the third processing line PL3, the high resolution image frames HRI undergo a vectorization algorithm. Striking structures in the high resolution image frames are translated into vector elements such as polylines and polygons, and their vector element properties, also called simply vector properties, are determined. By means of the vector properties, the appearance and position of the allocated vector elements are defined.

All allocated vector elements of are put into a list LAV, which contains the vector properties of the allocated vector elements and the frame number of the (high resolution) image frame in which the vector element was found. The list LAV is sent via the mobile communication network MCN via a third data channel CH3 to the control center (see Fig. 8). This requires only few bandwidth as compared to image frame transmission. Fig. 8 further illustrates the variant of the inventive method for safe train remote control of Fig. 7, with respect to the part taking place at the control centre COC.

Via the first data channel CHI, the low resolution image frames LRI are received at the ATO trackside rail traffic operation and management centre COC, also called simply control centre COC here. The control centre COC is stationary, e.g. integrated into an interlocking building, and comprises a train remote control console (not shown here) at which a remote driver can control the automatic train.

Via the second data channel CH2, the list of identified objects LIO is received. Based on a map data base MDB of the control centre COC, in particular the ref- erence object appearance information contained therein for each reference object identified by its object ID, the information of the list LIO is used to reconstruct the (high resolution) object appearances HROA of the identified objects. For this purpose, the for every listed identified object, its object ID is used to read out the appearance information of the corresponding reference object, and the rela- tive object image insertion properties are applied in order to scale, rotate and place the object correctly.

Via the third data channel CH3, the list of allocated vector elements LAV is re- ceived. The information of the list LAV is used to reconstruct the (high resolution) vector element appearances of the allocated vector elements.

Then the high resolution object appearances HROA originating from second data channel CH2 are overlaid onto the low resolution image frames LRI, resulting into overlay image frames OLI. The overlay image frames OLI are displayed at a dis- play of the remote train control console of the control centre COC to the remote driver. The high resolution object appearances HROA inserted in the overlay im- age frames OLI have enough resolution such that the remote driver can recog- nize all information relevant for safe train control, such as the switching state of the railway signal "Signal 0158" here. Further, the remote train driver can get a general impression of the landscape ahead of the train by means of the underly- ing low resolution image frames LRI of the overlay image frames OLI.

Further the vector element appearances VEA can be overlaid into the overlay im- age frames OLI, too, in order to improve the detailedness of the overlay image frames OLI (not shown here). Alternatively, and in the example shown, the vec- tor element appearances VEA are used to check the high resolution object ap- pearances HROA for plausibility before the overlay with the low image frames LRI is done. In the example, all the identified objects of data channel CH2 match with vector elements of data channel CH3. Therefore, the overlay image frame OIF is shown normally. If no sufficient matches were found, a warning message would be displayed that the overlay image frames may be corrupted.

Note that in another variant, both lists LIO and LAV may be transmitted via the mobile communication network MCN over one common data channel (i.e. the second channel CH2, so the third data channel CH3 is done without then), if de- sired (not shown in further detail).

Figs. 9 and 10 illustrate procedures that can be applied to verify a timely provi- sion of image data in the course of the present invention. The procedure may in particular be applied for transmitting data packets belonging to image data via the channels CHI, CH2 and/or CH3 in Figs. 7 and 8.

Fig. 9 illustrates the data transmission of packets or a corresponding protocol flow for the present invention.

The automatic train ATR wants to send a data packet DP to the control centre.

For this purpose, a mobile com router ATR-MCR of the automatic train ATR sends the data packet DP into the mobile communication network MCN. When the data packet DP is sent, the time Tis is read out at a UTC-clock UTCC of the automatic train ATR and noted; the time Tis is in particular included in a header of the data packet DP (first/top picture). The control centre COC receives the data packet DP by its mobile com router COC-MCR. At this occasion, the time Tic of the UTC-clock UTCC of the control centre COC is read out and noted (second picture).

Next, the mobile com router COC-MCR of the control centre COC sends out an acknowledgement ACK, for which the sending time T2 _S is read out at the UTC- clock UTCC of the control centre COC and noted. The time T2 _S is in particular in- cluded in a header of the acknowledgement ACK (third picture).

Finally, the acknowledgement ACK is received at the mobile com router ATR-MCR of the automatic train ATR. The time T _2C is read out at this occasion at the UTC- clock UTCC of the automatic train, and is noted (fourth/bottom picture).

Knowing T _1S , T _1C , T _2S and T _2C , a round trip delay ΔT _RoundTrip =T _1C -T _1S +T _2C -T _2S may be calculated. From other/former data packet sending events, an average round trip delay ΔT _{RoundTrip_AVG} is has also been determined. Further, a clock difference AT _Ci ocksync=0.5*(Tic-Tis+T2s-T2c) may be calculated. From other/former data packet sending events, an average clock difference ΔT _{clocksync_AVG} has also been determined.

As illustrated in Fig. 10, for a particular piece of image data sent resp. received, compare the data package DP of Fig. 9 above, the round trip delay ΔT _RoundTrip mi- nus the average round trip delay ATR _0U n _dThP _AV _G is compared to a threshold valueT _Threshold ^RT for round trip delay. Preferably, the threshold value T _Threshold ^RT for round trip delay is chosen depending on a standard deviation of round trip delays of the the past, typically applying a safety factor (also called confidence estaimation value).

If ΔT _RoundTrip -ΔT _{RoundTrip_AVG} >T _Threshold ^RT , then a warning message is displayed to the remote driver, indicating that the displayed image frame based (at least partial- ly) on the respective piece of image data is (or at least may be) not up to date. Further, in the example illustrated, the clock difference ΔT _Clocksync minus the aver- age clock difference ΔT _{clocksync_AVG} is compared to a threshold value T _Threshold ^CD for clock difference.

If ΔT _clocksync - ΔT _{clocksync_AVG} >T _Threshold ^CD , then the UTC-clocks of the automatic train and the control centre are not properly synchronized, what may blur the true round trip delays. Accordingly, it cannot be assured that the respective piece of image data is up to date, and a corresponding warning message is displayed to the remote driver.

If the checks do not reveal a risk of delayed image data delivery, the image frame based (at least partially) on the respective piece of image data is displayed regularly to the remote driver, without a warning message.

In summary, the present invention proposes a safe train remote control of an automated train, wherein the track section ahead of the automated train is watched with a video camera system producing high resolution image frames.

The high resolution image frames are converted to low resolution image frames and transmitted via a mobile communication network to a train remote control console in a trackside control centre via a fist data channel. Further, the high resolution image frames undergo a pattern recognition, and for identified objects corresponding to reference objects in a map data base, a reconstruction infor- mation is transmitted via the mobile communication network to the train remote control console via a second data channel. At the train remote control console, high resolution object appearances for the identified objects is reconstructed us- ing the reconstruction information and a corresponding map data base. The high resolution object appearances are overlaid onto the low resolution image frames received via the first data channel and displayed at the train remote control con- sole. Thus both safety relevant details and a general impression of the track sec- tion ahead are made available to a remote train driver at low bandwidth con- sumption of the mobile communication network. List of reference signs

ACK acknowledgement

ATR automatic train

ATR-MCR mobile com router (on automatic train)

CAM camera system

CHI first data channel

CH2 second data channel

CH3 third data channel

COC automatic train operation trackside rail traffic operation and man- agement centre / control centre

COC-MCR mobile com router (on control centre)

DP data packet

HRI high resolution image frame

HROA high resolution object appearance

LIO list of identified objects

LAV list of allocated vector elements

MCN mobile communication network

MDB map data base

LRI low resolution image frame

OLI overlay image frame

PL1 first processing line

PL2 second processing line

PL3 third processing line

UTCC UTC-clock

VEA vector element appearance