FIELD OF THE INVENTION

This invention relates to calibrating and/or analysing the reliability of one or more models of a physical system, for example a set of models for road traffic simulation.

BACKGROUND

Models of physical systems are often calibrated or trained (i.e. parameter values of the model are selected) for a specific task by selecting parameters that fit data that mimics the data (e.g. real-world data) that will be seen in the specific task. Calibration is often formulated as an optimization problem in which the parameters values of the model are selected that minimize the gap between the simulated model and reality.

Simulation is commonly used for large-scale systems by providing a highly adaptable platform to evaluate different system operation conditions and system behaviours by letting engineers modify a large number of parameters. However, this flexibility requires the model to be calibrated for each application in different conditions and settings. Calibration requires data, which can be time-consuming and costly to collect. One core drawback is the use of optimization methods which bring a high computation cost. A new direction in this research focuses on derivative free optimization algorithms, which are essentially direct-search algorithms that can be used to fit simulations to real-world data.

Other types of algorithms, such as Particle Swarm Optimization, are found in the literature, while some studies rely on statistical sampling of the search space with methods such as an Exhaustive search method or a Latin Hypercube sampling technique, which are not optimization algorithms by themselves. All of these techniques do not use derivatives of the calibration problem, as its explicit formulation is either unknown, too complex, or not differentiable (as with minimax problems).

The Mesh Adaptive Direct Search (MADS) algorithm is a derivative-free method specialized for constrained black box optimization. Its general principle is to generate candidate solutions on a discretization of the N-dimensional space of the black box parameters called the mesh, and to evaluate these candidates with the black box. Another approach is looking at derivative-free local and global methods. The derivative-free local methods are typically employing sampling with bias/rules toward improvement of the solution by using only function values. For the calibration process, such methods may be good for noisy, unreliable or expensive derivatives but may converge to local extremes (i.e. a local configuration of the model parameters). The derivative-free global approaches use a broad exploration with selective exploitation using function values in a typically computationally intensive convergence to global extremes. Such methods provide non-smooth responses on continuous variables under bound and general constraints.

One of the most important approaches that try to compensate for the typical drawbacks of optimization methods are the derivative-free approaches. Classical optimization used in calibration tasks may rely on provided derivatives of objective functions, finite differencing, and algorithmic differentiation. Derivative-free approaches may be good tools when: 1 ) automatic differentiation is not possible (i.e. black box or very complex models), 2) finite differencing is noisy or wrong, 3) models are expensive (i.e. finite differencing is not practical), and when 4) high-accuracy is not required.

When approaching and fixing punctual problems in the optimization process, all state-of-the- art approaches do not seem to alleviate completely the need for an iterative evaluation of a function in prescribed points of interest and noise and scale need special handling.

Considering the described apparatus and method may be implemented on car driver model calibration and reliability analysis, the prior art in the field of driver model calibration is also considered.

There are prior art systems that have developed systematic methodologies to calibrate driver behaviours models, such as the car-following, lane change, and the merge/yield. Using large open-source datasets (e.g. New Generation Traffic Simulation - NGSIM) or Extended Floating Car data. Punzo, Vincenzo, and Fulvio Simonelli. 2005. “Analysis and comparison of microscopic traffic flow models with real traffic microscopic data.” Transportation Research Record: Journal of the Transportation Research Board 1934.1 : 53-63 considered calibrating driver models (i.e. car-following) using trajectory data obtained from individual vehicles. The general methodology and the choice of distance metric (i.e. to quantify the variability among drivers) and objective functions (i.e. optimization criteria - typically traffic key performance indices) have been followed by subsequent researchers. For instance, Soria et al. investigated how different car-following models, incorporated in commercially available simulation tools (i.e. AIMSUN, CORSIM, and MITSIM), perform in different operational conditions, including congestion or weather (rain or clear sky). The study provided insights to the relationship between car-following parameters and different driver types, a relationship that, according to the authors, had attracted limited research.

More recently, and for the first time, Rakha, Hesham, John Sangster, and Jianhe Du. 2013. Naturalistic Driving Data for the Analysis of Car-Following Models. No. VT-2010-01 considered using naturalistic driving data to calibrate and compare four different driver models. This study focused on the cost and benefits of using naturalistic data and concluded that “any project seeking to use naturalistic data should plan for a complex and potentially costly data reduction process to extract mobility data.” Specifically, the radar detection technology used for data collection was found unreliable, requiring manual verification using the forward-facing video camera to extract trajectory segments, for which a complete and reliable set of measurements existed. In addition, speed measurements coming from the vehicle network were found more reliable from GPS data that tended to oscillate. As a result, only 50 percent of the corridorspecific extracted trajectories were used for analysis. Despite problems with the reliability of the measurements, the authors noted that the unique combination of driver information, coupled with the vast amounts of recorded data, can shed light on a number of important topics, such as driver heterogeneity (interdriver variation), and the relationship between driver behaviour and roadway type (intradriver variation). After cleaning the data, the authors used a methodology similar, but not identical, to Punzo and Treiber, to calibrate and validate four driver models (i.e. car-following) using naturalistic driving trajectory data. They concluded that the Rakha-Pasumarthy-Adjerid (RPA) model had the best model fit and was the most capable in explaining the variability of behaviour in the dataset.

Recently, Kesting, Arne, and Martin Treiber. 2008. “Calibrating car-following models by using trajectory data: Methodological study.” Transportation Research Record: Journal of the Transportation Research Board 2088.1 : 148-156 built on the methodology of Punzo and Simonelli and developed a detailed methodological framework for calibrating car-following models. Specifically, Kesting and Treiber investigated the minimum number of traffic states (i.e. driving regimes: cruising, approaching, creeping, free-acceleration, steady-state) required for fully calibrating a car-following model against a given trajectory, the temporal resolution of the input data, and the impact of the form of the trajectory comparison function on the results. The number of traffic state regimes contained in the data should relate to the traffic state regimes modelled by the driver model at hand (e.g. IDM, Gipps, Optimal Velocity, or other). Different microsimulation models describe driver behaviour through a different set of regimes; all of which must be contained in the observed data to perform calibration. In contrast, and if driver type can be somewhat inferred, trajectories pertaining to any combination of traffic states can be validated using special distance measures (e.g. the Frechet distance). Even though clustering techniques can be used to group similar trajectories, trajectory similarity does not necessarily imply driver type similarity because there are situations, such as stop- and-go traffic, in which there can be relatively little differentiation between drivers.

In summary, current approaches to calibrate and validate real and simulation physical models may be based on time-consuming (optimization) approaches that use aggregated data (e.g. minute’s aggregates - to form a spatio-temporal trajectory). But such aggregated data may be insufficient to calibrate or validate (simultaneously) the major components that comprise a simulation or real system model (i.e. driver lane change, car-following, merge/yield in a road traffic context).

Unintentionally, and in search of the best fit, the modeller may modify model’s parameters in a way that unrealistic behaviour at the physical level is being produced involving, for instance, too many transitions from acceleration to deceleration or excessively high number of lane changes for a driver in a traffic context. Additionally, without a proper spatio-temporal analysis tool and the proper validation metrics at this level, modellers may not have a way to measure the impact of the calibration process on system’s behaviour.

It is desirable to develop an apparatus and method that overcomes one or more of the above problems.

SUMMARY

According to a first aspect there is provided a modelling and/or prediction apparatus for calibrating a set of one or more models of a physical system, the modelling and/or prediction apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the modelling and/or prediction apparatus to: obtain the set of one or more models of the physical system; obtain real-world data from the physical system for one or more scenarios; obtain rules about the physical system operation; and calibrate the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system and in dependence on the rules about the physical system. By calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system and in dependence on the rules about the physical system the calibrated model may produce more accurate and realistic results.

In some implementations, the rules about the physical system act as constraints in the machine learning. In this way, the machine learning may fit to the real-world data but may be limited within the constraints set by the rules. This may provide a more accurate and realistic model.

In some implementations, the rules about the physical system comprise fuzzy logic. In this way, the rules may be applied to the machine learning in a fuzzy domain way.

In some implementations, the rules about the physical system comprise first order logic. In this way, the rules may be applied to the machine learning to relate a situation to a result, for example through a premise - conclusion formulation.

In some implementations, the rules about the physical system are predetermined. In this way, the rules may provide expert knowledge from domain knowledge of the physical system.

In some implementations, the real-world data from the physical system for the one or more scenarios is spatio-temporal data. In this way, the real-world data may comprise information about the space and time properties, or dynamics, of the components of the physical system.

In some implementations, the rules about the physical system are based on phenomenological observations identified in variation patterns of parameters of the set of one or more models within observed data. In this way, the rules, or expert knowledge, may be based on experience of what the simulation of the physical system should be like.

In some implementations, the program code causes the apparatus to convert the real-world data from the physical system for the one or more scenarios into a fuzzy domain to generate fuzzy input data. In this way, the application of the rules may be carried out in the fuzzy domain.

In some implementations, the program code causes the apparatus to apply the fuzzy logic to the fuzzy input data by means of fuzzy logic inference to generate fuzzy output data. In this way, the rules may be applied through fuzzy logic which may allow approximate inference to be carried out. The approximate inference may be more suitable to providing an accurate and realistic calibration.

In some implementations, the program code causes the apparatus to apply the fuzzy logic to the fuzzy input data by making one or more decisions in dependence on the fuzzy input data following the fuzzy logic. In this way, the decisions may allow the fuzzy logic rules to be applied in stages to the fuzzy input data.

In some implementations, the program code causes the apparatus to convert the fuzzy logic output data into a real domain to generate real output data. In this way, the output of the fuzzy inference may be converted back to a form which may be usable by the calibration module.

In some implementations, the program code causes the apparatus to calibrate the set of one or more models of the physical system in dependence on the real output data. In this way, the calibration module may use the output of the fuzzy inference to calibrate the model.

In some implementations, calibrating the set of one or more models of the physical system in dependence on the real output data comprises mapping the fuzzy logic output to a configuration of parameters of the set of one or more models using a fuzzy regression algorithm. In this way, the parameters of the model may be adapted to be provide a more accurate and realistic model, based on the fuzzy inference stage.

In some implementations, the program code further causes the apparatus to: cluster the real- world data from the physical system for the one or more scenarios to generate one or more groupings of the real-world data from the physical system for the one or more scenarios, and generate further rules about the physical system in dependence on the one or more groupings of the real-world data from the physical system for the one or more scenarios. In this way, the clustered data may provide further rules which may be implemented in the calibration of the models.

In some implementations, the further rules about the physical system comprise further fuzzy logic, and the program code further causes the apparatus to apply the further fuzzy logic to the fuzzy input data by means of fuzzy logic inference to generate the fuzzy output data. In this way, the further rules generated form the real-world data may provide mitigation for situations in which limited, or no, expert knowledge rules are available. In some implementations, the program code further causes the apparatus to obtain the real- world data from the physical system for the one or more scenarios from sensors configured to sense at least part of the physical system. In this way, the real-world data may be extracted from the physical system.

In some implementations, the physical system is a road traffic system. In this way, the apparatus may be used for road traffic physical systems.

According to a second aspect there is provided a method for calibrating a set of one or more models of a physical system, the method comprising: obtaining the set of one or more models of the physical system; obtaining real-world data from the physical system for one or more scenarios; obtaining rules about the physical system; and calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system in dependence on the rules about the physical system. By calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system and in dependence on the rules about the physical system the calibrated model may produce more accurate and realistic results.

According to a third aspect there is provided a reliability apparatus for assessing a reliability of a calibrated model of a physical system, the modelling apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the reliability apparatus to: obtain the calibrated model of the physical system; obtain rules about the physical system; obtain real-world data from the physical system for multiple scenarios; and generate one or more reliability measures of performance for the calibrated model of the physical system in dependence on a comparison between the calibrated model of the physical system with the real-world data from the physical system and the rules about the physical system. By generating one or more reliability measures of performance for the calibrated model of the physical system in dependence on the rules about the physical system the apparatus may more accurately assess a model’s reliability.

In some implementations, the program code further causes the apparatus to rank the reliability of the calibrated model against one or more other calibrated models by means of a statistical evaluation based on the reliability measures of performance. In this way, the different calibrated models may be compared based on the ranking and an assessment made on the reliability. In some implementations, the reliability measures of performance comprise one or more of average time loss, average speed, and average waiting time. In this way, reliability measure of performance may be based on the physical characteristics of the system.

In some implementations, the reliability of measures of performance are indicative of one or more of goodness of fit, predictive power, and plausibility. In this way, the measure of performance may assess how close the calibrated model is to a reliable level.

According to a fourth aspect there is provided a method for assessing a reliability of a calibrated model of a physical system, the method comprising: obtaining the calibrated model of the physical system; obtaining rules about the physical system; obtaining real-world data from the physical system for multiple scenarios; and generating one or more reliability measures of performance of the calibrated model of the physical system in dependence on a comparison between the calibrated model of the physical system with the real-world data from the physical system and the rules about the physical system. By generating one or more reliability measures of performance for the calibrated model of the physical system in dependence on the rules about the physical system the method may more accurately assess a model’s reliability.

According to a fifth aspect there is provided a modelling and/or prediction apparatus for generating a set of one or more models of a physical system, the modelling and/or prediction apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the modelling and/or prediction apparatus to: obtain a set of one or more models of the physical system; obtain real-world data from the physical system for one or more scenarios; obtain rules about the physical system; calibrate the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system in dependence on the rules about the physical system; obtain real-world data from the physical system for multiple scenarios; and generate one or more reliability measures of performance of the calibrated model of the physical system in dependence on a comparison between the calibrated model of the physical system with the real-world data from the physical system for the multiple scenarios and the rules about the physical system. By combining the calibration and reliability stages, a model may be calibrated and subsequently have the reliability of the calibration assessed. According to a sixth aspect there is provided a method for generating a set of one or more models of a physical system, the method comprising: obtaining a set of one or more models of the physical system; obtaining real-world data from the physical system for one or more scenarios; obtaining rules about the physical system; calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system in dependence on the rules about the physical system; obtaining real-world data from the physical system for multiple scenarios; and generating one or more reliability measures of performance for the calibrated set of one or more models in dependence on a comparison between the calibrated set of one or more models with the real-world data from the physical system for multiple scenarios and the rules about the physical system. By combining the calibration and reliability stages, a model may be calibrated and subsequently have the reliability of the calibration assessed.

BRIEF DESCRIPTION OF THE FIGURES

The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

Figure 1 schematically illustrates an exemplary model calibration problem including models, a calibration routine and scenario-based reliability analysis.

Figure 2A schematically illustrates exemplary driver behaviour models.

Figure 2B schematically illustrates exemplary driver behaviour variations.

Figure 3A schematically illustrates the stages of an exemplary modelling and/or prediction apparatus.

Figure 3B schematically illustrates the real-world data from the physical system.

Figure 4 schematically illustrates the stages of an exemplary modelling and/or prediction apparatus.

Figure 5 schematically illustrates the structure of an exemplary calibration module of a modelling and/or prediction apparatus. Figure 6 schematically illustrates the fuzzy data stages of an exemplary calibration module of a modelling and/or prediction apparatus.

Figure 7 illustrates sample trajectory data of the real-world data from the physical system.

Figure 8 schematically illustrates the fuzzy data fuzzyfication stages of an exemplary calibration module of a modelling and/or prediction apparatus.

Figure 9 schematically illustrates the fuzzy data rule evaluation stages of an exemplary calibration module of a modelling and/or prediction apparatus.

Figure 10A illustrates regression functionality evaluation by a fuzzy regression.

Figure 10B illustrates regression functionality evaluation by a least squares regression.

Figure 11 A illustrates regression principle for upper and lower bounds for the membership function.

Figure 1 1 B illustrates regression principle for fuzzy regression construction.

Figure 12 schematically illustrates the fuzzy data defuzzyfication stages of an exemplary calibration module of a modelling and/or prediction apparatus.

Figure 13A schematically illustrates the clustering of the real-world data from the physical system.

Figure 13B illustrates different regimes of clustering of the real-world data from the physical system.

Figure 14 schematically illustrates the stages of an exemplary reliability module of a modelling and/or prediction apparatus.

Figure 15 schematically illustrates the structure of an exemplary modelling and/or prediction apparatus. Figure 16 illustrates an example method for calibrating a set of one or more models of a physical system.

Figure 17 illustrates an example method for assessing a reliability of a calibrated model of a physical system.

Figure 18 illustrates an example method for generating a set of one or more models of a physical system.

Figure 19 schematically illustrates an example of an apparatus configured to perform the methods described herein.

DETAILED DESCRIPTION

The modelling and/or prediction apparatuses and methods described herein concern calibrating, assessing, and generating models of a physical system.

Embodiments of the described apparatus and method may tackle one or more of the problems previously mentioned by calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system and in dependence on the rules about the physical system. By using the rules, the calibrated model may produce more accurate and realistic results.

Additionally, embodiments of the described apparatus and method may tackle one or more of the problems previously mentioned by generating one or more reliability measures of performance of the calibrated model of the physical system in dependence on a comparison between the calibrated model of the physical system with the real-world data from the physical system and the rules about the physical system. By using the rules, the apparatus may more accurately assess a model’s reliability.

The described apparatus and method targets, but is not limited to, large-scale physical systems, for example road traffic, modelling for both simulation and real-world applications. In particular, the described apparatus and method focuses on model calibration and reliability assessment, for example driver models. More precisely, the described apparatus and method are directed to the calibration of behaviour models in which physical models of dynamics, available (recorded) sensory data, and expert knowledge may be combined. Exploiting these three elements, the system may exploit the known empirical description of the system’s dynamics, embed domain knowledge in the parametrization, and use machine learning to refine the parameters according to data peculiarities.

The goal of the described apparatus and method is to tackle the problems described herein and minimize the differences between real and simulated system performance indices by judiciously calibrating system’s behaviour models parameters. The described apparatus and method aim to be a unified framework to calibrate and evaluate calibrated models reliability against empirical data. Instead of considering a purely data-driven approach to assess the calibration validity (i.e. parameter estimation), the described apparatus and method may combine data insights, with physical models, and with human-understandable descriptions of system’s behaviours.

The described apparatus and method may be designed as a framework which: 1 ) allows for clustering system behaviours given their full operation context, 2) offers an interpretable assessment of the quality of a calibrated model with respect to the observed data, and 3) can operate at both microscopic and the macroscopic levels. Proper calibration of models can leverage their potential in both real-world instantiations (e.g. driver models for autonomous driving predictive systems) and simulation engines (e.g. driver models for traffic optimization and control) by capturing the nonlinear dynamics specific to systems’ behaviours in various scenarios.

The described apparatus and method may aim to increase trust and utility of simulation as it may be possible to 1 ) make it more accurate and 2) know how accurate it is. The described apparatus and method may contribute to a transition in the usage of physical simulation tools, from tools to inform domain experts (e.g. traffic engineers) on small scale project, to tools used to validate large scale Al deployment with a much higher risk associated to error and simulation uncertainty. Additionally, the described apparatus and method are focused on improving the precision of the simulation and measuring the distance from real system operation in a reliable manner allow to bring transitive trust to the simulation trust itself and quantify how trustable the simulation output is.

The described apparatus and method are relevant for both real-world physical systems behaviour predictions (e.g. driver model in autonomous driving) and simulators (e.g. driver model in traffic simulation). One contribution of the described apparatus and method are providing the means to optimize a system’s model parametrization (i.e. calibration) in order to meet expected key performance indices - KPI (i.e. minimize an objective function of desired business metrics) using a physics- informed explainable machine learning system. Such a physics-informed machine learning system may combine the mechanistic description of system’s dynamics, with expert domain knowledge, and data-driven parameter refinement for plausible and efficient system behaviour modelling. The parameter choice may also take into account the full system operation context and clusters plausible parameter spaces given an inclusive and complete distance metric that captures inter-context and intra-context variations.

An additional contribution of the described apparatus and method may be the calibration capabilities, in which a calibrated model reliability analysis function is used. This module of the described apparatus and method may exploit data, performance indexes, and expert knowledge in order to assess a system’s behaviour models in terms of goodness of fit, predictive power, and the realism (plausibility) using statistical methods and tools adapted to the explainable machine learning system.

This constellation of tools and capabilities may allow the described apparatus and method to efficiently optimize a physical system’s behaviour models (e.g. driver model lane change, carfollowing, and merge-yield in a road traffic context) that account for expert knowledge and the physics of the driver dynamics using explainable machine learning. This way, the described apparatus and method may avoid costly optimization methods and dedicated model parametrizations. As the calibration is solely based on the physics of the system model, expert knowledge, and the spatio-temporal system operation data, the described apparatus and method generate a plausible and reliable model transferrable across a range of new operation scenarios.

The described apparatus and method may rely on efficient computation of system behaviour model parameters sets using light-weight rule-based inference and regression (i.e. learning the mapping from system operation regime to system model parameters) that enables it to perform a versatile calibration and reliability assessment of any system behaviour model.

The described apparatus and method propose a platform for simultaneous calibration and reliability analysis of physical systems described through parametrized mathematical models. The goal is to ensure that the resulting system model parametrization is highly adaptable to different operating conditions and system behaviours by adjusting the large number of parameters describing the system. However, this flexibility may require the model to be calibrated for each application in different operating regions, conditions and settings. Calibration may require data, which can be time-consuming and costly to collect. The described apparatus and method propose a calibration procedure for the physical system models, which describe how the system components interact with each other. These calibrated behaviours should be generic for the operating region regardless of the specific context settings and the proposed procedure seeks to allow this generalisation by allowing simultaneous simulations on many new scenarios.

The exemplary embodiments shown in the Figures relate to road traffic driver models calibration and reliability analysis. It would be appreciated that the described apparatus and method may be equally applied to other physical systems.

In this application scenario, it is desirable to go beyond the Intelligent Driver Model (IDM) and Adaptive Cruise Control (ACC), especially when targeting the fusion of mobility models for both manned and autonomous vehicles. One problem with existing large-scale traffic simulators is that if the vehicles do not behave as they would in real life, then the conclusions drawn therefrom may be misleading or even wrong. Hence, this may result in the simulations being less useful to derive certifications and insurance guidelines as well as operational policies.

Reference is now made to Figure 1 which schematically illustrates an exemplary model calibration problem 100 including models 101 , a calibration routine 102 and scenario-based reliability analysis 103.

The versatile and reliable calibration of the model of the physical system, or driver model 101 , may assume a proper extraction of the driver dynamics and may enable its robustness in face of scenario variations. Independent of the driver behaviour model component (i.e. lane change, car-following, merge/yield), the calibrated model of the physical system would preferably be parametrized using the optimal set of parameters which minimizes the difference between the model output and the empirical data, as depicted in Figure 1 . Such optimal set of parameters preferably also allow the versatility of the model in front of scenario variations. Driver model 101 calibration may be a multi-factorial problem that needs to exploit the internal peculiarities of the driving scenario data (e.g. correlations, covariate shift) in order to refine the dynamics the physical model 101 of the driver and can replicate empirical data observations acquired from traffic. To describe a practical implementation of the problem the described apparatus and method aims to solve, a road traffic driver model calibration and reliability analysis is presented. Every driver may change behaviour based on how alert they are under different driving conditions, such as adverse weather, accidents, mandatory or discretionary lane changes. Therefore, estimating a single set of driver parameters for the entire journey may be an oversimplification. Intra-driver variation may be incorporated into the driver model by introducing event-oriented parameter changes or time-dependent adjustments (e.g. driving in the morning versus night). Intra-driver variations may be one of the main reasons why even the best-calibrated carfollowing models or the prior art have an average minimum error around 20 percent. Another plausible reason may be that the prior art systems may not identify and model the driver’s anticipatory reactions to traffic past the leader vehicle or to lane-changing manoeuvres properly.

On the other side, inter-driver variations pertain to differences in the driving behaviour of the entire population and have to do with physiological characteristics, vehicle characteristics, or localized attitudes that differentiate drivers between different states, cities, and countries. Microsimulation models often assume a universal distribution of driver and vehicle characteristics neglecting such differences. In addition to establishing the distribution of driver parameters, it also is important to acknowledge the correlations between parameter values that pertain to the same driver type (i.e., an aggressive driver may have a higher desired speed and acceleration targets).

Reference is now made to Figure 2A which schematically illustrates exemplary driver behaviour models 201 a, 201 b, 201c; and Figure 2B which schematically illustrates exemplary driver behaviour variations 202.

An example of driver variability 202 is provided in Figure 2B where multiple drivers’ trajectories are aggregated to analyse the interaction diagram between spacing and speed as well as the best model fit and was the most capable in explaining the variability of behaviour in the dataset. Black lines represent the combinations of spacing and speed in the dataset. Grey lines represent model estimates from the best-fit car-following model. Based on Figure 2B, one can see that the observed variability in the relationship between speed and spacing can only be partially replicated by such model results. The described apparatus and method propose a novel system for calibrating and assessing the reliability of a physical system model. In order to explore the capabilities of the described apparatus and method, it has been implemented on a highly nonlinear large-scale system, that is road traffic.

Reference is now made to Figure 3A which schematically illustrates the stages of an exemplary modelling and/or prediction apparatus. Reference is now made to Figure 3B which schematically illustrates the real-world data from the physical system.

In its general architecture, as well as for the implementation on driver models, the described apparatus and method may have a modular structure, depicted in Figure 3A.

The calibration module 320 may be configured to combine mechanistic models of driver behaviours, or models of the physical system 321 , with traffic engineering domain expertise, or rules about the physical system, 322, and a data-driven explainable machine learning mechanism to calibrate known models, such as driver behaviour models (i.e. car-following, lane change, and merge/yield).

In order to produce a calibrated driver model 324, the described apparatus and method may exploit the dynamics of the driver behaviours from the known mathematical/physical models 321 of driver behaviour and computes their parameters based on available data from multiple scenarios (i.e. consider normal and anomalous traffic events: rain, accidents, work-zones, social events etc.). In order to adjust to intra-driver and inter-driver variations, the described apparatus and method may embed systematic (rule-based) traffic domain expertise, or rules 322, to refine models’ parameters based on trajectory data and the properties of the model (e.g. robustness, parsimony, orthogonality).

In situations when partial or no expert knowledge, or rules 322, is available, the system may be able to autonomously extract clusters 522 of driver behaviours from trajectory data 310 using relevant distance measures that capture context of the driver (i.e. distance metrics that capture the inner structure of the data and the environment of driver decisions). T o avoid costly optimization routines, the system may use approximate explainable reasoning and inference that allows it to generalize and generate a plausible driver behaviour model configuration suitable for arbitrary scenarios and that can be easily transferable across scenarios. Aside the calibration module 320, the described apparatus and method may comprise a driver model reliability assessment module 304. The reliability assessment module 304 may offer a flexible choice of objective functions to assess the goodness of fit and the predictive power of a certain (i.e. physics-informed or externally calibrated) driver behaviour model. The reliability assessment module 304 may embed empirically extracted systematic expert rules 342 to detect driver behaviour regimes from driver trajectory data 310 and combine it with the mathematical model parameters, in a similar way to the calibration module 320. Finally, the reliability assessment module 304 may offer statistical and hypothesis testing routines to comparatively assess the goodness of fit, the predictive power, and the realism (plausibility) of the parameter choice of a calibrated model 324 from the calibration module 320, or an independently calibrated model 341 .

The main problems the described apparatus and method aims to solve may include:

• The efficient calibration of a physical system’s behaviour models 321 (e.g. car driver lane change, car-following, merge/yield models in a road traffic context) that account for expert knowledge 322 and the physics of the system’s dynamics using explainable machine learning.

• The avoidance of costly optimization methods and dedicated system model parametrizations by exploiting the physics of the system model 321 , expert knowledge 321 and the spatio-temporal system operation data 310 to provide a plausible and reliable calibrated model 324 for a wide range of operating scenarios that can be easily transferable among scenarios.

• Scalability emerging from the capability of the described apparatus and method to cluster plausible parameter space regions that reflect system’s operating context, when there is partial or no expert knowledge 322 available.

• Efficient computation of system’s behaviour models parameter sets using light-weight rule based inference (i.e. extract from spatio-temporal operating data of the system to operating regime) and regression (i.e. learning the mapping from operating regime to system’s model parameters). • Versatile and reliable calibration of any physical system’s behaviour model given available driver spatio-temporal operation data 310, performance indices, and domain knowledge 322 through statistical testing.

Reference is now made to Figure 4 which schematically illustrates the stages of an exemplary modelling and/or prediction apparatus.

The aim of the described apparatus and method is to calibrate a physical system’s model (e.g. driver model) and quantify the reliability of the calibrated model through a physics-informed learning system utilizing:

• An explainable machine learning model capable of combining the mechanistic description of the system’s dynamics (e.g. driver’s dynamics), the available system’s spatio-temporal operation data (e.g. driver trajectory data: velocity, acceleration, space gap and time gap to other drivers), and domain expertise given in the form of if-then rules to generate a parameter configuration;

• An adaptive mechanism capable of exploiting the available system information to generate a plausible configuration of model parameters even with limited or no expert knowledge by clustering system’s operating regimes from spatio-temporal data (e.g. driver trajectory data); and

• A flexible statistical evaluation system that provides a qualitative assessment of a model’s goodness of fit, predictive power, and realism (plausibility).

This functionality is embedded in the composing modules of the proposed system, as shown in Figure 4.

As shown in Figure 4, the apparatus may comprise a modular structure 300. The modular structure 300 may comprise several parts that operate in concert on the available system’s spatio-temporal operation data 310 (e.g. driver trajectory data), as the data flow diagram shows. In particular, the modular structure 300 may comprise a physics informed calibration module 320 and a reliability analysis module 340.

Reference is now made to Figure 5 which schematically illustrates the structure of an exemplary calibration module 320 of a modelling and/or prediction apparatus. A calibration of a set of one or more models 321 of a physical system may be carried out by the calibration module 320 of a modelling and/or prediction apparatus.

Calibration is the process of finding optimal parameter values of a model (or a group of models in our case) such that the model "fits" a specific set of real-world data. This process may minimize the error between the output of the model and the real-world dataset for a specific scenario. Instead of considering a purely data-driven approach to calibrate and assess the calibration validity (i.e. parameter estimation), the calibration module 320 may combine data insights 310 with human-understandable descriptions of driver behaviours and their mechanistic representation (model). The overall system may be designed as a framework which allows for an interpretable assessment of the calibrated model 324 with respect to the observed data.

In particular, the apparatus may be configured to obtain a set of one or more models 321 of the physical system. The models 321 may be uncalibrated. As shown in Figure 4, there may be more than one model 321 a, 321 b, 321 c. The different models 321 a, 321 b, 321 c may relate to different aspects. For example, in Figure 4 there is a lane change model 321 a, a car following model 321 b and a merge/yield model 321 c. It is shall be appreciated that the models 321 may relate to other forms of traffic, or non-traffic related models. It is preferable that the different models 321 in the set relate to a similar physical system. For example, all of the models 321 a, 321 b, and 321 c relate to traffic. In this way, the calibration of the models may be applicable to the different models 321 a, 321 b, 321 c as they may have overlapping characteristics.

In this traffic related implementation of the apparatus, the models 321 may simulate the speed, position and or gap between vehicles, for example. The model 321 of these simulations may be defined by the equations shown in Figure 5.

The apparatus may be configured to obtain real-world data 310 from the physical system. Preferably, the real-world data 310 may be for one or more scenarios. The scenarios may correspond to the different models 321 a, 321 b, 321c. In this way, the real-world data 310 may correspond to each of the different models 321 a, 321 b, 321c. Additionally, the real-world data 310 may include information about each of the different models 321 a, 321 b, 321c which may provide more data which may improve the overall calibration as there is a broader range of training data. Similarly, in this traffic related implementation of the apparatus, the real-world data 310 may relate to the speed, position and or gap between vehicles, for example. The physical system may be a road traffic system.

The apparatus may be configured to obtain the real-world data 310 from the physical system by means of sensors configured to sense at least part of the physical system. In other words, the real-world data 310 may be generated by one or more sensors. By sensing the physical system, this may provide the apparatus with the required data 310. The sensors may form part of the apparatus. Alternatively, the sensors may be a separate system and the sensed real-world data 310 is provided to the apparatus.

The real-world data 310 from the physical system for the one or more scenarios may be spatiotemporal data. In other words, the real-world data 310 may comprise information about space and time. In this traffic related information of the apparatus, the space data may relate to the position of a vehicle and the time data may relate to the time as which the vehicle was in a certain position.

The apparatus may be configured to obtain rules 322 about the physical system operation. The rules 322 may be predetermined. The rules 322 may be set by a user, and/or the rules may be set as certain conditions at the start of the calibration process.

The rules 322 may comprise fuzzy logic. Alternatively, or additionally, the rules 322 may comprise first order logic. The first order logic may comprise conditions and results.

In particular, the rules 322 may act as constraints in the machine learning. In other words, during the calibration of the model 321 , the rules 322 may provide limits of certain parameters of the model 321 .

In this traffic related implementation of the apparatus, the rules 322 may relate to the desired speed, minimum space gap, and/or desired time gap. The rules for the desired speed may be that speed and the time gap are above data-drive limits Vc and Tc, respectively, and the absolute acceleration is below a limit Ac. The rules for the minimum space gap may be that the gap S is less than Sc. The rules for the desired time gap may be that T<Tc and none of the above conditions apply. The rules 322 about the physical system may be based on phenomenological observations identified in variation patterns of parameters of the set of one or more models within observed data. In other words, the rules 322 may be based on the real-world data 310. The real-world system, which is to be simulated, may be observed to find certain rules 322, constraints and/or situations which repeatably occur. In this way, these rules 322 may be used to calibrate the system in an accurate and realistic way.

The apparatus may be configured to calibrate the set of one or more models of the physical system by using machine learning. The machine learning may fit the real-world data 310 from the physical system and in dependence on the rules 322 about the physical system. In other words, the rules 322 may provide constraints to the machine learning, and the apparatus may aim to fit the real-world data 310 to the models 321 while using the rules 322 to constrain how the machine learning is fitted.

The calibration module 320 may exploit a rule-based data-driven approach to refine the physical model of the system’s dynamics (in the example instantiation, a driver behaviour model). The rules may not be rigid, but they may be natural language based and act upon phenomenological observations identified in the variation patterns of model’s parameters within measured/observed data.

Additionally, the system may allow the mathematical treatment of subjective judgment, inference, and decisions underlying the imprecise human reasoning process using approximate inference tools (e.g. fuzzy logic and fuzzy inference). The motivation of the system may be to capture and quantify non-rigid non-deterministic behaviours from data using a fusion between data driven insights, the physical model 321 , and expert rules / domain knowledge 322 for inferring driving regimes that map to driver model parameters (e.g. IDM car-following) using rules extracted over time.

The internal functionality of the calibration module 320 is depicted in Figure 5. Using the information carried by the physical model 321 (e.g. IDM equations) and the trajectory data 310, the system may evaluate the domain expert rules 322 to extract (i.e. classify) the driving regime the driver finds themselves in (i.e. cruising, free acceleration, creeping/standing, or steady state). Based on the identified regime, the system may find the mapping to the most plausible configuration of parameters of the driver model 324 (e.g. IDM parameters) that describe the regime. This task may be a regression task in which the rule-based decision making is mapped to the actual model to be calibrated. These two steps, rule evaluation and regression, may be core to the calibration of the models 321 .

The calibration module 320 may employ a fuzzy logic inference system.

Reference is now made to Figure 6 which schematically illustrates the fuzzy data stages of an exemplary calibration module 320 of a modelling and/or prediction apparatus. The real-world data 310 may be converted into the fuzzy domain 602, in which decisions 605 are made, and then converted back into the real domain 607 for use during calibration.

The apparatus may be configured to convert the real-world data 310 from the physical system for one or more scenarios into a fuzzy domain 602 to generate fuzzy input data 604. In other words, the spatio-temporal data may be driver trajectory data 310 that is mapped in the fuzzy domain 602 through the fuzzyfication interface responsible to map real data to fuzzy membership values. The fuzzification of the real-world data 310 may be carried out by membership functions 602a.

Reference is now made to Figure 7 which illustrates sample trajectory data 701 , 702, 703, 704 of the real-world data 310 from the physical system. The fuzzyfication module 602, may map the input data, acceleration 701 , velocity space 702, space gap 703, and time gap 704 as shown in Figure 7, to the fuzzy domain 604.

Reference is now made to Figure 8 which schematically illustrates the fuzzy data fuzzyfication stages of an exemplary calibration module 320 of a modelling and/or prediction apparatus. The fuzzyfication module 602, may map the input data 701 , 702, 703, 704 through mapping functions 801 , 802, 803, 804 describing the partitioning of the input data space into regions. Figure 8 shows the mapping functions 801 , 802, 803, 804 corresponding to the different input data 701 , 702, 703, 704. Each input data 701 , 702, 703, 704 may be partitioned into different categories, e.g. very small, small, medium, high, very high, illustrated by the different shading in the functions in Figure 8. Each input variable 701 , 702, 703, 704 may have a different shape describing the variation pattern among the partitions over the domain span. Acceleration 701 and Velocity 702 may follow Gaussian shapes mapping 801 , 802 describing smoother partition switching, whereas the space gap 703 and time gap 704 may use triangle shape mapping functions 803, 804 (i.e. membership functions), as shown in Figure 8. The apparatus may be configured to obtain the rules 322, or knowledge base 601 which comprises the fuzzy logic 603 or domain knowledge.

The apparatus may be configured to apply the fuzzy logic 603 to the fuzzy logic input data 604 by means of fuzzy logic inference 605 to generate fuzzy logic output data 606. In particular, the apparatus may be configured to apply the fuzzy logic 603 by making one or more decisions in dependence on the fuzzy input data 604 following the fuzzy logic 603. In other words, the fuzzy logic inference 605 may carried out by means of (expert) rules evaluation 605a. The rules evaluation 605a may evaluate the result of the decision by the first order logic.

Reference is now made to Figure 9 which schematically illustrates the fuzzy data rule evaluation stages of an exemplary calibration module 320 of a modelling and/or prediction apparatus. The core of the fuzzy inference system is the actual expert rule evaluation 605. A set of exemplary sample rules 901 is provided in Table 1 , and also shown in Figure 9.

Table 1

These rules describe, in a human understandable (vague, imprecise) way the transitions in driver behavior along the simulation scenario providing the input data (velocity 702, acceleration 701 , space 703 and time gap 704). Internally, the rule evaluation accounts for building a “decision tree-like” structure which combines all the interacting variables describing the driver’s dynamics and the type of interactions that they have with each other, as shown in Figure 9.

For the mapping from driver behavior regimes identified in the trajectory data 310 (and derived quantities: acceleration 701 , velocity 702, space gap 703, and time gap 704) to the (changes in) parameters of the driver model, the system implements a regression model. Given a set of data samples, the goal is to minimize the total vagueness resulting from the (fuzzy) regression through the tuning of its parameters.

For the mapping from driver behaviour regimes identified in the trajectory data 310 (and derived quantities: acceleration 701 , velocity 702, space gap 703, and time gap 704) to the (changes in) parameters of the driver model 321 the system may implement a regression model. Given a set of data samples 701 , 702, 703 704, the goal is to minimise the total vagueness resulting from the (fuzzy) regression through the tuning of its parameters. Fuzzy regression, a nonparametric method, may be useful in estimating the relationships among variables where the available data is limited and imprecise, and variables are interacting in an uncertain, qualitative, and fuzzy way. Such a model offers means to model casual relationships. In the described apparatus and method, there may be ambiguity in describing a crisp measure of the dependent variable (i.e. how to modify the IDM parameter of interest from trajectory data 310). Unlike conventional regression analysis, where deviations between observed and predicted values reflect measurement error, deviations in fuzzy regression reflect the vagueness of the system structure expressed by the fuzzy parameters of the regression model. The fuzzy parameters of the model are considered to be possibility distributions, which corresponds to the fuzziness of the system.

Fundamentally, this may be considered a probabilistic-based method (fuzzy least squares calculates the fuzzy regression coefficients using least squares). The method is relatively robust against outliers compared to the possibilistic-based methods.

Reference is now made to Figure 10A which illustrates regression functionality evaluation by a fuzzy regression 1001. Reference is now made to Figure 10B which illustrates regression functionality evaluation by a least squares regression 1002. When comparing the fuzzy linear regression, as shown in Figure 10A, and a statistical linear regression model, as shown in Figure 10B, it can observed that while the fuzzy regression 1001 shows something akin to a confidence interval. The interval differs from the confidence intervals derived from a statistical linear regression model 1002.

The confidence interval from a statistical regression model 1002 shows the certainty that the modelled relationship fits within. It is 95% certain that the true relationship between the variables is as displayed. More precisely, the fuzzy regression 1001 displays the central value and the support boundaries determined by the left and right spread. Statistical least-squares regression 1002 shows the 95% confidence interval for the regression fit. On the other hand, the support of the fuzzy regression model prediction 1001 shows the range of possible values. The dependent variable can reach any value from the set, but the values more distant from the central tendency will have smaller degree of membership. It appears that the values closer to the boundaries are vaguely disappearing from the set as if they bleached out (gradient towards white in the fuzzy regression model plots).

Reference is now made to Figure 1 1 A which illustrates regression principle for upper and lower bounds for the membership function 1 101. Reference is now made to Figure 11 B which illustrates regression principle for fuzzy regression was construction 1102. An illustration of the fuzzy regression algorithm 521 and degree of fitting of driving regime X to driver model parameters Y is given in Figures 11 A and 11 B respectively. Figure 11 A indicates how the upper and lower bounds for the membership functions are constructed. A0 is the intercept and A1 is the slope of a linear relationship between X and Y. Figure 11 B illustrates how the fuzzy regression was constructed. The core idea of fuzzy regression is to minimize the total fuzziness criterion with constraints that all measurements fall between the H-cut upper and lower boundaries.

The apparatus may be configured to convert the fuzzy logic output data 606 into a real domain 607 to generate real output data 608. The decision may be decoded from the fuzzy domain to the real-world using the defuzzyfication interface 607. The defuzzification of the fuzzy logic output data 606 may be carried out by rule inference decoding 607a. This way all the processing, inference, and decision making is performed in a combined data-driven, physics enhanced, and domain expert informed way.

Reference is now made to Figure 12 which schematically illustrates the fuzzy data defuzzification stages of an exemplary calibration module 320 of a modelling and/or prediction apparatus. The output variables 606 follow the same design steps and capture, for each of the driving regime 1201 , 1202, 1203, 1204, 1205 (and possible among driving regimes) the activation (i.e. analog to probability) of a certain output, as shown in Figure 12. The output 608 is calculated based on the input data and the rules describing the expert knowledge embedded in the system regarding the transition among driving regimes 1201 , 1202, 1203, 1204, 1205.

The apparatus may be configured to calibrate the set of one or more models 321 of the physical system in dependence on the real output data 608. In other words, the real output data 608 may be used to generate the calibrated models 324. In particular, the apparatus may be configured to calibrate the set of one or more models 321 of the physical system in dependence on the real output data 608 by mapping the fuzzy logic output 606 to a configuration of parameters of the set of one or more models using a fuzzy regression algorithm 521.

Reference is now made to Figure 13A which schematically illustrates the clustering 522 of the real-world data 310 from the physical system. Reference is now made to Figure 13B which illustrates different regimes 1301 , 1302, 1303 of clustering 522 of the real-world data 310 from the physical system. Some of the rules 322 may be determined through a clustering mechanism of the real-world data 310.

In order to extend the flexibility of the system, when limited or no expert knowledge or rules 322 are available, the rules 322 can be extracted automatically from the real-world data 310 using a clustering mechanism (e.g. subtractive clustering). This way, the system can autonomously partition the input data space into “similar partitions” corresponding to different driver behaviours, as identified in the trajectory data.

In this way, the rules 322 used for the calibration may be fully based on, partially based on, or not at all based on the real-world data 322. This level of usage of the rules 322 based on the real-world data 322 may be dependent on whether expert knowledge, or rules that are predetermined are available. In this way, the apparatus may be able to utilise the rules 322 whether expert knowledge rules are available or not.

The apparatus may be configured to cluster the real-world data 310 from the physical system for one or more of scenarios to generate one or more groupings 523 of the real-world data 310 from the physical system for one or more scenarios. The clustering may be carried out to find real-world data 310 which is close, or is clustered, together. As shown in Figure 13A, real- world data 310 may be clustered into more than one grouping 523. In Figure 13A, there are three groupings 523a, 423b, 523c. The three groupings 523a, 523b, 523c may contain data that is close together. It may be appreciated that more or less than three groupings 423 may be generated.

The apparatus may also be configured to generate further rules 322 about the physical system in dependence on the one or more groupings 523 of the real-world data 310 from the physical system for the one or more scenarios. As the groupings 523 may include similar real-world data 310, which is close together, this may provide a suitable basis of generating further rules 322. Similar data may indicate that the real-world system has a repeatable situation which occurs, and as such a corresponding rule may be suited for that situation. For example, it may the that it can be seen from the real-world data 310 that during certain types of corners, the vehicles slow down by a larger amount.

The clustering component 522 of the calibration module 320 may be responsible to find and extract clusters within the driver trajectory data 310 using a relevant and appropriate distance measure to cope (eventually) with the data irregularities (i.e. uneven timeseries, scenario noise, scenario variability). From a driver behaviour perspective, clustering may form the basis for classification and driver modelling algorithms. The purpose of driver trajectory clustering, shown in Figure 13B, is to identify natural groupings of data from the large dataset (i.e. produced by simulation or recorded on field) to produce a concise representation of driver’s behaviour.

In a possible instantiation the clustering information to generate a fuzzy inference system that best models the data behaviour using a minimum number of rules. The rules may partition themselves according to the fuzzy qualities associated with each of the data clusters. As we typically do not have a clear idea how many clusters there should be for a given set of data (i.e. trajectories of multiple drivers in urban or motorway scenarios), a solution is subtractive clustering. This is a fast, one-pass algorithm for estimating the number of clusters and the cluster centres for such data.

The result of the subtractive clustering on the specific traffic model is shown in Figure 13B. The fuzzy classification has classified the data into the steady-state regime 1301 , the approaching regime 1302 and the cruising regime 1303. These regimes 1301 , 1302, 1303 may subsequently be used to generate the further rules 322. The further rules 322 may also comprise further fuzzy logic 603. The further fuzzy logic 603 may be used in the fuzzy inference 605 to apply the rules 322.

The apparatus may also be configured to apply the further fuzzy logic 603 to the fuzzy input data 604 by means of fuzzy logic inference 605 to generate the fuzzy output data 606. In other words, the output of the clustering 522, shown in Figure 5, may be included in the decision making 422. By generating the further rules 322 and further fuzzy logic 603 this may provide better constraints on the machine learning. Additionally, in situations where there is no, or limited, expert rules 322, the further rules 322 generated from the clustered data 522 may mitigate the lack of expert rules 322 and provide the required constraints on the machine learning.

Reference is now made to Figure 14 which schematically illustrates the stages of an exemplary reliability module 340 of a modelling and/or prediction apparatus. Validation (or reliability analysis) is the process of estimating the error of a calibrated model 341 compared to the real-world 310. This means estimating the error between the calibrated model 341 and the real-world 310 for all (arbitrary) possible scenarios. In the present instantiation for driver models calibration, the driver model reliability assessment module 340 of the described apparatus and method, depicted in Figure 14, provides a qualitative analysis of a calibrated model 341 .

The models 341 subject to the reliability analysis are picked from a pool of calibrated driver models 341 , be it through the calibration module 340 of the system or models 341 that were externally calibrated.

In particular, the apparatus may be configured to obtain a calibrated model 341 of the physical system. The calibrated model 341 may have been calibrated by the calibration apparatus as described herein. Alternatively, the calibrated model 341 may have been calibrated externally, for example by a third party. In this way, the reliability apparatus may be suitable for assessing a range of models from a range of different providers, for example in a plug-in and play type configuration.

In this traffic related implementation of the apparatus, the calibrated model 341 may simulate the speed, position and or gap between vehicles, for example. The calibrated model 341 of these simulations may be defined by the equations shown in Figure 14.

The apparatus may be configured to obtain rules 342 about the physical system. The rules 342 may be the same rules 322 as described herein with regards to the calibration module 320. In this way, the rules are the same in both the calibration module 320 and the reliability module 340. This may enable to reliability module 340 to assess the calibrated model 341 to the same criteria that it was calibrated. This may provide a means for checking if the calibrated model 341 is accurate and realistic. Alternatively, the reliability rules 342 may differ from the calibration rules 322. This may provide a means for assessing the calibrated model 341 against different criteria. This may be helpful for assessing external, or third part, calibrated models 341 . The reliability rules 342, whether they differ from the calibration rules 322 or not, may comprise a similar structure or features to the calibration rules 322, as described herein. In particular, the reliability rules 342 may act as constraints for machine learning. The reliability rules 342 may comprise fuzzy logic, and may be applied to the reliability assessment in the fuzzy domain. The reliability rules 342 may comprise first order logic. The reliability rules 342 may be predetermined.

The apparatus may be configured to obtain real-world data 310 from the physical system for multiple scenarios. In other words, the apparatus may obtain real-world data 310 for different scenarios of the calibrated models 341 . In this way, the calibrated model 341 may be assessed from the perspective of different scenarios and situations of the model calibrated 341 . This may allow the apparatus to provide a wider assessment of the reliability of the calibrated model in different scenarios.

In this traffic related implementation of the apparatus, the real-world data 310 may relate to the speed, position and or gap between vehicles, for example. The physical system may be a road traffic system. The real-world data 310 may be obtained in the same way as described herein with regards to the calibration apparatus 320. In particular, the real-world data 310 may be sensed from the physical system.

The reliability module 340 utilizes the trajectory data 310 for a large number of scenarios together with the expert rules 342 to parametrize a battery of statistical tests 345 that quantify, given the objective functions 343, the statistically significant differences among models 341 .

The model performance statistical evaluation computes relevant Measures of Performance (MOP), for example average time loss, average speed, and average waiting time, and ranks models depending on performance.

In particular, the apparatus may be configured to generate one or more reliability measures of performance 345 for the calibrated model 341 of the physical system in dependence on a comparison between the calibrated model 341 of the physical system with the real-world data 310 from the physical system and the rules 342 about the physical system. In other words, the apparatus may compare the inputted calibrated model 341 with the real-world data 310 and the rules 342. The real-world data 310 and the rules 342 may comprise constraints or criteria upon which the reliability measure of performance 345 is based. The reliability measure of performance 345 may include one or more of average time loss, average speed, and average waiting time. In this way, the performance of the calibrated model 341 may be assessed. The reliability of measures of performance 345 may also be indicative of one or more of goodness of fit, predictive power, and plausibility. In this way, the closeness of the calibrated model 341 to the real-world data 310 may be assessed.

The apparatus may be configured to perform statistical evaluation based on the reliability measures of performance 345. The statistical evaluation may be based on objective functions 343. The objective functions 343 may be defined by the equations shown in Figure 14.

The system performs a battery of statistical tests (i.e. a combination of omnibus ANOVA and post-hoc pairwise T-test) and adjusts the ranking depending on significance.

In particular, the apparatus may be further configured to rank the reliability of the calibrated model 341 against one or more other calibrated models 341 by means of a statistical evaluation based on the reliability measures of performance 345. A qualitive model assessment 346 may be used to rank the calibrated models 341 based on qualitive criteria.

The evaluation of the best algorithms, or calibrated models 341 , may depend on ranking for subsets of relevant metrics (i.e. the metrics with significant difference).

Reference is now made to Figure 15 which schematically illustrates the structure of an exemplary modelling and/or prediction apparatus. In particular, the end-to-end system runtime functionality is depicted in Figure 15.

The stages of the apparatus shown in Figure 15 include both the calibration module 320 and the reliability module 340. In this way, the apparatus in Figure 15 may calibrate one or more models 321 and subsequently perform a reliability assessment on such calibrated models 341 . In this way, one or more models of a physical system may be generated.

In particular, the apparatus may be configured obtain a set of one or more models 321 of the physical system. The models 321 may be uncalibrated. The models 321 may comprise the same features as described herein with regards to the calibration module 320 and the reliability module 340. The obtaining of the models 321 may be carried out in the same way as described herein with regards to the calibration module 320 and the reliability module 340. The apparatus may be configured to obtain real-world data 310 from the physical system for one or more scenarios. The real-world data 310 may comprise the same features as described herein with regards to the calibration module 320. The obtaining of the real-world data 310 may be carried out in the same way as described herein with regards to the calibration module 320.

The apparatus may be configured to obtain rules 322, 324 about the physical system. The rules 322, 324 may comprise the same features as described herein with regards to the calibration module 320 and the reliability module 340. The obtaining of the rules 322, 324 may be carried out in the same way as described herein with regards to the calibration module 320 and the reliability module 340.

The apparatus may be configured to calibrate the set of one or more models 321 of the physical system using machine learning to fit the real-world data 310 from the physical system in dependence on the rules 322 about the physical system. The calibration of the set of one or more models 321 may be carried out in the same way as described herein with regards to the calibration module 320.

The apparatus may be configured to obtain real-world data 310 from the physical system for multiple scenarios. The real-world data 310 may comprise the same features as described herein with regards to the reliability module 340. The obtaining of the real-world data 310 may be carried out in the same way as described herein with regards to the reliability module 340.

The apparatus may be configured to generate one or more reliability measures of performance 345 of the calibrated model 341 of the physical system in dependence on a comparison between the calibrated model 341 of the physical system with the real-world data 310 from the physical system for the multiple scenarios and the rules 322, 342 about the physical system. The generation of the reliability measures of performance 345 may be carried out in the same way as described with regards to the reliability module 340.

Reference is now made to Figure 16 which summarises an example of a method 1600 for calibrating a set of one or more models of a physical system. At step 1601 , the method 1600 comprises obtaining the set of one or more models of the physical system. At step 1602, the method 1600 comprises obtaining real-world data from the physical system for one or more scenarios. At step 1603, the method 1600 comprises obtaining rules about the physical system. At step 1604, the method 1600 comprises calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system in dependence on the rules about the physical system.

Reference is now made to Figure 17 which summarises an example of a method 1700 f for assessing a reliability of a calibrated model of a physical system. At step 1701 , the method 1700 comprises obtaining the calibrated model of the physical system. At step 1702, the method 1700 comprises obtaining rules about the physical system. At step 1703, the method 1700 comprises obtaining real-world data from the physical system for multiple scenarios. At step 1704, the method 1700 comprises generating one or more reliability measures of performance of the calibrated model of the physical system in dependence on a comparison between the calibrated model of the physical system with the real-world data from the physical system and the rules about the physical system.

Reference is now made to Figure 18 which summarises an example of a method 1800 for generating a set of one or more models of a physical system. At step 1801 , the method 1800 comprises obtaining a set of one or more models of the physical system. At step 1802, the method 1800 comprises obtaining real-world data from the physical system for one or more scenarios. At step 1803, the method 1800 comprises obtaining rules about the physical system. At step 1804, the method 1800 comprises calibrating the set of one or more models of the physical system using machine learning to fit the real-world data from the physical system in dependence on the rules about the physical system. At step 1805, the method 1800 comprises obtaining real-world data from the physical system for multiple scenarios. At step 1806, the method 1800 comprises generating one or more reliability measures of performance for the calibrated set of one or more models in dependence on a comparison between the calibrated set of one or more models with the real-world data from the physical system for multiple scenarios and the rules about the physical system.

Reference is now made to Figure 19 which schematically illustrates an example of an apparatus 1900 configured to perform the methods described herein. In particular, the apparatus 1900 may implement the model calibration method 1600, the model reliability assessment method 1700 and the model generation method 1800. The methods may be implemented on the same apparatus 1900, or different apparatus 1900. The different apparatusl 900 may have the same structure, as described below. The apparatus 900 may be implemented on an electronic computing device, such as a laptop, PC or automotive computing device.

The apparatus 1900 comprises a processor 1901 configured to process the datasets in the manner described herein. For example, the processor 1901 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The apparatus 1900 comprises a memory 1902 which is arranged to communicate with the processor 1901. Memory 1902 may be a non-volatile memory. The processor 1901 may also comprise a cache (not shown in Figure 19), which may be used to temporarily store data from memory 1902. The apparatus 1900 may comprise more than one processor 1901 and more than one memory 1902. The memory 1902 may store data that is executable by the processor 1901. The processor 1901 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor 1901 to perform its methods in the manner described herein.

Specifically, the modelling and/or prediction apparatus 1900 may comprise one or more processors 1901 and a memory 1902 storing in non-transient form data defining program code executable by the one or more processors 1901 to implement the models.

Below is described features and benefits of the described apparatus and method.

Going beyond the instantiation, key points of the described apparatus and method are the introduction of a new physical system model calibration and calibration reliability analysis that:

• Introduces a novel unified approach for the calibration and assessment of a physical system’s behaviour models (e.g. driver model car-following, lane-change, merge/yield) that avoids expensive optimization routines which might offer optimal parametrization but implausible values.

• Combines available spatio-temporal operation data (e.g. driver trajectory data) with mechanistic models of system’s behaviours, domain expert knowledge under the form of human-understandable rules, and an explainable machine learning approach.

• It may operate on various types of data (i.e. driver trajectory, traffic counts, floating car data) that describes system’s dynamics and exploits existing physical description of the system’s behaviours

• It offers the possibility to evaluate externally calibrated system models using a battery of statistical tests, objective functions, and spatio-temporal system operation data from multiple scenarios for a qualitative assessment of the calibrated models from multiple scenarios (e.g. for car driver model it considers normal and anomalous traffic events: rain, accidents, work-zones, social events).

• The system may embed systematic domain expertise to refine models’ parameters to adjust to intra-, and inter-operation regime variations based on the available spatiotemporal data (e.g. driver trajectory data) and properties of the model (e.g. robustness, parsimony, orthogonality).

• When limited expert knowledge is available the system extracts autonomously clusters from spatio-temporal data using relevant distance measures that capture context of the system operation (e.g. for driver model it uses distance metrics that capture the inner structure of the data and the environment of driver decisions).

• The system uses approximate explainable reasoning and inference for fast generalization and light-weight execution that can generate a plausible system behaviour model configuration suitable for arbitrary scenarios and can be easily transferable across scenarios.

• The system provides a flexible choice of objective functions to assess the goodness of fit and the predictive power of a certain (i.e. physics-informed or externally calibrated) system behaviour model.

Regardless of the deployment scenario, the described apparatus and method provides benefits such as:

• New approach for calibrating physical system behaviour models (e.g. driver models) useful for predictive models (e.g. in autonomous driving) and large-scale simulations (e.g. road traffic) for system optimization and control.

• An end-to-end unified pipeline for physical systems model calibration and reliability analysis that uses light-weight data processing mechanisms and explainable learning approaches to aggregate available data.

• A versatile framework that combines all available information about a system’s dynamics (e.g. driver trajectory data, systematically extracted domain expert rules about driving regimes, mathematical models of driver models) and explainable machine learning algorithms.

• Capturing intra-, and inter-system operating regime variations based on spatiotemporal operation data and properties of the model (e.g. robustness, parsimony, orthogonality) for extracting a model parameter configuration that is suitable for arbitrary scenarios and can be easily transferable across scenarios. • Combines an efficient data encoding (mapping) to a human understandable domain, rules evaluation, and regression that exploits the expert knowledge that can be embedded in the system and a light-weight inference mechanism easier to transfer to new scenarios.

• The reliability assessment provides a flexible choice of objective functions to assess the goodness of fit, the predictive power, and the plausibility of a certain (i.e. physics- informed or externally calibrated) system behaviour model.

• The reliability can be applied to self-calibrated models or externally calibrated models through statistical tests, objective functions, and spatio-temporal data from multiple scenarios for a qualitative assessment of the calibrated models from multiple scenarios.

The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the claims.