CROSS-REFERENCE

[0001] This application claims priority to United States Provisional Application No. 63/071,995, filed August 28, 2020 which is entirely incorporated herein by reference.

BACKGROUND

[0002] An autonomous vehicle is a vehicle that may be capable of sensing its environment and navigating with little or no user input. An autonomous vehicle system can sense its environment using sensing devices such as Radar, laser imaging detection and ranging (Lidar), image sensors, and the like. The autonomous vehicle system can further use information from global positioning systems (GPS) technology, navigation systems, vehi cl e-to- vehicle communication, vehicle-to- infrastructure technology, and/or drive-by-wire systems to navigate the vehicle.

[0003] A single highly automated vehicle or autonomous vehicle can generate one to five terabytes (1-5TB) of raw data per hour. Operating at 14 to 16 hours per day may mean generating as much as 50 terabytes per vehicle per day or 20 petabytes per vehicle per year. A modest fleet of 5,000 highly automated vehicles (there are 14,000 taxis in New York City alone) may generate over 100 exabytes of raw data annually. Such data may be generated by, for example, an autonomous vehicle stack or automated vehicle stack which may include all supporting tasks such as communications, data management, fail safe, as well as the middleware and software applications. Such data may also include data generated from communications among vehicles or from the transportation infrastructure. An autonomous vehicle stack or automated vehicle stack may consolidate multiple domains, such as perception, data fusion, cloud/over the air (OTA), localization, behavior (a.k.a. driving policy), control, and safety, into a platform that can handle end-to-end automation. For example, an autonomous vehicle stack or automated vehicle stack may include various runtime software components or basic software services such as perception (e.g., application-specific integrated circuit (ASIC), field- programmable gate array (FPGA), graphics processing unit (GPU) accelerators, single instruction multiple data (SIMD) memory, sensors/detectors, such as cameras, Lidar, radar, GPS, etc.), localization and planning (e.g., data path processing, double data rate (DDR) memory, localization datasets, inertia measurement, global navigation satellite system (GNSS)), decision or behavior (e.g., motion engine, error-correcting code (ECC) memory, behavior modules, arbitration, predictors), control (e.g., lockstep processor, DDR memory, safety monitors, fail safe fallback, by-wire controllers), connectivity, and input/output (I/O) (e.g., radio frequency (RF) processors, network switches, deterministic bus, data recording) and various others. Such data may be generated by one or more sensors and/or various other modules as part of the autonomous vehicle stack or automated vehicle stack.

SUMMARY

[0004] A significant amount of autonomous or automated vehicle data can be valuable and may be needed to be identified, selected, processed, transmitted and stored at the vehicle, edge infrastructure, and cloud contexts against different priorities of cost, timing, and privacy.

[0005] Recognized herein is a need for methods and systems for managing autonomous or automated vehicle data in a manner that is safe, secure, cost-effective, scalable, and fosters open applications.

[0006] National Highway Traffic Safety Administration (NHTSA) has issued a Standing General Order requiring manufacturers and operators of vehicles equipped with Society of Automotive Engineers (SAE) Level 2 advanced driver assistance systems (ADAS) or SAE Levels 3-5 automated driving systems (ADS) to report crashes. The order requires vehicle and equipment (including software) manufacturers of Level 2 ADAS or Levels 3-5 ADS systems and vehicles and operators of ADS-equipped vehicles to report crashes where the Level 2 ADAS or Level 3-5 ADS system was engaged during or immediately before the crash. These data may help the agency identify potential safety issues and impacts resulting from the operation of advanced technologies on public roads and increase transparency. Access to ADS data may show whether there are common patterns in driverless vehicle crashes or systematic problems in operation.

[0007] The present disclosure provides systems and methods for managing and recording vehicle data. In particular, the provided data management systems and methods can be applied to data related to various aspects of the automotive value chain including, for example, vehicle design, test, and manufacturing (e.g., small batch manufacturing and the productization of autonomous vehicles), creation of vehicle fleets that involves configuring, ordering services, financing, insuring, and leasing a fleet of vehicles, operating a fleet that may involve service, personalization, ride management and vehicle management, maintaining, repairing, refueling and servicing vehicles, and dealing with accidents and other events happening to these vehicles or by a fleet. As used herein, the term “vehicle data,” generally refers to data generated by any types of vehicle, such as a connected vehicle, a connected and automated vehicle, or a connected and autonomous vehicle, unless context suggests otherwise. The term “autonomous vehicle data” as utilized herein, generally refers to data generated by an autonomous vehicle. Although embodiments of the present disclosure have been described with respect to autonomous vehicles, it should be appreciated that the embodiments can be applicable or adapted for automated vehicles.

[0008] In some embodiments, the provided data management system may comprise a data orchestrator onboard an autonomous or an automated vehicle. The data orchestrator may be capable of orchestrating and managing vehicle data. In some cases, autonomous vehicle data may comprise data generated by the autonomous vehicle stack (e.g., data captured by the autonomous vehicle’s sensors), as well as driver and passenger data. The data orchestrator may be configured to determine which of (which portion of) the vehicle data is to be communicated to which data center or third-party entity, and when such data is transmitted. For example, some of the autonomous vehicle data may need to be communicated immediately or when the autonomous vehicle is in motion, whereas other data may be communicated when the autonomous vehicle is stationary (while waiting for the next assignment/task or being maintained).

[0009] In an aspect, a system is provided for managing vehicle data of a vehicle. The system comprises: a data repository configured to store: (i) data related to one or more remote entities that request one or more subsets of the vehicle data and (ii) a description of the one or more subsets of the vehicle data; a communication module configured to issue a query to a vehicle data recorder or one or more databases onboard the vehicle based at least in part on the description of the one or more subsets of the vehicle data; and a decision engine configured to execute a data transmission rule for transmitting the one or more subsets of the vehicle data, wherein the data transmission rule comprises: (i) a selected portion of the vehicle data to be transmitted; (ii) a timing to transmit the selected portion of the vehicle data; and (iii) a remote entity of the one or more remote entities for receiving the selected portion of the vehicle data.

[0010] In some embodiments, the data repository, the decision engine and the communication module are provided onboard the vehicle. In some embodiments, the one or more remote entities comprise a cloud application, a data center, a fog server, a third-party server, or another different vehicle. In some embodiments, the vehicle data recorder comprises a query engine configured to receive the query. In some cases, the query engine is configured to check one or more data records stored on a memory of the vehicle data recorder to determine whether the one or more data records meet the description of the requested one or more subsets of the vehicle data. In some instances, upon determining the one or more data records meet the description, the communication module is configured to issue another query to request a transmission of the one or more data records. In some instances, the query engine is configured to automatically transfer one or more data records from the vehicle data recorder to a database coupled to the system upon detection of an event. [0011] In some embodiments, a data record stored in the vehicle data recorder or the one or more databases includes metadata. In some cases, the metadata is related to an event or a condition of the vehicle. In some instances, the metadata is associated with a series of data records. For example, the metadata is used by a query engine of the vehicle data recorder to retrieve the series of data records. In some cases, the metadata is generated by a sensor that captures at least a portion of the vehicle data.

[0012] In some embodiments, the vehicle is (i) a connected vehicle, (ii) a connected and automated vehicle, or (iii) a connected and autonomous vehicle. In some embodiments, the system further comprises a knowledge base to store a machine learning-based predictive model or a user-defined rule to determine the data transmission rule. In some cases, the knowledge base is onboard the vehicle.

[0013] In another aspect, a method is provided for managing vehicle data of a vehicle. The method comprises: storing, in a data repository, (i) data related to one or more remote entities that request one or more subsets of the vehicle data and (ii) a description of the one or more subsets of the vehicle data; issuing, by a communication module, a query to a vehicle data recorder or one or more databases onboard the vehicle based at least in part on the description of the one or more subsets of the vehicle data; and executing a data transmission rule for transmitting the one or more subsets of the vehicle data, wherein the data transmission rule comprises: (i) a selected portion of the vehicle data to be transmitted; (ii) a timing to transmit the selected portion of the vehicle data; and (iii) a remote entity of the one or more remote entities for receiving the selected portion of the vehicle data.

[0014] In some embodiments, the data repository, the decision engine and the communication module are provided onboard the vehicle. In some embodiments, the one or more remote entities comprise a cloud application, a data center, a fog server, a third-party server, or another different vehicle.

[0015] In some embodiments, the method further comprises receiving the query by a query engine of the vehicle data recorder. In some cases, the method further comprises checking, by the query engine, one or more data records stored on a memory of the vehicle data recorder to determine whether the one or more data records meet the description of the requested one or more subsets of the vehicle data. For instances, upon determining the one or more data records meet the description, the method may comprise issuing another query to request a transmission of the one or more data records by the communication module. In some cases, the query engine is configured to automatically transfer one or more data records from the vehicle data recorder to a database coupled to the system upon detection of an event. [0016] In some embodiments, a data record stored in the vehicle data recorder or the one or more databases includes metadata. In some cases, the metadata is related to an event or a condition of the vehicle. In some instances, the metadata is associated with a series of data records. For example, the method further comprises retrieving, by a query engine of the vehicle data recorder, the series of data records using the metadata. In some cases, the metadata is generated by a sensor that captures at least a portion of the vehicle data.

[0017] In some embodiments, the vehicle is (i) a connected vehicle, (ii) a connected and automated vehicle, or (iii) a connected and autonomous vehicle. In some embodiments, the method further comprises providing a knowledge base to store a machine learning-based predictive model or a user-defined rule to determine the data transmission rule. In some cases, the knowledge base is onboard the vehicle.

[0018] Another aspect of the present disclosure provides a non-transitory computer readable medium comprising machine executable code that, upon execution by one or more computer processors, implements any of the methods above or elsewhere herein.

[0019] Another aspect of the present disclosure provides a system comprising one or more computer processors and computer memory coupled thereto. The computer memory comprises machine executable code that, upon execution by the one or more computer processors, implements any of the methods above or elsewhere herein.

[0020] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in this art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure.

Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

INCORPORATION BY REFERENCE

[0021] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and/or take precedence over any such contradictory material. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein), of which:

[0023] FIG. 1 schematically illustrates data flow between a data orchestrator and a data center. [0024] FIG. 2 shows examples of data flows across various data centers, autonomous vehicles and remote entities.

[0025] FIG. 3 shows an example of an Application Table.

[0026] FIG. 4 schematically shows an example of a data orchestrator in communication with one or more remote entities.

[0027] FIG. 5 illustrates an example of a predictive models knowledge base.

[0028] FIG. 6 shows an example of a model creator interacting with a metadata database and cloud data lakes for training and developing a predictive model.

[0029] FIG. 7 illustrates a method of creating a predictive model.

[0030] FIG. 8 shows example components of a data management system.

[0031] FIG. 9 shows an example of data ingestion pipeline and functions performed by a pipeline engine.

[0032] FIG. 10 shows an example data ingestion process.

[0033] FIG. 11 illustrates an example of metadata generated by data processing such as alignment, metadata generated by an application and/or a sensor.

[0034] FIG. 12 shows an example of scenario metadata.

[0035] FIG. 13 shows a computer system that is programmed or otherwise configured to implement the data management system.

[0036] FIG. 14 shows examples of varieties of applications in a lifecycle of automated and autonomous vehicles.

[0037] FIG. 15 illustrates an example of dynamically updating predictive models in vehicles.

[0038] FIG. 16 schematically shows data transmission managed with aid of an OEM, in accordance with some embodiments of the invention.

[0039] FIG. 17 shows a data transmission process between a data orchestrator and one or more cloud applications.

[0040] FIG. 18 schematically illustrates a multi-tier data architecture.

[0041] FIG. 19 schematically shows an example of a data orchestrator for managing data transmission between a vehicle layer and fog layer, and between the fog layer and a cloud layer. [0042] FIG. 20 shows an environment in which the data orchestrator may be implemented. [0043] FIG. 21 and FIG. 22 show examples of a data orchestrator fronts different in-vehicle systems such as vehicle data recorders, microcontrollers or electronic control units (ECU). [0044] FIG. 23 schematically shows a diagram of a data orchestrator including a plurality of functional components

DETAILED DESCRIPTION

[0045] While various embodiments of the invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed.

[0046] As used herein, the terms “autonomously controlled,” “self-driving,” “autonomous,” and “pilotless,” when used in describing a vehicle, generally refer to a vehicle that can itself perform at least some or all driving tasks and/or monitor the driving environment along at least a portion of a route. An autonomous vehicle may be an automated vehicle. Such automated vehicle may be at least partially or fully automated. An autonomous vehicle may be configured to drive with some or no intervention from a driver or passenger. An autonomous vehicle may travel from one point to another without any intervention from a human onboard the autonomous vehicle. In some cases, an autonomous vehicle may refer to a vehicle with capabilities as specified in the National Highway Traffic Safety Administration (NHTSA) definitions for vehicle automation, for example, Level 4 of the NHTSA definitions (L4), “an Automated Driving System (ADS) on the vehicle can itself perform all driving tasks and monitor the driving environment - essentially, do all the driving - in certain circumstances. The human need not pay attention in those circumstances,” or Level 5 of the NHTSA definitions (L5), “an Automated Driving System (ADS) on the vehicle can do all the driving in all circumstances. The human occupants are just passengers and need never be involved in driving.” It should be noted that the provided systems and methods can be applied to vehicles in other automation levels. For example, the provided systems or methods may be used for managing data generated by vehicles satisfying Level 3 of the NHTSA definitions (L3), “drivers are still necessary in level 3 cars, but are able to completely shift safety-critical functions to the vehicle, under certain traffic or environmental conditions. It means that the driver is still present and will intervene if necessary, but is not required to monitor the situation in the same way it does for the previous levels.” In some cases, an automated vehicle may refer to a vehicle with capabilities specified in the Level 2 of the NHTSA definitions, “an advanced driver assistance system (ADAS) on the vehicle can itself actually control both steering and braking/accel erating simultaneously under some circumstances. The human driver has to pay full attention (“monitor the driving environment”) at all times and perform the rest of the driving task,” or Level 3 of the NHTSA definitions, “an Automated Driving System (ADS) on the vehicle can itself perform all aspects of the driving task under some circumstances. In those circumstances, the human driver has to be ready to regain control at any time when the ADS requests the human driver to do so. In all other circumstances, the human driver performs the driving task.” The automated vehicle may also include those with Level 2+ automated driving capabilities where Al is used to improve upon Level 2 ADAS, while consistent driver control is still required. The autonomous vehicle data may also include data generated by automated vehicles.

[0047] An autonomous vehicle may be referred to as unmanned vehicle. The autonomous vehicle can be an aerial vehicle, a land vehicle, or a vehicle traversing water body. The autonomous vehicle can be configured to move within any suitable environment, such as in air (e.g., a fixed-wing aircraft, a rotary-wing aircraft, or an aircraft having neither fixed wings nor rotary wings), in water (e.g., a ship or a submarine), on ground (e.g., a motor vehicle, such as a car, truck, bus, van, motorcycle or a train), under the ground (e.g., a subway), in space (e.g., a spaceplane, a satellite, or a probe), or any combination of these environments.

[0048] Whenever the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0049] Whenever the term “no more than,” “less than,” or “less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “no more than,” “less than,” or “less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0050] The term “real-time,” as used herein, generally refers to a response time of less than 1 second, tenth of a second, hundredth of a second, a millisecond, or less, such as by a computer processor. Real-time can also refer to a simultaneous or substantially simultaneous occurrence of a first event with respect to occurrence of a second event.

[0051] The present disclosure provides methods and systems for data and knowledge management, including data processing and storage. Methods and systems of the present disclosure can be applied to various types of vehicles, such as a connected vehicle, a connected and automated vehicle, or a connected and autonomous vehicle. Connected vehicles may refer to vehicles that use any of a number of different communication technologies to communicate with the driver, other cars on the road (vehi cl e-to- vehicle [V2V]), roadside infrastructure (vehicle-to- infrastructure [V2I]), and the “Cloud” [V2C], The present disclosure provides data orchestrators that may be used in various contexts, including vehicles (e.g., autonomous vehicles) and nonvehicle contexts. Data orchestrators of the present disclosure may be used for managing data from various sources or for various uses, such as Internet of Things (loT) platforms, cyberphysical software applications and business processes, and for organizations in energy, manufacturing, aerospace, automotive, chemical, pharmaceutical, telecommunications, retail, insurance, healthcare, financial services, the public sector, and others.

Data management system

[0052] The present disclosure provides systems and methods for managing vehicle data such as autonomous vehicle data or automated vehicle data. In particular, the provided data management systems and methods can be applied to data related to various aspects of the automotive value chain including, for example, vehicle design, test, and manufacturing (e.g., small batch manufacturing and the productization of autonomous vehicles), creation of vehicle fleets that involves configuring, ordering services, financing, insuring, and leasing a fleet of vehicles, operating a fleet that may involve service, personalization, ride management and vehicle management, maintaining, repairing, refueling and servicing vehicles, and dealing with accidents and other events happening to these vehicles or by a fleet. The data management system may be capable of managing and orchestrating data generated by a fleet at a scale of at least about 0.1 terabyte (TB), 0.5 TB, 1 TB, 2 TB, 3 TB, 4 TB, 5 TB, or more of raw data per hour. In some instances, the data management may be capable of managing and orchestrating data generated by a fleet at a scale of at least about 50 TB, 60 TB, 70 TB, 80 TB, 90 TB, 100 TB of raw data per hour. In some instances, the data management may be capable of managing and orchestrating data generated by a fleet at a scale of at least 1 gigabyte (GB), 2 GB, 3 GB, 4 GB, 5 GB or more of raw data per hour. The data management system may be capable of managing and orchestrating data of any volume up to 0.5 TB, 1 TB, 2 TB, 3 TB, 4 TB, 5 TB, 50 TB, 60 TB, 70 TB, 80 TB, 90 TB, 100 TB or more of data per hour. The data management system can be the same as those described in International Patent Application W02020097221, filed November 6, 2019, which is incorporated herein by reference in its entirety.

[0053] In some embodiments, the data and knowledge management system may be in communication with a data orchestrator that resides onboard an autonomous or automated vehicle. The data orchestrator may be capable of managing vehicle data. The data orchestrator may be a data router. The data orchestrator may be configured to route the vehicle data in an intelligent manner to the data and knowledge management system. The data orchestrator may be configured to determine which of the autonomous/automated vehicle data or which portion of the autonomous/ automated vehicle data is to be communicated to the data and knowledge management system of which data center or third party entity, and when this portion of autonomous/automated vehicle data is transmitted. For example, some of the autonomous/automated vehicle data may need to be communicated immediately or when the autonomous/automated vehicle is in motion, whereas other data may be communicated when the autonomous/automated vehicle is stationary (while waiting for the next assignment/task or being maintained). The provided data management system may also comprise a predictive model creation and management system that is configured to train or develop predictive models, as well as deploy models to the data orchestrator and/or the components of the autonomous vehicle stack, or the components of the automated vehicle stack. In some cases, the predictive model creation and management system may reside on a remote entity (e.g., data center). The provided data management system may further comprise a data and metadata management system that is configured to store and manage the data and associated metadata that is generated by the autonomous/automated vehicle, and process queries and API calls issued against the data and the metadata. The data orchestrator, or the data and knowledge management system, can be implemented or provided as a standalone system. It should be noted that any method and systems described herein with respect to autonomous vehicle or autonomous vehicle data are also applied to automated vehicle or automated vehicle data.

[0054] FIG. 1 schematically illustrates the data flow between a data orchestrator 100 and a data center 120. In some, cases, the data orchestrator 100 may be configured to automate the data management process, including, for example, data creation, data cleansing, data enrichment, and delivering data across data centers, systems, and third party entities. Data collected from the autonomous vehicle 110 may comprise data captured by the autonomous vehicle’s sensors. Such sensors can include, for example, the navigation system, sensors onboard the vehicle such as laser imaging detection and ranging (Lidar), radar, sonar, differential global positioning system (DGPS), inertial measurement unit (IMU), gyroscopes, magnetometers, accelerometers, ultrasonic sensors, image sensors (e.g., visible light, infrared), heat sensors, audio sensors, vibration sensors, conductivity sensors, chemical sensors, biological sensors, radiation sensors, conductivity sensors, proximity sensors, or any other type of sensors, or combination thereof. Data collected from the autonomous vehicle may also comprise fleet data (e.g., data from vehicle operating system), driver data (e.g., driver mood, driver alertness level, driving style, etc.), passenger data (e.g., data from user experience platform such as access to music, game, data from user device such as a mobile application), and various others. The data sent by the data orchestrator of a vehicle is received by the data and knowledge management system residing in a data center. The data may include data streams from one or more sensors (e.g., the output of a video camera, sensor fusion data), batch data, and/or individual records (e.g., a purchase transaction made by a passenger while being transported, individual record or series of records produced by a vehicle subsystem, e.g., a system monitoring the vehicle’s engine health, or the subsystem monitoring the condition of tires, a result of a user interaction with one of the applications running on the vehicle).

[0055] In some embodiments, the data orchestrator 100 may be an edge intelligence platform. For example, the data orchestrator may be a software-based solution based on fog or edge computing concepts which extend data processing and orchestration closer to the edge (e.g., autonomous vehicle). While edge computing may refer to the location where services are instantiated, fog computing may imply distribution of the communication, computation, and storage resources and services on or in proximity to (e.g., within 5 meters or within 1 meter) devices and systems in the control of end-users or end nodes. Maintaining close proximity to the edge devices (e.g., autonomous vehicle, sensors) rather than sending all data to a distant centralized cloud, minimizes latency allowing for maximum performance, faster response times, and more effective maintenance and operational strategies. It also significantly reduces overall bandwidth requirements and the cost of managing widely distributed networks. The provided data management system may employ an edge intelligence paradigm that at least a portion of data processing can be performed at the edge. In some instances, machine learning model may be built and trained on the cloud and run on the edge device or edge system (e.g., hardware accelerator). Systems and methods of the disclosure may provide an efficient and highly scalable edge data orchestration platform that enables real-time, on-site vehicle data orchestration.

[0056] The software stack of the data management system can be a combination of services that run on the edge and cloud. Software or services that run on the edge may employ a predictive model for data orchestration. Software or services that run on the cloud may provide a predictive model creation and management system 130 for training, developing, and managing predictive models. In some cases, the data orchestrator may support ingesting of sensor data into a local storage repository (e.g., local time-series database), data cleansing, data enrichment (e.g., merging third-party data with processed data), data alignment, data annotation, data tagging, or data aggregation. Raw data may be aggregated across a time duration (e.g., about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 seconds, about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 minutes, about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 hours, etc). Alternatively, or in addition, raw data may be aggregated across data types or sources and sent to a remote entity as a package.

[0057] The data orchestrator may deliver data across data centers, cloud applications, or any component that resides in the data centers (e.g., associated with third-party entities). The data orchestrator may determine which of the data or which portion of the data is to be transmitted to which data centers and/or entities and when to transmit this portion of data. For example, some of the autonomous vehicle data (e.g., a first portion of data or package of data) may need to be communicated immediately or when the autonomous vehicle is in motion, whereas other data (e.g., a second portion of data or package of data) may be communicated when the autonomous vehicle is stationary (while waiting for the next assignment/task or being maintained). In a further example, a first portion of data may be transmitted to a data center hosting a fleet manager application for providing real-time feedback and control based on real-time data, whereas a second portion of data (e.g., batch data) may be transmitted to an insurance company server to compute insurance coverage based on the batch data. In some embodiments, data delivery or data transmission may be determined based at least in part on a predictive model and/or hand-crafted rules. In some embodiments, data transmission may be initiated based on the predictive model, hand-crafted rules, and repository that stores data about the destination and transmission protocol. In an example, the data orchestrator 100 may support services for data aggregation, and data publishing for sending aggregated data to the cloud, different data centers, or entities for further analysis. Details about the data orchestrator and the predictive model are described later herein.

[0058] A predictive model creation and management system 130 may include services or applications that run in the cloud or an on-premises environment to remotely configure and manage the data orchestrator 100. This environment may run in one or more public clouds (e.g., Amazon Web Services (AWS), Azure, etc.), and/or in hybrid cloud configurations where one or more parts of the system run in a private cloud and other parts in one or more public clouds. For example, the predictive model creation and management system 130 may be configured to train and develop predictive models. In some cases, the trained predictive models may be deployed to the data orchestrator or an edge infrastructure through a predictive model update module. Details about the predictive model update module are described with respect to FIG. 15. In some cases, the predictive model creation and management system 130 may also be able to translate machine learning models developed in the cloud into sensor expressions that can be executed at the edge. The predictive model creation and management system 130 may also support ingesting data transmitted from the autonomous vehicle into one or more databases or cloud storages 123, 125, 127. The predictive model creation and management system 130 may include applications that allow for integrated administration and management, including monitoring or storing of data in the cloud or at a private data center. In some embodiments, the predictive model creation and management system 130 may comprise a user interface (UI) module for viewing analytics, sensor data (e.g., video), or comprise a management UI for developing and deploying analytics expressions, deploying data orchestration applications to the edge (e.g., autonomous vehicle operating system, edge gateway, edge infrastructure, data orchestrator), monitoring predictive model performance, and configuring a predictive model. It is noted that although the predictive model creation and management system is shown as a component of the data center, the predictive model creation and management system can be a standalone system.

[0059] A model monitor system may monitor data drift or performance of a model in different phases (e.g., development, deployment, prediction, validation, etc). The model monitor system may also perform data integrity checks for models that have been deployed in a development, test, or production environment.

[0060] Data monitored by the model monitor system may include data involved in model training and during production. The data at model training may comprise, for example, training, test and validation data, predictions and scores made by the model for each data set, or statistics that characterize the above datasets (e.g., mean, variance and higher order moments of the data sets). Data involved in production time may comprise time, input data, predictions made, and confidence bounds of predictions made. In some embodiments, the ground truth data may also be monitored. The ground truth data may be monitored to evaluate the accuracy of a model and/or trigger retraining of the model. In some cases, users may provide ground truth data to the model monitor system or a model management platform after a model is in production. The model monitor system may monitor changes in data such as changes in ground truth data, or when new training data or prediction data becomes available.

[0061] The model monitor system may be configured to perform data integrity checks and detect data drift and accuracy degradation. The process may begin with detecting data drift in training data and prediction data. During training and prediction, the model monitor system may monitor difference in distributions of training data, test, validation and prediction data, change in distributions of training data, test, validation and prediction data over time, covariates that are causing changes in the prediction output, and various others. Alerts on model accuracy may be generated and delivered when new ground data becomes available. The model monitor system may also provide dashboards to track model performance/model risk for a portfolio of models based on the training/prediction data and model registration data collected as part of data drift, accuracy and data integrity checks.

[0062] The model monitor system may register information about the model and the data that was used to train/build the model. The model monitor system may define but may not restrict a model to be an artifact created or trained by applying an algorithm to the training data, and then deployed to make predictions against real data. A model may be associated with an experiment and may evolve over time as different data is provided to the model and/or parameters are tuned. The model monitor system may comprise a model ID generator component that generates a model ID (e.g., model id) uniquely associated with a model. The model ID may be deploymentwide unique and monotonically increasing as described elsewhere herein.

[0063] During prediction time, once a model is registered with the model monitor system, predictions may be associated with the model in order to track data drift or to incorporate feedback from new ground truth data.

[0064] The model monitor system may allow users to perform data checks. For example, users may perform data checks on the training and prediction data that has been registered with the system. Various data checks may be provided by the model monitor system, including but not limited to, values outside/within a range either in batch mode or across different sliding/growing time windows, data type checks either in batch mode or across different sliding/growing time windows, data distribution hasn’t changed at all over time as an indicator that something is suspect, or changes in volume of prediction/training data being registered over time.

[0065] The provided data management system may employ any suitable technologies such as container and/or micro-service. For example, the application of the data orchestrator can be a containerized application. The data management system may deploy a micro-service based architecture in the software infrastructure at the edge such as implementing an application or service in a container. In another example, the cloud applications and/or the predictive model creation and management system 130 may provide a management console or cloud analytics backed by micro-services.

[0066] Container technology virtualizes computer server resources like memory, central processing unit (CPU), and storage that are managed by an operating system (OS) with negligible overhead without requiring replication of the entire OS kernel for each tenant (and hence unlike a hypervisor technology). Containers were developed as a part of the popular Linux open-source operating system and have gained significant traction in software development and datacenter operations (“DevOps”) with the availability of advanced administration frameworks like Docker and CoreOS. Other container orchestration framework such as Kubernetes may also be utilized. Kubemetes provides a high-level abstraction layer called a "pod" that enables multiple containers to run on a host machine and share resources without the risk of conflict. A pod can be used to define shared services, like a directory or storage, and expose it to all the containers in the pod. There is growing demand to consume software and analytics for processing sensor data over nearline compute infrastructure very close to physical sensor networks in the Internet of Things (loT) use-cases (that include physical locations like factories, warehouses, retail stores, and other facilities). These compute nodes include, for example, servers from medium-size (e.g., a dualcore processor and 4 gigabytes of memory) to miniaturized size (e.g., a single core processor core with less than 1 gigabyte of memory) which are connected to the Internet and have access to a variety of heterogeneous sensor devices and control systems deployed in operations. The data management system provides methods for deploying and managing container technologies intelligently in these edge compute infrastructure settings.

[0067] The data center or remote entity 120 may comprise one or more repositories or cloud storage for storing autonomous vehicle data and metadata. For example, a data center 120 may comprise a metadata database 123, a cloud data lake for storing autonomous vehicle stack data 125, and a cloud data lake for storing user experience platform data 127. A user experience platform as described herein may comprise hardware and/or software components that are operating inside of a vehicle’s cabin. The user experience platform can be configured to manage the cabin’s environment and the occupants’ interactions, for example cabin temperature, per occupant entertainment choices, each occupant’s vital signs, mood and alertness, etc. In some cases, the metadata database 123 and/or the cloud data lake may be a cloud storage object.

[0068] An autonomous vehicle stack may consolidate multiple domains, such as perception, data fusion, cloud/OTA, localization, behavior (a.k.a. driving policy), control and safety, into a platform that can handle end-to-end automation. For example, an autonomous vehicle stack may include various runtime software components or basic software services such as perception (e.g., ASIC, FPGA, GPU accelerators, SIMD memory, sensors/detectors, such as cameras, Lidar, radar, GPS, etc.), localization and planning (e.g., data path processing, DDR memory, localization datasets, inertia measurement, GNSS), decision or behavior (e.g., motion engine, ECC memory, behavior modules, arbitration, predictors), control (e.g., lockstep processor, DDR memory, safety monitors, fail safe fallback, by-wire controllers), connectivity, and I/O (e.g., RF processors, network switches, deterministic bus, data recording). The autonomous vehicle stack data may include data generated by the autonomous stack as described above. The user experience platform data 127 may include data related to user experience applications such as digital services (e.g., access to music, videos or games), transactions, and passenger commerce or services. For example, the user experience platform data may include data related to subscriptions to access content, e.g., an annual subscription to a music streaming service, a news service, a concierge service, etc.; transaction-based purchase of goods, services, and content while being transported, as well as when vehicles intermittently stop, such as at refueling stations, restaurants, coffee shops, etc. (e.g., a recharging station operator, such as an energy company, can partner with a coffee shop chain to offer discounts in coffee drinks to passengers who purchase while refueling a vehicle); and redemption of loyalty points, e.g., automakers and fleet operators can reward their customers for their loyalty, using a system similar to that used by airlines or hotel chains where the loyalty points can be redeemed in much the same way these and other industries use such programs. In some cases, the user experience platform data 127 may also include third-party partner data such as data generated by a user mobile application. A user can be a fleet operator or passenger.

[0069] The cloud applications 121, 122 may further process or analyze data transmitted from the autonomous vehicle for various use cases. The cloud applications may allow for a range of use cases for pilotless/driverless vehicles in industries such as original equipment manufacturers (OEMs), hotels and hospitality, restaurants and dining, tourism and entertainment, healthcare, service delivery, and various others. In particular, the provided data management systems and methods can be applied to data related to various aspects of the automotive value chain including, for example, vehicle design, test, and manufacturing (e.g., small batch manufacturing and the productization of autonomous vehicles), creation of vehicle fleets that involves configuring, ordering services, financing, insuring, and leasing a fleet of vehicles, operating a fleet that may involve service, personalization, ride management and vehicle management, maintaining, repairing, refueling and servicing vehicles, and dealing with accidents and other events happening to these vehicles or by a fleet.

[0070] FIG. 2 shows examples of data flows across various data centers, autonomous vehicles, and entities. As shown in the example, data generated during autonomous vehicle fleet (AV fleet) 201 and by consumers using autonomous vehicle fleets 202 may be transmitted to various remote entities with aid of a data orchestrator 220. The various remote entities may include, for example, government 211, fleet leasing company 209, insurance company 207, fleet manager 205, fleet operator 203, digital services 217, other transport services 219 (e.g., such as train and shuttle, ride-sharing, ride-hailing service, shared trip or private trip, walk, bicycle, e-scooter, taxi, etc), platform provider 215, and original equipment manufacturer (OEM) 213. Data transmitted to the various entities may or may not be the same. For instance, data transmitted to digital services 217 (e.g., include more consumer related data or passenger related data) may be different from data (e.g., include more AV fleet data or sensor data) transmitted to fleet operators 203. Data may be transmitted to the various entities at different time points and/or frequency. For instance, sensor data stream may be sent to fleet manager 205 or platform provider 215 in real-time or while the vehicle is in motion, whereas a message package comprising batch data may be sent to government 211 or fleet leasing company 209 while the vehicle is at rest or at lower frequency. In some embodiments, an application repository or Application Table may be used to store information related to data transmission between the vehicle/vehicle application and a remote entity/cloud application. The Application Table may be a component of the data orchestrator described elsewhere herein. In some cases, one or more cloud applications (e.g., cloud applications 121, 122 or tenant applications) running on a cloud or remote entity may register in an Applications Table. The cloud application may be linked to one or more applications (e.g., edge applications, local applications) running on the autonomous vehicle or operating system of the autonomous vehicle. The Application Table may store data related to the specific vehicle data (e.g., type of data, pointer to the data to be transmitted) that a cloud application is interested in, the application (e.g., application running on the autonomous vehicle) that generates the specific vehicle data, applications and/or data centers a specific data is to be transmitted to (e.g., cloud applications 121, 122, location of an application on a server), data transmission scheme or protocol (e.g., timing of transmission such as delay time or frequency, communication protocol, compression or encryption method used for transmission), and various others (e.g., regulatory rules regarding privacy before data is transmitted).

[0071] The data orchestrator 220 may also be part of a connected vehicle, a connected and automated vehicle, or a connected and autonomous vehicle. FIG. 20 shows an environment in which the data orchestrator may be implemented. The vehicle 2001-1, 2001-2 may be privately owned or may be part of a fleet. The vehicle may be used for passenger transportation, long-haul or short-haul logistics, last-mile delivery (e.g., delivery within 5 miles, 4 miles, 3 miles, 2 miles, or 1 mile), or have mixed use (e.g., passengers and packages). It is noted that abovementioned data can be stored as any other suitable data structures. In some cases, the Application Table may be stored in a local storage and managed by the data orchestrator. In addition to or alternatively, the Application Table may be managed by both the data orchestrator and the predictive model creation and management system.

[0072] In some cases, the applications running on cloud or a remote entity (e.g., public clouds such as Amazon Web Services (AWS), and Azure, or private cloud) may register in the Application Table of a particular vehicle’s data orchestrator (or the data orchestrators of a fleet of vehicles) through a publish/subscribe scheme. In some cases, an application that is running on the fog/edge servers or a remote entity may register in the Applications Table through a publish/subscribe scheme. In some embodiments, a Registering Application may specify the Vehicle IDs from which it needs to receive data and/or the particular Vehicle Application(s) running on the corresponding vehicles it needs to receive data from.

[0073] In some embodiments, data requests that are generated by the Registering Applications may be organized and managed by a Cloud’s Subscription Module and the data requests may be communicated Over The Air (OTA) to one or more relevant vehicles via a message. A message may include one or more requests for one or more vehicle applications. In some cases, a request included in a message received by a vehicle may be registered in the Application Table. The Subscription Module may be configured to manage the data requests or registering application request. For instance, the Subscription Module may be capable of aggregating multiple registering application requests thereby beneficially reducing communication bandwidth consumption. For example, multiple registering application requests about requesting data from the same vehicle application (e.g., the Pothole Detector application) may be aggregated. In other examples, multiple registering application requests about requesting data from the different vehicle application running on a specific group of vehicles (e.g., all BMW Model 3 vehicles manufactured between 2010-2015) may be aggregated and packaged into a single message.

[0074] FIG. 3 shows an example of an Application Table 300. The Application Table 300 may be part of a data orchestrator (see, e.g., FIG. 4). An entry of the Application Table 300 may store data as described above. For example, a row of the Application Table may include the name of the application (e.g., Pothole detector) that is running in the autonomous vehicle (i.e., vehicle application). The application may generate data to be communicated to a remote entity, data center, or cloud application. A row of the Application Table may also include a flag (e.g., transmission flag) indicating whether new data is available for transmission to one or more applications running in specific data centers, an identifier of the vehicle where the data is generated (e.g., Vehicle ID), a cloud application (e.g., application name, location of the application on the cloud) or data center requesting data from this application and where the data is to be sent (e.g., app name, loc), the type of data to be transmitted (e.g., video stream, CAN data), a pointer to the actual data (e.g., Stream 1) to be transmitted from the vehicle, time of transmission (e.g., transmission timing delay), compression type (e.g., lossless), encryption type (e.g., RSA), and regulation rules. It is noted that the illustrated Application Table is merely an example. Any other data related to data transmission can be included in the Application Table. [0075] In some cases, one or more entries may be set by the local/vehicle application. For example, a transmission flag indicating whether requested data is available for transmission may be set by the local/vehicle application. In some cases, one or more entries may be set by the data orchestrator. For example, vehicle ID or regulatory rules may be set by the data orchestrator.

[0076] In some embodiments, the cloud data lakes may organize data around each vehicle in a fleet. For example, data from a particular AV Stack and a particular User Experience Platform may be organized and stored in association with a corresponding vehicle (e.g., Vehicle ID). As described above, a vehicle may register in the cloud data lake and may be identified by its Vehicle ID, the various data-acquisition applications it uses, the sensors that are accessed by each data-acqui sition application, the capabilities of each sensor, (e.g., a sensor can capture data every 5 seconds, or a sensor can capture video of 720p resolution) and others. In some cases, a user, an entity in the network, or a party registered to the system may be allowed to automatically derive additional information such as the vehicles, make, model, and year of manufacture of a vehicle using the Vehicle ID. In some cases, a vehicle can be part of a fleet (e.g., a corporate fleet, fleet a car rental company, a collection of privately-owned vehicles made by a specific OEM) which registers with the data management system.

Data orchestrator

[0077] A data orchestrator may be local to or onboard the autonomous vehicle. In some examples, the data orchestrator resides on the autonomous vehicle. As described above, a data orchestrator may also be part of a connected vehicle, a connected and automated vehicle, or a connected and autonomous vehicle. The provided data management system may employ an edge intelligence paradigm that data orchestration is performed at the edge or edge gateway. In some instances, one or more machine learning models may be built and trained on the cloud/data center and run on the vehicle or the edge system (e.g., hardware accelerator).

[0078] In some cases, the data orchestrator may be implemented using in part an edge computing platform or edge infrastructure/system. The edge computing platform may be implemented in software, hardware, firmware, embedded hardware, standalone hardware, application specifichardware, or any combination of these. The data orchestrator and its components, edge computing platform, and techniques described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These systems, devices, and techniques may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. These computer programs (also known as programs, software, software applications, or code) may include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and/or device (such as magnetic discs, optical disks, memory, or Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor.

[0079] In some embodiments, the provided data orchestrator may be capable of determining which of the autonomous vehicle data or which portion of the autonomous vehicle data is to be communicated to which data center or third party entity, and when this portion of data is transmitted. The data transmission or data delivery may be determined using the Application Table, rules, and predictive models. The predictive model may be a machine learning-based model.

[0080] Machine learning has evolved as a key computation construct in automating discovery of patterns in data and using the models built to make intelligent predictions in a variety of applications. Artificial intelligence, such as machine learning algorithms, may be used to train a predictive model for data orchestration. A machine learning algorithm may be a neural network, for example. Examples of neural networks include a deep neural network, convolutional neural network (CNN), and recurrent neural network (RNN). The machine learning algorithm may comprise one or more of the following: a support vector machine (SVM), a naive Bayes classification, a linear regression, a quantile regression, a logistic regression, a random forest, a neural network, CNN, RNN, a gradient-boosted classifier or repressor, or another supervised or unsupervised machine learning algorithm.

[0081] FIG. 4 schematically shows an example of a data orchestrator 410 in communication with one or more remote entities 420, 430. In some embodiments, the data orchestrator 410 may comprise a decision engine 413 and a data communication module 415. In some cases, the data orchestrator 410 may optionally comprise a data processing module 411. In some cases, the data processing module 411 may provide pre-processing of stream data and batch data. In alternative embodiments, the data processing module 411 may reside on the cloud 420 such as a component of the data and metadata management module 425. The data orchestrator 410 may be coupled to or have one or more local databases such as an applications repository 405 and/or a predictive models knowledge base 407. The applications repository 405 may store Application Tables as described above. The predictive models knowledge base 405 may be configured to store machine learning models and/or hand-crafted rules for determining a data transmission (scheme). The data transmission scheme may specify which of the autonomous vehicle data to be communicated to which data center or third party entity, and when such data is transmitted. The predictive models knowledge base 405 may store other models in addition to the machine learning models used by the data orchestrator. For example, the predictive models knowledge base 405 may store models that are used for the vehicle's autonomous mobility, models used for personalization of a cabin(s) of the vehicle and other functions performed inside the cabin(s), and/or models used for the safe and optimized operation of a fleet.

[0082] The data orchestrator 410 may be in communication with a predictive model management module 421. The predictive model management module 421 can be the same as the predictive model creation and management system 130 as described in FIG. 1. The predictive model management module 421 may reside on a remote entity 420 such as a data center, a cloud, a server, and the like. In some cases, the predictive model management module 421 may include services or applications that run in the cloud or an on-premises environment to remotely configure and manage the data orchestrator 410 over a network. In some cases, the predictive model management module 421 is a standalone system. In some cases, the predictive model management module 421 may be a component of a data center and the data center may host one or more applications 423 that utilize the autonomous vehicle data.

[0083] The aforementioned applications repository 405 can be the same as the Application Tables or include the Application Tables as described above.

[0084] The predictive models knowledge base 407 may store machine learning models and/or hand-crafted rules. In knowledge-based environments, the availability and leveraging of information, coupled with associated human expertise, is a critical component for improved process, implementation, and utilization efficiencies. A knowledge base provides a plethora of information about a specific subject matter in multiple data sources that can be accessed from global locations with Internet access, or other relevant technologies.

[0085] The applications repository 405, predictive models knowledge base 407, one or more local databases, metadata database 427, and cloud databases 429 of the system may utilize any suitable database techniques. For instance, structured query language (SQL) or “NoSQL” database may be utilized for storing the fleet data, passenger data, historical data, predictive model or algorithms. Some of the databases may be implemented using various standard data- structures, such as an array, hash, (linked) list, struct, structured text file (e.g., XML), table, JavaScript Object Notation (JSON), NOSQL and/or the like. Such data-structures may be stored in memory and/or in (structured) files. In another alternative, an object-oriented database may be used. Object databases can include a number of object collections that are grouped and/or linked together by common attributes; they may be related to other object collections by some common attributes. Object-oriented databases perform similarly to relational databases with the exception that objects are not just pieces of data but may have other types of functionality encapsulated within a given object. In some embodiments, the database may include a graph database that uses graph structures for semantic queries with nodes, edges and properties to represent and store data. If the database of the present invention is implemented as a data-structure, the use of the database of the present invention may be integrated into another component such as the component of the present invention. Also, the database may be implemented as a mix of data structures, objects, and relational structures. Databases may be consolidated and/or distributed in variations through standard data processing techniques. Portions of databases, e.g., tables, may be exported and/or imported and thus decentralized and/or integrated.

[0086] In some embodiments, the data management system may construct the database for fast and efficient data retrieval, query and delivery. For example, the data management system may provide customized algorithms to extract, transform, and load (ETL) the data. In some embodiments, the data management system may construct the databases using proprietary database architecture or data structures to provide an efficient database model that is adapted to large scale databases, is easily scalable, is efficient in query and data retrieval, or has reduced memory requirements in comparison to using other data structures. For example, a model tree may be stored using a tree data structure with nodes presenting different versions of a model and node parameters representing a model’s goal, performance characteristics and various others.

[0087] In some embodiments, the data orchestrator may be applied to a multi-tier data architecture. FIG. 18 schematically illustrates a multi-tier data architecture 1800. In the illustrated example, the data orchestrator may be a software-based solution based on fog or edge computing concepts as described above. In some cases, the multi-tier data architecture may comprise a vehicle layer (e.g., in-vehicle data 1810), a fog layer (e.g., fog/edge data 1820) and a cloud layer (e.g., cloud data 1830). The multi-tier data architecture may comprise any number of layers. For instance, a fog layer may comprise one or more layers. Data at the vehicle layer may comprise in-vehicle data 1810 generated by the user experience platform 401 and/or the vehicle stack 403, sensors onboard the vehicle, and various other sources as described elsewhere herein. Data at the vehicle layer may be the same as the autonomous vehicle data as described above. Data at the fog layer (e.g., fog/edge data 1820) may be generated, managed and directly accessed by the data orchestrator. The fog/edge data 1820 may comprise data after processed by the data processing module 411. The data processing module 411 may support ingesting of sensor data into a local storage repository (e.g., local time-series database), data cleansing, data enrichment (e.g., decorating data with metadata), data alignment, data annotation, data tagging, data aggregation, and various other data processing. The fog/edge data 1820 may also comprise intermediary data to be transmitted to the cloud according to a transmission scheme.

[0088] The data orchestrator may be configured to or capable of determining which of the vehicle data or which portion of the vehicle data stays in the in-vehicle database, is to be moved/transmitted to the fog layer database (e.g., fog/edge database), and which of the fog/edge data or which portion of the fog/edge data is to be communicated to which data center or third party entity, when and at what frequency this portion of data is transmitted. In some cases, data that is off-loaded or moved to the edge/fog database may be deleted from the in-vehicle database for improved storage efficiency. Alternatively, data in the in-vehicle database may be preserved for a pre-determined period of time after it is off-loaded to the edge/fog database.

[0089] FIG. 19 schematically shows an example of a data orchestrator 1910 for managing data transmission between a vehicle layer and fog layer, and between the fog layer and a cloud layer. The data transmission or data delivery among the multiple layers may be determined using the Application Table, rules, and predictive models. The predictive model may be a machine learning-based model as described above.

[0090] The data orchestrator 1910 can be the same as the data orchestrator 410 as described above. For example, the data orchestrator 1910 may comprise a decision engine 1913 and a data communication module 1915. In some cases, the data orchestrator 1910 may optionally comprise a data processing module (not shown). In some cases, the data processing module may provide pre-processing of stream data and batch data transmitted from the in-vehicle database 1920. The in-vehicle database 1920 may be on-board a vehicle and store vehicle data (e.g., in-vehicle data 1810). The data orchestrator may manage data transmission between an in-vehicle database 1920 and a fog/edge database 1930, and between a fog/edge database 1930 and a cloud database.

[0091] The data orchestrator 1910 may be coupled to or have one or more local databases such as an applications repository 405 and/or a predictive models knowledge base 407 as described above. The applications repository 405 may store Application Tables as described above. The predictive models knowledge base 405 may be configured to store machine learning models and/or hand-crafted rules for determining a data transmission (scheme). The data transmission scheme may specify which of the vehicle data or which portion of the vehicle data stays in the in- vehicle database 1920, is to be moved/transmitted to the fog layer database (e.g., fog/edge database 1930), and when and/or at what frequency such data is transmitted. The data transmission scheme may also specify which of the fog/edge data or which portion of the fog/edge data is to be communicated to which data center or third-party entity, when and at what frequency this portion of data is transmitted. The predictive models knowledge base 405 may store other models in addition to the machine learning models used by the data orchestrator. In some cases, the predictive models knowledge base 405 may not or need not be the same as the knowledge base of the system which may store models that are used for the vehicle's autonomous mobility, models used for personalization of a cabin(s) of the vehicle and other functions performed inside the cabin(s), and/or models used for the safe and optimized operation of a fleet.

[0092] FIG. 5 illustrates an example of a predictive models knowledge base 500. In some embodiments, a predictive models knowledge base 500 may comprise an Automotive Ontology 501 and one or more model trees 503. In some embodiments, the predictive models knowledge base may include both hand-crafted rules and machine learning-based predictive models. The hand-crafted rules and machine learning-based predictive models may independently or collectively determine rules or protocols regulating data transmission. For example, the rules may specify applications and/or data centers a given aggregation of data (e.g., Message package) is to be transmitted to, the aggregation of data to be transmitted, data transmission scheme (e.g., timing of transmission such as delay time or frequency, communication protocol, compression or encryption method used for transmission), and various others (e.g., regulatory rules regarding privacy before data is transmitted).

[0093] The hand-crafted rules may be imported from external sources or defined by one or more users (e.g., the hand-crafted rules may be user-defined rules). In some cases, the hand-crafted rules may be provided by a remote application that requests data from the vehicle. In some cases, a data transmission scheme may be determined based on a request from a remote application. In some cases, the request may be a request sent from a remote third-party application (e.g., application 423, 430) to an intermediary component (e.g., original equipment manufacturer (OEM)). For instance, an insurance application may request certain type of data from an OEM system associated with a vehicle (e.g., data collected by OEM-embedded devices) at a predetermined frequency (e.g., a week, two weeks, a month, two months, etc) for purpose of understanding whether the driver may be driving excessively compared to the insurance rate he is paying, creating new insurance products, providing discounts to drivers for safety features, assessing risk, accident scene management, first notice of loss, enhancing claims process and the like.

[0094] The request may contain information about the type of data needed by the application, the frequency with which the data are needed, a period of time for such type of data to be transmitted or other information. In some situations, when the data transmission is infrequent or the amount of data to be transmitted is relatively small, a data transmission scheme may be generated based on the aforementioned request without using an intelligent transmission scheme such as one that can be created using the machine learning models. For instance, a requesting application (e.g., insurance application) may send to the OEM system associated with a target vehicle (or group of vehicles) a request indicating the type of data and the frequency of such data are needed from the target vehicle. In some cases, the request may specify a group of vehicles. For instance, the request may specify a particular model (e.g., Audi A8), a model year (e.g., 2017), a model with specific driving automation features (e.g., A8 with lane change monitor), and the like. The OEM system may pass the request (e.g., send a request message to relay the request) to the data orchestrator of the respective target vehicle. Upon receiving the request, the data orchestrator may push the request to a queue and send back a response message to the OEM system to acknowledge receipt of the request. The OEM system may then send a message to the requesting application indicating the request has been logged.

[0095] Next, the data orchestrator may transmit the requested data based on the information contained in the request. The data orchestrator may send the requested data directly to the requesting application. In such cases, information related to data transmitted from the data orchestrator to the remote application (e.g., requesting application) may be communicated through an intermediary entity (e.g., OEM system). For example, in addition to passing the request/response message, the OEM system/application may send a message to the data orchestrator instructing the data orchestrator to delete the transmission request from the queue when a transmission period is completed (e.g., upon receiving a completion message from the data orchestrator). The data orchestrator may then delete the entry from the queue and send a message to the OEM system indicating the entry is deleted. The OEM system may send a message to the requesting application indicating the request is completed.

[0096] The predictive models knowledge base 500 may store other models in addition to the machine learning models used by the data orchestrator. For example, the predictive models knowledge base 500 may store models that are used for the vehicle's autonomous mobility, models used for cabin(s) personalization and other functions performed inside the vehicle and/or a cabin(s) of the vehicle, and/or models used for the safe and optimized operation of a fleet. Models stored in the predictive models knowledge base 500 may include predictive models used by the data orchestrator, predictive models that are being used by the Autonomous Vehicle Stack, models that are used by the user experience platform, or a fleet management system. Alternatively, predictive models that are being used by the Autonomous Vehicle Stack, and models that are used by the user experience platform may be stored in a predictive models knowledge base managed by the respective Autonomous Vehicle Stack or the user experience platform separately.

[0097] The Automotive Ontology 501 can be developed manually by one or more individuals, organizations, imported from external systems or resources, or may be partially learned using machine learning systems that collaborate with users (e.g., extracting automotive terms from natural language text). In some cases, a portion of the Automotive Ontology may be based on data from the model tree. For example, description of a goal and/or insight of a model may be stored in a node of the model tree whereas the description of the goal and/or insight may also be a part of the Automotive Ontology.

[0098] The predictive models knowledge base 500 may store other ontologies or models. In some cases, scenario metadata may be created to specify the characteristics of the scenario using a specific metadata which is then used to retrieve the appropriate vehicle data from the database. The predictive models knowledge base may include hierarchical scenarios ontology that can be used to create new scenarios as well as to create a scenario in various levels of details. For instance, a scenario described at a higher level of detail (i.e., higher level information about the scenario), may be used to create a low-fidelity simulation or predictive model, whereas the same scenario described at a lower level of detail (i.e., more detailed lower level information about the scenario) may be used to produce a high-fidelity simulation or predictive model.

[0099] The one or more model trees 503 may be a collection of tree structures. A tree structure may comprise one or more nodes 507 with each node including the characteristics of a predictive model and pointers to the data (e.g., training data, test data) that are used to generate the predictive model. The actual data (e.g., training data, test data) may be stored in the cloud database 429. The cloud database 429 can be the same as the cloud data lakes 125, 127, or include either of or both the cloud data lakes 125, 127. The hierarchy of nodes in a given model tree may represent the versions of a particular predictive model and the relationships between the models. The characteristics of a predictive model may include, for example, a predictive model’s goal/function, model performance characteristics and various others. A node 507 may also store model parameters (e.g., weights, hyper-parameters, etc.), metadata about the model parameters, a model’s performance statistics, or model architecture (e.g., number of layers, number of nodes in a layer, CNN, RNN). In some cases, a node 507 may further include information about the computational resource(s) (e.g., one graphics processing unit (GPU), two GPUs, three CPUs, etc.) required to execute a model. A node may include all or any combination of the data as described above. [00100] In some cases, the various predictive models may be stored using different model tree structures. A knowledge base may have different model tree structures depending on, for example, where the predictive models are being used. For example, the model tree structure for storing the predictive model used by a user experience platform may be different from the model tree structure storing the predictive model used by the data orchestrator.

[00101] The model tree may be dynamic. For example, a new node may be created in response to changes to the model’s original architecture, changes to the model’s performance characteristics, or changes to the training data, or test data.

[00102] In some cases, the predictive model knowledge base may also store hand-crafted rules. The hand-crafted rules can be developed manually by one or more individuals, organizations, or imported from external systems or resources. The hand-crafted rule and the predictive model may be applied independently, sequentially or concurrently.

[00103] Below shows an example of a rule regarding data transmission in accordance with the Application Table described in FIG. 3:

If the Transmission Flag of the Pothole Detector App is Set

Then

Determine when the data must be transmitted

For each Vehicle lD with Transmission_Flag=SET

Receive the video stream from the vehicle

Apply all applicable regulation rules

Encrypt the data set that results after applying the applicable regulation rules and creating the Enrypted Data File

Compress the Enrypted Data File creating the Compressed Data File

Create a Message Package consisting of [Compressed Data File, File Size, Transmission Delay, Data Center Address List]

Send the Message Package to the Communication-Module queue

[00104] In some embodiments, the data transmission scheme may also specify how data are transmitted. For instance, the data transmission scheme may specify compression methods (e.g., lossless compression algorithm, lossy compression algorithms, encoding, etc), or encryption methods (e.g., RSA, triple DES, Blowfish, Twofish, AES, etc) used for transmission. In some cases, a data compression method and/or encryption method may be determined for a transmission based on rules. For example, a rule may determine the compression method and/or encryption method according to a given type of data, the application that uses the data, destination of the data and the like. The rules for determining data compression method and/or encryption method may be stored in a database accessible to the data orchestrator such as the predictive models knowledge base as described above. In some cases, the rule for determining the data compression method and/or encryption method may be part of the rule for determining the data transmission. For instance, a ruleset for determining the encryption method or compression method may be called (e.g., by ruleset identifier) for determining the data transmission scheme.

[00105] The rules for determining the compression method and/or encryption method may be hand-crafted rules. For example, pre-determined or hand-crafted rules about compression method and/or encryption method may be applied upon receiving a transmission request specifying the type of data, data related to an application, destination of data, and the like. Such hand-crafted rules may be stored in a database accessible to the data orchestrator such as the predictive models knowledge base as described above. In some cases, the compression method and/or encryption method may be determined by machine learning algorithm trained models. For instance, when a pre-determined rule set for data compression or encryption is not available (e.g., ruleset identifier is not available, type of dataset is not seen before, etc), the trained model may be applied to the set of data to be transmitted and generate a rule for compressing or encrypting the set of data. In some cases, the rule set generated by the trained model may be stored in the predictive models knowledge base for future data transmission (scheme).

[00106] Referring back to FIG. 4, the data processing module 411 may support ingesting of sensor data into a local storage repository (e.g., local time-series database), data cleansing, data enrichment (e.g., decorating data with metadata), data alignment, data annotation, data tagging, data aggregation, and various other data processing. Data from the user experience platform 401 and/or the vehicle stack 403, sensors onboard the vehicle, and various other sources as described elsewhere herein may be ingested and processed by the data processing module. For instance, the data processing module may collect or ingest data from the sensors via one or more protocols (e.g., MQ Telemetry Transport, OPC Unified Architecture, Modbus, and DDS). The data provided or outputted by the sensors may be a binary data stream. The transmission or delivery of this data from the sensors to the data processing module can be push or pull methods. In some cases, the data processing module may enrich the incoming data from the sensors by decoding the raw binary data into consumable data formats (such as JavaScript Object Notation) or also merging with additional necessary and useful metadata. In some embodiments, metadata may relate to sensors that capture sensory data (e.g., GPS, Lidar, camera, etc.), pre-processing on data (e.g., aligning and creating time series), and various applications and/or predictive models that operate on the data for a specific use case or application (e.g., avoiding pedestrians, pattern recognition, obstacle avoidance, etc). Alternatively, such data processing may be performed by an application on the cloud. For example, the data processing module 411 may reside on the cloud 420 rather than the data orchestrator. Details about the data processing method and metadata creation are described later herein.

[00107] The decision engine 413 may be configured to execute rules in the predictive models knowledge base 407. For example, the decision engine may constantly look up for rules in the predictive models knowledge base 407 eligible or ready for execution, then execute the action associated with the eligible rules and invoke the data communication module 415 to transmit the results (e.g., aggregated data, Message Package) to the destination (e.g., requested data center 420, application 431, remote entity, third party entity 431, etc.).

[00108] The data communication module 415 may send processed data or a selected portion of the autonomous vehicle data to a destination in compliance with the rules. Below shows an example of a procedure performed by the data communication module:

Step 1 Check Queue Status

Step 2 If the Queue_Status=l Then

For every Message_P ackage

Examine File Size and Transmission Delay to determine Transmission Channel Type

Select Transmission channel of Transmission Channel Type

Send Compressed Data File over selected Transmission channel to each address in Data Center Address List

If Transmission_Succeess=l Then Delete Message Package from Queue Else GoTo Send

Step 3 Goto Step 1

[00109] Various communication protocols may be used to facilitate communication between the data orchestrator and the cloud or remote entity. These communication protocols may include VLAN, MPLS, TCP/IP, Tunneling, HTTP protocols, wireless application protocol (WAP), vendor-specific protocols, customized protocols, and others. While in one embodiment, the communication network is the Internet, in other embodiments, the communication network may be any suitable communication network including a local area network (LAN), a wide area network (WAN), a wireless network, an intranet, a private network, a public network, a switched network, and combinations of these, and the like. The network may comprise any combination of local area and/or wide area networks using both wireless and/or wired communication systems. For example, the network may include the Internet, as well as mobile telephone networks. In one embodiment, the network uses standard communications technologies and/or protocols. Hence, the network may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G/3G/4G or Long Term Evolution (LTE) mobile communications protocols, Infra-Red (IR) communication technologies, and/or Wi-Fi, and may be wireless, wired, , asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, or a combination thereof. Other networking protocols used on the network can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), and the like. The data exchanged over the network can be represented using technologies and/or formats including image data in binary form (e.g., Portable Networks Graphics (PNG)), the hypertext markup language (HTML), the extensible markup language (XML), etc. In addition, all or some of links can be encrypted using conventional encryption technologies such as secure sockets layers (SSL), transport layer security (TLS), Internet Protocol security (IPsec), etc. In another embodiment, the entities on the network can use custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above. The network may be wireless, wired, or a combination thereof.

Predictive model creation and management system

[00110] The predictive model management module 421 can be the same as the predictive model creation and management system as described in FIG. 1. The predictive model management module 421 may include services or applications that run in the cloud or on-premises environment to remotely configure and manage the data orchestrator 410 or one or more components of the data orchestrator (e.g., predictive models knowledge base).

[00111] In some embodiments, the predictive model management module 421 may comprise a model creator and a model manager. In some cases, a model creator may be configured to train, develop or test a predictive model using data from the cloud data lake and metadata database. The model manager may be configured to manage data flows among the various components (e.g., cloud data lake, metadata database, data orchestrator, model creator), provide precise, complex and fast queries (e.g., model query, metadata query), model deployment, maintenance, monitoring, model update, model versioning, model sharing, and various others. For example, the deployment context may be different depending on edge infrastructure and the model manager may take into account the application manifest such as edge hardware specifications, deployment location, information about compatible systems, data-access manifest for security and privacy, emulators for modeling data fields unavailable in a given deployment and version management during model deployment and maintenance.

[00112] The data management provided by the predictive model management module can be applied across an entire lifecycle of the automated and autonomous vehicles. For example, the data management may be applied across a variety of applications in the vehicle design phase, vehicle/fleet validation phase or the vehicle/fleet deployment phase. FIG. 14 shows examples of varieties of applications in a lifecycle of automated and autonomous vehicles. For instance, data management can be used in creating new models or updating existing models in the vehicle design phase, in the vehicle/fleet validation phase, or in the vehicle/fleet deployment phase.

[00113] FIG. 6 shows an example of a model creator 600 interacting with a metadata database 427 and a cloud data lake 429 for training and developing a predictive model. The trained predictive model may be tested for performance, then the predictive model meets the performance requirement may be inserted into the predictive model knowledge base 407. In some embodiments, the cloud database 429 may include both cloud data lakes 125, 127 as described in FIG. 1

[00114] The model creator may be configured to develop predictive models used by the data orchestrator, predictive models that are being used by the Autonomous Vehicle Stack, predictive models that are used by the user experience platform, by a fleet management system and various others. The model creator may train and develop predictive models that are used for the vehicle's autonomous mobility, for vehicle cabin(s) personalization and other functions performed inside the vehicle and/or vehicle cabin(s), for the safe and optimized operation of a fleet, and/or various other applications in addition to data management and data orchestration.

[00115] FIG. 7 illustrates a method 700 of creating a predictive model. The method or process may be performed by the model creator as described above. In order to generate a predictive model, model goals and performance characteristics (e.g., accuracy) may be determined (operation 701). Additionally, desired data characteristics (e.g., completeness, validity, accuracy, consistency, availability and timeliness) may be determined (operation 702). Next, labeled data or datasets may be selected from the database (e.g., the cloud data lake) for training the model (operation 703). In some cases, retrieving data from the database may include querying metadata within the metadata database with the data characteristics, then retrieving data from the cloud data lake based on the metadata query result. If no data is returned from the cloud data lake, data characteristics may be adjusted (i.e., repeat operation 702). In some cases, the returned data or dataset may be sampled prior to the next step.

[00116] In some cases, the labeled data or dataset may be analyzed for appropriateness in view of the model goal (operation 704). For example, the labeled dataset may be determined whether is sufficient for the predictive goal, e.g., developing a predictive model that enables an autonomous vehicle to make right-hand turns automatically. Various suitable methods can be utilized to determine the appropriateness of the labeled dataset. For example, statistical power may be calculated and used for the analysis. Statistical power is the likelihood that a study will detect an effect when there is an effect there to be detected. If statistical power is high, the probability of making a Type II error, or concluding there is no effect when, in fact, there is one, goes down. Statistical power is affected chiefly by the size of the effect and the size of the sample used to detect it. Bigger effects are easier to detect than smaller effects, while large samples offer greater test sensitivity than small samples.

[00117] The analysis result produced at operation 704 may determine whether the dataset need to be corrected. The result of the appropriateness analysis may indicate whether the dataset meet the appropriate requirement, a level of appropriateness, or whether need to be corrected. For example, when the appropriateness of the labeled dataset is calculated and is below a predetermined threshold, the dataset may be determined to not meet the appropriateness requirement and may need correction. Upon determining the dataset does not need correction, the dataset may be used for training the predictive model (operation 706). In some cases, training a model may involve selecting a model type (e.g., CNN, RNN, a gradient-boosted classifier or repressor, etc), selecting an architecture of the model (e.g., number of layers, nodes, ReLU layer, etc), setting parameters, creating training data (e.g., pairing data, generating input data vectors), and processing training data to create the model. In some cases, if the dataset is analyzed and determined to need data correction, correction may be performed (operation 705). In the case when the dataset cannot be corrected, a new or different dataset may be selected from the database (i.e., repeating operation 703).

[00118] A trained model may be tested and optimized (operation 707) using test data retrieved from the predictive model knowledge base 407. Next, the test result may be compared against the performance characteristics to determine whether the predictive model meet the performance requirement (operation 708). If the performance is good i.e., meets the performance requirement, the model may be inserted into the predictive model knowledge base 407 (operation 709). [00119] In some cases, inserting a new model into the predictive model knowledge base may include determining where the new model is inserted in the model tree (e.g., added as a new node in an existing model tree or in a new model tree). Along with the new model, other data such as model goal, model architecture, model parameters, training data, test data, model performance statistics may also be archived in the model tree structure. Next, the predictive model performance may be constantly monitored by the model creator or model manager (operation 710). If the trained model does not past the performance test, the process may proceed to determine whether the poor performance is caused by the data characteristics or the model characteristics. Following the decision, operation 701 (e.g., adjusting performance characteristics) and/or operation 702 (e.g., adjusting data characteristics) may be repeated.

[00120] In some cases, upon the creation of a new predictive model or an update/change made to an existing predictive model, the predictive model may be available to the selected vehicles. For instance, once a predictive model is updated and stored in the predictive model knowledge base, the predictive model may be downloaded to one or more vehicles in the fleet. The available predictive model may be downloaded or updated in the one or more selected in a dynamic manner. FIG. 15 illustrates an example of dynamically updating predictive models in vehicles.

[00121] As described above, predictive models may include models that are used for the vehicle's autonomous mobility, for vehicle cabin(s) personalization and other functions performed inside the vehicle and/or vehicle cabin(s), for the safe and optimized operation of a fleet, and/or various other applications in addition to data management and data orchestration. A new model may be created in order to enable the vehicle to address a new situation. A model may be updated in order to improve an overall performance based on new data that has been collected and stored in the cloud data lake. In some cases, a list of the predictive models that are used by a particular vehicle in a fleet or a set of vehicles accessible by a system is maintained in a vehicles database.

[00122] In some cases, such update, change or creation of a new model may be detected automatically by a component of the predictive model management module. For example, with reference to FIG. 15, a predictive model update module 1501 may be notified by the predictive model knowledge base 1503 when a new model is created or an existing model has been updated. The predictive model update module 1501 may then select one or more vehicles to receive a copy of the updated model. The one or more vehicles may be selected or determined based on subscription, utilization of the model or other criteria. The predictive model update component may also determine when the model is updated in the selected vehicle. For instance, the predictive model update component may determine that the model needs to be updated immediately, when the vehicle is at rest (e.g., during maintenance, cleansing, repair, etc.), or on an as-needed basis. For example, in the case of a vehicle that is part of a ride-hailing fleet, a predictive model for making right hand turns at night when there is a gaming taking place in the San Francisco baseball part may be needed only if the vehicle is assigned to complete a ride that involves going through the impacted area (e.g., to pick up a passenger, to drop off a passenger, or passing through that area in the process of picking up or dropping off a passenger somewhere else). The predictive model update module 1501 may be part of the Cloud’s Subscription Module as described above.

[00123] Referring back to FIG. 4, the cloud or data center 420 or the provided vehicle data management system may also comprise a data and metadata management module 425. The data and metadata management module 425 may perform various functions including data processing conducted by the data processing module 411, as well as metadata creation and management. The data and metadata management module may be configured to store and manage the data and associated metadata that is generated by the autonomous vehicle, and process queries and API calls issued against the data and the metadata. Details about the data and metadata management module are discussed in connection with FIG. 8 to FIG. 12, for example.

[00124] The cloud or data center 420 may further comprise cloud applications 423, and a user interface (UI) module 425 for viewing analytics, sensor data (e.g., video), and/or processed data. The UI may also include a management UI for developing and deploying analytics expressions, deploying data orchestration applications to the edge (e.g., autonomous vehicle operating system, edge gateway, edge infrastructure, data orchestrator), and configuring and monitoring the data orchestration.

[00125] FIG. 8 shows example components of a vehicle data management system 800 and particularly the components of the vehicle data management system that reside on a remote entity (e.g., data center). In some embodiments, the vehicle data management system 800 may comprise a data and metadata management system and a predictive model creation and management system. The data and metadata management system can be the same as the data and metadata management module as described above. The data and metadata management system may be configured to store and manage the data and associated metadata that is generated by the autonomous vehicle, and process queries and API calls issued against the data and the metadata. In some embodiments, the vehicle data management system 800 may comprise a data and metadata management system including at least a pipeline engine 801 and a predictive model creation and management system 803. In some embodiments, the data and metadata management system may comprise other functional components such as a database query engine 805, metadata query engine 807, data system management 815, data system archiving rules 817, data system security 819, database APIs 821, regulation rules 823 and cloud-cloud communication 825. For example, the metadata database 809 can be accessed using a Metadata Query Language through the database query engine 805. In another example, data in the data lakes 811, 813 can be accessed as a result of metadata queries or be accessed directly using the database query engine 805. The cloud-cloud communication 825 may include various communication protocols such as VLAN, MPLS, TCP/IP, Tunneling, HTTP protocols, wireless application protocol (WAP), vendor-specific protocols, customized protocols, and others. The cloud-cloud communication may comprise an interface that may be used by systems (e.g., cloud-based systems) and/or devices and may utilize a variety of APIs (e.g., APIs using REST architecture, SOAP, Web Services, Enterprise Service Bus protocol or any data exchange protocol designed to provide approaches for system to system and/or process to process communication). In some embodiments, the vehicle data management system 800 may further comprise a metadata database 809, a cloud data lake for storing autonomous vehicle stack data 811 and a cloud data lake for storing user experience platform data 813.

[00126] In some embodiments, one or more of the components as described above may interact with one or more cloud applications or enterprise applications (e.g., maintain fleet 831, manage fleet 833, map update 835, configure fleet 837). The cloud applications may be hosted on the remote entity and may utilize vehicle data managed by the data management system. In some cases, the cloud application may have a database or knowledge base 832, 834, 836, 838 that is created by the predictive model creation and management system 803. In some cases, the cloud application may have permission to access and manipulate data stored in the cloud data lake for storing autonomous vehicle stack data 811, the cloud data lake for storing user experience platform data 813, or the metadata stored in the metadata database 809. In some cases, data may be dispatched to the cloud applications and, in order to dispatch data to the corresponding cloud applications (as identified in the metadata or Application Table), the predictive model creation and management system may have the addresses of all of the resources (i.e., applications) on the cloud listed locally in a table for quick lookup.

[00127] The pipeline engine 801 may be configured to preprocess continuous streams of raw data or batch data transmitted from a data orchestrator. For instance, data may be processed so it can be fed into machine learning analyses. Data processing may include, for example, data normalization, labeling data with metadata, tagging, data alignment, data segmentation, and various others. In some cases, the processing methodology is programmable through APIs by the developers constructing the machine learning analysis. [00128] FIG. 9 shows an example of data ingestion pipeline 900 and functions performed by a pipeline engine. The pipeline 900 may include a plurality of functions for processing data that is being ingested in streams or in batch. The data may include, for example, simulation data, data from the vehicle such as fleet data, operating environment data, transportation data, vehicle telemetry, data generated by AV stack and user experience platform, third-party partner data (e.g., data from user mobile application), sensor data (e.g., GPS, IMU, camera, Lidar, infrared sensor, thermal sensor, ultrasonic sensor, etc), geolocation data, and various others as described elsewhere herein. The stream data may comprise a variety of types of data including, without limitation: time series data such as spatio-temporal point measurements of an environment, multi-dimensional data such as gridded measurements from Radar, Lidar, satellites, sonars, and/or an output of a simulation process formed in array-oriented data format or any other data format suitable for representing multi-dimensional data, visualization data such as map tiles, metadata, raw data such as raw input data from sensors, documents, digital services, source code of data collection, integration, processing analysis, and various others. The batches may be tenant specific, application specific, and grouped into context aware sub-groups for parallel processing. The batches may be generated and transmitted from the data orchestrator as described elsewhere herein.

[00129] The pipeline 900 may be customizable. For example, one or more functions of the pipeline 900 may be created by a user. Alternatively, or in addition to, one or more functions may be created by the management system or imported from other systems or third-party sources. In some cases, a user may be permitted to select from a function set (e.g., available functions 920) and add the selected function to the pipeline. In some cases, creating or modifying a pipeline may be performed via a graphical user interface (GUI) provided by a user interface module (e.g., user interface module 425 in FIG. 4). For example, a set of available functions may be displayed within a GUI. A user may select, within the GUI, a graphical element representing the CREATE SCENARIOS function by clicking the function or add the function to the current pipeline by drag-and-drop.

[00130] In some cases, the graphical user interface (GUI) or user interface may be provided on a display. The display may or may not be a touchscreen. The display may be a light-emitting diode (LED) screen, organic light-emitting diode (OLED) screen, liquid crystal display (LCD) screen, plasma screen, or any other type of screen. The display may be configured to show a user interface (UI) or a graphical user interface (GUI) rendered through an application (e.g., via an application programming interface (API) executed on the user device, on the cloud or on the data orchestrator). [00131] In some embodiments, the plurality of functions may comprise third-party functions such as ingestion 901, filtering 905, cleaning 907, tagging 909, augmentation 911, annotation 913, anonymization 915, and various others (e.g., simulate). For example, data cleaning 907 may include removing noise from data (e.g., noise reduction in image processing), correcting erroneous data (e.g., one camera is malfunctioning and shows no light but it’s daytime), establishing common data formats (e.g., use metric system, all numbers to third decimal, etc.), or preparing data such that it can quickly and easily be accessed via APIs by intended data consumers or applications. In another example, data augmentation 91 Imay include combining synthetic with real data for more complete data sets to test autonomous vehicle models, enhancing captured data with data from partners to enable certain types of predictions, combining traffic congestion data with weather data to predict travel time, combining several data sets to create information-rich data, (e.g., combine vehicle operating data, with city transportation infrastructure data, and congestion data to predict vehicle arrival times during specific times of the day). In a further example, data tagging 909 or annotation 913 may include annotation of multimedia data (e.g., image, Lidar, audio) that happens at every level and creation of metadata. Metadata may be created during the movement of data in the data management environment. For instance, an image may need to be retrieved annotated (most likely with some manual intervention) and then re-indexed. The created metadata may be incorporated into the metadata catalog. Other metadata such as manually or automatically generated metadata of various types may also be inserted in the metadata catalog. The plurality of functions may also comprise proprietary functions such as data alignment 903 and create scenarios 921.

[00132] FIG. 10 shows an example data ingestion process 1000. In some cases, stream data and/or batch data may be ingested in the pipeline engine. In some cases, the ingested stream data may be delivered to a stream processing system 1001 and the ingested batch data may be delivered to an extract-transform-load (ETL) system 1003. The ETL system 1003 may perform traditional ETL functionalities or customized functionalities. For instance, the ETL system may transform the ingested batch data to a format more useful to a user. For example, the data transformation may include selecting only certain columns to load into a format, translating coded values, deriving new calculated values, sorting data, aggregating data, transposing or pivoting data, splitting a column into multiple columns, and other processing.

[00133] Though stream processing system 1001 and ETL system 1003 are discussed herein, additional modules or alternative modules may be used to implement the functionality described herein. Stream processing system and ETL system are intended to be merely exemplary of the many executable modules which may be implemented. [00134] In some cases, data alignment may be performed by the ETL system or the stream processing system. In some cases, data captured by different sensors (e.g., sensors may capture data at different frequency) or from different sources (e.g., third-party application data) may be aligned. For example, data captured by camera, Lidar, and telemetry data (e.g., temperature, vehicle state, battery charge, etc.) may be aligned with respect to time. In some cases, data alignment may be performed automatically. Alternatively, or in addition to, a user may specify the data collected from which sensors or sources are to be aligned and/or the time window during which data is to be aligned. In an example, the result data may be time-series data aligned with respect to time. It should be noted that data can be aligned along other dimensions such as application, data structure, and the like.

Metadata

[00135] The vehicle data management system may provide metadata management. In some cases, metadata creation and management may be provided by the data and metadata managements system as described above. In some cases, metadata may allow for selection of a subset of data or a portion of the autonomous vehicle data based on the metadata. In some embodiments, metadata may provide information about sensors that capture sensory data (e.g., GPS, Lidar, camera, etc.), pre-processing on data (e.g., aligning and creating time series), and various applications and/or predictive models that operate on the data for a specific use case or application (e.g., avoiding pedestrians, pattern recognition, obstacle avoidance, etc). Metadata may be created onboard the vehicle. For example, metadata may be generated by the sensors or applications running on the vehicle. In another example, metadata may be generated by the data orchestrator onboard the vehicle. Metadata may be generated remote from the vehicle or by a remote entity. For example, metadata about data processing (e.g., alignment) may be generated in the data center or by a cloud application. In some cases, at least a portion of the metadata is generated onboard the vehicle and transmitted to a remote entity. In some cases, at least a portion of the metadata is generated by a component (e.g., cloud application or pipeline engine) provided on a remote entity. The created metadata may be stored in a metadata database managed by the data management system. As an alternative or in addition to, the metadata may be stored in a database having at least some or all of the data used to generate the metadata.

[00136] FIG. 11 illustrates an example of metadata generated by alignment, application and sensor. For example, when different sensor data 1111, 1113 are aligned, metadata (e.g., alignment-created metadata 1103) may be created to provide alignment information (e.g., structure padding, frequency, time window, etc). In some cases, metadata about the sensor or sources producing the data (e.g., sensor-created metadata 1105) may be created. For example, the sensor-created metadata may include information about the sensor, identifier of the sensor, data type, and others. In some cases, metadata about the application (e.g., application-created metadata 1101) may be created by the application that process and/or generate the data. For example, the application-created metadata 1101 may provide information about the name of the application, developer of the application, application version and various others.

[00137] In some embodiments, the data management system may generate metadata of metadata for fast retrieving or querying data from the database. For example, scenario metadata may be created to specify the characteristics of the scenario using a specific metadata which is then used to retrieve the appropriate vehicle data from the database. FIG. 12 shows an example of scenario metadata 1200. In the following example, a scenario data object describing a scenario where a vehicle is making a right-hand turn to merge from a city street onto a freeway during a cloudy morning may be defined as:

Scenario_Name: e.g., right-hand merge

Scenario Type: e.g., right-hand merging into freeway

Static Objects: e.g., tree

Dynamic Objects: e.g., moving vehicle

Environment: e.g., cloudy morning

Scene Description: e.g., vehicles entering freeway from urban surface street

Trigger Rules: e.g., start selections 10 seconds before the vehicle approaches the intersection and stop 20 seconds after the merge completes

Data: e.g., time-series-segmentl

[00138] Below is an example process of creating a new scenario object:

Step 1 Specify the Scenario Type

Step 2 Specify each Static Object, each Dynamic Object, and the Scene Description using the metadata in the MetadataCatalog

Step 3 Issue a query using the Metadata Query Language using the selected metadata to create the ResultSet

Step 4 If the ResultSet is empty Then determine if NewTimeSeriesData can be physically collected or simulated, Else go to Step 7. Step 5 If NewTimeSeriesData needs to be simulated Then determine if the entire data set needs to be simulated Or whether only part of the data set needs to be simulated and then merged with physically collected data. Go to Step 6.

Step 6 Process the NewTimeSeriesData (collected, simulated, or combination thereof) using the Data lngestion Pipeline

Step 7 For each TimeSeries in the NewTimeSeriesDataSet a. Select the TimeSeries segment that satisfies the specified metadata in Step 2 using the Triger Rules to create the TimeSeriesSegment. b. Create a new ScenarioObject c. Associate a name with the selected segment d. Fill in the metadata from Step 2 ed. Insert the TimeSeriesSegment in the ScenarioObject

Computer systems

[00139] The vehicle data management system, data orchestrator, or processes described herein can be implemented by one or more processors. In some embodiments, the one or more processors may be a programmable processor (e.g., a central processing unit (CPU), a graphic processing unit (GPU), a general-purpose processing unit or a microcontroller), in the form of fine-grained spatial architectures such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and/or one or more Advanced RISC Machine (ARM) processors. In some embodiments, the processor may be a processing unit of a computer system. FIG. 13 shows a computer system 1301 that is programmed or otherwise configured to implement the data management system. The computer system 1301 can regulate various aspects of the present disclosure.

[00140] The computer system 1301 includes a central processing unit (CPU, also “processor” and “computer processor” herein) 1305, which can be a single core or multi core processor, or a plurality of processors for parallel processing. The computer system 1301 also includes memory or memory location 1310 (e.g., random-access memory, read-only memory, flash memory), electronic storage unit 1315 (e.g., hard disk), communication interface 1320 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 1325, such as cache, other memory, data storage and/or electronic display adapters. The memory 1310, storage unit 1315, interface 1320 and peripheral devices 1125 are in communication with the CPU 1305 through a communication bus (solid lines), such as a motherboard. The storage unit 1315 can be a data storage unit (or data repository) for storing data. The computer system 1301 can be operatively coupled to a computer network (“network”) 1030 with the aid of the communication interface 1320. The network 1030 can be the Internet, an internet and/or extranet, or an intranet and/or extranet that is in communication with the Internet. The network 1030 in some cases is a telecommunication and/or data network. The network 1030 can include one or more computer servers, which can enable distributed computing, such as cloud computing. The network 1030, in some cases with the aid of the computer system 1301, can implement a peer-to- peer network, which may enable devices coupled to the computer system 1301 to behave as a client or a server.

[00141] The CPU 1305 can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 1310. The instructions can be directed to the CPU 1305, which can subsequently program or otherwise configure the CPU 1305 to implement methods of the present disclosure. Examples of operations performed by the CPU 1305 can include fetch, decode, execute, and writeback.

[00142] The CPU 1305 can be part of a circuit, such as an integrated circuit. One or more other components of the system 1301 can be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[00143] The storage unit 1315 can store files, such as drivers, libraries and saved programs. The storage unit 1315 can store user data, e.g., user preferences and user programs. The computer system 1301 in some cases can include one or more additional data storage units that are external to the computer system 1301, such as located on a remote server that is in communication with the computer system 1301 through an intranet or the Internet.

[00144] The computer system 1301 can communicate with one or more remote computer systems through the network 1030. For instance, the computer system 1301 can communicate with a remote computer system of a user (e.g., a user device). Examples of remote computer systems include personal computers (e.g., portable PC), slate or tablet PC’s (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personal digital assistants. The user can access the computer system 1301 via the network 1030.

[00145] Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 1301, such as, for example, on the memory 1310 or electronic storage unit 1315. The machine executable or machine readable code can be provided in the form of software. During use, the code can be executed by the processor 1305. In some cases, the code can be retrieved from the storage unit 1315 and stored on the memory 1310 for ready access by the processor 1305. In some situations, the electronic storage unit 1315 can be precluded, and machine-executable instructions are stored on memory 1310.

[00146] The code can be pre-compiled and configured for use with a machine having a processor adapted to execute the code, or can be compiled during runtime. The code can be supplied in a programming language that can be selected to enable the code to execute in a precompiled or as-compiled fashion.

[00147] Aspects of the systems and methods provided herein, such as the computer system 1301, can be embodied in programming. Various aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of machine (or processor) executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Machine-executable code can be stored on an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. “Storage” type media can include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

[00148] Hence, a machine readable medium, such as computer-executable code, may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as may be used to implement the databases, etc. shown in the drawings. Volatile storage media include dynamic memory, such as main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[00149] The computer system 1301 can include or be in communication with an electronic display 1335 that comprises a user interface (LT) 1340 for providing, for example, a graphical user interface as described elsewhere herein. Examples of UI’s include, without limitation, a graphical user interface (GUI) and web-based user interface.

[00150] Methods and systems of the present disclosure can be implemented by way of one or more algorithms. An algorithm can be implemented by way of software upon execution by the central processing unit 1305. The algorithm can, for example, trained models such as predictive model.

[00151] In some embodiments, at least a portion of the vehicle data may be transmitted to a remote entity (e.g., cloud applications) according to a pre-determined data transmission scheme that is not generated using Al algorithms. For instance, in some situations, when the data transmission is infrequent or the amount of data to be transmitted is relatively small, a data transmission scheme may be generated based on a request from a cloud application without using the machine learning models. In such situations, the data transmission may be managed by an intermediary entity (e.g., original equipment manufacturer (OEM)) that processes/passes requests and responses between the remote entity and the data orchestrator residing on the target vehicle. The intermediary entity may act as a proxy to pass the unmodified or processed data transmission requests/responses between the remote entity and the data orchestrator. In some cases, the intermediary entity may determine one or more target vehicles to transmit vehicle data based on the request. In some cases, the intermediary entity may further aggregate or assemble at least a portion of the vehicle data and send it to the requesting application. In some cases, the intermediary entity may generate metadata describing the vehicle data and/or information about the transmission (e.g., data source, data processing method, etc) and transmit the metadata to the requesting application. FIG. 16 schematically shows data transmission managed with aid of an OEM 1630, in accordance with some embodiments of the invention.

[00152] An OEM 1630 may manage basic vehicle data and functionalities. The OEM 1630 may communicate directly with a remote entity such as one or more cloud applications, enterprise cloud or other third party entities 1640-1, 1640-2 as described elsewhere herein. The OEM may provide runtime software components or basic software services such as perception (e.g., ASIC, FPGA, GPU accelerators, SIMD memory, sensors/detectors, such as cameras, Lidar, radar, GPS, etc.), localization and planning (e.g., data path processing, DDR memory, localization datasets, inertia measurement, GNSS), decision or behavior (e.g., motion engine, ECC memory, behavior modules, arbitration, predictors), control (e.g., lockstep processor, DDR memory, safety monitors, fail safe fallback, by-wire controllers), connectivity, and I/O (e.g., RF processors, network switches, deterministic bus, data recording). The OEM may collect or manage telematics data generated by the aforementioned software services or sensors. The telematics data may include, for example, speed related data (e.g., harsh acceleration, speeding, frequent acceleration), stop related data (e.g., harsh braking, frequent stopping, frequent braking), turn related data (e.g. harsh turning, acceleration before turn, overbraking before exit, swerving), data related to routes normally driven (e.g., highways versus local roads, areas with known traffic congestion, areas with high/low accident rates) or others (e.g., fatigued turning, usually driving on the fast lane, usage of turn indicators). An OEM 1630 may be in communication with one or more vehicles 1610-1, 1610-2 and/or one or more data orchestrators 1620-1, 1620-2.

[00153] In some embodiments, an intermediary entity such as the OEM may manage a data and knowledge management system which is configured to determine which predictive model(s) from the predictive model management module to send to a selected vehicle, or fleet of vehicles, and which component(s) may receive these models. In some cases, the model(s) may be transmitted OTA to the related vehicle(s) through the Cloud Subscription Module. In some cases, a remote application may request data from one or more vehicles by sending a request to the OEM. For instance, an insurance application may request certain type of data from an OEM system associated with a target vehicle (e.g., data collected by OEM-embedded devices) at a predetermined frequency (e.g., a week, two weeks, a month, two months, etc) for purpose of detecting fraud, creating new insurance products, providing discounts to drivers for safety features, assessing risk, accident scene management, first notice of loss, enhancing claims process and the like. The OEM may then pass the request to the data orchestrator associated with the target vehicle to coordinate a data transmission. The requested type of data may be transmitted from the data orchestrator to the requesting application 1640-1, 1640-2 directly.

[00154] FIG. 17 shows a data transmission process between a data orchestrator 1720 and one or more cloud applications. In some cases, a data transmission between one or more data orchestrators and one or more cloud applications may be coordinated and managed with aid of an intermediary entity such as vehicle OEM 1730. The data orchestrator may reside locally with a vehicle as described elsewhere herein.

[00155] In some embodiments, the one or more cloud applications may send request 1710 to the vehicle OEM 1730 requesting certain type of vehicle data. For example, the request 1710 may contain information about the type of data needed by the application (e.g., App 1), the frequency with which the data are needed, a period of time for such type of data to be transmitted, or other information such as the target vehicle identification number. For instance, a requesting application App 1 (e.g., insurance application) may send to the vehicle OEM 1730 associated with a target vehicle a request indicating the type of data and the frequency of such data are needed from the target vehicle.

[00156] The vehicle OEM 1730 may pass the request 1711 (e.g., send a request message to relay the request) to the data orchestrator of the target vehicle. The request 1711 passed to the data orchestrator may be unmodified request that is the same as the original request 1710. Alternatively, or in addition to, the vehicle OEM 1730 may process the request 1710 received from the cloud application App 1 and determine which vehicles/data orchestrators are the target vehicles/data orchestrators to receive the request 1711. For example, the original request 1710 may request telematics data from a type of vehicles for enhancing claims process without specifying a target vehicle (e.g., not knowing the vehicle ID), then the vehicle OEM 1730 may identify the target vehicles meeting the requirement of the vehicle type and send the requests 1711 to the identified target vehicles/data orchestrators. In some cases, the request may specify a group of vehicles. For instance, the request may specify a particular model (e g., Audi A8), a model year (e.g., 2017), a model with specific driving automation features (e.g., A8 with lane change monitor), and the li e. The OEM system may pass the request (e.g., send a request message to relay the request) to the data orchestrator of the respective target vehicle. As mentioned above, the vehicle OEM may act as a proxy to pass the requests and responses between the data orchestrator and the requesting application. This may advantageously add a layer of security since the vehicle ID or other vehicle information may not be exposed to the third party (e.g., cloud applications). [00157] Upon receiving the request 1711, the data orchestrator may push the request to a queue and send back a message to the vehicle OEM 1730 to acknowledge receipt of the request. The vehicle OEM 1730 may then send a message (i.e., response) to the requesting application indicating the request has been logged.

[00158] The one or more data orchestrators associated with the target vehicles may transmit the requested vehicle data to the requesting application based on the information contained in the request 1711. For example, the one or more data orchestrators may send the requested data (e.g., data packets) directly to the requesting application.

[00159] In some cases, in addition to passing and relaying the request/response messages, the vehicle OEM may send instructions to coordinate data transmission. For example, the vehicle OEM may send a message to the data orchestrator instructing the data orchestrator to delete the transmission request from the queue when a transmission period is completed (e.g., upon receiving a completion message from the data orchestrator). The data orchestrator may then delete the entry from the queue and send a message to the vehicle OEM indicating the entry is deleted. The OEM system may send a message to the requesting application indicating the request is completed.

[00160] In another aspect of the present disclosure, the data orchestrator may be implemented or integrated with any existing local data storage devices, vehicle recorder systems, event data recorder system and the like, onboard the vehicle. For example, the data orchestrator can be easily deployed to an existing vehicle data storage/recorder system and responsible for composing orchestrated workfl ows/dataflows that are defined as elsewhere herein. Each dataflow can be determined using the deep learning-based data transmission mechanism as described above and a transmission request may be registered with the Subscription Module (e.g., catalog registry) to support remote access. In some cases, the data orchestrator or data orchestration framework may provide ad-hoc transmission schemes and data exchange layer enabled by the automatic update capabilities and the cloud-based model management component.

[00161] FIG. 21 and FIG. 22 show examples of a data orchestrator communicating with different in-vehicle systems such as vehicle data recorders, microcontrollers or electronic control units (ECUs). The data orchestrator may retrieve data from the various data storages devices such as in-vehicle database, ECU databases and Vehicle Data Recorder using metadata-based query language. In some cases, the data orchestrator may be capable of obtaining vehicle data using the Decision Engine (e.g., 1913 in FIG.19) to extract data from the In-Vehicle Database (e.g., 1920 in FIG. 19) in a process as described in FIG. 19. In some cases, the data orchestrator may obtain the vehicle data via the Decision Engine by querying one or more ECU databases based on the application entry in the Application Table (e.g., 300 in FIG. 3). For example, each vehicle ECU may control a number of sensors and the sensor data are stored in the corresponding ECU database. The data orchestrator may query the sensor data using metadata-based query language that is established between the data orchestrator and the ECU Database. In some cases, the data orchestrator may obtain the vehicle data via the Decision Engine by issuing a query to the Vehicle Data Recorder based on the application entry in the Application Table. The data orchestrator may query the sensor data using metadata-based query language that is established between the data orchestrator and the VDR.

[00162] FIG. 21 schematically illustrates an in-vehicle architecture including interaction between a vehicle data recorder 2100 and a data orchestrator 2110. The data architecture may further include a cloud-based Subscription Module or Application request consolidator 2130, fog server (e.g., Edge Station) 2120, and the Cloud-Based Data Management System 2140. The data orchestrator 2110 and the vehicle data recorder 2100 may reside locally with a vehicle as described elsewhere herein.

[00163] In some cases, vehicles can be equipped with an event data recorder (EDR) also known as vehicle data recorder (VDR). Vehicle data recorder device 2100 can continuously record information about the vehicle’s speed, braking, acceleration, angular momentum, and various other vehicle data. This information may not be retained in permanent storage unless when the vehicle is in an accident, in which case the EDR may permanently save the data for a time period (e.g. five seconds) preceding the accident. In the United States, the EDR data can be downloaded under the Fourth Amendment by law. The event data recorders are generally located beneath the carpeting of the vehicles, making it difficult to access the devices and data without physically intruding in the vehicle owner’s car to plug into the download port located in the car or to remove the EDR module for later inspection. The EDR may be configured to record a predetermined amount of data elements (e.g., fifteen identified data elements) that provide a snapshot of a vehicle’s essential mechanical functioning such as speed and direction.

[00164] The data orchestrator 2110 may request the EDR data in a secured manner and send the data to an entity (e.g., law enforcement entity, devices, systems) in response to a legitimate request. For example, the EDR data may be orchestrated and transmitted with aid of the data orchestrator 2110 under the federal regulation of EDRs: Driver Privacy Act of 2015 and regulations promulgated by the National Highway traffic Safety Administration (NHSTA). In some cases, the data orchestrator 2110 described herein may be capable of constantly checking the EDR data upon a request and may orchestrate data transmission as soon as the data becomes available. [00165] The data orchestrator 2110 may comprise a data exchange layer and/or abstraction layer to interface with the vehicle recorder system. The abstraction layer of the orchestration engine may abstract the complexity of the underlying software, data structures, microservices thereby providing a uniform, simplified and secured means to access the vehicle data. The data exchange layer may be in communication with, for example, a query engine 2103 of the vehicle data recorder 2100, for requesting data stored on the vehicle data recorder.

[00166] As shown in the example, a vehicle data recorder 2100 may comprise a FIFO (First- In, First-Out) memory 2101 that stores data, and a Query Engine 2103 that responds to queries (from external devices such as the data orchestrator) by accessing the data stored in the FIFO memory. In some cases, the FIFO memory 2101 may be written with vehicle information sampled periodically using a ring buffer in a First-In, First-Out manner. The sampling frequency can be determined in advance on a vehicle information type basis. The vehicle data recorder 2100 may write the vehicle information that is sampled periodically in the ring buffer 2101, which causes the ring buffer to hold the latest vehicle information of the length (data amount) corresponding to a predetermined recording period. As an example, the data that is written in the ring buffer 2101 can be determined by the Domain Controllers that are part of the vehicle’s architecture. For instance, data is automatically recorded in the ring buffer 2101 upon receiving the data from the Domain Controller. A Domain Controller such as domain control unit (DCU) or multi-domain controller (MDC) is a centralized architecture that is typically integrated with powerful hardware computing capacity and availability of sundry software interfaces which enable integration of core functional modules. Domain controller may have lower requirements on function perception and execution hardware and provide standardized interfaces for data interaction. In some cases, a portion of the data may be retained until it is copied or retrieved by the data orchestrator and another portion may be overwritten when the buffer is full. Such data retaining policy may be determined by the VDR. Alternatively, or additionally, a secondary storage device may be utilized to store selected data when the buffer is full.

[00167] In some cases, each record stored in the FIFO memory may include metadata. The metadata can be the same as the metadata described elsewhere herein. For instance, the metadata may be created by the sensor that generated the sensor data, or created from the data ingesting process or data processing process for writing the sensor data to the memory of the vehicle data recorder. For example, metadata about the sensor or sources producing the data (e.g., sensor- created metadata) may include information about the sensor, identifier of the sensor, data type, and others. In another example, when different sensor data are aligned, metadata (e.g., alignment- created metadata) may be created to provide alignment information (e.g., structure padding, frequency, time window, etc). The metadata can be transmitted to the cloud metadata database and managed by the cloud-based data and metadata management system 2140 as described elsewhere herein (e.g., FIG. 8).

[00168] In some cases, the metadata may be related to conditions, events, internal and outside environment of the vehicle, such as “hard-breaking event,” “collision event,”, “gunshot event,” and the like. The metadata may be associated with a series of data records. For instance, a series of data records may correspond to an event tagged by the metadata. In some cases, such metadata may be generated by applications such as an advanced driver assistance system (ADAS) installed on the vehicle operating system. The metadata can be generated using any suitable techniques, for example using the recording device for voice recognition, imaging device for face recognition, and any suitable sensors for motion information, gunshot detection information, vehicle sensor information, license plate detection information, text detection information, and/or any other suitable technique. Such metadata may be used to tag a series of appropriate data records, and associate with each record a time-stamp. In some cases, Query Engine 2103 may utilize such metadata to retrieve the associated data records that are generated during the desired multi-second sequence.

[00169] In some cases, the Query Engine 2103 may comprise or accommodate one or more daemons. The one or more daemons may be processes that run in the background and perform operations at predefined times or in response to certain events. For example, the daemons may flag and capture important data even if there is no query that needs to be executed. In some cases, the one or more daemons or processes may be part of the Query Engine and can be activated when specific events occur (e.g., overheating of the electric vehicle’s battery system), and capture data from the sensors that are specified in the daemon using preprogrammed instructions. The query engine may be configured to automatically transfer one or more data records from the vehicle data recorder to a database coupled to the data orchestrator upon detection of an event (without receiving a transmission request). For instance, a copy of the event’s data may be automatically transferred by the Query Engine to the data orchestrator 2110 and can be stored in the database coupled to the data orchestrator (e.g., data orchestrator database).

[00170] In some cases, a query that has been received by the Query Engine 2103 and does not immediately return data from the FIFO memory 2101, may become persistent and be constantly checked against the contents of the FIFO memory until the requested data records are returned.

[00171] The Application Request Consolidator or Subscription Module 2130 can be the same as the cloud subscription module as described elsewhere herein. The Subscription Module 2130 may act as an intermediate broker between a fleet of vehicles and one or more Cloud-Based Data Management Systems. The Subscription Module may manage and organize data requests that are generated by the Registering Applications. For example, the Subscription Module may be configured to manage the data requests or registering application request. The Subscription Module may be capable of aggregating multiple registering application requests thereby beneficially reducing communication bandwidth consumption. For example, multiple registering application requests about requesting data from the same vehicle application (e.g., the Pothole Detector application) may be aggregated. In other examples, multiple registering application requests about requesting data from the different vehicle application running on a specific group of vehicles may be aggregated and packaged into a single message. In alternative cases, in the absence of a Subscription Module, the data orchestrator of a vehicle in the fleet may be able to establish direct connection with one or more Registering Applications from a set of applications.

[00172] One or more applications running on cloud or a remote entity (e.g., public clouds such as Amazon Web Services (AWS), and Azure, or private cloud) may register in an Application Table (e.g., Application Table as illustrated in FIG. 3) of a particular vehicle’s data orchestrator (or the data orchestrators of a fleet of vehicles) through a publish/subscribe scheme. In some cases, an application that is running on the fog/edge servers or a remote entity may register in the Applications Table through a publish/subscribe scheme. In some embodiments, a Registering Application may specify the Vehicle IDs from which it needs to receive data and/or the particular Vehicle Application(s) running on the corresponding vehicles it needs to receive data from.

[00173] The Subscription Module may receive data request from the Registering Application. Below is an example of a Data Request from a Registering Application which may include data fields such as:

[00174] A Data Request ID;

[00175] A Registering Application lD;

[00176] A Data Center lD specifying where the application with the Registering Application lD is running and where the data will be sent once it is received from each vehicle’s data orchestrator;

[00177] A Vehicle lD Set, specifying a set of vehicles from which the Registering Application is to receive data;

[00178] A Vehicle Application lD, specifying the vehicle application from which the data is requested (resource), e.g., ADAS application;

[00179] A Requested Data Description, including a description of the data that is to be collected and sent from the vehicle with the particular Vehicle lD and the specific Vehicle Application lD, e.g., three seconds before a hard-braking event, and three seconds after the hard-braking event. A data description may be expressed as a query that is based on the metadata associated with each record of the captured data and may be sent by the data orchestrator to the Query Engine as described above;

[00180] A Transmission Delay, specifying a time delay of transmitting the requested data or whether the data can be stored on the vehicle and offloaded either to an edge server or sent to the requesting data center when the vehicle is in a specific operating state (e.g., re-charging).

[00181] In an exemplary process of data transmission, the Subscription Module 2130 may perform one or more of the following operations:

[00182] 1. Generate a message (e.g., SM Message) for each received Data Request ID and for each member of the Vehicle lD Set in the Data Request ID. An example of a SM Message may include: SM Message lD, Data Request ID, Vehicle lD, Vehicle Application lD, Requested Data Description, Transmission Delay, and Received Data.

[00183] 2. Send the generated SM Message to the data orchestrator of each destination vehicle using the Over The Air (OTA) update function as described above.

[00184] 3. Track the relation between the Data Request ID and the corresponding

SM Message lD.

[00185] In some cases, the Subscription Module may process, organize the received data requests, and generate a message to the data orchestrator for a transmission request consolidated based at least in part on a requested data description. In an exemplary process for requesting vehicle data, the Subscription Module may perform one or more of the following operations:

[00186] 1. Collect the set of Data Request received during a user-defined time period, e.g., every hour, into the Data Requests Set.

[00187] 2. Organize the Data Requests in the Data Request Set that refer to the same

Vehicle Application lD into separate subsets Data Request Subseti, ...,

Data Request Subsetn., such that data requests that refer to the same Vehicle Application are organized in the same subset. One or more subsets may be created.

[00188] 3. In each Data Request Subsetj identify the set of unique

Requested Data Description and organize them into separate subsets

Consolidated Data Request Subseti, . . ., Consolidated_Data_Request_Subset _m such that data requests in the same consolidated data request subset share the same data descriptions. This may allow for issuing a single query to the Vehicle Data Recorder for multiple requests. One or more consolidated data request subsets may be created.

[00189] 4. Create the Vehicle lD Set that includes the IDs of all the vehicles that are required to respond to the data request in the Consolidated Data Request Subsetk.

[00190] 5. For each Data_Request_Subseti- _n and for each member of the Vehicle lD Set in each Consolidated_Data_Request_Subseti.nl of the Data Request Subset, generate a SM_message. The SM_message may include SM_Message_ID, Vehicle_ID, Vehicle Application, Requested Data Description, Transmission Delay, Received Data.

[00191] 6. Send the SM_Message to the data orchestrator of each destination vehicle using an

Over The Air (OTA) update mechanism.

[00192] The fog server/edge station 2120 may include a fog layer database or other components as described above. Data at the fog layer may be generated, managed and directly accessed by the data orchestrator. The fog/edge data may comprise data after it has been processed by a data processing module of the vehicle data recorder. The data processing module may support ingesting of sensor data into a local storage repository (e.g., local time-series database), data cleansing, data enrichment (e.g., decorating data with metadata), data alignment, data annotation, data tagging, data aggregation, and various other data processing. The fog/edge data may also comprise intermediary data to be transmitted to the cloud according to a transmission scheme. For example, the requested vehicle data may be transmitted from the data orchestrator 2110 to the cloud-based data management system 2140 directly without going through the subscription module 2130. In another example, the fog/edge data may be transmitted from the fog/edge stations 2120 directly to the cloud-based data management system 2140. In some cases, the data orchestrator 2110 or the fog/edge server 2120 may notify the Subscription Module when the requested data transmission has been performed.

[00193] The data orchestrator may be configured to or capable of determining which of the vehicle data or which portion of the vehicle data stays in the in-vehicle database, is to be moved/transmitted to the fog layer database (e.g., fog/edge database), and which of the fog/edge data or which portion of the fog/edge data is to be communicated to which data center or third party entity, when and at what frequency this portion of data is transmitted. For example, the data orchestrator may determine the transmission rule or transmission scheme using a machine learning algorithm trained model and/or user defined rules as described elsewhere herein. In some cases, data that is off-loaded or moved to the edge/fog database may be deleted from the in- vehicle database for improved storage efficiency. Alternatively, data in the in-vehicle database may be preserved for a pre-determined period of time after it is off-loaded to the edge/fog database.

[00194] In some cases, a vehicle may not be equipped with a vehicle data recorder. The data orchestrator as described herein may be capable of interfacing a vehicle data recorder, microcontrollers or electronic control units (ECU) onboard a vehicle.

[00195] In the case of direct integration with microcontrollers or ECUs, the data exchange layer of the data orchestrator may translate the hardware component input events into higher level API interaction that software applications can use at its expected level of abstraction and not have to drop to lower level communication protocols to interact with hardware elements. FIG. 22 shows an example of data orchestrator 2110 integrated with one or more ECU databases or ECUs 2200.

[00196] One or more ECU databases 2200 may reside with the vehicle. In some cases, an ECU may have its own ECU database. The data orchestrator 2110 may be configured to determine to which ECU database to send a query. An ECU Database may store the data generated by the sensors controlled by the corresponding ECU’s application. In some cases, the vehicle’s architecture may include a Vehicle Database that consolidates the data from the one or more ECU Database. In such cases, the data orchestrator may issue the query directly to the Vehicle Database in a similar process of issuing a query to a Vehicle Data Recorder. It should be noted that although ECU databases are described and illustrated in the figure, the data can be stored in any storage devices that may or may not have a database management system. The data orchestrator may be capable of querying data from such data storage devices directly using any suitable querying language such as structured query language (SQL).

[00197] Methods and data orchestrators for managing vehicle data of a vehicle are provided. The data orchestrator comprises: a data repository for storing: data related to one or more remote entities that request one or more subsets of the vehicle data and a description of the one or more subsets of the vehicle data; a communication module to issue a query to a vehicle data recorder or one or more databases onboard the vehicle based on the description of the one or more subsets of the vehicle data; and a decision engine to execute a data transmission rule for transmitting the vehicle data, and the rule comprises a selected portion of the vehicle data to be transmitted; (ii) when to transmit the selected portion of the vehicle data; and (iii) a remote entity for receiving the selected portion of the vehicle data.

[00198] FIG. 23 schematically shows a diagram of a data orchestrator 2300 including a plurality of functional components. The data orchestrator 2300 can be the same as the data orchestrator described in FIG. 4 or elsewhere herein. In the illustrated example, the data orchestrator 2300 may comprise an Applications Table 2301, a decision engine 2303, a knowledge base 2305, a data orchestrator database 2307, and a communication module 2309. The data orchestrator may be in communication with the Subscription Module and the in-vehicle ECU databases and/or vehicle data recorder as described above.

[00199] The data orchestrator 2300 may be in communication with the Subscription Module. For example, in response to receiving the SM Message, the data orchestrator 2300 may create a new record in the Applications Table 2301. The Application Table 2301 can be the same as the Application Table as described in FIG. 3. Next, the data orchestrator may issue a query to the Vehicle Data Recorder based on the Requested Data Description contained in the received SM_Message.

[00200] An example of the record in the Application Table 2301 may include a plurality of data fields such as:

[00201] 1. A Vehicle Application lD that is generating data to be communicated.

[00202] 2. A Vehicle lD of the vehicle providing the data.

[00203] 3. The type of data that is to be transmitted.

[00204] 4. A pointer to the vehicle data that is to be transmitted from the vehicle that generated the requested vehicle data. This pointer may point either to the locations in the Vehicle Data Recorder (by issuing the query issued to the Vehicle Data Recorder), or to the data orchestrator database 2307. If the pointer is not NIL (i.e., empty) and the Transmission Flag is SET, then it indicates that there is data ready to be transmitted. In the case that the vehicle is not equipped with a Vehicle Data Recorder, each Vehicle Application that is connected to the data orchestrator 2300 may be configured for setting the Transmission Flag when the new data is stored in the vehicle database (e.g., ECU databases or consolidated vehicle database).

[00205] 5. The timing of the transmission. This specifies whether the data is to be transmitted immediately or be transmitted under certain conditions (e.g., initiate transmission when the vehicle is in a specific state such as re-charging).

[00206] 6. The destination of the transmission. The transmission’s destination may be the

Subscription Module that sent the particular SM Message (in which case the SM Message lD, and Data Request ID are included in the location description), one or more edge/fog server(s), or one or more Registering Application. [00207] 7. The type of compression that is to be used on the data to be transmitted, or the data that is to be stored in the data orchestrator database.

[00208] 8. The type of encryption that will be used on the data to be transmitted, or the data that is to be stored in the data orchestrator database.

[00209] 9. The regulation rules that to be applied regarding privacy before the data is transmitted or stored in the data orchestrator database.

[00210] In the absence of a Subscription Module, the data orchestrator may receive a message directly from a Registering Application and create a new record in the Applications Table. Below illustrates elements of an example of a record created in the Applications Table in absence of a Subscription Module:

[00211] 1. A Vehicle Application lD that is generating data to be communicated.

[00212] 2. A Vehicle lD of the vehicle providing the data.

[00213] 3. A flag indicating whether new data is available for transmission to one or more

Registering Application. The flag’s values can include: DATA CENTER, or EDGE SERVER. DATA CENTER indicates that the Requesting Application is running in specific data centers. EDGE SERVER indicates that the data are to be stored in the data orchestrator database and offloaded to a fog/edge server.

[00214] 4. The type of data that is to be transmitted.

[00215] 5. A pointer to the actual data that will be transmitted from the vehicle that generated the requested data. This pointer points either to the appropriate data in the Vehicle Database, or in the Vehicle Data Recorder, depending on the data’s location.

[00216] 6. The timing of the transmission in case the data needs to be transmitted to a data center.

[00217] 7. The type of compression that is to be used on the data to be transmitted, or the data that is to be stored in the data orchestrator database.

[00218] 8. The type of encryption that will be used on the data to be transmitted, or the data that is to be stored in the data orchestrator database.

[00219] 9. The regulation rules that to be applied (if appropriate) regarding privacy before the data is transmitted or stored in the data orchestrator database.

[00220] 10. The locations of each Requesting Application that is requesting data from the

Vehicle Application and where the data is to be sent by the Communications Module. [00221] In some cases, the data orchestrator 2300 may store a set of transmission rules for transmitting data without the data request message (e.g., SM Message request). For example, the Knowledge Base 2305 may store a set of data collection and archiving policies that can be executed automatically. The data collection and archiving policies may be expressed in the form of if-then rules (e.g., hand-crafted rules) or one or more predictive models as described above. In some cases, the processes of executing the data collection and archiving policies may operate as a daemon where the daemon may constantly check whether the invocation condition is satisfied. For example, the Knowledge Base may include an Airbag Deployment Policy, operating as a daemon, periodically issues a query to the Vehicle Data Recorder to check if an

Airbag Deployment Event has been recorded. When the Vehicle Data Recorder responds that an Airbag Deployment Event has been recorded, the data orchestrator may automatically issue a follow-on query to the Vehicle Data Recorder to request, for example, data collected ten seconds before the Airbag Deployment Event. The data records may be retrieved based on the metadata tagged to the data records as an Airbag Deployment Event. This may beneficially allow for data orchestrator to receive useful/requested data from the vehicle data recorder despite that the vehicle data recorder is designed to only store the data records in a recent time window (i.e., FIFO buffer).

[00222] The communication module 2309 can be the same as the communication module as described in FIG. 4. For example, the communication module 2309 may send processed data or a selected portion of the vehicle data to a destination in compliance with the transmission rules. When a transmission is initiated, the communication module 2309 may send the data utilizing an available communication channel. In some cases, the data may be erased from the data orchestrator database 2307 after a transmission is completed.

[00223] The decision engine 2303 can be the same as the decision engine as described in FIG. 4. For example, the decision engine 2303 may be configured to execute rules in the knowledge base 2305. The decision engine 2303 may constantly look up for rules in the knowledge base 2305 eligible or ready for execution, then execute the action associated with the eligible rules and invoke the data communication module 2309 to transmit the results (e.g., aggregated data, Message Package) to the destination (e.g., requested data center, Subscription Module, application, remote entity, third party entity, etc.). The decision engine may also store data returned by the Vehicle Data Recorder or the ECU databases in the data orchestrator database 2307.

[00224] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the invention be limited by the specific examples provided within the specification. While the invention has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it shall be understood that all aspects of the invention are not limited to the specific depictions, configurations or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is therefore contemplated that the invention shall also cover any such alternatives, modifications, variations or equivalents. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.