FIELD

[0001] Ulis discfosare relates generally to nearb-syffibolic computing, or knowledge-isfiised learning, for entity prediction.

BACKGROUND

[0002] In general, the field of autonomous driving typically Involves processing a multitude of data streams from an array of sensors. These, data streams are then used to detect, recognize, and track object'' m a scene Foi example, m cnnipmei MMOU, a scene ts often represented as a ''Cl ul labeled liounding boxes, which, are provided around the objects that are detected within a frame. However, scenes are often more complex than just a set of recognized objects. While machine learning techniques have been able to perform these abject recognition tasks, they tend to lack the ability to fully utilize the interdependence of entities and semantic relations within a scene. These machine learning techniques, when taken atone, are not configured to provide high-level scene understanding, which is accurate and complete.

SUMMARY

[0003] The following is a srimmaiy of certain embodiments described in detail below. The described aspects: are presented merely to provide the reader with a brief summary of these certain embodiment s and the description of these aspects is not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a Variety of aspects that may not be explicitly set forth below.

[0004] According to at least one aspect, a computer implemented method for knowledge-based entity prediction is disclosed. The method inelades obtaining a knowledge graph based on (i) labels associated with a scene and (ii) an ontology. Hie knowledge graph includes nodes and edges. A set of the nodes represent the labels associated with the scene. Each edge represents a relation between relaxed pairs of nodes. The method includes identifying a path with: multiple edges having multiple relations from a source node to a target node via at least one intermediary node between the source node and the target node. Tire method includes reifying the path by generating a reified relation to represent the multiple relations of the path in which the reified relation is represented as anew edge that directly connects the sour ce node to tire target node. The method includes generating a reified knowledge graph structure based on the knowledge graph. The reified knowledge graph structure includes at least the somce node, rhe target node, and: the new edge. The method includes training a machine learning system to learn a latent space defined by rhe reified knowledge graph structure.

[0005] According to at least one aspect, a data processing system comprises one or more non- tiansKoiy cnmpmei loadable riorage media and one u- mote ptocessors The one m rnoic non- trausitory computer readable storage media store computer readable data including instructions that are executable to perform a method. The one or more processors are in data communication with the one or more non-transitory computer readable storage media. The one or more processors are configured to execute the computer readable data and perform the method. The method includes obtammg a know ledge gmph bared on < i) labeh d^ociatcd with a scene and < u) an ontology. The knowledge graph includes nodes and edges. A set of the nodes represent the labels associated with the scene . Each edge represents a relation between related pairs of nodes. The method includes identifying a path with multiple edges having multiple leiations from a source node to a target node via at least one intaniediaiy node between the source node and the target node. The method includes reifying the path by generating a reified relation to represent tire multiple relations of the path in which the reified relation is represented as a new edge that directly connects the source node to the target node. The method includes generating a reified knowledge graph structure based on the knowledge graph. The reified knowledge graph structure ; includes at least the source node, the target node, and the new edge. The method includes training a machine learning system to learn a latent space defined by the reified knowledge graph structure.

[0006] According to at least one aspect, a computer-implemented method includes obtaining a knowledge graph with data structures that include at least a first triple and a second triple The first triple includes a first scene instance, a first relation, and a first entity instance. 'Tire first relation relates the first scene instance to the first entity instance. Hie second triple includes the first entity instance, a second relation, and a first class. The second relation relates the fest entity instance to the fest class. The method includes identifying a path based on the first triple and the second triple. The path is defined from the first scene to the first class with the first entity instance being between the first scene instance and the first class. The path includes at least the first relation and the second relation. The method includes reifying the path by generating a teified telation to represent the first relation and the second mlanoii such that die icified relation directly relates the first scene to the first class. The method includes coiistraciins a reified knowledge graph structure with, a reified triple. The reified triple includes the first scene, the reified relation, and the first class, The method includes training a machine learning system to learn a latent space of the reified knowledge graph structure.

[0007] These and other features, aspects, and advantages of the present invention are discussed in the following detailed description in accordance with the accompanying drawings throughout which like characters represent similar or like parts.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. I. is a diagram of a non-limiting example of a system relating to knowledge-based entity prediction according to an example embodiment of this disclosure.

[0009] FIG. 2 is a diagram of a noil-limiting example of a scene along with a conesponding knowledge graph according to an example embodiment of this disclosure.

[0010] FIG. 3 is a diagram of an example Of a knowledge-based entity prediction system as a post processing step for computer vision entity prediction techniques according to an example embodiment of this disclosure.

[0011] FIG. 4 is a diagram of an example of a process relating to knowledge-based entity prediction according to an example embodiment of this disclosure.

[0012] FIG. 5 A is a diagram of an example of a sequence scene in relation to a frame scene according to an example embodiment oftills disclosure.

[0013] FIG. 5B is a diagram of an: example of a basic structure of a scene according to an example embodiment of this disc losure .

[0014] FIG; 6 A is a diagram of a nomlimiimg example of a first knowledge graph structure according to an example embodiment of this disclosure. [0015] FIG; 6B is a diagram of a iion-Iimiting example of a second knowledge graph structure cueonhre to m example embodiment ot nis di'Cloune

[0016] 6C is a diagram of a m'li-hmitmg example nt .1 thud knowledge gjaph structure according to an example embodiment of this disclosure.

DETAILED DESCRIPTION

[0017] The embodmients described herein, which have been shown and described by way of example, ami many of their advantages will be imderstood bx the fmegomg description. and it will be apparent that various changes can be made in the form, construction, and arrangement of the components without departing horn the disclosed subject matter or without sacrificing o ne or more of it s a dvantages . Indeed,, the described forms of these embodiments are merely explanatory. These embodiments are susceptible to various modifications and alternative forms, and the following claims are intended to encompass and include such changes and not be limited to the particular forms _: disclosed, but rather to cover all modifications, equivalents, and alternatives falling with the spirit and scope of thi s disclosure.

[0018] FIG. 1 is a diagram of a nomlmiiimg example of a system 100 configured to perform knowledge-ba sed entity prediction (KEP) according to an example embodiment of this disclosure. KEP includes the ta sk of predicting the inclusion of one or more potentially unrecognized or missing entities in a scene, given the current and background knowledge of the scene that arc represented as a knowledge graph (KG). The system 100 includes at least a processing system 110 with at least one processing device. For example, the processing system 1 10 include,- at least an elecnomc piocesson a Lend al piocessnig unit (CPI ^r ) a giapltus processing unit (GPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any suitable processing technology, or any number and combination thereof The processing system 110 is operable to provide the fimctionality as described herein.

[0013] The system 100 includes a memory system 120, which is operatively connected to the processing system 110. In an example embodiment, the memory system 120 includes at least one non-transitory computer readable storage medium, wliich is configured to store and provide access to various data to enable at least the processing system 110 to perform the operations and functionality, as disclosed herein. In an example embodiment, the memory system 120 comprises a single memory device or a plurality of memoiy devices. The memory system 120 c.m include ek-ctiieal. electronic, maanetic, optical, seinieonducmi, eicesiomagnetic, or any suitable storage technology that is operable with the system 100. For instance, m an example embodiment, the. memory system 120 can include random access memory (RAM), read only iiicmoiv (ROM! tlads memoiy, a disk time. a memoiy ea-d, an optical storage device, a magnetic s torage device, a memory module, any suitable type of memory device, or any number and combination thereof With respect to the processing system 110 and/or other components of tire system 100, the memoiy system 120 is local, remote, or a combination thereof (e.g., partly local and partly remote). For example, the memory system 120 can include at least a cloudbased storage system (e.g. cloud-based database system), which, is remote from the processing; system 110 and/or other components of the system 100.

[0020] The memory system 120 includes at least a KEP system 130, a machine learning system 140, training data 150, and other relevant data. 160, which are stored thereon. The KEP system 130 includes computer readable data with instructions, which, when executed by the processing system 110, is configured To provide KEP. The computer readable data can include instructions, code, routines, various related data, any software technology/, or any number and combination thereof. In addition, the machine learning system 140 includes a knowledge graph embedding (KGE) model a KGE, algorithm, any suitable artificial neural network model, or any number and combination thereof Also, the training data 150 includes a sufficient amount of sensor data, label data, KG data, KG structure data, various loss data, various weight data, and various parameter data, as well as any related machine learning data that enables the system 100 to provide the KEP, as described herein. Meanwhile, the other relevant data 160 provides various data (e.g. operating system, etc.), which enables: the system 100 to perform tire fenetions as discussed herein.

(0021] The system 100 is configmed to include at least oae sensor system 170. The sensor system 170 includes use or more sensors. For example, the sensor system 170 includes ail image sensor, a camera, a radar sensor, a li ght detection and ranging (LIDAR) sensor, a thermal Sensor, an ultrasonic sensor, an infrared^ sensor, a motion sensor, an audio sensor, an inertial rneasmemeut unit ( EMU), any suitable sensor, or any number and combination, thereof. The sensor system ,170 is operable to conmiunicate; with one or more other components (e.g., processing system 110 and memory system 120) of the system 100. Mote specifically, for example, the processing system 1 10 is configured to obtain the senses data directly or indirectly from one or more sensors of the sensor system 170. The sensor system 170 is local, remote, or a combination theieuf re g.. partly’ local and partly remotto. Upon leceivnig the sensor data, toe processing system, 110 is configured to process tltis sensor data m connection with tire KEF system 130, the machine learning system 140., the training data 150, or any number and combmation thereof.

[0022] In addition, the system 100 may include at least one other component. For example, as shown in FIG.1, the memory system 120 is also configured to store other relevant data 160, which relates to operation of the system 100 in relation to one or more components (e.g., sensor system 170, I/O devices 180, and other iunciional modules 190). fir addition, the system 100 is configured to include one or more I/O devices 180 (e.g;, display device, keyboard device, speaker device, etc.), which relate to the system 100. Also, the system 100 includes other functional modules 190. such as any appropriate hardware, software, or c&iubmatinn thereof that assist with or contribute to the functioning of the system 100. For example, toe other functional modules 190 mclude commimication technology that enables components of the system 100 to communicate with each other as described herein. In this regard, toe system 100 is operable to at least train, employ, or both train and employ the machine learning system 140 (and/or the KEP system 130) to perfonir KEP,

[00.23] FIG. 2 is a conceptual diagram 200 that illustrates various aspects relating to a nonlimiting example of a KEP task according to an example embodiment. In this regard, FIG. 2 illustrates a non-limiting example of a scene 202 along vyith a corresponding KG 204. In FIG. 2, the scene 202 is taken by an ego vehicle (not shown), which is configured to have a certain level of autonomy, while driving through a residential neighborhood 214 on a Saturday afternoon. In this example, the ego vehicle includes a perception module, which detects and recognizes a ball 210 bouncing on the road, as well as another sedan 212. In this case, based on this scene 202 and the detection of the ball 210, there may be an inquiry as to /“What is the probability that a child is nearby, perhaps chasing after toe ball?” The answer to this inquiry involves a prediction task, which requires knowledge of the scene 202 that may be outside the scope of traditional computer vision techniques. Fur example, this prediction task requires an understanding of the semantic relations between various aspects of the scene 202. e.g. that a ball is a preferred toy of children, that children often live and play in residential neighborhoods, that children rend to play outside on Saturday afienroons, etc. In contrast to traditional computer vision techniques, the KFP system 130 is configured to leveiage a km uv ledge-based approach ro provide an answer to that question and also predict the mnecuginzed/missing entity in that scene fbr this SEP task.

[8024] In addition, FIG. 2 shows that the scene 202 may be represented by a knowledge base, which includes an assertion companent (“ABox”) 206 and a terminological component (“TBox”) 208. The ABox 206 hicludes specific instances, which are denoted as round nodes in FIG. 2. More specifically, in FIG. 2, the Abox 206 includes at least a “scene” node 216, a ‘Tall” node 218, a “car” node 220, and a “Pacific Heights, SF, CA” node 222, which refer to specific instances within the scene 202. The “ball” node 218 refers to the detection of the basketball within the scene 202. The “car” node 220 refers to the detection of the sedan 212 within file scene. The “Pacific Heights, SF, CA” node 222 refers to the detection of the location of the scene 202 1 be Vtox 206 further mchides a ' node 22 1 to represent a possible missmg oi unrecognized entity instance within the scene 202,

[0025] The TBox 208 describes a domain of interest by defining classes and properties as a domain vocabulary. Meanwhile, the Abox 206 inc hides assertions, which use the vocabulary defined by the TBox 208. For example, in FIG. 2, the TBox 208 includes entity classes, which are denoted as rectangular nodes. More specifically, the TBox 208 includes at least a “Bali” node 226, a “Child” node 228, a “Car” node 230, and a “Residential Neighborhood” node 232, which refer to the entity classes of the corresponding specific instances within the Abox 206. In addition, the TBox 208 also includes at least a “Toy” node 234, a “Person” node 236, a ' Vehicle” node 238, and a “Locafton” node 240, which refer to the entity classes of the corresponding specific entity instances. As shown in FIG, 2, each of the nodes in the KG 204 is connected to at least one other related node via a relation (e.g., “includes” relation 242, “type” relation 244, “location” relation 246, “subclass” relation 248, “playswith” relation 250, and “livesin” relation 252 ). which indicates a relationship between these two related nodes. For example, in FIG. 2 the 'Cat” node 230 is an entity class, which is related to the“Vehicle” node 238, which happens to be another entity class, thereby indicating that the entity class of “Car” is a subclass of the entity class of “Vehicle.”

[0026] FIG. 2 firrther illustrates a number of relationships between related nodes of the KG 204. For example, the KG 204 includes the “scene” node 216, which refers to die actual driving. scene 202. and the “ball” node 218. which refers to a detection of the actual ball 210 s.e g.. basketball) in die din mg scene 202 in the real world. Hie "scene” node 216 and the "ball’’ node 218 me connected via the “includes” -relation to indicate that the scene 202 includes the ball 210. The “ball” node 218 is related to the “Ball” node 226 via the “type” relation 244 to indicate that “ball” is a specific instance of a type associated with the cla ss of “Ball ” Resource De scription Fmmewutk Schema (REE’S Web ( hrtology I anguage {OWL), OJ any suitable language may be used as a modeling langua ge in the KG 204. As shown in FIG. 2, the sn ucture of this KG 204 is advantageous m providing piedicrinm at the entity class level n e. TRov level) such that die output with respect to tile niissing/unrecognized entity refers to an entity class (e.g,, “Child”) ins tead of the sp ecific instance of that entity (e.g.. Bob) for a downstream application, which in this case relates to autonomous driving.

(0027] Also m FIG. 2, she KG 204 includes a node 224 ro represent a potential entity instance, which is unrecognized or missing in the given scene 202. The entity instance may be missing op unrecognized for various reasons, such as hardware limitations, occluded entities, poor iieki-of view, degraded visuals, device failure, etc. Given the Cunent and background knowledge of the scene 202, the task of predicting this unrecognized or missing entity (e.g., a: child), as indicated by the “?” node 224, tends to be outside the scope of traditiona l computer vision techniques. In contrast, the KEP system 130 is configured to provide an answer for tins KEP task via a knowledge- infused learning approach, which integrates or infuses knowledge into one or more machine learning models.

(0028] FIG. 3 is a diagram of an autonomous driving (AD) perception pipeline 300, which includes the employment of the trained machine learning model of the KEP system 130 to perform KEP. The AD perception pipeline 300 is a process that contributes to scene uudci-nandmg by nki-tifiuig >emoi detections within a given scene and pnwidmg 'emautb? entity labels for" that scene, hi tiiis example, the AD perception pipeline 300 includes at least a computer-vision (CV) entity recognition system 304, a KG generation system 308, and the KEP system 130 hi this case, as shown m FIG 3 the KEF xX'-tum 130 pciinmis KEP as a postprocessing task with respect to the: recognition task performed by the CV entity ⁷ recognition svsiem 364.

[0029] The CV entity reeognitim system 304 employs visual object detection techniques to genemtc a set al cutitx labels 3bu as output m response to receiving sensei dak. 302 as rmun For example, the CV entity recognition system 304 may receive the sensor data 302 from the sensor system 170. The sensor data 302 may include raw images, video, LIDAR point clouds, other sensor data , or any combination thereof, lire sensor data 302 may be two-dimensional (2D) sensor data (e.g. _x camera images) or tliree -dimensional (3D) sensor data (e.g., 3D point clouds). The CV entity recognition system 304 may generate 2DZ3D bounding-boxes about the detections to enable those detections to be identified. The CV entity recognition system 304 may employ object recognition techniques, semantic segmentation tgchfiiques, or a combination thereof Semantic segmentation takes a more granular approach by assigning a semantic category to each pixel in an image. The CV entity recognition system 304 identifies a set of detections (e.g. one or more detections) in the sensor data 302 and provides a set of entity labels 306 (e.g. , one or more entity labels) for that set of detections. In this example, the CV entity recognition system 304 includes at least one machine learning system to perform this recognition task of generating the set of entity labels 306, for example, by classification techniques. The set of entity labels 306 are then used by the KG system 308.

[0030] As shown in FIG. 3, KG system 308 obtains the set of entity labels 306 as input. The KG system. 308 generates at least one KG 310 with semantics based at, least on the set of entity labels 306 according to an ontology. For example, the KG system 308 may generate a node for each entity ⁷ label from. the set of entity ⁷ labels 306. The KG system 308 may also generate a node for other entity labels (e.g,, additional entity labels 312). In an example embodiment, the set of entity labels 306 and/or the additional entity labels 312 represent entity class nodes in the KG 310 based on the ontology. The KG system 308 may further include the generation of relations between related pairs Of nodes. For instance, referring to FIG. 2 as a non-limiting example, the KG system 308 generates ielations such as the huchides” relation 242, “type” relation 244, “location” relation 246, “subclass” relation 248, “playswitlf ’ relation 250, and “livesm” relation 252. The KG system 308 includes obtaining the set of entity' labels 306 as input, constructing at least one KG 310 with semantics based prr the set of entity labels 306- according fo _: an oniofogy, and providing the KG 310 as output.

[0031 ] The KI P sy stem HO is configured to obtwm m le^ ene die K(? 3 Iff a> input The KFP system 130 is configured to output a set of additional entity labels 312 for a given scene instance. Tins set of additional entity labels 312 represents entities that are highly likely to be in the scene, but may have been missed or unrecognized during the CV entity recognition system 304. These entities may be missing or unrecognized by the CV entity recognition system 304 for various reasons, such asfwdware limitations, occluded entities, poor field-of view, degraded visuals, etc. As shown in FIG. 3, the KG system 308 may then obtain and use these additional emit} labels 312 to complete rhe KG 310. In addition, the KEP system 130 is configured to provide the set of additional entity labels 312 to at least one downstream application or system. In this regard, with respect to the task of scene understanding, the KEP system 130 is advantageous in providing the additional entitv labels 312 for a gi ven scene instance that may have been missed or" not recognized by the C V entity recognition system 304. The KEP system 130 contributes to pro viding; a more complete view of a given scene than the limited view provided by the CV entity recognition system 304.

[0032[ F R*- 4 ^a diagram of an example of a process 400 relating to knowledge-infused KEP according to an example embodiment. In this regard, FIG. 4 illustrates the process 400 as a pipeline architecture. In this example, the process 400 includes four phases: (1) a first phase 402 of KG construction. (2) a second phase 404 of path reification, s.3} a third phase 406 of fawwledge graph embedding i.KGE) learning, and (4) a fouitb phase 40b of entity prediction. In addition, FIG.4 also illustrates an example hr which the KEP ss stein 130 provides the second phase 404, the third phase 406, and the fourth phase 408 upon receiving at least one KG, which is constructed during tire first phase 402. The process 400 may be performed by at least one processor of the processing system 110 (FIG. 1 k any suitable data processing system, or any number and combination thereof. Also, the piccess 400 may include more phases or less phases than, the four phases of FI( » I provided that the process 400 is able to achieve Imowledge- infosed KEP, as discussed herein. [0033] At the first phase 402, in an example, fe process includes performing KG constaction. As shown in FIG.4, file process includes constructing at least one K.G 310 based on one or more danu-eti- 410 and an ontology 412. More specifically. as an example, the process includes ehtanung at least one dataset 410, which includes sensor data and at least one set of labels for that sensor data. The dataset 410 may be within at least one suitable domain depending on the application. For instance, in the domain of autonomous driving, the dataset 410 includes raw data generated by theSensor system 170 (e.g., a camera, LIDAR, RAD.4R, GPS, IMU, any sensor, or any combination thereof) along with conespoudmg annotations or labels. The dataset 410 may include complex driving scenarios (e.g,, steep hills, construction, dense traffic. pedestrians, various times of day , various lighting conditions, etc,). The dataset 410 may include numerous driving sequences of a predetermined duration (e.g. 8 seconds per driving sequence), which include at least camera images and LIDAR sweeps. Each sequence may be sampled into frames with a predetermined frequency (e.g., 1 UFPS ). In addition, the dataset 410 may include high quality annotations (i.e,, at least one set. of labels;- with, for example, bounding box labels (e.g., cuboid labels), semantic segmentation labels, any suitable labels, or any combination thereof. More specifically, for example, the semantic segmentation labels may include granular label details: such as smoke, ear exhaust, vegetation, drivable surface, etc.

[0034] The process also includes generating or obtafriins an ontology 412. For example, in FIG. 4, the process inc ludes generating or obtaining a driving scene ontology (DSC)). The DSO provides a formal structure along with semantics tor representing information about scenes. The DSO is configured to describe any driving scene regardless of its source (e.g., dataset 410). That is, the DSO is configured to be data agnostic. In this example, a scene is defined as an observable volume of space and time. More colloquially, a scene refers to a situation in which objects may appear (e.g. vehicle) and events may occur (e.g. lane change maneuver).

[0035] FIG. 5 A ilhisfrates different types of scenes of the DSO according to an example embodiment. For example, the DSO includes at least a sequence scene (“SequenceScene” 502) and a frame scene (“FrameScene” 504). FIG. 5 A also shows the “SequenceScene” 502 relative to the “FrameScene” 504, as well as their relations to each other. The “SequenceScene” 502 and die “Frames cene” 504 are represented as a type of scene instance and are firns denoted as round nodes, hi tins example, the “SequenceScene” 502 represents a situation in which an ego-vehicle drives over an interval of time and along a path of spatial locations. In this regard, for instance, the “SequenceScene” 502 represents a type of scene, which is captured, for instance, by an egovehicle as video. Also, in this example. “FraineScene” 504 represents a type of scene, which is captured, for instance, by an ego-vehicle at. a specific instant of time and point in space. The “FrameScene'' 5(14 is captured, for instance, as an image. The “FiameScene” 504 is generated by sampling the frames of a video. The “FrameScene” 504 may be a part of the “SeqnenceScene” 502 if the time ihstant and spatial point of that “FrameScene’’ 504 are within the time mterval and spatial path of the “SequenceScene” 502, as shown in FIG. 5A. In this regard, the “FrameScene” 504 and the “SequenceScene” 502 are connected via the “isPartOf’ relation 506 to indicate that the frame scene is a part of the sequence scene. In addition, the “SequenceScene” 502 and the “FrameScene” 504 are connected via the “hasPart”relation 508 to indicate that the sequence scene includes at least one frame scene. In this regard, as shown in FIG. 5A, the SequenceScene 502 may contain any number (“M”) of "Fratne Scenes” 504 in a sequence, where “M" represents an integer number greater than 1 . A FrameScene” 504 may occur before another “FrameScene’’ 504, as indicated via the “occursBefore” relation 510. For example, the second “FrameScene” 504 occurs before the “FrameScene” 504. Also, a “FrameScene” 504 may occur after another “FrameScene” 504, as indicated via die “occursAfter” relation 512. For example, the second “FrameScene” 504 occurs after the first “FrameScene’’ 504.

[0036] As shown in FIG, 5 A and FIG, 5B, the DSO may represent time in seveial ojs. As one example, each “FrameScene” 504 is annotated with a time mstmit, which is encoded as “Datelnne” 514 via the “liasTime” relati.on 530. Each “SequenceScene” 502 is annotated with mo time iusfants, which tepiesem die beginning and end of J ume interval. As anorhet example, scenes may be linked to other scenes based on their relative temporal order, using the “occiirsBefore” relation 510 and/or the “occurs After” relation 512, In addition, as indicated in FIG. 5B, spatial information (“SpatialRegiori” 518) is linked to the “FrameScene” 504 through die “hasLocation” relation 534. The range of “hasLocation” 534 is the “SpatialRegiori” 518, which may be expressed as a “subClass’’ relation 536 via “Geometw” 522 with latitude and longitude coordinates or via “Address” 520 with address data. (e.g., country, province, city, street, etc,). [0037J FIG; 5B iUusteates an example ©f fee basic statctiire of a scene 500, as defined by die DSO, according to an example embodiment. As: discussed above, the scene 500 may Include a sequence scene as a scene mslauce oi a frame scene as a scene iuskance FIG. 5B includes rectangular nodes to denote that these elements occur at file entity class level. More specifically, as sbtrtw m FIG 5B, the “>eene" ela-^s ^00 mcludes an 'I iiriy" class 51o In tins case, rhe “Entity” class 516 is a perceived object or event. For instance, as a non-limiting example, the “Entity” class 516 may include moving vehicles, parked cars, pedestrians, ambulances, pedestrians with wheelchairs, etc. The “Entity” class 516 is- linked to die scene 500 through the “includes” relation 532. The “Entity” class 516 is divided into two classes (or entity types), “Object” class 524 and “Event” class 526, Via a “subClass” relation 536. The “Object” class 524 is a subclass of the “Entity” class 516. The “Event” class 526 is a subclass of the “Entity” class 516. The “Object” class 524 may participate in the “Event” class 526, as represented by “isParticipantOf” (and/or “hasParficipant”) relation 538. In an example embodiment, the “Object.” class: 524 and the “Event” class 526 are derived from the dataset 410 (e.g., the bounding box labels and the segmentation labels). Table 1 lists the primary relations associated with the “Scene” class 500 as defined by the DSO. [0038] Referring back to the process 400 (FIG. 4), at the first phase 402, the system 100 is configured to Integrate information from external sources when constructing the KG 310. For example, the ty-stem 100 may imegrate additional knowledge about a scene into the KG 3 b') As a non-limiting example, the system 100 may integrate lacatimr attributes (e.g., latitude and longitude coordinates for each feme. < Jpen Stieet Map ri ’SM) data oddiess mfoimarmn, < raM tags, etc, i that enrich the spatial semantics of the scene. The integration of additional knowledge into the KG 310 may result in additional entities being connected to the scene instance, as deemed necessary or appropriate to enhance the KG 310 and/or the performance of the KEP system 130.

[0035] Afterwards, the KG 310 is constructed by converting the scene data contained in the dataset 410 (along with the additional infonnarion from external sources if available) to a format (e.g.. RDF ² format), which is conformant with the ontology 412 (e.g. DSO). The relevant scene is queried and extracted from the dataset 410, making this process trivially straightforward.: As an example, the RDF can then be generated using an RDF library (e.g., RDFLib ⁴ Python library version: 4.2.2 or any suitable library). For instance, in FIGS. 3 and 4, the KG 310 is constructed as a driving scene knowledge graph (DSKG).

[0040] The DSKG contains data structures that include triples, where each triple is of the form of <h, r, t>, where h=head, 1-1 elation. ajid t=tail and where h and i represent nodes and r represents an edge. For example, the DSKG includes triples of the fem of <seenei, includes, earj > to indicate drat an entity instance (carj) is mcfcded in a scene instance (scenes) . In a number of the examples disclosed herein, the entity instances are expressed with all lowercase letters (e.g. carp while their corresponding entity classes are expressed in title case (e.g. Car). An entity instance is linked to its class iirDSO through triples of the form of <carj type. Cafe hr fids context, it :may be tempting to formulate KEP as a linked path (LP) problem with the objective to complete triples of the form of <scenei, includes, ?>, where represents the element to be predicted. This formulation, however, would entail predicting a specific entity instance rather than predicting the class of an entity. Similar to C V-based object recognition, the objective of KEP should be to predict the class of an entity in the scene - e.g. predicting Car rather than cap. Jh other words, most LP models are: unable to complete the triple of the form of <h, r, t > when there is no r that directly links h and t hr the training data, even if h and t are linked tfoough a path of n-hops (n > 1) in the KG, such as <h, n, ti < ti, r2, t >. Idris is the issue faced by the KEP system 130 with the DSKG, as a scene instance is connected to an entity sub-class only via a 2-hop path. Due to this requirement, foe KEP system 130 cannot simply rely on LP Inn straightforward manner. Therefore, upon generating the KG 310 <e g , DSKG i, the process 400 proceed* to the second ph.we 404 m os ercome ihis technical problem.

[0041] At the second phase 404, in an example, the process 400 includes performing path reification; With path reification, for a given scene instance te.g., “scene” node 414), the system 100 is configur ed to provide a technical solution for KEP by detennining the entity class (e.g,, “Entity Type” node 418) ofan entity instance (e.g., “Entity” node 416). However, since the entity class (e.g., '‘Entity Type” node 418) is not immediately available through a direct link from the scefie instance (e.g., “scene” node 4-14). the system 100 is configured to formulate this KEP task as a path prediction problem (i.e. predicting the path from a scene instance to the entity class). The path may be of any length (e.g,, m-liop where m represents ail integer greater than I). To solve this path prediction problem, the system 100 introduces or creates a pew relation (e.g., “iricludesType” relation 420) to the base KG 310 (e.g., DSKG). Hie system 100 uses this new relation to reify a mhlti-bop path (e.g., 2 -hop path). More specifically, in this example, the system 100 generates a new relation (e.g., “mckidesType” relation 420). which directly links a source node (e.g,, “scene” node 414) with a target node (e.g,. “Entity Type” node 418) with a single-hop. In this regard, the “hieludesType” relation 420 is a combination of the “includes” relation 422 and the “type” relation 424. This requirement can be more formally defined as follows:

(0042) Let Si be the i ^tl scene instance mode ia DSKG (sy e S ) where S represents the set of all scene instance nodes in DSKG, ej be the j* entity instance node and T be a subclass of Entity in the DSO such that (?e E where E = ^Car, Animal, Pedestrian, ...} c C). in this case, the system 100 is configured to perfonu path reification as follows:

[0043] With path reification, the DSKG is transformed into a DSKGR structure (i.e. DSKG with reified paths). Since the “iacludesType” relation is now present during training, the system 100 is configured to use or re-use link prediction (LP) mefoods to address foe KG 310 incompleteness issue. More specifically, LP is a technique for predicting a missing link, such as predicting the head <?, i, t> or predicting the tail <h, r, ?> of a triple for a single hop (or a single relation). As a result of the? creation of this reified relation (“includesType” relation 420) to provide a single: hop , the KEP can now he mapped to EP m order io complete tuples of die form ? } in DSKGR. Before path reification, the KEP could not effectively be mapped to LP in ordei to predict the entity subclasses or the entity types of the DSKG due to the multiple hops (or niultiple relations) that existed between the source node (’’scene” instance) and the target node (“Entity Type” class).

[0644] Although DSKGK IS desenbed above as a KG \truetuie with reified padis. the sx rtcm 160 is not limited to this particular pattern for the reified KG structure. In this regard, the system 100 is configured to generate a reified KG structure with reified paths based on the KG 310 in a variety of patterns. The panents may differ with respect to how entity instance iuibimaimn is represented along the path fem a scene instance to an entity class. FIGS. 6A, 6B, and 6C provide examples of three different reified KG structures. which may be generated by the system 100.

[00451 FIG. 6A is a diagram of a non-limitmg example of a portion of a first knowledge graph structure 600 according to a.u example embodiment. The first knowledge graph structure 600 is DSKGR, which is considered to represent a “complete graph” structure and die most expressive representation with respect to DSKG when compared to the second and third knowledge graph structures, respectively. DSKGR includes all of the nodes and all of the edges of the DSKG that are associated the reified paths. For example, as show in FIG. 6A, DSKGR. includes entity instances 604 and 606 (“pedestrian #1” node? and “pedestrian #2” node) together with the corresponding relations 610 and 612 (e,g., “includes” relation and “type” relation) of the entity instances 604 and 606. DSKG®. also includes: the entity types 608 ('“Pedestrian” class node) together with coiTespoHding reified relations 614 (e.g. “mcludesType” relation). As shown in FIG. 6 A. the fest knowledge graph structure 600 inchides the information from the base DSKG while benefirting from the inclusion of each reified relation 614 (“inctadesType” rela tion) such that each entity type 608 is directly linked to the scene instance 602 (“scene #1” node) via a single hop. Each reified relation 614 is advantageous hi enabling the KEP system 130 to predict an entity type 608 (or entity class) of an unrecognized or missing entity instance for a particular scene instance.

[0046] FIG. 6B is a diagram of a non-limiting example of a portion of a second knowledge graph structure 618 according to an example embodiment. The second knowledge graph structure 618 is DSKGty, which represents a ¹ Tippartite graph.” The second knowledge graph structure 618 is a more compact representation than the first knowledge graph structure 600. Also, the second knowledge graph stiucmie f>18 is a more compact representation than the third know ledge graph structure 620. More specifically, the second knowledge graph structure 618 contains each scene instance 602 (e.g.. “scene #1” node), each reified relation 614 (e.g.. “includesType” relation), and each eimty ripe 608 (e.g,, “Pedestrian” class node) This pattern results in a bipartite-graph structure with leified relations 614 directly linking scene instances 602 (or source nodes) to entity types 608 (or target nodes) via single hops, respectively. Each reified relation 614 is advantageous in enabling the KEP system 130 to predict an entity type or entity class of an unrecognized or mixsing entity instance for a particular Scene instance.

[0047] As shown in FIG. 6B, the second knowledge graph stoichire 618 discards or does not include the entity instances 604 and 606 (e.g., pedestrian #1 and pedestrian #2) of the DSK.G. The second knowledge graph structure 618 also discards or does not include the relations 610 and 612 (e.e. “includes” relation and “type” relation) from the base DSKG that are linked to the entity instances 604 and 606, That is, unlike the first knowledge graph structure 600, the second knowledge graph structure 618 does not contain each multi-hop path (e.g., “includes” relation and the ¹ Type” relation) from the scene instance 602 to the respective: entity type 608. In this regard, the resulting entity instance cardinality for each scene instance is reduced to zero in the second knowledge graph; stnwtae 618 compared to the DSKG . Meanwhile, the second knowledge graph staKtuie o l8 maintains the same entity class cardinality as the DSKG,

[0048] FIG. 6C is a diagram of a non-lhnitag example of a portion of a third knowledge graph stau rue 620 according to an example embodiment. The third knowledge graph structure 620 is DSKGprw, which represents a “prototype graph.” The third knowledge graph structure 620 includes a single prototype instance 622 (e.g,_ “[prototype] pedestrian” node) to represent all entity instances 604 and 606 of a particular entity type 608. In this regard, DSKGprat replaces all entity instances 604 and 606 (e.g. “pedestrian #1” node and “pedestrian #2 node) with a single prototype instance 622 (e.g., “[prototype] pedestrian” node) for each linked entity type 608 (e.g. “Pedestrian” node). The prototype instance 622 represents all of the entity instances 604 and 606 linked to a particular entity type 608 for the scene instance 602, In addition, DSKGiw includes prototype jelatious (e g . Alidades ' lekmon 6 i0 and "type” lelahoii 612 > to connect the prototype instance 622 in a valid manner to the other related nodes (e,g., scene instance 602 and entity type 60S). In DSKGM, the resulting entity instance cardinality tor a scene instance 602/ is equal to the entity’ class: cardinality.

[0043] Referring back to FIG. 4, at the second phase 404, the system I 00 is configured to generate the reified paths and construct a reified KG stnietmc that includes at least those leified paths. More specifically, the system 100 is configured to transform the KG 310 into a reified KG structure by creating reified relations to transform the multi-hop paths into single-hop paths. The reified KG structure may include the first KG structure (FIG.6A), the second KG structure (FIG. 6B), the third KG structure (FIG. 6C), any suitable KG structure with reified paths, or any number and combination thereof provided that the reified KG structure includes at least the reified relations m relation to the scene instance and the entity class (or entity type). Alter the second phase 404 is complete, the process advances to the third phase 406.

[0050] At the thu d phase 406, in an example, the process includes performing KGE learning. In this third phase, the system 100 is configured to translate the reified KG structure into at least one KGE 42b. which encodes the leiiied KG snneluie in a lov-dimemiomil, Litem feanite vccmt representation 426. More specifically, the system 100 uses at least one machine learning algorithm (e.g., KGE algorithm) to leant a representation of the reified KG structure with reified paths, which was constructed at the second phase 404. In this regard, the KGE learning is performed with an EP objective to generate a latent space, which may be useful for various downstream applications and various oilier tasks such, as querying, entity typing, semantic clustering, etc.

[0(1511 In an example embodiment, the system 100 is configured to learn one or more KGEs 428 using one or more KGE algorithms and re-using the teamed latent space for KEP. In this regard, the process may include selecting one or more KGE algorithms. A non-limiting example of a KGE algorithm includes TrarisE, HolE, CouvKB, or any number and combination thereof. As a first example, TransE is a representative KGE model, which leanis relations between nodes as a gcomeuic tianslatioii in the embedding space This, however, hunts its ability to handle symmetrie/transifive /relations, I:N relations and N: 1 relations. As a second example, HolE uses the circular correlation among head and tail of a triple with its relation embedding to team an efficient cqinpressiou of a foil expressive bi-liitear model This, allows both nodes and rela tions to be represented in As a third example. ConvKB learns a high-level feature, map of the input triple by passing a conca tenated node/relation embeddings through a, convolution layer with set of iiheti. fo filters r |H] * The Get score is ihen computed by using a dense Ia>er vrdi only one neuron and weights W. After fire system 100 learns the latent space or embedding space from the reified KG structure using at least one KGE algorithm, the system 100 is configured to use the one or more KGEs 428 for KEP, as discussed below.

[0052] At the fourth phase 408, in an example, the process mcltides performing KEP. More specifically, the system 100 is configured to perform KEP with at least one learned KGE 428, as indicated by Algorithm i. In addition, Table 2 provides a list of notations, which are used in Algorithm I, as shown below.

[0053 J Algorithm 1 is performed by the system 100. particularly the K.EP system 130,-via. at least one processor uf the processing system 110 or by any suitable processing device. As an overview, the system 100 is configured to receive the KGE 428 as input and provide a set a predicted entity classes for each scene instance (sp) as output. Moro specifically, the system 100 is configured to perform a method for each scene instance ($■ ) within the set of all scene instances. For each scene instance, the system 100 is configured to obtain a set of test triples such that each test triple includes a particular scene instance (50 and an entity class that relates to that particular scene instance (s _E ) within the set of all entity classes . For each triple in the set of test triples , the system 100 generates a set of negative triples (x _ne gj, which serve to determine how likely a candidate (f) is linked to the given scene instance ($f) via the given relation (r =fincIudesType” relation). In this case, each negative triple includes the scene instance (S;), the “includesType” relation, and a candidate (t '). Each candidate (t ^f ) represents an entity class within the set of all ensity classes (E) that the given scene instance (.syb may possibly include. Alter the sei of negative triples is generated for the given scene instance, the system 100 is configured to retrieve embeddings via a lookup function from the learned KGE 428 based on each negative triple, as indicated in line 7 of Algorithm 1. The system 100 is configured to generate a score (e.g., a likelihood score 430) for each negative triple of the scene instance (s0 based on the retrieved KG embeddings, as indicated in line 8 of Algorithm I . The system 100 is configured to sort the negative triples based on the scores via tire argsort function and obtain a set of the top-k labels, where ‘k’ represents a predetermined threshold (e.g., a preselected number of labels). The top-k labels represent the k-highest ranked labels (e.g., candidates or entity classes) for the given scene instance. The system 100 is configured to aggregate the set of top-k labels for the set of test triples (3J) to obtain a set of predicted entity classes for the given scene instance The system 100 is configured to provide the set of predicted entity classes as output. The system 100 is thus configured to obtain a set of predicted entity classes, which are highly likely linked to the scene instance (Sf). Tire system 100 determines that the set of predicted entity classes (E*) include one m more nu<>mg m umecomnzed entity class*- of the rot-uc m^iance (s, ) [0054] As mdicated in Algorithm 1, the objective of KEP is to predict a specific link captured by triples, where each tuple is denoted as (fo r. t) vidi ‘h’ representing a head (ci a mi le), ‘r’ representing a relation (or edge and T representing a tail (or a node). To enable this more specific link /prediction based on the reified KG stTuctee, the system 100 is configured to leani the KGE representation of die nodes u e heads and tails) and the edges ii e . relations) wine die LP objec-i\e Then for each s^ene msiauce st die KGE 12 n- queued Hsing * mcludesl ype ’ telanon to find tile missing k-entity class labels Zfo £ E (as indicated in lines 5-18 of Algorithm 1). Algorithm 1 succinctly describes this KEP process via at least one KGE 428, which is trained using at least one KGE algorithm. The computational complexity of Algorithm 1 is $(AT X M) where AT = |Sf andM = [E|.

[0055] FirrthenuOre, there are a number of differences between KEP, as presented above via Algorithm 1, and the traditional LP setup. For example, in contrast to Algorithm I, the KGE algorithms for LP team to maximize the estimated plausibility $(fo r. t) for any valid triple while miiiimiziiig it for any invalid, or negative, triple. Such KGE models can then be used to infer any missing link by obtaining the element (head or tail? with the highest plausibility to complete the triple (for, 0- However, as expressed m Algorithm 1, the processing system 11.0 is configured to perform KEP in a different manner than die traditional LP setup.

[0056] A.> ■hsenxi-ed lieiein, she embodiments inclnde a number of abram/menns leatmex. well as benefits. For example, foe embodiments ar e configured to perform KEP, which improves scene understanding by predicting potentially unrecognized entities and by leveraging heterogeneous, high-level semantic knowledge of driving scenes. The embodiments provide an innovative nemo-symbolic solution for KEP based on knowledge- infused learning, which (i) introduces a dataset agnostic ontology to describe forcing scenes, {n) uses an expressive, holistic representation of scenes with KGs, and (iii) proposes an effective, non-standard mapping of the KEP problem, to foe problem of LP using KGE. The embodiments further demonstrate that kiiowledge-infosed learning is a potent tool, which may be effectively utilized to enhance scene understanding for at least partially autonomous driving systems or other application systems.

[0057] Overall, the embodiments introduce the KEP task and propose an innovative knowledge- infused learning approach. The embodiments also provide a dataset agnostic ontology to describe driving scenes. The embodiments map the KEP to the problem of KG link prediction by a technical solution that: overcomes various limitations: through a process that includes at least path reification.

[0058] That is. the above description is intended to be illustrative, and not restrictive, and provided in the context of a particular application and its requirements. Those skilled in the art can appreciate from the foregoing description that rhe present invention may be implemented in a variety of forms, and that the various embodimentsmay be implemented alone or in combination. Therefore, while the embodiments of the present invention have been described in connection with particular examples thereof the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments, and the true scope of the embodiments and/or methods of the present invention are pot limited to the embodiments shown and described, since various modifications will become apparent to the skilled practitioner upon a "study of &e drawings, specification, and following claims. For example, components and ftmciionality may be separated or combined differently than in the manner of the various described embodiments, and may be described using different terminology. These and other variations, modifications, additions, and improvements may fall within the scope of the disclosure as defined in the claims that follow.