RELATED APPLICATION INFORMATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63/253,169, filed on October 7, 2021, U.S. Provisional Patent Application No. 63/308,081, filed on February 9, 2022, and U.S. Utility Patent Application No. 17/958,597 filed on October 3, 2022 incorporated herein by reference in their entireties.

BACKGROUND

Technical Field

[0002] The present invention relates to machine learning and more particularly to codisentangled time series/text multi-modal representation learning for controllable generation.

Description of the Related Art

[0003] Time series data are prevalent in the loT era. For instance in industrial monitoring, a large number of sensor streams constitute complex time series. Machine learning software are developed to automatically detect and classify time series patterns. However, compared to natural language comments from human experts, the output of Al software are usually simple 1-of-N labels which cannot cover the multitude of aspects the operators want to know about the data. This lack of comprehensiveness and naturalness in feedbacks has hindered the adoption of Al time series analytics in the industrial field. SUMMARY

[0004] According to aspects of the present invention, a computer-implemented method for multi-model representation learning is provided. The method includes encoding, by a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation. The method further includes generating, by a trained text generator, a natural language text that explains the input TS segment, responsive to the TS-shared latent representation, the TS-private latent representation, and a text-private latent representation.

[0005] According to other aspects of the present invention, a computer program product for multi-model representation learning is provided. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes encoding, by a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation. The method further includes generating, by a trained text generator, a natural language text that explains the input TS segment, responsive to the TS-shared latent representation, the TS-private latent representation, and a text-private latent representation.

[0006] According to still other aspects of the present invention, a computer processing system for multi-model representation learning is provided. The computer processing system includes a memory device for storing program code. The computer processing system further includes a hardware processor operatively coupled to the memory device for running the program code to encode, using a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation;. The hardware processor further runs the program code to generate, using a trained text generator, a natural language text that explains the input TS segment, responsive to the TS- shared latent representation, the TS-private latent representation, and a text-private latent representation.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

[0008] The disclosure will provide details in the following description of preferred embodiments with reference to the following figures wherein:

[0009] FIG. 1 is a block diagram showing an exemplary computing device, in accordance with an embodiment of the present invention;

[0010] FIG. 2 shows an exemplary text-annotated time series dataset derived from maintenance logs and time series historian software, in accordance with an embodiment of the present invention;

[0011] FIG. 3 is a block diagram showing an exemplary use case, in accordance with an embodiment of the present invention;

[0012] FIG. 4 is a diagram showing an explaining of factors of variations, in accordance with an embodiment of the present invention;

[0013] FIG. 5 is a diagram showing an exemplary architecture of a multi-modal VAE, in accordance with an embodiment of the present invention; [0014] FIG. 6 is a diagram showing an exemplary training procedure, in accordance with an embodiment of the present invention;

[0015] FIG. 7 is a diagram showing an exemplary first use case, in accordance with an embodiment of the present invention;

[0016] FIG. 8 is a diagram showing an exemplary second use case, in accordance with an embodiment of the present invention;

[0017] FIG. 9 is a diagram showing an exemplary third use case, in accordance with an embodiment of the present invention;

[0018] FIG. 10 is a block diagram showing an exemplary environment to which the present invention can be applied, in accordance with an embodiment of the present invention; and

[0019] FIG. 11 is a block diagram showing another exemplary environment to which the present invention can be applied, in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0020] Embodiments of the present invention are directed to co-disentangled time series/text multi-modal representation learning for controllable generation.

[0021 ] Embodiments of the present invention utilize text-annotated time series data (e.g. obtained from maintenance records) to train a neural network model that generates readable and accurate explanatory texts given a time series segment (e.g., from loT sensor streams). In addition, the factors of variations in the data of both modalities are automatically identified to make the generation process interpretable and controllable. This is achieved by a novel architecture based on multimodal variational autoencoder and a novel training objective called “co-disentanglement”. The goal is to implement a virtual expert that helps operators better understand their domain time series data and make more informed decisions. As used herein, the term “private”, as in TS-private or text-private means that the referred information or features are uniquely contained in one of the TS or text modalities and not the other. Also, the term “shared”, as in TS-shared or text-shared means that the referred information or features are common to both the TS and text modalities.

[0022] FIG. 1 is a block diagram showing an exemplary computing device 100, in accordance with an embodiment of the present invention. The computing device 100 is configured to perform co-disentangled time series/text multi-modal representation learning for controllable generation.

[0023] The computing device 100 may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a rack based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor- based system, and/or a consumer electronic device. Additionally or alternatively, the computing device 100 may be embodied as a one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically disaggregated computing device. As shown in FIG. 1, the computing device 100 illustratively includes the processor 110, an input/output subsystem 120, a memory 130, a data storage device 140, and a communication subsystem 150, and/or other components and devices commonly found in a server or similar computing device. Of course, the computing device 100 may include other or additional components, such as those commonly found in a server computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory 130, or portions thereof, may be incorporated in the processor 110 in some embodiments.

[0024] The processor 110 may be embodied as any type of processor capable of performing the functions described herein. The processor 110 may be embodied as a single processor, multiple processors, a Central Processing Unit(s) (CPU(s)), a Graphics Processing Unit(s) (GPU(s)), a single or multi-core processor(s), a digital signal processor(s), a microcontroller(s), or other processor(s) or processing/controlling circuit(s).

[0025] The memory 130 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory 130 may store various data and software used during operation of the computing device 100, such as operating systems, applications, programs, libraries, and drivers. The memory 130 is communicatively coupled to the processor 110 via the I/O subsystem 120, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor 110 the memory 130, and other components of the computing device 100. For example, the VO subsystem 120 may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, platform controller hubs, integrated control circuitry, firmware devices, communication links (e.g., point-to- point links, bus links, wires, cables, light guides, printed circuit board traces, etc. ) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem 120 may form a portion of a system-on-a-chip (SOC) and be incorporated, along with the processor 110, the memory 130, and other components of the computing device 100, on a single integrated circuit chip.

[0026] The data storage device 140 may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device 140 can store program code for co-disentangled time series/text multi-modal representation learning for controllable generation. The communication subsystem 150 of the computing device 100 may be embodied as any network interface controller or other communication circuit, device, or collection thereof, capable of enabling communications between the computing device 100 and other remote devices over a network. The communication subsystem 150 may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

[0027] As shown, the computing device 100 may also include one or more peripheral devices 160. The peripheral devices 160 may include any number of additional input/output devices, interface devices, and/or other peripheral devices. For example, in some embodiments, the peripheral devices 160 may include a display, touch screen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and/or other input/output devices, interface devices, and/or peripheral devices. [0028] Of course, the computing device 100 may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements.

For example, various other input devices and/or output devices can be included in computing device 100, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and/or wired input and/or output devices can be used. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized. These and other variations of the processing system 100 are readily contemplated by one of ordinary skill in the art given the teachings of the present invention provided herein.

[0029] As employed herein, the term “hardware processor subsystem” or “hardware processor” can refer to a processor, memory (including RAM, cache(s), and so forth), software (including memory management software) or combinations thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and/or a separate processor- or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem can include one or more on-board memories (e.g., caches, dedicated memory arrays, read only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories that can be on or off board or that can be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input/output system (BIOS), etc.). [0030] In some embodiments, the hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and/or one or more applications and/or specific code to achieve a specified result.

[0031] In other embodiments, the hardware processor subsystem can include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application-specific integrated circuits (ASICs), FPGAs, and/or PLAs.

[0032] These and other variations of a hardware processor subsystem are also contemplated in accordance with embodiments of the present invention

[0033] FIG. 2 shows an exemplary text-annotated time series dataset 200 derived from maintenance logs 210 and time series historian software, in accordance with an embodiment of the present invention.

[0034] From the text-annotated time series dataset 200, one can extract synchronized pairs of time series segments and natural language texts. This allows learning of domainspecific relationships between time series and texts to facilitate domain data understanding and staff training. No existing analytics system utilizes such multimodal data jointly, despite their informativeness and the high cost of acquiring them.

[0035] A description will now be given regarding an embodiment of the present invention.

[0036] First, a dataset comprising pairs of time series segments and free-form comment texts is used to train the model (FIG. 6). The time series can be either univariate or multivariate. Then at test time, the trained model takes a potentially unseen time series as input, and generates texts that can be used as interpretation or explanation of the input. The user can run the model repeatedly, and each time the model will generate a different text, explaining the input in different ways (FIG. 7). A reference text can be optionally provided as input to impose the style of the generated text (FIG. 8). An example usage of this system is in FIG. 2. The user can further obtain explanation of the factors of variations in the data using the proposed latent dimension traversal procedure (FIGs. 4 and 9).

[0037] FIG. 3 is a block diagram showing an exemplary use case 300, in accordance with an embodiment of the present invention. The use case is for explaining sensor time series in natural language. In this example, a user provides a multi-variate time series segment 310 in need of explanation. The human-friendly analytics software system 320 generates a set of comment texts 330 for the given time series (showing just one text among many sampled).

[0038] FIG. 4 is a diagram showing an explaining of factors of variations, in accordance with an embodiment of the present invention.

[0039] Explaining factors of variations. Starting from a time series 410 and its corresponding generated text 420, a user can manipulate them 410, 420 simultaneously by changing a particular dimension of the latent representation vector 430. The lower part of FIG. 4 illustrates the effect of such manipulation.

[0040] Previous works focused on either time series classification or text generation. Embodiments of the present invention are first to address the time series-to-text generation task in an end-to-end manner, addressing both sub-tasks jointly in a single model. It is also the first work to propose a co-disentanglement algorithm, which encourages consistency between the learned factors across two modalities. This algorithm simultaneously addresses the weaknesses of two existing families of methods: (1) VAE-based multimodal data modeling methods whose learned representations lack interpretability; and (2) disentangled representation learning methods that aim to identify factors for a single modality but cannot take advantage of multiple modalities.

[0041] FIG. 5 is a diagram showing an exemplary architecture 500 of a multi-modal VAE, in accordance with an embodiment of the present invention.

[0042] The architecture 500 includes a time series part 501 and a text part 551.

[0043] The time series part 501 receives a time series segment 511 by a time series encoder 512 which outputs a TS-private latent representation 513 and a TS-shared latent representation 514 to a time series generator 515 that, in turn, outputs a generated time series 516.

[0044] The text part 551 receives text 561 by a text encoder 562 which outputs a text- shared latent representation 563 and a text-private latent representation 564 to a text generator 565 that, in turn, outputs generated text 566.

[0045] FIG. 6 is a diagram showing an exemplary training procedure 600, in accordance with an embodiment of the present invention. The training procedure 600 involves a training dataset 601 having a time series database 601 A (of historical sensor data) and text logs 60 IB with free-form human-written comments.

[0046] The training procedure 600 involves data pre-processing 610 including time series augmentation 611 and text augmentation 612.

[0047] The training procedure 600 further involves training 620 the multi-modal disentangled VAE by optimizing the described objective using stochastic gradient descent. [0048] FIG. 7 is a diagram showing an exemplary first use case 700, in accordance with an embodiment of the present invention.

[0049] A time series segment 701 is received. The time series segment 701 is preprocessed 702. A trained time series encoder 720 outputs a TS-shared latent representation 721 and a TS-private latent representation 722. A trained text generator 730 receives the TS-shared latent representation 721 and the TS-private latent representation 722 and also a test-private latent 731 sampled from previously, and outputs a piece of natural language text 732 that explains the input time series segment 701 for human users. [0050] FIG. 8 is a diagram showing an exemplary second use case 800, in accordance with an embodiment of the present invention.

[0051] A time series segment 801 is received for pre-processing 802, and the pre- processed time series is input to a trained time series encoder 810 which outputs a TS- shared latent representation 811 and a TS-private latent representation 812.

[0052] A list of example text 821 with a desired style is received for pre-processing 822, and the pre-processed text is then received by a trained text encoder 823 that outputs a textprivate latent representation 824. In place of the list of example text 821 to finally obtain the text-private latent representation 824, one can select 825 a text-private latent representation from a list of style latent prototypes to obtain the text-private latent representation 824.

[0053] The TS-shared latent representation 811, the TS-private latent representation 812, and the text-private latent representation 824 are input to a trained text generator 830 to output of a piece of natural language 831 in the desired style that explains the input time senes. [0054] FIG. 9 is a diagram showing an exemplary third use case 900, in accordance with an embodiment of the present invention.

[0055] A time series segment 901 is received for pre-processing 902, and the pre- processed time series is input to a trained time series encoder 910 which outputs a TS- private latent representation 911 and a TS-shared latent representation 912.

[0056] A loop 921 is commenced for each dimension z of the shared latent representation to visualize the generated pairs as a function of changing perturbation. To that end, the loop 921 receives time-series 941 resulting from the perturbed latent representation 933 and the explanation text 953 to visualize 959 the generated pairs as a function of changing perturbation.

[0057] A loop 931 is commenced for each perturbation from -1 to 1 to perform the following. Add 932 the perturbation to dimension z of TS-shared latent representation to output perturbed TS-shared latent representation 933. Input the TS-private latent representation 911 and the perturbed TS-shared latent representation 933 to a trained time series generator 940 to output a time series 941 resulting from the perturbed latent representation 933. Input the perturbed TS-shared latent representation 933 and a textprivate latent representation 951 sampled from prior to a trained text generator 952 to output explanation text 953.

[0058] Data analysts obtain 960 insights about the factors of variations in time series and corresponding semantics of each factor.

[0059] Data Preprocessing [0060] Acquire a multi-modal dataset consisting of pairs of time series segment and natural language comment text in English. The time series can include more than one representations.

[0061] Each time series is normalized by the minimum and maximum values so the new range is between 0 and 1. Augmented copies of each time series are generated by adding Gaussian noise.

[0062] The set of available text is augmented by back-translation. Each text is translated to a foreign language and then translated back to English using a machine translation tool such as Google Translate. The resulting text will have the same meaning as the original text but will likely use different words. Using K foreign languages, each text is converted to K rephrased texts with identical semantics. Each rephrased text is associated with the time series to form a new data pair. Each text is tokenized into a token sequence using a tokenizer such as SentencePiece. A start-of-text symbol <s> is inserted at the beginning of the text, and an end-of-text symbol </s> is appended to the end of the text.

[0063] The augmented copies of the original time series-text pairs are used to produce additional pairs. In the resulting dataset, denote the number of pairs by n/, the i’th data pair by (x ⁽ⁱ⁾ , y ⁽ⁱ⁾ ) where x ⁽ⁱ⁾ is the time series segment and is the text.

[0064] Network Model Architecture

[0065] The multi-modal VAE neural network is illustrated in FIG 5. It includes (1) a time series encoder 512, (2) a text encoder 562, (3) a time series generator 515 and (4) a text generator 565.

[0066] The time series encoder 512 can be implemented as a RNN, a transformer or any sequence model. One possible implementation is a dual-attention RNN for multivariate input series. The time series encoder 512 takes as input a (possibly multivariate) time series segment. At each timestep outputs a hidden state vector that summarizes the accumulated information at each timestep of the time series. The hidden state vector of the last timestep is used as the TS-shared and TS-private latent representation of the input time series segment.

[0067] The time series generator 515 can be implemented as a RNN, a transformer or any sequence model. The time series generator 515 takes a pair of shared 513 and private latent representation vectors 514 as the condition, and outputs a (possibly multivariate) time series of a specified length.

[0068] The text encoder is a transformer stack. It takes a tokenized text sequence. The output of the last transformer layer at the position of the </s> token is used as the text- shared 563 and text-private latent representations 564.

[0069] The text generator 565 is a transformer stack. The text generator 565 takes a pairs of shared 563 and private latent representation vectors 564 as the condition and outputs a token sequence 566. Optionally, it can take a partially completed token sequence as additional input, and output a token sequence where the input is a prefix. The forward passing of the generator 565 works in an autoregressive position-by-position manner. At each position, the embedding of the input token is looked up and passed to the self-attention layer. A subsequent projection layer generates a probability distribution over all tokens in the vocabulary. According to this distribution, a text token is sampled as the output at the current position.

[0070] Training [0071] The neural network is trained end-to-end in iterations. At each training iteration, a batch of B time series-text pairs is sampled and the following objective L is computed. The objective is maximized by stochastic gradient descent (SGD).

[0072] £ £1 + £2 +I- £3

[0073] The complete set of objective can be decomposed into three groups L ₁ , L ₂ and L ₃ as follows.

[0074] Group 1 : Single-modal objectives,

[0075] These include:

[0076] Minimizing the reconstruction error of data from the latent representations from same modality's encoder (Terms 1 and 2). Here function r measures the reconstruction error as the following.

[0077] The text reconstruction error is computed using the cross entropy. At each position, the cross entropy between the generated token probability distribution and the true token from the training text is computed. This is averaged over each sequence and over the minibatch.

[0078] The time series reconstruction error is computed as the mean squared error (MSE) between the input time series and the reconstructed one, both as n-by-T matrices. This is averaged over the minibatch.

[0079] Minimizing the KL divergence between latent and standard Gaussian prior (Terms 3-6) [0080] Group 2: Multi-modal objectives,

[0081] These include:

[0082] Minimizing the error of reconstructing one modality's data using the other modality's shared latent representation (Terms 1 and 2)

[0083] Maximizing the mutual information between the two shared representations in both modalities, (term 3). Here I denotes mutual information between two random variables. The mutual information is estimated using the NWJ lower bound.

[0084] Minimizing the mutual information between shared and private component of the representation (term 4 and 5). The mutual information is estimated using the CLUB upper bound.

[0085] Group 3 : Factor disentanglement objectives

[0086] These include:

[0087] Minimizing the total correlation between the factors in the representation of each modality (terms 1 and 2). Here TC denotes total correlation between multiple random variables.

[0088] Maximizing mutual information between matched factor pairs across modalities (term 3). The mutual information is estimated using the NWJ lower bound.

[0089] Training a randomly initialized network using the complete set of objectives can be difficult as optimization is likely to get stuck in the many local optima at the beginning. We adopt a three-stage warm-start approach to alleviate this difficulty. The training proceeds in three stages:

[0090] At stage 1, only the single-modal objectives are trained. This stage lasts for e.g. 1000 iterations.

[0091] At stage 2, the multi-modal objectives are added. This stage lasts for e.g. another 1000 iterations.

[0092] At stage 3, the factor disentanglement objectives are further added. This stage lasts until convergence.

[0093] In stage 1, in each iteration, a minibatch of data (B time series-text pairs) is sampled and stochastic gradient descent (SGD) is used to update the parameters of encoders/decoders. In stage 2 and 3, we alternate updating the encoders/decoders with updating the estimators. After each iteration of encoder/decoders update, we perform 5 iterations of estimator update.

[0094] In training the text generator, a mixture of teacher-forcing and free-running training modes is used to improve the generation quality. After the text generator outputs each token, we randomly choose one of these two modes to determine the input at the next position. In teacher forcing mode, the input at the next position is the true token at that position in the training text. In the free-running mode, a token is sampled from the vocabulary according to the output distribution of the current position, and used as the input at the next position.

[0095] Use Case 1 : Generating Text Explanation from an Input Time Series (without Particular Text Style) - FIG. 7 [0096] A time series segment 701 that the user wants explanation is preprocessed and passed through the trained time series encoder 720. The shared latent 721, in combination with a text-private latent sampled 731 from a standard Gaussian prior, are fed into the trained text generator 730 as condition. The initial input to the text generator 730 is the start-of-text symbol <s>. Free-running mode is used and the sampled output token at each position is used as the input for the next position. This dynamic decoding process repeats until a </s> symbol is generated. The generated token sequence is joined to form a text string that will be returned to the user. Repeated samplings can generate a diverse set of texts.

[0097] Use Case 2: Generating Text Explanation from an Input Time Series (with Particular Text Style) - FIG. 8

[0098] The procedure is similar to the first use case, except that the text-private latent is obtained in a different way.

[0099] If the user has one or more reference texts with the desired style, these texts can be fed into the trained text encoder 823, and the mean of the resulting text-private latent representations is used. The user can also select from a list of text-private representations that are known to correspond to a particular style. Such a list can be pre-trained using a curated corpus of texts in different styles.

[0100] Use Case 3: Explaining Factors of Variation - FIG. 9

[0101] One can use the trained model to analyze what the latent factors of variations in the dataset are, and understand the manifestation of each factor in terms of time series and of text. The user selects a time series segment as reference. This time series segment is fed to the trained time series encoder and the TS-private and TS-shared latent vectors are obtained. We perturb each dimension of the TS-shared latent vector with a range of magnitudes. For each perturbed dimension and each perturbation magnitude, we feed the new TS-shared latent, together with the original TS-private latent to the trained time series generator to obtain a new time series. We also feed the new TS-shared latent, together with a text-private latent sampled from a standard Gaussian prior, to the trained text generator to obtain a new text. The collection of pairs of new time series and new texts from each perturbation are visualized. Such visualization provides human data analysts with insights about the semantics and quantitative effect of each factor of variation in their domain data. [0102] FIG. 10 is a block diagram showing an exemplary environment 1000 to which the present invention can be applied, in accordance with an embodiment of the present invention.

[0103] The environment 1000 includes a multi-modal representation learning system 1010 and a controlled system 1020. The multi-modal representation learning system 1010 and the controlled system 1020 are configured to enable communications therebetween. For example, transceivers and/or other types of communication devices including wireless, wired, and combinations thereof can be used. In an embodiment, communication between the multi-step-ahead forecasting system 1010 and the controlled system 1020 can be performed over one or more networks, collectively denoted by the figure reference numeral 1030. The communication can include, but is not limited to, multi-variate time series data from the controlled system 1020, and forecasts and action initiation control signals from the multi-step-ahead forecasting system 1010. The controlled system 1020 can be any type of processor-based system such as, for example, but not limited to, a surveillance system, a manufacturing system (e.g., an assembly line), an Advanced Driver- Assistance System (ADAS), and so forth.

[0104] The controlled system 1020 provides data (e.g., multi -variate time-series data) to the multi-step-ahead forecasting system 1010 which uses the data to generate text explanations of the time series data.

[0105] The controlled system 1020 can be controlled based on an explanation generated by the multi-modal representation learning system 1010. For example, based on an explanation indicating that a machine will fail in x time steps, a corresponding action (e.g., power down machine, enable machine safeguard to prevent injury/etc., and/or so forth) can be performed at t<x in order to avoid the failure from actually occurring. As another example, based on a trajectory of an intruder, a surveillance system being controlled could lock or unlock one or more doors in order to secure someone in a certain place (holding area) and/or guide them to a safe place (safe room) and/or restrict them from a restricted place and/or so forth. Verbal (from a speaker) or displayed (on a display device) instructions could be provided along with the locking and/or unlocking of doors (or other actions) in order to guide a person. As a further example, a vehicle can be controlled (braking, steering, accelerating, and so forth) to avoid an obstacle that is predicted to be in a car’s way responsive to an explanation. As a yet further example, the present invention can be incorporated into a computer system in order to forecast impending failures and take action before the failures occur, such as switching a component that will soon fail with another component, routing through a different component, processing by a different component, and so forth. It is to be appreciated that the preceding actions are merely illustrative and, thus, other actions can also be performed depending upon the implementation, as readily appreciated by one of ordinary skill in the art given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

[0106] In an embodiment, the multi-modal representation learning system 1010 can be implemented as a node in a cloud-computing arrangement. In an embodiment, a single multi-modal representation learning system 1010 can be assigned to a single controlled system or to multiple controlled systems e.g., different robots in an assembly line, and so forth). These and other configurations of the elements of environment 1000 are readily determined by one of ordinary skill in the art given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

[0107] FIG. 11 is a block diagram showing another exemplary environment 1100 to which the present invention can be applied, in accordance with an embodiment of the present invention.

[0108] The environment 1100 includes a controlled system 1120 that, in turn, includes a multi-modal representation learning system 1110. One or more communication buses and/or other devices can be used to facilitate inter-system, as well as intra-system, communication. The controlled system 1120 can be any type of processor-based system such as, for example, but not limited to, a surveillance system, a manufacturing system (e.g., an assembly line), an Advanced Driver- Assistance System (ADAS), and so forth.

[0109] Other than system 1110 being included in system 1120, operations of these elements in environments 1100 and 1100 are similar. Accordingly, elements 1110 and 1120 are not described in further detail relative to FIG. 11 for the sake of brevity, with the reader respectively directed to the descriptions of elements 1110 and 1120 relative to environment 1000 of FIG. 10 given the common functions of these elements in the two environments 1000 and 1100.

[0110] The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0111] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0112] Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

[0113] Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user'ss computer, as a stand-alone software package, partly on the user'ss computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field- programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

[0114] Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

[0115] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

[0116] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0117] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions. [0118] Reference in the specification to “one embodiment” or “an embodiment” of the present invention, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

[0119] It is to be appreciated that the use of any of the following “/”, “and/or”, and “at least one of’, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and/or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.

[0120] The foregoing is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the present invention and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention. Having thus described aspects of the invention, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.