CROSS-REFERENCES TO RELATED APPLICATION

[0001] This application claims priority to and is a complete specification of Application No. 201941039636, titled “DECODING REAL DATA SIGNAL INTO HUMAN LANGUAGE”, filed on October 01, 2019.

FIELD OF THE INVENTION

[0002] The disclosure generally relates to interpreting real-data signals and, in particular, to a system and method for interpreting real-data signals using neural networks.

DESCRIPTION OF THE RELATED ART

[0003] Artificial Intelligence (AI) has created a great value and buzz across the technological landscape in recent years. Artificial Intelligence has found use in diverse fields such as language processing, text completion, image reconstruction, autonomous cars, voice assistance, and in the fields of robotics, drone movement control, virtual reality applications, video games, graphics simulations, etc. The success of AI algorithms in various fields has opened new ways of creating products like never before.

[0004] There has been rapid advancement in communication technologies in terms of the quality of message signals by reducing noise through various techniques. However, there are still areas of improvement in the communication techniques. Conventionally, human communication may involve conversion of brain signals to speech signals through vocal cords, and the conversion of speech signals to electrical signals through transducers in a mobile device for transmission, and so on. The multiple conversion of data signals may lead to unintended noise introduction and misinterpretation. For instance, the misinterpretation may happen when the vocal cords receive instructions from the brains signals and also due to irreversibility in air, friction, or due to insufficient or improper training to the vocal cords or speech. Further, the oral communication is a bigger challenge for speech impaired people who cannot talk. There is a requirement for a simplified mode of communication directly reading the brain signals without needing to interpret speech generated by brain signals using vocal cords. The need for a technology in the space of mobile communication to bring a new way of communication that makes conversations real, simple and correct.

[0005] The existing methods and techniques used to interpret brain signals, and other real data signals representing electrical activity, involve physical laws and mathematical transformations. An example of physical laws may involve using classical mechanics and Newton’s laws of motion. The other way of interpreting the real data signal is using mathematical transformations like Fourier and Laplace transforms. The mathematical transformations reveal a particular scaling of the real data signal in frequency domain and the scaling is typically correlated with the information or physics of what the real data signal is trying to convey. A repeatability of similar scaling with similar series of data is recorded as the general characteristics of the real data signal when transformed by mathematical transformation. However, such transformation tools provide limited interpretation and are understood only by a few researchers and scientists across the globe and hence the physics or mathematical interpretation of the real-data signals cannot be widely understood.

[0006] There have been various attempts towards decoding real-data signals to speech signals. KR20200088263A discloses a method and system for text-to -speech conversion by generating a speech spectrum from embedding vectors. US20200187841 A1 discloses a system and method for measuring perceptual experiences brain activity measurement device. US20190333505A1 discusses systems and methods for decoding intended speech from neuronal activity. However, these and other publications do not provide an accurate interpretation of real-data signals and are accessible only by researchers and scientists. SUMMARY OF THE INVENTION

[0007] According to one embodiment of the present subject matter, a method of interpreting real-data signals is disclosed. The method includes receiving, by a processor, a plurality of audio signals representing phonetic data and text data in a natural language. The plurality of audio signals are converted into a plurality of digital signals. The plurality of digital signals and associated phonetic data and text data are provided as training data to a point-to-point recurrent neural network (PPRNN) engine. The method includes receiving real-data signals as an input to PPRNN engine, wherein the real-data signal represents an electrical activity. Next, the method involves predicting one or more of phonetics, letters, words, or sentences associated with the real-data signals using the PPRNN engine. The predicted phonetics, letters, words, or sentences are converted into speech signals in the natural language.

[0008] In various embodiments, the method includes training the PPRNN to predict the one or more of phonetics, letters, words, or sentences associated with the real-data signals. The training includes minimizing a loss function to optimize the training data. In various embodiments, the real-data signal is a brain signal. In various embodiments, the method includes transmitting the converted speech signals wirelessly to a receiver device.

[0009] According to another embodiment, a system for interpreting real-data signals is disclosed. The system includes a memory unit and a processor coupled to the memory unit. The processor is configured to receive a plurality of audio signals representing phonetic data and text data in a natural language. The processor converts the plurality of audio signals into a plurality of digital signals and provides the plurality of digital signals and associated phonetic data and text data as training data to a point-to-point recurrent neural network (PPRNN) engine. The processor is then configured to receive real-data signals as an input to PPRNN engine, wherein the real-data signal represents an electrical activity. The processor predicts one or more of phonetics, letters, words, or sentences associated with the real-data signals using the PPRNN engine. The predicted phonetics, letters, words, or sentences are converted into speech signals in the natural language.

[0010] In various embodiments, the system includes a communication unit configured to transmit the converted speech signal. In various embodiments, the system includes a display unit configured to render the predicted the one or more of phonetics, letters, words, or sentences associated with the real-data signals. In various embodiments, the system includes one or more sensors configured to detect and measure the real-data signal. In various embodiments, the PPRNN engine includes a predetermined number of hidden layers, wherein each hidden layer includes a plurality of recurrent neural network nodes.

[0011] This and other aspects are described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The invention has other advantages and features, which will be more readily apparent from the following detailed description of the invention and the appended claims, when taken in conjunction with the accompanying drawings, in which:

[0014] FIG. 1 illustrates a flow diagram for a method for interpreting real-data signals, according to one embodiment of the present subject matter.

[0015] FIG. 2 illustrates a block diagram of a system for interpreting real-data signals, according to one embodiment of the present subject matter.

[0016] FIG. 3 illustrates a simplified block diagram showing the aggregation of data to create a database, according to an embodiment of the present subject matter.

[0017] FIG. 4 illustrates a block diagram of the training process, according to one embodiment of the present subject matter.

[0018] FIG. 5 illustrates a block diagram of the PPRNN engine architecture, according to one embodiment of the present subject matter.

[0019] FIG. 6 illustrates architecture of a recurrent neural network used in the PPRNN engine, according to one embodiment of the present subject matter.

[0020] FIG. 7 illustrates digital wave like signals for the various pronunciations for the word “chocolate”, according to one example of the present subject matter.

[0021] FIG. 8 illustrates a phonetic letter and its corresponding wave like digital signal, according to one example of the present subject matter.

[0022] FIG. 9 illustrates a phonetic word and its corresponding wave like digital signal, according to one example of the present subject matter. [0023] FIG. 10 illustrates a phonetic sentence and its corresponding wave like digital signal, according to one example of the present subject matter.

[0024] FIG. 11A and FIG. 11B illustrate input and predicted wave signals, according to one example of the present subject matter.

[0025] FIG. 12 illustrates wave like digital audio signal of an essay, according to one example of the present subject matter.

[0026] FIG. 13 illustrates output test wave like digital audio signal, according to one example of the present subject matter.

[0027] FIG. 14 illustrates an ionic current signal generated from a nanoporous membrane, according to one example of the present subject matter.

[0028] FIG. 15 illustrates a sample wave like digital EEG signal output from electrodes placed on human brain, according to one example of the present subject matter.

DET AILED DESCRIPTION

[0029] While the invention has been disclosed with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt to a particular situation or material to the teachings of the invention without departing from its scope.

[0030] Throughout the specification and claims, the following terms take the meanings explicitly associated herein unless the context clearly dictates otherwise. The meaning of “a”, “an”, and “the” include plural references. The meaning of “in” includes “in” and “on.” Referring to the drawings, like numbers indicate like parts throughout the views. Additionally, a reference to the singular includes a reference to the plural unless otherwise stated or inconsistent with the disclosure herein.

[0031] The present subject matter describes methods and systems for interpreting real- data signals. A flow diagram for a method for interpreting real-data signals is illustrated in FIG.l, according to various embodiments of the present subject matter. The method includes receiving a plurality of audio signals representing phonetic data and text data in a natural language at block 102. The plurality of audio signals are converted into a plurality of digital signals at block 104. The plurality of digital signals and associated phonetic data and text data are provided as training data to a point-to-point recurrent neural network (PPRNN) engine at block 106. Further, the method includes receiving real-data signals as an input to PPRNN engine at block 108. The real-data signal may represent an electrical activity. In various embodiments, the real-data may refer to data representing electrical activity in brain signals, plant sciences, ocean engineering, flood modeling, climate sciences, tsunami's, cyclones to decode the real data signal obtained from such fields into human language form. The method involves predicting one or more of phonetics, letters, words, or sentences associated with the real -data signals using the PPRNN engine at block 110. The predicted phonetics, letters, words, or sentences are converted into speech signals in the natural language at block 112.

[0032] In various embodiments, the method may include training the PPRNN to predict the one or more of phonetics, letters, words, or sentences associated with the real-data signals. The training may include minimizing a loss function to optimize the training data. In various embodiments, the real-data signal may be a brain signal. In various embodiments, the method may include transmitting the converted speech signals wirelessly to a receiver device.

[0033] A system for interpreting real-data signals is illustrated in FIG. 2, according to another embodiment of the present subject matter. The system 200 may include a memory unit 202 and one or more processing units or processors 204 coupled to the memory unit 202. The processor 204 may be configured to receive a plurality of audio signals representing phonetic data and text data in a natural language. The processor 204 may convert the plurality of audio signals into a plurality of digital signals and provide the plurality of digital signals and associated phonetic data and text data as training data to a point-to-point recurrent neural network (PPRNN) engine 206. The processor 204 may then be configured to receive real-data signals, which may represent electrical activity, as an input to PPRNN engine 206. The processor 204 may predict one or more of phonetics, letters, words, or sentences associated with the real-data signals using the PPRNN engine 206. The predicted phonetics, letters, words, or sentences may be converted into speech signals in the natural language.

[0034] In various embodiments, the system 200 may include one or more sensors 214 configured to detect and measure the real-data signal. In various embodiments, the system 200 may include a communication unit 216 configured to transmit the converted speech signal. In various embodiments, the system 200 includes a display unit 218 configured to render the predicted the one or more of phonetics, letters, words, or sentences associated with the real-data signals.

[0035] In various embodiments, the PPRNN engine may include a training component 208, a validating component 210, and a testing component 212. Each component of the PPRNN engine may receive different datasets. For example, the plurality of audio signals and associated phonetic data and text data in a natural language may be segregated into one or more of a training dataset, a validation dataset, and a test dataset. In some embodiments, the training dataset may encompass 70% of the received data, the validation dataset may include 15% of the received data, and the test dataset may encompass 15% of the received data. Alternatively, the training dataset may encompass 80% and the validation dataset may encompass 20% of the received data. In various embodiments, the training data may be accessed from a remote database.

[0036] A simplified block diagram 300 to illustrate aggregation of data to a database is illustrated in FIG. 3, according to one embodiment of the present subject matter. The database 302 may be created using the plurality of audio signal data 304-1 and the associated phonetics and text data 304-2. The audio signal data 304-1 may include speech or voice data, such as conversational data, audio data in a video, voice notes, songs, and the like. The phonetics and text data associated with the audio signal data 304-1 may also be obtained. In some embodiments, the data 304-1,2 may be obtained from social media 306-1, telephonic data 306-2, sensor data 306-3, media 306-4, smart devices 306-5, and so on. The social media data may include various audio posts or video posts on social networking platforms, the sensor data may include audio or textual data collected by sensors, such as microphones used in computers or phones, the media may include audio data in radio, television or other conventional sources, and the smart devices may include internet of things devices.

[0037] The audio signal data 304-1 and the associated phonetics and text data 304-2 may be subjected to a data preparation step 308. In some embodiments, the data preparation 308 may be configured to pre-process the received audio signal and associated phonetics and text data by eliminating errors and adjusting for missing values. In some embodiments, the data preparation may involve various steps of data cleansing, data augmentation, data standardization on the training dataset. Further, the data preparation may also be configured to analyze the prepared data to create the database 302. In some embodiments, the database 302 may be stored in a separate remote data store that may be accessed through a network.

[0038] Referring back to FIG. 2, the training component 208 may be configured to receive the training data, which may include a plurality of audio signals and associated phonetic data and text data in a natural language. The phonetic data may include speech sounds of the plurality of audio signals and the text data may include letters, words, or sentences associated with the audio signals. In various embodiments, the plurality of audio signals may be converted into digital signals and then provided as training data.

[0039] In various embodiments, historical data in the database 110 may be used for training one or more learning models, such as machine learning models, neural networks, etc., which may be used for making predictions or recommendations. The historical data may include a training dataset and a validation dataset. A block diagram of the prediction process is illustrated in FIG. 4, according to one embodiment of the present subject matter. The historical data 402 may be used for training the prediction models 404 using a neural network engine 206, such as point-to-point neural network engine. The current data 406 may be provided to the prediction models 404 as a test dataset. The current data 406 may include real-data signals, such as brain signals. The prediction models 404 may provide the predicted outputs 408, such as phonetics and text data associated with the real-data signals. The predicted outputs 408 may be provided and stored in the database 302. In some embodiments, future predictions may be performed based on the previously predicted outputs.

[0040] A block diagram of the PPRNN engine architecture 500 is illustrated in FIG. 5, according to various embodiments of the present subject matter. As shown, the training data 502 may be provided as input to the PPRNN engine. In various embodiments, the training dataset 502 may be a multi-dimensional dataset having three or more dimensions. For example, the training dataset may have dimensions of 25x25x1670. The training dataset 502 may include a plurality of training samples 504-N, which may be depicted as a grid point. Each training sample 504-N may be provided as input to a recurrent neural network (RNN) layer 506. The RNN layer may include a plurality of RNNs 506-N.

[0041] In various embodiments, the mathematical formulation of the PPRNN may be given by:

PPRNN =JJ _M tanh(h _ji + b _2i )d _j dQi . (1) hji = W _!i h _{j i} + W _2i Xj_! + b . (2) where x is the input, h is the hidden cell state and Wl, bl and W2, b2 are the weight and bias matrices for hidden-hidden and input-hidden connections. W is the domain of interest and M is the number of examples for training.

[0042] The four boundary conditions may be denoted by bl, b2, b3 and b4. An RNN- LSTM may be assigned to each interior grid point, giving a total of 23x23 = 529 RNNs. The inputs to the model may be the four boundary conditions, bl, b2, b3 and b4, for each example. The boundary condition values may be stacked into single vector and given as input to the model, along with the audio signals and text and phonetic data at the grid point in that example, which is the value to be trained for. The text and phonetic data the model predicts, is denoted by T _pred (x; y). The error between the actual and predicted text and phonetic data may be minimized by iteratively running the training examples through the network. The Sum of Squared error loss function may be used, which takes the form

[0043] The indexing code developed, splits and rearranges the training data into a suitable form to input the audio signals and boundary conditions of each example. In various embodiments, the PPRNN at each grid point may be separately trained in this way to determine the text and phonetic data at the point. The total average error (Sum of Squared Error Loss Function) and standard deviation over all the points may be minimized using the L-BFGS optimizer. The final weight matrix 508 so obtained may be recorded and used to predict the text and phonetic data distribution for a real data signals with four new boundary conditions 510. In the square Dirichlet case, 1670 training examples may be used to get a good prediction, with a 5 hidden layer RNN. In various embodiments, the number of layers and the number of nodes in each layer may be tuned to obtain an accurate prediction 512.

[0044] In various embodiments, different RNNs may be used for each grid point to remove the problem of mesh dependence, as each RNN may be trained separately. Further, the PPRNN also facilitates predictions by training only a subset of the number of grid points, to reduce training time and focus on the points of interest. The PPRNN does not depend on the type of geometry or boundary conditions or discretization of the domain, since a new network is trained at each grid point, which leads to mesh and geometry independent predictions.

[0045] A schematic of the recurrent neural network used in the PPRNN engine is illustrated in FIG. 6. Each interconnection may have a specific weight attached that is initially assigned randomly or with the use of special initializations. The weights may be later updated according to the error the network produces. As shown, the network may include an input layer 602 of dimension d, one hidden layer 604 of dimension J, and an output layer 606 of dimension K.

[0046] The weights between the input and the hidden layer may be given by W _h , of dimension dxJ and the weight matrix between the hidden layer and the output layer may be given by Wo of dimension JxK. The weight of the connection of the 1 ^th input node to the j* hidden node may be given by wij ^h connection of the j* hidden node to the k* output node may be given by wjk°. In addition to the weights, vectors b ^h and b°, called bias vectors may also be defined to account for the case when all the node values are zero. Finally, the trainable parameters are given by, {w ^h , b ^h , w°, b ⁰ }. In various embodiments, the parameters may be updated based on the loss function after each data point is input using back- propagation method. The loss function defines the difference between the true value and the predicted value. In various embodiments, may be one of Sum of Squared error (SSE) and the Cross Entropy (CE) functions.

[0047] In various embodiments, the plurality of audio signals may be converted to digital wave like signals. Various phonetic sounds of common English words when pronounced in different forms and their corresponding wave like signals may be obtained using signal processing. The digital wave like signals for the various pronunciations for the word “chocolate” are illustrated in FIG. 7. The signal processing may include receiving an audio filed of the audio data signal and converting to sampled data in range of [-1,+ 1]. An example of a phonetic letter and its corresponding wave like digital signal is illustrated in FIG. 8. Another example of the wave like digital signals of the phonetic English letters is illustrated in FIG. 9. The obtained wave like digital signal of the phonetic letter may be passed through the PPRNN algorithm to get trained for the phonetic letters and the wave like digital signals to predict the new phonetic letters associated with the new wave like digital signals. In various embodiments, the procedure may be repeated for all words in a natural language, such as English, and corresponding phonetic audio sounds to learn the wave like digital signal corresponding to each English word. A sample digital wave signal for an English sentence “Hi John, How was your day today?” may be represented as shown in the wave signal in FIG. 10.

EXAMPLES

[0048] Example 1: Comparison of an essay between Matlab Audioread() function and PPRNN engine

[0049] The real-data signal was a lecture in a classroom. The real-data was A Thermodynamic lecture on Entropy, which was given as a test essay to convert into wave like digital signal. The lecture was as follows: “Good afternoon folks, Today we are discussing the seventh chapter in Thermodynamics and that is Entropy

. With this I have covered Clausius Inequality, definition of entropy and entropy as a property. Next class, we will be covering entropy for solids, liquids and gases, and entropy calculation using steam tables and the concepts of Lost Work and Irreversibility”. The corresponding actual wave like digital signal of the assay was obtained using audioread() function in Matlab. The wave signal was fed as input to the trained PPRNN engine and the result is shown in FIG. 11B. Both the input depicted in FIG. 11A and the results in FIG. 11B match very well with error < 1%. PPRNN was now trained on audio speech and phonetics, words and sentences to generate wave like digital signals.

[0050] Example 2: Converting wave like digital signals into phonetics, words and sentences using the trained PPRNN

[0051] An example of a wave like digital audio signal of an essay on Thermodynamic lecture on reversible cycles is illustrated in FIG. 12, according to one example. PPRNN was trained to give the phonetics words and sentences for the above wave like digital audio signal. PPRNN was trained on a plurality of wave signal of audio data and the corresponding output phonetics, words, sentences and essay. Below is the output essay corresponding to the FIG. 12 wave like digital audio signal.

[0052] Output essay: “Good afternoon guys, Today we are going to cover the last chapter in Thermodynamics, Reversible cycles . So we have covered Rankine cycle, Brayton cycle and vapor compression cycle and understood how to calculate efficiency for heat engine, COP for refrigerator and also understood the working principles of all these three cycles as heat engine, reversible heat engine (refrigerator) and heat pump.”

[0053] PPRNN was trained on a given wave like digital audio signal and after training we propose to test on a new wave like digital audio signal if it can reproduce the phonetics, words, sentences and essay. FIG. 13 shows the output test wave like digital audio signal. PPRNN outputs the below essay as the prediction:

[0054] Output essay: “Good afternoon guys, today we are looking at second half of

Entropy that we covered last week . So we looked at entropy for solids, liquids and gases, and entropy calculation using steam tables and the concepts of Lost Work and Irreversibility. With this chapter 7 entropy is completed and we will discuss the last chapter reversible cycles on Monday”.

[0055] Example 3: Compare the PPRNN predicted output essay with the actual essay

[0056] Actual Lecture essay: “Good afternoon guys, today we are looking at second half of Entropy that we covered last week . So we looked at entropy for solids, liquids and gases, and entropy calculation using steam tables and the concepts of Lost Work and Irreversibility. With this chapter 7 entropy is completed and we will discuss the last chapter reversible cycles on Monday”. [0057] The PPRNN engine produced 99% accuracy in reproducing the phonetics, words, sentences and the complete essay from the audio signal. Finally, the PPRNN engine was tested on any real data wave like digital signal to convert to speech/human language. FIG. 14 shows an example of an ionic current signal generated from a nanoporous membrane. The ionic current was generated from a nanoporous membrane when the nanoporous membrane is filled with electrolyte solution and driven by an electric field. PPRNN was used to output from the electrical wave like digital signal and the phonetics, words and sentences and grammar using the training PPRNN engine obtained from any wave like digital audio signals. PPRNN engine learns the patterns of each phonetic, words and sentences while getting trained for wave like digital audio signal and preset mles and guidelines for learning a real data signal like an electrical signal considered in the said example to follow those rules and output the corresponding phonetics, words and sentences. The above electrical signal was decoded using PPRNN engine using the preset mles and guidelines of decoding real data signals into phonetics, words and sentences by first training the audio wave signals into phonetics, words and sentences and speech and validating the trained audio signals with audioread() matlab function.

[0058] Example 4: Comparison of truth values and predicted values for given digital data converted from audio signals

[0059] The audio signals of words, such as Hallelujah, How, (sigh while speaking), audio tune (no sound) were provided as test values to the PPRNN engine for predicting the digital data and associated words. Table 1 shows the tmth values and the corresponding predicted values.

Table 1: Comparison of tmth values and predicted values by PPRNN engine

[0060] Example 5: Decoding brain signals into speech

[0061] A real data signal like brain signal obtained from EEG tests of humans and converted into speech/human language using the trained PPRNN engine. FIG. 15 shows an example of a sample wave like digital EEG signal output from electrodes placed on human brain. [0062] For example we the about brain signal was converted into the following essay or speech: “Hi John, how are you doing?” .

Good to see you, thank you.”

[0063] The brain-signals to speech conversion technology allows direct conversion of brain signals and interpretation of the signals to speech without losing the data or leading to misinterpretations due to irreversibility like air, friction when the brain signals are passed to vocal cords and then talk. The disclosed methods may be used in mobile phone and other smart device technologies that can directly read the brain signals of human beings and directly communicate to other person as speech.

[0064] Although the detailed description contains many specifics, these should not be construed as limiting the scope of the invention but merely as illustrating different examples and aspects of the invention. It should be appreciated that the scope of the invention includes other embodiments not discussed herein. Various other modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the system and method of the present invention disclosed herein without departing from the spirit and scope of the invention as described here.