FIELD OF THE INVENTION

[001] The present invention generally relates to indicators for assisting with deep breathing and meditation. In particular, the present invention is directed to a system and method for providing breathing guidance based on real-time audio analysis.

BACKGROUND

[002] Meditation has been shown to have health benefits but can be difficult to learn to do properly and maintain regular practice. Breathing techniques can be an important part of the meditative process. However, it can be difficult for many to maintain proper breathing for meditation for more than a few minutes. A number of attempts have been made to assist with meditation and proper breathing, including guided meditations as well as visualizations that guide people’s breathing (e.g., indicators signaling a user when to breathe). For many users, however, these may be of limited value for increasing duration and maintaining regular practice as prior art systems are thematically repetitive. Therefore, there is a need for an improved breathing guidance system.

SUMMARY OF THE DISCLOSURE

[003] A method of providing visual breathing indicators that are displayed to correlate with audio being listened to by a user is provided that includes receiving an audio signal, determining a current tempo of the audio signal, determining a time index with a plurality of time indices associated with times that audio events are predicted to occur based on the current tempo, selecting one of a plurality of visual breathing indicators to display at one of the plurality of time indices, determining a duration to display the selected visual breathing indicator, and displaying the selected visual breathing indicator at the selected time index for the determined duration.

[004] A system for displaying visual breathing indicators that are coordinated with audio being listened to by a user includes an audio detector, the audio detector detecting sounds and producing a signal representative of the detected sounds, a display, a computing device coupled to the audio detector and the display, the computing device configured to receive the signal and including a processor and a memory, the memory including a plurality of visual breathing indicators and a set of instructions executable by the processor, the instructions including determining a current tempo of the audio sounds based on the signal, determining a time index that includes a plurality of predicted time indices when musical events will occur based on the current tempo, selecting one of the plurality of visual breathing indicators to display at a selected one of the plurality of predicted time indices, determining a duration to display the selected visual breathing indicator, and sending the selected visual breathing indicator to the display for display at the selected time index for the determined duration.

[005] In another embodiment, a method of providing breathing indicators that are correlated with audio being listened to by a user includes receiving data about a song to be played to a user, generating an FFT derived format of the song from the data, determining a tempo for the song from the FFT derived format using a convolutional neural network, selecting one of a plurality of breathing indicators to activate at a selected time index based on the determined tempo, determining a duration to activate the selected breathing indicator based on the tempo and a breathing guide protocol, activating the selected breathing indicator at the selected time index for the determined duration, selecting a second one of the plurality of breathing indicators to display at a second selected time index based on a change in the determined tempo, determining a second duration to activate the second selected breathing indicator based on the changed tempo and the breathing guide protocol, and activating the second selected breathing indicator at the second selected time index for the second determined duration.

BRIEF DESCRIPTION OF THE DRAWINGS

[006] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

FIG. 1 is a diagram showing a system for providing indicators for breathing based on real-time audio analysis in accordance with an embodiment of the present invention;

FIG. 2 a process diagram outlining a method for determining tempo of audio in accordance with an aspect of the present invention;

FIG. 3 a process diagram outlining another method for determining tempo of audio in accordance with an aspect of the present invention; FIGS. 4A-4D illustrate exemplary visual breathing indicators provided sequentially on a display in accordance with an embodiment of the present invention;

FIG. 5 is a process diagram for providing visual indicators for breathing based on real-time audio analysis in accordance with an embodiment of the present invention;

FIG. 6 is a process diagram outlining a process for determining visual indicators to display based on a determined current tempo of audio in accordance with an embodiment of the present invention; and

FIG. 7 is an exemplary computing system for use with a system providing visual indicators for breathing based on real-time audio analysis according to an embodiment of the present invention.

DESCRIPTION OF THE DISCLOSURE

[007] In the present invention, breathing guidance is provided to a user based on real-time audio detection of music or other audio being listened to by the user. An audio detection system analyzes current audio being played (or to be played in the near future), which may be selected by the user or other audio provider, and determines a current tempo. Breathing indicators are then provided to the user based on and in correlation with the determined tempo to coincide with the audio being played by the user. The system continuously re-determines the tempo (or the tempo of music to be played in the near future). Thus, if the tempo changes within a song, for example, or a new song begins, the system will adjust the breathing indicators that are to be displayed accordingly in real-time. In this way, a user can choose the audio to which to meditate while still receiving appropriate visual breathing indicators for that selected audio. With this system, a user can choose a certain type of music to set the tone for a particular type of meditation, such as for calming down, getting inspired, or trying to fall asleep. By providing tempo-correlated visual indicators that guide users to breathe in sync with music, users can engage in calming breathing exercises while listening to their favorite music from any source, including whichever music app they choose even if the app plays songs at random. This provides an enjoyable and creative experience, which can have the effect of increasing the duration and frequency of meditative activity.

[008] In an embodiment, is a system for providing indicators for guiding the breathing of a user (i.e., listener), the user plays, or is otherwise in the presence of, audio, typically music. The audio is detected, either via a direct connection to a device that is playing the audio or by first capturing sounds from the user’s surroundings. The audio is continuously analyzed to determine a tempo of the audio. For simplicity, the following description will assume that the audio is a musical song, but it will be understood that other sounds may be used even if they do not constitute what may traditionally be considered music.

[009] Based on the current tempo determination, an indicator, including a visual indicator, such as an animation, and/or an auditory overlay or vibration loop, is provided on a display or other suitable apparatus that guides the breathing of the user. The indicators guide the user to breathe based on the current tempo, i.e., the beats per time of the song, and are presented in synchrony in terms of introduction and duration of the indicators with the tempo of the music. The indicators may correspond with a variety of time signatures for breathing. For example, the visual indicators may create a breathing guide schema that coaches the user to breathe in for four beats, hold it in for the next four beats, breathe out during the next four beats, then to hold it out for the next four beats, and so on. Alternatively, the visual indicators may create a breathing guide schema that coaches the user to breathe in for eight beats, hold it in for the next four beats, then breathe out for the next four beats. Any variation of breathing loop duration can be used, preferably, however, it would fit into a loop that adds up to 16 beats for songs in 4/4 or 12 beats for songs in 3/4. Regardless of the particular form of guidance used, the visual indications are preferably presented on the display such that the indicators correspond in time with any detected beat.

[0010] As the visual indicators are being displayed in this manner, the system continues to monitor the audio being played (or to be played). If changes are detected in the tempo, either because a new song begins or the tempo of the current song changes, the visual indicators displayed to the user are changed to remain appropriate for and synchronized with the changed tempo.

[0011] Turning now to the figures, and particularly FIG. 1, a breathing visualization system 100 according to an embodiment of the present disclosure is shown schematically. At a high level, system 100 includes an audio detector 104, a computing device 108, and a display 112.

[0012] Generally, audio detector 104 detects audio signals, either by detecting sounds present in the user’s surrounding environment (e.g., live music or music played from a device to which audio detector 104 is not in direct electronic communication) or by receiving a signal of the music that is being played as it is being played (or before it is scheduled to be played), for example, by monitoring the output signal being sent to headphones or speaker. Upon detecting sounds or receiving a signal representative of the music being played (or scheduled to be played), audio detector 104 sends a signal representative of the current music to computing device 108. If audio detector 104 is used to detect sounds from the environment, the sounds are transformed into a signal representative of the detected sounds in time and that signal is sent to computing device 108.

[0013] Computing device 108 receives the signal from audio detector 104 and analyzes the signal to determine the current tempo of the music. The current tempo may be determined, for example, by analyzing features in the signal that carries the musical information after converting the audio signal into an image (e.g., a spectrogram) or other format using but not limited to the Fast Fourier Transform algorithm (FFT). As outlined in FIG. 2, a tempo determination process 140 for estimating tempo includes receiving, at step 142, an audio signal. The audio signal is converted to an image, such as a spectrogram by FFT or other FFT derived format, at step 144, followed by a preprocessing routine that removes non-music noise from the audio signal at step 145. For a plurality of frequency bands, such as those associated with bass, midrange, and/or treble, peaks are detected at step 146. Preferably, filtering is used to eliminate the counting of peaks that are too close together in time. Each detected peak for each frequency band is associated with a relative time of occurrence, and for every pair of peaks in a given band, a time interval between peaks is determined at step 148. Each interval is then inverted to give a tempo value, preferably in beats per minute (bpm) at step 150. The bpm for each pair of peaks in a frequency band is compiled and an estimated tempo for the frequency band is selected statistically (e.g., by removing outliers, selecting the average of values that fall within a range containing the most values) at step 152. This process can be repeated for each frequency band such that a plurality of tempo estimates are generated. These tempo estimates may be increased in number as sampling continues, until a best estimate tempo emerges, which is then deemed the current tempo. This process will be ongoing so that the current tempo may change over time.

[0014] In the alternative, and as outlined in FIG. 3, the real-time tempo may be determined through the implementation of a pre-processing routine in conjunction with a deep learning architecture trained to recognize the tempo of a song and output tempo (in bpm, for example), beat location, and measure start location. In tempo determination process 160, an audio signal is received at step 162. The audio signal is converted to an image, such as a spectrogram by FFT or other FFT derived format, at step 164. Non-music noise is removed from the audio signal at step 166 by detecting and isolating the likely musical portion from the rest of the audio signal. Subsequently, a convolutional neural network trained on segments of audio files represented as images with labeled time stamp metadata using but not limited to spectrograms derived from the Fast Fourier Transform algorithm or other FFT derived format is implemented that correlates the images or other FFT derived format of the detected audio signal with images or other FFT derived format formed from segments of audio with known tempos at step 168. The training dataset created to build the convolutional neural network is created by one or more humans, determining which portions of songs include viable music (e.g., excluding extended silences), followed by a script that converts the song into usable training data. As more data is accumulated, this process can be automated further reducing the amount of human involvement in training data creation. The neural network is built to be used by devices with limited computing resources. Original audio image resolution can be utilized by many mobile device models. Alternatively, a reduced resolution audio image or other FFT derived format is created to be processed by mobile device models with fewer computational resources, including CPU and memory.

[0015] Based on the determination of the current tempo, computing device 108 selects from stored options or generates a series of visual breathing indicators for guiding the breathing of the user who is listening to that music. The breathing indicators typically are displayed in small groups or a series and are designed to be synchronized with the tempo/beat of the detected music and include breathing guidance that is appropriate for the current tempo, such as visual instructions conveying to the user to breathe in for eight beats, hold it in for the next four beats, then breathe out for the next four beats.

[0016] The generated or selected series of breathing indicators, such as visual breathing indicators 116 (e.g., 116A-116C as shown in FIGS. 4A-4D), are then sent to display 112 for display to the user. The breathing indicators are displayed in essentially real-time with the music being played or detected. In FIG. 4D, an illustrative diagram depicts indicators 116 being displayed at determined times, e.g., T1-T6. As the breathing indicators are being displayed, audio detector 104 continues to detect the music being played and sends updated signals representative of the music to computing device 108. Computing device 108 analyzes the signal and modifies, or maintains the status quo of, as applicable, the series of breathing indicators to be displayed on display 112. Alteratively, if computing device 108 has access to a digital copy of the music being played or to be played, computing device 108 analyzes the metrics of the music that is scheduled to be played to the user and determines which series of breathing indicators is to be shown and when so the series corresponds to the music as it is played.

[0017] In an exemplary embodiment, computing device 108 can be, but is not limited to, a smartphone, smartwatch, tablet, portable computer, or augmented or virtual reality player. It will be understood that computing device 108 may include display 112 and audio detector 104 as built-in components.

[0018] A process for providing visual breathing indicators based on characteristics of audio being played, such as breathing visualization process 200, is outlined in FIG. 5. Breathing visualization process 200 involves detecting audio sounds being played or recognizing an audio signal being played or scheduled to be played at step 204, determining a current tempo of the detected music at step 208 as outlined above, selecting or preparing a series of visual breathing indicators to guide a user’s breathing based on the determined current tempo at step 212, and displaying the series of visual indicators on a display for the user at step 216 such that the indicators are shown at appropriate time indices to be in sync with the determined tempo. This synchronization may be based on a known time index if the music is in a digital format and recognized, or based on predicted time indices based on the determined current tempo of the music being played. Preferably, steps 204, 208, 212, and 216 are repeated continuously so that the displayed breathing indicators remain synchronized with the current tempo of the music even if the tempo (or song) changes. Continuously in the context of this invention means with sufficient frequency such that the displayed indicators are updated, if necessary, at appropriate times within several seconds or less as the music is being played. Alternatively, when a digital audio signal is accessible and analyzed in advance of being played, continuously means that the displayed indicators are updated in order to coincide with changes in the music scheduled to be played.

[0019] A process for determining appropriate series of visual breathing indicators to display based on the determined current tempo, such as exemplary indicator algorithm 300, is outlined in FIG. 6. In process 300, a signal representative of music being played by or for the user is received at step 304. The signal is analyzed at step 308 to determine the tempo as described above. At step 310, the determined tempo is analyzed to determine whether it is new or has otherwise changed. If the tempo remains the same, the signal for the now current music is again received at step 304 and an updated current tempo is determined at step 308. If the tempo has changed, which includes the music starting or a new song being played, a time index is created or updated at step 312. The time index includes the time indices in the future at which certain musical events will likely occur based on the current tempo, assuming the song doesn’t change or stop. For example, the time index may include a beat to occur at 30 seconds from the current clock time. From the currently determined tempo and a selected or associated appropriate breathing guide schema (as discussed above), a series of visual breathing indicators are developed or selected at step 316. The breathing indicators for the selected series are then assigned time indices at which they are to be shown and durations for which they are to be shown at step 320. For example, if the tempo is found to be 60 beats per minute, a first breathing indicator instructing the user to breathe in could be set to be displayed at a time index that would correspond with the playing of a predicted beat peak and would be scheduled to be displayed for 8 seconds. Then a second breathing indicator instructing the user to hold the breath would be scheduled to be displayed immediately after the first visual indicator and be shown for 4 seconds, at which point a third indicator in this series of breathing indicators would be displayed instructing the user to breath out for 4 seconds (corresponding to 16 beats). Thus, these breathing indicators will be timed to display consecutively and such that they are synchronized with the appropriate musical occurrences based on the analysis of the current music signal in step 308 and the predictive time index for the relevant musical occurrences that includes predicted time indices for future relevant musical occurrences developed at step 312. The series to be displayed and the time indices at which each indicator in the series is updated at step 324, which schedules the selected indicators to be shown on a display at the designated times. Note that if the indicators are not visual, then the indicators will be scheduled to be activated (e.g., emit sound or vibrate) at appropriate times for the selected durations.

[0020] The accuracy of these predicted time indices depends on the continuation and constancy of the music. Therefore, the signal is continuously analyzed so that changes in tempo can be accounted for in the selection of appropriate breathing indicators and the time indices at which they are to be shown (as well as their duration). In the alternative, for situations in which data for an entire song (or playlist) is accessible, the breathing indicators and time indices for display of those indicators can be based on that data, and can thus be correlated without any lag since any changes in tempo would be determined prior to those changes being played. In this alternative, the time indices would not be predicted based on the current tempo, but based on available data about the song being played. However, the musical selections would still be monitored for any changes (e.g., skipped songs, changed playlist), so that the selected indicators and times to be shown could be updated or altered as appropriate.

[0021] FIG. 7 shows a diagrammatic representation of one embodiment of a computing system, e.g., computing device 108, in the exemplary form of a system 400, within which a set of instructions for causing a processor 404 to perform any one or more of the aspects and/or methodologies, such as process 300, of the present disclosure.

[0022] System 400 can also include a memory 408 that communicates with other components, via a bus 412. Bus 412 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0023] Memory 408 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., a static RAM “SRAM”, a dynamic RAM “DRAM”, etc.), a read only component, and any combinations thereof. In one example, a basic input/output system 416 (BIOS), including basic routines that help to transfer information between elements within system 400, such as during start-up, may be stored in memory 408. Memory 408 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 420 embodying any one or more of the aspects and/or methodologies of the present disclosure. In another example, memory 408 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof.

[0024] System 400 may also include a storage device 424, such as, but not limited to, the machine-readable storage medium described above. Storage device 424 may be connected to bus 412 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 424 (or one or more components thereof) may be removably interfaced with system 400 (e.g., via an external port connector (not shown)).

[0025] Storage device 424 and an associated machine-readable medium 425 may provide nonvolatile and/or volatile storage of machine-readable instructions, data structures, program modules, and/or other data for system 400. In one example, instructions 420 may reside, completely or partially, within non-transitory machine-readable medium 425. In another example, instructions 420 may reside, completely or partially, within processor 404. Systems 400 may be connected via a network interface 440 to a network 444 so that music provider services 448 can be accessed for music selection.

[0026] System 400 may further include a video display adapter 452 for communicating a displayable image to a display device, such as display 112. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof.

[0027] In this way, a method of providing visual breathing indicators that are displayed to correlate with audio being listened to by a user is provided that includes receiving an audio signal, determining a current tempo of the audio signal, determining a time index with a plurality of time indices associated with times that audio events are predicted to occur based on the current tempo, selecting one of a plurality of visual breathing indicators to display at one of the plurality of time indices, determining a duration to display the selected visual breathing indicator, and displaying the selected visual breathing indicator at the selected time index for the determined duration.

[0028] Additionally or alternatively, the method further includes selecting a second visual breathing indicator from the plurality of visual breathing indicators and displaying the second visual breathing indicator at a second time index for a second determined duration.

[0029] Additionally or alternatively, the method further includes converting the audio signal to a spectrogram or other FFT derived format.

[0030] Additionally or alternatively, the method further includes removing non-music noise from the spectrogram. [0031] Additionally or alternatively, determining a current tempo includes determining a plurality of frequency peaks in a frequency band.

[0032] Additionally or alternatively, the method further includes determining a time interval between each pair of peaks in the plurality of frequency peaks in the frequency band.

[0033] Additionally or alternatively, determining a current tempo includes determining a second plurality of frequency peaks in a second frequency band.

[0034] Additionally or alternatively, the method further includes determining a second time interval between each pair of peaks in the second plurality of frequency peaks in the second frequency band.

[0035] Additionally or alternatively, the method further includes determining the tempo is based on the time interval and the second time interval.

[0036] Additionally or alternatively, determining the tempo includes correlating the digital file with a set of pre-selected digital files representing segments of audio with known tempos.

[0037] Additionally or alternatively, the digital file is a spectrogram or other FFT derived format.

[0038] Additionally or alternatively, the first visual breathing indicator instructs the user to breathe in, wherein the first time index corresponds to when a beat peak is predicted, and wherein the first duration is eight seconds.

[0039] Additionally or alternatively, the second visual breathing indicator the user to hold a breath, wherein the second time index corresponds to when the first duration ends, and wherein the second is four seconds.

[0040] Additionally or alternatively, the method further includes selecting a third visual breathing indicator from the plurality of visual breathing indicators to display at a third time index, determining a third duration to display the third visual breathing indicators, and displaying the third visual breathing indicator, wherein the third time index corresponds to when the second duration ends, wherein the third visual breathing indicator instructs the user to breathe out, and wherein the third duration is four seconds.

[0041] In another embodiment, a system for displaying visual breathing indicators that are coordinated with audio being listened to by a user includes an audio detector, the audio detector detecting sounds and producing a signal representative of the detected sounds, a display, a computing device coupled to the audio detector and the display, the computing device configured to receive the signal and including a processor and a memory, the memory including a plurality of visual breathing indicators and a set of instructions executable by the processor, the instructions including determining a current tempo of the audio sounds based on the signal, determining a time index that includes a plurality of predicted time indices when musical events will occur based on the current tempo, selecting one of the plurality of visual breathing indicators to display at a selected one of the plurality of predicted time indices, determining a duration to display the selected visual breathing indicator, and sending the selected visual breathing indicator to the display for display at the selected time index for the determined duration.

[0042] Additionally or alternatively, the instructions further include selecting a second one of the plurality of visual breathing indicators to display at a selected second one of the plurality of predicted time indices, determining a second duration to display the second selected visual breathing indicator, and sending the second selected visual breathing indicator to the display for display at the second selected time index for the second determined duration.

[0043] Additionally or alternatively, the instructions further include determining an updated current tempo.

[0044] Additionally or alternatively, determining the current tempo includes converting the signal to a spectrogram or other FFT derived format, determining a frequency for each of a plurality of peaks in the spectrogram or other FFT derived format in a frequency band, and determining a time interval between pairs of each of the plurality of peaks.

[0045] Additionally or alternatively, non-music noise is removed from the signal.

[0046] Additionally or alternatively, the current tempo is determined based on multiple time interval determinations between pairs of each of the plurality of peaks. [0047] Additionally or alternatively, determining the current tempo includes converting the signal to a spectrogram or other FFT derived format, removing non-music noise from the signal, and determining the current tempo by using a pre-trained convolutional neural network trained from segments of audio files with known tempos.

[0048] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions, and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.