TECHNICAL FIELD The present disclosure relates to breath monitoring and in particular to devices, systems and methods for monitoring the breathing of a patient such as a person suffering from asthma.

BACKGROUND

Many children suffer from various illnesses and diseases throughout their childhood and some consequences of them are tragic. Asthma is a widespread disease around the globe, where it is ranked third as the main cause for hospitalization, for children under the age of 15. Asthma can be a serious respiratory condition which has a serious impact on a person’s quality of life and health and can lead to death. Average figures suggest that 383,000 people die due to asthma every year. From these 202,990 are children.

Additionally, asthma creates and causes many secondary ailments leading into different sicknesses and or amplifying the adverse effects of other diseases. A major problem faced by patients especially children and doctors alike is the effective administration of asthmatic medication which to a large extent are inhaler-based medication. It has been estimated that 93% of children suffering from asthma do not know how to use inhalers properly and they also suffer from severe breathing problems. The main reason for this is that children do not like to use inhalers as treatments due to the lack of motivation in the treatment process and a proper tracking mechanism to see the progress of the asthma condition of the child. Most asthma patients are unaware of correct breathing exercises and unwilling to do breathing exercises since there is no motivation factor. SUMMARY OF THE INVENTION

According to a first aspect of the present disclosure, a device for detecting breath sounds of a subject is provided. The device comprises: a housing having a portion configured to be insertab!e in a first ear of the subject; a first microphone arranged within the housing to capture sounds within the ear of the subject; and a second microphone arranged within the housing to capture sounds external to the ear of the subject.

In an embodiment, the device further comprises a processing module configured to combine signals from the first microphone and the second microphone to generate a breathing sound signal for the subject. in an embodiment, the processing module is configured to apply a noise reduction algorithm to the signals from the first microphone and the second microphone. The noise reduction algorithm may be a normalized least mean squares algorithm.

In an embodiment, the device further comprises a communication interface configured to transmit an indication of the breathing sound of the subject to a user device.

In an embodiment the second microphone is arranged within the housing such that second microphone faces the pinna or outer ear of the first ear of the subject when the portion of the housing configured to be insertable in the first ear of the subject is inserted in the first ear of the subject.

In an embodiment the device further comprises a second housing portion configured to be insertable in a second ear of the subject, the second housing portion having a speaker arranged therein.

According to a second aspect of the present disclosure, a method of monitoring breathing of a subject is provided. The method comprises: receiving on an indication of breathing sound signal from a breath monitoring device; identifying breathing events by classifying parts of the sound signal; determining a temporal length of an identified breathing event; and generating a breathing event indication.

In an embodiment the breathing events are inhaling and / or exhaling.

In an embodiment generating a breathing event indication comprises controlling a displayed object on a display as part of a game. The game may be configured to simulate breathing exercises.

In an embodiment classifying parts of the sound signal comprise using a machine learning classifier to classify parts of the sound signal.

In an embodiment, the method further comprises generating a mel-scaled spectrogram from the indication of the breathing signal and wherein classifying parts of the sound signal comprises using the mel-scaled spectrogram.

In an embodiment the breathing events comprise inhaler usage and wherein generating a breathing event indication comprises estimating an inhaled dosage.

According to a third aspect of the present invention an electronic device comprising a processor and a storage device is provided. The storage device stores computer executable instructions for causing the processor to carry out a method comprising: receiving on an indication of breathing sound signal from a breath monitoring device; identifying breathing events by classifying parts of the sound signal; determining a temporal length of an identified breathing event; and generating a breathing event indication.

In an embodiment, the breathing events are inhaling and / or exhaling.

In an embodiment, generating a breathing event indication comprises controlling a displayed object on a display as part of a game. The game may be configured to simulate a breathing exercise. In an embodiment, classifying parts of the sound signal comprises using a machine learning classifier to classify parts of the sound signal.

In an embodiment, the method further comprises generating a mel-scaled spectrogram from the indication of the breathing signal and wherein classifying parts of the sound signal comprises using the mel-scaled spectrogram.

In an embodiment, the breathing events comprise inhaler usage and wherein generating a breathing event indication comprises estimating an inhaled dosage.

According to a fourth aspect of the present disclosure a breath monitoring system is provided. The breath monitoring system comprises: a user device configured to monitoring breath sounds of a patient and generate a symptom diary for the patient, the symptom diary comprising indications of breathing events for the patient; a breath monitoring and analysis server configured to receive the symptom diary from the user device and to store indications of the breathing events in a database; and a physician terminal configured to allow a doctor to access the indications of breathing events stored on the database. In an embodiment, the breath monitoring system further comprises an air quality measuring device comprising a plurality of sensors configured to measure indications of environmental conditions in the vicinity of the patient, wherein the breath monitoring and analysis server is configured to receive indications of the environmental conditions in the vicinity of the patient and store the indications of the environmental conditions in the database.

The plurality of sensors may comprise at least one of the following: a particulate matter sensor, a volatile organic compound sensor, a carbon dioxide sensor, a temperature sensor and a humidity sensor.

In an embodiment, the user device is an electronic device as described above.

Embodiments of the present invention provide a product that assists Asthma Patients in particular children and gives a perfect motivation for the patients to carry out breathing exercises properly and keep on using the inhalers accordingly. The product comes with a headset that will identify the breathing patterns of the user, and mobile games, that motivate the children to use the inhaler and get healthier as they play. The game also supports single player as well as multi-player features. Embodiments of the present invention facilitate enhanced administration of specific inhaler-based medicines for asthma simultaneously having the ability to assimilate and evaluate data to support a symptom diary, identification of preventive measures.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, embodiments of the present invention will be described as non limiting examples with reference to the accompanying drawings in which:

Figure 1 shows a breath monitoring system according to an embodiment of the present invention;

Figures 2a and 2b show an earpiece breath monitoring device according to an embodiment of the present invention; Figures 3a and 3b illustrate a processing of signals captured by an earpiece breath monitoring device according to an embodiment of the present invention;

Figure 4 shows a block diagram of an earpiece breath monitoring device according to an embodiment of the present invention;

Figure 5 shows a block diagram of a user device for processing breathing sounds of a user according to an embodiment of the present invention;

Figure 6 is a flowchart showing a method of breath monitoring according to an embodiment of the present invention;

Figure 7 shows a mel spectrogram used in embodiments of the present invention to classify sounds; Figure 8 shows an example of analysis of breathing sounds carried out in an embodiment of the present invention; Figure 9 is a block diagram showing an air quality measuring device according to an embodiment of the present invention; and

Figure 10 shows a breath monitoring system according to an embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide a system for monitoring breathing and inhaler usage by patients such as children who are asthma suffers. The system may include games that give motivation for children to carry out breathing exercises properly and keep on using the inhalers.

Figure 1 shows a breath monitoring system according to an embodiment of the present invention. The breath monitoring system 100 comprises an earpiece breath monitoring device 110 which is inserted into the ear of the patient in a similar manner to a normal earpiece. The earpiece breath monitoring device 110 comprises microphones arranged to capture and isolate the breath sounds of the patient. The breath monitoring system 100 further comprises a user device 120 such as a smart phone or tablet computer. The user device 120 may be coupled to the earpiece breath monitoring device 110 via a wireless network connection such as a Bluetooth connection. In alternative embodiments, the user device 120 and the breath monitoring device are coupled by a wired connection. The user device 120 runs a software application which identifies breathing patterns of the patient from the signals captured by the earpiece breath monitoring device 110. The software application may include games that are controlled by the breathing of the patient. Such a game motivates children to use the breathing inhaler and carry breathing exercises properly and get healthier as they play. The games may also support single player as well as multi-player features. The software application may be configured to identify inhaler usage in addition to analyzing breathing patterns of the patient.

The breath monitoring system 100 further comprises an air quality measuring device 130 which is located close to the patient and comprises a plurality of sensors for detecting air pollutants which may be causes of asthma. The air quality measuring device 130 may be portable and may be carried with the patient. The user device 120 and the air quality measuring device 130 are connected to a network 140, which may be for example the internet.

A monitoring and analysis server 150 is connected to the network 140 and receives data from the user device 120 and the air quality measuring device 130 over the network 140. The monitoring and analysis server 150 may implement a cloud platform for monitoring and analyzing the breathing patterns of the patient over time. For example, the monitoring and analysis server 150 may assimilate and evaluate data to support a symptom diary for the patient.

A physician terminal 160 is coupled to the monitoring and analysis server 150 and allows a doctor to access data for the patient, for example to review inhaler usage or breathing patterns of the patient.

Figures 2a and 2b show an earpiece breath monitoring device according to an embodiment of the present invention. The earpiece breath monitoring device 200 comprises two microphones. The signals from the two microphones are used to isolate the breathing sounds of a patient from other noise sounds. As shown in Figure 2a, the earpiece breath monitoring device 200 comprises a housing 210 which has a cylindrical portion 212 that is sized to fit into the ear canal of the patient. A soft rubber bush 214 is arranged over the cylindrical portion 212. A cable outlet 216 extends downward from the housing 210. The earpiece breath monitoring device 200 comprises an exterior microphone 220 arranged within the housing 200 to collect sound signals outside the ear of the patient. The earpiece breath monitoring device 200 shown in Figure 2a is intended to be inserted into the right ear of a patient with the cable outlet 216 facing downwards. Thus, the exterior microphone 220 will face backwards towards the outer ear or pinna of the patient when the earpiece breath monitoring device 200 is in use. This minimizes the breathing sounds captured by the exterior microphone 220 because it faces away from the patient’s mouth. Thus the exterior microphone 220 captures a relatively large amount of exterior noise and a relatively small amount of the patient’s breathing sound. Therefore the efficiency of isolation of the patient’s breathing sounds by combining the signal captured by the two microphones is maximized.

Figure 2b shows a view of the earpiece breath monitoring device 200 with the soft rubber bush 214 removed. As shown in Figure 2b, the cylindrical portion 212 comprises a ridge 218 to hold the soft rubber bush in place. An in-ear microphone 230 is located with the cylindrical portion 212 to capture sounds within the patient’s ear canal.

As described above, the in-ear microphone 230 captures sounds from within the patient’s inner ear and the exterior microphone 220 captures sounds outside the patient’s ear. In embodiments of the present invention the signals captured by the two microphones are used to isolate the patient’s breathing sounds from other noise sounds. This process is described in below with reference to Figures 3a and 3b. Figures 3a and 3b illustrate a processing of signals captured by an earpiece breath monitoring device according to an embodiment of the present invention. As shown in Figure 3a, a breathing sound signal 310 is captured by the in-ear microphone 230. The in-ear microphone 230 also captures a noise sound signal 320. The exterior microphone 220 captures the noise sound signal 320. The outputs from the in-ear microphone 230 and the external microphone 220 are provided to a processing module 330 which isolates the breathing sound signal 310 by applying a noise cancellation algorithm such as the normalized least mean squared (NLMS) algorithm.

As shown in Figure 3b, the in-ear signal 315 received by the in-ear microphone 230 is a combination of the breathing sound signal 310 and the noise sound signal 320. The exterior signal 325 received by the exterior microphone 220 is the noise sound signal 320 alone. The processing module 330 applies a noise cancellation algorithm to reduce the noise in the in-ear signal 315 to provide a filtered breathing sound signal 340. This filtered breathing sound signal 340 is used in the later processing to analyze the breathing of the patient and / or to control the game.

Figure 4 shows a block diagram of an earpiece breath monitoring device according to an embodiment of the present invention. The earpiece breath monitoring device 400 shown in Figure 4 comprises two earpieces: a monitoring earpiece 410 which comprises two microphones as described above with reference to Figures 2a and 2b, and a speaker earpiece 420 which is configured to produce sound in the same manner as a conventional earpiece. The monitoring earpiece 410 and the speaker earpiece 420 are connected to a processing and interface component 430. The processing an interface component 430 comprises a processing module 432 and a communication interface 434. The processing module 432 is configured to process the signals received from the monitoring earpiece 410. The communication interface 434 allows the earpiece breath monitoring device 400 to send and receive signals from device such as a smart phone or tablet computing device. The communication interface 434 may be implemented as a wireless interface such as a Bluetooth interface. Alternatively, the communication interface 434 may be implemented as a wired interface such as an analogue interface provided by a 3.5mm audio jack or a digital interface such as a universal serial bus (USB) interface. The processing carried out by the processing module 432 comprises a noise reduction algorithm such as that described above with reference to Figures 3a and 3b. The communication interface 434 may transmit a filtered breathing sound signal to a user device for processing and may receive indications of sound signals from the user device for transmission to the user through the speaker earpiece 420.

Figure 5 shows a block diagram of a user device for processing breathing sounds of a user according to an embodiment of the present invention. The user device 500 may be for example a smart phone device, tablet computing device, or other computing device. The user device 500 comprises a processor 510, a communication interface, a wireless network interface 530, a display 540 and a program storage 550. The processor 510 is a central processing unit (CPU) which is capable of executing computer executable instructions. The communication interface 520 allows communication with devices such as the earpiece breath monitoring device described above. In some embodiments, the communication interface 520 may be implemented as a wireless interface such as a Bluetooth interface. Alternatively, the communication interface may be a wired interface such as an analogue interface provided by a 3.5 mm audio jack or a digital interface such as a universal serial bus (USB) interface. The wireless network interface 530 allows the user device 500 to connect to wireless networks and thereby connect with the internet through a wireless router.

The program storage 550 stores computer program instructions which may be in the form of computer program or application which can be executed by the processor 510. The computer program storage 550 stores a sound classification module 552, a breath analysis module 554 and a game module 556. The sound classification module comprises computer program instructions for classifying sounds received from the earpiece breath monitoring. The sounds may be classified as an inhale breathing sound, an exhale breathing sound, an inhale using an inhaler, or noise sounds. The sound classification module 552 may also comprise computer program instructions for identifying asthma symptoms and events such as coughing, wheezing and other breathing related sounds. The breath analysis module 554 comprises computer program instructions for analyzing the breathing sounds classified by the sound classification module 552. For example, the breath analysis module 554 may comprise computer program instructions for determining the length of time of breathing actions of the patient, such as the length of time of an exhale or inhale. The breath analysis module may comprise computer program instructions for determining a dose of medicine or drug inhaled by the patient using an inhaler. The game module 556 comprises computer program instructions for a breathing based games. The games may involve displaying a character or other object on the display 540 of the user device 500. From the analysis of breathing signals of the patient determined by the breath analysis module 554, the movement of the character may be controlled. For example, by breathing in, the patient controls the character to move upwards and by breathing out the patient controls the character to move downwards. Thus, by playing the game, a child can learn to control their breathing.

There are two main benefits of using games. Firstly, games can motivate the patients, particularly children, to use inhalers properly. There are standard inhaler breathing techniques and the games may be designed to simulate those breathing techniques. Secondly, the games may simulate recommended breathing exercises.

For example, doctors recommend a breathing exercise called belly breathing (diaphragm breathing). It helps to use diaphragm effectively for breathing. In diaphragm breathing, patient need to take a slow deep breath and exhale slowly through the nose. Patient needs to stretch the stomach rather than the chest. The deep breath can be simulated through the games. When patient is using a metered dose inhaler, patient needs to take a deep breath in order to take medication into the mouth. Then he/she needs to hold the breath for about 10 secs and slowly breath out. This technique can be simulated via our games while using the inhalers. Figure 6 is a flowchart showing a method of breath monitoring according to an embodiment of the present invention. The method 600 shown in Figure 6 is carried out on the user device 500 shown in Figure 5.

In step 602, the communication interface 520 of the user device 500 receives an indication of a breath sound signal from the earpiece breath monitoring device. As described above, a noise reduction algorithm is applied by the earpiece breath monitoring device before the indication of the breath sound signal is transmitted to the user device 500. In step 604, the sound classification module 552 of the user device 500 classifies sounds in the breath sound signal. The classification module 552 may classify sounds as breath sounds or noise sounds. It is noted that while the processing carried out on the earpiece breath monitoring device will remove some of the non breath sounds from the capture signals, some noise signals may still remain. For example, the voice of the patient travels through facial muscles and thus cannot be filtered out by the noise reduction algorithm implemented on the earpiece breath monitoring device. Additionally, some external noise may not be completely canceled out by the noise reduction. The sound classification module 552 may implement a machine learning based classification algorithm. In an embodiment, the sound classification module generates a mel-spectrogram from the input breath sound signal. Figure 7 shows an example of a mel-spectrogram.

Figure 7 shows a mel-spectrogram used in embodiments of the present invention to classify sounds. Such a mel-spectrogram may be used to train a deep neural network. Firstly, a deep neural network is built. The neural network is trained by breathing and non-breathing samples. Inputs for the model are mel-spectrograms of those thousands of sound samples. Deep neural network iteratively learns the features of mel-spectrogram which can be used to identify inhaling and exhaling sounds among others. By feeding mel-spectrogram as an input to the trained model, it can output whether the sound sample is an inhaling sound, an exhaling sound or another sound. As shown in Figure 7, portions of the sound are identified as an inhale sound 710 and an exhale sound 720.

Returning to Figure 6, in step 606, the breath analysis module 554 of the user device 500 determines a temporal length of breathing events classified in step 604. For example, the breath analysis module 554 may determine the length of inhale and / or exhale events.

Figure 8 shows an example of analysis of breathing sounds carried out in an embodiment of the present invention. As shown in Figure 8, parts of the sound are classified as an inhaling part 810 and an exhaling part 820 of a breathing cycle 830. The breath analysis module 554 determines the temporal length of the inhaling part 810 and the exhaling part 820.

In some embodiments, the breath analysis module 554 may determine a dose of medication inhaled by the patient. This may comprise determining length of an inhale event for an inhaler and from this determining the amount of medication inhaled. The dose may be determined from a cumulative count of the number of puffs taken on the inhaler. Inhalers generally have a specified amount of medication delivered in a single puff or actuation. Doctors recommend a minimum time window to keep inhaling in order to enter full medication into lungs. This differs according to inhaler type.

In an example, the medication delivered in each puff is d, the prescribed inhaling time period for a single puff is tr, the total number of puffs detected is n, and the actual recorded inhaling time periods are tai, ta2, ... ,ta _n .

Then in the i ^th actuation, if t _r >tai, the actual dose di inhaled will be: d

di—— x to-i

t _r

Otherwise, (if t _r £ tai) the actual dose di inhaled in the i ^th actuation will be:

di = d

The total dose Dtotai can then be calculated as:

D _totai ^¾ d + d ₂ H - l· d _n

Where: d-i— d (if t _r ^ tcif).

The breath analysis module may use the equations above to estimate the dose of medication inhaled by the patient.

In a set up phase, the patient may enter details of the inhaler used and the medication and this data may be stored on the user device 500 to allow calculation of the dose of medication. In step 608, the user device 500 generates an event notification. This may involve the game module 556 controlling an object displayed on the display 540 of the user device. Step 608 may also involve recording breathing events and sending an indication of the recorded events to the monitoring and analysis server 150 using the wireless network interface 530 of the user device 500.

Figure 9 is a block diagram showing an air quality measuring device according to an embodiment of the present invention. As shown in Figure 9, the air quality measuring device 130 comprises a particulate matter sensor r 910, a volatile organic compound sensor 920, a carbon dioxide sensor 930, a temperature sensor 940, a humidity sensor 950, and a communication interface 960. The air quality measuring device 130 is configured to detect and measure several air pollutants which are causes of asthma and also to monitor environmental conditions. The particulate matter sensor 910 may be a smoke particle detector which detects particles having a diameter of 2.5 microns or less. The communication interface 960 may be a wireless network interface and allows the outputs of the sensors to be transmitted to the monitoring an analysis server 150.

Figure 10 shows a breath monitoring system according to an embodiment of the present invention. The breath monitoring system 1000 shown in Figure 10 may include the features of the breath monitoring system 100 shown in Figure 1.

The breath monitoring system 1000 comprises a monitoring and analysis server 150. The monitoring and analysis server 150 is coupled via a network to a user device 120 and a physician terminal 160.

The breath monitoring and analysis system 1000 further comprises an air quality measuring device 130 which may be configured as shown in Figure 9. The air quality measuring device comprises a communication interface 960 and sensors 1032. The sensors 1032 may comprise the sensors shown in Figure 9. The user device 120 may be a mobile device and may include the functionality described above with reference to Figure 5. The user device 120 runs games 1022 and a patient app 1024. The games 1022 are designed to encourage the user to perform breathing exercises and use inhales properly. The patient app 1024 implements a symptom diary which stores information about asthma symptoms such as coughing, wheezing and asthma attacks. The symptom diary may also record inhaler usage and medication dosages. Additionally, the symptom diary may record breathing exercises carried out by the user and information on the results of such breathing exercises. The symptom diary may also record environmental data captured by the air quality measuring device 130.

Information captured by the symptom diary of the patient app 1024 is sent to the breath monitoring and analysis server 150 which stores the data in a database 1052. The breath monitoring and analysis server 150 may execute machine learning algorithms to predict asthma attacks from the patient diary data. In some embodiments, the breath monitoring and analysis server 150 may receive environmental data directly from the air quality measuring device 130. The physician terminal 160 is coupled to the breath monitoring and analysis server 150 and allows a doctor to access and analyze patient data using a doctor app 1062 which runs on the physician terminal.

The patient app 1024 may send the symptom diary data to the breath monitoring and analysis server 150 on a daily basis. The symptom diary data may include number of puffs of medication inhaled, time spent on breathing exercises and other data. The data is collected by the breath monitoring and analysis server 150 and stored in the database 1052. A doctor can view the symptom diary data using the doctor app 1062 running on the physician terminal 160. The symptom diary data may be protected and encrypted to ensure that access is limited to a doctor or a group of doctors authorized to access the patient data. This invention facilitates enhanced administration of specific inhaler-based medicines for asthma simultaneously having the ability to assimilate and evaluate data to support a symptom diary, identification of preventive measures.

Whilst the foregoing description has described exemplary embodiments, it will be understood by those skilled in the art that many variations of the embodiments can be made within the scope and spirit of the present invention.