Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
THERAPY SYSTEM AND DEVICE FOR SPEECH ARTICULATION
Document Type and Number:
WIPO Patent Application WO/2002/039423
Kind Code:
A1
Abstract:
The invention is a system for people with speech disorders causing the inability to engage in a fluent articulation of speech. With the increase in access to multimedia computers, speech training can be made available to patients with no continuous assistance required from speech therapists. Moreover, as computer hardware becomes more and more miniaturised, some speech training or correction algorithms can now be implemented into small wearable devices that patients can use in everyday life. The invention defines some of the new applications of computer technology to speech articulation correction and training, particularly useful in cases of stuttering. The system described in the invention can send data associated with the patient profile to a miniaturised speech therapy device working in real time, namely CDSTU - Compact Digital Speech Therapy Unit. This devie is implementing real-time algorithms for the auditory feedback loop modification which are controlled by the personal computer and offers also the possibility of recording, transmitting and playing back the speech material scheduled to the training. Also, a system for computer-based interactive speech therapy to people with speech disorders is claimed.

Inventors:
CZYZEWSKI ANDRZEJ (PL)
SKARZYNSKI HENRYK (PL)
KOSTEK-CZYZEWSKA BOZENA (PL)
Application Number:
PCT/PL2000/000081
Publication Date:
May 16, 2002
Filing Date:
November 13, 2000
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
CZYZEWSKI ANDRZEJ (PL)
SKARZYNSKI HENRYK (PL)
KOSTEK CZYZEWSKA BOZENA (PL)
International Classes:
G09B5/00; G09B7/00; G09B19/04; G10L15/22; (IPC1-7): G10L11/00; A61F5/58; G10L21/00
Domestic Patent References:
WO1994000085A11994-01-06
Foreign References:
US5794203A1998-08-11
US4662847A1987-05-05
US4421488A1983-12-20
Other References:
AWAD S S: "THE APPLICATION OF DIGITAL SPEECH PROCESSING TO STUTTERING THERAPY", IEEE INSTRUMENTATION AND MEASUREMENT TECHNOLOGY CONFERENCE,XX,XX, vol. 2, 19 May 1997 (1997-05-19), pages 1361 - 1367, XP002920499
AWAD S S: "COMPUTER ASSISTED TREATED FOR MOTOR SPEECH DISORDERS", VENICE, IT, MAY 24 - 26, 1999,NEW YORK, NY: IEEE,US, 24 May 1999 (1999-05-24), pages 595 - 600, XP000871775, ISBN: 0-7803-5277-7
BAHILL A T ET AL: "How the testing techniques for a decision support system changed over eight years", CONFERENCE PROCEEDINGS. 1993 INTERNATIONAL CONFERENCE ON SYSTEMS, MAN AND CYBERNETICS. SYSTEMS ENGINEERING IN THE SERVICE OF HUMANS (CAT. NO.93CH3242-5), PROCEEDINGS OF IEEE SYSTEMS MAN AND CYBERNETICS CONFERENCE - SMC, LE TOUQUET, FRANCE, 17-20 OCT., 1993, New York, NY, USA, IEEE, USA, pages 12 - 17 vol.5, XP002168647, ISBN: 0-7803-0911-1
TERRY BAHILL A ET AL: "KNOWLEDGE EXTRACTION CHANGES THE WAY AN EXPERT THINKS", PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON SYSTEMS, MAN, AND CYBERNETICS,US,NEW YORK, IEEE, vol. -, 18 October 1992 (1992-10-18), pages 917 - 921, XP000366598, ISBN: 0-7803-0720-8
Attorney, Agent or Firm:
Czyzewski, Andrzej (Gdynia, PL)
Download PDF:
Claims:
Claims
1. A computerbased interactive speech therapy service to people with speech disorders installed on a multimedia computer with a sound card and a headset or on a network server to which the multimedia computer is connected, the service performing acquisition of descriptive data about the patient's speech disorder and automatically detecting defects of articulation on the basis of voice analysis of the patient said system making automatic diagnosis of the speech disorder and providing information about adequate means of therapy or schedule of necessary speech training.
2. A computerbased interactive speech therapy service according to claim 1, wherein training speech articulation is organised using a working in realtime computer software for modifying signals in the auditory feedback loop which is between connected to the computer's sound port microphone used by the patient and connected to the same computer's sound port earphones used by the patient.
3. A stuttered speech processing software working in realtime, the software implemented to a computer with a sound card and a known headset connected to it said software causing a delay of speech signal received from headset's microphone or causing reverberation or frequency shifting or generating the chorus effect or masking of speech signal with noise or with another sound.
4. A microcomputer device for the realtime impaired speech processing, the device communicating with a personal computer through a twoway digital interface said device having its signal input connected to the patient's microphone or laryngophone and its signal output connected to patient's headphones or loudspeakers said device being able to cause speech signal delaying or reverberation or frequency shifting or generating the chorus effect or masking of speech signal with noise or with another sound according to settings received by said device from the personal computer.
5. A device as claimed in claim 4, wherein previously stored speech samples can be received from the personal computer through the serial interface by the said microcomputer device and patient's speech samples can be stored in the said microcomputer device memory or transmitted via said microcomputer device interface to the personal computer for speech play back or speech analysis. Training System and Device for Speech Articulation and Speech Understanding Abstract The invention is a system for people with speech disorders causing the inability to engage in a fluent articulation of speech. With the increase in access to multimedia computers, speech training can be made available to patients with no continuous assistance required from speech therapists. Moreover, as computer hardware becomes more and more miniaturised, some speech training or correction algorithms can now be implemented into small wearable devices that patients can use in everyday life. The invention defines some of the new applications of computer technology to speech articulation correction and training, particularly useful in cases of stuttering. The system described in the invention can send data associated with the patient profile to a miniaturised speech therapy device working in real time, namely CDSTUCompact Digital Speech Therapy Unit. This device is implementing realtime algorithms for the auditory feedback loop modification which are controlled by the personal computer and offers also the possibility of recording, transmitting and playing back the speech material scheduled to the training. 5 patent claims.
Description:
THERAPY SYSTEM AND DEVICE FOR SPEECH ARTICULATION Background of the invention The invention is a system for people with speech disorders causing the inability to engage in a fluent articulation of speech. With the increase in access to multimedia computers, speech training can be made available to patients with no continuous assistance required from speech therapists. Moreover, as computer hardware becomes more and more miniaturised, some speech training or correction algorithms can now be implemented into small wearable devices that patients can use in everyday life. The invention defines some of the new applications of computer technology to speech articulation correction, particularly useful in cases of stuttering.

There are numerous electronic speech correction applications in which analogue processing circuitry is used for processing the acoustical signal. The Russian patent description No. 66.680 covers the electronic echo-corrector of speech which consists of the following elements linked in series: microphone, input amplifier, system for echo generation, output amplifier and electroacoustic transducer in the form of earphones. The system, according to the invention, uses a specially structured subsystem for echo generation, built from twenty inductor-capacitor blocks linked in series with individual correcting amplifiers. The speech signal delay reached by the system ranges from 0 to 120 ms at discrete values. The system has to use a large number of inductor-capacitor-based filters and passive intermediate elements. Hence the non-uniform transfer characteristics at various values of the delay coefficient. A device based on the Russian invention does not qualify for miniaturisation or digitisation.

Another known and commercially used echo-reverberator speech corrector for stutterers is covered in the Polish patent description No. 130.362. This device consists of a microphone linked with the input of the input amplifier, output of which is connected to the input of a known echo and reverberation circuit with further output to the amplifier and speaker. What is important in this invention is that the circuit for echo and reverberation generation contains an input low-pass filter connected to an analogue-to-digital converter, and is then connected with a binary shift register, which in turn feeds the digital-to- analogue converter, connected via the output of the low-pass filter to the output of the amplifier. The input of the input amplifier and output of the output amplifier are connected with a telephone antilocal splitting system which enables connection with the telephone network. The solution described in this patent seems technologically outdated.

Nevertheless, its uniqueness lies in the proposed access to the echo producing device via a public telephone system. Even though the echo-reverberator speech corrector system meets the assumed functions, it does have some drawbacks resulting from the fact that it has to be used via a telephone line. This can be both costly and inconvenient as it leaves the telephone line engaged for long periods of time.

The Polish patent No. 168523 describes a method of digital correction of speech for stutterers through electronic signal processing in an acoustic feedback loop that splits the acoustic signal obtained from the stuttering person via a microphone or contact microphone into two frequency bands from which the bandwidth with higher frequencies is passed in the analogue domain after amplification to the output of an electroacoustic transducer, and the lower bandwidth is sampled and processed with an analogue-to-digital converter with a regulated sampling frequency. The signal obtained in this fashion is digitally shifted downwards and simultaneously delayed. The lower bandwidth of frequencies processed in this way is then mixed with a high-pass analogue signal and, as an output signal, is regulated as to the amplitude and then sent to the electroacoustic transducer, for example earphones or a bone vibrator. The solution described in this patent is conceptually advanced, yet technologically outdated mainly because of the rapid progress of digital technology and development of new architectures of computing systems and communication devices. Moreover, as results from research experiments, transmission of any portion of unchanged signals can diminish the effect of auditory loop modification on speech fluency.

In the US patent No. 4,421,488 entitled"Aid for curing or mitigating stammering" an aid for curing or mitigating stammering is described which comprises two earphones and a microphone connected with an amplifier, and a delay unit in which the signal from the amplifier to one earphone is delayed in relation to the signal which is transmitted to the other earphone. Numerous research results show that this solution is not optimal in most stuttering cases, because the effect of increased fluency is better when the original speech signal is masked by the echo. Transmitting the original signal to one of the patient's ears can diminish the expected effect of auditory feedback modification on speech fluency.

Similarly, in the US patent No. 4,662,847 a device and method for transmitting electronic speech signals from a user through two paths is described, one of which is synchronous and the other asynchronous. The synchronous speech signal is transmitted while the user is normally speaking and, at any pause in phonation, the device switches to the asynchronous path and transmits speech in a delayed auditory feedback mode and continues to do so until a pre-selected change occurs in the user's speech. The basic problem connected with the practical implementation of this device is automatic detection of stuttering artefacts which are related to pausing speech. That is because pausing speech production can also be normal and is not always related to stuttering.

The US patent No. 5,940,798 presents a belief according to which the speech motor control system of a stutterer has set the Kalman gain too high, thereby creating an unstable control system that in turn causes stuttering. A feedback modifier is then proposed backing the stutterer's speech to the stutterer hearing system employing known methods of signal processing which were previously applied to this task, claiming that the values of delay and frequency transposition should be adequately small. Practically, the only new input from this patent description is the theoretical interpretation of the speech motor behaviour based on the Kalman filtration concept.

Similarly, the US patent No. 5,794,203 and its innovation are more theoretical than practical. In the patent, methods such as echo generation and frequency shifting are considered as the biofeedback system for speech disorders.

The US patent No. 5,478,304 also presents the concept of delaying the voice signal, this time, however, it is related theoretically to the natural time shift of acoustic waves as they are transmitted through the patient's tissues and skull bones.

The US patent No. 4,685,448 introduces a specific solution to the speech acquisition system that is transmitted to an electro-mechanical transducer attached to the outer surface of the laryngeal or glottis region of the neck of a user and is then delayed like in many other speech correctors for stutterers.

The authors of the US patent No. 5,961,443 describe the application of earlier methods of correcting stuttered speech, namely echo and frequency shifting in the auditory feedback loop, but they claim that a wearable compact device they have designed, configured for positioning in or adjacent to the ear canal of an individual may be treated as a therapeutic device. The patent does not offer any major conceptual innovation, because the general idea of a small therapeutic device was published earlier in research literature (for example: Czyzewski A., Kostek B., Roland-Mieszkowski M.,"Digital speech aid to decrease stuttering-clinical results and patients'reactions". Abstr. in Journal of Fluency Disorders 1994, vol. 19, No. 3, pp. 1-5). Nevertheless, the patent referred to, specifies some details, primarily mechanical, of a compact wearable stuttered speech corrector design.

All the above mentioned patented solutions do not address the application of multimedia computers or today's communication technology to the problem of speech therapy. On the other hand, the existing computer programmes for stutterers do not implement adequate real-time speech processing methods for the correction of disordered speech. For example, the software called"Dr. Fluency"TM is a computerised stuttering treatment programme offered by Speech Therapy Systems Ltd. (www. dfluency. com) and Speak Gentle @ is a programme published in 1998 by H. Walden and W. Egger. These programmes offer various speech exercises for stutterers, generate training material, however, they do not provide any means for a direct electronic analysis of speech or its correction through modification of the auditory feedback loop.

The invention according to the present description offers a more extensive and better use of personal computers by providing a fully integrated system of speech therapy employing real-time signal processing and network communication features of contemporary computers.

Disclosure of the Invention The invention offers a new type of interactive speech therapy service to people with speech disorders. The service uses state-of-the-art computer technology and telecommunications. In particular, the computer software designed according to the concept described in the invention, can be installed on local hosts or on computer network stations. The software provides vital assistance to a speech therapist or patient in the area of diagnosis and rehabilitation of speech disorders, especially stuttering. Another function the system can easily perform is providing directed information to patients who have various speech disorders and problems with understanding speech.

The system in the invention is described using figures and a description to be provided in detail in the next section. The idea underlying the system is a programmed speech therapy training algorithm which consists of diagnostic tools and rehabilitation devices connected with it. The device is a programmable recorder and player or modifier of speech in the auditory feedback loop which can be realised on the basis of computer software or as a specialised hardware unit.

It is assumed that the user of the system has access to a typical multimedia computer with a sound card and a headset. Installed on the computer is a local or network software that meets the below functions. The first function the system described in the invention has to perform is data acquisition where information about the patient's medical history is collected. This is done through a system of electronic questionnaires. The next function is analysis of the speech signal articulated by the patient when prompted by the computer. The analysis is performed when the patient reads out fragments of a specially selected text displayed on the screen. To implement the function, the computer is provided with a speech analysis software. As speech is received via a microphone, the software detects errors and articulation defects. Next, the results of the electronic questionnaire and the patient's voice are analysed. Based on that the system automatically diagnoses any possible speech disorders and how strong they are. The system informs the patient who has been diagnosed with a potential speech disorder that they need to consult a speech therapy or phoniatry specialist to have the diagnosis confirmed. In particular, the patient- specialist contact can be facilitated using teleconference features which can be integrated into the system in question. The patient may contact a specialist and have the diagnosis confirmed. Then the patient may look up a therapy programme in the system's menu to match the disorder. The programme recommends the frequency and type of speech articulation or understanding training which the computer will help to set up. If the speech disorder involves stuttering, the essential part of the therapy is training the articulation using a programme for modifying signals in the auditory feedback loop which is between the microphone used by the patient and the earphones. In the invention, the signals in the loop can be modified by the computer or by external device. The software installed in the computer works in real time and causes a delay of the signal or reverberation or frequency shifting or generates the chorus effect or masking of speech with noise or another sound.

As an option, once selected and matched to the patient's needs, the signal modification algorithm settings can be sent from the computer to a miniaturised sound processor, as described in the next paragraph. The patient can use the processor together with a small microphone and earphones in everyday life, meaning away from the computer station as discussed earlier. In the case of patients who have trouble understanding speech, the external electronic device can also receive word material to train speech understanding.

The material can be a set of lexical units and structures which the patient should listen to systematically. The miniature external device can also be used to record the patient's speech which can then be transmitted to the computer for further playback and analysis to assess articulation.

The diagnostic and rehabilitation system significantly assists one in diagnosing speech disorders and problems with speech understanding. This feature is especially important for screening testing. Also, once the disorder has been diagnosed, the system in the invention helps to organise a speech therapy training in just a short time. In cases of stuttering, the system ensures a direct access to electronic correction tools which facilitate training and learning to speak fluently. The system also assists in electronic speech prosthesis fitting. In particular, the package of services described in the invention can be made available via the Internet and on typical personal computers which makes cost effective dissemination of the service possible. By offering a solution as described above, the system covered by the invention creates a new dimension of medical and telemedical speech therapy and phoniatry service.

Detailed Description of the Invention The electronic questionnaire in the integrated system for diagnosing and rehabilitating speech disorders as described in the invention, contains several to about a dozen questions about the occurrence of basic disorders that impair verbal communication.

Preferably, questions about the particular disorders are asked in an indirect manner, meaning that they pertain to everyday situations. Based on that conclusions are drawn on speech articulation and speech understanding problems or problems with writing down a text that is being dictated to the subject. The assumption is not that the patient, who can just as well be a child, has the necessary knowledge to be able to answer specialist questions. In this way, the system makes a preliminary test of how the speech articulation organs and kinesthetic speech control function. As an option, the computer system using a speech synthesis programme or pre-recorded sound can dictate fragments of a text which the subject is expected to input into the computer using the keyboard or a speaker- independent speech recognition system. Also the test that involves typing of a text being dictated can be repeated with the image of the person face reading it displayed on the computer screen. Thanks to that it is possible to compare the results of speech understanding with and without lip reading. This can lead to conclusions as to the role of the subject's visual speech centre. In this way, the data collected through the electronic questionnaire by the system described in the invention, can be supplemented with the results of verbal, auditory and auditory and visual tests all of which increases the probability of a correct diagnosis. The data, once logically processed by the computer using an algorithm produced by speech therapy and phoniatry specialists, and following the analysis of the results of the tests which use multimedia techniques, can then help to establish whether the subject falls under a risk group, in particular a group of people who potentially may have difficulty with communicating or have no such problems. In particular, the analysis of the data set acquired in the course of tests described in the invention, can be supported by decision algorithms from the soft computing area, such as neural networks, fuzzy logic, rough set method or genetic algorithms. Thanks to that the algorithm which learns on the basis of the data sets can automatically recognise the particular patient profiles.

Having recognised the patient profile using the system described in the invention, the system can then, as an option, refer the subject for consultation with a specialist to have further testing done in order to verify the automatic diagnosis. Next, the patient who has been diagnosed with speech problems is instructed on how to use the therapeutic tools that are available in the system. In particular, for each of the risk groups and each diagnosed speech disorder degree, the system has a different therapy procedure. Based on the procedure speech articulation or speech understanding exercises are programmed.

They way it works is that the computer displays a programmed material to be read out and at the same time activates speech analysis software. In the system described in the invention the material can either be displayed on the computer screen or using a known software based speech synthesis or speech play back. The role of the patient is to read out the displayed text or repeat utterances. The utterances can be received by a microphone connected to the computer's sound interface and analysed by a digital speech processing software.

The system described in the invention can also send data associated with the patient profile to a miniaturised speech therapy device working in real time, namely CDSTU-Compact Digital Speech Therapy Unit. The design of the miniaturised speech therapy device CDSTU and its operation are described on the basis of Fig. 1. The device consists of a microcomputer 1 which contains a known two-way sound interface 2.

Connected to its input is the microphone from the headset 3. Connected to its output are the earphones from the same headset. The subject can listen to the material via the earphones which are part of the headset 3 connected with the device 1. As an option, the device 1 in its internal memory can also record patient speech using the microphone from the headset 3 and send it to the computer 5 via the interface 4 for playback and analysis.

The device 1 communicates with the external computer 5 via a typical interface 4, e. g. a USB (Universal Serial Bus) which can also be used to send the settings regarding the modification of the auditory feedback loop. The system operates like this : the microcomputer device 1 through the two-way serial interface 4 receives data from an external computer 5 which in particular, could be a personal computer equipped with software that enables the sending of speech samples or programme settings. The programme installed in the CDSTU device 1 after disconnecting this device from the computer is able to processes the speech signal according to received algorithm whose role is to delay the speech signal or generate reverberation or chorus effect or perform frequency shifting up or down the spectrum scale or generate a masking noise at a given loudness level. Optionally, the microcomputer device 1 can receive speech signal samples for playback from computer 5 to which it is connected via the interface 4, or send to the computer 5 samples of patient speech recorded in the microcomputer device's memory 1 using the microphone from the headset 3 in order to carry out analyses employing the software installed on the computer 5 whose objective is to determine whether the recorded and transmitted speech was correctly articulated by the patient.

Brief description of tAle drawing Fig. 1 presents block diagram of the Compact Digital Speech Therapy Unit providing microcomputer speech therapy device and shows the connection between this device and the multimedia personal computer.