TECHNICAL FIELD:

The invention relates to a support and rehabilitation device consisting of a camera, a microprocessor system, a communication unit, an oscillator unit that creates the necessary waveform for skin stimulation, and a conductor array components for prosopagnosia (face blindness) patients to distinguish faces by enabling the image taken with the camera to be recognized with artificial intelligence aided face recognition algorithm, and enabling patients to distinguish faces through their skin with electrical stimulation by generating separate signals for each face, and also by using it for rehabilitation purposes to create new pathways in the brain by causing neural plasticity and thus restoring the facial recognition function.

PRIOR ART

Prosopagnosia, also called face blindness, comes from the Greek words prosopon (face) and agnosia (non-knowledge). It is a cognitive disorder in which the person cannot recognize faces, including their own. There is no problem in the intellectual level of people with this disorder. The term refers to a condition that initially follows acute brain injury (acquired face blindness), but there is also a congenital or developmental form of the disorder, with a prevalence rate of 2.5% within all cases (Gruter, Gruter, Carbon, 2008). The specific brain area usually associated with prosopagnosia is the fusiform gyrus, located in the temporal lobe, which is activated particularly in response to faces. Thanks to the functionality of the fusiform gyrus, complex details of faces are recognized in greater detail than inanimate objects. The fusiform gyrus, located in the right hemisphere of the cerebral hemispheres mentioned above, is more active in face recognition. It is not yet known whether the fusiform gyrus is specific only for the recognition of human faces. Lesion-induced prosopagnosia results from damage to the occipito-temporal lobe and is mostly found in adults. Although there have been several attempts for improvement, no therapy has produced a definitive and permanent solution. In the current literature, there are applications with face recognition systems that display the name of the person, to whom the face belongs, on the screens, or devices that make audio notifications, but these systems do not have the capability to restore face recognition ability by creating new pathways in the brain.

BRIEF DESCRIPTION OF THE INVENTION:

Thanks to the invention, prosopagnosia patients with face recognition disorder will both be able to distinguish faces and be rehabilitated as life goes on. Thus, they will be able to alleviate the problems they face in maintaining their daily lives. Since the device is portable and wearable, instant data flow will be possible, electrical stimulations instead of audio stimulations, and additionally using mobile phone messages, if desired, will prevent possible interaction confusions with the environment. In the later stages of rehabilitation, the brain will begin to analyze facial details thanks to electrical stimulation, and daily life will be continued by the patient with the device until the device is no longer needed.

The microprocessor system, which recognizes the images taken by the camera thanks to the artificial intelligence-aided algorithm embedded in it, delivers the electrical stimulation, which is predetermined for each face and taught to the patient, to the patient's skin. The binding of image and electrical stimulation increases the activity in the visual cortex region of the brain, creates discrimination impulses and the treatment process begins.

The proposed invention is the first device developed for the treatment of patients with prosopagnosia. Unlike the existing ones, it is both a supportive and therapeutic system. No other device is capable of rehabilitating.

LIST OF FIGURES:

Figure 1. Contact Unit of Stimulator Device Working with Artificial Intelligence Aided Sensory Substitution Method for Prosopagnosia Patients

Figure 2. Block Diagram of Artificial Intelligence Aided Sensory Substitution Stimulator Device for Prosopagnosia Patients

Figure 3. Flow Chart of Artificial Intelligence Aided Sensory Substitution Stimulator Device for Prosopagnosia Patients THE EQUIVALENTS OF THE NUMBERS USED IN THE FIGURES:

1. Communication Unit

1.1. USB

1.2. Ethernet

1.3. Bluetooth

2. Microprocessor

3. Camera

4. Stimulator

4.1. Signal Generator

4.2. Power Amplifiers

4.3. DEMUX

5. Contact Unit

DETAILED DESCRIPTION OF THE INVENTION

The invention is a sensory substitution device with facial recognition. Sensory substitution devices consist of three main components. These are the camera (3) that detects the image from the environment, the stimulator (4) that contacts the user and transmits visual information to the user's skin with electrical stimuli via contact unit (5), and a communication unit with a microprocessor (2), where image processing and face recognition algorithms executed, that enables the communication between the camera (3) and the stimulator (4), creates the appropriate stimulation pattern, and interacts with a mobile phone or personal computer. Communication unit (1), contains USB (1.1), Ethernet (1.2) and Bluetooth (1.3). In the invention, the images detected by the camera (3) are transferred to the microprocessor (2) unit. The faces detected by the face recognition algorithm written in the microprocessor (2) unit were defined, named and a separate signal pattern was created for each face. This signal generated is delivered to the patient via the stimulator (4), which is in contact with the patient through the contact unit (5), and it is ensured that they can distinguish faces through the incoming signals. Stimulator (4) contains signal generator (4.1), power amplifiers (4.2), and 1x100 DEMUX (4.3). At the same time, the name tags of the face in the image are sent to the mobile phone or personal computer via Bluetooth (1.3), providing the necessary notifications for the patient's adaptation period. First of all, a stimulator (4) circuit is designed for this device. The designed circuit was implemented and produced on an organic PCB suitable for the human contact unit (5). Our stimulator (4), originally designed as a 10 x 10 matrix, provides an image with a resolution of 100 pixels. Higher resolutions are possible.

The contact unit (5) produced is double-layered and has a length of 104 mm and a width of 76 mm. PCB is a highly flexible product with a thickness of 0.6 mm. PCB is a highly flexible product with a thickness of 0.6 mm. It came in the form of a 10 x 10 matrix that will provide stimulation to the user after production.

This device stores a 100-bit pattern for recognized persons and applies the pattern of recognized persons to the patient as long as the relevant person is in the image. In addition, the system ensures that the stimulator always gives the same patterns to the patient, even if it comes from photo and video sequences. In this way, after a short training with the device, the patient can distinguish and recognize the faces detected by the device camera during the normal course of life, with the photos of his family and those he wants introduced to the system.

Facial recognition systems are very advanced and very useful systems as a result of the advancement of artificial intelligence methods. Today, although convolutional neural networks are particularly successful in image recognition, they have implementations that require systems with powerful processors and large memories. For the proposed portable system, it is not possible to achieve this with high performance. For this reason, face detection, segmentation, and recognition processes are carried out with complex artificial intelligence and face recognition algorithms that can be applied with embedded hardware.

Facial recognition systems work in two stages. In the first step, it is determined whether there is a face in the image and if there is, where this face is. Then, this detected face is segmented by the recognition algorithm and compared with the tagged faces in the database, and then it is decided who it belongs to.

Face detection is the first step. A very efficient object detection method, Haar classifier is used to achieve this. It is a machine learning based method.

Initially, the algorithm needs to be trained, and a large number of positive (with faces) and negative (without faces) images required for this training. Then feature extraction is done. If all Haar properties like location, scale, type are taken into account, we get more than 160,000 attributes for a 24 x 24 image, but this is a huge number for processors and this must be reduced. The AdaBoost algorithm is used in order to avoid excessive feature instances and to extract enough features that are useful to us.

AdaBoost algorithm can be used with many other types of learning algorithms to improve performance. The output of other learning algorithms (weak learners) is combined into a weighted sum representing the final output of the reinforced classifier. AdaBoost is used to remove unnecessary features and select only relevant features. AdaBoost is used to remove unnecessary features and select only relevant features. For example; a vertical edge is a relevant feature when performing nose detection, but it won't work for lips. With AdaBoost, parameters from these unrelated features are disabled, reducing more than 160,000 parameters in a 24 x 24 image down to 2500.

The AdaBoost algorithm tries to separate the face and non-face objects in the image in order to detect faces, i.e., it performs a classification process. For this, it starts by assigning equal weight values to the points in the first iteration and constructs the best possible classifier. In the second iteration, the weights of the misclassified points in this first iteration are changed to be equal to the (total correct weight/total incorrect weight) x first weight formula, and thus their total correct weights and total incorrect weights are equalized. Each of these iterations constitutes weak classifiers. For the final result, they are summed up. The higher the number of iterations, the higher the final success rate of the classifier will be. Let us explain this algorithm with an example. Let there be an environment with five red and five blue dots. The AdaBoost algorithm starts the classification process by giving each point a weight value of 1 in the first iteration.

At the end of the first iteration, three blue dots were classified as red. We need to re-equalize the total weights of the incorrectly classified dots with the total weights of the correctly classified dots. Therefore, we multiply the initial weights of the blue dots by the ratio of the total number of correctly classified dots to the total number of incorrectly classified dots. The initial weight value is 1, the total weight of the correct dots is 7, the total weight of the wrong dots is 3. The value 2.33 resulting from the 1*(7/3) operation is assigned as the new weight value of the incorrectly classified dots. The weights of the correctly classified dots are not changed and the second iteration is started with these weight values.

At the end of the second iteration, the classification process becomes as seen in Table 1. This time one red and two blue dots are in the wrong classes. The formula described above is applied again and the new weight values of the three incorrectly classified dots are calculated as 3.66. At the end of the third and final iteration, we obtain three sets of classified dots. These three are the weak learners mentioned above. Now it's time to combine these weak learners. All classes formed as a result of 3 iterations are taken into account while doing this combining process. When making this calculation, the log (correct class/wrong class) values of each class are used.

Table 1 - Combination Coefficients of Each Region

We can examine the methods used for face recognition in two classes as traditional and modern methods. EigenFaces (Original Faces), FisherFaces, and LBPH (Local Binary Pattern Histogram) algorithms are the most common ones among the traditional methods that detect geometric features of faces and compare them with faces in the database.

Modern methods used in face recognition include machine learning and deep learning techniques. In this method, the algorithm fed with new images goes through the learning process and starts to recognize faces better over time. Once a deep learning algorithm has learned enough patterns or features from the data, it will be able to extract features of a digital image that it has not seen before, and will be able to recognize who owns the image by comparing these samples or features with previous images stored in the database.

In this device, LBPH (Local Binary Pattern Histogram), one of the face recognition methods mentioned above, is used. The LBPH method was used because the deep learning method requires high processing power just like convolutional deep learning artificial neural networks. The LBPH algorithm makes classification using histogram values. Like other classification algorithms, LBPH must also be trained with visual data. Our faces are suitable for classification by LBPH as they consist of micro patterns. LBPH analyzes images in 3*3 matrices and attaches special importance to the value in the centre of this matrix. The pixel values in the centre are compared with the neighbouring pixel values of this centre. Pixels larger than the centre value are set to 1 , and smaller pixels to 0. Then, the binary values of each block are converted to a histogram by converting them to decimal radix. Finally, these block histograms are combined to form a single feature vector for an image containing all the features we are interested in.

Table 2 - Conversion of Pixel Values to Binary Format and Histogram Value in LBPH

In Figure 2, the block diagram of the device is shown. Here, the faces detected by the camera (3) are recognized by the face recognition algorithm embedded in the microprocessor (2) unit, and the stimulation of the recognized face is transferred to the stimulator (4) in order to be delivered to the patient's skin. Distinguishing faces with the stimulator (4) initially requires a certain amount of time. For this, it is also possible to send the information about the owner of the face to the user's mobile phone in text form via Bluetooth (1.3) available in the communication unit (1) in order for him to recognize the faces that he was not able to distinguish in the first place. In addition, all settings other than the standard operation of the device can be made via a bluetooth (1.3) connection with a mobile phone or personal computer. The device is the first system developed for the treatment of prosopagnosia patients.