Technical Field

The present solution generally relates to smart learning platform enabling hybrid learning.

Background

Distance learning became a topic during a global pandemic forcing children to participate school from their homes. However, despite the pandemic, home school and distance learning have always been a target of interest to educate school-age children. Sometimes, distance learning would be desired for short periods of time, for example when a child is too sick to participate classroom learning, but well enough to follow lessons remotely. For others, remote learning can save time and therefore increase learning efficiency. For example, when a student lives on a distance from the school, the student doesn’t have to spend a long time travelling each day. Instead, the time spent in traveling can be used for learning. Each student has individual preferences in relation to their learning. Some of the students may learn a particular subject matter better if they can study the subject at their own time and pace, using an intelligent learning system that provides materials and exercises that best help them to learn.

Summary

The present embodiments are targeted to provide a solution by means of which hybrid learning is enabled. Hybrid learning offers the possibility to significantly save cost in providing education and at the same time improve academic outcomes.

Various aspects include a system, a method, an apparatus and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims. According to a first aspect, there is provided a smart learning system comprising a tracking component, an analysis component, and an interface for communications of data with one or more applications executed on user devices, wherein the smart learning system is configured to receive by the tracking component a first data set generated by one or more applications in connection with execution of said applications, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications; analyze, by the analysis component, the first data set to determine issues relating to the learning and/or motivation of the student; select, by the analysis component, an action to be applied as a response to a determined issue, wherein the action is one or more of the following: an automatic rectification of the issue at the student user device; an indication of the issue on a teacher user device; an indication of the issue on a student user device; a suggestion on a possible rectification on a student user device; a suggestion on a possible rectification on a teacher user device; an entry to a log for an analysis.

According to a second aspect, there is provided an apparatus comprising at least means for tracking, means for analysis and means for communications of data, wherein the means for tracking is configured to receive a first data set generated by one or more applications in connection with execution of said applications in said apparatus, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications; the means for analysis is configured to analyze the first data set to determine issues relating to the learning and/or motivation of the student; the means for communications is configured to deliver information on the determined issues to another device, and as a response from said another device, to receive an information on an action to be applied to the determined issue.

According to a third aspect, there is provided a method for a smart learning system having an interface for communications of data with one or more applications executed on user devices, the method comprising receiving by the tracking component a first data set generated by one or more applications in connection with execution of said applications, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications; analyzing, by the analysis component, the first data set to determine issues relating to the learning and/or motivation of the student; selecting, by the analysis component, an action to be applied as a response to a determined issue, wherein the action is an automatic rectification of the issue at the student user device; an indication of the issue on a teacher user device; an indication of the issue on a student user device; a suggestion on a possible rectification on a student user device; a suggestion on a possible rectification on a teacher user device; an entry to a log for an analysis.

According to a fourth aspect, there is provided a computer program product comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to receive by the tracking component a first data set generated by one or more applications in connection with execution of said applications, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications; analyze, by the analysis component, the first data set to determine issues relating to the learning and/or motivation of the student; select, by the analysis component, an action to be applied as a response to a determined issue, wherein the action is one or more of the following: an automatic rectification of the issue at the student user device; an indication of the issue on a teacher user device; an indication of the issue on a student user device; a suggestion on a possible rectification on a student user device; a suggestion on a possible rectification on a teacher user device; an entry to a log for an analysis.

According to an embodiment, the first data set also comprises data that has been captured by one or more recording applications on the user.

According to an embodiment, a second data set recorded from a classroom learning environment is received; the recorded second data set is analyzed to determine a learning phase; whereupon the analysis component is configured to use the second data set when analyzing the first data set.

According to an embodiment, a third data set generated by one or more applications on the teacher user device is received, and the third data set is delivered to the student user device. According to an embodiment, the tracking component is configured to track, based on the received first data set, also attentiveness and emotional state of a student.

According to an embodiment, the analysis component is configured to obtain history data and to use the history data when analyzing the first data set.

According to an embodiment, the analysis component is implemented as a machine learning algorithm.

According to an embodiment, the computer program product is embodied on a non-transitory computer readable medium.

Description of the Drawings

In the following, various embodiments will be described in more detail with reference to the appended drawings, in which

Fig. 1 shows a smart learning system according to an embodiment;

Fig. 2 shows an arrangement of devices according to an embodiment;

Fig. 3 shows an apparatus according to an embodiment;

Fig. 4 shows an example of a hybrid learning environment; and Fig. 5 is a flowchart illustrating a method according to an embodiment.

Description of Example Embodiments

The following description and drawings are illustrative and are not to be construed as unnecessarily limiting. The specific details are provided for a thorough understanding of the disclosure. Flowever, in certain instances, well- known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, reference to the same embodiment and such references mean at least one of the embodiments.

Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure.

The present embodiments are related to a smart learning system, also referred to as smart learning platform, an example of which is illustrated in Figure 1. The smart learning system is a digital tool for assisting learning in hybrid learning environment comprising face-to-face learning, i.e. classroom learning environment, and digital learning occurring online or offline. The smart learning system uses data from several sources to optimize the learning between these two learning environments.

The smart learning system 100 generally comprises a first component 100 for classroom learning and a second component 120 for distance learning. The second component 120 for distance learning can be utilized in both online and offline learning mode. Both the first and the second components comprise various computer modules to carry out functionalities of the smart learning system. The modules and their respective functionalities are discussed in more detailed manner later.

The smart learning system 100 may communicate with various client applications 99 through an interface 105. The client applications may be independent third-party applications that are used as learning and/or teaching platforms by students and teachers. One or more of such applications 99, or at least their respective user interfaces, may be located on a client device. The client device for a student comprises one or more learning applications/platforms or their interfaces. The client device for a teacher comprises one or more teaching applications/platforms or their interfaces. An application may be any educational application or a platform that is used for receiving schoolwork and educational material from a teacher user, and receiving completed homework and other school-related data from a student user. Said applications communicate with the smart learning system 100, whereupon the smart learning system 100 may analyze the learning curve of the student and to control and adjust the educational material of the teacher.

The data that is received from the applications may be processed by a machine learning (ML) component 130. The machine learning component 130 operates with the various modules of the smart learning system to generate intelligent decisions based on the received data according to the needs of said modules. The machine learning component may implement a machine learning algorithm, which may be based on deep learning. For example, a (deep) neural network - as an example of an machine learning algorithm - comprises an input layer and an output layer, and a plurality of hidden layers therein between. Each of the layers comprises units that are configured to implements an activation function based on an input it receives. The output of the activation function is forwarded to the units of the next layer, if the output exceeds a certain threshold.

The machine learning component 130 can be referred to as “global machine learning service”. In addition to the global machine learning service, there can be a ML algorithm located on a student device, and a ML algorithm located on teacher’s device. These ML algorithms are referred to as “local machine learning algorithms”. The operation of these algorithms is discussed later in this description.

The operation of any of the machine learning algorithms is based on a training, which is performed by using a training dataset comprising data that is representative of the actual data being received by the machine learning algorithm. The purpose of the training is to find parameters (also referred to as “weights”) by means of which the loss of the algorithm will be minimized (optimization). The training can occur in a supervised manner or unsupervised manner. In the context of smart learning, the data having been gathered through years can be used as a training data set. Such data comprises digital educational material; exams; assignments; historical data on students’ input to such exams and assignments; students’ progress on the educational material; students’ feedback on the lessons; etc. In addition to the historical data, also data that is gathered during the smart learning and/or during lessons, is continuously fed to the machine learning algorithm to enable continuous learning of the algorithm. The training dataset may comprise video and audio recordings of a student (near a computer or at the classroom), student device activity during class and corresponding labels (i.e., was the student focused at a particular time frame). Dataset may be anonymized in order to protect privacy of the students.

Figure 2 illustrates devices participating to the overall system operation. The smart learning platform with the first and second components can be arranged into a server 200. The smart learning platform is connectable by a teacher device 210 and a plurality of student devices 220. The server 200 may comprise, or be in connection also with, a global machine learning model 230 and a database 240. The teacher device 210 and the plurality of student devices 220 may communicate with the server 200 through first and second components (as discussed with reference to Figure 1). According to an embodiment, the teacher device 210 uses the first component for the classroom learning for carrying out the teaching in the classroom. According to an embodiment, the teacher device 210 uses the second component for distance learning for providing teaching over network. The student device 220 uses the second component when the student is participating the lesson online or offline. According to an embodiment, the student device 220 has a first component to be utilized when participating the lesson in the classroom.

A student device 220 can be a smart phone, a tablet device, a laptop computer a personal computer or any other similar device. A teacher device 210 may be a smart phone, a tablet device, a laptop computer, a personal computer or any similar device. In addition to the student and teacher devices, the system may comprise tracking devices, such as video and/or still cameras, and/or microphones, and/or audio recording means. The student device 220 is usable by a student while studying remote, but sometimes also when studying at the classroom. The student device 220 may comprise a local ML algorithm by means of which data from devices (watches, video camera, microphone, navigation tracker, etc.) is synchronized. The local ML algorithm is configured to locally determine how focused the respective student is at the moment, and to send statistics to the global ML service.

An example of a local ML algorithm is a CNN-based emotion detector comprising, for example, three fully-connected layers that are able output a single emotion score that indicates the emotion of a student. The emotion detector could have been trained with anonymized videos of students in a class, and finetuned with videos from home web-cameras of remote students. The emotion score of the local emotion detector can be fed with an activity data to a global ML service which is able to output a single engagement score for the student.

Another example of a local ML algorithm is a CNN-based performance detector. The performance detector is able to output a performance score for a student based on data that is input to the algorithm The input data may comprise speed of a student for completing an exercise, a correctness of the completed exercise, activity of the student. The performance scored will be utilized when determining the phase of the student when compared to other students.

The engagement score can be used alongside with the performance score of a student to see, whether a low performance of the student correlates with the engagement score, or whether the student is really trying but not able to achieve the results.

The teacher device 210 is also usable at the classroom environment, but the teacher is also able to use the teacher device 210 outside the classroom. The teacher device 210 may comprise a local ML algorithm to track students by using cameras/microphones installed in classroom, and to send statistic to the global ML service.

The tracking devices may be integrated to the student device and/or teacher device, but the system may contain independent tracking devices being installed in classrooms or student’s room at home.

Figure 3 illustrates an example of a user device 300. The user device can be a teacher device or a student device for the purposes of a system shown in Figure 2. The device 300 comprises a processor, a memory, a communication interface and a user interface. The memory comprises a computer program code for causing the device to carry out various functionalities. The memory may also contain other data concerning one or more of the following: teacher’s profile (age, gender, educational background, experience, PD (Professional Degree) history, work time management/follow-up); students’ profile (age, gender, educational history, special education needs, other medical history); school’s profile (country, location, size, curriculum, co-ed/segregated, learning environment specifications, specialized teachers); class’ profile (number of students, students, student-teacher ratio); planned learning activities (project topics, standards, subject-matter, level, duration, learning material, teachers, students); project based learning modules (SDGs (Sustainability Development Goals), standards, subjects, core competences, content areas, desired end product, assessment, differentiation, activities to reach the objectives, materials); learning activity (time, duration, action of the teacher/student); observation/assessment (time, duration, assessment result), hardware used for learning (how much, when, where); software used for learning (how much, when, where); self-evaluation data or everyone in the organization.

The communication interface enables wired or wireless short or long range communication with other devices. Examples of the communication networks comprises any data transfer technology, such WLAN (wireless local area network), wireless mobile networks of different generations (3G - 5G and forward); LAN (local area network); etc. Data transfer networks that are utilized by the smart learning system comprises the available or the future technologies, and therefore they are not discussed further in this specification.

The user interface comprises means for a user (e.g. a teacher or a student) to provide data to the system and to view data from the system. The user interface may be a graphical user interface being tailored for certain user roles. The user interface is able to show data retrieved from a database, and receive inputs from users. The user interface comprises graphical elements to be used for example in connection of accomplishing assignments or of requesting assistance. In addition, the user interface may display or play recordings made in the classroom.

The main aspect of the hybrid learning in which the teacher device and the student device and the smart learning platform operate, is the independence of time and of a physical place. In general, when time and place are taken into consideration, the learning may be

1 ) “classic learning” occurring at the same time, at same physical location;

2) “remote learning” occurring at the same time, at different physical locations; 3) “asynchronous learning” occurring at different times; at different physical locations;

4) a learning occurring at different times, at same physical location.

With respect to an idea of hybrid learning, options 2 and 3 come up. Therefore term “hybrid learning environment” refers to following learning situations:

1 ) a situation, where some of the students are located remotely from the class environment, and where some students are located in the class. Thus, the hybrid class environment combines both classroom learning and distance learning (“remote learning”) simultaneously.

2) a situation, where all students are self-learning at different locations and at different times (“asynchronous learning”).

These aspects create a few challenges relating to student monitoring, material selection, student co-operation, and student preferences.

Student monitoring is possible in a conventional, i.e., classic, class learning environment. In such situation, a teacher is able to monitor students to see how they participate and follow the lesson. The teacher is also able to react to problems, for example, when a student is being stuck. For the remote participants, i.e. students participating to distance learning online, the student monitoring is much harder, especially if the distance learning occurs at the same time with the classroom learning. The harder the monitoring becomes, when the student is in the distance learning environment, but in offline mode. In such a situation, the teacher has basically no means to evaluate student’s engagement to the self-learning.

With respect to the material selection, there are different methods, materials and assessments to be used for remote learning and classroom learning. At the classroom learning, the particular learning objective can be achieved by using a material A and activity B. However, if the student is participating remotely, a different set of material X and self-learning activity Y may be a better combination. Therefore, the challenge occurring at the hybrid learning environment, is to select these materials. Student co-operation is also an issue raising from a remote learning. In all learning environments, people learn also from each other, not just from the teacher. This is possible at the classroom environment. However, in a hybrid environment, the students should also be able to learn from each other by communicating, by participating and by sharing views.

Thus, the present solution is targeted to at least the following issues occurring in a hybrid learning environment:

1 ) how to teach students simultaneously, when some of the students are located in a classroom and some of the students participate remotely;

2) how to correct gaps in learning resulting from a temporal distance learning when a student returns to classroom learning environment;

3) how to combine distance learning and classroom learning environment for a single student so that both learning situations support each other as good as possible;

4) how to provide instructions to a student at the distance learning environment, when the student is offline and/or not performing the studies at the same time with the students at the classroom learning environment.

In order to tackle the matter, the machine learning model according to present embodiments is configured to track at least the remote students, and to compare their progress speed against the progress speed of students in the classroom learning environment. Based on the comparison, the machine learning model, for example the performance detector, determines whether the remote students need support, and whether they are on schedule.

In addition, the machine learning model observers the actions and behaviour of at least the remote students through the software on their computer. The observation is on how are they focusing on the assignments or is their focus targeted somewhere else (e.g. some social media application). In addition, the local machine learning algorithm, such as an emotion detector, at the student’s device is configured to analyze the emotional state of the remote students by using a webcam capturing as means to obtain data on facial expressions of the remote student and/or by using other sensors that are capable of measuring physical and psychical parameters from a student.

According to an embodiment, the machine learning model also observes the actions and behaviour of the students through their computers and/or through an additional monitoring system comprising various data recording devices. The observation is targeted to students’ focus and attentiveness at the classroom environment.

The system according to present embodiments is configured to continuously give an overview of students’ progress, at least when being remote, and pays more attention on students that are stuck in an assignment, proceeding too slowly, not focusing or doing some other things etc. Such observations may be corrected automatically by the system, and/or indicated to a teacher or to the student, and/or stored to a memory for further analysis.

Thus, a method according to an embodiment comprises at least:

1 ) receiving tracked data relating to one or more features of a student in a learning environment, in particular, but not limiting to, the distance learning environment;

2) analyzing the tracked data with respect to corresponding threshold values to determine issues relating to learning;

3) automatically applying an action as a response to a determined issue, wherein the corrective action is one or more of the following: an automatic rectification of the issue at the student user device; an indication of the issue on a teacher user device; an indication of the issue on a student user device; a suggestion on a possible rectification on a student user device; a suggestion on a possible rectification on a teacher user device; an entry to a log for an analysis.

For carrying out the first step, the smart learning system further comprises a tracking component. In addition, the smart learning system has a data transfer network with a student device in order to receive tracked data on a student in a distance learning environment. The tracking component controls various functionalities and applications at the student device to enable them to provide data concerning the activity of the student. In addition, the tracking component receives data on student’s pointer’s movements on a display of the student devise. The tracking component functions with the student user device in both simultaneous and non-simultaneous learning, wherein simultaneous refers to learning occurring at the same time with the classroom learning and non- simultaneous refers to learning occurring outside normal lesson hours. For example, the tracking component is able receive data from the education software the student uses, i.e. which page the student is at the moment, how long the student stays on certain pages, which assignments have been completed, which assignments have been skipped, how long a student stays on a certain assignment, which actions do they take, etc. In addition, the tracking component is receive data from other applications or web sites being opened at the student device, i.e. which application is currently active, where the student is inputting data, etc. In addition, the tracking component may receive data from device’s camera equipment to analyze facial expressions, eye movement, overall pose, etc. In addition, the tracking component may receive data from device’s audio capturing means to analyze sounds and words made by the student. In addition, the tracking component is able to receive data on student’s wearable device, either directly or through corresponding application at the student device. Such wearable devices may comprise a smartwatch, an activity wrist device, a ring, or other health tracker, etc.

Image and video detection neural network may be utilized to extract bounding boxes on student’s faces, to identify student’s emotions, which in combination with a set of features relating to the persons closeness to the screen, to the followed page on the education material, to the input devices the student is using at the moment, will reveal whether the student is focused on a lesson. By this, the machine learning algorithm may determine the level of concentration and tension when communicating with the teacher, and also to determine the level of involvement in the learning process.

The facial expression analysis, i.e., students’ emotions, can be performed as follows, for example: a face of each student is recoded at 5 frames per second during the time the student is participating to the class, either in a classroom learning environment or remote learning environment. The emotion detector according to the present embodiments has several emotional labels, for example focused, distracted, happy, sad, angry. The emotion detector is configured to determine that a student has a certain emotion at a time-frame if a score for this emotion is higher than a certain predefined value, for example, 0.5. It is appreciated that the predefined value can deviate from the given example. According to an embodiment, the present system may be enabled to measure a marker called “engagement score” for each student, whereupon the goal is to have an engagement score higher than a minimal threshold (for example 0.3) for each students.

Engagement score may be calculated using scores for emotions that are detected at each timeframe, as well as using the data being collected during the class. Such data may comprises number of times the student answered or asked a question, the grade of the test results, etc. Such a data may be collected dynamically and is fed into the network.

If the engagement score is too low (e.g., <0.3), the system may signal both teacher and a student, encouraging the teacher to work with a respective student more, and the student to be more active. If the engagement score is too high (e.g., >0.8), a student may be told that he/she is doing great, but he/she could give answering opportunities to other students as well.

According to a further embodiment, the tracking component may also be configured to receive data on students in a classroom learning environment. Such tracked data comprises observed actions and behaviour of students in the classroom, which have been obtained by capturing images and/or video on students. The interest is on where the students sit, what and where are they looking at, and what are they doing.

The data that is gathered from the classroom learning environment may be used as basis when determining the phase of learning for the remote students, i.e. defining a threshold value for the phase of learning. Since the teacher is acting in the classroom learning environment physically, s/he is able to see the progress of the students, and adjust the teaching accordingly. Therefore, it is important to indicate whether there are remote students that are not following anymore.

To complete a second step of the method, the smart learning system comprises an analysis component to analyze the tracked data with respect to corresponding threshold values to determine issues relating to learning. The analysis is based on a raw data that is obtained from the students. The purpose of the second step is to determine how a remote student proceeds in learning when compared to the students participating in a classroom environment. For example, it is determined whether the remote student is able to accomplish the exercises in a given time and correctly, when compared to the class-room students.

The threshold may be calculated according to the performance score of the the students in a classroom environment. The performance score for one student can be defined through several factors, e.g., speed, correctness, activity. When the performance of each student in a classroom has been calculated, an average of the performances ( performance _ave ) can be determined. This average may be used as a threshold to which the performance of the remote student {performance^ ) can be compared: set threshold = performance _ave

If performance _rem < threshold

Then action

It is appreciated that a performance score of any student in the classroom learning environment can be compared to the threshold as well to see whether the student is learning as planned.

In the analysis, the system, in particular the global machine learning model, matches the progress speed of different students or different student groups (either in the same room or partially present and partially remote) doing the same activities against each other. Due to the matching, the system notices if some group or a student is going slower, lagging behind and might need an attention from the teacher. In addition, the system is able to determine, if some students are going remarkably faster than others, and will need more activities/material before the lesson is over. In addition, the system is able to highlight students that are not focusing, e.g. doing other things, chatting with each other, etc. In addition, the system is able to highlight students and/or groups, where their emotional state needs attention.

Any matching or comparison can also be made against historical data. The system stores historical data concerning the previous students and groups working with the same material, and their speed and level of progress, and can use this data to compare the current students by means of the global machine learning model.

As a result of the second step, the smart learning system provides data concerning e.g. progress of learning; phase of learning; motivation of the students; current status of learning; status of students; alertness of students; attention of students; displacement activities of students; activities of students in general; etc.

For a third step, the smart learning system may comprise a correcting component that is able to automatically apply a corrective action to determined issues, wherein the corrective action is an automatic rectification of the issues at a student device or an indication of the issue on a teacher device. Instead or in addition, the correcting component may just indicate the issues on a student user device and/or a teacher user device.

For example, if the smart learning system notices that a student tries to start doing something, and s/he will not have time to complete during the lesson, the smart learning system is able to suggest alternative activity/material that the student will have time to complete. As an another example, the smart learning system may notice that a student or a group completes their task ahead of the schedule, whereupon it may suggest additional materials and activities that they can still complete during the lesson.

The system is able to present a dashboard view to a teacher, highlighting all the issues it discovers, allowing the teacher to quickly take corrective action through that view, suggesting different corrective actions per type of content that is displayed in this view. If the teacher indicates to the system that certain things are not worth highlighting, the system can learn this. Such issues are shown less in the future, whereas other that the teacher often takes action should remain prioritized for the teacher. In addition, the data that is gathered from the classroom learning environment can also be utilized outside the lesson hours, i.e. to a remote student that is operating non-simultaneous manner with the students in the classroom learning. The data, for example, recordings or certain parts of the recordings being analyzed according speech recognition means, can be played on a student user device to help in accomplishing assignments.

The present embodiments can be applied in various learning environment combinations. For example, the learning environment can comprises classroom environment and distance learning environment, which are occurring at the same time or at different times. In addition, the distance learning can occur online or offline. Regardless the type of the distance learning, the purpose is to provide a view to student’s progress in distance learning, and to solve possible problems being occurred therein, in a classroom learning environment.

Technically, the solution is based on co-operation of the first and second components used in the smart learning system.

The first component is configured to obtain data on the classroom learning environment. The data comprises video and/or audio and/or image data concerning recordings made in the classroom learning environment. The data may also comprise textual input data or any other input data made by a teacher and concerning remarks on the learning progress and/or educational material being dealt with.

The second component is configured to provide educational material to students in distance learning environment, and obtain data from the distance learning environment. The data provided from the distance learning environment can comprise an output of local ML algorithm installed in the student device. Instead or in addition, the data gathered from the distance learning environment comprise video and/or audio and/or image data concerning recordings made on a student in the distance learning environment. The data may also comprise textual input data or any other input data made by the student and concerning the learning progress of the student as well as accomplished assignments. In addition, the data may contain traced digital navigation paths of the student.

The educational material used the classroom learning environment and provided to a teacher via the first component, may be synchronized with the educational material used in the distance learning. In addition, the educational material used in the distance learning can be complemented with recordings made during the classroom learning.

The present embodiments are clarified by means of a use case. The first use case relates to a hybrid learning situation combining both classroom learning and distance learning occurring simultaneously. Student X participates the lesson online from a distance learning environment. Students Y, Z participates the lesson at the classroom. The educational material that is provided to the student X is synchronous to the educational material that is presented by the teacher to students Y, Z. The smart learning system tracks student’s X progress on the material, and especially assignments a - c, and notices that the student X is not able to finish assignment b. At the same time, the smart learning system tracks the learning of the students Y, Z, and teaching process in the classroom environment. From the classroom learning environment, the smart learning system gathers recordings on teacher’s and students speech, from which the smart learning system is able to recognize the topic and the phase, and possible problems the students have. From the educational material the teacher is using, the smart learning system notices that the students Y, Z are proceeding to assignment d already, while student X is still struggling with the assignment b. As a corrective action, the smart learning system tries to help student X by playing, on the student’s device, the recording of the teacher’s speech concerning the topic of the assignment b or special advices directly targeted to the assignment b. If the smart learning system notices that the given assistance does not help the student X, the smart learning system is configured to indicate the situation to the teacher.

In a second example use case, which extends the first use case, the students Y and Z in the classroom also have personal digital devices, which they use for accomplishing assignments provided by the teacher. In such a case the learning progress can be digitally monitored and tracked, whereupon comparison between the learning progress of remote students and learning progress of classroom students can be more easily determined.

The first and second use case refers to tracking students in the classroom. There are two embodiments for implementing this. As a first embodiment, the local students (i.e. students at the classroom) may have the same digital equipment and the same educational software as the remote students by means of which the lessons are viewed and assignments are accomplished. As a second embodiment, the local students are tacked by an additional tracking mechanisms and devices. A classroom may be installed with dedicated cameras for capturing image data (video / still) on the classroom. In addition to the cameras, other recording sensors may be used as well. The data may be provided to a computer program having algorithms to identify and track users. Instead, the cameras may be equipped with such algorithms to perform student identification and tracking. The first and the second embodiments may be separate embodiments, or they can be combined. When combined, the smart learning system implements a data fusion which combines the tracking data obtained from the computer and additional sensors, such as cameras. The tracking data obtained from the classroom is utilized when evaluating the progress and possible difficulties of the students at the classroom environment. In addition, the tracking data, and the evaluation results in particular, may be utilized when evaluating the progress and possibilities of the remote students. If tracking data is unavailable from the classroom environment, then the remote students and their progress is being evaluated by utilizing history data of earlier remote students.

In a third example use case a part of the learning happens in online and part in offline environment. In such arrangement, the smart learning system will track the online assignments and notify the teacher on the gaps in accomplishing the assignments as well as recommend the study material for the next online and offline classes. If there are gaps in online or offline learning results compared to lesson plans, the system will recommend additional lessons for the student, the additional lessons focusing on the gaps in the classroom setting. Since the machine learning model continuously gathers data on student’s progress and possible problems, the system, with the help of machine learning model can identify individual learning styles. For example, if student repeatedly has problems with a certain type of online or offline learning methods, the system can recommend different learning methods.

Figure 4 illustrates further an example of the learning environments, and the operation of the smart learning system. As shown in Figure 4, the solution is applicable in a classroom learning environment 400 and a distance learning environment 405. The classroom learning environment 400 refers to a physical environment, where teacher and students operate. The classroom learning environment 400 comprises at least a teacher user device 401 , and in some embodiments, also a plurality of student user devices. In addition, the classroom learning environment comprises a camera 402 with a microphone for recording video/audio/image material on the classroom learning. The distance learning environment 405 comprises a student user device 406 having a camera 407 and one or more applications 408, 409 executed on the student user device 406.

A smart learning system 450 comprises a tracking component 453, an analysis component 455, and an interface 457 for communications of data with one or more applications executed on a teacher user device 401 and/or a student user device 406. The tracking component 453 is configured to receive a first data set generated by one or more applications 408, 409 in connection with execution of said applications 408, 409. At least one of said application is a learning application and at least one other of said applications is a monitoring application. The monitoring application is configure to track and store data relating to user’s actions on a student user device 406. The tracking component 453 is configured, based on the received first data set to determine for example one or more of the following: learning progress of the student, attentiveness of the student, emotional state of the student. After this, the analysis component 455 is configured to analyze the tracked data to determine issues relating to the learning and/or motivation of the student.

In addition, the tracking component 453 may be configured to receive a second data set recorded from a classroom learning environment 400, in particularly from the one or more applications being executed on the teacher user device 401 and/or a camera 402. An analysis component 455 is configured to analyze the recorded second data set to determine a learning phase of the students in the classroom learning environment; and to compare the first data set to the second data set.

The analysis component 455 is further configured to select a corrective action to be applied to a determined issue, despite whether the determined issue is determined based on the first and second data, or only the first data. Initially, the corrective action may an automatic rectification of the issue at the student user device 405. In addition, or alternatively, the corrective action is an indication of the issue on a teacher user device 401 .

The functionalities of the tracking component and the analysis component may be provided by the first and the second component discussed above. Therefore, the first component tracks and analyzes the learning progress occurring at the classroom learning environment, and the second component tracks and analyzes the learning progress occurring at the distance learning environment. Both the first and the second component may utilize the machine learning model by providing corresponding input data to the machine learning model. The machine learning model outputs a respective result, which is transmitted to the component in question. Each component may control the respective learning environment according to the result from the machine learning model. For example, if the second component receives data from the distance learning environment, which data comprises video recordings taken on the student and data on pointer navigation on the student device, such data may be provided to the machine learning model which determines that the user lacks motivation and surfs on the internet. This result of determination is delivered to the second component, which selects more interesting assignments to be transmitted to the student device, or challenges the student in other way. Thus, the second component controls the assignment selection and/or the learning application based on the result received from the machine learning model. The first component may function in similar manner by controlling either teaching applications in teacher device, or - if available - learning applications in student devices.

Figure 5 illustrates a method according to an embodiment as a flowchart. The method comprises at least steps for receiving 510 by the tracking component a first data set generated by one or more applications in connection with execution of said applications, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications; analyzing 520, by the analysis component, the first data set to determine issues relating to the learning and/or motivation of the student; selecting 530, by the analysis component, an action to be applied as a response to a determined issue, wherein the action is an automatic rectification of the issue at the student user device; an indication of the issue on a teacher user device; an indication of the issue on a student user device; a suggestion on a possible rectification on a student user device; a suggestion on a possible rectification on a teacher user device; an entry to a log for an analysis.

An apparatus according to an embodiments comprises means for implementing the method according to various embodiments. With respect to the method of Figure 5, the apparatus comprises means for receiving (i.e. a tracking component) a first data set generated by one or more applications in connection with execution of said applications, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications means for analyzing (i.e., the analysis component) the tracked data to determine issues relating to the learning and/or motivation of the student; and means for selecting (i.e., the analysis component) an action to be applied as a response to a determined issue, wherein the corrective action is one or more of the following: an automatic rectification of the issue at the student user device; an indication of the issue on a teacher user device; an indication of the issue on a student user device; a suggestion on a possible rectification on a student user device; a suggestion on a possible rectification on a teacher user device; an entry to a log for an analysis. In addition, the apparatus comprises processing means, such as one or more processors, memory and a computer program code being stored in said memory. The computer program code is executable by said one or more processors to implement the steps of a method according to various embodiments.

An apparatus according to an embodiment, which can be implemented as a student user device, comprises at least means for tracking, means for analysis and means for communications of data, wherein the means for tracking is configured to receive a first data set generated by one or more applications in connection with execution of said applications in said apparatus, wherein the first data set comprises data relating to at least user’s actions on a student user device with respect to at least one of said one or more applications; the means for analysis is configured to analyze the first data set to determine issues relating to the learning and/or motivation of the student; the means for communications is configured to deliver information on the determined issues to another device, and as a response from said another device, to receive an information on an action to be applied to the determined issue.

Various aspects of the embodiments are set out in the independent claims. Other aspects comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims. It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications, which may be made without departing from the scope of the present disclosure as, defined in the appended claims.