Field of the Invention

The present invention relates to a computer-implemented method for determining topics on which a user is working, and reading actions and reading activities thereof through the screenshots of the user. The present invention, more specifically, belongs to field of information technology, and relates to supporting personal information management and personal knowledge management processes which allow identifying, monitoring and analyzing reading activities by means of determining topics on which individuals are working and the reading activities they perform by analyzing texts obtained with OCR (optical character recognition) method from screenshots of information processing devices of the individuals with the help of topic modelling which is a machine learning technique.

Background of the Invention

In the applications known in the state of the art, texts can be obtained from screenshots by optical character recognition (OCR) method in information processing devices and these texts can be saved in a text file.

Turkish patent document no TR 2016 17663 B titled "A Method Providing Access to Screenshots on Which the User Works with the Help of Keywords" discloses a method which enables individuals to access past screenshots on their computers with the help of keywords. The said method allows the user to see at what time and at what intensity they were working on a topic and to remind that time. Furthermore, it allows simultaneous indexing and searching since the individual uses the cloud storage unit on more than one computer.

United States patent document no US20060284838 discloses a method which saves screenshots to files and allows the user to access their computer remotely. According to one embodiment of the invention, it captures the screenshots and acquires text via optical character recognition (OCR), and saves these in the storage unit as day/week/month.

In applications known in the state of the art, methods for extracting the text from images captured by wearable cameras with OCR have also been developed. Kimura et. al. tried a system which allows outputting text from images comprising all types of texts, also including computer screenshots, by OCR in case the said images are captured by head-mounted cameras [1].

However, in these methods known in the state of the art, it is not included to determine the reading actions and reading activities performed by the user " through time" by analyzing these text files acquired from screenshots or camera images by optical character recognition (OCR) method.

In other methods known in the state of the art, the content of text files can be analyzed with technique called "topic modeling" and it is possible to obtain the topics that can determine the content in these documents. Each topic consists of a certain number of words, each of which has a different weight on the topic. Therefore, it can be shown statistically in what weight of mixtures of different topics the content in a document can be created. By means of a technique based on LDA (Latent Dirichlet Allocation) algorithm, which was initially developed by Blei et al. in 2003 [2], content analysis of many text documents can be performed [3] However, these applications do not include to obtain the topics on the said screens by analyzing the text files obtained from the screenshots and camera images by "topic modeling" technique regularly "during a period of time" and to determine the reading actions and reading activities of the user by analyzing these topics.

In other applications known in the state of the art, the actions and activities of the user can be recognized and identified with the help of data obtained from the processes performed by the computer users on the computer [4, 5] However, in the applications developed for this purpose, there are softwares based on capturing intensive data which run in the background of the user’s computer and capture keyboard and mouse usage, screen touching movements of the user; record the software by the user, the files they open and the windows they view by capturing them in detail. The use of these softwares, specifically in corporate environments, causes security problems. On the other hand, since the development of these applications requires detailed coding for each device type and operating system, generalization to these devices and operating systems is costly.

United States patent document no US20130110565 suggests a system which enables to manage the activities of remote users from a central point by means of capturing data belonging to all interactions of the user on the computer, including the screenshot of the user. The suggested system involves review and evaluation of the captured data by a human. However, in these applications known in the state of the art, the method of recognizing and determining the actions and activities of the users by the same device or an information processing device allocated for this job with the help of the topics determined by utilizing screenshots of the users is not disclosed.

In United States patent document no US20070299631A1, keystrokes, file accessed, web sites visited, communication events are recorded into a user log in order to determine activities of a user. The suggested system also envisions data collection, evaluation and sharing of multiple users and multiple devices. However, these applications do not include the method of determining the reading actions and activities with the help of the topics determined by utilizing the screenshots of the users.

In the applications known in the state of the art, data in various forms such as applications currently running, screen touches, keystrokes, mouse movements, file movements, visited websites, etc. are required to be collected from the devices used by the users, and a low level of research - development and programming is required for each device and operating system in order to collect this data. In the invention of this document, by means of an algorithm detailed in the following sections, the topic on which the user is working on can be determined and the reading actions and reading activities of the user can be determined by using these topics only by capturing screenshots regularly and continuously from the device used by the user.

Furthermore, even though capturing screenshots and camera images continuously and obtaining text from these images is known in the state of the art, arranging text files in a way to enable obtaining topic models belonging to different temporal dimensions by means of using LDA algorithm so as to determine the topics on the user is working in "different temporal dimensions", which is disclosed with process steps of determining topic models in the present invention, is not a method known in the state of the art" .

Moreover, the method of determining the user’s reading actions within "time intervals" from topic models obtained by means of using the LDA algorithm from text files acquired from screenshot or camera images for "different time intervals” is not explained in the state of the art.

In addition to these, in the state of the art, the method of determining the user's reading activities within "temporal dimensions" from the topic models obtained with the LDA algorithm "for different temporal dimensions" from text files acquired from screenshot and camera images is also not explained. References

[1] Kimura, T., Huang, R., Uchida, S., Iwamura, M., Omachi, S., & Rise, K. (2013). The reading-life log— technologies to recognize texts that we read. In 12th International Conference on Document Analysis and Recognition (ICDAR), IEEE, pp. 91-95.

[2] Blei, D.M., Ng, A.Y, Jordan, M.I. (2003). Latent Dirichlet allocation, Journal of Machine Learning Research, 3, pp.993- 1022.

[3] Blei, D. M., & Lafferty, J. D. (2009). Topic Models. Text Mining: Classification, Clustering, and Applications, 71-89. http://doi.org/10.1145/1143844.1143859

[4] Grefenstette, G., & Muchemi, L. (2016). On the Place of Text Data in Lifelogs, and Text Analysis via Semantic Facets Present and Future of Textual Lifelog Data Induction of Semantic Facets in Textual Lifelog Data, 30(i), 2015- 2017.

[5] Okamoto, M. (2014). Topic -by-Topic Activity Estimation for Knowledge Work Lifelog, 29-39. http://doi.org/10.1007/978-3-662-44651-5_3

Problems Solved with the Invention

With the computer-implemented method developed, the topics on which an individual is working, and the reading actions and activities carried out on these topics can be determined and identified by continuously capturing the texts in the screenshots of the information processing devices used by the individual via screenshot capturing method and analyzing these texts. For this purpose, an algorithm consisting of the following four steps for the solution process of the problem is used:

(1) As long as the desktop, laptop and tablet computers and smart phones used by the individual are turned on, screenshots are captured from these devices every 30 seconds by default. The texts contained in these images are extracted with the OCR technique, which is a technique known in the state of art, and text files are created and named with date-time tags, saved in folders named with date-device name tags and transferred through a cloud service and gathered in a storage unit.

(2) The keywords identifying the topics of this compilation consisting of documents of different temporal dimensions and the topic models comprising compositions of these topics in the said documents are obtained by using LDA (Latent Dirichlet Allocation) algorithm, which is a method in the state of the art, on text files belonging to different time intervals determined by day, month, year or by the user.

(3) According to an algorithm of "reading action determination" included in the invention and which is not a method known in the state of the art; the reading actions that correspond to the topics on which the individual is working in the selected time dimension are determined by scanning the topics and compositions in the topic model belonging to a selected temporal dimension and device in a given time interval, taking advantage of the change of the dominant topics in the images over time.

(4) According to an algorithm of "reading activity determination" included in the invention and which is not a method known in the state of the art; the reading activities performed by the individual are determined with the help of a similarity search, which enables to gather similar topics higher than a determined threshold value in different temporal dimensions such as day, month, year, of a time interval with its start and end date given and in different devices.

With the computer-implemented method of the present invention; for example, a student can successfully compile the reading and writing actions of the thesis they have been working on for a year from among the screenshots of other unrelated actions and activities they have performed during this period, and obtain a set of images consisting of screenshots only from the thesis preparation moments, and use it as a proof of doing the said work.

It is a more device -independent and more flexible method compared to other methods, since the reading activities of the user can only be obtained from the regularly screenshots captured and it can be applied on the information processing device (electronic device) used by the user without needing to use any of its properties other than screen capture.

Therefore, any computer program developed in order to determine the topics on which the user is working and reading actions and activities can be stored in a portable USB memory and the user can run the program in this USB memory on the computer on which they will be working at that moment, and determine their activities they perform without interruption between the devices they have by means of utilizing the said computer-implemented method. Similarly, a mobile application to be developed by utilizing the said method can be downloaded to the user's smartphone or tablet, and therefore the same process can be performed on these devices. In another aspect, web application add-ons to be developed by utilizing the said method can be downloaded to the user's internet browser, and therefore the screenshots of the user only in the internet environment can be captured and processed. In another aspect, software to be developed by utilizing the said method can be embedded in the operating system of a digital book reader, therefore page images of all digital books the user reads on this device can be captured, and the topic, action and activity determination processes are carried out. In another aspect, software developed by utilizing the said method can be embedded in a wearable camera device, and therefore the images in line of vision of the user can be captured and the texts in these images are extracted and processed.

Seven important sets of gains can be obtained as a result of determining the topics on which the individual is working, reading actions and activities on these topics. These output sets are explained below, being numbered between (5) and (11):

(5) The reading activities performed by the individual in certain time interval, temporal dimension and devices are compared with the past reading activities, and similar activities can be recognized and it can be listed to the user according to the similarity rate.

(6) It is possible to identify the reading activities performed by the individual in certain time interval, temporal dimension and devices utilizing the suggestions made by the system by means of recognizing the past reading activities and/or comments of the individual. As a result of this identification, past reading activities have a descriptive name and the user can search on past activities with keywords.

(7) As a result of acquiring topic models from screenshot and camera image texts, temporal distribution of the topics on which the individual is working can be obtained utilizing the said topic models. Therefore, it can be reported graphically when an individual becomes interested in a particular topic, during what time, day or month they are interested, and when they quit being interested. This process also allows temporally observing multiple topics on which the individual is interested at the same time.

(8) The evolution of any topic on which the individual is working in the course of time can be determined and viewed by temporal analysis of the topic models. Therefore, as soon as an individual begins to review a topic, they can observe how the concepts that make up the topic turn into concepts when they end reviewing the topic.

(9) After determining the reading activities, the trees formed by the sub-activities that these activities cover in different time intervals and temporal dimensions can be calculated. Reading activity trees calculated using a coverage ratio given at the beginning can hierarchically display which sub-activities a parent activity covers along with the timelines of the activities. Being able to determine the parent, side and sub activities of a reading activity provides a powerful tool for the individual to make sense of the relevant reading activity and to associate it with other activities.

(10) It is possible to obtain reading activity analytics after the reading activities are determined. Reading activity analytics create statistics of individual’s reading activities in different time intervals distributed in different temporal dimensions such as yearly, monthly and daily relating to topics, actions, time of staying on the image and number of screen/camera images in different dimensions from any temporal level to the temporal level at which only one image is captured (in other words time periods varying from one year to 30 seconds by default) and provide the opportunity to analyze them.

(11) After the reading activities are identified, the individual can create reading activity portfolios. Reading activity portfolios allow the screen/camera images of the entire set of reading activity identified on a yearly, monthly or daily level or its part filtered with start-end dates to be copied to a separate folder and packed. The individual has a proof that they have performed the said activity with an identified reading activity portfolio.

The claimed invention and the gains that can be acquired with the invention allow every information device that can save screenshots (smartphone, smart television, game console, digital book reader, tablet, laptop computer, desktop computer, etc.) and every wearable camera that can capture image to be transformed into a smart device which can determine the user’s reading actions and activities by means of recognizing the topics on which the user is working on and which can report these actions and activities based on topics.

Detailed Description of the Invention

An algorithm belonging to a computer-implemented method which is developed to fulfd the objective of the present invention, and enables to determine the topics on which the user is working, reading actions and reading activities from screenshots and wearable camera images of the user is illustrated in the accompanying figures, in which;

Figure 1 is the flowchart of the algorithm of the computer-implemented method that enables to determine the topics on which the user is are working, reading actions and reading activities from the screenshots and wearable camera images.

Figure 2 is the flowchart of the algorithm belonging to the process which allows capturing a screenshot or camera image from each device continuously and obtaining text.

Figure 3 is the flowchart of the algorithm belonging to the process which allows obtaining topic models from texts in different temporal dimensions. Figure 4 is the flowchart of the algorithm belonging to the process which allows determining reading actions from topic models.

Figure 5 is the flowchart of the algorithm belonging to the process which allows determining reading activities from reading actions.

The components shown in the figure are each given reference numerals as follows: 100. A computer-implemented method for determining the topics on which the user is working, reading actions and reading activities from screenshots and wearable camera images

200. Continuously capturing screenshot or camera image from each device and obtaining text,

210. Capturing image which contains text,

220. Obtaining text from image containing text with optical character recognition (OCR) method,

230. Creating unique fde name with date and time tags in order to save image and text separately,

240. Creating unique image and text folders with date and device tags on the cloud storage service,

250. Saving image file with a unique file name to a unique image folder,

260. Saving text file with a unique file name to a unique text folder,

300. Obtaining topic models from texts in different temporal dimensions,

310. Obtaining start and end date and temporal dimension (day, month, year, etc.) information for the preparation,

320. Copying text files to temporary text folders corresponding to temporal dimension in high volume external storage unit,

330. Obtaining the start and end date and temporal dimension information for model creation,

340. Obtaining topic models from text files in temporary text folders with LDA (Latent Dirichlet Allocation) algorithm and saving them in topic models folders corresponding to the temporal dimension.

400. Determining reading actions from the topic models,

410. Receiving temporal dimension in day, month or year format with the start- end dates and selecting the device,

420. Scanning each document in composition file,

430. Determining the documents in which the dominant topic is changed,

440. Identifying reading actions corresponding to documents between two dominant topic changes. 450. Listing reading actions of a selected topic.

500. Determining reading activities from the reading actions.

510. Receiving temporal dimension in day, month or year format with the start- end dates and selecting the device,

520. Obtaining topics-actions list,

530. Determining the similarities of the topics,

540. Obtaining reading activities by gathering similar topics,

550. Saving reading activities,

560. Listing topics of a selected reading activity and reading actions.

The computer-implemented method (100), which is the subject matter of the invention that enables to determine the topics on which the user is are working, reading actions and reading activities from the screenshots and wearable camera images, is shown in Figure 1.

The said computer-implemented method (100) is run on a computer system which comprises at least one electronic device comprising at least one data entry device for entering data, at least one internal or external data storage unit for connecting to the said electronic device, at least one image recording device, at least one cloud storage unit, and a data communication network for providing data communication between the said cloud storage unit and the electronic device.

The said electronic device can be a smart phone comprising a screen, smart television, game console, digital book reader, tablet, laptop computer, desktop computer, a game console that can be connected to a screen, etc.

The said data entry unit which is used for receiving a time interval (start and end time), temporal dimension (daily, monthly/yearly or temporal dimension of a certain day/month/year length) or number of topics (daily/monthly/yearly number of topics) from the user may be a unit such as a keyboard, mouse, touchscreen, wireless/wired game remote controller, etc.). The said image recording device may be glasses with a camera, a wearable camera such as a necklace with a camera, an action camera that can be mounted on a helmet, a camera that can be mounted on clothing, a camera carried by the user or a camera that captures the image in line of vision of the user during reading, etc. At least one of the said image recording devices may comprise a data transfer cable (for example a USB cable), or a wireless data communication unit to the electronic device and/or cloud server (for example a module such as Wi-Fi, 3G/4G/4.5G/5G... module, GSM module, GPRS module, etc.) for transferring the captured camera images to the electronic device.

In the computer-implemented method (100), the process of continuously capturing screenshot and/or the image taken by the image recording device from each electronic device owned by the user and obtaining text from these images (200) is carried out as the first step. For this purpose, images are taken from the screen of the electronic device or the image recording device at certain time intervals determined before by utilizing the clock of the operating system of the electronic device used by the user (for example in every 30 seconds) or determined by the user via the data entry unit (210).

In the preferred embodiment of the invention, one of screenshot capturing techniques known in the state of the art is utilized. For example; a technique is used wherein commands of the operating system used in the device allowing to copy the data in the graphic memory where the screenshot is kept and to create an image file are used.

In the preferred embodiment of the invention, image capturing functions present in the operating system embedded in wearable cameras are utilized.

The obtained screenshot and camera images are converted in to text with optical character recognition (OCR) method (220). Files of the converted texts and the pictures taken from the screen/camera image are named with the same unique (different from each other) tags such that they will the date on which the screenshot or camera image is taken and the time data consisting of hour-minute- second (230).

In the preferred embodiment of the invention, image files are preferably named as “year month day hour minute second. jpeg” the year, month and day information belonging to the date of the capture and the hour, minute and second information belonging to the time of the capture; and text files are preferably named as “year month day hour minute second. txf

Unique (different from each other) folders are created in the cloud storage unit, which can be accessed through the electronic device or the wearable camera, according to the date and device name in order to store the files, if they were not created before (240).

In the preferred embodiment of the invention, image folders are named as “\year\year.month\year.month.day\device_name” the year, month and day information belonging to the date of the capture and the device_name being the name of the device on which the capture is carried out; the text files are named as “\year\year.month\year.month.day\device_name-OCR” . Both folders groups are opened under a main folder (For example, a folder named " l LifeLogging") in the cloud storage.

Screenshot and camera image files named with unique (different from each other) tags are saved in the corresponding uniquely named image folder (250). Uniquely named text files are also saved in the corresponding uniquely named text folder (260).

In the preferred embodiment of the invention, the screenshot and camera image files captured at all dates and times and the text files obtained from the images can be stored on all electronic devices of the user without being confused with each other with the unique naming application, and it can be seen from the folder/fde names on which electronic device, on what date and hour-minute-second they are captured. The folder created and the file saved in any electronic device can be accessed simultaneously from other electronic devices used by the user since the user uses the cloud storage unit.

In the preferred embodiment of the invention, image and text files are regularly (once in every minute, hour, day, week, etc.) transferred to an electronic device assigned for processing data, with the same folder format to a high-volume external data storage unit from the cloud storage unit, and they are moved from the cloud storage unit. Therefore, it is prevented that the ever-growing life log data in the cloud storage takes up increasing disk space on all electronic devices of the user at the same time. If the user has enough storage space in the cloud storage unit for long-term recording, this process will not be required and the data can be processed in the cloud storage unit.

In the computer-implemented method (100) of the present invention, the process of obtaining the topic models of different temporal dimensions from the texts (300) is carried out in the solution step after the solution step of continuously capturing the screenshot and/or camera image in every electronic device owned by the user and obtaining text (200).

The solution step of obtaining topic models in different temporal dimensions from the texts (300) of the process consists of two consecutive stages, namely the preparation stage which can be applied at the same time or at different times and the stage of creating model.

In the said preparation stage, it is enabled that a working copy of the text files to be processed belonging to a time interval and temporal dimension (310) received from the user is created (320); and in the model creating stage, it is enabled that the topic models are obtained by using working copy date in a time interval and temporal dimension received from the user (330). Words that do not add meaning to the textual content extracted from the screenshot or camera image during obtaining topic models can be placed in a fde, such as a "stoplist.txt" fde, and given as input to the LDA algorithm so that they are not taken into account when determining the topics in the LDA algorithm. Therefore, although it does not contain any text, the meaningless and contaminated text data generated by OCR method from the images are also deleted and ignored in the following stages (340). At this stage, if desired, it is continued to the process by going back to the beginning (310).

In the stage of data preparation, the start and end date and temporal dimension information are received separately from the user for preparation (310). The start and end dates should be two dates preferably within the same year. However, the temporal dimension can be "daily", "monthly" or "yearly". The second process of this preparation step is to copy (320) the text fdes to temporary text folders corresponding to the temporal dimension in the high volume external data storage unit. Therefore, it is ensured that temporary copies of the original life log data, which will be used during creating the topic models, are created during the preparation step. When the daily temporal dimension is selected, the fdes in the folder belonging to a day between the start and end date are copied in the exact way to a temporary folder for the corresponding day. When the monthly temporal dimension is selected, all of the fdes in the folders belonging to the days of a month between the start and end date are copied to a temporary folder for the corresponding month. When a yearly temporal dimension is selected, all of the fdes in the folders belonging to the days of that year are copied to a temporary folder of that year.

In the preferred embodiment of the invention, text fdes stored in the external data storage unit with the naming system

“ \LifeLogging\year\year . month \year. month. day\device name- OCR\year.month.day_hour.minute.second.txt” are copied to temporary folders named “MempDaily”, ‘ \TempMonthly ” or “ \TempYearly ” depending on the selected start-end dates and selected temporal dimension. As a result of this, folders are created in the source folder within the cloud storage unit as many as the number of days in TempDaily, as many as the number of months in TempMonthly, as many as the number of years in TempYearly, and they are accumulated as long as they are not deleted. The preparation process is a one-way process and as a result of copying, if there are previously copied files in the temporary folders, they are deleted and replaced with the final versions of the original files. The folders of temporary files are preferably gathered under a shared folder (for example V/'emp ) for easy access.

In the stage of creating topic models of the solution step, the start and end date and temporal dimension information are received from the user for model creation (330). The start and end dates should be two dates within the same year. However the temporal dimension can be "daily", "monthly" or "yearly". In the second process of the model creation stage, the LDA (Latent Dirichlet Allocation) Algorithm is applied on the text files in the temporary folders, the topic models are obtained (340) and written to a topic model folder that will correspond to the time intervals in each temporary folder. After this process, if desired, it is returned to the beginning (310) and continued the process of obtaining the topic model for other dates.

In the preferred embodiment of the invention, the topic models obtained from files in YiempDaily folder are placed in “\TopicModelsDaily” folder, topic models obtained from files in “ \TempMonthly ” folder are placed in \ 7 'o picMode Is Mon ihly . and the topic models obtained from “ \TempYearly ” folder are placed in \TopicModels Yearly folder. They are preferably gathered under a shared folder (for example ‘ TopicModels ”) for easy access of topic models. In the preferred embodiment of the invention, the "MALLET Topic Modeling" package with open source code, which is an application known in the state of the art and developed in Java language, is used in the calculation of topic models. The "import-dir" and "train-topics" commands of the "MALLET Topic Modeling" tool are used in order to obtain topic models. The "import-dir" command is used to determine the folder in which the data fdes the topic models of which are to be calculated and the basic parameters. The output of this command is a “.mallet” fde. The "train-topics" command, which is the second command, takes the " .mallet" fde, which is the output of the first command, as input and generates the composition file preferably named "composition.txt" which contains probability distributions on of the topics on the text files with the keys txt comprising the keywords of the topics.

The "keys txt" file contains the topic numbers of the topic model belonging to the collection and the first 20 words by default that make up each topic. In the preferred embodiment of the invention, 20 topics for the daily topic model, 60 topics for the monthly topic model, 200 topics for the yearly topic model are determined by default. The algorithm can also be implemented such that it will take the number of different topics from these values as a parameter.

In the "composition.txt" file, first the folder and name of the file, and then the probability values that enable the content of the file to be determined by the topics in the topic model are included for each text file of the collection. The probability values consist of values between 0 and 1 and are arranged in the order of the topic number used in the "keys.txt" file for each text file. The total of the probability values of the topics identifying the content of that file is 1 for each text file.

In the preferred embodiment of the invention, examples of folder and file names of " keys.txt " and "composition.txt" files obtained with the "MALLET Topic Modeling" tool are given below: Example of daily topic model is

“\TopicModels\TopicModelsDaily\year\year.month\year.mon th.day\device \composition.txt” and

“\TopicModels\TopicModelsDaily\year\year.month\year.mon th.day\device\keys.tx t”; example of monthly topic model is

“\TopicModels \TopicModelsMonthly\year\year. month \device \composition. txt ” and example of yearly topic model is

“\TopicModels\TopicModelsYearly\year\device\composition .txt” and “\TopicModels \TopicModelsYearly\year\device \keys. txt” .

In the preferred embodiment of the invention, although it has been suggested to determine the topic models only in daily, monthly or yearly temporal dimensions in order to make the algorithm clear, the same algorithm can be easily applied to obtain topic models of temporal dimensions of any day length depending on the number of days. Therefore, topic models belonging to different temporal dimensions such as at least one day, at least one week, at least one year periods, for example two-day, two-week, seasonal, semi-annual, two-year, etc. depending on the preference of the user.

The first stage of the solution step of determining the reading actions from the topic models (400) is the process of taking the temporal dimension from the user in the form of day, month or year and selecting the device (410). Then each text document in the composition file is scanned in the topic model of the selected day, month or year and the device (420). During this scanning process, the dominant topic of each text document is determined. The dominant topic is the topic with the highest probability value among the topics forming the content in a text file. The documents in which the dominant topic changes are determined during scanning cycle (430). If the sequence number of the topic with the highest probability value in two consecutive images changes, it means that the dominant topic has changed. Therefore, a new “reading action” is obtained for documents between both dominant topic changes (440). If the dominant topic were changing in each image, the number of reading actions to be obtained for that day, that month or that year would be as many as the number of screen/camera images of that day, that month or that year. In reality, each reading action usually consists of one or more images since the dominant topic remains on the screen or in the image for more than 30 seconds, which is the default value, while the individual is reading a text on a screen or at their line of vision. In this case, the number of total reading actions is generally less than the number of images. There may be many reading actions distributed over different times within the temporal dimension in one topic. In other words, more than one non-consecutive reading action distributed in the temporal dimension may belong to the same dominant topic. In the extreme case, it is obtained at least zero reading action for each dominant topic and at most as many reading actions as the total number of images captured. A “topics-actions” list is obtained as a result of the algorithm. In this list, a list of reading actions for each topic is kept, sequentially according to the number of topics throughout the selected temporal dimension. However, the list of numbers of images belonging to the reading actions is kept for each reading action. Thus, all captured images belonging to the selected temporal dimension and the selected device are each assigned to a reading action, and thus each reading action is assigned to a topic. At this stage, the user selects a topic from the “topics-actions” list and can view the list of reading actions belonging to the selected topic, and take the list of screenshot and camera images belonging to a reading action selected from the said list (450). If the screenshot and camera images are displayed in a minimized form, all screenshot and camera images of a selected topic are displayed simultaneously, and therefore reading actions performed during the selected temporal dimension can be viewed. After this process, if desired, the process of determining the reading action can be continued by selecting a new time interval, a new temporal dimension or a new device by going back to the beginning (410).

In the preferred embodiment of the invention, the list of reading actions belonging to each one of, for example, 20 topics in one day, 60 topics in one month, or 200 topics in one year is determined with this algorithm.

The first stage of the solution step of determining the reading activities from the reading actions (500) is the process of taking the temporal dimension in the form of day, month or year and selecting the device (510). Then, the process steps numbered (420) (430) and (440) in Figure 4 are applied as the algorithm of determining reading actions from topic models, and thus the "topics-actions" list belonging to the selected temporal dimension and device is obtained (520). A matrix of topic similarity values is obtained by comparing each topic in the topic model of the selected temporal dimension and device according to the similarity of their words with another topic (530). Then, similar topics obtained by a similar topics search algorithm on the similarity values matrix are brought together. The similar topics search algorithm is comprised of two stages. In its first stage, the topics in the "topics-actions" list are ranked from the highest number of reading actions to the lowest number of reading actions. Then, for each topic in this list, within the similarity values matrix, different topics with a similarity greater than a certain threshold value are searched. When a similar topic is found, the previous topic is quitted, and the search process is continued with the new topic. The search is continued until there is no similar topic similar with each found topic. If the topics similar to a topic have not already been placed, they are placed within the same similar topic group. This cycle is continued for topics that have not been placed into any similar group in the list of “topics-actions”. At the end of the cycle, similar topics in the "topics-actions" list are grouped. Each of the similar topics groups found is called a "reading activity" (540). Then, reading activities are recorded (550). The process of recording the reading activity is performed by means of text files which contain topics grouped in the form of topic groups, similar to folders created to contain information about the selected temporal dimension, the selected device and the threshold value used in the algorithm.

After the process of recording the reading activity, the user can select an activity from the reading activities list at this stage, and get the list of the topics of the selected activity and the list of actions related to these topics, and the list of screens and camera images of these actions (560). If all screenshot or camera images of a selected topic are displayed in a minimized form, the user can view the improvement of the reading activity in the selected temporal dimension by viewing the screenshot and camera images of that reading activity as a whole. After this process, it can be continued to the process of determining reading activity for different time intervals, different temporal dimensions or different devices by going back to the beginning (510).

In the preferred embodiment of the invention, "cosine similarity algorithm", which is a method known in the state of the art, is used in order to determine the topic similarities.

The folder names format, “ED”, used for storing the reading activities determined in the preferred embodiment of the invention for topic models in different temporal dimensions with the similarity threshold value ranging from 00-10 are as follows:

Example of folder/file name format for recording reading activities in the daily topic model:

“\ Detection \Daily\year. month.day\device'\ActivitiesED. txt

Example of folder/file name format for recording reading activities in the monthly topic model:

“\Detection \Monthly\year. month \device 'ActivitiesED. txt ” Example of folder/file name format for recording reading activities in the yearly topic model:

“\Detection\Yearly\year\device\ActivitiesED. txt”

By means of gathering the reading activities under a main folder, for example '"Detection", it is allowed to easily distinguish the results of the process from the "\Temp..." folders where temporary process files are kept and "\TopicModels ... " folders where the calculated topic models are kept.

In the preferred embodiment of the invention, 20 topics in the daily topic model, 60 topics in the monthly topic model, 200 topics in the yearly topic model have been received, and the content of an exemplary reading activity file is as “[(9 3 7 11 18) (1 2 4) (5 6) (10 8 15) (20) (19 12 14) (13 17 16)]”. In the reading activity file in this example, the sequence numbers of 20 different topics belonging to the daily model have been collected under seven different reading activities. In other words, 7 different reading activities were carried out belonging to that day, and the third reading activity is comprised of reading activities belonging to the topics no 5 and 6 of the daily model. Similarly, reading activities of 60 topics obtained in the monthly time selection are recorded to the corresponding activity text file in the monthly activities folder; and the reading activities of 200 topics obtained in the yearly time selection are recorded to the corresponding activity text file in the yearly activities folder.

In another preferred embodiment of the invention, the process of recording reading activities (550) is carried out as batch processing for a selected time interval, temporal dimension and device. In this approach, the user can enable the reading activities of all devices on a selected day, a selected month, all days of a selected month, a selected year, all months of the selected year and all days of a selected year, respectively, with default threshold values to be calculated and recorded automatically. The user can list and review the reading actions belonging to different reading activities in a selected temporal dimension and on the selected device, and the screenshot and camera images of these actions by means of the reading activities obtained at this stage of the algorithm. This type of review will allow the user to plan future reading activities, monitor and control current reading activities, and evaluate past reading activities. On the other hand, the reading activities obtained in this stage can be used as data for acquiring outputs mentioned in “Other Advantages of the Invention” and numbered between (5) and (11).

The fde and folder names mentioned in this document are given as example in order to better understand the subject, and different unique names can also be given.

As a result, the computer-implemented method of the present invention (100) enables the actions and activities for reading performed by the individuals on printed-written texts or topics on which they are working on information processing devices to be determined via images captured by the same information processing devices or a different information processing device assigned for this or a video recording wearable camera, and enables to obtain and use information required for the individual to plan, control and evaluate their activities.

The user can see which reading actions and activities they performed on which topics in different time intervals in the past on an information processing device the screenshot of which can be recorded by means of the said method, and therefore they have the opportunity to access similar reading activities they performed in the past and the system can make suggestions to identify this activity.

The computer-implemented method (100) of the invention which determines the topics on which the individuals; who are known as information workers and who perform information work on a physical medium such as paper, book, notebook, or on one or more desktop, laptop or tablet computers throughout the day; are working via the said information processing devices or image recording devices, monitors reading actions they perform and enables to list the reading activities which the individuals perform in different time intervals by means of analyzing these actions will contribute to the efficiency of the user in information processing.