ELECTRONIC SYSTEMS, METHODS AND APPARATUS FOR GROUP

TEACHING AND EXAMINATION

FIELD OF THE INVENTIO [00011 The present invention relates to systems, methods and apparatus for group teaching and examination, also known as testing or assessment. In particular, embodiments relate to examination of a group of users via a computer network,

BACKGROUND TO THE INVENTION [0002] Examinations are an important part of education. Examinations are typically prepared by an educator, such as a teacher, and include questions which are provided to a group of students to complete. The students' answers to the questions are then marked by the educator who determines a grade for each student.

[0003] One problem often encountered when providing an examination to a group of students is that the students can copy answers for the questions from each other. Typically, examinations are supervised in an effort to prevent such copying. However, the success of such supervision is greatly dependent on the vigilance of the supervisor and employment of a supervisor usually incurs salary and training costs.

[0004] Recently, educational institutions have started to provide examinations via computers over a communications network. This can significantly reduce the overall cost of conducting an examination, However, whe providing an examination over a network to students at a remote location, it can be much more difficult to supervise the students and prevent the students from copying each other's answers. Consequently, it can be difficult to accurately assess the skill level of the students,

[0005] A further problem with examinations is that they usually require a teacher to spend a large amount of time to prepare the examination and mark each student's answer(s) to the examination. This incurs large monetary costs to the educational institutions.

[0006] The pressure to minimise the time and monetary limitations can result in practices that reduce the quality of the examination. For example, teachers often reuse questions from previous examinations or practice examinations. Consequently, students are examined on questions with which they are already familiar and to which they have often alread been provided with an answer.

[0007] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge.

OBJECT OF THE INVENTION

[0008] It is a preferred object of the embodiments of the present invention to provide a system and/or a method and/or an apparatus that addresses or at least ameliorates one or more of the aforementioned problems of the prior art and/or provides a useful commercial alternative. SUMMARY OF THE INVENTION

[0009] Generally, embodiments of the present invention relate to examination, also known as testing or assessment, and in particular relate to examination of a group of users via a computer network.

[0010] According to one aspect, but not necessarily the broadest aspect, the present invention resides in an electronic method of examining a group of users, the method comprising:

receiving at a server, a plurality of questions and a plurality of question groups in which the questions are distributed;

determining via the server or receiving at the server, a selection of the question groups for use in an examination;

determining via the server for each user in the group of users, a selection of questions from each selected question group;

generating via the server for each user in the group of users, an examination comprising the selected questions; and

transmitting the examinations to the respective users in the group of users for completion.

[0011] Suitably, the selection of the question groups and/or the questions is a random selection that is determined via the server.

[0012] Suitably, the selection of the question groups and/or the questions is a predetermined selection that is received at the server,

[0013]Suitabiy, the electronic method comprises setting via the server according to a predetermined value, a maximum number of question groups and/or questions to be selected for each examination.

[0014] Suitably, the electronic method further comprises flagging one or more questions within a question group as being for a practice examinatton and flagging one or more questions within a question group as being for a real examination.

[0015] Suitably, the electronic method comprises flagging the questions as being associated with one or more levels of skill and basing the selection of questions on a skill level of the respective user.

[0016] Preferably, the questions in a question group are equivalent.

[00 7] Preferably, the electronic method further comprises receiving at the server completed examinations from the users.

[Q018]Suttably, the electronic method comprises marking the completed examinations electronically.

[0019] Suitably, the electronic method comprises providing a graphical user- interface enabling a teacher to mark the completed examinations.

[0020] Preferably, the electronic method comprises generating via the server statistics for each question group from the marked completed examinations.

[0021] Preferably, the electronic method comprises providing a graphical user interface displaying the marks of each user for the examination and enabling a teacher to select and view each user's answer to each question.

[0022] Preferably, the electronic method comprises providing a graphical user interface enabling a teacher to generate the plurality of questions and distribute the questions into question groups.

[0023]Suitabiy, the electronic method comprises receiving at the server an answers to a question of the examination when a user completes the question and proceeds to the next question.

[0024]Suttably, the electronic method comprises storing the answer in the form of global variables and receiving the answer at the server as one or more text strings generated by converting the global variables.

[0025] Suitably, the electronic method comprises receiving at the server a recording of one or more user inputs, such as mouse clicks, touchscreen inputs and/or keystrokes, made by the user while the user was answering a question.

[0026] Suitably, the electronic method comprises providing an animation of the user answering the task, for example, to a teacher using the recording, [0027] Suitably, the electronic method comprises providing a graphical user interface including one or more buttons to control the animation.

[0028] Suitably, the recording comprises one or more text strings.

[0029] According to another aspect, the present invention resides in a computing apparatus for examining a group of users, the computing apparatus comprising:

a processor;

a communications device coupled to the processor and a communications network; and

a memory coupled to the processor, the memory storing computer program code components which are executed by the processor to effect:

receiving at the computing apparatus, a plurality of questions and a plurality of question groups in which the questions are distributed;

determining via the processor or receiving at the computing apparatus, a selection of the question groups for use in an examination; determining via the processor for each user in the group of users, a selection of questions from each selected question group; generating via the processor for each user in the group of users, an examination comprising the selected questions; and

transmitting the examinations from the computing apparatus via the communications network to the respective users in the group of users for completion.

[0030] Further aspects and/or features of the present invention will become apparent from the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

[0031 ] In order that the invention may be readily understood and put into practical effect, reference will now be made to embodiments of the present invention with reference to the accompanying drawings, wherein like reference numbers refer to identical elements. The drawings are provided by way of example only, wherein:

[0032] FIG 1 is a schematic diagram of a system for examining a group of users in accordance with embodiments of the present invention;

[0033] FIG 2 is a schematic diagram of a computing device for use in the system shown in FIG 1 ;

[0034] FIG 3 is a schematic diagram of a method that can be performed in the system shown in FIG 1 ;

[0035] FIG 4 is an exemplary graphical user interface for providing an examination question to a user in accordance with embodiments of the present invention; [00361 FIG 5 is an exemplary graphical user interface for choosing a set of examination questions to review in accordance with embodiments of the present invention;

[0037] FIG 6 is an exemplary graphical user interface for review a user's answers to, and/or marks for, a question in accordance with embodiments of the present invention; and

[0038] FIG 7 is an exemplary graphical user interface for reviewing the results of a group of users in accordance with embodiments of the present invention, [0039] Skilled addressees will appreciate that elements in the drawings are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the relative dimensions of some elements in the drawings may be distorted to help improve understanding of embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0040] Embodiments of the present invention relate to examination, also known as testing or assessment, and in particular relate to examination or testing of a group, such as a grou of students, via a computer network. For example, the group of users in a subject, such as a class in a school or a subject in an out of school online tutoring organisation with teachers and students all over the world.

[0041] FIG 1 is a schematic of a computer-implemented system 100 in which an electronic method for examination of a group can be implemented. In the system 100, a computing device 102, hereafter called a server, is coupled to a communications network 1 10. The server provides aspects of an electronic method for examination of a group. The server 102 stores content and/or testing software which provides a "blank canvas" to which course material and examination material can be added. The software can be used on the server via the communications network 110 or downloaded to a computing device coupled to the communications network 1 10 for use. The use of a blank canvas enables the software to be language independent. That is, despite the internal programming code of the software being in English, the software will still work in other languages. For example, examination questions can be written in Chinese for a Chinese language test. A student provides an answer in Chinese characters and this answer is compared to Chinese characters that have been input into and stored in the software. If these characters match, then the answer is marked correct.

[0042] A first computing device 104 is coupled to the communications network H Q to enable a teacher to prepare, mark and review examinations for a group. The teacher can generate a plurality of questions for examinations and distribute the questions into question groups. The teacher can generate and distribute the questions using the downloaded software on the first computing device 104 or the teacher can generate and distribute the questions using the software on the server 102 via the communications network. The teacher can amend the questions or add further questions via the software at any time. The teacher can connect to the server 102 via the first computing device 104 to mark completed examinations, select questions and/or question groups for use in examinations, and review marked questions, examinations and/or statistics for the group. The teacher can save the current state of the questions or exami ations at any time. [0043] A second computing device 106 is coupled to the communications network 1 10 to enable a user, such as a student, to receive an examination, complete the questions in the examination, and transmit the answers to the questions to the server 102 for marking. Once the questions have been marked, the student can access their marks for each question on the server 102 via the second computing device 106.

[0044] With reference to FIG 2, a computing apparatus 102 to provide aspects of an electronic method for examination of a group of users is provided according to another aspect of the present invention. The apparatus, previously referred to as the server 102, stores computer program code components 208 on a computer readable medium 206 at least some of which are selectively executed to provide examination or testing of a group of users in accordance with embodiments of the present invention as described herein. The apparatus will usually comprise a processor 202, a memory 204 coupled to be in communication with the processor 202 and a communications device 210 in communication with the processor. The memory 204 stores computer program cod components 208 at least some of which are selectively executed via the processor 202 to perform aspects of the present invention as described herein, for example, to transmit or respond to the receipt of examination related requests via a communications network, such as the communications network 1 10, connected to the apparatus 102.

[0045] in a preferred embodiment, the memory 204 stores computer program code components which are executed by the processor 202 to effect the following: receiving at the computing apparatus, a plurality of questions and a plurality of question groups in which the questions are distributed; determining via the processor or receiving at the computing apparatus, a selection of the question groups for use in an examination; determining via the processor for each user in the group of users, a selection of questions from each selected question group; generating via the processor for each user in the group of users, an examination comprising the selected questions; and transmitting the examinations from the computing apparatus via the communications network to the respective users in the group of users for completion.

[0046J With reference to the first and second computing devices (104 and 106} of the present invention, examples of such computing devices include, but are not limited to, personal, laptop, netbook and notebook computers, personal digital assistants (PDAs), portable multimedia devices, mobile telephones and other computing devices,

[0047] The computing devices comprise a processor operatively coupled to one or more input devices, such as a keyboard, mouse, touch screen, controller and/or other suitable input device. The processor is operatively coupled to one or more output devices, such as a screen, printer and/or other suitable output device. The processor is also operatively coupled to a storage medium in the form of memory. The computing devices further comprise a well known communication means to enable the computing device to communicate with other computing devices via wired and/or wireless data connections.

[0048] FIG 3 is a flow diagram of an electronic method 300 of examining a group of users, such as students, according to one embodiment of the present invention.

[0049] At step 302, a plurality of questions and a plurality of question groups in which the questions are distributed are received at the server 102. The questions and question groups can be received from the software on the first computing device 104, or the questions and question groups can be received by being created on the server 102 using tools provided over the communications network 1 10. fn some embodiments, a graphical user interface is provided enabling a teacher to generate the pluralit of questions and distribute the questions into question groups. The questions in a question group may all be on the same topic or the same type of question. For example, all questions in a question group might relate to capital cities or presidents of the United States. The questions in a question group can be equivalent to enable the questions in a question grou to be used interchangeably in an examination.

[0050] At step 304, a selection of the question groups for use in an examination is determined via the server 102. The selection of the question groups can be a random selection or a predetermined selection. For example, the selection of the question groups can be received from a teacher via the first computing device 104.

[0051] At step 306, a selection of questions from each selected question group for each user in the group is determined via the server or received at the server. The selection of the questions can be a random selection that is determined via the server or a predetermined selection that is received at the server. For example, the selection of the questions can be received from a teacher via the first computing device 104.

[0052] in some embodiments, at step 304, the electronic method further comprises setting a maximum number of question groups, questions and/or questions in each question group to be selected for each examination. The maximum number can be set via the server 102 according to a predetermined value, such as a predetermined value received from the first computing device 104. Subsequently, a predetermined number of question groups, questions and/or questions in each question group can be randomly selected.

[0053Jin some embodiments, at step 304, the electronic method further comprises flagging one or more questions within a question group as being for a practice examinatio and flagging one or more questions within a question group as being for a real examination. The selection of questions in the question group is then limited to the chosen examination type. This can prevent questions that are used in practice examinations from being reused in real examinations.

[0054] In some embodiments, the electronic method comprises flagging the questions as being associated wit one or more levels of skill The selection of the questions and/or the question groups is then based on a skill level of the respective user.

[00551 At step 308, for each user in the group an examination is generated via the server. The examinations comprising the selected questions for each respective user.

[0056] At step 310, the examinations are transmitted from the server 102 to the computing devices 106 of the respective users in the group for completion. The users can complete the examination questions via a graphical user interface (GUI), such as the GUI shown in FIG 4,

[0057] At 312, answers to questions or completed examinations are received at the server from the second computing devices 106 of the users. [0058] In some embodiments, the user's answers are held in global variables on the server 102 or the second computing device 106 until a button is pressed to submit the completed examination. When the button is pressed the user's answers are converted from global variables into text strings and sent to a database as text strings.

[0059] In some embodiments, the text string for the current question is sent to the database when the user proceeds to the next question, for example, by clicking a button. In the advent of an interconnection or software failure, this enables the user to continue the examination from the last completed answer.

[0060] In some embodiments, a recording of one or more user inputs, such as mouse clicks, touchscreen inputs and/or keystrokes, made by the user are received at the server. The one or more user inputs of the user are recorded while the user is answering the questions. For example, a time of the user inputs, a location of a mouse click, a mouse button clicked and/or a key pressed can be recorded. The one or more user inputs that are recorded are converted into one or more strings and transmitted from the second computing device 106 to the database. For example, the one or more strings can be text strings, such as an alpha numeric string or a text only string. A single string can be transmitted for each task.

[0061] In preferred embodiments, each user inputs is sequentially recorded and stored into the string. At the end of each question, task or set of tasks, the string is transmitted to and stored in the database. For example, the string is transmitted when the user presses the send/mark button.

[0062]At 31 4, the completed examinations are marked. In some embodiments, the completed examinations are marked electronically. For example, multiple choice and/or short answer questions can be automatically marked by the server 102. The questions can be marked automatically at the completion of each question or at the completion of the examination.

[0063] in some embodiments, a graphical user interface is provided enabling a teacher to mark the completed examinations or parts of the completed examinations. The graphical user interface can be provided by software on the first computing device 104 which receives the completed examinations from the server 102, or the graphical user interface can be provided to the first computing device 104 over the communications network 110 by the server 102.

[0064] in some embodiments, the user's answers are retrieved for review from the databas as text strings and used to recreate the user's graphical answer as seen during examination. As the teacher marks each question, the results of marking th question are sent to the database.

[0065] In some embodiments, an animation of the user answering the task is provided to the teacher using the recording. The teacher can retrieve the one or more user inputs of a user from the database, for example, to determine how the user went about answering the task. In preferred embodiments, the one or more recorded user inputs are presented to the teacher, for example, along with the recreated graphical answer of the user.

[0066] in some embodiments, graphical user interface is provided including one or more buttons to control the animation. The one or more buttons can be selectable navigation icons to navigate through the user's answer, for example, by time or by user input event. The navigation icons can be similar to the buttons on a CD playe or tape recorder, for example, a "Play" icon, a "Pause" icon, a "Fast forward" icon, a "Rewind" icon and a "Stop" icon can be provided. The teacher can review a video animation of the user's answer using the selectable navigation icons. For example, pressing a selectable icon converts the string to an animation of the user answering the task, which is presented to the teacher in real time, slow motion or fast motion, in such a way, the server or the first computer can recreate the graphics of a user's answer for quality control purposes and/or to address any disputes.

[0067] In some embodiments, at any stage during the playing of the animation, the teacher can pause and edit the user's animation. For example, the teacher can add further user inputs to the animation. The further user inputs can be added to the string and transmitted to the database, for example, as an edited version of the answer.

[0068] The user can then access and view an animation of their original user inputs and/or an animation of the edited version of the user inputs for comparison to the original. In such a way, the teacher can leave feedback for the user, by showing, for example, what the user should have done when answering the task. In some embodiments, the animation and/or the questions are provided aurally, for example, to enable visually impaired users to complete the examination and/or to provide feedback to visually impaired users.

[0069] In some embodiments, when the user proceeds to the next question, a notification is sent to the first computing device 104. For example, the server 102 records that a user has completed a question and transmits a message "user(s) X has completed question(s) Y" to the first computing device 104. The teacher can then review each user's answer animation and amend the animation to provide feedback for the user. For example, feedback can be provided while the user completes the next question, in some embodiments, the feedback can include a mark or series of marks for the questionfs), However, the feedback may not include a mark.

[00701 in some embodiments, a help button is provided to the user while the user is completing the questions}. The user's can press the help button to notify the teacher that the user needs help, for example, if the user is having difficulties with a question. In preferred embodiments, pressing the help button transmits a request for help to the teacher, including the user's answer to the question so far, for example, the video animation of the user's answer so far. In some embodiments, the user is also presented with a comment section, where the user can provide details of a problem they are having. Comments provided by the user in the comment section are transmitted with the request for help to the teacher.

[0071] In some embodiments, a user can also transmit a help request to a group forum. Another user in the group can then respond to the help request. In some embodiments, a user can also transmit a request for help to a user outside the group, for example, the user can request help from a peer. In these embodiments, the teacher can be notified, for example, that user A is now helping user B. All user inputs of user A and/or user B can be recorded for later review by the teacher or either user.

[0072] in some embodiments, a graphical use interface displaying the marks of each user for the examination is provided. The graphical user interface enables a teacher to select and view each user's answer to each question. In some embodiments, statistics are generated via the server for each question group from the marked completed examinations,

[0073] in some embodiments, the teacher can send a request from the first computing device 104 to the server 102 to update the questions, groups or examinations at any time. For example, if a question becomes redundant or the answer changes the teacher can update the correct answer to be a new answer.

[0074] in some embodiments, a user can send a request from the second computing device 104 to the server to select questions or groups for an examination, or delete questions or groups from an examination.

[0075] FIG 4 shows an exemplary graphical user interface (GUI) 400 for providing an examination question to a user according to one embodiment of the present invention. The GUI comprises a question 410 and a means 420 to answer the question 410. In this example, the examination question is a multiple choice question and the means 420 comprises a list from which a user can choose one or more answers 422. The one or more answers 422 can be selected by a user via check boxes 424.

[0076] A button 430 can be clicked to proceed to the next question of the examination. The question is marked automatically when the button is clicked. A progressive tally 440 is provided comprising the number of questions answered correctly by the user in the examination thus far and the totai number of questions in the examination, in alternative embodiments, the progressive tally 440 comprises the number of questions answered correctly by the user and the number of questions the user has answered thus far.

[0077] in some embodiments, the GUI 400 can comprise a drag and drop functionality and drawing toolbar. For example, a trigonometry question might ask a user to draw an equiiateral triangle, and the user could select the line tool to draw the triangle. In such an embodiment, the question can be marked by a teacher, or marked correct automatically if the triangle is drawn within predetermined thresholds.

[00781 in such an embodiment, the position of every bitmap and text can be stripped from the user's answer and converted to a string to copy the answers, but in a list format. The name of the graphic and its location is thus copied and transmitted to the database. This avoids the need to send a large graphic file that would take a long time to upload. An exact copy of all the graphics used for the examination is stored in the database. The server can then recreate the graphics of users' answers for review and/or marking,

[0079] FIG 5 shows an exemplar graphical user interface (GUI) 500 for choosing a set of examination questions to review according to one embodiment of the present invention. The GUI 500 comprises buttons 510 which can be clicked to review the user's answers and/or marks for a specific set of questions from the examination. A first button 512 can be clicked to review all of the user's answers to the examination questions. A second button 514 can be clicked to review all of the user's incorrect answers to the examination questions. A third button 516 can be clicked to review all of the user's correct answers to the examination questions. An example of a marked answer to the question 410 is shown in FSG 8.

[0080] FIG 6 shows a graphical user interface (GUI) 800 for reviewing a user's answers and/or marks for a question according to one embodiment of the present invention. GUI 600 comprises the same or a similar layout to GUI 400, and an indicium 640 to illustrate whether the user has answered the question correctly or incorrectly. The user has selected the check bo 622 which is not the correct answer. The indicium 640 is therefore a cross to indicate that an incorrect answer has been selected. In this embodiment, a correct answer is indicated by a tick. However, it is envisioned that other symbols that are commoniy used to represent that an answer is correct or incorrect may be used. A button 630 is provided to proceed to the next question for review. Depending on the choice of the user in GUI 500, the next question may be the next question in the examination, the next incorrectly answered question in the examination or the next correctly answered question in the examination.

[0081 J FIG 7 shows an exemplary graphical user interface (GUI) 700 for reviewing the results of a group according to one embodiment of the present invention. The GUI 700 comprises the name of a group 704, and a table 702 comprising the names 710 of the user in the group 704, the users' results 740 for each question in the examination, and the user's total score 750 for the examination. Each result in the table comprises a link 744 to enable the respective answer to be reviewed. The link 744 can take the user to a GUI such as the GUI 600 shown in FIG 6.

[0082] Attributes 712 of each user, the date of the test 720 and the time taken 730 by each user to complete the examination are also shown in the table 702. The attributes include the user's date of birth (DOB), gender, cultural group and whether the student is involved in instrumental music. However, other attributes that are relevant to a user's skill or expected skill level can also be included. The table comprises an age 760 of each user, and a calculated equivalent age 770 of each user that is based on one or more of the results 740» total scare 750, time taken 730 and attributes 712 of the user. In some embodiments, for example, where the questions or groups are flagged by skill levels, the equivalent age can be based on the results for questions in each group. The age 760 and/or the equivalent age 770 can be calculated via the processor 202 of the server 102, the first computing device 104 and/or the second computing device.

[0083] Hence, embodiments of the present invention provide custom built, computer and/or teach marked examinations. A teacher can automatically generate different examinations for each user in a group and distribute the examinations to the users. Specialised software is provided for preparing the examination questions efficiently and later editing the questions. The questions can be marked automatically.

[0084] Embodiments of the present invention thus benefit teachers and students by streamlining the process of conducting examinations and by more accurately assessing the students. This results in lower time and monetary costs and a simpler, higher quality examination process.

[0085] Furthermore, the present invention can assist in preventing copying in examinations by generating examinations with different selections of questions for each user and increase the quality of assessment via examinations by not reusing questions that have previously been presented to a user.

[0086] in this specification, adjectives such as first and second, and the like may be used solely to distinguish one eiement or action from another element or action without necessarily requiring or implying any actual such relationship or order. Where the context permits, reference to an integer or a component or step (or the like) is not to be interpreted as being limited to only one of that integer, component, or step, but rather could be one or more of that integer, component, or step etc.

[0087] In this specification, the terms "comprises", "comprising" or similar terms are intended to mean a non-exclusive inclusion, such t at an apparatus that comprises a list of elements does not include those elements solely, but may well include other elements not listed.

[0088] Throughout the specification the aim has been to describe the invention without limiting the invention to any one embodiment or specific collection of features. Persons skilled in the relevant art may realize variations from the specific embodiments that will nonetheless fall within the scope of the invention.