progress report (2) Page 1 of 14 11/02/2008 Progress Report – Development of A unique student learning aid (Also as “THE ITIOT V. 2.0”) RE: Teaching and Learning Development Fund –2007 Improving Student Satisfaction at UWA (ISS) Scheme The School of Anatomy & Human Biology received $ 18,000 to design and develop a unique student learning aid known as “The ITIOT V. 2.0” The project was assigned a project leader (Guy Ben-Ary) and was initiated in August 2007. The implementation of the project was divided into 3 main parts: 1. Performing a need analysis and writing up a design document describing various aspects (features specifications and technical issues) of the project. 2. The implementation phase (coding and interface design) 3. Testing & debugging. The first stage of the project was signed off at the beginning of 2008 and now moved to its second stage of coding. This stage is due to begin only at the end of February 2008 due to issues related to access to the appropriate programmers. It is planned that the second stage will end by June 2008. Then the testing phase will start and the final product will be delivered as scheduled by September 2008 Please find attached to this document a copy of the design document that summarises the needs analysis process for this application. Regards Guy Ben-Ary Manager CELLCentral School of Anatomy & Human Biology progress report (2) Page 2 of 14 The re-developed ITIOT Specification 12/12/2007 Investigated and Composed by Guy Ben-Ary & Nguyen Ly (CELLCentral) The Original ITIOT application was designed and developed by Dr. Lutz Slomianka General................................................................................................................................ 3 Access to the ITIOT............................................................................................................ 3 The Sign Up/Log In Features.............................................................................................. 4 Units, Practices and topics .................................................................................................. 5 The exam............................................................................................................................. 6 Tracking function................................................................................................................ 6 Managing the statements..................................................................................................... 7 Description of the Structure of Knowledge Items used by the Itiot ................................... 8 1. The Source .................................................................................................................. 9 No Source.................................................................................................................... 9 Special use of the source............................................................................................. 9 The Source in the Knowledge Item .......................................................................... 10 2. The Possible Correct Answers Array........................................................................ 10 The Possible Correct Answers Array in the Knowledge Item.................................. 10 3. The Question Type.................................................................................................... 11 The Question Type in the Knowledge Item.............................................................. 11 4. The Subject Area....................................................................................................... 11 Special use of the subject area .................................................................................. 11 The Subject Area in the Knowledge Item................................................................. 12 5. The Absolute Drop Value ......................................................................................... 12 The Absolute Drop Value in the Knowledge Item ................................................... 13 6. The Non-Self Subject Area Permission Probability - short: Non-Self Permission .. 13 The Non-Self Permission in the Knowledge Item .................................................... 14 How does a "script-ready" knowledge item look like ?................................................ 14 progress report (2) Page 3 of 14 General The main purpose of this document is to describe in detail the required redevelopment of the ITIOT application. The ITIOT is a unique student learning aid that is used extensively in first year Human Biology and has features not found in any other commercial product. It was developed By Dr. Lutz Slomianka in 1999. Back then he developed the ITIOT using multiple static pages embedded with java scripts. Although the engine and technology (that this application is based on) needs to be replaced and re-assessed, the logic the application was based on is unique and the new application will duplicate it. The ITIOT allows students to learn through practicing multiple-choice questions. The uniqueness of this system resides in the way it generates multiple choice questions. In essence, the system uses statements rather questions as the core dataset. Each statement (or image) is considered to be a “Knowledge Item”. Questions are constructed from a selection of statements based upon their characteristics – the knowledge item’s properties. These properties including subject categories which permits questions to be constructed where all the statements are appropriately inter-related. It also has the capacity to give prior probabilities for the use of statements, depended on weather the statement is the correct or incorrect answer. Statements can also have a graphics file associated with them that is displayed with the statement so, for example, laboratory- based information can be easily in cooperated in the learning experience and computer based assessment. In general, the following enhancements are required to the current application: Re-writing the application as a window based software for robustness, efficiency of code, unity of usage etc. Develop a solid and well designed “technological back end” including a thorough administration interface. Adding feedback to the quiz questions Flexibility in the number of statements for each question Import statements Export quizzes Complexity in subject categorization Access to the ITIOT The old version of the ITIOT had 2 formats: 1. Practice mode – the ITIOT runs under the web CT platform. The practices are unlimited due to the special structure of the ITIOT. All the student needs to do is to choose a practice and/or a sub practice and start the exercise. progress report (2) Page 4 of 14 2. When the students want to take the compulsory test, they need to do it via the computers in the Histology area. The ITIOT today is a collection of scripts they are distributed between multiple computers and servers. This fact poses severe maintenance problems. The new version will be depended on one central engine and the quizzes could be access via different security access in various locations. The Sign Up/Log In Features The idea here is to detach the ITIOT application from the central student administration of UWA. The ITIOT will deal with the registration of its users independently from other applications on campus. Students will be given the URL of the ITIOT and be asked to fill in the registration form. The fields for this form will be: 1. Name 2. Student Number 3. E-mail Please note: Students might be enrolled to one or more practices. For each practice (Practice means a whole set of quizzes usually set per University Unit) the administrator could set up the type of registration: 1. Open registration – Students just go and register without any need of confirmation 2. Restricted registration - Students register and then a request is sent to the manager of the quiz to approve their registration 3. Batch Registration - The student registration requests would be checked against a file that will be uploaded prior to student’s registration. a. This file will have the list of students allowed to use the ITIOT and will be formatted in a pre-defined format (Tab Delimited – see appendix for a sample of a file as such). b. The names of the students who do not appear in this file will be sent to the manager of the practice. These students will be asked to see the manager to sort the registration out.. c. The manager of the practice will be able to add manually students if needed. d. The file with the lists of students change on a weekly basis in the first couple of week of semester. Therefore the list will be replaced when a new one is generated. Once a new list is uploaded the old one can be deleted and the “new sign ins” will be checked according to the new list. There is no need to compare between the 2 lists and delete the students that dropped from the unit. The deletion of students from the registered students will always be done manually by the manager of a practice/unit progress report (2) Page 5 of 14 Units, Practices and topics Every Unit (IE – HB102, HB100) has multiple practices assigned to it (see chart below – the HB 100 route). When designing the ITIOT the practice manager could divide it to multiple topics and each topic could then be divided into subtopics. When in practice mode the student could choose if he/she would like to have a quiz covering all of the topics and/or one or more topics and/or one or more subtopics While taking the practice the ITIOT would indicate, immediately if the student answered the question correct. If he/she didn’t answer correctly the application would indicate what the correct answer is. Students could see only the practices that they are registered too. Sub topic 1 Sub topic 2 Sub topic 3 Practice – HB101-2007 Topic 1 Practice – HB101-2007 Topic 2 Practice – HB101-2007 Topic 3 ITIOT Practice – HB100-2005 Practice – HB100-2007 Practice – HB100-2006 HB100 HB101 HB102 progress report (2) Page 6 of 14 NEW FEATURE –FEEDBACK Item based feedback - In the new version of the ITIOT feedback will be given to the student why the answer to the question is wrong and why the correct one is correct. This will be achieved by adding a field- “feedback” to the Knowledge items. Which would be a non compulsory field and the practice manager could use it or not. Question type feedback – if the student got the answer wrong and when the question type allows it (correct/incorrect statements, true or false) the engine will generate another type of feedback – related to the question type – IE: “ please note that you are asked to choose the INCORRECT STATEMENT” The exam The exams would be available to be taken by all students between 2 dates. They would have to “hide” behind a special log in page. The students would be able to take the exams only in the histology lab – So a range of IP’s should be associated with the exams 1. The students should get their marks as soon as they submitted the exam. 2. Students should get an immediate feedback for the exams – show them which of their answers was wrong or right and why. 3. The tests should have a question & time limit (15 questions and they are required to answer them in 15 minutes). 4. An options of exporting Quizzes will be included in the management module – IE - 30 questions without repeats – forming a local quiz – Here there should be a work flow that will ask the use to input the desired number of questions for the quiz. Then define how many questions he/she wants from each topic/subtopic of the practice – 3 first questions from subject 1. 4 from subtopic 2.3 and so on. In the test we would like to allow the students to go back and edit their questions before they submit the test – review functions before submit Tracking function A tracking function will be needed for the ITIOT so that for every practice the administrators could get statistical data such as: What time each of the students logged into the practice How long they spend working each of the sessions How many questions they tried to answer What time were they using it progress report (2) Page 7 of 14 Managing the statements Each practice (quiz) will be normally assigned to a University unit (I.E. HB102). Each Unit could have multiple practices - (I.E. HB102-2005, HB102-2006, HB102- 2006/Singapore etc...) Each practice could be divided into topics and sub-topic (3-4 nodes of sub classifications) Each Unit and therefore subjected practices will have a manager assigned to it. The manager could access all the statements and all the admin tools. There would be multiple teachers each one of them in charge of 1 or more topics. The manager would be able to either “open a new practice” under a unit’s node and start from scratch: design the practice structure with topics and sub topics and assign them to teachers located in the Units Database. Or they could duplicate a practice I.E. HB102-2005/UWA to HB102-2006/Singapore the next years. In that case the manager could edit, after the duplication, details if necessary – I.E. new teachers or topics etc. If a new teacher is assigned to a new topic he/she would have to build it from scratch: add statements manually or write them as a text file and export them to the application. If it’s a topic that existed in previous years she could choose to use the “old statements” from previous years or input new statements. Teachers could see all the statements of other teachers in a certain Unit (multiple practices) and use them (or “borrow them”) if wanted. But the teachers won’t have access to other units and their attached practices. So units and practices will be assigned to a manager and statements and topics are assigned to teachers. The Upload facility - from text files with a predetermined structure – probably looks like that (the old format of the knowledge items): Knit("none", ["Natural selection is differential survival"], 1, -101, 1, 0); Knit("none", ["Natural selection always produces efficient organs"], 1, -101, 1, 0); Knit("none", ["Natural selection has foresight and purpose"], 1, -101, 1, 0); Knit("none", ["Natural selection always results in increased size"], 1, -101, 1, 0); Knit("none", ["Natural selection always results in increased complexity"], 1, -101, 1, 0); Knit("none", ["All evolution is due to Natural Selection"], 1, -101, 1, 0); Knit("none", ["Natural selection acts primarily for the good of a species"], 1, -101, 1, 0); Knit("none", ["Natural selection is the only evolutionary force"], 1, -101, 1, 0); The feedback would have to be added later and therefore some budget of this project should be kept for Data-entry activities especially of the feedback field of the knowledge items progress report (2) Page 8 of 14 A “block edits” facility needs to be in place where after the practice manager makes a test for the students on a topic or multiple topics, the managers can block the access of the teachers to these topics and prevent them from changing the statements The manager could export statements so she could send them to teachers to be checked. The teachers could input individual or groups of statements. It is very important that the workflow will be as simple as possible such as – Do you want to input new statements Choose a subject meter Choose the type of question Input the statements and feedback Most of the values of properties should be set to default and the teachers could choose to change them if its required. A good search facility for statements will be integrated so te manager could find faulty statements with an ease. Description of the Structure of Knowledge Items used by the Itiot The following description was written by LUTZ and is describing in detail the logic of the ITIOT engine, this logic will be adopted in the redevelopment of the application. The itiot generates questions based on "knowledge items" and their properties. As far as the itiot is concerned, there are no questions and no ways to manipulate questions because a question "is only there" once the itiot is finished with its' job. The only thing the itiot really knows about are knowledge items. They are the itiot's design principle. The itiot is actually pretty flexible as long as it is allowed to follow this principle. What the itiot can do depends on which properties you give to the knowledge items. Currently, each knowledge item has 6 elements which define the properties of the knowledge item and the behavior of the itiot. Questions are set by selecting a "correct" knowledge item and pairing it with three "wrong" knowledge items - "wrong" only in the sense that they can not be the correct answer to the question associated with the "correct" knowledge item. Otherwise, each of the "wrong" knowledge items may be selected to provide the correct answer for a subsequent question. This means that each knowledge item used by the itiot must be the correct answer to a question. The six elements which define the properties of a knowledge item are: 1. the source progress report (2) Page 9 of 14 2. the possible correct answers array 3. the question type 4. the subject area 5. the absolute drop value 6. the non-self permission probability These six elements are combined into just one line of simple text. This line can be pasted directly into the itiot script. Only a few other alterations of the script are likely to be required to adapt the itiot to a new set of knowledge items. 1. The Source The source can but does not have to contain the directory path and filename of information associated with the knowledge item. It can be a picture or an html file. The information will be displayed in a utility window a few seconds after the question has appeared in the main window of the itiot. The itiot will assume that a question will be set in an html file if the source contains the sequence of letters "htm". In this case, the itiot will not display a question in the question field of the main window. "htm" should therefore not be part of the name of an image associated with the knowledge item. No Source If there is no information to be displayed, the source must contain the sequence of letters "none". Special use of the source The source is also used to determine if a knowledge item has "recently" been used as the correct answer to a question. Knowledge items can share a source, but the fraction of unique sources among all sources must be greater than the value of the "norepeatsfrac" variable set in the declaration of variables of the itiot script. The "norepeatsfrac" variable determines the number of uniques knowledge items used as correct answers in a sequence of questions. If there are 100 knowledge items and the "norepeatsfrac" variable is set to 0.5, the idiot will use 50 different knowledge items as correct answers in any sequence of 50 questions. By making the selection dependent on the source it is possible to ask for the same correct answer twice or trice or ..., but only if the "source context" is different. This feature can be overridden by saving the same source content multiple times but with different file names. ATTENTION! progress report (2) Page 10 of 14 If the fraction of unique sources is smaller than the "norepeatsfrac" value the itiot will eventually freeze the browser. The itiot will automatically add a unique identifier to the source if its value is "none". The itiot will not add a unique identifier if the value is e.g. "none4", "xnone", "nonebingo", "nun" etc. The Source in the Knowledge Item Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, "non-self" permission) ; 2. The Possible Correct Answers Array The possible correct answers array contains the possible correct answers to the question (see 3) associated with the knowledge item. Only the first entry of the array will be used as the right answer. Other elements are used to exclude possible wrong answers. Examples If the array looks like ["here"], "here" can be used as a right answer and anything different from "here" can be used as a wrong answer. If the array looks like ["there","here","everywhere"], "there" can be used as a right answer, but, in addition to "there", "here" and "everywhere" are excluded as possible wrong answers. Note that it is possible to construct 3 knowledge items which contain the following, functionally different correct answer arrays: ["there", "here", "everywhere"] ["here", "there", "everywhere"] ["everywhere", "there", "here"] or any combination of any number of the possible correct answers - the itiot will behave accordingly. Possible correct answers could also be letters, numbers, combinations thereof or a statement like "none of the other choices". The Possible Correct Answers Array in the Knowledge Item Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, "non-self" permission) ; progress report (2) Page 11 of 14 3. The Question Type The question type is an integer which assigns the knowledge item to a group. Wrong answers will only be drawn from knowledge items which belong to the same question type group as the knowledge item which supplied the right answer, i.e. the question type establishes an exclusive group of knowledge items. The question type value is also used to draw the "correct question" from the Qtype array of the itiot script. Note that it is the knowledge item which determines the question and not vice versa. Note that the same "written" question, e.g. "What is this?", can be associated with different question type values, e.g. 1, 2, 3 etc. and that multiple question types can be associated with a question type group if they are asked via html files. It is possible to create subgroups within the question type group (see 4). ATTENTION! Each question type group must contain enough knowledge items to form a question, i.e. at the very least 4. The browser will sooner or later freeze if a question type group with less than 4 knowledge items exists. The minimum number of knowledge items necessary increases if the possible correct answers (see 2) of a knowledge item exclude other knowledge items as possible wrong answers. The Question Type in the Knowledge Item Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, non-self permission) ; 4. The Subject Area The subject area is an integer which assigns the knowledge item to a subgroup of knowledge items within the question type group. How exclusive the knowledge item "behaves" depends on the "non-self permission" value (see 6). Special use of the subject area Question of the "Identify the correct/incorrect statement" type To identify this type of questions the itiot uses the sign of the subject area. If the itiot encounters a negative subject area value, it assumes that all knowledge items with the progress report (2) Page 12 of 14 same absolute value as the negative subject area value are assessed by this type of question. The sign of the subject area establishes exclusive sign groups of knowledge items within a given question type group. If the subject area value of the right answer is positive, the subject areas of all wrong answers must be negative and vice versa. Examples: If a knowledge item has the subject area value -4, the itiot will assume that all knowledge items with subject areas values of 4 or -4 are assessed by questions of the type "Identify the correct/incorrect statement". If the "non-self" permission value (see 6) is larger than 0, knowledge items with a subject area value of 4 do not have to exist, but there must be at least three other knowledge items of the same question type with positive subject area values or the itiot will sooner or later freeze the browser. If a knowledge item has the subject area value -4 and there exists a knowledge item with the subject area value 4, there must be at least 2 additional knowledge items with negative subject area values or the itiot will sooner or later freeze the browser. A kowledge item with a subject area value of 7 can not be assessed by this type of question if there is no knowledge item with a subject area value of -7. It is possible to assign negative and positive values to different absolute values of the subject area, e.g. subject area values of -1, -2, -3, 4, 5, 6 etc. In this case knowledge items with positive subject areas must be assessible by a question different from the "Identify the correct/incorrect statement" type. Such a question will have to be asked using an external source of the question (see 1). Knowledge items with negative subject area values and items with subject areas of the same absolute value will behave similar to "ordinary" knowledge items with regard to the "non-self" permission value (see 6). The Subject Area in the Knowledge Item Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, non-self permission) ; 5. The Absolute Drop Value The absolute drop value determines the likelihood with which the knowledge item is used if it is "hit" by the random number generated by the itiot. If the drop value is 0.1, the knowledge item will only be used one out of every ten times it is hit. progress report (2) Page 13 of 14 Example In the table used as an example below, there are two entries for "lynx". While it may be sensible to have those two entries it may not be sensible to ask for "lynx" twice as often as for any other animal or to have it appear twice as often as a wrong choice. The frequency of the use of "lynx" can be adjusted by setting its' absolute drop value to 0.5. ATTENTION! The absolute drop value determines the frequency of the use of a knowledge item as a correct answer and as a wrong answer. Decreasing the absolute drop value to 0 therefore means that the item is never asked for and that it will never be a wrong answer. The Absolute Drop Value in the Knowledge Item Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, non-self permission) ; 6. The Non-Self Subject Area Permission Probability - short: Non-Self Permission The non-self permission is, together with the subject area, the thing that makes the idiot tick ... sort of. If this value is 0, a possible wrong answer must be a member of the same subgroup (see 4) as the right answer. If this value is 0, the subgroup becomes an exclusive group as far as this knowledge item and only this knowledge item is concerned. The subgroup must therefore contain at least 4 knowledge items if one knowledge item of the subgroup has a non-self permission value of 0 (see 3). If this value is 1, a knowledge item will be accepted as a wrong answer regardless of its subject area. If the value of the 'non-self' permission is 0.33, the chance that any possible wrong answer belongs to a subject area different from the subject area of the right answer is 0.33. Since there are three wrong answers, one of them will be likely to belong to a subject area different from the subject area of the right answer. Example: correct answer subject area non self permission comment tiger 1 0 wrong answers will always be carnivores (subject area 1) progress report (2) Page 14 of 14 lion 1 0 wrong answers will always be carnivores (subject area 1) cougar 1 0 wrong answers will always be carnivores (subject area 1) lynx 1 0.1 wrong answers will almost always be carnivores (subject area 1) but other european animals (subject area 2) will appear with a likelihood of 10% (non-self permission 0.1) lynx 2 0.5 a second entry for lynx - this time it is grouped with European animals (subject area 2), but other carnivores (subject area 1) may still be "good" wrong answers - their frequency is here set to 0.5 (non-self permission 0.5) hedgehog 2 1 Well, a hedghog is sort of a carnivore and a European animal - free mix and match of all animals in the question (non-self permission 1) rabbit 2 0 wrong answers will always be European animals (subject area 2) deer 2 0 wrong answers will always be European animals (subject area 2) The Non-Self Permission in the Knowledge Item Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, non-self permission) ; How does a "script-ready" knowledge item look like ? Knit ("abc...", ["abc...", ....], integer, integer, value between 0 and 1, value between 0 and 1) ; In this scheme, the elements of the knowledge item are placed like this: Knit ("source", ["possible correct answers", ....], question type, subject area, absolute drop value, "non-self" permission) ;