WEBVTT 00:01:09.000 --> 00:01:09.900 BPangerang lands in NE Victoria 00:01:18.000 --> 00:01:18.900 Gadigal people of the Eora nation 00:01:21.000 --> 00:01:21.900 Dja Dja Wurrung (Central Victoria) 00:01:22.000 --> 00:01:22.900 I pay my respects to the Wurundjeri people 00:01:34.000 --> 00:01:34.900 Wangal people of the Eora nation. 00:01:34.000 --> 00:01:34.900 Ngunnawal people 00:01:37.000 --> 00:01:37.900 in Randwick, NSW, so same Gadigal lands as Raechel 🙂 00:03:19.000 --> 00:03:19.900 I keep losing Raechel's narrative. Is it just me? 00:03:21.000 --> 00:03:21.900 I keep losing Raechel's narrative. Is it just me? 00:03:46.000 --> 00:03:46.900 Good morning from Germany. I had some troubles to login via Firefox, but with Microsoft Edge it worked like a charm. 00:04:40.000 --> 00:04:40.900 Hazel, am using Chrome and its working just fine 00:05:26.000 --> 00:05:26.900 Thanks, Nina. May just be our internet connections then! I really appreciate the local response 🙂 00:07:59.000 --> 00:07:59.900 This is exactly what you would need to do in performing or auditing a bank reconciliation (I also teach accounting and used to practise!) so I can vouch for its authenticity! 00:08:34.000 --> 00:08:34.900 Calculating a drug dose based on a person's body weight 00:08:50.000 --> 00:08:50.900 Does not necessarily need to have a single correct answer. 00:09:20.000 --> 00:09:20.900 Design a data-acquisiton system to read a specific temperature sensor. 00:09:38.000 --> 00:09:38.900 Q: Raechel did you design this directly in Blackboard or use other software? 00:11:48.000 --> 00:11:48.900 Moodle has that sort of combined question type (if you can work out how to use it ;-)). 00:11:53.000 --> 00:11:53.900 Note - Moodle cloze question will allow a mix of selected and typed responses within the one question. 00:12:41.000 --> 00:12:41.900 see here https://docs.moodle.org/en/Embedded_Answers_(Cloze)_question_type 00:12:59.000 --> 00:12:59.900 Which one of the elements of authenticity have you used in the design process? 00:13:30.000 --> 00:13:30.900 there are also external services which allow the combination of answer types. We are using Mobius by Digitaled. Links into LMS 00:15:03.000 --> 00:15:03.900 I am always very creative in coming up with good exam questions (even in multiple choice format) when checking student solutions for assignments. 00:16:21.000 --> 00:16:21.900 That's one where it would be nice if the LMS could handle any order - but I don't konw any that could/ 00:18:45.000 --> 00:18:45.900 in Top Hat you can opt for the text answers not to be case sensitive 00:18:54.000 --> 00:18:54.900 Sally Jordan did some interesting research on automated marking of free-text answers (from around 2007). Was surprisingly possible. 00:19:00.000 --> 00:19:00.900 don't dropdown boxes have the same issue as MCQ, i.e. misguiding them by the wrong answers? 00:20:17.000 --> 00:20:17.900 With hundreds of students, many will ignore instructions and will get false wrong answers. 00:20:33.000 --> 00:20:33.900 E.g. http://oro.open.ac.uk/20868/" 00:23:08.000 --> 00:23:08.900 Calculated formula with randomised values. 00:23:17.000 --> 00:23:17.900 Fill in blanks 00:23:49.000 --> 00:23:49.900 cloze (we used that for a multi-part listening test in chinese) or drag and drop hot spot images e.g histology. 00:24:14.000 --> 00:24:14.900 in economics, authenticity is more about the question's content than the format 00:24:30.000 --> 00:24:30.900 The big users of automated online assessment for practice a the OU are Science, Maths and Languages. 00:25:30.000 --> 00:25:30.900 Some LMSs have statistics you can look at after the test is run, to see if they questions were really roughly equally difficult. 00:25:35.000 --> 00:25:35.900 is there a way on these LMS to rank questions according to their difficulty? 00:26:25.000 --> 00:26:25.900 equity of question pools by topic too. so each student get a fair topic coverage... so stratify the random pools by topic and difficulty. (called 'category' in Moodle quiz). 00:26:27.000 --> 00:26:27.900 I have set weekly formative assessment quizzes with my accounting students all through pandemic. About 15% using them! 00:26:47.000 --> 00:26:47.900 Huge amount of time to write the questions - different formats although not as fancy as Raechel's! 00:26:49.000 --> 00:26:49.900 @Ben Freyens yes realism is the main element. Cognitive challenge in the design is also another one and feedback and self-evaluation opportunity is central all through the process of authenticity. 00:26:58.000 --> 00:26:58.900 Getting students to create multiple response questions for each other 00:27:02.000 --> 00:27:02.900 It takes a lot of time to create these things the first year you teach the course. In future years, you can re-use, so it is much easier. 00:27:03.000 --> 00:27:03.900 how do you motivate students to take the quizzes? 00:27:06.000 --> 00:27:06.900 @Tim Hunt (OU) - I am interested in the automated online assessment for language; would you be able to share an example or link to the one at OU? 00:27:22.000 --> 00:27:22.900 Do your students do practice quizzes if there are no marks associated? Mine don't. 00:28:17.000 --> 00:28:17.900 Kerry I discuss how the process of self-test helps with learning and show them the evidence, and I also leave open so they can redo them. I also structure them as part of their weekly activities but it is hard to encourage the students who really need to do them to do them! 00:28:51.000 --> 00:28:51.900 I even ask them to pause video and then do questions so I've embedded them ,but it's so tempting not to do that and just watch the video through... 00:29:07.000 --> 00:29:07.900 we did one in an online class yesterday and most students did attempt it 00:29:31.000 --> 00:29:31.900 with some systems must do question before video continues 00:29:39.000 --> 00:29:39.900 some LMS will havs quiz stats by question you can check to look patterns. in Moodle there are quiz stats. 00:29:41.000 --> 00:29:41.900 yes, ours doesn't allow that unfortunately" 00:29:46.000 --> 00:29:46.900 @Darlyne - it is mostly pretty basic. Testing simple text entry. For Moodle, we have a human-marked record audio question type. Also for Moodle, there are a set of language tools (which require payment) called Poodll, which claim to have some auto-grading of speaking, but we have not used those. 00:29:56.000 --> 00:29:56.900 The Moodle quiz stats are really useful 00:30:17.000 --> 00:30:17.900 Will the slides be shared? 00:30:18.000 --> 00:30:18.900 see here https://docs.moodle.org/en/Quiz_statistics_report 00:30:23.000 --> 00:30:23.900 I usually make use of case studies for MCQs to align with the real world and not just testing content 00:30:25.000 --> 00:30:25.900 others may share info about other LMS 😉 00:30:41.000 --> 00:30:41.900 Do you have any examples of authentic exam questions from disciplines where there is a lower likelihood of a right-wrong answer, e.g. in the humanities? 00:30:41.000 --> 00:30:41.900 Thank you it was brilliant 00:30:45.000 --> 00:30:45.900 Thks Alice. I have tiny marks associated with weekly quizzes (1% for all of 12 of them) , but it seems to make a difference I also close access every week to keep them on track, but that is not always well received. 00:30:47.000 --> 00:30:47.900 @Darlyne basic technology - the usefulness of the learning experience depends on what the question author does with them. Don't be afraid to do cool things with basic tech - which I think was the message of this talk. 00:31:18.000 --> 00:31:18.900 That was really helpful, Raechel, thank you. Some great comments in the chat box too. One question, you cited Ibbett & Wheldon and Paxton, can you provide further details please. 00:31:18.000 --> 00:31:18.900 I'm using a real life case study that runs throughout the module, and set formative questions on this - bu not summative (yet). Details https://accountingcafe.org/2021/12/13/embedding-a-single-company-case-in-an-ifrs-course/ 00:31:25.000 --> 00:31:25.900 @Tim Hunt (OU) Thank you for the info and explanation. 00:31:42.000 --> 00:31:42.900 @Alice Shepherd (Univ of Leeds) I heard that we can embed quizzes in Microsoft Stream where they have to complete it before they can view the remaining video but not tried it yet. 00:31:57.000 --> 00:31:57.900 Thank you very interesting. 00:32:03.000 --> 00:32:03.900 Hi @Lisa Marshall UoLeeds UK ooh I'll have a look at that, thanks. 00:32:18.000 --> 00:32:18.900 @Alice Shepherd (Univ of Leeds) We do the same. All the summative assessments are based on that same scenario. 00:32:19.000 --> 00:32:19.900 Here is a talk of me about similar experiences in Moodle: "Experience with an Online Assessment in a Lecture about Fundamentals of Electrical Engineering" https://youtu.be/_64QhWNF-nA 00:32:46.000 --> 00:32:46.900 Authenticity to me means relating to the job tasks the graduate will eventually do in the workplace 00:32:54.000 --> 00:32:54.900 Thanks for the link @Mathias Magdowski #3 00:32:57.000 --> 00:32:57.900 Have you had any issues with student motivation in carrying out these quizzes? 00:33:50.000 --> 00:33:50.900 The workplace assumes that all students will move into a defined role 00:34:42.000 --> 00:34:42.900 have you published on this, and are there any other publications you'd recommend? 00:34:46.000 --> 00:34:46.900 Cognitive challenge in the design and link with real life problems id very important 00:34:53.000 --> 00:34:53.900 I am interested in the phd research. I get it that students with good course outcomes are correlated wit serious formative quiz attempts, but is that a casual relationship? That is the big question. Do you know? 00:35:10.000 --> 00:35:10.900 Thank you for the informative session Raechel. I've some more ideas to apply now. 00:35:31.000 --> 00:35:31.900 @Kerry or is it the keen students who would do well anyway who do the quizzes?!?! 00:35:50.000 --> 00:35:50.900 how closely aligned are the formative style questions to the exam? 00:36:08.000 --> 00:36:08.900 sorry last question, is there a way on these LMS to rank questions according to their difficulty? 00:36:10.000 --> 00:36:10.900 In film / drama, they talk about getting the audience to suspend their disbelief get the most out of the experience. When full authenticity is not possible in education, I wonder if we could usefully use a similar concept. It needs to be authentic enough that we can persuade the students to suspend their disbelief and go with it. 00:38:41.000 --> 00:38:41.900 2000 students in 1st year nursing course - 5 practice quizzes before main quiz - 1998 attempts - 1st quiz - down to 935 attempts in quiz 5 - all questions in categories in main quiz of 30 questions - students love practice quizzes 00:41:20.000 --> 00:41:20.900 Thank you Rachel and James, Very interesting. 00:41:22.000 --> 00:41:22.900 And that 'difficulty' that Moodle is showing may also relate to how well the question is designed - a poorly designed qn will be difficult but not because it is 'hard' 00:41:43.000 --> 00:41:43.900 For students used to 'authentic' virtual reality simulations will they 'suspend disbelief' for text based activities (Everyone happily used text based adventures originally) 00:41:54.000 --> 00:41:54.900 Thanks - that was very informative. And thanks to the organisers for timing this so we can join from the UK 😀 00:41:59.000 --> 00:41:59.900 thanks Raechel (and Matthew) - another very interesting session. 00:42:38.000 --> 00:42:38.900 Thank you for an informative session. 00:43:04.000 --> 00:43:04.900 have to duck - thanks for the session 00:43:11.000 --> 00:43:11.900 Thank you. 00:43:41.000 --> 00:43:41.900 gotta go, thank you for the presentation. 00:43:59.000 --> 00:43:59.900 very true - MedEd in UK very wedded to SBA MCQ format - some schools flirting with VSAQs but some manual checking required 00:44:07.000 --> 00:44:07.900 Do you have experience with students taking MCQ exams as open exams (with resources, materials to consult)? 00:45:43.000 --> 00:45:43.900 I mean many times they are used as summative assessments, at the end of the semester, and without material. This add a lot to anxiety and stress. Would the use of material reduce stress? 00:46:37.000 --> 00:46:37.900 Thanks very much 00:46:46.000 --> 00:46:46.900 @Monica2 - we found during first lockdown that by reducing time in the mcq, an open book format performed very similarly to our closed book exams 00:46:51.000 --> 00:46:51.900 Thanks for the sharing 00:46:59.000 --> 00:46:59.900 Can you use peer marking for things that cannot be automatically marked? 00:47:17.000 --> 00:47:17.900 Thank you! Very interesting. 00:47:43.000 --> 00:47:43.900 Fantastic session - thank you all. Sorry, I have to leave now, but really glad to have joined you this morning. 00:48:20.000 --> 00:48:20.900 Thanks, Kevin, interesting. I was assuming open exams would reduce anxiety. 00:48:42.000 --> 00:48:42.900 In Moodle at least, peer-marking is a different sort of activity, with different workflow for the students 00:48:56.000 --> 00:48:56.900 The "Peer Review" plug-in of FeedbackFruits can be included into many learning management systems: https://feedbackfruits.com/peer-review 00:50:02.000 --> 00:50:02.900 We use PeerScholar in Blackboard 00:50:53.000 --> 00:50:53.900 you can also mark the peer marker 00:51:00.000 --> 00:51:00.900 We like Gradescope, especially for STEM subjects. Lots of flexibility in terms of formats of questions and responses. Students can also scan in their written papers with their phone, fill in a form that links their question numbers to their written pages, then upload. Lecturers can mark the whole batch of student papers by question and add reusable comments, also allocating marks as they go. It plugs into Canvas 00:51:12.000 --> 00:51:12.900 Thank you for the session. 00:51:14.000 --> 00:51:14.900 Thank you. Interesting talk. 00:51:20.000 --> 00:51:20.900 tHANKS HEAPS 00:51:21.000 --> 00:51:21.900 Many thanks - plenty of food for thought! 00:51:26.000 --> 00:51:26.900 Thank you 00:51:28.000 --> 00:51:28.900 Thanks for your insights! 00:51:33.000 --> 00:51:33.900 Thank you 00:51:38.000 --> 00:51:38.900 Thank you. This was very interesting and helpful! 00:51:47.000 --> 00:51:47.900 Thank you.