Ghizzi 5:00 PM Good morning from Bristol UK ;) NUS 5:01 PM Good afternoon from Singapore! Iris Nube (IB) 5:03 PM Good morning from the Hague, the Netherlands Alexander [FU Berlin] 5:04 PM Good morning from Berlin :-) Trevor Meek - U of Bradford UK 5:04 PM Good morning all from Bradford, UK. NUS 5:05 PM Hi, Alexander, jeffery here! how r u? Alexander [FU Berlin] 5:05 PM hey Jeffery :D doing fine here. How do you do? Tim Hunt 5:06 PM Hellow from Milton Keynes, UK Daniel R. Schneider 5:07 PM Hello from Zurich, Switzerland Mathew Hillier 5:07 PM Greetings late comers! please use : Tools > Audio > Audio set up wizard Mathew Hillier 5:08 PM lecturers already complaining about student's handwriting in exams here quite a lot :-) Alexander [FU Berlin] 5:08 PM same here ;-) Mathew Hillier 5:10 PM i guess you run out of computer labs in short order .. if you have say 1500 in a first year engineering course! Remi Ayoko 5:16 PM I am good thank Mathew Hillier 5:17 PM please feel free to type in comments here! (to everyone!) 5:17 PM Remi Ayoko 5:17 PM Ok thanks Matthew! Rebecca Hamer 5:18 PM Were there other question formats tried, so other than MCQ? Tim Hunt 5:18 PM But, In principle you could use any question type that Moodle/QuesitonMark/... supports. Mathew Hillier 5:19 PM :-) Remi Ayoko 5:19 PM What kind of preparations did the students have Alexander [FU Berlin] 5:19 PM maximum: how many students can participate per round? Rebecca Hamer 5:20 PM I notice no mathematics calculation including the steps. Does any of the used platforms support this? Alexander [FU Berlin] 5:20 PM 300? excellent!!! Rebecca Hamer 5:20 PM But no steps included in the response, as it was MCQ. Mathew Hillier 5:21 PM what is the big screen for? Tim Hunt 5:21 PM If you are interested in computer-assessment of maths, see http://www.transformingassessment.com/events_7_november_2012.php and http://www.transformingassessment.com/events_3_april_2013.php Ghizzi 5:21 PM i was wondering about that too Mathew Hillier 5:21 PM thanks tim :-D back up location was a lab? 5:22 PM Tim Hunt 5:22 PM I am hoping that you can say more about reliability of Wi-Fi We hit that problem too. 10% problems is about what we experienced too. 5:23 PM Mathew Hillier 5:23 PM wow. i wouldn’t let someone reformat my laptop! i doubt students here would wear that Ghizzi 5:24 PM no cant see that working here either Rebecca Hamer 5:24 PM how high stakes were these exams. What do you offer students with problems? Do they have to resit? Mathew Hillier 5:24 PM you had loaner laptops? Alexander [FU Berlin] 5:25 PM how much time does it take to during the preparation to deal with those 10% laptop problems? Rebecca Hamer 5:25 PM So they got less time then, had to start over? Rebecca Hamer 5:26 PM 59% is not that bad actually Mathew Hillier 5:26 PM ok for focus group! if you offer free pizza helps too 5:27 PM Trevor Meek - U of Bradford UK 5:27 PM I agree in my experience <20% is norm post exam for feedback Alexander [FU Berlin] 5:27 PM what was the exact question you asked them in the survey? Rebecca Hamer 5:27 PM Yes, we have something like that, except for mathematics Remi Ayoko 5:27 PM Did the students tell you what they liked specifically about the exam? Rebecca Hamer 5:29 PM familiarity of the keyboard is important, especially if you are using uncommon characters (in other languages) Mathew Hillier 5:30 PM i have found the pre-exam testing/mock run is very important to filter out the dodgy laptops! Rebecca Hamer 5:30 PM That is a helpful tip, running a pre-exam. Tim Hunt 5:31 PM I would say that if you are experimenting with new technology, a pre-exam is essential, not just helpful. Mathew Hillier 5:31 PM wireless will be a limit for high def videos etc. 100+ all downloading a video will break just about any wireless network in a room. @tim agree!!! 5:31 PM bordom! 5:32 PM Rebecca Hamer 5:32 PM I mean instead of a one-on-one check... Alexander [FU Berlin] 5:32 PM @mathew: exactly. it even starts getting slower when using audio for language tests... Rebecca Hamer 5:33 PM Is full lockdown possible if you are not online, but onscreen? Remi Ayoko 5:33 PM Please can you comment on how you mark the exam? Mathew Hillier 5:33 PM Q Alex - that is why we are looking at bootable USB sticks as an approach. we can store multiple GB on the USB and not need networks. 5:33 PM John Dermo 5:34 PM That is an interesting point about the holding room. There was once a time when you could hold students in academic isolation with notes and books, but nowadays you can't deprive them of their devices, or they cannot revise other subjects. Mathew Hillier 5:34 PM store* Rebecca Hamer 5:34 PM thanks Daniel R. Schneider 5:34 PM attending the Apple developer conference WWDC in San Francisco in the past I know it is possible to provide very powerful WiFi for hundreds of users, even 500 - 1000 simultaneous users in a large hall. But you need a lot of hotspots and technicians who are very experienced Alexander [FU Berlin] 5:34 PM on the other hand: USB has other disadvantages when it comes to reliability... Tim Hunt 5:35 PM @Daniel, but it is not easy or cheap. Mathew Hillier 5:35 PM yes, agree each approach will have limits. Tim Hunt 5:35 PM If you know how to do it, please tell us :-) Ghizzi 5:35 PM did you require a minimum standard on their devices for the exam? RAM, graphics cards etc? Daniel R. Schneider 5:35 PM @Tim no, surely not cheap Mathew Hillier 5:36 PM we will hear from folks at UBC who do capture all transactions over their network. Alexander [FU Berlin] 5:36 PM after having tried several progressive approaches, we voted for a very conservative approach ;-) Tim Hunt 5:36 PM But, I guess it would not hurt to get an idea of the budget for what setting up Wifi at ADC is, then we would know how far away we are. Daniel R. Schneider 5:37 PM @Mathew: are the Linux USB solutions really compatible with Macs? I didn't attend the last webinar and I don't know if they used only PCs or also Macs Mathew Hillier 5:37 PM we have made it so. will boot most macs and most pcs using the same stick. 5:37 PM Tim Hunt 5:37 PM I love those three "What do we want" choices. Great summary Mathew Hillier 5:37 PM we all want to know! Ghizzi 5:37 PM yep me too hehe Alexander [FU Berlin] 5:37 PM Thanks!!!! R.NAIDOO 5:38 PM HOW EXPENSIVE IS THE TECHNOLOGY Daniel R. Schneider 5:38 PM my experience with Linux on Macs is that there are often problems with keyboard or the trackpad (as Apple's trackpads are behaving different as on most standard PCs...) Tim Hunt 5:39 PM You said you had 10% technical problems. What are your feeling about that? We had a similar level at the OU and thought it was 'too high'. I wonder what our target reliability should be? Daniel R. Schneider 5:39 PM Thanks for that information Kelly Hall (Edinburgh University) 5:39 PM How did you find the space to fit 300 people? Did the University work with your to refurbish a space for the exams? Or was a space already fit for purpose? Rebecca Hamer 5:39 PM @Tim, we would need to agree that 10% is way too high Alexander [FU Berlin] 5:40 PM How much time does it take to deal with those 10% laptop problems? Rebecca Hamer 5:40 PM I was still wondering about the students who had to restart. Where they given extra time, or were they disadvantaged? Kelly Hall (Edinburgh University) 5:40 PM Thanks! Alexander [FU Berlin] 5:40 PM pardon? I see! 5:41 PM Tim Hunt 5:41 PM Just to point out that an open-source alternative to Respondus Lockdown Browser that was mentioned here is http://www.safeexambrowser.org/. Daniel Schneider is giving a talk about that a week today: http://www.transformingassessment.com/moodle/calendar/view.php?view=day&cal_d=18&cal_m=09&cal_y=2014 Mathew Hillier 5:41 PM thanks tim! yes the open source 'safeexam browser' talk is coming up in the series Alexander [FU Berlin] 5:41 PM does the LDB detect being run in a VM? Daniel R. Schneider 5:41 PM Thanks for mentioning, Tim ;-) Alexander [FU Berlin] 5:42 PM ok Rebecca Hamer 5:42 PM So they did not lose the earlier responses, but had to start where they started experiencing problems? Could they go back to responses from before the problems? 5:42 PM Mathew Hillier 5:42 PM the questions were stored in the server? and responses 5:42 PM ok 5:42 PM Alexander [FU Berlin] 5:42 PM How many tech persons do you need to perform an exam with 300 participants? Mathew Hillier 5:42 PM so the wireless becomes a big risk failure for whole groups 5:42 PM Alexander [FU Berlin] 5:42 PM hahaha ;-) Mathew Hillier 5:42 PM rather then individuals Alexander [FU Berlin] 5:43 PM ok Ghizzi 5:43 PM do require a minimum standard in the students devices? RAM, OS, graphics cards etc Tobias Halbherr 5:43 PM why do you use wireless at all? why not use ethernet cables? R.NAIDOO 5:43 PM WHAT FRAMEWORK DID YOU USE TO ANALYSE YOUR QUESTIONNAIRE Tim Hunt 5:44 PM Wireless is the concern for us too at the Open University, UK, and the problem is that it is very hard to do much about. For us, it is the part of the system we have least control over. Alexander [FU Berlin] 5:44 PM @tobias: I second that question ;-) James Denholm-Price 5:44 PM Did you use existing WiFi infrastructure or add more access points just for the exam? Trevor Meek - U of Bradford UK 5:44 PM The students seemed a bit close to each other in the photo you showed of the lecture theatre - were there problems with them seeing each others screens? Tim Hunt 5:44 PM I was already planning to watch tomorrow. Now even more so. Ghizzi 5:44 PM second that Trevor Meek - U of Bradford UK 5:45 PM Thanks... Mathew Hillier 5:45 PM randomisation would help greatly me too!!! 5:45 PM Ghizzi 5:45 PM yes that's true, Rebecca Hamer 5:45 PM Sorry have to leave. Mathew Hillier 5:45 PM randomise questions and randomised response items? ah ok. 5:45 PM James Denholm-Price 5:45 PM Did you have any issues with accessibility, e.g. for visually impaired students? Or does BYOD help mitigate that? John Dermo 5:45 PM If you use a very small font, might there be a problem with accessibility? We have to allow for larger fonts for students who need this. James Denholm-Price 5:45 PM (small fonts?) Daniel R. Schneider 5:46 PM @Tim: Unfortunately I don't have a contact to the Apple guys doing WiFi at WWDC. I spoke with them and they mentioned that they're adding many access points which they place on different heights, some under the chairs, some at 2m height, some on the ceiling. They are using a live monitoring system to analyze where there are problems to improve the coverage for the next session Tobias Halbherr 5:46 PM how do you deal with fairness issues when you randomize question order? was there no correlation between overall test scores and difficulty of the first item(s) in the test? Remi Ayoko 5:46 PM Thanks for your time today, I need to leave..thanks everyone Ghizzi 5:46 PM most students with accessibility issues here have solutions on their own devices one advantage, extra time is the only external shouldn't be different to paper Tim Hunt 5:47 PM Thanks Daniel. Ghizzi 5:47 PM I dyslexia with irlen use various software on comp to adjust Mathew Hillier 5:47 PM as discussed monday the definition of 'fair' varies greatly Daniel R. Schneider 5:48 PM Usual halls have WiFi access point mounted only at the ceiling, which covers a large room, but that's not the goal: You need many access points covering only small "cells" like that you can increase the number of clients connected 5:48 PM Tim Hunt 5:48 PM The main thing with random variants is to analyse the data afterwards. Is there a difference in Facility index (average score) between between students who got different questions allocated? If not, relax. If so, consider if you need to try to adjust John Dermo 5:49 PM How do you stop students taking the exam on their own device from outside the room? John Dermo 5:49 PM Thanks - Kelly Hall (Edinburgh University) left the Main Room. ( 5:49 PM ) - Alexander [FU Berlin] 5:49 PM Thanks! James Denholm-Price 5:49 PM Thanks :-) - Karen left the Main Room. ( 5:50 PM ) - Alexander [FU Berlin] 5:50 PM Jeffery: thanks a lot & all the best... ;-) 5:50 PM NUS 5:50 PM You take care too alexander! Hele 5:50 PM Thanks! Tobias Halbherr 5:50 PM thank you John Dermo 5:50 PM Thank you very much! Ghizzi 5:50 PM Cheers all thank you ;) Tim Hunt 5:50 PM @John. With Moodle / SEB, then Moodle verifies that the student is using SEB. As with any exams, students must show Photo ID in the exam room. Server logs show if one student was logged in from two computers. Trevor Meek - U of Bradford UK 5:50 PM Thanks. Tim Hunt 5:50 PM Thanks for a great session. Daniel R. Schneider 5:50 PM Thank you! Alexander [FU Berlin] 5:51 PM What was the exact question in the survey? Tim Hunt 5:51 PM @John, it is not bullet-proof, but not necessarily worse than paper exams. Alexander [FU Berlin] 5:51 PM (huge audio drop outs here atm) John Dermo 5:52 PM @Tim can't they install SEB and access the exam from outside the room, though, if they have internet access? Alexander [FU Berlin] 5:52 PM ok... I see... Ghizzi 5:52 PM have you used in your teaching program as quizzes or revision tests, to familiarise students with process? Tim Hunt 5:52 PM @John, if they are not in the exam room, the invigilator won't have ticked them off the list, so their answers being present will be suspicious. invigilator 5:52 PM Ghizzi 5:53 PM thank you Daniel R. Schneider 5:53 PM @John: You should also use an exam password which you communicate only inside the hall, when all students are ready to start the exam Invigilators of course have to make sure no one is using his/her phones 5:54 PM Tim Hunt 5:54 PM Also, you you put in a list of usernames who should have access to this exam. Only those people have access. Daniel R. Schneider 5:54 PM plus they should not be allowed to wear watches considering smart watches are getting smaller... 5:54 PM Tim Hunt 5:54 PM So the worst case is a student telling their friend their username/password, and the friend sitting outside the exam hall. Ghizzi 5:55 PM hehe topical Tim Hunt 5:55 PM At that point you need to rely on server logs, which is tricky. Daniel R. Schneider 5:55 PM You should use several security elements, not only the single sign on credentials Tobias Halbherr 5:56 PM ...high-tech cheating, in the long-term, is a loosing game for invigilators, imho. We need to start changing the exam culture, shifting away from assessing with closed question formats towards open ended questions and problems of high complexity. Daniel R. Schneider 5:56 PM same here, cheating means exam failed. In our case few students would take that risk Tobias Halbherr 5:57 PM ...the product of a complex open ended question does not necessarily have to be complex to mark code judges would be an example 5:57 PM Daniel R. Schneider 5:58 PM Touch screens? We plan to use tablets for some exams 5:58 PM Tim Hunt 5:58 PM There was some work on touch-screens for maths at the OU, about 5 years ago in the early days of tablets. But, not exams. Let me see if I can find a link 5:58 PM Alexander [FU Berlin] 5:58 PM gotta leave. thanks again from Berlin. See you tomorrow!!! bye... Tobias Halbherr 5:58 PM in our online-exams diagrams drawn by handed our simply handed in on paper Tobias Halbherr 5:59 PM diagrams built in software are handed in as files Tim Hunt 5:59 PM http://www.open.ac.uk/opencetl/physics-innovations-centre-excellence-teaching-and-learning/activities-projects/use-tablet-pcs-tutors-mark-physics-assignments Daniel R. Schneider 5:59 PM the next release of Safe Exam Browser will have touch support on Windows 8/8.1 computers Tim Hunt 5:59 PM That was teachers grading students work. Daniel R. Schneider 5:59 PM and there is a freehand drawing question type in Moodle (for example) Tobias Halbherr 6:00 PM ...I will log out and check out the first part of your presentation, which I missed. Thank you & good bye 6:00 PM R.NAIDOO 6:00 PM WHAT THEORY DO YOU USE TO ALLOCATE MARKS Tim Hunt 6:00 PM Thank for you for the discussion. Bye 6:00 PM Ghizzi 6:00 PM very useful cheers guys :) Sami #2 6:01 PM Thanks guys NUS 6:01 PM bye Mathew Hillier 6:01 PM bye all! Mathew Hillier 6:01 PM thanks@ ! 6:01 PM Trevor Meek - U of Bradford UK 6:01 PM Thanks. Great discussion. Bye all. Daniel R. Schneider 6:01 PM bye to all R.NAIDOO 6:01 PM Please forward email address to me naidor@unisa.ac.za