WEBVTT

00:00:00.000 --> 00:00:00.900
<v Ash Prosser>
Hi from Perth, Western Australia 🙂

00:00:10.000 --> 00:00:10.900
<v Eleonora Moura #2>
Hi everyone, from St Albans, UK

00:00:11.000 --> 00:00:11.900
<v andy grindrod>
Morning from Leicester UK

00:00:28.000 --> 00:00:28.900
<v John Woollard>
Hi, from University Southampton, UK

00:00:49.000 --> 00:00:49.900
<v Karin Moses>
I am receiving your voice intermittently

00:01:00.000 --> 00:01:00.900
<v Karin Moses>
All good now

00:01:05.000 --> 00:01:05.900
<v Laurine Hurley>
all good here, Karin

00:01:24.000 --> 00:01:24.900
<v James Wood>
Hello everyone 

00:01:43.000 --> 00:01:43.900
<v Fiona Harvey>
Hello 👋 

00:01:49.000 --> 00:01:49.900
<v Rachel Forsyth (Lund, Sweden)>
Hello @James Wood ! 

00:02:08.000 --> 00:02:08.900
<v Emma D-W (Dundee)>
Hello @Fiona Harvey 

00:02:12.000 --> 00:02:12.900
<v Cynthia Ogilvy>
Kia ora (Greetings) from New Zealand.

00:03:16.000 --> 00:03:16.900
<v James Wood>
Morning @Rachel Forsyth (Lund, Sweden) !

00:04:50.000 --> 00:04:50.900
<v Mojy>
Hello @Ash Prosser fellow Perth academic 🙂  

00:06:55.000 --> 00:06:55.900
<v Emma D-W (Dundee)>
[For those outside England/UK - Level 4 = 1st year UG & Level 5 = 2nd year]

00:10:18.000 --> 00:10:18.900
<v Rachel Forsyth (Lund, Sweden)>
invigilation outside UK may be supervised/proctored

00:11:57.000 --> 00:11:57.900
<v Mike>
why use multiple choice (1 correct answer) rather than multiple response ?

00:13:55.000 --> 00:13:55.900
<v Fabio Aricò>
I guess all depends on what/how you want to assess, and also the degree of certainty/precision, which is discipline specific.

00:17:32.000 --> 00:17:32.900
<v Mike>
Is this image accessible?

00:17:37.000 --> 00:17:37.900
<v Alison Willows>
Is there not an accessibility issue with using image only in questions? (without alt text)

00:19:00.000 --> 00:19:00.900
<v Emma D-W (Dundee)>
It's a shame that so little has been done on MCQ for Higher Order, one of the first presentations I did was at the Computer Aided Assessment Conference in Loughborough in 2001 - though we did go a little beyond MCQ, but it was all objective marking (QuestionMark at the time) [Ref of paper https://repository.lboro.ac.uk/articles/conference_contribution/Using_computer-aided_assessment_to_test_higher_level_learning_outcomes/9490043/1
The assessment we were looking at the time was in a computer lab (not all students 22 years ago had their own laptops // internet at home 🙂 ]

00:19:46.000 --> 00:19:46.900
<v Lai>
the link you gave is not working @Emma D-W (Dundee) 

00:20:11.000 --> 00:20:11.900
<v Lindesay>
😩 

00:20:15.000 --> 00:20:15.900
<v Annette>
remove the square bracket and it workds

00:20:43.000 --> 00:20:43.900
<v Mike>
the latest chatgpt can analyse images

00:20:44.000 --> 00:20:44.900
<v Emma D-W (Dundee)>
Thanks, Annette - and here's it again to save the editing. https://repository.lboro.ac.uk/articles/conference_contribution/Using_computer-aided_assessment_to_test_higher_level_learning_outcomes/9490043/1

00:20:45.000 --> 00:20:45.900
<v Fabio Aricò>
Indeed. MCQ testing can be high order if engineered correctly. It can also be combined with very effective active learning techniques in the classroom.

00:22:45.000 --> 00:22:45.900
<v Michael Sankey #2>
is anybody else's sound dropping in and out?

00:22:58.000 --> 00:22:58.900
<v James Wood>
Mine is perfect. 

00:23:00.000 --> 00:23:00.900
<v Fabio Aricò>
Working well here Michael

00:23:00.000 --> 00:23:00.900
<v Cesar Ortega (Curtin)>
Sound is flawless

00:23:10.000 --> 00:23:10.900
<v Michael Sankey #2>
OK thanks

00:23:52.000 --> 00:23:52.900
<v Mike>
how many students used chatgpt?

00:25:03.000 --> 00:25:03.900
<v Emma D-W (Dundee)>
@Fabio Aricò  - I agree, it took quite some time to design the set of questions we'd used that time. [And I was trying to get QMP to do things it wasn't necessarily designed to do ... it did take some time to find work rounds e.g. for multiple response & score 1/right answer, but not allow them to select all answers ... ]

00:25:26.000 --> 00:25:26.900
<v Alice Shepherd>
MCQs don't all have to have same weighting

00:25:28.000 --> 00:25:28.900
<v Rachel Forsyth (Lund, Sweden)>
Sorry, I have to go, but will catch up with the recording. 

00:26:11.000 --> 00:26:11.900
<v Emma D-W (Dundee)>
Agree re. the 1 min/ question, that having longer to answer some questions is useful (and weightings of different questions]

00:26:56.000 --> 00:26:56.900
<v Laurine Hurley>
"some knowledge is more equal than other knowledge"!

00:27:02.000 --> 00:27:02.900
<v Lai>
I probably did not fully get it right: does the speaker used chatgpt to test the MCQ she designed and use the output of chatgpt to adapt the question even better? I am using free version of chatgpt, but I am not able to input image. is it correct that with the paid version (20dollar per month) user can use multimedia input with the chatgpt?

00:27:05.000 --> 00:27:05.900
<v Susanne Voelkel>
We tend to have a mixture of MCQ, multiple answer, fill the gaps, ordering etc. types of questions, with each question having a different anticipated length

00:27:12.000 --> 00:27:12.900
<v Laurine Hurley>
ie, time per question

00:27:27.000 --> 00:27:27.900
<v andy grindrod>
Thanks Chahna

00:27:37.000 --> 00:27:37.900
<v Emma D-W (Dundee)>
Thanks, Chahna! 

00:27:52.000 --> 00:27:52.900
<v Alice Shepherd>
instructions and test organisation has to be clear if you're going to use different weightings, set against randomisation of questions from a bank.  I think the approach of practice tests to set expectations is a good one.  Can also prompt a useful conversation about test management eg managing time, skipping Qs etc.  Analogous to 'game management' that we hear so much of in football commentary?!?! 

00:30:10.000 --> 00:30:10.900
<v Emma D-W (Dundee)>
Agree, @Alice Shepherd  - admittedly when we were  looking at extending objective testing, it was very much for invigilated settings, rather than in their own spaces, so question banks etc., were less of an issue. 

00:31:22.000 --> 00:31:22.900
<v Alice Shepherd>
Yes @Emma D-W (Dundee) .  I appreciate the nuances here, having been involved with professional assessment which is all proctored.  

00:31:35.000 --> 00:31:35.900
<v Laurine Hurley>
the term "Augmenting Intelligence" may be more accurate than Artificial?

00:32:21.000 --> 00:32:21.900
<v Emma D-W (Dundee)>
Are yours human or computer proctored, @Alice Shepherd 

00:32:55.000 --> 00:32:55.900
<v Alice Shepherd>
mixture.  students can choose to attend an exam centre with a human invigilator, or sit remotely with AI and human proctoring 

00:33:19.000 --> 00:33:19.900
<v Emma D-W (Dundee)>
👍 

00:33:36.000 --> 00:33:36.900
<v Mike>
we cannot accurately predict what AI technologies our students future professional world will use

00:34:38.000 --> 00:34:38.900
<v Emma D-W (Dundee)>
Tools, I agree, but text / images / coding etc., they'll have access to; so I'd see it more along the lines of working with AI, rather than "How to use tool x" (much as we do with many tools that we use with students) 

00:35:52.000 --> 00:35:52.900
<v Mike>
We don't know what AI will be capable of 

00:36:04.000 --> 00:36:04.900
<v Fabio Aricò>
No news under the sun Mike, we are always behind...teaching for jobs that haven't been created yet. But we can just do our best and at least attempt to visualise the world to come. Critical ability is something that transcends time and very much needed.

00:36:28.000 --> 00:36:28.900
<v Emma D-W (Dundee)>
Fabio said much more clearly what I'd tried to say 🙂 

00:36:39.000 --> 00:36:39.900
<v Mark Bassett>
Thanks Fabio!

00:38:18.000 --> 00:38:18.900
<v Mike>
Who visualised chatgpt capability before it became available?

00:38:56.000 --> 00:38:56.900
<v Mark Bassett>
Ah - osmosis - what a great false friend!

00:40:10.000 --> 00:40:10.900
<v Fabio Aricò>
Same with Google and all the lot... AI is not AI like in SciFi movies, it's a chatbot. About time we demystify it.

00:41:23.000 --> 00:41:23.900
<v Emma D-W (Dundee)>
And it many ways it's the public facing view of things like the Turing Test or Deep Blue Chess machine - now much more easily available to all rather than restricted to AI / Computer Science research groups etc. 

00:41:25.000 --> 00:41:25.900
<v Mike>
I don't think you understand how the new AI works Fabio

00:41:36.000 --> 00:41:36.900
<v Mike>
it is not a 

00:41:53.000 --> 00:41:53.900
<v Karin Moses #2>
 Generative AI is changing week by week. It can now generate a reflective response, a critically evaluative one and  has a capacity to use images in its generative production and response to prompts.

00:42:21.000 --> 00:42:21.900
<v Mike>
Generative AI can create speech, music, video etc.

00:42:58.000 --> 00:42:58.900
<v Mathew Hillier>
some good examples of what gen AI can do here https://twitter.com/emollick

00:43:10.000 --> 00:43:10.900
<v Mark Bassett>
Right - we have to teach/scaffold learners with literacies that our tasks require/demand - academic, information, and digital 🙂

00:43:13.000 --> 00:43:13.900
<v Mike>
Companies can add their own data

00:45:17.000 --> 00:45:17.900
<v Mike>
OpenAI developers did not predict what their generative AI could do ... it surprises them

00:45:26.000 --> 00:45:26.900
<v Alice Shepherd>
they can but a lot of companies will want to protect their proprietary data...

00:45:43.000 --> 00:45:43.900
<v Cesar Ortega (Curtin)>
But data is already in the warehouses of Google, Facebook and the likes.

00:46:07.000 --> 00:46:07.900
<v Fabio Aricò>
No news under the sun again: Google is also there for business and we need to educate our students to that and to everything on the internet

00:46:07.000 --> 00:46:07.900
<v Mark Bassett>
Rant away!

00:46:27.000 --> 00:46:27.900
<v Alice Shepherd>
agree, but it may not be the proprietary data.  My discipline is accounting.  Externally reported data is only partial.  

00:46:31.000 --> 00:46:31.900
<v Brett Fyfield>
Great presentation Margaret!

00:48:04.000 --> 00:48:04.900
<v PT>
Agree with Brett. A great presentation Margaret!

00:48:21.000 --> 00:48:21.900
<v Margaret Bearman>
why thank you - and again HT to Juuso and rola

00:48:22.000 --> 00:48:22.900
<v Alice Shepherd>
both really interesting presentations and thought-provoking

00:48:34.000 --> 00:48:34.900
<v Fiona Harvey #3>
That was brilliant thank you! I totally agree re the critical questions. Neil Postman said in 1992(!) “what is the problem that this technology is solving?” And that’s my go to now with AI. I love it and worry about it at the same time 

00:48:54.000 --> 00:48:54.900
<v Mike>
Do all assessments need to be checked if they can be passed using chatgpt etc.?

00:49:14.000 --> 00:49:14.900
<v Emma D-W (Dundee)>
Thanks for both presentations -really interesting. 

00:49:27.000 --> 00:49:27.900
<v Alice Shepherd>
thanks @Chahna Gonsalves (KCL) 

00:49:31.000 --> 00:49:31.900
<v James Wood>
 Yes, thank you to both presenters, lots of food for thought as we adapt our assessment policies and guidance for September! 

00:50:51.000 --> 00:50:51.900
<v Marion Palmer>
Thank you Chahna and Margaret for the presentations. Margaret, I like the update of the digital assessment design model in terms of developing community through assessment tasks but it's surprising how few examples there are in  the literature. 

00:50:59.000 --> 00:50:59.900
<v Margaret Bearman>
sorry for the shameless plug - but please see https://doi.org/10.1111/bjet.13337 and https://link.springer.com/chapter/10.1007/978-3-030-41956-1_5 for some of my thoughts (with Rola and Rose Luckin) on the boundaries around AI

00:51:09.000 --> 00:51:09.900
<v Kerrie King>
I’m really concerned about the amount of time we find ourselves putting into creating assessments now.  

00:51:10.000 --> 00:51:10.900
<v Margaret Bearman>
Agree Marion

00:51:32.000 --> 00:51:32.900
<v Cesar Ortega (Curtin)>
The conversation should not be about whether ChatGPT is really intelligent or not. We should discuss how we and our students are going to use this new tool. There is great potential, but also great risks. Like any tool, harm or benefit come from the one using it.

00:51:37.000 --> 00:51:37.900
<v Margaret Bearman>
I think this comes to Phill Dawson's point - we should assure less learning because of the expense but 

00:51:45.000 --> 00:51:45.900
<v Margaret Bearman>
spend a lot of time on formative work

00:52:15.000 --> 00:52:15.900
<v Emma D-W (Dundee)>
And make sure we think about Generative AI generally, not ChatGPT specifically. 

00:52:19.000 --> 00:52:19.900
<v Margaret Bearman>
Learning to work with the black box: Pedagogy for a world with artificial intelligence
https://bera-journals.onlinelibrary.wiley.com/authored-by/Bearman/Margaret and 
https://bera-journals.onlinelibrary.wiley.com/authored-by/Ajjawi/Rola
First published: 22 May 2023 https://doi.org/10.1111/bjet.13337

00:53:00.000 --> 00:53:00.900
<v Mike>
since chatgpt gives a different response every time it is difficult to check whether it passes tests always

00:53:32.000 --> 00:53:32.900
<v Mathew Hillier>
re AI literacy framework - I also dabbled in that too. <https://teche.mq.edu.au/2023/03/a-proposed-ai-literacy-framework/

00:54:20.000 --> 00:54:20.900
<v James Wood>
A question for Margaret: How should we best ‘tool up’ our teachers to prepare for authentic assessment in a digital world? Many of our teachers are so busy working on research in their own areas. How can we best induct them to prepare authentic digital assessments? Our students need critical digital literacies, but probably most teachers don’t have it, and many of our students may be more advanced than some of our teachers. 

00:54:45.000 --> 00:54:45.900
<v Chahna Gonsalves (KCL)>
@Kerrie King , I agree that we need to be mindful of the time. In my case, all the time put into preparation of the questions was balanced by automated marking and short follow up conversations with a few (3) students who wanted it. Teaching in a team of 1 or two made this worth it but but i think we have to be reasonable about the approached we take in terms of workload

00:56:13.000 --> 00:56:13.900
<v Mark Bassett>
programmatic assessment 🙂

00:56:28.000 --> 00:56:28.900
<v Mathew Hillier>
agree there is indeed tensions in the design space for assessment. 'authentic v scale v integrity'.

00:56:32.000 --> 00:56:32.900
<v Chahna Gonsalves (KCL)>
@Mike the responses for me stayed quite similar so I felt quite sure that my decision of whether it could answer a question correctly meant the question needed to change

00:56:58.000 --> 00:56:58.900
<v Chahna Gonsalves (KCL)>
We need a variety of assessment types

00:56:59.000 --> 00:56:59.900
<v Joanne Bradbury>
Thank you to both presenters, very interesting presentations

00:57:02.000 --> 00:57:02.900
<v Mark Bassett>
Thank you Chahna and Margaret

00:57:09.000 --> 00:57:09.900
<v Eric Holm>
Thank you 

00:57:14.000 --> 00:57:14.900
<v PT>
Thank you both.

00:57:17.000 --> 00:57:17.900
<v Alice Shepherd>
👏 

00:57:18.000 --> 00:57:18.900
<v Penny Wheeler>
👏 👏 👏 

00:57:18.000 --> 00:57:18.900
<v Matt Townsend - Cardiff University>
Many thanks

00:57:19.000 --> 00:57:19.900
<v andy grindrod>
Thanks all - really thought provoking

00:57:21.000 --> 00:57:21.900
<v Emma D-W (Dundee)>
From the tests I've done, the similarity of the answers from ChatGPT varies depending on the openness of the question you've asked it. 

00:57:22.000 --> 00:57:22.900
<v Nuria>
Thank you to both presenters!

00:57:28.000 --> 00:57:28.900
<v Jillian Hendry #3>
Fantastic session, thank you to all 🙂

00:57:28.000 --> 00:57:28.900
<v Alison Willows>
thank you both, very interesting and lots to think on

00:57:28.000 --> 00:57:28.900
<v Johanne>
Thanks - excellent presentations

00:57:30.000 --> 00:57:30.900
<v Chahna Gonsalves (KCL)>
but automated marking (***with feedback) when it's valuable 🙂

00:57:30.000 --> 00:57:30.900
<v PT>
Thank you Michael and the organising team

00:57:31.000 --> 00:57:31.900
<v Maxine Mitchell - Murdoch Uni #4>
thank you - very thought provoking!

00:57:33.000 --> 00:57:33.900
<v Alpana A>
Thank you 

00:57:34.000 --> 00:57:34.900
<v Liz Hardie>
Thank you both, really interesting

00:57:34.000 --> 00:57:34.900
<v andy grindrod>
👏 

00:57:35.000 --> 00:57:35.900
<v Denise>
👏 

00:57:36.000 --> 00:57:36.900
<v Lyn Marks>
thank you both

00:57:39.000 --> 00:57:39.900
<v Brett Fyfield>
Great to have your perspectives!

00:57:45.000 --> 00:57:45.900
<v Emma D-W (Dundee)>
[e.g. it rarely disagrees with itself on the number of wives that Henry 8th had ...]

00:57:45.000 --> 00:57:45.900
<v Chahna Gonsalves (KCL)>
Thank you all. Please keep in touch

00:57:45.000 --> 00:57:45.900
<v Matthew>
Cheers Team!

00:57:47.000 --> 00:57:47.900
<v Mojy>
Thank you both

00:57:48.000 --> 00:57:48.900
<v Tim Linsey>
Thank you, great!

00:57:49.000 --> 00:57:49.900
<v James Wood>
👍 Excellent session. Thank you very much, and thank you for all the questions and perspectives in the chat. Hope to see you at AHE. 

00:57:50.000 --> 00:57:50.900
<v melanie>
many thanks

00:57:57.000 --> 00:57:57.900
<v Emma D-W (Dundee)>
Really super, thanks! 

00:58:01.000 --> 00:58:01.900
<v Kerrie King>
Thanks everyone 

00:58:03.000 --> 00:58:03.900
<v Eric Holm>
👍 

00:58:03.000 --> 00:58:03.900
<v Sue Hackett>
Thanks a million

00:58:09.000 --> 00:58:09.900
<v Cynthia Ogilvy>
Thanks

00:58:12.000 --> 00:58:12.900
<v Maryam Rajid #2>
Thanks

00:58:18.000 --> 00:58:18.900
<v Marion Palmer>
thank you

00:58:18.000 --> 00:58:18.900
<v Joanne Bradbury>
thank you, bye

00:58:18.000 --> 00:58:18.900
<v Sabine>
thank you

00:58:19.000 --> 00:58:19.900
<v Mark Bassett>
Night all!

00:58:20.000 --> 00:58:20.900
<v Fiona Harvey #3>
Thank you that was great, both presentations were great

00:58:20.000 --> 00:58:20.900
<v Michael Sankey #2>
cheers