1 June 2016: Adaptive Comparative Judgement for online grading of project-based assessment

Presenter: Professor Richard Kimbell (Goldsmiths College, University of London).

Hosted by Professor Geoffrey Crisp, PVC Education, University of New South Wales and Dr Mathew Hillier, Monash University, Australia. Starting 07:00AM UTC/GMT. Duration 60 Minutes.

Adaptive Comparative Judgement (ACJ) offers an alternative to marking, especially for performance assessments for which achievement can be difficult to describe in mark schemes. The ACJ system uses a web-browser to deliver pairs of candidates’ work to judges. The judges apply their subject expertise to select which of the two sets of work is better in terms of meeting appropriate educational objectives. A quality parameter is estimated for each piece of candidates’ work using the Rasch logistic model to analyse the judges’ decisions. Placing the pieces of work on a measurement scale gives them a rank order.

This webinar looked at how ACJ delivered through online technologies can be a valid and highly reliable alternative to traditional analytical marking. This alternative has been underused in educational assessment and measurement in the past, mainly due to the lack of supporting technologies to facilitate large-scale testing for large number of judgements and judges.  

In this webinar Richard presented and engaged with ACJ, explored the problems it solves for both summative and formative assessment and highlighted the benefits it offers for peer to peer review with a focus on raising awareness and transforming assessment.

Further information

Session Recording

Multiple formats are available.

play via you tube

You Tube version (Flash video) - recommended version.

A screen cast of the session.

Start playing via You Tube (note: this is a single file): https://youtu.be/ftsDbDPqb2w (You can choose to view in standard definition or high definition and in full screen mode).

Blackboard Collaborate archive version

This uses the classroom system itself to re-play a recording of the event. You are also able to jump forward, back and pause the recording.

Note: links shown inside this recording are only 'clickable' when displayed in the lower left 'text chat' area.

If you are having trouble accessing the classroom system please read the help page.

SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 "
SMTP -> get_lines(): $data is "220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 "
SMTP -> get_lines(): $data was "220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 "
SMTP -> get_lines(): $str is "220-We do not authorize the use of this system to transport unsolicited, "
SMTP -> get_lines(): $data is "220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 220-We do not authorize the use of this system to transport unsolicited, "
SMTP -> get_lines(): $data was "220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 220-We do not authorize the use of this system to transport unsolicited, "
SMTP -> get_lines(): $str is "220 and/or bulk e-mail. "
SMTP -> get_lines(): $data is "220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 220-We do not authorize the use of this system to transport unsolicited, 220 and/or bulk e-mail. "
SMTP -> FROM SERVER:220-quagmire.securedserverspace.com ESMTP Exim 4.91 #1 Mon, 25 Mar 2019 18:50:47 -0500 220-We do not authorize the use of this system to transport unsolicited, 220 and/or bulk e-mail.
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] "
SMTP -> get_lines(): $data is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] "
SMTP -> get_lines(): $data was "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] "
SMTP -> get_lines(): $str is "250-SIZE 52428800 "
SMTP -> get_lines(): $data is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 "
SMTP -> get_lines(): $data was "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 "
SMTP -> get_lines(): $str is "250-8BITMIME "
SMTP -> get_lines(): $data is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME "
SMTP -> get_lines(): $data was "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME "
SMTP -> get_lines(): $str is "250-PIPELINING "
SMTP -> get_lines(): $data is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME 250-PIPELINING "
SMTP -> get_lines(): $data was "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME 250-PIPELINING "
SMTP -> get_lines(): $str is "250-AUTH PLAIN LOGIN "
SMTP -> get_lines(): $data is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME 250-PIPELINING 250-AUTH PLAIN LOGIN "
SMTP -> get_lines(): $data was "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME 250-PIPELINING 250-AUTH PLAIN LOGIN "
SMTP -> get_lines(): $str is "250 HELP "
SMTP -> get_lines(): $data is "250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME 250-PIPELINING 250-AUTH PLAIN LOGIN 250 HELP "
SMTP -> FROM SERVER: 250-quagmire.securedserverspace.com Hello transformingassessment.com [208.117.38.64] 250-SIZE 52428800 250-8BITMIME 250-PIPELINING 250-AUTH PLAIN LOGIN 250 HELP
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "334 VXNlcm5hbWU6 "
SMTP -> get_lines(): $data is "334 VXNlcm5hbWU6 "
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "334 UGFzc3dvcmQ6 "
SMTP -> get_lines(): $data is "334 UGFzc3dvcmQ6 "
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "235 Authentication succeeded "
SMTP -> get_lines(): $data is "235 Authentication succeeded "
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "250 OK "
SMTP -> get_lines(): $data is "250 OK "
SMTP -> FROM SERVER:250 OK
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "250 Accepted "
SMTP -> get_lines(): $data is "250 Accepted "
SMTP -> FROM SERVER:250 Accepted
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "354 Enter message, ending with "." on a line by itself "
SMTP -> get_lines(): $data is "354 Enter message, ending with "." on a line by itself "
SMTP -> FROM SERVER:354 Enter message, ending with "." on a line by itself
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "250 OK id=1h8ZMd-00CCMx-HN "
SMTP -> get_lines(): $data is "250 OK id=1h8ZMd-00CCMx-HN "
SMTP -> FROM SERVER:250 OK id=1h8ZMd-00CCMx-HN
SMTP -> get_lines(): $data was ""
SMTP -> get_lines(): $str is "221 quagmire.securedserverspace.com closing connection "
SMTP -> get_lines(): $data is "221 quagmire.securedserverspace.com closing connection "
SMTP -> FROM SERVER:221 quagmire.securedserverspace.com closing connection