1 June 2016: Adaptive Comparative Judgement for online grading of project-based assessment

Presenter: Professor Richard Kimbell (Goldsmiths College, University of London).

Hosted by Professor Geoffrey Crisp, PVC Education, University of New South Wales and Dr Mathew Hillier, Monash University, Australia. Starting 07:00AM UTC/GMT. Duration 60 Minutes.

Adaptive Comparative Judgement (ACJ) offers an alternative to marking, especially for performance assessments for which achievement can be difficult to describe in mark schemes. The ACJ system uses a web-browser to deliver pairs of candidates’ work to judges. The judges apply their subject expertise to select which of the two sets of work is better in terms of meeting appropriate educational objectives. A quality parameter is estimated for each piece of candidates’ work using the Rasch logistic model to analyse the judges’ decisions. Placing the pieces of work on a measurement scale gives them a rank order.

This webinar looked at how ACJ delivered through online technologies can be a valid and highly reliable alternative to traditional analytical marking. This alternative has been underused in educational assessment and measurement in the past, mainly due to the lack of supporting technologies to facilitate large-scale testing for large number of judgements and judges.  

In this webinar Richard presented and engaged with ACJ, explored the problems it solves for both summative and formative assessment and highlighted the benefits it offers for peer to peer review with a focus on raising awareness and transforming assessment.

Further information

• Slide Set: TA_webinar_1_june_2016_slides.pdf [4.9MB] [PPTx 46.8MB]
• Chat Log (edited for clarity): TA_webinar_1june2016_chat_log.txt
• Richard's staff page.
• The adaptive comparative judgement project page at DigitalAssess (the provider of the software that drives ACJ).
• Student comment video (shown towards the end of the session)
• Additional resources mentioned in chat
  • Digital Platform for the Assessment of Competences (D-PAC) project
  • Wikipedia article: Adaptive comparative judgement
  • Whitepaper: Whitehouse C. and Pollitt A. (2012) Using Adaptive Comparative Judgement to Obtain a Highly Reliable Rank Order in Summative Assessment [PDF]
  • Journal Article: Kimbell, R (2012) The origins and underpinning principles of e-scape [PDF], International Journal of Technology and Design Education, 22(2), pp 123-134.
  • Report: Bramley, T (2015) Investigating the reliability of Adaptive Comparative Judgment [PDF], Cambridge Assessment Research Report, 23 March [PDF]
  • NoMoreMarking (open source project).

Session Recording

Multiple formats are available.

You Tube version (Flash video) - recommended version.

A screen cast of the session.

Start playing via You Tube (note: this is a single file): https://youtu.be/ftsDbDPqb2w (You can choose to view in standard definition or high definition and in full screen mode).

Blackboard Collaborate archive version

This uses the classroom system itself to re-play a recording of the event. You are also able to jump forward, back and pause the recording.

Note: links shown inside this recording are only 'clickable' when displayed in the lower left 'text chat' area.

• Click: https://au-sas.bbcollab.com/site/external/jwsdetect/playback.jnlp?psid=2016-05-31.2354.M.603B1516A8EFB6764BC75F99AC55DB.vcr&sid=2010034
• Next:
  • in Firefox/Internet explorer you may be promoted to open with/run or save the file - choose open with/run.
  • in Chrome a prompt will appear in the status bar at the bottom of the window - choose 'keep', then click the file name.
• The Blackboard collaborate player should begin launching via JAVA.

If you are having trouble accessing the classroom system please read the help page.