Grading without TAs: The Eliph system for peer evaluation

This article presents the Eliph system for improving peer evaluation of computer programs. Eliph is part of a larger platform for programming education and uses fine-grained data of code revision history collected by the platform. Eliph was evaluated in a real classroom, and the user study verified that using the system resulted in higher-quality peer evaluation.

Article  |  Fall 2017

A major challenge in online education, where there may be thousands of students in a class, is evaluating and giving feedback on students’ work. For some of those classes, automatic evaluation is an option, but in most classes where the assignments are open-ended, automatic evaluation is not effective or even possible. One practical solution for these classes is peer evaluation, often used in essay- or project-based classes. While previous studies have looked at the benefits, challenges, and systems for peer evaluation of various classes, the challenges and needs of peer evaluation for computer programming classes are unique and have been overlooked.

Jungkook Park, a graduate student in Prof. Alice Oh’s research group, developed Eliph, a more effective peer evaluation system for programming based on fine-grained revision history of students’ solutions. With Eliph, students evaluate other students’ work with access to much more information than just the final submission. They can play back, keystroke by keystroke, the entire history of the submission from the skeleton code provided by the TAs, to all the intermediate outputs, to the final submission.

Prof. Oh and her research team tested Eliph by deploying the system in a computer science course and running a user study with 60 students. The results showed that the comments from the peer evaluation with Eliph clearly exhibited better understanding of the original code authors’ intentions and provided more helpful comments for the authors.

Prof. Oh and Jungkook Park’s research made significant contributions in three ways. First, it presented the positive results of peer evaluation in the domain of programming education, where large-scale classes are quite common. Second, it highlighted the importance of fine-grained data collection and visualization using a Web-based education platform. Third and foremost, it demonstrated a new system for improved peer evaluation with the playback feature of fine-grained revision history.

This research was presented at ACM Computer Supported Cooperative Work (CSCW) 2017 and won the Honorable Mention award.

 For a podcast interview with Prof. Alice Oh, click on the following link : http://www.kaist.ac.kr/html/en/news/podcast.html