Perusall: University learning-teaching innovation employing social annotation and machine learning

Abstract

This paper presents the learning-teaching innovation process of a University course. The traditional elements of the teaching-learning process (lecture, study, exam) involving students in ongoing activities have changed. The paper focuses on the learning changes introduced by social annotation activities carried out through the Perusall web environment. In particular, Perusall functionalities that assess students' participation were examined. These rely on multiple indicators set by the teacher, and a Machine Learning algorithm, which assesses the quality of annotations. A study was carried out to examine the validity of this process by analysing the relationship between Perusall algorithm's scores and teacher's scores, and how students perceive the automated scoring. The relationship was investigated through the Spearman correlation coefficient and Kendall's coefficient of concordance. Thematic analysis was used to analyse the qualitative data concerning students' perceptions. The results indicate that the Perusall algorithm provided scores quite similar to those of the teacher, and that students positively perceived the automated scoring.