Student engagement and motivation
Between 2012 and 2015, ACARA conducted a number of research trials into student engagement with aspects of online assessments beyond tailored test design. These additional studies have investigated student engagement with technology enhanced items, with challenging (difficult) items, and with the delivery of spelling using audio files. While these trials were limited in terms of sample size, the results from them are promising and form part of an ongoing research program conducted by ACARA to support the move to NAPLAN online.
When working with the technology enhanced items, students were asked to interact with the technology in the context of NAPLAN test questions. This involved using drop down menus, moving items across the screen, highlighting text and objects or playing an audio or video file. Students were observed across all year groups engaging with the technology and appeared motivated by the experience of working online.
Students participating in the trials recounted the test as more engaging than paper-based NAPLAN tests and said they felt more confident in their performance. The audio delivery of the spelling test via headphones or speakers meant that ACARA was able to test spelling in the way that it is taught and assessed in everyday school practice. Observations of students participating in the spelling test showed that they adapted rapidly to the dictated format, and found it easy and convenient to operate.
The trials also found that no students reported feeling penalised for typing their responses. Overall, students described typing as being easier for changing and correcting answers, and the audio delivery of the spelling words was widely perceived by students to be fair and valid.
Another activity investigated the cognitive and behavioural engagement of students with NAPLAN items delivered within the new tailored (multistage) test design, with a particular focus on the testlet which contains the most challenging and difficult items. These items are outside of the current range of NAPLAN year-level tests. Students’ knowledge, thinking skills and strategy use in dealing with these challenging items were monitored. The research also investigated how the inclusion of these items related to the intended assessment outcomes.
The main goal was to investigate the extent to which these items are able to target high-level knowledge and skills that the most able students should have achieved. For numeracy, the research found that students who are mathematically capable of solving challenging numeracy items were able to access and engage with most of the questions presented in the testlets containing the most challenging set of questions.
For reading, the research found that students at each year level drew on wide-ranging reading skills relating to each of the four reading development levels: literal comprehension, interpretation, critical reading and creative reading. Overall the knowledge and strategies demonstrated by the students who participated in these studies indicated that the items in these most challenging testlets were fundamentally appropriate for students with higher levels of numeracy and reading achievement.
A study was also conducted into student motivation factors and effects. Of particular interest was the effect of the fixed and branched forms with regards to students’ performance, students’ motivation while doing the test, and students’ subjective experience of online NAPLAN numeracy tests. Findings shed further light on the positive motivational effects of adaptive online testing and extend current understanding beyond NAPLAN performance outcomes to the important area of student motivation.
See the research reports: