Development and Evaluation of Search Tasks, LEAP Group, Oxford University
Description
We worked with the Local Electron Atom Probe (LEAP) group at the Materials Science Department of Oxford university to both devise and carry out search taks for Information Retreival studies for the RDM Research Data Search product development. Our start point for doing this was a framework of learning objectives and cognitive complexity, outlined and discussed in a paper by Diane Kelly. This framework was then used to try and define objective task complexity ratings, by using the learning objectives, required outcomes and mental activities involved in completing the tasks. This was done by questioning of the participants. In effect, the participants helped us define the tasks, shortly before carrying them out. As the cognitive processes required are cumulative, more processes equals greater complexity. We ended up with five levels of cognitive complexity. For search task evaluation and study, participants were first asked to complete a pre-task questionnaire. This contained questions about participants' interest in and knowledge of the task, and subjective questions about perceived task complexity. There were also subjective questions about expected task difficulty, in relation to specific expectations of presumed challenges associated with completing the task, e.g. evaluating the results and determining when they had enough information to stop. The post-task questionnaire contained similar items to the pre-task questionnaire, with the aim of comparing expectations with experience. Additionally there were questions about enjoyment and engagement, and some overall judgements around difficulty and satisfaction. Participants' search behaviours were logged manually with real time observation and repeated analysis of recordings. The values were averaged and reported by cognitive complexity level. The participants were 12 members of the LEAP group based in Oxford. We will revisit to test the same tasks on DataSearch when we have suitable and enriched matching data.