Methodology developed in practice

Research approaches in practice

Design & development

The key approach in my practice has been the design and development of educational software, multimedia resources, systems and ultimately courses. This design approach has been in a context where new technology offers new and unknown opportunities and despite disquiet about technology-led approaches, has inspired creativity and innovation in my practice. The key to this research approach has been a combination of developing design methodology and rigorous evaluation in real-world contexts. In this sense, I have been unconsciously engaged in a 'design science' approach as discussed in the Theoretical and Conceptual Framework section of this dissertation.

Participant action research

As my work developed, I became increasingly conscious that I was developing a participant action research approach (Denzin and Lincoln 2005, 33-34) to complement design & development. This was the result of a growing interest and opportunity to design courses, degree programmes and ultimately secondary (Notschool.net project) and higher education organisation ([P9] Ultraversity project). In each case the concept of co-research with students became ever more explicit.

Specific Methods Used

Prototyping, iterative development and field testing

In developing new interactive educational software, an early discovery was that the traditional waterfall method (Bell and Thayer 1976), of identification of user needs followed by specification, implementation and testing, would not work. Participants (including myself) in the design process were discovering new needs, had little ability to specify unknown designs offering new practices and found themselves learning through the process of development in an 'expansive' sense (Engstrom 1999). A further complication in practice was that the computers in use had a range of features and capabilities and the design team would often have diverse understandings of what could be achieved. So the method employed was of prototyping initial ideas to produce a working design, not fully debugged nor complete, to inform the next steps and inspire further invention.

Prototyping was only the beginning of course, and was followed by cycles of development and field testing, often in classrooms by the teacher participants in the design and development process, whose understanding was also growing. Alongside the successive improvement in the software itself, there was a parallel and important task to develop the teachers' and students' guidance material which underwent a similar process.

As Mor puts it:

The design element in a design study may refer to the pedagogy, the activity, or the tools used. In some cases, the researchers will focus on iterative refinement of the educational design while keeping the tools fixed, whereas in others they may highlight the tools, applying a free-flowing approach to the activities. In yet others they will aspire to achieve a coherent and comprehensive design of the activity system as a whole.

(Mor 2010, 27)

Analysis of software designs

Frequently in the design process, a failure in use would be identified in broad terms - a teacher or student would report that some aspect was unclear, difficult to use or simply baffling. At this point it was important to analyse the software design (and the practice) to discover where improvement needed to be made. At first this was done informally and with tacit knowledge of 'what works', but this task was improved to make use of insights from the worlds of visual design and from human computer interface. The input from visual design and cognition theory (Marr 1982, Scrivener 1984, Gregory 1966) offered clarity about the simplest ideas of placement and the overlapping of graphical elements on the screen, the treatment of 'white space', typography and combinations of colours. The input from human computer interface theory was primarily from Donald Norman's Four Stages of User Activity (Norman 1983b), regarding the task analysis of operating equipment. We designed our own interpretation of this model to guide colleagues in our team - An Analysis of a Single Interaction (Millwood and Riley 1988), but oriented to operating a piece of educational software.

Online and interactive questionnaire surveys

In later practice, relating to the development of courses and self-evaluation, surveys that directly questioned participants became an additional method used (Lodico et al 2010, 12). The advent of low-cost online surveys, which also gave the researcher (academic or participant) an immediate and easily repeated summary analysis, meant this became an important method. I engaged in methodological development, writing new software to take advantage of the particular strengths of interactive designs. The strengths are that instead of using coding systems, letters or numerical values to rank or select choices, this can be achieved through interactive objects and sliders, building on kinaesthetic and visual thinking of respondents to help them more directly make the judgements about an issue being surveyed.The first venture in this direction came in the design of Making Choices (Millwood 1993), a tool for modelling decisions by interactively dragging choices into rank order and the COGs passport (Millwood 2004), a tool for transition between primary and secondary schooling.  COGS helped learners evaluate their competencies by dragging elements in a geometric design. This survey method was developed most recently in the design of interactive learning needs analysis for health professionals and volunteers in the charity Macmillan Cancer Support.

Videography

In several projects, understanding the holistic context and seeing the detailed activity became important. In these cases, making video of the activity or of the discussion to evaluate it was employed, although this could prove challenging to access and analyse (Jewitt 2012). Shrum et al describe the 'fluid wall' and the 'videoactive context' to emphasize that (a) the camera is an actor, and (b) both behaviour and observation occur in both directions -- in front of and behind the camera (Shrum et al 2005). These notions have been frequently employed for projects in which I have been active to explore digital creativity with children.

In some cases, the video was transcribed and the transcription added to the video as a 'text track' which was searchable. Added value could be obtained by adding text tracks for chapters and for keyword analysis, permitting the video to be used as the vehicle for exploration and dissemination of research findings, not simply the data gathered. ( [P9] Ultraversity Project)

Although lowering of costs of equipment, media and the labour necessary to process have made this an attractive option, there are drawbacks associated with the quantity of data generated and the difficulties of processing, but these are offset in many cases by the direct and rich way in which knowledge of the research context can be communicated.

Structured Interview

In creating innovation in higher education, it became important to evaluate the experience of students and tutors in greater depth. In these cases we developed interview frameworks, conducted the interviews, recorded the audio transcribed and then employed an interpretive phenomenological analysis (Smith et al. 2009) to the data to discover in a grounded sense, the key themes of their response to our innovations (Millwood and Powell, 2009). These methods were particularly helpful in identifying conceptual knowledge in novel contexts offered by technology, although limited in reliability due to modest sample sizes and the potential for researcher bias (Brocki & Wearden 2006 101).