Methodology

Approach to Practice

The design of software, resources and systems I have developed has been undertaken in the context of a series of research projects. The inquiry paradigm of these projects started with creative curriculum development in the early part of my career and later became overlaid with a more explicitly collaborative action research approach (Lewin 1973, 205-6; Argyris and Schön 1978), within and on the education system itself. My mature visualisation of this approach is shown in the following figure, developed as a wall poster:

Figure 2: A model of action research

This paradigm is not without its problems, as Somekh and Zeichner put it:

This action research approach and all these methods have been employed in my practice as part of the many research and development projects undertaken, and they are described and exemplified in the section Methodologies in Practice.

Approach to Thesis

When setting out to develop this thesis, I decided on a reflective methodological approach and related methods to construct it, but, keeping true to my practice of design and what drives me, the design and development of a resource in the shape of a web site was a vital element. The steps taken to arrive at the substance and presentation of my thesis are discussed in the section Methodology for this thesis.

This web-site can be found at http://phd.richardmillwood.net

Values

The research and development I have been engaged in has always been values-driven, in a nutshell 'to change the world for the better'. Over time these values have developed, but they clearly indicate a subjective viewpoint, and thus run the risk of introducing bias and overlooking issues. The approach has been to recognise these values, identify the risks and maintain the moral perspective.

The most important of these values are discussed below.

Collaboration

All the research and development I have engaged in recognises that collaboration is essential. The positive features for me are:

• a continuing dialogue of critical friendship;
• the benefit of diverse strengths and perspectives;
• where possible, the democratic involvement of beneficiaries and stakeholders to improve ideas, evaluation and uptake.

The price on occasion has been an 'averaging' of creative ideas to achieve consensus, but on reflecting back over this work, this has rarely prevented innovation.

Social justice

The idea that research and development might address inequalities of opportunity in society through education has been central. This means that methodologies such as experimental design, which favour one group over another through a treatment group and a control group, have been avoided in the work I have undertaken, and more naturalistic and qualitative methods used to evaluate outcomes. In the design process it has meant paying attention to diversity, culture and gender issues and making positive efforts in the design of innovations to address these. This has occasionally needed to counter technology-led innovation, which so often simply addresses 'normal' users. In some cases, a compromise has been needed between exploring new designs and addressing accessibility, whilst maintaining a critical view of the issue.

Transparency and participation

My first software development work as a full-time researcher at Computers in the Curriculum in 1980 obliged me to work closely and to be led in pedagogical issues by teacher-groups and individual teachers leading on items of software. This demanded a transparency in planning and participation in design which I came to value. The returns were a growing awareness of practising teachers' knowledge and concerns combined with a practical means to deliver on the values of social justice. This in turn led to design and development criteria informed by my practice working with the teachers. In later projects, especially at Ultralab in the 90s this approach was extended to include school students as 'co-researchers', inviting them to understand the goals of our developments and to contribute at every level.

Delight

The idea that delight was essential to successful learning was a central tenet in Ultralab in the 90s, permitting us to develop software for learning with features intended to inspire delight in learners. But it wasn't until 2006 that I realised how much tacit knowledge had been developed. The opportunity to articulate this fully for Teachers' TV's School Matters series, Happiest Days? (Millwood 2007) and for a poster An Analysis of Delight (Millwood 2010) helped me to recognise the nuanced detail of this concept and how deeply it had become key to my design and development. The influence of David Hargreaves (1975) in his book Interpersonal Relations and Education and John Heron in his Feeling and Personhood: Psychology in Another Key (1992) has been central to developing this concept of delight.

Approaches

Design & development

The key approach in my practice has been the design and development of educational software, multimedia resources, systems and ultimately courses. This design approach has been in a context where new technology offers new and unknown opportunities and despite disquiet about technology-led approaches, has inspired creativity and innovation in my practice. The key to this research approach has been a combination of developing design methodology and rigorous evaluation in real-world contexts. In this sense, I have been unconsciously engaged in a 'design science' approach:

A design science of education should be based on a linguistic framework which offers an intermediate level of systematisation, rising above anecdotes but remaining grounded in reality. Such a framework would allow us to capture the structure of educational situations, the challenges they engender, as well as the means of addressing them, in forms which should empower learners and teachers to control their practice as much as it allows researchers to inspect it scientifically.

(Mor 2010, p14)

I would argue that the analytical perspectives I present in the Theoretical and Conceptual Framework are my version of Mor's 'linguistic framework'.

Participant action research

As my work developed, I became increasingly conscious that I was developing a participant action research approach (Denzin and Lincoln, pp33-34) to complement design & development. This was the result of a growing interest and opportunity to design courses, degree programmes and ultimately secondary (Notschool.net project) and higher education organisation ([C18] Ultraversity Project). In each case the concept of co-research with students became ever more explicit.

Specific Methods Used

Prototyping, iterative development and field testing

In developing new interactive educational software, an early discovery was that the traditional waterfall method (Bell and Thayer 1976), of identification of user needs followed by specification, implementation and testing, would not work. Participants (including myself) in the design process were discovering new needs, had little ability to specify unknown designs offering new practices and found themselves learning through the process of development in an 'expansive' sense (Engstrom 1999). A further complication in practice was that the computers in use had a range of features and capabilities and the design team would often have diverse understandings of what could be achieved. So the method employed was of prototyping initial ideas to produce a working design, not fully debugged nor complete, to inform the next steps and inspire further invention.

Prototyping was only the beginning of course, and was followed by cycles of development and field testing, often in classrooms by the teacher participants in the design and development process, whose understanding was also growing. Alongside the successive improvement in the software itself, there was a parallel and important task to develop the teachers' and students' guidance material which underwent a similar process.

As Mor puts it:

The design element in a design study may refer to the pedagogy, the activity, or the tools used. In some cases, the researchers will focus on iterative refinement of the educational design while keeping the tools fixed, whereas in others they may highlight the tools, applying a free-flowing approach to the activities. In yet others they will aspire to achieve a coherent and comprehensive design of the activity system as a whole.

(Mor 2010, p27)

Analysis

Frequently in the process, a failure in use would be identified in broad terms - a teacher or student would report that some aspect was unclear, difficult to use or simply baffling. At this point it was important to analyse the software, and the practice, to discover where improvement needed to be made. At first this was done informally and with tacit knowledge of 'what works', but later after making sense of knowledge from experts, this task was improved to make use of insights from the worlds of visual design and from human computer interface. The input from visual design theory offered clarity about the simplest ideas of placement and the overlapping of graphical elements on the screen, the treatment of 'white space', typography and combinations of colours. The input from human computer interface theory was primarily from Donald Norman regarding the task analysis of operating equipment, and resulted in our own interpretation to guide colleagues in our team expressed as An Analysis of a Single Interaction (Millwood and Riley 1988).

Survey

In later work, relating to the development of courses and educational practice, evaluation through direct questioning of participants became an additional method used. Particularly with the advent of online surveys with their immediate and low-cost summary analysis, this became an important method. Development in this methodology to take advantage of the particular strengths of interactive designs came in the design of Making Choices (Millwood 1993), a tool for modelling decisions by interactively dragging choices into order and the COGs passport a tool for transition between primary and secondary schooling.  COGS helped learners evaluate their competencies by dragging elements in a geometric design. This design thread has been developed most recently in the design of interactive learning needs analysis for health professionals and volunteers in the charity Macmillan Cancer Support.

Videography

In several projects, understanding the holistic context and seeing the detailed activity became important. In these cases, making video of the activity or of the discussion to evaluate it was employed, although this could prove challenging to access and analyse. In some cases, (Ultraversity Project 2006), the video was transcribed and the transcription added to the video as a 'text track' which was searchable. Added value could be obtained by adding text tracks for chapters and for keyword analysis, permitting the video to be used as the vehicle for dissemination of research findings, not simply the data gathered.

Structured Interview and grounded theory approaches

In creating innovation in higher education, it became important to evaluate the experience of students and tutors in greater depth. In these cases we developed interview frameworks, conducted the interviews, recorded the audio transcribed and then employed an interpretive phenomenological analysis (Smith et al. 2009) to the data to discover in a grounded sense, the key themes of their response to our innovations (Millwood and Powell, 2009).

Philosophical Approach – Pragmatism

The philosophical approach of Pragmatism - that the function of thought is as an instrument or tool for prediction, action, and problem solving (Peirce, 1935, James, 1898) - has inspired my work and specifically been employed in the production of this thesis. It has guided me to gather my work practice, discover those aspects which have made the greatest contribution and attempt to link them to theoretical perspectives through the thesis web site.

Methodological Approach – Autoethnography

Although it is clear that the approach I have taken is of autoethnography, there are variants, and my approach has been closest to that defined by Ellis (2004) - “research, writing, story, and method that connect the autobiographical and personal to the cultural, social, and political”. Although I set out to describe and look critically at my experience, there is also the deliberate attempt to find theory in this thesis, and a move from my tacit theories to those articulated in the analyses, [A1], [A2], [A3] and [A4], published with this thesis, where the intent is to provide reliable tools to other designers. I hope that this desire and the positive outcomes of much of the practice I have been engaged in will counter the criticisms levelled at auto-ethnographers as "unscientific, or only exploratory, or subjective" (Denzin & Lincoln, 2005, p8).

The five factors described by Richardson (2000, pp 15-16) for evaluating  such work are used here to justify my position, and for you the reader to judge my success as set out in table 2.

Table 2: Richardson's factors for evaluating autoethonographic work

Specific Method

Designing the thesis web site

From the outset, the processes of gathering, categorising, reflecting, selecting and presenting were identified as knowledge and information management tasks, which from the author's perspective demanded a content management system (CMS). The practice of designing such a system was aligned with the author's experience and ambition, offering not only a vehicle for development but also dissemination and participation. The structuring, semantic tagging, work-flow, language translation, accessibility, visual design and multimedia features of the Plone CMS were seen as appropriate for the task based on experience using this CMS for the websites of key relevant professional organisations in recent years - Ultralab, Core Education UK and the National Archive of Educational Computing.

Gathering the evidence

The first step was to enter the events in my practice using the 'Event' content type in Plone and collecting these in the Portfolio section. Each event consisted of a title, summary, description, start and end date for each of the elements of my practice. I chose to be broad in scope, creating an auto-biographical account which is more complete than required for this thesis, but allowed decisions on relevance, importance and contribution to made through a second pass. An important consequence of this process was the positive effect of building a rounded account of my life experience leading to a holistic picture. The outcome is a list of around 400 items of practice.

Categorising the evidence

Each entry was tagged as belonging to one of seven categories that emerged from considering the kinds of practice I had engaged in:

1. education - events in my formal lifelong education;
2. employment - posts held;
3. project - research and development projects undertaken;
4. professional - positions of professional activity, e.g. societies, examination, advice;
5. conference - participation in conferences;
6. publication - papers and other media published formally;
7. teaching - activity where my rôle was to teach others.

Adding reflections and selecting key contributions

To create a manageable portfolio for assessment of this thesis, a selection of events was made that seemed to offer potential for the development of a doctoral thesis through a process of reflection. These events were edited to include a paragraph or more of reflection, identifying the key elements within them that influenced the development of my design practice and assessing the contribution made in what were frequently collaborative activities. As well as clarifying the nature of my contribution, I assessed the proportion of it in crude percentage terms.

From these items, a further selection was made to shorten the list to form the basis of a claim for examination. These items are tagged 'claim' and show with a white background and bold text in the portfolio timeline as shown in Figure 3.

Figure 3: Claim items highlighted in white on a timeline of my professional practice

Each of these items has been shared with the original key collaborators, where possible, and they have formally agreed to my judgment of contribution. This process was followed as part of registration for this degree and is submitted to the Board of Studies for Research Degrees as one of the conditions of registration.

On reflection, I consider the estimate of my contribution in percentage terms to be too limited, and in future would propose the use of more qualitative terms ranked as:

• leader;
• one of a pair;
• member of small team;
• critical friend and
• member of large team.

Identifying originality, impact and importance

A final process of identifying the originality, impact and importance of the items of practice on which I based the claim was undertaken and referenced to evidence. In some cases the practice was very public and on the large scale and may be readily judged for these factors by other academics and practitioners in the field.

The outcome of this process forms the basis of the Claim made in that section of this thesis.