One of the approaches that we have adopted to support projects with their evaluation activities is to establish pair/teams of evaluation buddies.
An evaluation buddy is basically another project who can:
- check that evaluation plans are sensible, credible, valid, do-able, relevant to programme aims
- provide peer review of content, interfaces, and other practical outputs from the projects
- sharing ideas, experiences and interim outcomes of evaluation
- swaping time and expertise, e.g. analyse data for one another (extra objectivity
- sharing resources e.g. external consultants buddy up for e.g. workshops, critical moments (writing up etc)
- act as a sounding board and be a neutral ear to articulate findings, thougts and any concerns about evaluation outcomes.
It is a support mechanism that can develop into long term relationships for future OER work. Of course it does not preclude the development of other useful relationships across projects. We describe a bit more about this approach in the toolkit.
We held the first of a series of meetings with these pairs/groups this week. This meeting included three projects REACTOR, Opening up a Future for Business and CORE-SET. These projects have several areas of commonality which we explored in the meeting.
Each project gave a brief overview of their project and the evaluation questions/issues that thay were looking at.
Led by University of Liverpool, this project builds on the Pilot Phase project CORE-Materials and expands the remit to various stakeholders outside HE (including private companies and 3rd sector agencies). They see the value of adopting a baselining approach which offers a more systematic approach to evidence gathering. They have developed an instrument to gather current understanding and practice of their stakeholders and will apply that at various stages throughout the project to evidence change. This is likely to reveal some interesting information about readiness of other sectors to engage with OER and highlight perceived barriers and enablers.
Led by Doncaster College this project builds on the UKOER phase 2 project SPACE and aims to develop 3D interactive resources for environmental technology which will be made available on web from project website, but also to be downloadable onto mobile devices. Also focussing on a range of sectors outside HE this project has also invested significant energy into engaging very different groups of stakeholders (including sector skills councils, private companies, public sector, as well as HE in FE) Early research to identify what teachers and students within these curriculum areas want or need has been supported by the wider networks of these agencies. They also bring different resources to input to the OERs and offer opportunities to make sure they reflect the curriculum needs of many stakeholders.
Opening up a Future for Business
Led by Southampton Solent University this project has an FE & 6th form college audience as it aims to collate and produce an OER to enable 16-19 year olds, thinking about their future, to move forward with confidence in to studying Business and Management topics in Higher Education. The project emphasised the need to reconceptualise their approach as the project has progressed. By examining how FE teachers already use OERs they discovered several institutional barriers such as blocking of some sites, to very protective and risk averse management of access to services on the web. Copyright restrictions on materials produced by other educational institutions have affected what teachers use and what can be adapted as OErs. The project is looking at how openness spreads within institutions. All of these issues are impacting on platform choice for their disaggregable book – with Moodle emerging as a strong choice as many of their stakeholder institutions use this VLE.
Common themes and strands
The following areas/linkages were noted as significant for this group of projects and we anticipate interesting lessons emerging around these:
- Linking with students as important stakeholder group
- Students for evaluating and testing
- Students as project employees
- Student-generated content
- Subject area linkages- engineering, environmental technologies, business
- Linking with a wide range on non HE stakeholders – not just educators and students in these organisations (lessons around stakeholder engagement, practice change, challenges/enablers)
- Choice of Platforms (links to different stakeholder requirements, constraints and influence)
Isobel and I found this meeting really valuable and the projects also highlighted the usefulness of this approach to enable sharing of similar evaluation instruments, sharing processes (not just evaluation related), talking through issues and reflecting on emerging lessons at regular intervals, and in the longer term possibly in presenting evaluation outcomes to a wider audience. We also discussed the evaluation toolkit and how this can help with reflection on lessons learned and preparing for reporting.
This buddy team did note the value of meeting as they prepare their Interim reports and are planning another meeting in June. We will meet again as projects prepare for their final reports too.