Evaluation toolkit
We are delighted to launch our new Evaluation Tookit which aims to benefit UKOER projects by providing a structured way to collate their findings and observations and can also be used in the reporting process. It also provided lots of information, advice, examples and videos, on evaluation, sources of evidence, and buddying. To help projects get the most out of the toolkit, we have produced a guide.
- UKOER Evaluation toolkit on the wiki
The interactive element of the toolkit provides two alternative routes into the structure – through the programme themes, or through key evaluation focus areas. Both routes utilise the same underlying mechanism to record findings, observations and evidence – a series of google forms and spreadsheets – but the higher level user interface is very different.
- The key focus area route, via links on dedicated wiki pages, is useful for those familiar with our evaluation framework. Each focus area has an individual wiki page which provides a description of that area and some specific issues and aspects that the programme is investigating.
- The programme themes route offers a more visual approach via an interactive mind map (using spicynodes) where you can explore the themes of the UKOER3 programme and drill down to find the specific issues and aspects that relate to the themes you are investigating. We have produced a quide to using the spicynodes maps.
Whichever route projects choose, they can opt to receive an email record of their entries for future reference and report writing.
As projects approach writing their interim reports, we hope this provides an opportunity to reflect on, and perhaps reframe, the evaluation questions they started with, in the light of experiences to date, as it becomes clearer where their valuable contributions lie.
We welcome feedback on the toolkit, and expect to revise and improve it in the light of comments.
Open educational practices – what do we mean?
To conclude our posts for open education week, I’ve been asked to introduce our new briefing paper on open educational practices.
I think we’ve been talking about open practices since the pilot phase of UK OER – you can find it in some of our recommendations. When we set out, I for one had the idea of a perfectly formed OER as openly licensed, self-contained, professionally tagged with educational metadata, and probably hosted in an open repository. Very quickly it was clear that open release would be messier and more interesting than that. OERs were caught up in a host of other emerging practices, such as guiding students to freely available content (with open licensing being only one way of judging value), or using third party services to support students outside of institutional learning environments. The people who were interested in making open content had other open agendas too, some of them very radical.
One problem with tracing the links between open practices is that they tend to be ‘owned’ by different agencies. Even within JISC there are fascinating developments in open education at large, in open access publishing, and in open research data management, that I am only just finding out about. Can we build better connections? The Curriculum Delivery programme has reported on new pedagogical models in which open content makes good sense, while the Curriculum Design programme aims to make design teams more effective and collaborative – couldn’t high quality OERs be of some help here??
Finally, I’m interested in how ‘open’ embodies a different set of values for different people. Other members of the team have pointed out to me that open learning is not the same beast as open education. And our work with earlier phases of the programme confirmed that different stakeholders have very different perceptions of benefit and risk. It’s important to acknowledge the complexity of the open landscape and I hope our new briefing sheds a little light. Do also head over to where Amber Thomas has collated some visualisations of the wider open education landscape. And please comment!
UKOER3 themes and activities
moodle showing UKOER3 themes and areas of activity
We are currently going through UKOER3 project plans and drawing out key themes and areas of activity to map these against our synthesis and evaluation framework.
We use the framework as an iterative tool to support both evaluation activities of projects and synthesis activities of our team. It highlights core areas of inquiry and identifies evaluation questions that projects intend to address. We started with a working pilot phase framework in 2009 which has since been through several iterations. The latest version of the framework was created at the end of phase 2 in November 2011 and links to evidence from projects. We are currently revising this framework to reflect phase 3 activities and questions (coming soon!).
We have listed phase 3 projects and their evaluation questions and themes/areas on a new wiki page and the wordle is made from the terms in the third column of this page – themes/links. UKOER projects may find this wiki page useful to identify other projects to make links with. The funding call for the THEMES projects did identify 4 themes which has obviously shaped some of the focii.
A: Extend OER through collaborations beyond HE
B: Explore OER publishing models
C: Addressing sector challenges
D: Enhancing the student experience
It’s early days for phase 3 but what strikes me so far is that the issue of student created OERs has emerged as a more prominent focus this year. Identified by a few projects in the pilot phase as an area of interest, raising issues around ownership, digital litercaies, and licencing, it failed to attract alot of interest during phase 2 with only a few projects including student content as part of their OERs. I am pleased to see this re-emerge (under the banner of Enhancing the student experience) because I think this area needs investigating more.
Cross -sector partnerships continue to be a major theme and this phase will see several projects taking up the challenge of working with publishers – indeed new OER publishing models are being considered and hopefully developed during this phase. 3rd sector agencies as partners have also emerged with several projects working with charities and the voluntary sector – another area to watch with interest. Some projects plan to work with industry, NHS, skills councils and SMEs, continuing some interesting work that was carried out during phase two with these groups. Projects will continue to work across educational sectors and schools also feature as partners in some projects. We have been working on a briefing paper which highlights issues emerging during phase 2 around open practice across sectors. We expect this phase of activities to inform and expand on this work.
Digital literacies featured strongly during phase 2 and continues to be a focus area for many projects, as do research skills and academic practice, with the OMAC strand particularly focussing on this area. The OMAC (open materials for accredited courses) strand aims to build on the outcomes of phase 2 and focuses on the release of materials linked to the new UK Professional Standards Framework (UK PSF) for staff who teach in higher education.
A key focus of phase 2 was changing teaching practice and we noted a move of emphasis from OERs to open edcuational practices. This is a continued area of focus for projects this year and is likely to raise some interesting debate and discussion.We are currently preparing a briefing paper on open educational practice – so watch this space…
One of the challenges for UKOER project timescales is that they often underestimate the time taken to release the OERs and have little time to find out how they are used by different stakeholders. We may find that projects are able to report more about this (particularly academic use of OER) as some projects are intending to re-use existing OERs. Several projects have included a remit to investigate student use of OERs – another area which should be of significant interest to the community.
So, in summary, I think we are in for a really interesting time during phase three and I’m looking forward to working with the projects as they attempt to answer their evaluation questions. Not least I always look forward to those unanticipated findings that come from taking a few risks and being experimental…
If you have not had the chance to read the findings from phase 2 (we have been a bit tardy in reporting this here – but hope to do a series of blogs focussing on different aspects of this soon) then do check out our Phase 2 Synthesis report. It’s a meaty read but you can dip in and out of the different sections…
UKOER3 startup
The evaluation and sythesis team took part in the UKOER3 startup event in Birmingham on November 14th 2011. The event brought together the 18 projects in UKOER3 with the programme leaders and relevant support groups, including CETIS, JISC Legal, JorumOpen and the E&S team. I (Allison Littlejohn)was representing the E&S team, while the others from E&S ((Helen Beetham, Lou McGill and Isobel Falconer) participated via Twitter.
Our role in UKOER3 is to lead the evaluation of the programme from October 2011-October 2012. We are working with all 18 UKOER projects to identify key messages for the JISC, the HEA and the sector as a whole. We are delighted – and excited – to be leading the evaluation and synthesis of the programme, building on our work in phases 1 and 2 . During UKOER2 (20010/11) we identified key benefits and motivations for academics and institutions to create and release Open Educational Resources – see our UKOER2 final report http://tinyurl.com/bu5lvge which was launched the the UKOER3 startup.
In phase 3 we will evaluate the programme using the synthesis and evaluation framework we developed through phases 1 and 2. The framework has a number of key themes, each with a range of evaluation questions that are asked by project teams as they carry out their individual project evaluations. The data they collect was mapped to the questions in the programme evaluation framework, providing an overview of key issues and trends across the programme. Thus we identified key lessons learnt and outcomes and highlighted significant outputs that demonstrate evidence of this.
Watch this space….