Evaluation buddy meeting between REACTOR, Opening up a Future for Business and CORE-SET
One of the approaches that we have adopted to support projects with their evaluation activities is to establish pair/teams of evaluation buddies.
An evaluation buddy is basically another project who can:
- check that evaluation plans are sensible, credible, valid, do-able, relevant to programme aims
- provide peer review of content, interfaces, and other practical outputs from the projects
- sharing ideas, experiences and interim outcomes of evaluation
- swaping time and expertise, e.g. analyse data for one another (extra objectivity
- sharing resources e.g. external consultants buddy up for e.g. workshops, critical moments (writing up etc)
- act as a sounding board and be a neutral ear to articulate findings, thougts and any concerns about evaluation outcomes.
It is a support mechanism that can develop into long term relationships for future OER work. Of course it does not preclude the development of other useful relationships across projects. We describe a bit more about this approach in the toolkit.
We held the first of a series of meetings with these pairs/groups this week. This meeting included three projects REACTOR, Opening up a Future for Business and CORE-SET. These projects have several areas of commonality which we explored in the meeting.
Each project gave a brief overview of their project and the evaluation questions/issues that thay were looking at.
CORE-SET
Led by University of Liverpool, this project builds on the Pilot Phase project CORE-Materials and expands the remit to various stakeholders outside HE (including private companies and 3rd sector agencies). They see the value of adopting a baselining approach which offers a more systematic approach to evidence gathering. They have developed an instrument to gather current understanding and practice of their stakeholders and will apply that at various stages throughout the project to evidence change. This is likely to reveal some interesting information about readiness of other sectors to engage with OER and highlight perceived barriers and enablers.
REACTOR
Led by Doncaster College this project builds on the UKOER phase 2 project SPACE and aims to develop 3D interactive resources for environmental technology which will be made available on web from project website, but also to be downloadable onto mobile devices. Also focussing on a range of sectors outside HE this project has also invested significant energy into engaging very different groups of stakeholders (including sector skills councils, private companies, public sector, as well as HE in FE) Early research to identify what teachers and students within these curriculum areas want or need has been supported by the wider networks of these agencies. They also bring different resources to input to the OERs and offer opportunities to make sure they reflect the curriculum needs of many stakeholders.
Opening up a Future for Business
Led by Southampton Solent University this project has an FE & 6th form college audience as it aims to collate and produce an OER to enable 16-19 year olds, thinking about their future, to move forward with confidence in to studying Business and Management topics in Higher Education. The project emphasised the need to reconceptualise their approach as the project has progressed. By examining how FE teachers already use OERs they discovered several institutional barriers such as blocking of some sites, to very protective and risk averse management of access to services on the web. Copyright restrictions on materials produced by other educational institutions have affected what teachers use and what can be adapted as OErs. The project is looking at how openness spreads within institutions. All of these issues are impacting on platform choice for their disaggregable book – with Moodle emerging as a strong choice as many of their stakeholder institutions use this VLE.
Common themes and strands
The following areas/linkages were noted as significant for this group of projects and we anticipate interesting lessons emerging around these:
- Linking with students as important stakeholder group
- Students for evaluating and testing
- Students as project employees
- Student-generated content
- Subject area linkages- engineering, environmental technologies, business
- Linking with a wide range on non HE stakeholders – not just educators and students in these organisations (lessons around stakeholder engagement, practice change, challenges/enablers)
- Choice of Platforms (links to different stakeholder requirements, constraints and influence)
Isobel and I found this meeting really valuable and the projects also highlighted the usefulness of this approach to enable sharing of similar evaluation instruments, sharing processes (not just evaluation related), talking through issues and reflecting on emerging lessons at regular intervals, and in the longer term possibly in presenting evaluation outcomes to a wider audience. We also discussed the evaluation toolkit and how this can help with reflection on lessons learned and preparing for reporting.
This buddy team did note the value of meeting as they prepare their Interim reports and are planning another meeting in June. We will meet again as projects prepare for their final reports too.
TEPL-SIG webinar
On 2st February 2012 isobel Falconer and I led a webinar for the TEPL-SIG (Technology Enhanced Professional Learning – Special Interest Group) on Open Practices which aimed to encourage participants to consider their own practices. We considered open educational practice (OEP) but also touched on issues around wider open practices with a focus on how this might differ for people in different stakeholder roles (learner, teacher, employer -or even a mix of these). We were keen to take a broad view to emphasise that open practice in one aspect of your life can lead to or influence openness in other areas. The level of participation was excellent and I left hoping for an opportunity to take this further as we only had time to touch on a few issues.
The group included people from around the world with several different roles and levels of experience with OER or OEP. It was nice to have some of the UKOER project team members there as we were drawing on their work and the work of the UKOER programme to illustrate our discussions. We made a wiki page for participants to refer to prior to the session with specific reference to the new briefing paper on open practice across sectors. below are the slides that we used for the session.
Before talking about UKOER and findings we asked particpants a series of questions which really got the discussion flowing (see slides 2 and 3). We were basically asking if open practice is a new concept or if our existing practices are just changing – not rocket science to be sure but a useful way to get participants to reflect on their own practices and how these are impacted by organisational cultures. As expected we did have attendees from professional contexts where some organisational knowledge was seen as a commodity that shouldn’t be shared. We discussed issues around degrees of openness where, in some contexts, there were valid reasons for maintaining some restricted access – a good example of this being around patient data in the NHS.
Isobel offered some insights from the UKOER programme which led to discussion around the motivations for open sharing and the benefits in different contexts. We finished with by considering the challenges and barriers for different sectors.
What is always frustrating at events like this where you have a very fast moving text chat is that people say something really interesting and then chat moves on and you don’t have time to really delve into a particular aspect. The conversation touched on areas such as:
- inequalities and whether OER adds to the rich/poor divide or actually helps to challenge it
- issues around use – the differences between learner/teacher use and how the means of production impacts on quality and size of OER/OCW
- organisational showcasing
- notion of institutions paying ‘lip service’ to the principle of being ‘open’ online but not necessarily considering the value of this in a local community context
- sustainability and embedding of tools to support open processes (and challenge of integrating these into existing practice)
- issues around confidence of individual teachers
- fears of litigation
- questions about why we are even talking about this – are some initiatives already making our discussions obsolete (Coursera, Udactity, MITx)
- how current initiativess around assessment and accreditation are going to impact on things
One of the people attending suggested that we hold the webinar again during open education week which has not been possible – but we hope publishing this post will at least provide some useful information to support the activities during the week and you can watch the recording of the session too. we would be delighted if you would like to continue the conversations we started here in the comments section too.
UKOER phase 2
Synthesis activities for the first two phases of the HE Academy/JISC UKOER Programme have included cultural and institutional issues across a number of sectors. The Evaluation and Synthesis team’s final report on phase 2 synthesises evidence and outputs under five main headings:
- Practice change
- Development and Release Issues
- Cultural Considerations
- Institutional Issues
- Impacts and Benefits
and includes recommendations to the funders and to the stakeholders represented in the various strands of the programme. Further, detailed findings from each component strand of the programme addressed how different communities and cultures are progressing towards openness in their educational practice and in their management and use of educational resources:
- Release – projects undertaking release of new or repurposed OERs in key subject areas
- Open Materials for Accredited Courses (OMAC) – projects releasing materials linked to the national professional standards for staff who teach in higher education
- Cascade – projects cascading and embedding their own good practice in OER release to other contexts
- Collections – projects using a range of technologies to collate existing OER in thematic collections
On practice change, evidence is accumulating that teachers do share content informally, but do not necessarily consider IPR, or share resources openly. Thus they are engaging in some of the practices associated with OER, but they do not necessarily recognise OER terminology, and in asking about use/reuse of OER we may be asking the wrong questions. This finding has implications for development and release, where cascade strand projects noted that choice of search engine affected which resources were found and used, while collections projects found that their users expected searches to be “google-like” in their ease of use, personalisation, and production of relevant results. Users expressed frustration with the search functionality of many OER repositories. They were also disappointed in the scarcity of relevant OERs, a finding echoed by the release strand PORSCHE project. Most of the collections projects decided to include “grey” or “non” OERs to get round this problem – while clearly labelling them so that their non-licensed status was clear.
Thus a key question that is emerging as important to OER practice is what difference does it make to practices of development/use/reuse and sharing, that the resources are “open”?
Experience during this phase confirmed that OER practice cannot be divorced from other open practices.
In some disciplines, sharing practice through a range of open technologies has emerged as important as sharing resources, and is having an impact on the way subjects are being studied and taught. Considerations of OER use cannot be divorced from these wider changes to disciplinary knowledge practices. Social science subjects, for example, are being changed in radical ways by the availability of public social and research data online as well as the rise of new social/digital practices.
Findings from this phase confirm those from the phase 1 institutional strand, that there are different cultures of openness at different educational institutions. This is not as simple as a single dimension from closed to open: rather there are many different ways in which institutions can support open educational practices.
For all stakeholders, then, a key question is, why engage in open practices? But equally, what is it about a practice that makes it open?
Our final report on phase 2 offers no simple answers, but lots of evidence to support you in exploring these issues.
open practices across sectors: briefing paper
The HE Academy/JISC UKOER Programme has encouraged cross-sector approaches to OER development, and is increasingly focussing on broader issues around open educational practices (OEP). Synthesis activities for the first two phases of activity have included cultural and institutional issues across a number of sectors. Activities to date include a UKOER phase 2 programme webinar on OERs across sectors , a presentation at the JISC Innovating eLearning Online Conference in November 2011 – Open practice across sectors and a recent TELP-SIG webinar on open practices. It is anticipated that phase three activities will continue to address cross-sector issues and identify good practice for the wider community. It has become clear that a significant benefit of engaging with the concepts and challenges of OEP and OER is in the way it encourages cross sector understanding, collaboration and outcomes.
same horizon by lou mcgill cc-by-sa
The UKOER Phase 2 Synthesis report considered practice change in detail:
“Collaborative practice has emerged as important during this funding phase. Cross disciplinary approaches are beginning to have an impact at an institutional level and reveal a new benefit of open content – that it is easily shared and co-constructed across existing boundaries. Engaging with partners outside the academic sector has been challenging but has encouraged new partnerships, trust and levels of understanding. Several projects comment that working across boundaries to develop project outcomes (business/community/academy, staff/consultants, students/teachers) has been one of the most radical aspect of their experience and has the potential to change practice more widely.”
We invite you to read one of our new briefing papers which looks at the various motivations for different stakeholders across sectors in engaging with OEP and OER and it also identifies some synergies and differences between HE and HE in FE, NHS and employers.
One of the core questions around open educational practices seems to be around the terminology:
Recognising new or changing practice as ‘open’ or OEP has added to the complexity in the field as it needs defining and explaining – is existing practice becoming more open or does it require people to change their practice? During the JISC online conference discussions many people argued that some people have been engaged in open practice (and even producing OERs) for many years but do not use this terminology to describe it. Whilst some may prefer not to use new terminology, it can be useful to engage people with the concepts and generate institutional-wide understanding that can be fed into strategy, policy and practice. It can also be valuable to situate open practices as an extension of existing practices, which may generate less anxiety or resistence and establishing OEP as a credible and valid form of scholarship has been identified by projects as one way of normalising these practices.
What are your views on this? Are you already an open practioner?
UKOER3 themes and activities
moodle showing UKOER3 themes and areas of activity
We are currently going through UKOER3 project plans and drawing out key themes and areas of activity to map these against our synthesis and evaluation framework.
We use the framework as an iterative tool to support both evaluation activities of projects and synthesis activities of our team. It highlights core areas of inquiry and identifies evaluation questions that projects intend to address. We started with a working pilot phase framework in 2009 which has since been through several iterations. The latest version of the framework was created at the end of phase 2 in November 2011 and links to evidence from projects. We are currently revising this framework to reflect phase 3 activities and questions (coming soon!).
We have listed phase 3 projects and their evaluation questions and themes/areas on a new wiki page and the wordle is made from the terms in the third column of this page – themes/links. UKOER projects may find this wiki page useful to identify other projects to make links with. The funding call for the THEMES projects did identify 4 themes which has obviously shaped some of the focii.
A: Extend OER through collaborations beyond HE
B: Explore OER publishing models
C: Addressing sector challenges
D: Enhancing the student experience
It’s early days for phase 3 but what strikes me so far is that the issue of student created OERs has emerged as a more prominent focus this year. Identified by a few projects in the pilot phase as an area of interest, raising issues around ownership, digital litercaies, and licencing, it failed to attract alot of interest during phase 2 with only a few projects including student content as part of their OERs. I am pleased to see this re-emerge (under the banner of Enhancing the student experience) because I think this area needs investigating more.
Cross -sector partnerships continue to be a major theme and this phase will see several projects taking up the challenge of working with publishers – indeed new OER publishing models are being considered and hopefully developed during this phase. 3rd sector agencies as partners have also emerged with several projects working with charities and the voluntary sector – another area to watch with interest. Some projects plan to work with industry, NHS, skills councils and SMEs, continuing some interesting work that was carried out during phase two with these groups. Projects will continue to work across educational sectors and schools also feature as partners in some projects. We have been working on a briefing paper which highlights issues emerging during phase 2 around open practice across sectors. We expect this phase of activities to inform and expand on this work.
Digital literacies featured strongly during phase 2 and continues to be a focus area for many projects, as do research skills and academic practice, with the OMAC strand particularly focussing on this area. The OMAC (open materials for accredited courses) strand aims to build on the outcomes of phase 2 and focuses on the release of materials linked to the new UK Professional Standards Framework (UK PSF) for staff who teach in higher education.
A key focus of phase 2 was changing teaching practice and we noted a move of emphasis from OERs to open edcuational practices. This is a continued area of focus for projects this year and is likely to raise some interesting debate and discussion.We are currently preparing a briefing paper on open educational practice – so watch this space…
One of the challenges for UKOER project timescales is that they often underestimate the time taken to release the OERs and have little time to find out how they are used by different stakeholders. We may find that projects are able to report more about this (particularly academic use of OER) as some projects are intending to re-use existing OERs. Several projects have included a remit to investigate student use of OERs – another area which should be of significant interest to the community.
So, in summary, I think we are in for a really interesting time during phase three and I’m looking forward to working with the projects as they attempt to answer their evaluation questions. Not least I always look forward to those unanticipated findings that come from taking a few risks and being experimental…
If you have not had the chance to read the findings from phase 2 (we have been a bit tardy in reporting this here – but hope to do a series of blogs focussing on different aspects of this soon) then do check out our Phase 2 Synthesis report. It’s a meaty read but you can dip in and out of the different sections…
UKOER3 startup
The evaluation and sythesis team took part in the UKOER3 startup event in Birmingham on November 14th 2011. The event brought together the 18 projects in UKOER3 with the programme leaders and relevant support groups, including CETIS, JISC Legal, JorumOpen and the E&S team. I (Allison Littlejohn)was representing the E&S team, while the others from E&S ((Helen Beetham, Lou McGill and Isobel Falconer) participated via Twitter.
Our role in UKOER3 is to lead the evaluation of the programme from October 2011-October 2012. We are working with all 18 UKOER projects to identify key messages for the JISC, the HEA and the sector as a whole. We are delighted – and excited – to be leading the evaluation and synthesis of the programme, building on our work in phases 1 and 2 . During UKOER2 (20010/11) we identified key benefits and motivations for academics and institutions to create and release Open Educational Resources – see our UKOER2 final report http://tinyurl.com/bu5lvge which was launched the the UKOER3 startup.
In phase 3 we will evaluate the programme using the synthesis and evaluation framework we developed through phases 1 and 2. The framework has a number of key themes, each with a range of evaluation questions that are asked by project teams as they carry out their individual project evaluations. The data they collect was mapped to the questions in the programme evaluation framework, providing an overview of key issues and trends across the programme. Thus we identified key lessons learnt and outcomes and highlighted significant outputs that demonstrate evidence of this.
Watch this space….
After ‘that session’ at Alt-C
Well the cattle have been rounded up, the settlers have lit their home fires, and the preacher and the gadabout are safely departed to bother someone else. Apart from the comedy value of our session on open country (blogged by David Kernohan at http://followersoftheapocalyp.se/oer-the-cowboy-gospel-altc2011), what were the serious messages from the debate at Alt-C?
Amber Thomas, Helen Beetham, David Kernohan and Dave White at ALT-C 2011
David argued that ‘My contention was that OER was a symptom of a wider systemic issue, which takes in publishing, the idea of the public intellectual, online life, online practice and information as a right.’
David White (in the cowboy hat) offered the insight that the open use/reuse of online content is now so deeply embedded that on the demand side there is little need for the term ‘open’ – though arguably the term ‘educational’ might be more distinctive of the kinds of content being developed by the OER programme in the UK.
Amber as the ‘lady sheriff’ described how open standards (the ‘law’) can facilitate relations in open country but also emphasised that institutions have to continue to deliver learning at a time when existing models of higher education funding are under threat. We should beware snakeoil salesmen bearing the promise that open content can fill the void when teachers are dismissed.
Finally, my own slides (http://www.slideshare.net/hbeetham/open-country-hb) explore some issues emerging from final reports, which no doubt will be expanded on as we dig deeper. It has been exciting to see so many projects not only producing and sharing more resources under the UK OER banner, but reflecting deeply and productively on the wider issues of open education.
Photo: Josie Fraser
OER at Alt-C 2011
There are a number of papers and symposia at this year’s ALT conference that will be interesting to UK OER projects and followers. Links are to the Crowdvine site which requires a conference log-in. Papers may well be mirrored in other locations at a later date.
The Paradox of Openness: the high cost of giving online explores the real costs of online participation and sharing of resources. It asks, among other things, whether content sharing via open platforms is always ‘fair’ and whether academic communities can sustain open practices in times of financial constraint.
Making open educational resources happen comprises three short papers with several presenters from the UK OER programme. They explore designing for openness, supporting openness in institutions, and open content for interprofessional learning.
The World Collaboration session includes discussion of a proposed open learning exchange – not exactly open content, but drawing inspiration from the mutual sharing of content as a model.
Are we in Open Country? is a UK OER symposium with a wild west theme (!) that explores the promise and reality of working with open content. I plan to draw on emergent findings from some of the phase 2 projects, along with my co-presenters, to ask whether some of the wilder promises of the open content movement are relevant to life in present day universities and colleges, and what open practices are most valuable in uncertain times.
Three high profile UK OER projects combine to present Happening Resources, a session that touches on accessibility, OER in clinical education, and the role of the SCORE Fellows in advancing OER research and practice.
Finally, the session on the GLOmaker Tool could be of interest to those designing reusable open content from scratch, and for those concerned about the digital literacy of end-users.
Update on open content/open practices
A high-octane meeting this week with the folk from the TALL team and their OER Impact Study http://oerblog.conted.ox.ac.uk/. It let me to sketch out the area that I think we are gathering evidence about in the Eval and Synth team, which is a more detailed picture of the two right-hand quadrants in the diagram posted below (features of open content, and aspects of open practice). On the left side are some emerging practices in learning and teaching that could arguably be called ‘open’. OERs are *involved* in these practices, sometimes as signs that they are going on, sometimes as drivers to make them happen, sometimes just in the background. So OER practices are a sub-set of these more open learning and teaching practices and it’s our task in the Eval and Synth team to gather evidence about whether and how the subset of practices promoted by UK OER are supportive of these larger movements. On the right hand side are some features of content that arguably makes it more reusable in an educational context – makes it more likely to partake helpfully in some of the practices on the left. We are familiar from UK OER pilot projects with the challenges and costs of moving content in these directions. However, we need more evidence of whether – and how – these features of open content actually support educational re-use. We have focused a lot on open licensing, which is critical for repurposing, but the focus in this phase is more on reuse and it may be that other features of open educational content are actually more significant here. The ‘E’ may be just as important as the ‘O’.
Printable version OER venn (pdf)
Impact model diagram
This is the current version of our UKOER impact model. We have included a description of this and have identiified emerging issues from phase 2 of the programme in more detail on our wiki https://oersynth.pbworks.com/w/page/40291776/UKOER-Impact-Model
From the literature we know that OER release is influenced by the nature of the resources and the practices around OER creation, release and reuse. Resources release and the practices around them can be considered individually (ie the practices of an individual learner or teacher releasing or reusing resources) or socially (ie the practices of groups or collectives releasing or reusing resources).
The release of OERs and practices around these is situated within the wider educational context of Further and Higher Education. Within this broad context individual and social practices influence the release of different types of OERs, and the release of these resources, in turn, affect the institutionally based practices associated with them. Further, we recognise that these practices and resources exist within a wider societal context in which open practices and resources are evolving rapidly. These aspects of OER release are integrated in the UKOER Impact Model below.
Individual impact (the left hand side of the model) is being explored by other parts of this landscape are being explored by other funded projects, for example the OER Impact Study, led by the TALL group at the University of OxfordOER Impact Study, and Open Resources: Impact on Learners and Educators (ORIOLE), led by the UK’s Open University). Through our evaluation of the whole JISC UKOER programme, we are focusing on examining the right hand side of the model, taking a social focus.
