In an ideal world, corroborating evidence should be: i) independent; ii) robust and unbiased; and iii) publically available. However, finding evidence for some impacts is more challenging than others, and may require a substantial research effort in its own right. Designing a robust evaluation is a fairly technical task, and you will probably want to draw on social science expertise to design an evaluation that unambiguously demonstrates cause and effect and attributes impacts to your research.
To make this happen, you have a number options open to you. You could commission an independent consultant to design and carry out the evaluation for you.
However, despite the fact that the evaluation is independent, it is unlikely to look independent on a list of “sources to corroborate the impact” in an impact case study, if it is published on your University website. The alternative therefore is to work with a stakeholder or public representative organisation linked to your impact case study to co-design, carry out and publish the evaluation (potentially with assistance from social scientists in your institution). If they do not have the resources or expertise to carry out the evaluation themselves, then you might want to pay for the independent consultant to do the work for them. They will have to acknowledge your funding, but the evaluation is independent and it is published on their website, so it looks independent as well.
Not everyone has access to the funding to make this possible however. A further option therefore, which requires time and expertise instead of funding, is to conduct the evaluation yourself with social scientists in your institution and write it up as a peer-reviewed paper. In some disciplines, it may be possible to build an impact evaluation into the next paper you write that builds on your underpinning research. This may even add value to the paper, giving it greater academic impact. For some, it may be possible to write the evaluation as a stand-alone paper for a journal in their discipline. For other disciplines, this is not an option, and they are reliant on working with social scientists to co-design, carry out and publish the evaluation in journals from their own field. The challenge here is to pitch this as an opportunity, as there are few social scientists who are likely to want to spend significant amounts of time providing evaluation services to academics from other disciplines. However, with some exploration and discussion, it may be possible to get a social scientist excited about the research opportunities presented by an evaluation.
Although not independent, if you can design your evaluation robustly and get it published in a reputable peer-reviewed journal, it is unlikely that your evidence will be dismissed. In theory you might have “cooked” the results to make yourself look good, but a strong research design in a good journal is likely to re-assure most panel members that you’re not trying to pull the wool over their eyes. Here are three types of impact evaluations leading to papers that I’ve helped design.
Design from scratch
As Director of Engagement and Impact for the School of Natural and Environmental Sciences in Newcastle University, I am helped one of my colleagues design an evaluation study to corroborate his impact (before he then left the University). He approached me with evidence that a paper he had written had led to a massive spike in the consumption of a product across Europe. Sadly the evidence was only in the form of a correlation, and did not prove cause and effect. The spike in consumption was significant and sustained, and co-incided perfectly with the publication of his findings, but how could we demonstrate without doubt that his paper was the cause of this spike rather than a co-incidence?
He explained to me that he was planning to publish another paper with even more striking and far-reaching findings early next year, and we identified an opportunity to do things differently. This time he would go back to his funders to commission a large-scale survey of consumers in the UK and Germany immediately before and after the publication of his work. We would ask them about their consumption of the products he was studying, and whether any changes we detected after publication were in fact as a result of reading about his work. I pitched the opportunity to a social science colleague of mine, Lynn Frewer, who jumped at the opportunity to work with us to test some hypotheses she was developing about consumer behaviour. Together we designed a methodology that should be sufficiently robust and novel to get our findings published in a good journal.
In a second example, I evaluated the impact of my own research with interdisciplinary environmental scientist, Ros Bryce (University of the Highlands and Islands), funded through an EU project concerned with science-policy dialogue. We conducted interviews with members of the policy community, asking them to identify if they had come into contact with over 70 research findings, one of which was based on the work of my colleagues and I. We then used qualitative interviews and Social Network Analysis to trace those findings as they were communicated from person to person until they either reached policy or not. Life intervened and we didn’t finish writing the paper, but in the intervening period, I met a human geographer, Ruth Machen (Newcastle University), who was studying similar questions with the same group of policy-makers. The resulting paper, now under review in Evidence & Policy, combines insights from the qualitative and quantitative analysis that Ros and I conducted, with Ruth’s in-depth work. It makes a significant methodological contribution in addition to the new empirical understanding of the specific science-policy community we studied.
Policy report to research article
The second type of impact evaluation I’ve written up as a research article started life as a policy report. This example is more of an attempt to evidence rather than evaluate policy impact. I was faced with a problem where I had invested significant research effort to support a policy document (the United Nations Convention to Combat Desertification’s Global Land Outlook), but I did not have any control over the way our research would be cited in their report. I needed to create a publically available evidence trail that would prove that a significant part of the report arose from my research.
To do this, we wrote the first draft of our contribution to the report as a research paper, and then converted this into simpler language for inclusion in the report with a request that the report cited the research paper that we then submitted to the journal, Land Degradation and Development (it is accepted and in press). The journal article includes a footnote on the front page explaining, “a condensed version of this paper is part of the UNCCD’s Global Land Outlook, published 17 June 2018”. The idea was that whether the report cited our work or not, we would have a published link between the two outputs, with clear overlaps in the text and ideas contained within the two documents. In the end, the policy document cited a working paper version of our paper with the same title, and the link is robust.
Finally, I published a paper earlier this year in Global Environmental Change that contained a substantive section detailing impact evaluation work I conducted with PhD student, Kathleen Allen. We did the work in collaboration with the IUCN UK Peatland Programme, who funded her travel expenses to accompany me to run focus groups across the UK to evaluate the impact of research I had submitted to REF2014. The impacts were mixed, but made for interesting reading, enriching a paper that would otherwise have been more descriptive in nature. This enabled us to publish the work in a higher impact journal that might otherwise have been possible. I have not yet decided if I will use this paper as one of the “references to the research” or a “source to corroborate the impact”. It could be used for either purpose.
Reed MS, Bryce R, Machen R (under review) Pathways to policy impact: a new approach for planning and evidencing research impact. Environmental Science and Policy
Reed MS et al. (2017) A Place-Based Approach to Payments for Ecosystem Services. Global Environmental Change 43: 92-106.
Thomas RJ, Reed MS, Clifton K, Appadurai AN, Mills AJ, Zucca C, Kodsi E, Sircely J, Haddad F, von Hagen C, Mapedza E, Woldearegay K, Shalander K, Bellon M, Le QB, Mabikke S, Alexander S, Leu S, Schlingloff S, Lala-Pritchard T, Mares V, Quiroz R (in press) Modalities for Scaling up Sustainable Land Management and Restoration of Degraded Land. Land Degradation & Development
By Mark Reed