EFP Brief No. 249: Measuring Foresight Impact

This brief describes a new instrument for measuring the impact of foresight. The foresight impact measurement instru-ment consists of 54 measures covering a wide range of foresight activities and potential policy and other impacts. This instrument, developed primarily by Ron Johnston and the author, is the result of several sessions with leaders of many of the most active national foresight programmes and includes a variety of types of measurement categories – notably those that align with the policy cycle in terms of positioning foresight for future impacts on policies as they emerge or are developed. It also has been pilot-tested on two Canadian foresight programs – in both cases achieving strong participation rates, high frequency of written comments and positive assessments of most of the measures and very strong endorsements of several key measures. One of the cases, a national foresight project on animal health and food security is described in this brief. Essentially the instrument provides a baseline for interim evaluation – while the experience is still vibrant – and in so doing it can (1) provide a unique mix of qualitative and quantitative feedback for stakeholders, participants and sponsors; (2) be immediately applied if required to making the case for continuity, future foresight funding or new projects; (3) form a credible baseline against which more formal evaluation can be structured later; and (4) help create a key international benchmark data base entry and case example of public sector foresight impact measurements – and thus position the EFP well for the future.

The Impact-Value Challenge

A key recurring challenge for foresight initiatives – projects, programmes and pilots – has been how to actually demonstrate the value of foresight investments for government sponsors and stakeholders – who are mindful of accountability, are asked to justify the value of foresight investments for government mandates and are requested to provide cost-efficiency and cost-effectiveness analysis so that foresight can be compared with other prospective applications of limited government funds.

The methodology elaborated below is a response to this challenge, prepared by Professor Jack E. Smith with input from senior international foresight leaders from the US, Europe (UK, FR, NL, FN) Australia and Asia ( TH, CH, KR, SP). The methodology draws upon discussion papers presented by the author and Professors Jon Calof and Ron Johnston at five international meetings. The challenge was to assess how to effectively measure impacts of foresight for government sponsors, operating in the short to medium term of 1-3 years when ideally these foresight impacts occur over a (mid to long term) five- to fifteen-year time horizon.

Case Study on Animal Health and Food Security in Canada

In September 2011, the Fore-Can Project on Animal Health and Food Security completed a three year foresight-based assessment of major challenges and opportunities associated with the future management of animal health and food security systems in Canada. The project was well received, involved a wide range of stakeholders and effectively engaged key policy advisors and industry leaders. As with many foresight projects, questions of immediate and enduring impact were raised as the end of the project drew closer. Fortunately this timing has coincided with the development of a new series of long and short format impact measurement instruments as part of an international forum of foresight best practices (more below on this).

Accordingly, the Fore-Can management team decided to be the first project to apply the new instruments. The logic for starting to measure impact now is as follows:

  • Impact is a relatively imprecise and general term, which inspires almost as many distinct answers as there are participants – so having a new and fairly comprehensive instrument that can add precision and shape stakeholder thinking while they are still involved is both innovative and appropriate in addressing the diversity of interests.
  • Impact happens at all stages of a project, i.e., during, immediately after and beyond completion, especially if there is a follow-up of projects – often until much later: so a time-flexible and adjustable instrument- linked to current and recent activities and also designed to accommodate later impacts is needed.
  • The approach adopted uses a single instrument – as a long form where commentary plus scoring is solicited and as a short form where numbers of respondents will be larger; the narrative and the quantitative aspects are complementary.
  • It has been designed to apply upon completion when memories are fresh and the knowledge still current; it can also be applied at any point in the future or re-applied as a comparative measure of time dependent impacts.
  • In this way it can be applied today as a current measure of impact and simultaneously as a measure of positioning for future prospective impacts – as assessed by those most involved.
  • This is why it is described as a preliminary baseline impact measurement tool that captures expectations as well as examples.
  • Impact analysis is not the same as an evaluation but may provide needed input especially if baseline data has been collected during or just after completion since most evaluations occur much later.

The Impact Measurement Instruments

The deployment was quite straight forward as follows:

TFCI described the development process and demonstrated the two forms of the impact measurement instrument to the CFIA-led Fore-Can team. The project leader first sent the long form to 54 potential participants – of whom four declined to participate and four responded with many comments plus scoring. The short form was then sent to all, and ten more responses were received – mostly just with scoring of the 50+ variables; based upon the short notice and lack of solicitation before emails were sent, it is positive that 14 responses in total were received out of 50 potential ones. With more advance preparation, this rate of 28% could easily be doubled. TFCI then managed a dual analysis – combining the quantitative and the qualitative responses.

The Measures

The actual measurement, distributed amongst several different lenses (or measure groupings), consists of a total of 54 measures. The first lens or level of impact interest is in terms of general role effectiveness: wherein foresight is seen as generally playing or performing as many as five roles to differing degrees;

249_bild1

The second set of impact measures, lens or grouping, consists of several general benefits, as perceived main-ly by those directly involved. As the impact data base and diversity of cases grows, differing patterns of pro-tagonist and stakeholder appreciation may emerge.

249_bild2

A third set of measures is obtained by using a success factor lens, which is especially relevant for foresight process designers and planners:
249_bild3

A fourth set of lenses consists of seeing foresight main-ly as a macro or meta process, focused on foresight as essentially a learning process and that each foresight project educates someone, and usually all participants. Here the evaluation team collected testimonials, anec-dotes, personal stories etc. In the category “training & skills development” the evaluators acknowledged that foresight is often motivated by sponsors wanting to strengthen readiness, resilience and preparedness skills.
249_bild6

These categories also give credit to the notion that fore-sight is a key tool for risk assessment and the man-agement of uncertainty.

249_bild7

And finally, foresight is closely aligned with design and planning. Accordingly, the participants of the evaluation had the opportunity to give account of the changes induced by the foresight exercise such if their organisa-tion achieved new strengths, there was any evidence of foresight in adopted priorities or of new directions with foresight-derived origins.

Alignment with Policy Cycle

Further, in the impact design, three groups of measures were developed – related to successive stages of the policy cycle: pre-policy; policy implementation and post policy. Here the participants had to give a score (# score represents average out of 5 including all scores other than no response).

249_bild4

249_bild5

 

The Response

Overall, these preliminary impact results indicate that the project had both a significant impact on participants from a present time vantage point and a well-positioned potential for future impact as expressed by the clear and consistent trend in the results toward impact endorsement in most of the variables examined. The conclusion to be drawn is not only that the project was quite successful in operational terms, but also that its full impact may only be known some years hence, given the strong prospects for future impact that were cited by most respondents.

The lists of the top and bottom five impact elements provide a snapshot both of domains where there is strength or weakness but also reflect a high degree of alignment amongst the respondents. Also of note is that 2/5 of the highest and lowest impacts are from the critical success factors elements (questions # 6-13), and this suggests that the CSF list is a key differentiator of impact – as was intended by Calof and Smith when they undertook their study in 2007.

249_bild8

Room for Improvements

The findings and the comments together present a consistent picture of a project that was both successful in achieving its intended near-term objectives and is well positioned for future impact and development opportunities. The ratings questions worked well to elicit stakeholder differentiation, which is normally regarded as indicative of a good engagement process, and many of the excellent comments reinforce this.

Because of the clear and generally enthusiastic responses, prospects for continued support from the participants for follow-up activities appear positive.

The combination of a long form and short form for impact assessment was viable, but both formats could be improved. The long format should be tailored to interviews, with some additional guidance provided. While it worked very well to elicit substantive commentary, it clearly was too daunting in terms of the time commitment required for most, particularly in that the impact analysis was an unanticipated additional time commitment for all stakeholders. Given the generally responsive attitudes, it is reasonable to assume that with more lead time, improved instruments, structured impact discussions built into the last meeting and a clear link to next stage development ideas, a response rate of over 60% can be anticipated – double what was received with almost no advance notice and no context preparation. The short format worked very well but likely missed a relatively easy opportunity to obtain short commentary on each of the eight sections of enquiry – thus enabling participants to elaborate the basis for their scores. The next version of the impact instruments will embody these improvements.

Overall, the post project preliminary impact baseline measurement has been very productive: baseline data and a set of premises for future development and evaluation/assessment have been established, and much of the impact experience has been captured in comments and scores that validate the benefits of the project – notably while still vivid and current.

Key Issues Raised Relevant to Policymaking

The main implication is that policy authorities can now have access to a reliable interim foresight impact measurement instrument aligned with stages of the policy cycle – and as experience accumulates with its application, governments can begin to benchmark their foresight project impacts against other projects, nations, fields etc.

Finally, the measures used for examining foresight impacts could be equally applied to most policy staging – so that at least the perception of potential impacts of policies could be measured during the development process rather than waiting for full implementation – when it is likely too late to adjust them.

Authors: Jack Smith, TFCI Canada Inc. and Telfer School of Management, University of Ottawa, Canada. (JESMITH@TELFER.UOTTAWA.CA)
Sponsors: Canadian Food Inspection Agency
Type: FORE-Can: national foresight project on animal health and food security – measurement phase
Organizer: Dr. Shane Renwick (CFIA SHANE.RENWICK@INSPECTION.GC.)
Duration: 2011
Budget: € 10,000
Time Horizon: 2011
Date of Brief: July 2012

Download EFP Brief 249_Measuring Impact of Foresights

Sources and References

Jonathan Calof, Jack E. Smith, (2012) “Foresight impacts from around the world: a special issue”, foresight, Vol. 14 Iss: 1, pp.5 – 14