Posts Tagged ‘impacts’

EFP Brief No. 249: Measuring Foresight Impact

Wednesday, January 30th, 2013

This brief describes a new instrument for measuring the impact of foresight. The foresight impact measurement instru-ment consists of 54 measures covering a wide range of foresight activities and potential policy and other impacts. This instrument, developed primarily by Ron Johnston and the author, is the result of several sessions with leaders of many of the most active national foresight programmes and includes a variety of types of measurement categories – notably those that align with the policy cycle in terms of positioning foresight for future impacts on policies as they emerge or are developed. It also has been pilot-tested on two Canadian foresight programs – in both cases achieving strong participation rates, high frequency of written comments and positive assessments of most of the measures and very strong endorsements of several key measures. One of the cases, a national foresight project on animal health and food security is described in this brief. Essentially the instrument provides a baseline for interim evaluation – while the experience is still vibrant – and in so doing it can (1) provide a unique mix of qualitative and quantitative feedback for stakeholders, participants and sponsors; (2) be immediately applied if required to making the case for continuity, future foresight funding or new projects; (3) form a credible baseline against which more formal evaluation can be structured later; and (4) help create a key international benchmark data base entry and case example of public sector foresight impact measurements – and thus position the EFP well for the future.

The Impact-Value Challenge

A key recurring challenge for foresight initiatives – projects, programmes and pilots – has been how to actually demonstrate the value of foresight investments for government sponsors and stakeholders – who are mindful of accountability, are asked to justify the value of foresight investments for government mandates and are requested to provide cost-efficiency and cost-effectiveness analysis so that foresight can be compared with other prospective applications of limited government funds.

The methodology elaborated below is a response to this challenge, prepared by Professor Jack E. Smith with input from senior international foresight leaders from the US, Europe (UK, FR, NL, FN) Australia and Asia ( TH, CH, KR, SP). The methodology draws upon discussion papers presented by the author and Professors Jon Calof and Ron Johnston at five international meetings. The challenge was to assess how to effectively measure impacts of foresight for government sponsors, operating in the short to medium term of 1-3 years when ideally these foresight impacts occur over a (mid to long term) five- to fifteen-year time horizon.

Case Study on Animal Health and Food Security in Canada

In September 2011, the Fore-Can Project on Animal Health and Food Security completed a three year foresight-based assessment of major challenges and opportunities associated with the future management of animal health and food security systems in Canada. The project was well received, involved a wide range of stakeholders and effectively engaged key policy advisors and industry leaders. As with many foresight projects, questions of immediate and enduring impact were raised as the end of the project drew closer. Fortunately this timing has coincided with the development of a new series of long and short format impact measurement instruments as part of an international forum of foresight best practices (more below on this).

Accordingly, the Fore-Can management team decided to be the first project to apply the new instruments. The logic for starting to measure impact now is as follows:

  • Impact is a relatively imprecise and general term, which inspires almost as many distinct answers as there are participants – so having a new and fairly comprehensive instrument that can add precision and shape stakeholder thinking while they are still involved is both innovative and appropriate in addressing the diversity of interests.
  • Impact happens at all stages of a project, i.e., during, immediately after and beyond completion, especially if there is a follow-up of projects – often until much later: so a time-flexible and adjustable instrument- linked to current and recent activities and also designed to accommodate later impacts is needed.
  • The approach adopted uses a single instrument – as a long form where commentary plus scoring is solicited and as a short form where numbers of respondents will be larger; the narrative and the quantitative aspects are complementary.
  • It has been designed to apply upon completion when memories are fresh and the knowledge still current; it can also be applied at any point in the future or re-applied as a comparative measure of time dependent impacts.
  • In this way it can be applied today as a current measure of impact and simultaneously as a measure of positioning for future prospective impacts – as assessed by those most involved.
  • This is why it is described as a preliminary baseline impact measurement tool that captures expectations as well as examples.
  • Impact analysis is not the same as an evaluation but may provide needed input especially if baseline data has been collected during or just after completion since most evaluations occur much later.

The Impact Measurement Instruments

The deployment was quite straight forward as follows:

TFCI described the development process and demonstrated the two forms of the impact measurement instrument to the CFIA-led Fore-Can team. The project leader first sent the long form to 54 potential participants – of whom four declined to participate and four responded with many comments plus scoring. The short form was then sent to all, and ten more responses were received – mostly just with scoring of the 50+ variables; based upon the short notice and lack of solicitation before emails were sent, it is positive that 14 responses in total were received out of 50 potential ones. With more advance preparation, this rate of 28% could easily be doubled. TFCI then managed a dual analysis – combining the quantitative and the qualitative responses.

The Measures

The actual measurement, distributed amongst several different lenses (or measure groupings), consists of a total of 54 measures. The first lens or level of impact interest is in terms of general role effectiveness: wherein foresight is seen as generally playing or performing as many as five roles to differing degrees;

249_bild1

The second set of impact measures, lens or grouping, consists of several general benefits, as perceived main-ly by those directly involved. As the impact data base and diversity of cases grows, differing patterns of pro-tagonist and stakeholder appreciation may emerge.

249_bild2

A third set of measures is obtained by using a success factor lens, which is especially relevant for foresight process designers and planners:
249_bild3

A fourth set of lenses consists of seeing foresight main-ly as a macro or meta process, focused on foresight as essentially a learning process and that each foresight project educates someone, and usually all participants. Here the evaluation team collected testimonials, anec-dotes, personal stories etc. In the category “training & skills development” the evaluators acknowledged that foresight is often motivated by sponsors wanting to strengthen readiness, resilience and preparedness skills.
249_bild6

These categories also give credit to the notion that fore-sight is a key tool for risk assessment and the man-agement of uncertainty.

249_bild7

And finally, foresight is closely aligned with design and planning. Accordingly, the participants of the evaluation had the opportunity to give account of the changes induced by the foresight exercise such if their organisa-tion achieved new strengths, there was any evidence of foresight in adopted priorities or of new directions with foresight-derived origins.

Alignment with Policy Cycle

Further, in the impact design, three groups of measures were developed – related to successive stages of the policy cycle: pre-policy; policy implementation and post policy. Here the participants had to give a score (# score represents average out of 5 including all scores other than no response).

249_bild4

249_bild5

 

The Response

Overall, these preliminary impact results indicate that the project had both a significant impact on participants from a present time vantage point and a well-positioned potential for future impact as expressed by the clear and consistent trend in the results toward impact endorsement in most of the variables examined. The conclusion to be drawn is not only that the project was quite successful in operational terms, but also that its full impact may only be known some years hence, given the strong prospects for future impact that were cited by most respondents.

The lists of the top and bottom five impact elements provide a snapshot both of domains where there is strength or weakness but also reflect a high degree of alignment amongst the respondents. Also of note is that 2/5 of the highest and lowest impacts are from the critical success factors elements (questions # 6-13), and this suggests that the CSF list is a key differentiator of impact – as was intended by Calof and Smith when they undertook their study in 2007.

249_bild8

Room for Improvements

The findings and the comments together present a consistent picture of a project that was both successful in achieving its intended near-term objectives and is well positioned for future impact and development opportunities. The ratings questions worked well to elicit stakeholder differentiation, which is normally regarded as indicative of a good engagement process, and many of the excellent comments reinforce this.

Because of the clear and generally enthusiastic responses, prospects for continued support from the participants for follow-up activities appear positive.

The combination of a long form and short form for impact assessment was viable, but both formats could be improved. The long format should be tailored to interviews, with some additional guidance provided. While it worked very well to elicit substantive commentary, it clearly was too daunting in terms of the time commitment required for most, particularly in that the impact analysis was an unanticipated additional time commitment for all stakeholders. Given the generally responsive attitudes, it is reasonable to assume that with more lead time, improved instruments, structured impact discussions built into the last meeting and a clear link to next stage development ideas, a response rate of over 60% can be anticipated – double what was received with almost no advance notice and no context preparation. The short format worked very well but likely missed a relatively easy opportunity to obtain short commentary on each of the eight sections of enquiry – thus enabling participants to elaborate the basis for their scores. The next version of the impact instruments will embody these improvements.

Overall, the post project preliminary impact baseline measurement has been very productive: baseline data and a set of premises for future development and evaluation/assessment have been established, and much of the impact experience has been captured in comments and scores that validate the benefits of the project – notably while still vivid and current.

Key Issues Raised Relevant to Policymaking

The main implication is that policy authorities can now have access to a reliable interim foresight impact measurement instrument aligned with stages of the policy cycle – and as experience accumulates with its application, governments can begin to benchmark their foresight project impacts against other projects, nations, fields etc.

Finally, the measures used for examining foresight impacts could be equally applied to most policy staging – so that at least the perception of potential impacts of policies could be measured during the development process rather than waiting for full implementation – when it is likely too late to adjust them.

Authors: Jack Smith, TFCI Canada Inc. and Telfer School of Management, University of Ottawa, Canada. (JESMITH@TELFER.UOTTAWA.CA)
Sponsors: Canadian Food Inspection Agency
Type: FORE-Can: national foresight project on animal health and food security – measurement phase
Organizer: Dr. Shane Renwick (CFIA SHANE.RENWICK@INSPECTION.GC.)
Duration: 2011
Budget: € 10,000
Time Horizon: 2011
Date of Brief: July 2012

Download EFP Brief 249_Measuring Impact of Foresights

Sources and References

Jonathan Calof, Jack E. Smith, (2012) “Foresight impacts from around the world: a special issue”, foresight, Vol. 14 Iss: 1, pp.5 – 14

EFP Brief No. 218: Embedding Foresight in the Colombian Innovation System

Thursday, June 21st, 2012

This follow-up brief recapitulates the evaluation of the Colombian Technology Foresight Programme (CTFP). The foresight brief no. 119 (“Evaluating Foresight – The Colombian Case”) summarised the methodological aspects and principal focus of the framework on which the evaluation of the second cycle (2005 – 2008) of the CTFP was based. The objective of the present follow-up brief is to look back and summarise the implications of the evaluation of the CTFP by drawing on the key findings of the evaluation summary report. Concretely, it focuses on (1) the appropriateness and adaptation of the evaluation framework, as well as the effects for the spread of a foresight culture in Colombia that have been induced or stimulated by the evaluation of the CTFP and (2) the institutional mechanisms in support of the social appropriation of the CTFP’s output and results as well as the dissemination of the foresight knowledge generated by the CTFP to policy, industry and society as a whole.

Evaluation to Improve the Capacity for Learning

The development of Colombia’s Technology Foresight Programme (CTFP) has long been a reference point in the Latin American region. The CTFP is the first national foresight programme in Latin America that has been evaluated so far. The principal idea of the CTFP has been building a platform to create, distribute and utilise foresight knowledge in Colombia. It was intended to introduce new foci and new types of foresight practices and interventions in support of the strategic re-orientation of programmes and (sub-)sectors.

The focal point of the evaluation carried out in 2008 under the leadership of the University of Manchester was to reshape the objectives and activities of the second cycle of the CTFP (2005 – 2008). The conceptual framework of the evaluation was geared towards analysing foresight as a process.

This follow-up brief describes the methodological framework of the evaluation and discusses the learning process involved as well as the question whether the evaluation improved the aptitude for learning.

Evaluation of the CTFP

In Colombia, the evolution of long-term thinking in foresight has been largely driven by the role of COLCIENCIAS (Colombian Office of Science and Technology) as a node institution capable of facilitating inter-institutional alliances between various centres of excellence, on the one hand, and mobilising resources and engaging key stakeholders into a dynamic and self-reinforcing foresight learning process, on the other. Part and parcel of this learning process has been the comprehensive evaluation of the second cycle of the CTFP, which was geared towards identifying and supporting strategic sectors during the period between 2005 and 2008. Commissioned by COLCIENCIAS, the overall aim of the evaluation was to increase the CTFP’s capacity to shape and inform policy processes and actors.

Methodological Approach and Phases of the Evaluation Process

The evaluation of the CTFP was based on a methodological framework composed of a mix of seven diverse methods and activities that have been listed and described in some detail in the original foresight brief: (1) documentary analysis, (2) logic chart and indicators, (3) surveys, (4) interviews, (5) case studies, (6) benchmarking and (7) evaluation forum.

The evaluation process was divided in four phases:

Phase 1: Scoping – This phase had the principal objective to understand the main rationale of the evaluation process in order to design a coherent research process. The evaluation report states, “In addition to the traditional objectives of a Foresight programme evaluation (i.e. assessment of the impacts of the programme and the projects; assessment of the cost-effectiveness of the programme, and evaluation of the way in which Foresight is run in Colombia), COLCIENCIAS and SECAB [Secretaria Ejecutiva del Convenio Andres Bello] were particularly interested in identifying lessons and recommendations for the improvement of Foresight and horizon-scanning activities in the country.”

Phase 2: Understanding – This phase of the evaluation process was based on the collection of tacit and codified knowledge about the CTFP. “Tacit knowledge was collected through individual and group interviews with key stakeholders in COLCIENCIAS, other sponsors (e.g. ministries) and main stakeholders […]. Codified knowledge involved the compilation of major codified products (e.g. interim and final reports, books, journal publications and other important documents, such as individual project budgets and description of the programme’s expenses).”

Phase 3: Evaluating – This phase was based on a benchmarking of the CTFP against practices in other countries. The main objective here was to learn from other international best practices in establishing a national foresight initiative. The lessons shared from other countries included the UK, Malta, Russia, Spain and Hungary.

Phase 4: Learning – This phase involved conducting further analysis and preparing the final evaluation report.

Lessons Learned

The focus of the evaluation was on assessing what effects the second cycle had on policy and programme development. In particular, the impact of the CTFP on the design and of research policies was to be assessed as well as the effects of the CTFP on the promotion of national skills and the establishment of a national foresight culture in Colombia. In this sense, the evaluation was an important step towards synthesising the lessons derived from the national foresight exercise in terms of impact on skills, enhancement of capacities for strategic decision-making processes and policy design.

Key Findings of the CTFP Evaluation

The final report summarised the key findings of the report as follows:

(1) Regarding the overall objectives, the evaluation report states that the “CTFP objectives have been appropriate and successfully achieved. The programme has contributed to the creation of development visions and strategies for moving towards a knowledge-based society” through horizon-scanning and the building of foresight capacities in key sectors.

(2) As regards the value for money, the evaluation concluded that the CTFP achieved “a paradigm shift” by greatly contributing to the creation of a shared vision for “the productive transformation of Colombia into a knowledge-economy”. Furthermore, the evaluation states that the CTFP has begun to pay off since a diverse set of stakeholders have already adopted the vision brought forward by the CTFP in formulating their long-term objectives.

(3) Regarding the organisational structure of the CTFP, the evaluation observed that the institutional anchoring of the technical and decision-making groups in COLCIENCIAS during the second cycle notably “increased the CTFP’s capacity to shape and inform policy processes and actors. However, these changes also made the programme appear to be more of a COLCIENCIAS instrument than a national programme.”

(4) With respect to the approaches and mix of methods, the evaluation highlighted that “one original and effective feature of CTFP has been the combination of thee conceptual and methodological approaches: Foresight, horizon scanning and productive chain.”

(5) Regarding implementation and aftercare of the second cycle, the evaluation stressed the need for an aftercare strategy in the Colombian foresight programme. If the key support institutions of the programme “were to consider implementing such a strategy, this would probably increase the ability of Foresight to inform policy and shape research priorities. At the same time, it would also allow sufficient time for new networks to exploit the momentum created and consolidate institutional alliances.”

(6) With regard to the CTFP’s contribution to the spread of a foresight culture in Colombia, the evaluation states: “Some stakeholders still see foresight as being exclusively expert-oriented.” Therefore, it was recommended “that the general public be encouraged to participate in projects and training courses. This would probably require alliances with the private and productive sectors, in order to increase the financial and implementation feasibility of large-scale courses and projects.”

(7) Concerning the presence and visibility of the CTFP, the evaluation states that “while CTFP stands up well alongside programmes conducted elsewhere, it has limited visibility in the international academic and professional literature.” A clear implication derived from this is “that all major reports should be […] made available on the internet, and more widely disseminated through, for example, conference presentations and articles in relevant publications.”

(8) With regard to the impacts related to science, technology and innovation (STI), the evaluation stated that out of a total of 24 projects and more than 30 capacity-building courses, “nine projects had positive impacts on public and private polices and strategies; six projects had positive impacts on the agendas of STI programmes and institutions; five projects had positive impacts on the consolidation of research groups; two projects had positive impacts on the consolidation of S&T capacities; and two projects had positive impacts on international projects.”

(9) Concerning policy recommendations and strategies, these have been highlighted as “fundamental elements of CTFP outputs”. According to the evaluation, “the most significant influence of the CTFP on public policy has been the work on the STI Vision 2019.” Moreover, the CTFP’s biotechnology project had a significant influence on the policies and research priorities of COLCIENCIAS’ National Biotechnology Programme.

Networking Key to Spreading Foresight Culture

The strengthening of networks was a central pillar for achieving the principal purpose of the CTFP. In this sense, the enlargement of inter-institutional networks was instrumental for the realisation of a strategy to spread a foresight culture in Colombia. On the side of the sponsors and organisers of the CTFP, building networks to facilitate the involvement of different expertises in support of an interdisciplinary approach to foresight in many fields and sectors was a vital aspect for all those who participated in the CTFP.

Dissemination Strategy Falls Short of Potential

The CTFP produced a large number of high-quality scientific outputs. However, the strategy to disseminate the preliminary and final results has pursued an approach not conducive to enhancing the international visibility of the CTFP as a best practice across the region. Moreover, in disseminating the new knowledge generated, there was a tendency to emphasise tangible over non-tangible outputs. Since the degree to which the newly produced knowledge contributes to the opening of new strategic options or the strategic re-orientation of sectors is an important success criteria for foresight, it is vital to embed this new knowledge in people’s and organisations’ practices. This requires that the dissemination of the results be tailored to different target groups, which also calls for different means of communication appropriate to the respective group. Although the CTFP delivered many tangible outputs, such as scientific publications (predominantly in Spanish), the approach chosen to disseminate the outputs and results remained far behind the possibilities of alternative approaches that could have enhanced the interaction between the different governmental and scientific communities or increased the international visibility of the CTFP as a best practice from which others could learn.

Methodological Progress Towards Context-sensitive Use of Methodology

The choice of the methods is the most distinctive feature of a foresight process. The evaluation of the CTFP stated that “an average CTFP study involved more than 10 methods, with more or less half of these being horizon-scanning techniques (including bibliometrics, trend extrapolation and patent analysis) and the other half related to Foresight and productive chain approaches (e.g. scenarios, brainstorming, stakeholders mapping, key technologies, morphological analysis, among others).” Regarding the use of foresight, the evaluation focused on the selection of methods but did not monitor how well the methods used lined up with the STI implementation environment, specifically in regard to strengthening capacities in support of policymaking in the area of science, technology and innovation with due regard to local concerns in Colombia. Therefore, a future monitoring or evaluation framework might also focus on adjusting or reconfiguring the methods applied to fit the foresight process needs in Colombia.

Towards Enhancing Colombia’s Foresight Capabilities

Given that there is no ‘one-size-fits-all’ approach to evaluate foresight and that the evaluation of foresight activities cannot be carried out independently of the national context (Georghiou and Keenan, 2006), it is difficult to assess the interaction between foresight and evaluation with respect to the impact on policy instruments or the improvement of overall system coordination through strengthening linkages between innovation actors.

The concepts of effectiveness, efficiency, appropriateness and behavioural additionality on which the framework to evaluate the CTFP rests are key to understanding the focus of foresight activities that are carried out to reconfigure institutional set-ups and re-orient policy goals. Georghiou and Keenan (2006) state that the “[…] evaluation of foresight must include understanding of the interaction of foresight outputs with the strategic behaviour of policy and economic actors.” In this sense, the evaluation of the second cycle of the CTFP was an important step towards better understanding the drivers of the strategic behaviour on part of the key implementing institutions in the Colombian system. However, further advancing Colombia’s foresight capacities depends to a high degree on the institutionalisation of foresight in the Colombian context.

Improve Dissemination Through Alternative Communication Channels

According to the results of the evaluation of the second CTFP cycle, a key lesson that can be drawn refers to the appropriation of the insights gained from the foresight programme. Although the fact that the foresight programme was conducted by COLCIENCIAS was an important institutional driver, the lack of an aftercare strategy constitutes a weak point for the strategic influence of the foresight knowledge generated on the target groups and sectors. Therefore, an important recommendation to increase the future impact of the CTFP is that alternative dissemination and communication channels should be exploited to a greater extent. An improved dissemination strategy should also take into account the need for a stronger diversification of foresight capacity building in Colombia.

Towards Further Institutionalisation of Foresight in Colombia

The “shift from networks and individual exercises [….] to more institutionalisation towards centres of excellence“ is an important step to “take on responsibility for preserving knowledge and for allowing lessons learned to be carried forward in a long-term framework“ (Popper et al., 2010). In this sense, the evaluation revealed that a move away from the somewhat centralistic approach to anchor the foresight process in COLCIENCIAS towards a more effective institutional mechanism was a necessary step to better embed foresight in the Colombian STI system. COLCIENCIAS recent decision to institutionalise the foresight practices in the framework of the CTFP by establishing the Colombian Foresight Institute (COFI) at the Universidad del Valle (Cali) can be considered an important move to enhance the aptitude for learning and thus strengthen the contribution of foresight to reorienting the Colombian STI system. In this arrangement, multiple organisations will be able to conduct foresight.

Under the bottom line, we can conclude that the evaluation came at the appropriate time to develop recommendations on how the foresight outputs, results and knowledge generated during the second cycle of the CTFP could be better appropriated by the stakeholders and embedded in a broader strategic policy context. In particular, an improved dissemination strategy and the search for alternative ways of institutionalising foresight are central pillars for engaging future resources and a broad set of stakeholders in a dynamic and self-reinforcing learning process based on which a foresight culture can develop in line with the evolving STI policy system in the Colombian context.

Authors: Dirk Johann                                            dirk.johann.fl@ait.ac.at                                
Sponsors: COLCIENCIAS (Colombian Office of Science and Technology)  
Type: Evaluation of Foresight Programme  
Geographic coverage: Colombia
Organizer: COLCIENCIAS (Colombian Office of Science and Technology)

PREST / Manchester Institute of Innovation Research (MIoIR), MBS, University of Manchester

Duration: 6–9 months Budget: € 40k Time Horizon: 2020 Date of Brief: May 2012

Download EFP Brief No 218_ Embedding Foresight in the Colombian Innovation System

Sources and References

Georghiou, L. and M. Keenan (2006), “Evaluation of National Foresight Activities: Assessing Rationale, Process and Impact”, Technological Forecasting and Social Change, 73, 761-777

Popper, R., L. Georghiou, M. Keenan, I. Miles et al. (2010), Evaluating Foresight – Fully-fledged Evaluation of the Colombian Technology Foresight Programme (CTFP), Colombia: Universidad del Valle

EFP Brief No. 195: Influence of Foresight on Public Policy in Flanders

Friday, September 30th, 2011

This brief presents the findings of a research project aimed at understanding the influence of policy-oriented foresight on public policy in Flanders. A foresight identifying six strategic clusters for technology and innovation policy in Flanders is analysed. The results of this analysis show that the foresight-oriented technology assessment (FTA) did have a significant impact on the policy process, but the greater effect might prove to be in its role as a reference point for future FTAs, which will then give shape to long-term technology and innovation policy in Flanders.

Assessing the Impact of Foresight

Policy-oriented foresight or foresight in a public policy context is aimed at supporting the decision-making process. By anticipating as much as possible different, alternative developments, it seeks to contribute to effective long-term decision-making. The policy process itself, however, is non-linear and often depends on temporary “policy windows” (Kingdon, 1995), which leave the specific role of exercises aimed at strengthening the evidence base of public action a little unclear. This is also the case for policy-oriented foresight.

The literature on foresight contains many studies that identify factors for success. However, little empirical evidence is available that policy actors actually use the outputs of foresight exercises aimed at supporting policy decisions.

Based on the factors for success found in the foresight literature, our research project analysed three case studies. The first case examined was a foresight identifying six strategic clusters for technology and innovation policy in Flanders. This brief assesses whether and how the foresight influenced the Flemish government’s policy decisions. The analysis of influence was informed by the evaluation literature since studies on foresight rarely address this question.

The first section of this brief provides insight into different aspects of the foresight research in this particular case. The assessment of the influence of the foresight in the second section is followed by a first set of recommendations on how to improve the relationship between policy-oriented foresight and public policy.

Foresight in Flanders

The Flemish Council for Science and Innovation (VRWI) conducted a foresight on innovation and technology in Flanders from 2005 to 2006. The VRWI is a strategic advisory council in the policy domain of science and innovation. It advises the Flemish government on its science and technology policies either in a proactive manner or at the government’s request. The council is a multi-actor arena where different stakeholders in the field of science and innovation meet. These stakeholders mostly have an industrial background, but they also include scientists from universities or representatives from other knowledge institutions in Flanders as well as government representatives from the administrative or political level. The VRWI operates as a ‘boundary organization’ (Guston, 2001) between science, politics and society; this will prove important for the influence of the foresight exercise that the organisation conducted on public policy in Flanders.

Approach towards Technology & Innovation

The foresight was set up in 2005 at the VRWI’s own initiative. It aimed at “providing a long-term reference point for technology and innovation in Flanders” (Smits et al. 2006:10). The starting point for the study can be traced back as far as 1997 when the VRWI and other key stakeholders in the technology and innovation field felt the need to gain more insight into the scientific, economic and societal developments that might possibly influence the welfare and well-being of the citizenry in Flanders. Together with a group of university researchers, a process was organised to develop an appropriate methodology that would serve to assess this problem. Methodological inspiration was found in foresights from Japan and Germany as well as those conducted at the European level.

Initially, the foresight had been designed as a rather broad exercise, addressing different societal, economic and scientific problems. However, this broad focus was not retained. The VRWI had initiated the foresight proactively, but upon further consultation with stakeholders, among them political actors, the scope of the study was narrowed down to focus on technology and innovation. Central to the analysis were those developments in the science, technology and innovation field necessary for Flanders to remain competitive both within the European area and the globalised world.

Once the focus on technology and innovation had been established, the different steps to conduct the foresight were put in practice. The VRWI took the lead while cooperating closely with the scientific research group that had developed the methodological approach.

Methodology

The foresight was conducted in three consecutive steps, leading to the formulation of specific policy priorities for six sectors (or clusters) of technology and innovation in Flanders. The third step was the actual foresight. There, the future was assessed with a medium-term horizon of about ten years, i.e. 2015. As will be elaborated below, the study was highly participative and intensive.

In a first phase, a SWOT analysis (strengths, weaknesses, opportunities and threats) was conducted on the different sectors in the technology and innovation field in Flanders.

The second phase of the study consisted of linking the results of the SWOT analysis with those of a broader European foresight. Based on this synthesis, a high-level group of experts (the ‘captains of industry’ in Flanders) identified six strategic clusters for technology and innovation in Flanders. These cover a broad range of technological and innovative domains and are clustered on a thematic base. The six strategic clusters that were identified by the high-level group in this second phase are

  • Transport, logistics and services supply chain management
  • ICT and services in health care
  • Health care prevention and treatment; food and agriculture
  • New materials, especially nanotechnology, and the manufacturing industry
  • ICT for socio-economic innovation
  • Energy and environment for the services and manufacturing industry

The first two phases took about a year to be completed. The selective expert consultation at the end of phase two set the stage for a broader consultation of experts via the Delphi method in the third phase. This final phase was framed in terms of the six strategic clusters identified. The actors consulted were all R&D experts in the field of technology and innovation in Flanders. In total, 130 R&D experts participated in the third phase of the study, which took six months to be completed. The experts were divided along the six strategic clusters identified and, using the Delphi method, were asked to assess 160 possible future developments as well as the current and future capacities in the field in Flanders.

Via two or more rounds of discussion, a consensus was reached between the experts in each of the strategic clusters. They identified 30 specific priorities in technology and innovation. Additionally, 85 of the 130 experts agreed to evaluate factors critical for the achievement of these priorities. The VRWI Council then validated these results and formulated specific recommendations for the different stakeholders: universities, industry and government. These recommendations were of particular importance to the latter. A more detailed account on the role of government within a broader technology and innovation context and the use of the results of the study is provided in the next section.

Diffusion of Results among a Wide Range of Actors

The foresight was captured in two reports. Firstly, a summary report introduces the 30 priorities, the factors for success and the set of recommendations. Secondly, a more technical report elaborates upon the foresight process itself. It provides a detailed account of the three phases that constitute the foresight.

Additionally, the Council and its president took specific action to promote the results of the study among a broad range of actors. They did this by presentations and road shows to diffuse the results. An important step in this respect was to engage the support of universities and industry not only before and during the foresight process but also after the foresight was completed. This assured diffusion of the results among a first and important set of stakeholders. A third important stakeholder, government, was much less intensely involved in the foresight. Diffusion of the results of the study among political and administrative actors is, however, an important factor for the foresight to have an impact on technology and innovation in Flanders. The next section assesses to what extent this was accomplished.

Influence on Public Policy in Flanders

Did the foresight influence the Flemish government’s strategic decision-making on technology and innovation? To answer this question, we must first consider the concept of influence itself. Moreover, we must describe the broader technology and innovation policy context in Flanders before we can adequately address the question of influence on policy there. These are the issues we will now turn to.

A Framework for Assessing Policy Influence

From the perspective of policy actors, influence of evidence on policy can be viewed as knowledge utilisation. There are three dimensions for the analysis of influence on policy: source, intention and moment. This brief focuses in particular on the first dimension of influence: the source of influence.

The source of influence of foresight on policy can be product-related or process-related. Product-related influence is the influence of the output of foresight, i.e. the results of the study presented in a report. The results of foresight can influence policy in different ways. We differentiate between four types of product-related influence:

  • Direct instrumental influence is reflected in official policy documents.
  • Conceptual influence is seen as “enlightenment” (Weiss, 1980) of policy makers.
  • Agenda-setting influence means that new topics can be discerned, which were previously not under the attention of policy makers.
  • Political-strategic influence takes place when political actors legitimise or oppose government actions based on the study’s results.

The foresight can also have a significant process-related influence on policy and policy actors. The involvement of political or administrative actors at an early stage of the process might promote a more future-oriented view on policy or a better understanding of the possible added value of foresight for policy. Indirectly, this process-related influence might enhance the product-related use of foresight since it makes policy actors more receptive to its outcomes.

Importance of the Specific Policy Context

From a historical perspective, technology and innovation policy in Flanders can be characterised as predominantly technology-pushed or supply-driven. Policy ‘demand’ was and is to a large extent adapted to industrial and scientific ‘supply’. In other words, technology and innovation policy in Flanders is usually developed bottom up. Public funding is mostly responsive to the R&D policy of industrial actors and of universities. It is to a much lesser extent coordinated (let alone steered) by government within a broader strategic framework. The foresight tried to provide such a strategic framework by identifying future technology and innovation priorities. Via its recommendations, it also assigned a specific role to the Flemish government for the framework to be implemented. In principle, the Flemish government could assume different roles, ranging from a hands-off and encouraging position (bottom-up policy approach) to a hands-on and actively steering role (top down). The latter is in order when the task at hand requires making strategic policy choices and taking the necessary actions to enable them.

Assuming that the foresight served as necessary input for establishing a strategic framework on technology and innovation in Flanders, the involvement of a broad range of stakeholders was not only a logical consequence in a supply-driven policy domain. More importantly, it secured the necessary stakeholder support for implementing the priorities–which is especially critical when the strategic priorities are defined by a hands-on, steering government.

Policy Influence Analysed in Documents

The influence of the foresight on public policy was analysed through a thorough document analysis and interviews. The document analysis included strategic policy documents, policy briefs, white papers, parliamentary documents etc. The interviews were carried out with political and administrative stakeholders as well as members of the VRWI responsible for the study.

Challenge-driven Innovation Approach Inspired by Foresight Exercise

At first, the report had no influence on public policy. The foresight report was published in 2007, at the end of the then Flemish government’s legislative period when policy directions had already been decided upon. The policies in place were further enacted toward the end of the legislature. Later on, however, the results of the foresight eventually significantly influenced public policy in Flanders in several ways.

First, there is reference to its results in official strategic policy documents, such as the broader strategic framework project “Flanders in Action” set up by the Flemish government to make Flanders a frontrunner region in the social as well as in the economic domain. This emphasis on the strategic level was translated into the Flemish government’s policy note 2009-2014 on innovation policy and the related policy briefs and actions. More recently, in May 2011, the Flemish government approved a conceptual brief giving shape to a more hands-on strategic approach in technology and innovation in Flanders. The focus on a ‘challenge-driven innovation’ approach is particularly inspired by the strategic orientation provided in the foresight. Moreover, the establishment of innovation nodes can be traced back, among others, to the strategic clusters defined by the foresight in 2007. It therefore seems fair to say that the foresight has had an important direct instrumental influence on public policy in Flanders.

The study is also well known among a broad range of actors in the policy domain. Especially the first two phases of the study (SWOT and relative positioning of Flanders in Europe) have served as a knowledge base for political and administrative actors in government, marking an important conceptual influence of the foresight.

When we consider the study’s influence on the political agenda, political attention seems to have been mostly directed toward the perceived need to make strategic choices for the domain. This need was addressed in the foresight and played a key role in conducting the exercise. Contrary to a predominantly hands-off approach in the past, the government now has come to consider, accept and implement a more demand-driven approach when deciding on innovation policy at the strategic level.

Thus, we can discern an important, directly instrumental and conceptual influence on policy in terms of agenda-setting. This is a medium-term influence, i.e. the effects are observed three to five years after the study was published. By contrast, there does not seem to have been any politically strategic influence.

Process-related Trade-off for Technology & Innovation Actors

Additionally, the foresight has also had an important process-related influence. It is considered an important first exercise of its kind in the policy domain since it was aimed, quite explicitly, at bringing about a strategic, long-term vision and making policy choices in technology and innovation in Flanders. It has introduced a certain dynamic among the actors in the policy domain itself. Several actors indicate, for example, that a follow-up foresight is necessary to develop an adequate long-term strategic policy in the domain of technology and innovation since this foresight dates from 2006 with a horizon of 2015.

Authors: Ellen Fobé                                      ellen.fobe@soc.kuleuven.be

Marleen Brans                                marleen.brans@soc.kuleuven.be

Sponsors: Flemish government: Policy Research Centre – Governmental Organization in Flanders 2007-2011
Type: Assessment of influence of foresight on public policy
Organizer: Research project: Evidence-based policy-making: matching supply and demand of quantitative policy information and foresight. Project supervisor: Prof. Dr. Marleen Brans; researcher: Ellen Fobé
Duration: 2010-2011 Budget: N/A Time Horizon: 2007-2011 Date of Brief: June 2011  

 

Download EFP Brief No. 195_Influence_of_Foresight

Sources and References

Project link and research themes of the Policy Research Centre – Governmental Organization in Flanders

  • http://soc.kuleuven.be/sbov/eng/research/epr14.htm
  • http://soc.kuleuven.be/sbov/eng/index.htm

References

  • Guston, D. (2000): Between politics and science: assuring the integrity and productivity of research. Cambridge University Press, Cambridge.
  • Kingdon, J.W. (1995): Agendas, Alternatives and Public Policies. Second edition, Boston: Little, Brown and Company.
  • Smits, E., Ratinckx, E. & Thoen, V. (2006): Technology and innovation in Flanders: priorities. Brussels: Flemish Council for Science and Innovation.
  • Weiss, C. (1980): Knowledge creep and decision accretion. Knowledge: Creation, Diffusion, Utilization 1(3), pp.381-404.