Posts Tagged ‘evaluation’

EFP Brief No. 249: Measuring Foresight Impact

Wednesday, January 30th, 2013

This brief describes a new instrument for measuring the impact of foresight. The foresight impact measurement instru-ment consists of 54 measures covering a wide range of foresight activities and potential policy and other impacts. This instrument, developed primarily by Ron Johnston and the author, is the result of several sessions with leaders of many of the most active national foresight programmes and includes a variety of types of measurement categories – notably those that align with the policy cycle in terms of positioning foresight for future impacts on policies as they emerge or are developed. It also has been pilot-tested on two Canadian foresight programs – in both cases achieving strong participation rates, high frequency of written comments and positive assessments of most of the measures and very strong endorsements of several key measures. One of the cases, a national foresight project on animal health and food security is described in this brief. Essentially the instrument provides a baseline for interim evaluation – while the experience is still vibrant – and in so doing it can (1) provide a unique mix of qualitative and quantitative feedback for stakeholders, participants and sponsors; (2) be immediately applied if required to making the case for continuity, future foresight funding or new projects; (3) form a credible baseline against which more formal evaluation can be structured later; and (4) help create a key international benchmark data base entry and case example of public sector foresight impact measurements – and thus position the EFP well for the future.

The Impact-Value Challenge

A key recurring challenge for foresight initiatives – projects, programmes and pilots – has been how to actually demonstrate the value of foresight investments for government sponsors and stakeholders – who are mindful of accountability, are asked to justify the value of foresight investments for government mandates and are requested to provide cost-efficiency and cost-effectiveness analysis so that foresight can be compared with other prospective applications of limited government funds.

The methodology elaborated below is a response to this challenge, prepared by Professor Jack E. Smith with input from senior international foresight leaders from the US, Europe (UK, FR, NL, FN) Australia and Asia ( TH, CH, KR, SP). The methodology draws upon discussion papers presented by the author and Professors Jon Calof and Ron Johnston at five international meetings. The challenge was to assess how to effectively measure impacts of foresight for government sponsors, operating in the short to medium term of 1-3 years when ideally these foresight impacts occur over a (mid to long term) five- to fifteen-year time horizon.

Case Study on Animal Health and Food Security in Canada

In September 2011, the Fore-Can Project on Animal Health and Food Security completed a three year foresight-based assessment of major challenges and opportunities associated with the future management of animal health and food security systems in Canada. The project was well received, involved a wide range of stakeholders and effectively engaged key policy advisors and industry leaders. As with many foresight projects, questions of immediate and enduring impact were raised as the end of the project drew closer. Fortunately this timing has coincided with the development of a new series of long and short format impact measurement instruments as part of an international forum of foresight best practices (more below on this).

Accordingly, the Fore-Can management team decided to be the first project to apply the new instruments. The logic for starting to measure impact now is as follows:

  • Impact is a relatively imprecise and general term, which inspires almost as many distinct answers as there are participants – so having a new and fairly comprehensive instrument that can add precision and shape stakeholder thinking while they are still involved is both innovative and appropriate in addressing the diversity of interests.
  • Impact happens at all stages of a project, i.e., during, immediately after and beyond completion, especially if there is a follow-up of projects – often until much later: so a time-flexible and adjustable instrument- linked to current and recent activities and also designed to accommodate later impacts is needed.
  • The approach adopted uses a single instrument – as a long form where commentary plus scoring is solicited and as a short form where numbers of respondents will be larger; the narrative and the quantitative aspects are complementary.
  • It has been designed to apply upon completion when memories are fresh and the knowledge still current; it can also be applied at any point in the future or re-applied as a comparative measure of time dependent impacts.
  • In this way it can be applied today as a current measure of impact and simultaneously as a measure of positioning for future prospective impacts – as assessed by those most involved.
  • This is why it is described as a preliminary baseline impact measurement tool that captures expectations as well as examples.
  • Impact analysis is not the same as an evaluation but may provide needed input especially if baseline data has been collected during or just after completion since most evaluations occur much later.

The Impact Measurement Instruments

The deployment was quite straight forward as follows:

TFCI described the development process and demonstrated the two forms of the impact measurement instrument to the CFIA-led Fore-Can team. The project leader first sent the long form to 54 potential participants – of whom four declined to participate and four responded with many comments plus scoring. The short form was then sent to all, and ten more responses were received – mostly just with scoring of the 50+ variables; based upon the short notice and lack of solicitation before emails were sent, it is positive that 14 responses in total were received out of 50 potential ones. With more advance preparation, this rate of 28% could easily be doubled. TFCI then managed a dual analysis – combining the quantitative and the qualitative responses.

The Measures

The actual measurement, distributed amongst several different lenses (or measure groupings), consists of a total of 54 measures. The first lens or level of impact interest is in terms of general role effectiveness: wherein foresight is seen as generally playing or performing as many as five roles to differing degrees;


The second set of impact measures, lens or grouping, consists of several general benefits, as perceived main-ly by those directly involved. As the impact data base and diversity of cases grows, differing patterns of pro-tagonist and stakeholder appreciation may emerge.


A third set of measures is obtained by using a success factor lens, which is especially relevant for foresight process designers and planners:

A fourth set of lenses consists of seeing foresight main-ly as a macro or meta process, focused on foresight as essentially a learning process and that each foresight project educates someone, and usually all participants. Here the evaluation team collected testimonials, anec-dotes, personal stories etc. In the category “training & skills development” the evaluators acknowledged that foresight is often motivated by sponsors wanting to strengthen readiness, resilience and preparedness skills.

These categories also give credit to the notion that fore-sight is a key tool for risk assessment and the man-agement of uncertainty.


And finally, foresight is closely aligned with design and planning. Accordingly, the participants of the evaluation had the opportunity to give account of the changes induced by the foresight exercise such if their organisa-tion achieved new strengths, there was any evidence of foresight in adopted priorities or of new directions with foresight-derived origins.

Alignment with Policy Cycle

Further, in the impact design, three groups of measures were developed – related to successive stages of the policy cycle: pre-policy; policy implementation and post policy. Here the participants had to give a score (# score represents average out of 5 including all scores other than no response).




The Response

Overall, these preliminary impact results indicate that the project had both a significant impact on participants from a present time vantage point and a well-positioned potential for future impact as expressed by the clear and consistent trend in the results toward impact endorsement in most of the variables examined. The conclusion to be drawn is not only that the project was quite successful in operational terms, but also that its full impact may only be known some years hence, given the strong prospects for future impact that were cited by most respondents.

The lists of the top and bottom five impact elements provide a snapshot both of domains where there is strength or weakness but also reflect a high degree of alignment amongst the respondents. Also of note is that 2/5 of the highest and lowest impacts are from the critical success factors elements (questions # 6-13), and this suggests that the CSF list is a key differentiator of impact – as was intended by Calof and Smith when they undertook their study in 2007.


Room for Improvements

The findings and the comments together present a consistent picture of a project that was both successful in achieving its intended near-term objectives and is well positioned for future impact and development opportunities. The ratings questions worked well to elicit stakeholder differentiation, which is normally regarded as indicative of a good engagement process, and many of the excellent comments reinforce this.

Because of the clear and generally enthusiastic responses, prospects for continued support from the participants for follow-up activities appear positive.

The combination of a long form and short form for impact assessment was viable, but both formats could be improved. The long format should be tailored to interviews, with some additional guidance provided. While it worked very well to elicit substantive commentary, it clearly was too daunting in terms of the time commitment required for most, particularly in that the impact analysis was an unanticipated additional time commitment for all stakeholders. Given the generally responsive attitudes, it is reasonable to assume that with more lead time, improved instruments, structured impact discussions built into the last meeting and a clear link to next stage development ideas, a response rate of over 60% can be anticipated – double what was received with almost no advance notice and no context preparation. The short format worked very well but likely missed a relatively easy opportunity to obtain short commentary on each of the eight sections of enquiry – thus enabling participants to elaborate the basis for their scores. The next version of the impact instruments will embody these improvements.

Overall, the post project preliminary impact baseline measurement has been very productive: baseline data and a set of premises for future development and evaluation/assessment have been established, and much of the impact experience has been captured in comments and scores that validate the benefits of the project – notably while still vivid and current.

Key Issues Raised Relevant to Policymaking

The main implication is that policy authorities can now have access to a reliable interim foresight impact measurement instrument aligned with stages of the policy cycle – and as experience accumulates with its application, governments can begin to benchmark their foresight project impacts against other projects, nations, fields etc.

Finally, the measures used for examining foresight impacts could be equally applied to most policy staging – so that at least the perception of potential impacts of policies could be measured during the development process rather than waiting for full implementation – when it is likely too late to adjust them.

Authors: Jack Smith, TFCI Canada Inc. and Telfer School of Management, University of Ottawa, Canada. (JESMITH@TELFER.UOTTAWA.CA)
Sponsors: Canadian Food Inspection Agency
Type: FORE-Can: national foresight project on animal health and food security – measurement phase
Organizer: Dr. Shane Renwick (CFIA SHANE.RENWICK@INSPECTION.GC.)
Duration: 2011
Budget: € 10,000
Time Horizon: 2011
Date of Brief: July 2012

Download EFP Brief 249_Measuring Impact of Foresights

Sources and References

Jonathan Calof, Jack E. Smith, (2012) “Foresight impacts from around the world: a special issue”, foresight, Vol. 14 Iss: 1, pp.5 – 14

EFP Brief No. 239: Corporate Foresight – A Delphi Study

Friday, December 21st, 2012

The purpose of this paper is to provide new impetus to the design of strategy and innovation processes in companies. Its intention is to contribute to the discussion of methods of future studies and thereby to increase the practical relevance of future research in businesses. To this end, the specific requirements that these methods have to meet in order to be applicable in companies are presented and recommendations given both for companies and the profession of future research.


Looking into the Future: Methods of Future Studies

In every business, there is the need to gain insight into future trends to be able to respond to forthcoming challenges, but it is impossible to identify such trends without attempting to look into the future. As fantastic as it may seem, the application of the methods of future studies actually makes this look into the future possible. However, the use of the methods is often perceived as incompatible with the current workflow. Therefore, this study is primarily concerned with the question of how the methods of future studies can be best applied in business environments.


Making Strategy Processes More Profitable

The paper intends to give impetus to the discussion about methods both in the discipline of future studies and in businesses considering the specifics of future studies when applied to the business context. The main goal is to set the stage for improvements of the methodological quality of future studies when applied to businesses and to increase the relevance of future studies to businesses. It aims to supplement the discussion of methods in future research and thereby increase the practical relevance of future research in business. These requirements can serve decision-makers in companies and research to plan and evaluate the methods used to make strategy processes more profitable and efficient.


Methodological Background of Delphi

The methodology of this study consisted of a literature analysis, an empirical study and the deduction of theoretical and practical implications. The first step to answer the research questions was to examine the theoretical and conceptual background by means of a literature review. Subsequently, an empirical survey in the form of a preliminary and a main study was carried out. The preliminary study consisted of 15 expert interviews. Then a Delphi study was conducted in two rounds. The results of the empirical survey served to derive the requirements that the methods of future studies would have to meet in companies. Recommendations, both for the discipline of future studies and for companies, on how the methods can be modified so as to meet those requirements were described. The research project was based on the mixed-methods approach with an emphasis on qualitative research. In the preliminary and the main study, different qualitative methods were used. In the main study, quantitative data and qualitative data were triangulated.


Participants of the Study

A total of 204 experts were invited to participate in the study. Of those invitees, 58 took part in the first round of the consultation and 35 in the second round; 32 participants completed the entire survey. The experts chosen to participate in the survey were required to have wide experience in the use of methods of future studies in businesses. The goal was to involve experts with diverse professional backgrounds. Some experts had an academic background in areas of future studies and innovation management, some came from strategy and innovation departments of both SMEs and global corporations, and others from a background in management consulting and research and development departments.

Problems and Requirements in Applying Methods

The empirical results show that there are specific challenges in applying the methods of future studies in businesses. The methodological design and the implementation of the methods often prove to be difficult. Among the reasons for these problems are lack of knowledge, processes that take too long, limited human and financial resources as well as difficulties in communicating the results. The identification of these problem areas made it possible to derive a set of requirements that the methods of future studies have to meet so as to be applicable to businesses: they have to be easily learnable, transparent, motivational and easily communicable. Further, measurability, the capability to tie in with other methods, the scalability of the method and possibilities for collaboration are important.

Learnability, Transparency and Transferability

The methods have to be learnable with reasonable effort at different skill levels because there is often a lack of methodological knowledge in business settings and a knowledge gap between different hierarchical levels. The results of this study also show that there is not only a lack of knowledge about the necessary methodological steps but also uncertainty about the potential insight to be gained by applying the methods. Therefore, both the concepts of the methods applied and the ways in which they can be implemented have to be transparent. It is further necessary that the methods can be transferred both to and from other fields of application. This need arises from the ever-expanding range of methods, from limited human resources and from the diverse intentions that can motivate the use of the methods.

Motivational Potential, Communicability and Evaluation

The empirical data point to difficulties in motivating the people involved. Since it is crucial to produce and maintain motivation, the methods should satisfy the criteria of being motivational. The communicability of methods is also central in the corporate context. The study shows that there is scepticism about the discipline of future studies and its methods that needs to be addressed. Successful communication can also help to avoid false expectations, which otherwise are often perceived as serious obstacles. Another requirement is the measurability of the process and the outcome. The need arises because many experts believe that it is impossible to verify the outcome of the methods based on “hard data”. The results of the study show that many experts for this reason emphasise the value of the process itself.

Scalability, Flexibility and Collaboration

The empirical data show that the period for the implementation and evaluation of the methods is often perceived as being too long. This suggests that there is a need for temporal scalability. The length of the implementation period, and thus the costs, must be adaptable to the actual situation of the companies. Further, the methods should allow for joint implementation since knowledge from within the company has to be extracted and made explicit. The study reveals hurdles in this process; the involvement of all stakeholders is perceived to be difficult. An essential point of concern is the complexity of the research object, which requires that the chosen methods can be combined. The empirical evidence suggests that stakeholder participation is already used by many, but the potential is not yet exhausted in some places.


Overcoming Hurdles through a Joint Process of Methods Development

The study revealed a number of problem areas in the use of the methods. These problems can only be solved through a joint effort on part of the profession of future research and the companies. Focusing on methods only may prevent us from perceiving the limitations but also the opportunities in applying them in certain settings or situations. Therefore, the use of the methods can only be improved if we consider the specific requirements of the companies in question. On the other hand, looking at the operational procedures in a company only may in turn prevent the emergence of new perspectives. Concentration on daily routines may lead to ignorance of the world outside the company and therefore to missing new opportunities.

A joint process of developing and adapting methodology could result in devising methods capable of transferring and integrating knowledge and research results about the future instead of creating an abstract “methodology of future research”. The study allowed to derive suggestions for a potentially successful joint working process.


How Can the Hurdles Be Overcome?

The members of the profession need both strong methodological and excellent teaching skills to facilitate the learning of the methods. The methods of future research should be taught in different contexts: in higher education as well as in vocational training and further education programs at various skill levels. Companies should identify the knowledge gaps of their employees in order to address these specifically. In order to achieve transparency, it is necessary to disclose the processes involved and the criteria used for choosing a particular method. To satisfy this requirement in a company, it is possible to focus on internal transparency so that internal company knowledge does not have to be exposed to outsiders.

Practitioners of future studies need to be proficient in interdisciplinary and interface skills to promote the transfer of methods both from and to other fields of application. In addition, an intensive exchange between future studies, related disciplines and companies is very important. Expert knowledge about motivation is needed to motivate those involved in the use of future studies methods. Both the profession of future studies and the companies have to recognise the importance of motivation for the methodological process. Further, companies should enhance social and career-enhancing incentives to motivate employees involved in foresight processes.

The basis for the successful communication of the methods is a high level of expertise and communication skills of the practitioners of future studies. This includes both verbal communication itself and communication about communication formats. In addition, the professional management of expectations and individual communication concepts are important in dealing with stakeholders in the company.

Knowledge about evaluation concepts as used in the profession of future studies is necessary to be able to assess and measure the outcome. Reviewing the steps taken should be a natural part of every project. To be able to do so, businesses need to make structural adjustments such as define responsibilities and plan a budget for foresight processes. To make sure that methods can tie in with each other, it is necessary to be open to experimentation, extend the combination of methods and also to evaluate systematically.

For the profession of future studies this means that methodological approaches have to be extended with a focus on both internal and external methods of triangulation to create meaningful combinations of methods. In order to make sure that the methods can be implemented collaboratively, existing methodological concepts have to be expanded. Possibilities to collaborate
should be integrated in methods that have been unsuitable for collaboration so far.

Solid knowledge of project management is essential to be able to assure the temporal scalability of the methods. Therefore, practitioners of future studies need to be able to acquire such knowledge during their training. The profession should also actively participate in the discussion of how to adapt the time scale of the methods. A key aspect to improve the handling of methods could be the application of appropriate software solutions.


Limits and Areas in Need of Further Research

Whether such a thing as methods of future studies even exists is an issue that is still discussed controversially within the discipline of future studies. The lack of a scientific consensus both on the methodological canon and the classification of methods is responsible for the fact that this study could only provide a limited view on the application of methods of future studies in companies. It is impossible to presume that all participants of the preliminary and the main study would share a common understanding of the term “methods of future research”. Therefore, the range of existing methods is only inadequately presented and quantified.

The sample of this study is subject to restrictions: the fact that the participating experts were selected with a focus on their experience in strategy and foresight processes may potentially distort the picture. The reality of those companies that do not deal with strategy at all is not represented in this study either; hence the need for further research. The companies’ reasons for having only little contact with future studies or no contact at all have to be understood. This would be an addition to the results of the present study and might reflect even better on the individual perspectives of different protagonists.

Download EFP Brief No. 239_Corporate Foresight – A Delphi Study.


Sources and References

Ambacher, N. (2012): “Corporate Foresight – A Delphi study on the use of methods of future research, taking into account the needs of industry and research”, Master’s thesis at the Free University of Berlin, Berlin

For more information about the study and its results, please visit the project website:

EFP Brief No. 224: Technology Radar: Early Recognition of New Business Fields in Future Markets

Tuesday, October 23rd, 2012

New technologies are changing the market. All the more important it is for a company not to miss any relevant future technology. In the years 2009 and 2010, a global German high technology company used the support of the FutureManagementGroup AG to identify the ten most important emerging technologies in each of its four business units. The technologies should lie outside the current core technologies. The goal of the project was the early recognition of future markets in these technologies. For this purpose, we used a broad toolset in accordance with the Eltville Model of future management.

Future Management

The FutureManagementGroup AG (FMG), founded in 1991, is an international group of experts specialised in future management and the early recognition of opportunities in future markets. Using the “Eltville Model” and various future management methods and tools, we built a methodological bridge from management practice to futures research and back to daily business. Future management comprises the entirety of all systems, processes, methods and tools for early perception and analysis of future developments and their inclusion in strategy.


Figure 1: Future management as a bridge

Future management makes it easier, and in many cases possible at all, to use the results of futures research as a resource for orientation and inspiration in a business context.

The Five Futures Glasses

We use the “Eltville Model”, which offers a set of five distinctive and clear views on the future. We call them “the five futures glasses”. Each of the five futures glasses has its own specific characteristics, principles and modes of thinking:

  • The blue futures glasses look at the probable future → assumption analysis.

The guiding question is: How will our market(s), work and living environments change in the next five to ten years?

  • The red futures glasses look at possible surprises in the future → surprise analysis.

The guiding question is: How should we prepare for possible surprising events and developments in the future?

  • The green futures glasses look at the creatable future → opportunity development.

The guiding question is: Which opportunities for new markets, products, strategies, processes and structures will arise from these changes?

  • The yellow futures glasses look at the desired future → vision development.

The guiding question is: What does our company need to look like in five to ten years time in the sense of a strategic vision?

  • The violet futures glasses look at the planned future → strategy development.

The guiding question is: How do we need to design our strategy to realise the strategic vision?

The five futures glasses form the process model of the Eltville  Model. You cannot wear all five futures glasses at the same time or the future will remain unclear and confusing. You need to put your different futures glasses on one after the other to form a effective working process.

The second essential component of the Eltville Model is the results model, a semantic network of objects of thought that are used (future factors, assumptions, surprises, opportunities etc.)

The Eltville Model has been developed through research and in more than a thousand workshops and projects with leading corporations as well as with non-profit organisations around the world. It is a unique model that consistently resolves the confusion concerning the future, creates clarity and provides a productive way of working with sound insights and results.

Looking for Amazing Technologies

The most important goal of the project was to identify “amazing technologies” outside a client’s current capabilities but with a potentially high impact on the existing business of the client. We were asked to evaluate the exact relevance of these technologies for the client’s business to deduce new market opportunities of these technologies and evaluate their potential.

Our solution to accommodate these needs was a “future business radar”. The focus was on the blue futures glasses (assessment of technologies) and the green futures glasses (development of opportunities). Less focus had been given to the yellow futures glasses (assessment of opportunities and decision, which opportunities should be pursued). Not included were the violet futures glasses: With the completion of the project, the business units have individually taken responsibility for developing the strategy to enter the future markets that were identified as relevant to their business.

Technology Radar: the Project Process

Function Maps

After the definition of the project goals and the project timeline, the first step was the analysis of functions delivered by the four business units. In contrast to a product or a solution, a function describes the effects that a product is actually bought for. Questions to think about to identify the functions of a product are:

  • What is it that your customers actually pay for when they purchase your product?
  • What is the actual use that your customers would like to obtain from your product?

Concentrating on the functions opens up completely new business opportunities even for the combination of products with other products from outside the current portfolio. Functions can be described at three levels:

  1. Super-functions: Functions that are indirectly fulfilled by a product or service, for example through integration into other products (e.g. personal mobility in case of all automotive parts)
  2. Primary functions: Core functions of a product or service for which it was invented. The main reason for its existence (e.g. sealing).
  3. Secondary functions: Additional functions the product or service fulfils beyond its core use. They often are the decision criteria of customers if several products can fulfil the primary functions reasonably well (e.g. convenience, cost saving).


Figure 2: Levels of functions

The relevant functions were developed in a workshop with the project team consisting of representatives of all business units and enhanced through independent analysis by FMG. The functions were then transferred to visual maps, reviewed by the business units and jointly further developed by FMG and the project team.

Long List of Technologies:
Which Ones Are Potentially Relevant?

The long list of technologies was developed from extensive secondary research. All technologies that are described in current literature as emerging and/or as gaining importance in the future where considered for the long list. The single selection criterion for inclusion in the long list was the existence of a conceivable relation to a single function of one of the business units. The connection of a technology to a function is a valid indicator for its potential relevance. It shows that the technology can change the way in which the function is performed in the future. It can provide new solutions and products as well as change business models, thus changing value creation in the market. A total of 180 potentially relevant technologies have been identified.

An important source in the desk research was the FMG-FutureNet, a semantic database of futures knowledge. It is a knowledge network, modelled on the human brain, in which items of future information are saved and linked. We structure the available future knowledge and evaluate, summarise, substantiate and meaningfully link the individual items of futures information. In addition, we add information gained in our projects. As a result, the FMG-FutureNet has become a unique database of future markets.

For the technology radar project, we additionally evaluated websites, studies, books and magazines.

Short Lists of Technologies:
Evaluation of Technologies

The technologies from the long list were evaluated along two criteria: “impact on industry” and “reasonable time horizon”. The initial evaluation was done by representatives from the business units on a 9-point scale. A second evaluation was performed by FMG leading to some technologies with low rankings to be reconsidered. After a structured discussion process, each business unit selected ten technologies for deeper analysis. In total 32 different technologies were analysed and the results summarised in technology briefings.

Identification of Future Market Opportunities

A future market is a solution for important future problems or desires of certain people that develops or will generate significantly more revenue in the future. Examples of future markets include augmented reality glasses for smartphone users, robots that carry luggage and equipment for the military, or affordable space tourism for adventure travellers. The difference between a future market and a future trend or future technology is that one can additionally imagine which concrete solution people would actually be prepared to pay for and how you can make a profit out of it.

Future market opportunities were developed through analytical and creative thinking, including input like future factors and methods like meta-opportunities, which we would like to introduce here briefly.

Future factors are trends, issues and technologies that act as the driving forces of future change and allow us to collect knowledge about the future. They are based on existing knowledge of experts and futurists on possible and probable future developments. Future factors give indications on what, why and how the future is changing. Two types of future factors are important for the early recognition of future markets:

  1. Future factors in nature, society, business and politics that change the needs of end consumers. Examples are climate change, feminisation, entrepreneurisation, flexibilisation or globalisation
  2. Future factors in technology and science that will change processes and methods as well as products, services and solutions. Examples are nanotechnologies, dematerialisation, informatisation, micro-system technology, robotics or neurotechnologies.

Future factors primarily represent the view through the blue futures glasses but can also be used as a technique to support creative thinking. This is especially fruitful when future factors have no direct relation to the client’s industry.

Meta-opportunities are repetitive patterns that are recognisable in many future opportunities. These patterns are recipes and shortcuts for opportunity recognition. They illustrate models of best-practice thinking and stimulate the search for opportunities. Through the use of meta-opportunities, productivity and the value of opportunity development can be increased considerably.

Subsequently, the identified and developed future market opportunities were set in relation to the business units and to the functions fulfilled by the business units in particular. In addition, the technologies were analysed for the interrelations among each other. From 98 raw future market opportunities, ten were selected for each business unit to be described in a short portrait. The criterion of choice was the estimated market potential. The selected future markets were described following four main questions:

  1. Which problem is solved? Which desire is fulfilled?
  2. What is the solution?
  3. Whom is the solution delivered to?
  4. How is the solution special?

Finally, the time horizon of the future markets was evaluated from a technical and a demand perspective; the markets were classified in terms of their distance from current capabilities.

A Strong Case for Function-based Technology Assessment

An important goal of the project was not to miss any relevant technology. This was ensured by an overview scan and the analysis of the results of futures research concerning the emergence and further development of new technologies. Simultaneously, the technology radar served as a future business radar, as it identified the most promising future markets that lie in the most important technologies. Out of 180 technology candidates that were included in the long list, we created 41 differentiated and in-depth future market portraits.

The project has shown how function-based technology assessment can contribute to identify relevant technologies outside current competencies and businesses – an essential requirement to recognise potentially profitable future markets.

The most promising of the recognised future markets needed to be explored in more detail. Future markets can only be considered as realistic if there are enough arguments for their future market potential. Therefore, the next step for each business unit was to do detailed future markets research for selected markets. The future


Figure 3: Map of results

markets research provides a solid analysis of market prospects, key challenges and possible business models. It thus allows sound investment decisions for the development of a future market.

Authors: Enno Däneke   

Stefan Schnack

Sponsors: A German high technology company
Type: Sectoral forward-looking analysis
Organizer: FutureManagementGroup AG, Eltville, Germany
Enno Däneke,
Duration: 2009 – 2010
Budget: n.a.
Time Horizon: 2020
Date of Brief: July 2012

Download: EFP Brief No. 224_Technology Radar Eltville

Sources and References

Mićić, Pero (2010): The Five Futures Glasses: How to See and Understand More of the Future with the Eltville Model. Houndsmill, Basingstoke, Hampshire: Palgrave McMillan.

Mićić, Pero (2007): Phenomenology of Future Management in Top Management Teams. Leeds: Metropolitan University.

Mićić, Pero (2006): Das ZukunftsRadar. Die wichtigsten Trends, Technologien und Themen für die Zukunft, Offenbach: GABAL-Verlag.

For further information on future management, the Eltville Model and the Five Futures Glasses, please visit:


EFP Brief No. 218: Embedding Foresight in the Colombian Innovation System

Thursday, June 21st, 2012

This follow-up brief recapitulates the evaluation of the Colombian Technology Foresight Programme (CTFP). The foresight brief no. 119 (“Evaluating Foresight – The Colombian Case”) summarised the methodological aspects and principal focus of the framework on which the evaluation of the second cycle (2005 – 2008) of the CTFP was based. The objective of the present follow-up brief is to look back and summarise the implications of the evaluation of the CTFP by drawing on the key findings of the evaluation summary report. Concretely, it focuses on (1) the appropriateness and adaptation of the evaluation framework, as well as the effects for the spread of a foresight culture in Colombia that have been induced or stimulated by the evaluation of the CTFP and (2) the institutional mechanisms in support of the social appropriation of the CTFP’s output and results as well as the dissemination of the foresight knowledge generated by the CTFP to policy, industry and society as a whole.

Evaluation to Improve the Capacity for Learning

The development of Colombia’s Technology Foresight Programme (CTFP) has long been a reference point in the Latin American region. The CTFP is the first national foresight programme in Latin America that has been evaluated so far. The principal idea of the CTFP has been building a platform to create, distribute and utilise foresight knowledge in Colombia. It was intended to introduce new foci and new types of foresight practices and interventions in support of the strategic re-orientation of programmes and (sub-)sectors.

The focal point of the evaluation carried out in 2008 under the leadership of the University of Manchester was to reshape the objectives and activities of the second cycle of the CTFP (2005 – 2008). The conceptual framework of the evaluation was geared towards analysing foresight as a process.

This follow-up brief describes the methodological framework of the evaluation and discusses the learning process involved as well as the question whether the evaluation improved the aptitude for learning.

Evaluation of the CTFP

In Colombia, the evolution of long-term thinking in foresight has been largely driven by the role of COLCIENCIAS (Colombian Office of Science and Technology) as a node institution capable of facilitating inter-institutional alliances between various centres of excellence, on the one hand, and mobilising resources and engaging key stakeholders into a dynamic and self-reinforcing foresight learning process, on the other. Part and parcel of this learning process has been the comprehensive evaluation of the second cycle of the CTFP, which was geared towards identifying and supporting strategic sectors during the period between 2005 and 2008. Commissioned by COLCIENCIAS, the overall aim of the evaluation was to increase the CTFP’s capacity to shape and inform policy processes and actors.

Methodological Approach and Phases of the Evaluation Process

The evaluation of the CTFP was based on a methodological framework composed of a mix of seven diverse methods and activities that have been listed and described in some detail in the original foresight brief: (1) documentary analysis, (2) logic chart and indicators, (3) surveys, (4) interviews, (5) case studies, (6) benchmarking and (7) evaluation forum.

The evaluation process was divided in four phases:

Phase 1: Scoping – This phase had the principal objective to understand the main rationale of the evaluation process in order to design a coherent research process. The evaluation report states, “In addition to the traditional objectives of a Foresight programme evaluation (i.e. assessment of the impacts of the programme and the projects; assessment of the cost-effectiveness of the programme, and evaluation of the way in which Foresight is run in Colombia), COLCIENCIAS and SECAB [Secretaria Ejecutiva del Convenio Andres Bello] were particularly interested in identifying lessons and recommendations for the improvement of Foresight and horizon-scanning activities in the country.”

Phase 2: Understanding – This phase of the evaluation process was based on the collection of tacit and codified knowledge about the CTFP. “Tacit knowledge was collected through individual and group interviews with key stakeholders in COLCIENCIAS, other sponsors (e.g. ministries) and main stakeholders […]. Codified knowledge involved the compilation of major codified products (e.g. interim and final reports, books, journal publications and other important documents, such as individual project budgets and description of the programme’s expenses).”

Phase 3: Evaluating – This phase was based on a benchmarking of the CTFP against practices in other countries. The main objective here was to learn from other international best practices in establishing a national foresight initiative. The lessons shared from other countries included the UK, Malta, Russia, Spain and Hungary.

Phase 4: Learning – This phase involved conducting further analysis and preparing the final evaluation report.

Lessons Learned

The focus of the evaluation was on assessing what effects the second cycle had on policy and programme development. In particular, the impact of the CTFP on the design and of research policies was to be assessed as well as the effects of the CTFP on the promotion of national skills and the establishment of a national foresight culture in Colombia. In this sense, the evaluation was an important step towards synthesising the lessons derived from the national foresight exercise in terms of impact on skills, enhancement of capacities for strategic decision-making processes and policy design.

Key Findings of the CTFP Evaluation

The final report summarised the key findings of the report as follows:

(1) Regarding the overall objectives, the evaluation report states that the “CTFP objectives have been appropriate and successfully achieved. The programme has contributed to the creation of development visions and strategies for moving towards a knowledge-based society” through horizon-scanning and the building of foresight capacities in key sectors.

(2) As regards the value for money, the evaluation concluded that the CTFP achieved “a paradigm shift” by greatly contributing to the creation of a shared vision for “the productive transformation of Colombia into a knowledge-economy”. Furthermore, the evaluation states that the CTFP has begun to pay off since a diverse set of stakeholders have already adopted the vision brought forward by the CTFP in formulating their long-term objectives.

(3) Regarding the organisational structure of the CTFP, the evaluation observed that the institutional anchoring of the technical and decision-making groups in COLCIENCIAS during the second cycle notably “increased the CTFP’s capacity to shape and inform policy processes and actors. However, these changes also made the programme appear to be more of a COLCIENCIAS instrument than a national programme.”

(4) With respect to the approaches and mix of methods, the evaluation highlighted that “one original and effective feature of CTFP has been the combination of thee conceptual and methodological approaches: Foresight, horizon scanning and productive chain.”

(5) Regarding implementation and aftercare of the second cycle, the evaluation stressed the need for an aftercare strategy in the Colombian foresight programme. If the key support institutions of the programme “were to consider implementing such a strategy, this would probably increase the ability of Foresight to inform policy and shape research priorities. At the same time, it would also allow sufficient time for new networks to exploit the momentum created and consolidate institutional alliances.”

(6) With regard to the CTFP’s contribution to the spread of a foresight culture in Colombia, the evaluation states: “Some stakeholders still see foresight as being exclusively expert-oriented.” Therefore, it was recommended “that the general public be encouraged to participate in projects and training courses. This would probably require alliances with the private and productive sectors, in order to increase the financial and implementation feasibility of large-scale courses and projects.”

(7) Concerning the presence and visibility of the CTFP, the evaluation states that “while CTFP stands up well alongside programmes conducted elsewhere, it has limited visibility in the international academic and professional literature.” A clear implication derived from this is “that all major reports should be […] made available on the internet, and more widely disseminated through, for example, conference presentations and articles in relevant publications.”

(8) With regard to the impacts related to science, technology and innovation (STI), the evaluation stated that out of a total of 24 projects and more than 30 capacity-building courses, “nine projects had positive impacts on public and private polices and strategies; six projects had positive impacts on the agendas of STI programmes and institutions; five projects had positive impacts on the consolidation of research groups; two projects had positive impacts on the consolidation of S&T capacities; and two projects had positive impacts on international projects.”

(9) Concerning policy recommendations and strategies, these have been highlighted as “fundamental elements of CTFP outputs”. According to the evaluation, “the most significant influence of the CTFP on public policy has been the work on the STI Vision 2019.” Moreover, the CTFP’s biotechnology project had a significant influence on the policies and research priorities of COLCIENCIAS’ National Biotechnology Programme.

Networking Key to Spreading Foresight Culture

The strengthening of networks was a central pillar for achieving the principal purpose of the CTFP. In this sense, the enlargement of inter-institutional networks was instrumental for the realisation of a strategy to spread a foresight culture in Colombia. On the side of the sponsors and organisers of the CTFP, building networks to facilitate the involvement of different expertises in support of an interdisciplinary approach to foresight in many fields and sectors was a vital aspect for all those who participated in the CTFP.

Dissemination Strategy Falls Short of Potential

The CTFP produced a large number of high-quality scientific outputs. However, the strategy to disseminate the preliminary and final results has pursued an approach not conducive to enhancing the international visibility of the CTFP as a best practice across the region. Moreover, in disseminating the new knowledge generated, there was a tendency to emphasise tangible over non-tangible outputs. Since the degree to which the newly produced knowledge contributes to the opening of new strategic options or the strategic re-orientation of sectors is an important success criteria for foresight, it is vital to embed this new knowledge in people’s and organisations’ practices. This requires that the dissemination of the results be tailored to different target groups, which also calls for different means of communication appropriate to the respective group. Although the CTFP delivered many tangible outputs, such as scientific publications (predominantly in Spanish), the approach chosen to disseminate the outputs and results remained far behind the possibilities of alternative approaches that could have enhanced the interaction between the different governmental and scientific communities or increased the international visibility of the CTFP as a best practice from which others could learn.

Methodological Progress Towards Context-sensitive Use of Methodology

The choice of the methods is the most distinctive feature of a foresight process. The evaluation of the CTFP stated that “an average CTFP study involved more than 10 methods, with more or less half of these being horizon-scanning techniques (including bibliometrics, trend extrapolation and patent analysis) and the other half related to Foresight and productive chain approaches (e.g. scenarios, brainstorming, stakeholders mapping, key technologies, morphological analysis, among others).” Regarding the use of foresight, the evaluation focused on the selection of methods but did not monitor how well the methods used lined up with the STI implementation environment, specifically in regard to strengthening capacities in support of policymaking in the area of science, technology and innovation with due regard to local concerns in Colombia. Therefore, a future monitoring or evaluation framework might also focus on adjusting or reconfiguring the methods applied to fit the foresight process needs in Colombia.

Towards Enhancing Colombia’s Foresight Capabilities

Given that there is no ‘one-size-fits-all’ approach to evaluate foresight and that the evaluation of foresight activities cannot be carried out independently of the national context (Georghiou and Keenan, 2006), it is difficult to assess the interaction between foresight and evaluation with respect to the impact on policy instruments or the improvement of overall system coordination through strengthening linkages between innovation actors.

The concepts of effectiveness, efficiency, appropriateness and behavioural additionality on which the framework to evaluate the CTFP rests are key to understanding the focus of foresight activities that are carried out to reconfigure institutional set-ups and re-orient policy goals. Georghiou and Keenan (2006) state that the “[…] evaluation of foresight must include understanding of the interaction of foresight outputs with the strategic behaviour of policy and economic actors.” In this sense, the evaluation of the second cycle of the CTFP was an important step towards better understanding the drivers of the strategic behaviour on part of the key implementing institutions in the Colombian system. However, further advancing Colombia’s foresight capacities depends to a high degree on the institutionalisation of foresight in the Colombian context.

Improve Dissemination Through Alternative Communication Channels

According to the results of the evaluation of the second CTFP cycle, a key lesson that can be drawn refers to the appropriation of the insights gained from the foresight programme. Although the fact that the foresight programme was conducted by COLCIENCIAS was an important institutional driver, the lack of an aftercare strategy constitutes a weak point for the strategic influence of the foresight knowledge generated on the target groups and sectors. Therefore, an important recommendation to increase the future impact of the CTFP is that alternative dissemination and communication channels should be exploited to a greater extent. An improved dissemination strategy should also take into account the need for a stronger diversification of foresight capacity building in Colombia.

Towards Further Institutionalisation of Foresight in Colombia

The “shift from networks and individual exercises [….] to more institutionalisation towards centres of excellence“ is an important step to “take on responsibility for preserving knowledge and for allowing lessons learned to be carried forward in a long-term framework“ (Popper et al., 2010). In this sense, the evaluation revealed that a move away from the somewhat centralistic approach to anchor the foresight process in COLCIENCIAS towards a more effective institutional mechanism was a necessary step to better embed foresight in the Colombian STI system. COLCIENCIAS recent decision to institutionalise the foresight practices in the framework of the CTFP by establishing the Colombian Foresight Institute (COFI) at the Universidad del Valle (Cali) can be considered an important move to enhance the aptitude for learning and thus strengthen the contribution of foresight to reorienting the Colombian STI system. In this arrangement, multiple organisations will be able to conduct foresight.

Under the bottom line, we can conclude that the evaluation came at the appropriate time to develop recommendations on how the foresight outputs, results and knowledge generated during the second cycle of the CTFP could be better appropriated by the stakeholders and embedded in a broader strategic policy context. In particular, an improved dissemination strategy and the search for alternative ways of institutionalising foresight are central pillars for engaging future resources and a broad set of stakeholders in a dynamic and self-reinforcing learning process based on which a foresight culture can develop in line with the evolving STI policy system in the Colombian context.

Authors: Dirk Johann                                                                  
Sponsors: COLCIENCIAS (Colombian Office of Science and Technology)  
Type: Evaluation of Foresight Programme  
Geographic coverage: Colombia
Organizer: COLCIENCIAS (Colombian Office of Science and Technology)

PREST / Manchester Institute of Innovation Research (MIoIR), MBS, University of Manchester

Duration: 6–9 months Budget: € 40k Time Horizon: 2020 Date of Brief: May 2012

Download EFP Brief No 218_ Embedding Foresight in the Colombian Innovation System

Sources and References

Georghiou, L. and M. Keenan (2006), “Evaluation of National Foresight Activities: Assessing Rationale, Process and Impact”, Technological Forecasting and Social Change, 73, 761-777

Popper, R., L. Georghiou, M. Keenan, I. Miles et al. (2010), Evaluating Foresight – Fully-fledged Evaluation of the Colombian Technology Foresight Programme (CTFP), Colombia: Universidad del Valle