Posts Tagged ‘transparency’

EFP Brief No. 258: Perspectives on Use of Expertise in Futures Studies

Monday, December 21st, 2015

Numerous foresight activities rely on gathering expert knowledge, using e.g., the Delphi method. A crucial question for the quality of the exercise is who the relevant experts are. The question is particularly difficult when studying so-called wicked problems, which elude exact definition. Inclusive definition of expertise is called for in this brief, particularly because of the social power experts have. This brief tackles the complex issues of characterising expertise and taking full use of it in expert-based futures projects. Transparency of the chosen expertise must often be combined with upholding anonymity.

Facing Complex “Wicked” Problems with Methods Using Expert Knowledge

Numerous foresight methods are based on expert information, such as Delphi studies. In practical applications of these methods, the researchers have to make judgements on who is an appropriate expert.

This is particularly difficult in the case of the so-called wicked problems (Rittel and Webber 1973). Such problems are very complex with many definitions, and each definition seems to carry a presupposition of a solution. The solutions are based on a variety of theories, assumptions and values. Therefore, in many foresight applications an interdisciplinary approach is chosen, and experts are invited from different domains. In this way, both knowledge about different facets of a problem (such as technological, economic, and societal) is represented, and the different theories and concepts behind them are included.

For example, if climate change is seen primarily as a technological problem resulting from fossil energy use, the responses are likely to be expressed in technological or economic terms. If the problem is seen to result fundamentally from global inequalities, the results focus on international agreements and funding mechanisms. If the problem is seen to result from population growth and modern lifestyles, the answers are to be found in the cultural and social spheres. No single framework could encompass the ecological, economic, social, cultural and technological dimensions of climate change.

Obviously in futures research, a temporal aspect further complicates the issue. It is not enough to know what the state of the problem is; we would also need to know how the situation may change.

If we want to know how things are right now, or how they will be in the near future, often it makes sense to ask those who can be defined to be on the top of their field. This approach often means asking middle-aged or older people with long work histories. However, the aims of foresight processes are often about opening new visions, finding new trends or unexpected turn points, considering trends and changes on long time scales, and finding ways to reach futures we define as desirable. Then, narrowly defined expertise may not be an optimal solution.

 

Objective

In this brief I will outline certain definitions for expertise, and discuss their implications for the use of experts in futures studies. The aim is not to provide definite answers, but rather to inspire discussion and make foresight processes more sensitive to alternative forms of expertise.

Defining Expertise

  1. Expertise as cognitive property and a social construction

Expertise can be seen as both cognitive property and a social construction. From a cognitive perspective, expertise refers to knowledge and skills of a domain of activity. It can be acquired through education, experience or any other form of cognitive refinement. However, it is not necessarily linked to the social status of an expert. The social status often follows from formal degrees, higher professions and leading organisational positions. These properties are considered to indicate expertise, but they do not guarantee it, and similarly, a person without a socially acknowledged expert status may possess similar skills to an authorised expert.

Defining the content of expertise, i.e. the appropriate cognitive resources, or skills that matter in a domain of practice, is a social process (Turner 2001). The content of valid expertise changes over time and cultures. For example, medicinal practices that were considered valid in the 16th century Europe are very far removed from those practiced today in the western world, and different from traditional Chinese medicine. The content also depends on the definition of the problem at hand. With wicked problems that elude definition, it becomes increasingly difficult to determine what kinds of expertise should be included in a futures project.

 

  1. Different types of knowledge

Bogner and Menz (2009) distinguish between three different types of knowledge. First, there is technical knowledge, which we typically gain through education. Then, there is process knowledge, which is close to tacit knowledge. It is knowledge that is gained through working in a field, and consists of knowledge regarding the practices and modes of operation within a field, such as patterns of interaction and organisations.

The third type of knowledge Bogner and Menz (2009) call interpretative knowledge and it means a person’s subjective assumptions, views, interpretations, rules, etc. The key idea here is that all we learn through education or practice is interpreted and assimilated through our previous knowledge, values, and experiences. Therefore even identical education and work history could not produce two identical experts.

Such subjectivity has often been considered a problem that detracts from expertise, which is commonly thought to be objective. It is, however, quite impossible to avoid. Selinger and Crease (2002: 245) note that expertise is built upon the person, and the “prejudices, ideologies, hidden agendas, or other forms of cultural embeddedness that person might have” do not disappear during the process of becoming an expert.

More importantly, subjectivity may not even be a problem. It is through subjectivity that many important aspects of expertise emerge into foresight processes, such as ethical consideration, empathy, and sense of responsibility. They all derive from experts’ ability to personally engage with problems and their solutions.

If we accept that expertise itself is affected by the person, and reflects the social environment in which it has been accumulated, it becomes important to search for variety not only in terms of multidisciplinarity but also, for example, in terms of gender, age, and ethnicity.

 

  1. Why does the definition of expertise matter?

Defining someone as an expert gives him/her social power to define problems and suggest their solutions. Experts are therefore important actors in society. Expert knowledge is a vehicle for maintaining or changing the existing patterns of thought and action. While experts do not alone determine the future, they are powerful in defining what is real and possible in a society. In addition, expert knowledge is not entirely transparent, and there is limited possibility to hold experts accountable for their power.

In sustainability discourse, in particular, the widening of the definition of expertise has been called for. There may be a need to look at “counter-expertise” i.e. expertise beyond the establishment (e.g. non-governmental organisations). It is not only a matter of democracy; different backgrounds also produce different expertise.

Because expertise is not dependent in a straightforward way on formal degrees or titles, it can be argued that there is no such thing as a “lay expert”. A person may either have relevant expertise or not, regardless of the status as a layperson. However, it may make sense to describe where the expertise springs from.

Using Exertise in Futures Studies

Why is expertise useful in futures studies? It is not just that experts know a lot. Information might be found through other means. Instead, in futures projects, such as Delphi studies, it is often important to make intuitive and quick estimates about future possibilities, and about the impact of changing drivers on the topic in question. Intuitive judgement has even been considered a central ability of an expert (Dreyfus and Dreyfus 2005). Careful consideration does not replace but improves the expert’s intuition.

  1. The expertise matrix

The wide definition of expertise that was discussed above poses a new challenge to the expert selection. If we assume that various types of expertise need to be included in the panel, there need to be tools for keeping track of the variety. Expert matrix (introduced by Kuusi et al. 2006) is a simple tool for both ensuring the variety and for making it more transparent for the audience of a foresight exercise.

In the matrix, the desired variety is detailed, and during the assembly of the panel, the expertise of the panellists is marked in it. For example, in a research project regarding renewable energy growth, we listed various energy sources and forms on one hand, and various roles within a value chain on the other hand. Then we searched for e.g., an expert being familiar with biogas and energy transfer (Varho et al. forthcoming).

It is also possible to make a longer list of expertise characteristics. In another project, we wanted to have variation in terms of topics affecting the future of transport (such as fuels, engine technology, land use, and behaviour), transport modes, field of education, level of education, background organisation, age, and gender. These were listed in a table, in which each expert was characterised (Varho and Tapio 2013).

It is not necessary to have perfect cover in a matrix. Finding different people for all combinations of our transport project expert table, for example, would have meant thousands of experts. However, we were able to get significant variation, and the table also increased the transparency of the process considerably.

 

  1. Facilitating expert deliberation

As we all, also experts may get too focused on conventional wisdom and current or past situation. As we discuss the future possibilities, there is often a need for people who think “out of the box”. Some people are more able to do this, but it is possible to encourage this type of thinking in the whole panel.

One way to do so is to ask experts to describe both a probable and a preferable future (Amara 1981). This approach does not only accept but embraces the subjectivity of experts. It helps the experts to recognise their preferences (in preferred future) and also to aim for objectivity (in probable future). Even if total objectivity is impossible, it is an ideal that many experts strive for, and they may feel more comfortable when they have these two views to distinguish between.

Different methods for gathering data from experts may encourage the expression of new views. For example, the use of expert interviews allows new ideas and interpretations to be incorporated into the futures project. Usually in e.g. Delphi studies it is the research team that formulates the questions. New or alternative interpretations of the problem at hand may not emerge during questionnaire rounds. In addition, some experts may not feel comfortable giving numerical estimates of the future development. The combined use of numerical, visual, and verbal answers can encourage different types of experts to express their views (Varho and Tapio 2013).

Anonymity of expert panels is usually sought for. It is useful, among other things, for encouraging also those with less obvious (social) expert status to express their views. However, face-to-face meetings such as workshops can be introduced at some point of a project, for more thorough communication and co-learning.

Discussing the Use of Expertise

In this brief I have outlined some perspectives on experts and the use of expertise in futures projects. Some questions arise from these, although definitive conclusions cannot perhaps be drawn.

 

  1. What are experts good for?

Using expert views is often valuable in futures projects. It can be an “economical” approach, as experts are able to give estimates regarding complex systems. When the futures project aims more at estimating how the future unfolds than at creating new visions, conventional expertise is naturally important. But the further we look into the future, the less the existing knowledge may be relevant. Do we really need experts, or more specifically, those we define as experts?

In a recent project (see Tuominen et al. 2014) we asked both transport experts and high-school students to describe the future of Finnish transport. It was interesting to see that the students were able to describe in their essays very diverse futures that, in most part, reflected the alternative future visions of the experts. However, they were not able to give plausible numerical estimates that would have reflected the qualitative visions. It seems that expertise on a subject is valuable – at the very least – because experts are more able to give numerical estimates. Numbers, in turn, are often very useful for distinguishing the future views apart and for comparing them with existing or targeted levels. In addition, transport was a subject that all students have some experience on. A more esoteric subject might not have gained equally valuable answers from them.

 

  1. Are experts “experts” at all?

Given the subjectivity of experts that was discussed in this paper we need to ask if they should be defined as experts, or are they e.g., stakeholders. There may not be a conclusive answer to this question. It is likely, however, that when people are invited into the futures project as experts they aim more at objectivity than when they are defined as stakeholders. In the stakeholder position, they may even feel obligated to defend the interest of those they are invited to represent.

It is possible to include in an expert panel people who would not define themselves as experts. For example, we have included a high-school student in a project that otherwise addressed experienced professionals (Varho and Tapio 2013), because we believed that she would have valuable experiences and viewpoints to share from the perspective of today’s youth. This was considered valid, in particular, because the timeline of the futures project extended several decades into the future.

Collins (2013) has discussed three dimensions of expertise, namely “esotericity”, “accomplishment”, and “exposure to tacit knowledge of a domain”. Being a teenager is hardly expertise according to the first two dimensions, but to some extent it does fulfil the third dimension. Being immersed in a subculture gives a person the ability to see and interpret the world in ways that are not obvious to others.

  1. Is an expertise matrix useful?

An expertise matrix or another equivalent tool is important for finding appropriate experts for a futures project. In addition, participating experts should be described to increase the transparency and internal validity of the project (Kuusi et al. 2015), even when anonymity is maintained.

 

Authors: Vilja Varho        vilja.varho@luke.fi; vilja.varho@fidea.fi
Sponsors: n.a.
Type: Methodological discussion
Organizer: Natural Resources Institute Finland (Luke) www.luke.fi
Duration: n.a.
Budget: n.a.
Time Horizon: n.a.
Date of Brief: October 2015

Download EFP Brief No. 258: Perspectives on Use of Expertise

Sources and References

This brief is based on the following article, in which the findings are discussed in more detail:

Varho, V., Huutoniemi, K. 2014. Envisioning solutions – Expert deliberation on environmental futures. In: Huutoniemi, K., Tapio, P. (eds.) Transdisciplinary Sustainability Studies: A Heuristic Approach. Routledge, London & New York. pp. 140-157.

References

 

Amara, R. (1981) ‘The futures field. Searching for definitions and boundaries’, The Futurist, 15(1): 25–29

Bogner, A. and Menz, W. (2009) ‘The theory-generating expert interview: epistemological interest, forms of knowledge, interaction’, in: Bogner, A., Littig, B. and Menz, W. (eds.) Interviewing Experts. Palgrave Macmillan, Houndmills, UK. pp. 43–80.

Dreyfus, H.L. and Dreyfus, S.E. (2005) ‘Expertise in real world contexts’, Organization Studies 26(5): 779–792

Kuusi, O., Kinnunen, J., Ryynänen, O.-P., Myllykangas, M. and Lammintakanen, J. (2006) ‘Suomen Terveydenhuollon tulevaisuudet’, in: Terveydenhuollon tulevaisuus, Eduskunnan kanslian julkaisu 3/2006.

Kuusi, O., Cuhls, K. and Steinmüller, K. (2015) Quality Criteria for Scientific Futures Research. Futura 1/2015: 60-77.

Rittel, H.W.J. and Webber, M.M. (1973) ‘Dilemmas in a general theory of planning’, Policy Sciences, 4(2): 155–69.

Selinger, E.M. and Crease, R.P. (2002) ‘Dreyfus on expertise: The limits of phenomenological analysis’, Continental Philosophy Review 35: 245–279.

Tuominen, A., Tapio, P., Varho, V., Järvi, T. and Banister, D. 2014. Pluralistic backcasting: Integrating multiple visions with policy packages for transport climate policy. Futures, 60: 41-58.

Turner, S. (2001) ‘What is the Problem with Experts?’ Social Studies of Science 31(1): 123–149.

Varho, V. and Tapio, P. (2013) ‘Combining the qualitative and quantitative with the Q2 scenario technique – the case of transport and climate’, Technological Forecasting & Social Change 80(4): 611–630.

Varho, V., Rikkonen, P., Rasi, S. (forthc.) Futures of distributed small-scale renewable energy in Finland – A Delphi study of the opportunities and obstacles up to 2025. Under review in Technological Forecasting & Social Change.

EPF Brief No. 242: Quality and Leadership for Romanian Higher Education

Friday, December 21st, 2012

The project “Quality and Leadership for Romanian Higher Education” (QLHE) aimed to elaborate a vision of Romanian higher education in 2025 and a strategy consisting of specific policy guidelines to achieve it. Based on a large participatory foresight exercise, the project sought to contribute to improving the strategic management of universities and achieving a wide national consensus on the development of the Romanian higher education system.

Transforming the Higher Education System

The project was to help transform the framework of Romanian higher education, as it has been repeatedly stated that the system lacks a vision and long-term strategy. The Presidential Commission on Education issued a report claiming that “education in Romania is ineffective, irrelevant, and low in quality”. The whole reform process has been incoherent, ineffective and has lacked a long-term, shared vision of the future. Therefore, the education system was in urgent need of change. The transformation had to be endorsed by the academic community, policymakers, stakeholders and public opinion. In order to achieve broad consensus, the project carried out a foresight exercise – a large participatory exercise involving a substantial number of people from various target groups and a wide range of ideas, possible future scenarios, solutions, policy options etc.

The higher education system has been repeatedly evaluated as homogeneous, lacking diversity, outdated and out of tune with the realities of the dynamic and interconnected world around it. Prior to developing and achieving the final results, the project carried out activities to analyse the context and identify the major challenges and drivers of change in order to generate a clear and encompassing view of the environment, its needs, the existing obstacles and the potential opportunities. Panels of experts elaborated a series of studies concerning the analysis of the current state of Romania’s universities in relation to various aspects of society, the existing challenges, and the drivers of change in light of the main features of the Romanian social system. The resulting documents served as a point of reference for the subsequent activities.

Creating a Shared Vision

The goals of the project were to create a shared vision and a set of strategic recommendations for Romanian higher education and, in doing so, to develop the prospective analysis and leadership capacities of key actors through a series of workshops and training sessions on various topics of interest.

Another challenging objective was to develop and sustain a foresight community by creating an environment that would enable the emerging community to interact and exchange opinions. Thus, the project designed a web-based collaborative platform, The Foresight Wiki. The name indicates that the platform uses the wiki technology for developing collaborative websites and Web 2.0 technologies. This allows members of the future studies and foresight communities to write articles that any other member can edit. The platform represents an innovative tool providing a user-friendly interactive setting.

Bucharest Dialogues

The platform was not the only step to advance the development of the foresight community; a series of ten international debates, the Bucharest Dialogues, provided the platform with information and knowledge and gave the participants the opportunity to gain experience in the foresight process. These mutual learning workshops were designed as variations on the Bohm dialogues where experts can get together and discuss fundamental aspects of foresight. The Bucharest Dialogues invited foresight practitioners, managers and policymakers in a setup following David Bohm’s principles of dialogue. During a Bucharest Dialogue, key speakers would represent distinct voices within the foresight community, speaking on a broad, preestablished topic.

Mutual Learning Workshops

Both the Mutual Learning Workshops and the Bucharest Dialogues offered a great opportunity for knowledge, skills transfer and learning by allowing the Romanian experts to closely collaborate with more than one hundred international experts. Among the international experts that participated in the Romanian foresight exercise were representatives of institutions such as Fraunhofer ISI, The Institute for Prospective Technological Studies (IPTS), European Universities Association or UNESCO-CEPES (the European Centre for Higher Education), which acted as partner institutions, different international institutions, such as SAMI Consulting, UNIDO, and well-known individual experts, such as Murray Turoff, Roxanne Hiltz, Riel Miller, Peter Bishop, Ozcan Saritas, Denis Loveridge, Ziauddin Sardar, Wendy Schulz and others (for a full list of participants, see the ForWiki platform).

Large-scale Participative Approach

The context and the challenges addressed by this project and the objectives pursued were suited for a large-scale, participative, systemic foresight exercise. As mentioned above, such an approach was necessary since the lack of a systemic approach to change in higher education has not only generated a mélange of reforms but, more importantly, has also resulted in the absence of a clear vision of the future bearings of Romanian higher education.

The exercise started with a nomination/co-nomination process to identify the key stakeholders. It went on to combine panel work, workshops and online interaction. All these activities involved hundreds of participants who provided knowledge, feedback and recommendations during every step of the project.

A series of workshops and trainings were organised for the stakeholders. They focused on various topics of interest, such as foresight and strategic planning, public policy elaboration in higher education, public policy analysis, introduction to the Delphi method or critical thinking and helped to develop skills and abilities so that the whole transformation would actually occur from within the system and would represent a sustainable process, accepted and widely supported by the stakeholders. All these events were chaired by outstanding international experts.

The whole process highlighted interactivity and focused on sharing experience and new knowledge in an international context. One of the key features of the process was empowering stakeholders to contribute to a shared vision. There were two International Advisory Board meetings, international surveys, and various workshops and trainings facilitated by foresight experts. All the outputs were widely disseminated and constantly tested beyond the initial groups with the help of focus groups and a number of online surveys. At the same time, all results were presented to all participants and stakeholders in an appealing way, using films, attractive websites and platforms.

Following a bottom-up approach, the process started with expert panel analyses, which served as a starting point for the creation of four success scenarios on Romanian higher education in 2025. They were used as frameworks for the transformation of the system and expressed the most relevant and desired changes: University of Life and Jobs, Knowledge Constellation, Atheneum and Blue Ocean.

The scenario building was a vast process that combined three renowned and thoroughly tested methods: World Café, Cards and Integral Matrix Analysis. The scenario workshop was designed as a collaborative process in which the members of the expert panels and the invited stakeholders worked in a World Café setting with more than 70 participants. The participants and stakeholders “played” with the main concepts provided by the previously elaborated documents. They used cards and extracted
the most creative ideas. The goal was to outline a final vision for the higher education system, which was tested and altered in order to meet the requirements and desires of the community.

Elphi Platform

The project was innovative not only in carrying out the first foresight exercise on higher education in Romania but also in creating an adapted version of a Delphi questionnaire tailored to the needs of the Romanian higher education context. The questionnaire was provided on the online platform Elphi, which gave the stakeholders the opportunity to actively participate and in the shaping of the Romanian higher education strategy. A large number of respondents from academic, business, social
and policymaking environments participated. They analysed a series of policy proposals that had previously been drafted by nine different panels of experts in relevant areas. Experts were invited to provide arguments and dynamic rankings; their feedback was essential to improving the initial proposals in order to yield the most desirable policy proposals, adapted to the realities of Romanian higher education, while at the same time being future-oriented and bold enough to spur transformation.
The online platform was innovative in introducing a system of dynamically ranking arguments, providing respondents with an opportunity to refine their views and reach a final consensus. The involvement of a large number of experts also legitimised the recommended policies. Later on, these policies formed the core of the White Paper on Romanian Higher Education in 2015, the strategy document whose recommendations charted the first steps to be made towards the 2025 vision.

Measures of Change

The White Paper was to support the vision by suggesting concrete measures and policy proposals for change, designed for the medium term (2011-2015) and for immediate implementation. The first step in formulating the White Paper was to elaborate a series of policies that were tested and initially integrated into a Green Paper on Romanian Higher Education in 2015 by a group of experts – an intermediate step in developing the White Paper. The Green Paper proposed an approach in waves, in which the interest expressed by individual universities constituted the premise of transformations. According to this proposal, the process of transformation should be supported by financial assistance throughout a transition phase and strongly oriented towards autonomy, leadership and responsibility. Romanian higher education is currently perceived as an administrative service, with the state having the right to intervene in the universities’ internal affairs. Thus, university autonomy is weak and subject to administrative, fiscal and financial restrictions. As a potentially significant opportunity, participating universities should be offered the option to change their legal status. Universities must maintain their public interest status, but, at the same time, enjoy economic and fiscal freedoms specific to educational and research services.

The Green Paper was a consultative document; a large online consultation was opened around the key statements, and several university rectors and vice-rectors were interviewed. The integration of the opinions and comments expressed during this process by over 300 respondents supported the development of the White Paper.

Personalisation, Diversification, and Transparency as New Values

The vision and the White Paper were the products of a broad and complex process whose first stages were described in the sections above. Reflecting the success scenario elaborated by stakeholders, the 2025 vision document describes a future of Romanian higher education based on the values of personalisation, diversification and transparency. In short, the three principles describe the desired changes the system should undergo. Personalisation means more options for students in terms of flexible educational pathways that can be fit to their individual plans for the future. Diversity means institutional structures and a systemic configuration that allow for distinct trajectories for institutions with different missions and goals. Transparency highlights the importance of comprehensive, relevant and easily accessible information about the education system while working towards a reputation system for universities.

Innovative Aspects

In Romania, using the foresight methodology to build a vision of the higher education system and develop strategic recommendations (White Paper) represented an innovative approach. The Romanian higher education foresight exercise was the second national foresight process in this country. Such a toolkit had never been used in higher education before and, as such, it represented a major challenge to the team implementing it.

The foresight exercise was the preferred methodology because the project strove to go beyond the limits of common expertise and the traditional policymaking process in Romania, which had led to inconsistent higher education strategies. Moreover, the need for a systemic approach was implicit in the complexity of an education system that engages a variety of actors and their relationships and eventually influences the life of every citizen. Another innovative aspect was the use and adaptation of the online roundless Delphi, which was adjusted to the specific needs of the project and led to the creation of the Elphi platform.

Reform Approaches Find Society’s Consent

The process and the results were designed to raise awareness about the fact that the Romanian higher education system needs to be changed and that Romanian society supports this transformation. By participating in the process, a variety of actors and stakeholders legitimised the vision document and the strategy-setting White Paper. These two documents, together with the
workshops, training sessions, dialogues and debates organised throughout the three years of the project, set out an appropriate framework for the transformation of higher education. They supported a long-term vision designed to draw the picture of a desirable future, generate and stimulate forward-looking thinking about future challenges, provide the basis for decision-making in the present, and mobilise individual and collective action.

Although these ideas, solutions and policies were embraced by the key actors and stakeholders in the education system, the actual transformation of course requires more than visionary documents or the will of the actors involved. While, to date, there has been no official commitment to carry through with the proposed changes in law, a number of follow-up projects are currently empowering the universities in accordance with the principles set out in the vision (improving the system’s transparency, encouraging the collaboration of universities, and capacity-building for differentiation).

Download EPF Brief No. 242_Quality and Leadership for Romanian Higher Education.

 

Sources and References

Andreescu, L., Curaj, A., Gheorghiu, R. (2011): Unleashing individualization. Challenges for Personalization in Tertiary Education, Proceedings of the 7th International Conference on the Management of Technological Changes, ed. C.

Rusu, Greece, Alexandroupoli: Democritus University of Thrace.

Andreescu, L., Gheorghiu, R., Proteasa, V., Curaj, A. (2012): Institutional Diversification and Homogeneity in Romanian Higher Education: The Larger Picture, in Curaj, A. et al. (eds.): European Higher Education at the Crossroads, Dordrecht, Heidelberg, New York, London: Springer, pp. 863-885

Andreescu, L., Gheorghiu, R., Zulean, M., Curaj, A. (2012): Systemic Foresight for Romanian Higher Education, in Curaj, A. et al. (eds.): European Higher Education at the Crossroads, Dordrecht, Heidelberg, New York, London: Springer, pp. 995-1017

Andreescu, L., Gheorghiu, R., Zulean, M., Curaj, A. (2012): Understanding Normative Foresight Outcomes: Scenario
Development and the ‘Veil of Ignorance’ Effect, Technological Forecasting and Social Change, available online 26 October
2012 ISSN 0040-1625, http://dx.doi.org/10.1016/j.techfore.2012.09.013. (http://www.sciencedirect.com/science/article/pii/S0040162512002399)

www.edu2025.ro, last accessed 17 September 2012.

www.forwiki.eu, last accessed 17 September 2012.