Posts Tagged ‘accompanying research’

EFP Brief No. 234: Learning Effects of a Foresight Exercise: An Accompanying Social Research Study

Friday, December 21st, 2012

The purpose of the accompanying social research study to the Freightvision exercise (Brief No. 226) was twofold: First, we wanted to introduce a concept for accompanying social research of a large participatory foresight process in order to grasp immediate learning effects. Secondly, we analysed immediate learning effects in the course of a large participatory foresight process. The research questions guiding the empirical analysis were: How can we operationalise and measure learning in the context of a large foresight process? Learning thereby involves different levels of learning: individual learning, group learning, organisational learning, system-level learning etc. And how can we operationalise and measure networking, i.e. the establishment of personal ties that enable the exchange of information and hence learning in a large foresight process?

The Foresight Case Freightvision in Focus

The foresight case in focus intended to integrate new knowledge, perspectives and stakeholder groups into an established field. Creating channels for communication between participants from business, policy, civil society and R&D to overcome sectoral boundaries was an explicit goal from the beginning. Stakeholder participation in this case was defined as inviting representatives of research, business, policy and civil society explicitly as “experts” who take part in a strategic dialogue on long-term issues. The expertise of participants was sought as deliberative input and shaped the content and tangible results of the foresight process, leading to robust scenarios, recommended action plans, visions and background reports.
Given the large scale of the foresight exercise (up to 90 participants in four fora, budget >3 m EUR, duration > 3 years), deliberative participation was guaranteed through four large and highly interactive fora using large group intervention techniques derived from organisational development theory (world café, open chair discussion rounds, interactive poster sessions etc.). Methodologically, the Freightvision foresight assessed here relied on an overall architecture and methods of organisational development (OD) that focus particularly on changing the thinking and actions of stakeholders. The application of OD concepts and instruments throughout all phases of the foresight exercise was assumed to maximise interaction, collaboration and learning among stakeholders in this foresight system.

Methodologies of the Accompanying Social Research

Learning effects of foresight processes can occur in various dimensions, which we tried to capture in our accompanying social research study: i) the acquisition of social capital (e.g., establishing new contacts, building networks), ii) the acquisition of factual knowledge and understanding (new insights derived from discussions and multiple perspectives), and iii) the development of strategic alternatives (Amantidou & Guy, 2008). Following Lewin (1953), Schein (1995) and Grossman et. al. (2007), we distinguished and applied three different approaches of accompanying social research to analyse and assess the immediate learning effects of foresight. The three approaches were the practitioner model of field research, qualitative interviewing and content analysis.

Practitioner Model of Field Research

The accompanying research to evaluate the effects of the foresight process on participants and stakeholders was conducted by AIT – Foresight and Policy Development Department. The process involved 165 individual participants coming from private enterprises, interest groups representing the various transport modes, infrastructure providers, trade unions, environmental NGOs, research organisations and administration. Participation in Forum 1 to 4 was between 96 and 75 individuals.
In moderated workshops, we conducted a survey and several discussions as part of the foresight process. Within this foresight project group, organisational development (OD) researchers acted as counsellors trying to intervene in social systems in order to provoke change (Schein, 1995; Grossman, et al., 2007). In the context of these moderated workshops, the foresight counsellors and the foresight project group evaluated their roles during the stakeholder fora as well as other impacts by (1) reflecting on and adapting their own observations and patterns of intervention, (2) by evaluating the process as a whole and (3) by carrying out a qualitative survey of the project group in the moderated workshops after each stakeholder forum. The questions addressed mutual learning processes, short-term effects and the evaluation of the overall design and process of the stakeholder fora.

Participatory Ethnographic Research

According to Schein (1995) and Grossman et al. (2007), researchers participate in the day-to-day life of social systems yet try to minimise influence or set interventions. To capture various kinds of immediate impacts from the foresight case, telephone interviews were carried out after each of forum. Around 20% of the participants were interviewed by the research team, resulting in 71 interviews all in all (the interviews took 15-20 minutes each). Qualitative content analysis was applied to extract information from the interviews. The post-forum telephone interviews showed that participants were positive about the methodology. They were particularly positive about the high levels of interaction during the fora (working intensively in a productive atmosphere, using creative methods including wild cards and visualisation of the freight transport system in 2050), which helped the different stakeholder groups to better understand the motivations and backgrounds of various other stakeholder groups. The interviewees also mentioned that the project led to a systemic picture of the whole longdistance freight transport system across modes.

Experimental Social Research

In experimental social research, the observer implements a lab-like environment trying to minimise influence on the observed object. The research setting is designed to generate quantitative data that claims to describe “the reality of the observed object” apart from the observing researcher. In our case, a social network analysis (SNA) approach was applied, reducing the observed part of the complex communication and learning process to different categories of ties established between participants of the foresight fora. Assuming that actors are embedded in a web of social interrelations, SNA provides a set of methodologies and tools to understand internal communication, organisation and aspects of their formation (Heimeriks, Hörlesberger, & Besselaar, 2003; Coromina, Guia, Coenders, & Ferligoj, 2008). A questionnaire was designed and distributed both at the beginning and end of every forum, listing names of participants and asking each participant to quantify the level of acquaintance with all remaining ones. The difference in levels of acquaintance before and after every forum served as a proxy for the number and quality of ties established during the fora (qualitative and quantitative statistical network analysis was applied in order to extract information from the questionnaires).
The team of researchers conducting the accompanying social research were external observers. The network analysis based on pre- and post-forum questionnaires showed that the network of participants had already reached a high density after Forum II and that there were no signs of emerging closed clusters of unconnected sub-groups. New participants were integrated quickly (approximately one quarter were new in every forum), and the network density remained stable until Forum 4. Figure 1 shows a network of personal ties (or relationships) between participants based on personalised questionnaires returned a) before Forum I (March 2009, n = 41/96 questionnaires) and b) after Forum III (October 2009, n = 35/79 questionnaires). Stakeholders are coloured in black, all other project partners in grey. Geometric positions and distances are determined by the combined strength of a participant’s ties (participants are positioned closer if ties are stronger). The shape of a node is determined by the number of inward vs. outward ties and its volume by the total number of ties. Network “connectors” have more outgoing vs. incoming ties (ellipses pointed upward) and “authorities” vice versa (ellipses pointed sideward). All computations were performed using the software PAJEK (Chen, 2003).

Learning Effects

The immediate learning effect of a large-scale foresight project was analysed based on three methods of accompanying social research. First, the practitioner model was applied in an analysis of the foresight process in moderated workshops. Learning in this context mainly referred to the creation of cultural islands and increased the participants’ identification with the foresight process. Secondly, a qualitative analysis was conducted in an ex-postfacto analysis where individual learning resulting from the
foresight process in focus was captured in different questions.
The main result here is that the major achievement of a large participative foresight process with respect to learning is probably that details out of the social contexts and rationalities of various stakeholders add up to a multidimensional picture at the system level. This results in perceiving oneself as being part of a system and gives a clearer view of one’s own role in the system. Interdependencies between the various actors become more apparent, which on the whole results in a more comprehensive big picture at the system level. Thirdly, we tried to empirically grasp the increase of personal ties between participants of a large foresight process by means of a social network analysis. We assumed that these ties reflect some extent of exchange of information and hence can be expected to enable learning processes. Overall, the number of newly formed acquaintances more than tripled during the fora; the network diameter settled at a low size of three ties. A higher density, an average degree of centrality and a lower diameter reflect a higher flow of information. It becomes clearer how participants perceive their position within the network of stakeholders and their influence and future agendas (Schartinger et al., 2011).

Effects at European and National Level

A clearly discernible effect is the continued collaboration of the project team in the following FP7 calls, which can be attributed to the well-designed collaboration in the project team as active participants in the fora. In addition, the project team held briefing and debriefing sessions before and after the fora to discuss and optimise the networking process. Less can be said about the direct effect of the foresight in terms of relevance to policy documents, as the accompanying research ended shortly after the Freightvision project.

In Austria, the results were presented up to the highest ranks of the ministry of transport, which led to the ministry funding a follow-up project (Freightvision Austria, see EFP Brief No. 231) at the national level through the Transport Research Program IV2plus. Media coverage both at the sectoral level (some was very offensive even criticising the scientific evidence) the national level gives some indication of the relevance of the Freightvision process. After the final dissemination conference, DG TREN (MOVE) ordered extra copies of the last management summary for distribution throughout the directorate, which can be seen as a sign of the project’s relevance to internal discussion. In 2012, we conducted some additional interviews to find out whether Freightvision had any direct influence on the White Paper on Transport published in 2011.

Although some affirmative statements were made, it is not possible to verify such an influence. The Commission Staff Working Document on the White Paper shows no reference to Freightvision or other parallel FP7 Support Actions. However, several of the 36 measures from the project are mentioned in this document (e.g., CO2 labelling and integration into standards, e-freight, ecodriving training, liberalisation of cabotage, IST, ERTMCS/ETCS etc.).

Further Need for Follow-up Research

A further step in research on the effects of foresight would be to analyse in depth how participants of a foresight process deal with what they have learnt during the foresight process once they return to their usual surroundings and home environments. In principle, large participatory foresight processes induce participants to carry new impulses to their home organisations. Strategic dialogues and mutual learning processes during the foresight exercise can provide guidance in situations with high degrees of unpredictability and become effective in the organisations the participants originate from.

However, it is a great challenge to methodologically grasp the different kinds of effects over time and to isolate the contribution of foresight processes to complex and continuous processes like strategy finding and policy formulation. Determining the contribution of foresight exercises will always be achieved only in part.

Highly Controversial Stakeholder Responses

Although the process was built on a well-founded evidence base, including several models that are also cited in the recent White Paper, it was foreseeable that controversial positions would emerge in the normative phase of the foresight. For reasons of transparency, an effort was made to make dissent explicit and to document minority positions in working groups. Although it was clear that the project, financed through a FP7 support action, was no formal stakeholder consultation process in preparation of the White Paper, lobbying occurred to the extent that some participants at the final conference were on the verge of boycotting the event because of unfavourable conclusions for a specific interest group. Due to the explicit backing by many of the forum participants who attended the dissemination conference, it became clear that the overall results were valid and that the foresight process had been transparent and sound.

Sources and References

Amanatidou, E. and Guy, K. (2008), “Interpreting foresight process impacts: Steps towards the development of a framework conceptualising the dynamics of ‘foresight systems’”, Technological Forecasting and Social Change, Vol. 75, No. 4, pp. 539-557.
Chen, C. (2003), Mapping Scientific Frontiers. The Quest for Knowledge Visualization, Berlin: Springer.
Coromina, L., Guia, J., Coenders, G. and Ferligoj, A. (2008), “Doucentered networks”, Social Networks, Vol. 30, No. 1, pp. 45-59.
European Commission (2011), “Commission Staff Working Document – Accompanying the White Paper – Roadmap to a Single European Transport Area – Towards a competitive and resource efficient transport system”, SEC(2011) 391 final.
Grossmann, R., Lobnig, H. and Scala, K (2007), Kooperationen im Public Management. Theorie und Praxis erfolgreicher Organisationsentwicklung in Leistungsverbünden, Netzwerken und Fusionen, Munich: Juventa Verlag.
Heimeriks, G., H., Hörlesberger, M. and Besselaar, P. van den (2003), “Mapping communication and collaboration in heterogeneous research networks“, Scientometrics, Vol. 58, No. 2, pp. 391-413.
Lewin, K. (1953), Die Lösung sozialer Konflikte. Ausgewählte Abhandlungen über Gruppendynamik, Bad Nauheim: Christian-Verlag.
Schartinger, D., D. Wilhelmer, D. Holste, K. Kubeczko (2011), Assessing immediate learning impacts of large foresight processes. Submitted to Foresight Journal.
Schein, E.H. (1995), “Kurt Lewin’s Change Theory in the Field and in the Classroom. Notes toward a Model of Managed Learning”, available at: (accessed 2 December 2012).