S08 - Edler Et Al, 2012 - Marc
S08 - Edler Et Al, 2012 - Marc
S08 - Edler Et Al, 2012 - Marc
3347
Advance Access published on 15 February 2012
doi:10.1093/reseval/rvr002
Evaluation of research and innovation policy faces radical challenges arising from a new policy
emphasis upon demand-side measures and linked to this an understanding of innovation policy as
a means to achieve societal goals. This article considers the implications for the practice of
evaluation at both micro and meso-levels. It uses the exemplar of an evaluation design for the
European Unions Lead Market Initiative to expose the extent to which classical approaches to
evaluation are valid and where new issues arise. Some problems highlighted include the difficulty
of establishing a relevant baseline, the inability of public statistics constructed in supply-side
mode to capture actions, the need to engage with actors who do not necessarily see themselves
as part of the initiative being evaluated, long timescales and potential wide geographical scope,
measures that span from micro to macro, and blurred boundaries between implementation and
impact. It is concluded that there is a key role for evaluators to become involved in co-learning
and co-evolution of these policy instruments in a manner analogous to the relationship
between evaluation and policy development that characterized the emergence of collaborative
R&D support programmes.
Keywords: innovation policy; demand-side; lead markets; evaluation design.
1. Introduction
The evaluation of research and innovation policy faces
radical challenges within many OECD countries appears
to be on the brink of two major, interrelated changes: (1) a
turn towards more demand-oriented innovation policy and
(2) an understanding of innovation policy as an instrument
to achieve societal goals. For several decades, innovation
policy across Europe, and beyond, has largely been dened
and implemented as supply-oriented policy, supporting
capacities to produce new knowledge and innovation. In
recent years, however, a demand-based approach to innovation policy has experienced a fresh start (Wilkinson et al.,
2005; Aho et al., 2006; Georghiou 2007; Lember et al.,
2007; Myoken 2010; Edler 2010; Edler 2011; OECD
2011).1 Those demand-based approaches aim to identify,
create, and support demand for innovation. The instruments of demand-based policies are broad and include
public procurement of innovation, direct or indirect nancial support for the purchase of innovations, various kinds
of training, and awareness mechanisms to build up and
broaden absorptive capacity for innovation and the
The Author 2012. Published by Oxford University Press. All rights reserved. For Permissions, please email: [email protected]
34
J. Edler et al.
system failures;
. targeting innovations towards societal goals and policy
35
36
J. Edler et al.
capabilities that were previously noted. From an evaluation perspective the interesting questions then become,
how (and by how much) did the policy measures alter behaviour and have those new behaviours continued over
time?
At the meso-level evaluation is more likely to be manifested as ex ante appraisal where rationales are established
and realistic goals and targets articulated. Here the evaluation needs to engage with the context of the policy
measure and be situated in a model or theory of systemic
change, for example a realignment of research away from
thematic approaches and towards societal challenges.
From this rationale, in the simpler case of a demand-side
policy being conceived as a programme or initiative with
bounded goals, timescale and budget, it is possible to
follow the recommended evaluation approach of
producing a logic model that sets out the goals in terms
of their wider context and relates these to activities,
outputs, and outcomes. This is illustrated in the example
given in this article but it should be cautioned that a
second category of demand-side policies do not lend themselves so easily to this conceptualization.7 Direct attempts
to change behaviour through guidelines and training
would come in the second group. This is not to say that
these could not or should not be evaluated, only that cause
and effect become harder to relate.
A macro level for evaluation could also be envisaged.
This would seek to relate the broader economic and other
framework conditions that structure demand for innovations to some measure of innovative performance.
Increasingly, indicator studies explore these characteristics
but we shall not go further in this article (EU Com 2010;
Allman 2011).
37
38
J. Edler et al.
support broad areas in which different kinds of innovations can ourish. All areas are linked to wider societal
needs (sustainability, efcient and effective health care etc.)
and for all of them the Commission sees a strong economic
potential within Europe that can be realized through
concerted, coordinated, and exible action (EU COM
2007: p. 4).
For each of the six elds the Commission has dened
concrete Action Plans with a timeline of 35 years. The
action plans design a mix of demand-side policy measures:
. Legislation, i.e. to ensure that all regulation and legis-
Figure 1.
39
The Four Functions of Strategic Intelligence, the Evaluation Flow and Indicator Needs.
40
J. Edler et al.
Figure 2.
intelligence, this need would have to be dened more explicitly with more systematic consultation, extending to the
scan of potential corresponding global needs that promise
export potential.
For the implementation of the measure and its monitoring and evaluation, the needs and areas identied need to
be matched with clearly dened markets so that the market
developments can be traced and target actors and institutional framework conditions can be identied. The delineation of those markets and the tracing of development in
the markets can be approached in two ways. More traditionally, existing indicators and proxies for denition of
the six elds selected for the LMI can be derived according
to the corresponding industrial sectors or product classications. However, the six areas chosen do not easily t to
the given product classes or sector classications. A
second, often more suitable option for the denition of
the area to target is through dening the demanders and
other actors associated with the underlying societal need,
as lead markets are about innovations to satisfy certain
societal needs. In addition, the technological or procedural
solutions may be developed by actors that are located
in other market segments than those we would expect
based on traditional, backward looking classications.
Therefore, the traditional approach needs to be open to
dynamic changes. This means, as a principle, across all the
six markets, an interview programme and a survey
41
42
J. Edler et al.
Table 1.
Procurement
Standards
Legislation
Complementary actions
In addition to the market development impact based on the indicator and survey proposed:
Has the market become more innovation and competition driven, e.g. do we see more rm entries, more variety of
products, greater product differentiation etc.?
Have the actions taken inuenced specic context conditions of the defence area (defence procurement, established
networks, traditionally more nationally closed markets).
Are procurers across Europe more aware of benet in process innovation?
Are procurers more aware of the functionalities of innovative personal protective equipment (PPE)
Have tender texts changed signicantly (functional specication, value for money, variants, link to leading edge standards
as minimum requirements etc.)
Is there increased and upgraded procurement activity also in services related to PPE?
Have similar public services across countries coordinated and bundled their demand
Are small and medium sized companies among the winners, any increase in their share? Do we see changes in SME
involvement in supply chains?
Has the share of leading edge protective textile products in public procurement increased signicantly? Are the products
purchased increasingly leading edge?
Are EU standards becoming international standards?
Have voluntary certications become more common, and are users better aware?
Are the European regulations being accepted outside the EU? Have producers in Europe made use of improved design
registration (link to WIPO)?
Is there greater awareness as for need for PPE, greater willingness to pay leading edge premium across Europe?
What share of stakeholders is better informed about opportunities, both supply and demand side? Through what
channels?
Does RTD funding system provide enough support for next generation?
Have nancing opportunities been improved, has there been a piloting of innovation platforms?
Figure 3.
43
44
J. Edler et al.
Notes
1. The OECD Working Group on Science, Technology
and Innovation Policy has started intensive activities
2.
3.
4.
5.
6.
7.
8.
9.
to support demand-based innovation policies; the discussions in this group have demonstrated that many
countries have turned or plan to turn towards
demand-based policies (OECD 2011).
Evaluations of energy efciency programmes employing demand-side instruments can be found at
http://iepec.org; or www1.eere.energy.gov/ba/pba/
program_evaluation/evaluation_documents.html
In the USA, examples of indicators and measurement
exist, though, for example within the Federal Energy
Management Program (FEMP) there is a requirement
to refer to lists of energy efcient products (and thus in
principle products that are innovative or at least at an
early stage of their life cycle) in tender processes and
the compliance of procurers to that requirement can
be monitored http://www.energystar.gov/index.cfm?
c=fed_agencies.fed_ag_index.
Another early example has been the US EnergyRelated Inventions Program (Brown et al., 1994).
The energy efciency programmes in the USA,
referred to in Note 2, are prime examples of those
competence and awareness enhancing programme
(see http://iepec.org).
One means to overcome those asymmetries is constructive technology assessment, a process in which
societal actors not directly involved in the production
of a technology are included in the technology discourse at an early stage.
Horowitz 2010 elaborates on logic models in the
energy efciency diffusion debate that focuses on the
nal impact of the policy, energy saving, rather than
the innovation effect in terms of diffusion of energy
efcient products.
For the analysis of impact, such a procurement
analysis would be desirable on a global scale as well;
however, we lack a database beyond Europe comparable with TED.
The actual evaluation of the LMI as conducted illustrates this challenge: it appliedwith some
changesthe quantitative indicators suggested and
followed the major dimensions of the interview programme. In correspondence with what this article
argued, the evaluation report illustrates a set of
methodological challenges. Most importantly, it
tries to measure all sorts of innovation and market
effects quantitatively that one would expect to take
much more time to realize as a consequence of the
complex LMI. This is exacerbated by the fact that a
clear baseline for markets and actor behaviour was
not established sufciently, as existing databases do
not match what is dened as a lead market and
more generallyfail to monitor change in market
delineation as a consequence of innovations and
new suppliers entering the market (SEES/Oxford
Research 2011: p. 28).
45
References
Aho, E. et al. (2006) Creating an Innovative Europe, Report of
the Independent Expert Group on R&D and Innovation appointed following the Hampton Court Summit. Luke
Georghiou, Rapporteur.
Allman, K. et al. (2011) Measuring wider framework conditions
for successful innovation, NESTA report, January 2011.
Arnold, E. (2004) Evaluation Research and Innovation Policy:
A Systems World Needs Systems Evaluations, Research
Evaluation, 13/1: 317.
Barre, R. (2009) ERA indicators and monitoring, Expert Group
Report, Brussels <ftp://ftp.cordis.europa.eu/pub/era/docs/
era_indicators&monitoring.pdf> accessed September 2011.
Bartlett, C. A. and Ghoshal, S. (1990) Managing innovation in
the transnational corporation. In: Bartlett, C. A., Doz, Y. L.
and Hedlund, G. (eds) Managing the Global Firm,
pp. 215255. London/New York: Routledge.
Beise, M. (2004) Lead markets: Country-Specic Drivers of the
Global Diffusion of Innovations, Research Policy, 33/67:
9971018.
Beise, M. and Cleff., T. (2004) Assessing the Lead Market
Potential of Countries for Innovation Projects, Journal of
International Management, 10/4: 453477.
Blind, K. (2009) Standardisation: a catalyst for innovation,
Inaugural Address delivered at the occasion of accepting
the
appointment
as
extraordinary
Professor
of
Standardisation on behalf of NEN, the Netherlands
Standardisation Institute, RSM, Rotterdam.
. (2010) The Use of the Regulatory Framework to
Innovation Policy. In: Smits, R., Shapira, P. and
Kuhlmann, S. (eds) The Theory and Practice of Innovation
Policy An International Research Handbook, pp. 217246.
Cheltenham: Edward Elgar.
. (2012) The Inuence of Regulations on Innovation: A
Quantitative Assessment for OECD Countries, Research
Policy, 41/12: 391400.
Blind, K., Gauch, S. and Hawkins, R. (2010) How
Stakeholders Assess the Impacts of ICT Standards,
Telecommunications Policy, 34/3: 162174.
Blind, K., Gauch, S., Hawkins, R. and Jungmittag, A. (2005)
Trade and the Impact of Innovations and Standards: The
Case of Germany and the UK, Applied Economics, 37:
13851398.
Blind, K. and Jungmittag, A. (2008) The Impact of Patents and
Standards on Macroeconomic Growth: A Panel Approach
Covering Four Countries and Twelve Sectors, Journal of
Productivity Analysis, 29/1: 5160.
Brown, M. A., Wilson, C.R. and Franchuk, C.A. (1994) The
economic, energy and environmental impact of the
energy-related inventions program, Prepared for the Ofce
of Energy Efciency and Renewable Energy U.S.
Department of Energy, Washington <http://www.osti.gov/
accomplishments/documents/fullText/ACC0016.pdf>
accessed September 2011.
Edler, J. (2007) Bedurfnisse als Innovationsmotor. Berlin/
Karlsruhe: Sigma.
. (2010) Demand Oriented Innovation Policy.
In: Smits, R., Kuhlmann, S. and Shapira, P. (eds) The
Theory and Practice of Innovation Policy An International
Research Handbook, pp. 275302. Cheltenham: Edward
Elgar.
. (2011) Innovation in EU CEE: the Role of DemandBased Policy. In: Radosevic, S. and Kaderabkova, A. (eds)
Challenges for European Innovation Policy. Cohesion and
Excellence from a Schumpeterian Perspective, pp. 177208.
Cheltenham/Northhampton: Edward Elgar.
46
J. Edler et al.
47