Comparative Analysis of Social

and Public Policy

Week 2

Country rankings and league tables

1
 Country Rankings and League Table
:
Overview: Part 1

(1) Country scoring – some preliminaries

(2) Conceptual and methodological aspects of

comparative policy data and indicators

(3) Construction of rankings and league tables

2
 Overview: Part II

(4) Comparisons in the policy world

• Who compares, and in what context?
• Why do policymakers use country comparisons?
• When does comparison matter for policy?

(5) Academic vs policy world approaches to

comparison

3
(1) Country scoring and
ranking – some
preliminaries
• country ‘scoring’ is essentially a descriptive (numerical)
form of cross-national comparison

• it is a crucial precondition for (especially quantitative)

causal policy research

• but it is also beloved of national and international policy

makers: especially scoring as ranking.

• International league table = visual depiction of a country

ranking.
Selected Sources of Comparative
Country Data…
Global
• UN: https://unstats.un.org/unsd/demographic-social/index.cshtml [social and demographic]
• World Bank: https://data.worldbank.org/ [broad array of indicators]
• IMF: https://www.imf.org/en/Data [finance and economy]
• ILO: https://ilostat.ilo.org/ [labour market and employment]

Regions or Groups of Countries

• OECD: https://www.oecd.org/social/statistics.htm [mainly high income countries]

• ADBG: https://www.afdb.org/en/knowledge/statistics [Africa]
• Eurostat: https://ec.europa.eu/eurostat [Europe]
• CEPAL: https://statistics.cepal.org/portal/cepalstat/index.html?lang=en [LA & Caribbean]
• Asian Development Bank: https://www.adb.org/what-we-do/data/main [Asia]

• The UN, World Bank, OECD etc also offer some regional-level data websites.

5
… and indicators
poverty and inequality in rich countries (OECD [see also – Gini Index, World Bank]
https://data.oecd.org/inequality/poverty-rate.htm;

environmental performance index (Yale University): https://epi.yale.edu/

gender equality (e.g. Gender Development Index, UNDP):

https://hdr.undp.org/gender-development-index#/indicies/GDI

health systems (e.g. WHO, Commonwealth Fund)

https://en.wikipedia.org/wiki/Health_system#International_comparisons

employment protection levels; EPL (OECD);

http://www.oecd.org/els/emp/oecdindicatorsofemploymentprotection.htm

‘doing business’ (ease of); World Bank;

https://databank.worldbank.org/source/doing-business

And more! economic growth (GDP); unemployment, government debt, public

spending, social spending; life satisfaction; trust; crime, public sector performance ;
e.g. OECD: http://www.oecd.org/social/society-at-a-glance-19991290.htm 6
Object of country ranking
• Be clear about: what is being scored/ranked?

• A. policy outputs or design (‘settings’)

• e.g. benefit levels, social spending, capital spending, policy ‘mixes’,
institutional arrangements etc.

• B. policy (social) outcomes

• unemployment, poverty, homelessness, birth rates, obesity, mortality etc.

policy makers (in principle) learn most from the relationship of A

and B….but this is also the most difficult to establish
(2) Country rankings: conceptual
and methodological aspects
• How can we be confident that a ranking is valid?
• How can we interpret what a ranking actually tells us –
comparatively – about a policy issue?

• Several issues affect the validity of country rankings and

what these show us:

 concepts: definition and measurement (‘operationalisation’)

 data: validity, comparability, and availability

8
 Youth Unemployment Rates (orange)
and Youth Unemployment to Population
Ratios (blue), 2015
60.0

50.0

40.0

30.0

20.0

10.0

0.0

Source:
Eurostat

YUtoPOP YURATE
Example: conceptual definitions and
measurements

• Comparative interest: measuring sickness absence in public

administration in selected countries.
(Hoffmann & Van Dooren, 2017)

• conceptual issues
• sickness
• public administration
methodological issues (measurement)
• Sickness ratio = total no. of sickness days ÷ no. of annual
reference days x 100. But:
(i) head count or full-time equivalents (FTE)?
(ii) days per year: calendar or working days? 10
 From: Hoffmann & Van Dooren, 2017

Sick days Public sector count Reference days

Country/
Head Calendar
region count (FTE) days
Core PA Broad PA Working days
Austria x x x x
Bavaria x x x
Denmark x only for x minus annual
2010 (paid) leave

Finland x x x
Flanders x x x

Netherlands x not x x
education/
health

11
 Hoffmann & Van Dooren, 2017

Figure 1. Sick leave in days per employee for 2010

16

14

12

10

8 OECD

6 our results

12
 (3) Construction of
rankings & league tables

• Analytically, constructing country ranking and

league tables is a descriptive endeavour

• The choice of indicators will shape the ranking

system and the picture produced.

• As a process: empirical and evidence-based…

 but potentially also subjective and interest-based
13
 construction of rankings &
league tables: choosing
indicators
• Single or composite?

• Single: which to choose?

• Composite: weight of chosen indicators?

• What does chosen methodology imply in policy

terms?
 e.g. framing of policy problems; promotion of preferred types
of policy solutions? 14
Commonwealth Fund Healthcare
System Performance Ranking 2021

• Deliberate focus of methodology on patient perceptions and satisfaction:

“…we carefully selected 71 measures relevant to health care system

performance …[the ranking methodology] is unique in its inclusion of
survey measures designed to reflect the perspectives of patients and
professionals…Nearly three-quarters of the measures come from surveys
designed to elicit the public’s experience of its health system”.
(https://www.commonwealthfund.org/publications/fund-reports/2021/aug/mirror-mirror-2021-reflecting-
15
poorly#how-we-measured-performance)
(4) Country rankings and
policymaking

• Why (and how) do policy makers make use of country

rankings and league tables?

• When (and how) do country rankings influence real-

world policy debate & reform?

See seminar readings for week 1 and 2 for further examples and
insight into these issues.

16
 context: comparing across countries has
become more common in public policy….

Driving factors include:

• globalisation
• digitalisation
• political integration (global and regional)
• similar challenges and common crises (economic,
environmental, health)
e.g. post-2008 financial crisis, Covid-19, AI revolution, climate…
• trends of (intended) policy convergence?
17
Policymakers and
comparisons
• who uses comparisons?
Policymakers and wider (specialist) policy actors

• comparing with where?

(often) from countries which are similar (e.g. ‘families of
nations’, F. Castles ed. 1993) or in the same region.

• with what effects on policy?

 Comparison offers increased opportunities for policy
‘learning’ (or transfer) – but various motives exist and
policy impact depends on several factors & context.

18
 Why do policy makers
compare?

• An aide to policymakers’ processes of ‘puzzling’ and

‘powering’ when deciding what to do (Heclo, 1974):

1. problem-solving
‘puzzling’ (both rational and ideational)

2. legitimation-seeking
‘powering’ (strategic and interest-based)
> the political dimension of comparing
19
Problem-solving:
international league tables
and ‘best practice’

- League tables can be used for benchmarking

performance against others (‘laggards’ and
‘leaders’)

- Policymakers learning from ‘best practice’ and

international evidence.

- Often viewed as non-ideological and non-political:

comparison as a technical/objective procedure to
aid domestic policy making.
20
Problem-solving: rational approach of
comparing (and learning from
abroad)
6. check political
1. what exactly is the problem?desirability
2. which country is best to 7.
compare with and
check practical learn
from? feasibility
3. visit and talk to many people
8. consider existing
4. generalise (de-contextualise)policy programmes
5. adapt and design (re-contextualise)
9. the problem of
prospective evaluation
10. beware of political role
of comparison
Drawn from: Rose, R. (2002) Ten Steps in Learning Lessons from Abroad, EUI RSC, 2002/05
http://hdl.handle.net/1814/176321
 Critiquing the ‘best practice’ model

• suggests particular (best) policy to achieve a particular aim

• implies policy learning (and transfer) is possible and desirable

• ignores wider agenda (strategic policy aims; legitimacy)

• ignores potential conflicts between policy goals and long vs.

short-time effects (e.g. economic and climate policy aims)

• simplifies and disregards context (time; cultural norms etc)

• Rankings themselves shape problem perception (e.g. indicator

for ‘democracy’; public sector deficit)
22
Powering: the political dimension of international league tables

• Are best practice/ rational models reflective of the ‘real world’ of

policy making? (Only ‘puzzling’?)

• What about the interests of policy makers and the political

dimension of comparison? (‘powering’)
 Government/opposition and other policy actors also make strategic use
of comparisons and country rankings.

• League tables may create or provide:

external expertise (i.e. an ‘international consensus’)

external pressure (for competition and convergence)

external legitimation (of domestic agendas) 23

 Country rankings and domestic policy:
linking ‘puzzling’ and ‘powering’ together

• To understand how and when international comparison

influences policy, we (as academics) need theories of policy
reform and change

 e.g. Kingdon’s (1984) multiple streams analysis framework

problem; policy; politics (puzzling and powering combine
together)

• we need to analyse: where do comparisons come in? (e.g. via

case studies)

• other theories of policy change are also available 24

 (5) Comparisons in policy
world and academic
research
• Research/ academic world: comparative analysis:
identifying and explaining variation

• In the policy world – possibly other reasons for

comparisons (problem-solving, external legitimation)

• but may be similar at times (e.g. policy evaluation)

• at other times, political and academic interests can

be complementary (example next week….) 25