m

hi

1
The nature of surveys

The purpose of this chapter is to outline briefly book titled The Survey Method: The Contribution
what is meant by the term survey research. Any of Surveys to Sociological Explanation (1982). The
examination of texts on social research or the focus of the present book is to show what can be
popular media shows that the term survey is used achieved with a good survey and how to achieve
in many different ways. To avoid confusion we this.
must be clear at the beginning about how the term
will be used in this book. The way we define what
a survey is affects the range of topics for which we WHAT IS A SURVEY?
can use a survey, the way in which we conduct a
survey and our evaluation of the merits and short­ A survey is not just a particular technique for
comings of the survey method. collecting information: questionnaires are widely
The primary aim of this book is to provide used but other techniques, such as structured and
guidance on how to do good surveys. Some critics in-depth interviews, observation, content analysis
of the survey method have responded to problems and so forth, can also be used in survey research.
evident in many surveys by urging that we The distinguishing features of surveys are the form
move away from survey research altogether and of the data and the method of analysis.
employ entirely different research methodologies.
However, the solution to criticisms of surveys need
Form of data
not be to abandon survey research but to solve the
problems that the critics assume are inherent to Surveys are characterised by a structured or
the survey approach. systematic set of data which I will call a variable
Many criticisms of surveys are based on misun­ by case data grid. All this means is that we collect
derstandings of what surveys can be and are based information about the same variables or character­
on examples of poor surveys and the inappropri­ istics from at least two (normally far more) cases
ate use of survey research. This is not the place and end up with a data grid. In Table 1.1 each row
to review these criticisms or to argue how they represents a case (person) and each column repre­
are frequently misinformed. The best discussion of sents a variable or information collected about
these criticisms is available in Catherine Marsh’s each case. Since the same information is collected
4 PARTI THE SCOPE OF SURVEY RESEARCH

Table 1.1 A variable by case data grid

Variables

Political
Sex Age orientation Social class

Person 1 male 36 years progressive working

Person 2 male 19 years moderate lower middle
Cases Person 3 female 30 years progressive upper working
Person 4 male 55 years traditionalist upper middle
Person 5 female 42 years traditionalist middle

for each case the cases are directly comparable and or by many other means. There is no necessary
we end up with a structured or ‘rectangular’ set connection between questionnaires and survey
of data. research (see Figure 1.1).
The technique by which we generate data The absence of a necessary connection
about the cases can vary between surveys. We between the survey method and a particular data
might collect information by giving a question­ collection technique is reinforced by the fact that
naire to each person and then copying answers the cases in the variable by case data grid need not
from each questionnaire into the data grid. be people. Technically the case in the data grid
Because questionnaires are highly structured they is called a unit of analysis—it is the ‘object’ about
provide a straightforward way of obtaining infor­ which we are collecting information (see Web
mation for the data grid. However, the data for Pointer 1.1). While this frequently is a person it
the grid could be collected by other means such as need not be. We could construct a data grid in
interviewing or observing each case, by extracting which the unit of analysis was a country, a year or
information from records we have on each person virtually anything so long as we collect attributes

Research question

Case study Survey Experiment

Questionnaire Questionnaire Questionnaire

Interview (structured) Interview (structured) Interview (structured)
In-depth interview In-depth interview In-depth interview
Observation Observation Observation
Content analysis Content analysis Content analysis

Figure 1.1 A range of methods of research and techniques of data collection

 CHAPTER 1 THE NATURE OF SURVEYS 5

WEB POINTER 1.1 Units of analysis

Clear and brief description of the idea of units of analysis. www.socialresearchmethods.net/kb/unitanal.php

of that case (see section on units of analysis in links (simply to demonstrate that two things go
Chapter 3). If countries were the cases, a list of together does not prove a causal link).
countries would be listed down the side of the This style of research and analysis can be
table instead of people, and attributes of countries contrasted with other methods. For example, the
(e.g. population size, area, density, unemploy­ case study method focuses on particular cases and
ment rate) would be listed across the top. If years tries to develop a full and rounded understand­
were the cases, years (e.g. 1980, 1990, 2000, ing of the cases. The case study method does not
2010) would be listed down the side with attrib­ fundamentally rely on comparing cases but on
utes relevant to years across the top (e.g. inflation fully understanding the ‘wholeness’ of a particu­
rate in particular years, divorce rate). lar case and understanding particular attributes of
The variable by case data grid is fundamental a person (or an organisation or whatever the case
for survey analysis which is based on a comparison is) within the context of the case’s other charac­
of cases. It is this method of analysing data which teristics and history. The experimental method
is the second distinguishing feature of surveys. is similar to the survey method in that data are
collected in the form of variable by case data grid,
but is fundamentally different in that the variation
Methods of analysis between the attributes of people is created by inter­
One function of survey analysis is to describe the vention from an experimenter wanting to see if the
characteristics of a set of cases. Thus if we want to intervention creates a difference. For example, the
describe how a group of people will vote, we need experimenter who wants to know whether a drug
to know how each person in that group intends cures a disease would take a group of sufferers and
to vote. A variable by case data grid provides this divide them into two similar groups. The drug
information. would be administered to only one group and
But survey researchers are also interested in then the recovery rates of the drug and non-drug
causes of phenomena. The survey analyst tries to groups would be compared. Here the variation
locate causes by comparing cases. By looking at between the two groups (i.e. drug/non-drug)
how cases vary on some characteristics (e.g. some has been created by the experimenter. A survey
cases will be political progressives and others will approach would not create the variation but would
be traditionalists), the survey analyst will see if the find ‘naturally occurring’ variation; that is, find a
progressives are systematically different from the group of sufferers who did not have the drug and
traditionalists in some additional way. For example, compare them with a group of sufferers who did
in Table 1.1 there is variation between cases in how have the drug.
they vote. This is systematically linked to varia­ The problem for survey researchers is that they
tions in class: the progressives are working class and cannot be sure that the two groups are similar in
the traditionalists are middle class. Survey research other respects, whereas the experimenter begins
seeks an understanding of what may cause some with two similar groups and the only difference (in
phenomenon (e.g. vote) by looking at variation theory) is that only one group receives the treat­
in that variable across cases, and looking for other ment. Therefore any difference in recovery rates
characteristics which are systematically linked must be due to the drug. Apart from the poten­
with it. As such it aims to draw causal inferences tial ethical problems of experimental research (see
(e.g. class affects vote) by a careful comparison Chapter 5), these different approaches to obtaining
of the various characteristics of cases. It does not variation between groups lead to quite different
end there. The next step is to ask why class affects methods of analysis.
vote. Survey researchers need to be very careful, In any particular study a range of research
however, to avoid mistaken attribution of causal approaches can be used. A study of causes of
6 PARTI THE SCOPE OF SURVEY RESEARCH

industrial disputes could involve a survey of atti­ misleading. It is more helpful to distinguish
tudes of management and workers, a case study between two stages of the research process:
of a particular strike or a particular factory and collecting data and analysing data.
an experiment where groups of workers work At the data collection stage it is more useful
under different conditions to see if this affects the to distinguish between research methods that
frequency of disputes. yield structured and unstructured data sets than
The techniques by which data are collected between methods that are quantitative or quali­
using any of these methods can vary considerably. tative. I have already argued that the data grid
In a survey we could observe each case, interview required for survey research can be filled in using
them, give them a questionnaire and so on. a variety of data collection methods. The nature
In summary, survey research is one method of the data that are collected for the grid need not
of collecting, organising and analysing data. The be quantitative. Each cell in the grid might be
relevant data can be collected by a variety of tech­ filled with numeric or quantitative data (e.g. age,
niques and in many studies it may be appropriate income, years of education, score on an IQ test,
to use a range of research methods (see Figure 1.1). number of times assaulted) or it may be filled with
much more qualitative information. For example
the variable (column) about which we are collect­
QUANTITATIVE AND QUALITATIVE ing information can be used to indicate the nature
RESEARCH of a person’s marriage. We could simply ask them
to rate the marriage on a scale of 0 to 10 or we
Survey research is widely regarded as being inher­ could enquire at some depth about the nature of
ently quantitative and positivistic and is contrasted the marriage. In the course of our enquiry we
to qualitative methods such as participant obser­ may collect information about equality, conflict,
vation, unstructured interviewing, case studies ways of handling conflict, feelings of intimacy
and focus groups (see Web Pointer 1.2). Quan­ and so forth. The clues about equality, conflict
titative survey research is sometimes portrayed as and intimacy may come from specific interview
being sterile and unimaginative but well suited questions, may be based on observations or may
to providing certain types of factual, descrip­ be based on information gleaned from a number
tive information—the hard evidence. Qualitative of different topics discussed in a relatively unstruc­
methods are often regarded as providing rich data tured interview. Whether the data are quantitative
about real life people and situations, and being or are collected with a questionnaire or some other
more able to make sense of behaviour and to method does not go hand in hand with a survey.
understand behaviour within its wider context. The survey will simply collect systematic data that
However qualitative research is often criticised for allows for systematic comparison between cases
lacking generalisability, being too reliant on the on the same characteristics. Similarly we might
subjective interpretations of researchers and being collect quantitative data but this does not mean we
incapable of replication by subsequent researchers. are conducting a survey. If this is collected from
This distinction between quantitative and only one case (as in a case study) or is collected
qualitative research is frequently unhelpful and only spasmodically from some cases we do not

WEB POINTER 1.2 Positivism and survey research

A brief but useful description of the term positivism. www.socialresearchmethods.net/kb/positvsm.php

Howard Becker discusses some of the differences http://stuff.natehaas.com/pub/TheoryThe%20Necessary%20
and similarities of quantitative and qualitative Evil.htm
research. and
www.sfu.ca/-palys/Becker-EpistemologyOfQualitativeRese
arch.pdf
 CHAPTER 1 THE NATURE OF SURVEYS 7

have the structured data set that permits survey and provide accounts of how they did their
analysis. research (e.g. Bell and Newby, 1977). Like my
The quantitative/qualitative distinction is also own experience, theirs does not conform to the
not especially helpful at the data analysis stage. textbook models.
While statistical techniques are typically employed What ought to be done in a book like this?
to analyse survey research it is the logic of analysis To describe an ‘ideal-typical’ model of survey
that distinguishes survey research. The logic of research, in which each step of research is outlined,
survey analysis is that variation in one variable is is not to describe what researchers do. As such it
matched with variations in other variables. The can mislead. When you actually do some research
notion of co-variation is not an inherently statis­ you will find that you are not doing what you
tical concept—it is a logical concept that has been ‘should’. So should the book describe the reality?
systematically formulated by Mill (1879). Statis­ Perhaps. But which one? The course that a piece
tical methods have been developed as indices of research actually takes will be peculiar to that
of co-variation but this does not mean that the piece of research: it is affected by the research topic,
method of analysis is quantitative or statistical. the technique of data collection, the experience
Similarly, causal analysis is a common feature of and personality of the researcher, the ‘politics’ of
survey research. That is, survey data are used to the research, the types of people or situation being
evaluate whether one variable affects another (e.g. studied, funding and so on. I could describe my
does religion affect voting preference?). Again, experiences but like an ideal-typical model they
statistical tools are frequently used and are helpful would not reflect other people’s.
as part of the process of evaluating whether one I have decided to do a bit of both. I will
variable affects another but causal analysis is funda­ outline the key steps which a survey researcher
mentally a logical rather than a statistical process. must take at some stage and describe the reasoning
It is most useful to think of survey research behind the order in which it is normally suggested
by emphasising that it is a structured approach to they be taken. But, I will also point out that in
data collection and analysis and that it relies on practice some steps are omitted, things are done
a particular logic of analysis. Typifying survey out of order and we move backwards and forwards
research using the quantitative/qualitative distinc­ between steps. Guidelines that are provided are not
tion is misleading and emphasises the use of meant to be prescriptive. The guidelines I describe
statistical analysis and quantitative measures at the are like signposts or a map to provide some direc­
expense of highlighting the fundamental charac­ tion and give us clues as to where to go when we
teristics of the survey method. get lost. As you become more familiar with the
territory you can manage more easily without the
map and learn short cuts. What 1 describe will not
PRACTICE VS IDEAL TYPES always reflect your research experience but will
provide guidance. You should not try to follow
A basic difficulty when trying to describe how to each step slavishly. The prime goal of research
do research is the gap between textbook accounts should be to gain accurate understanding and, as
of how research should be done and how it actually a researcher, use methods and techniques which
is done. A number of valuable books have been enhance understanding. Use the method: do not
published in which some researchers ‘come clean’ let it use you.

KEY CONCEPTS

Case Variable
Data collection technique versus research method Variable by case data grid
Structured data set Variation
Unit of analysis
8 UttiM THE SCOPE OF SURVEY RESEARCH

FURTHER READING

Catherine Marsh’s book The Survey Method (1982) critique of survey research from a symbolic inter-
remains the best description of the survey method actionist perspective as do Blumer’s papers ‘What
available. Her outline and evaluation of the most is Wrong with Sociological Theory?’ (1954) and
substantial criticisms of surveys is direct, clear and ‘Sociological Analysis and the Variable’ (1956).
stimulating. Chapter 3 of The Sociological Imagination
Carr-Hill’s paper ‘Radicalising Survey Meth­ (1959) by C. W. Mills on abstracted empiricism
odology’ (1984) approaches the criticisms of is a well-known attack on certain forms of survey
surveys somewhat differently but suggests ways of research. A very useful discussion of positivism
overcoming some of the problems from a radical is contained in Halfpenny’s 1982 book Positivism
perspective. and Sociology: Explaining Social Life, which clarifies
Donsbach’s paper on ‘Survey Research at the much of the confusion in the way this concept is
End of the Twentieth Century’ (1997) provides used in sociology.
a critical evaluation of surveys as they are often Differences between the various types of
practised. research designs are discussed fully in de Vaus’
The exchange between Greeley (1996) ‘In book Research Design in Social Research (2001).
Defense of Surveys’ and Lewontin (1996) Chapter 4 of Madge’s classic text, The Tools of
‘In Defense of Science’ in relation to a particu­ Social Science (1965) provides a concise summary
lar survey (national survey of American sexual of Mill’s logic that is relevant to survey analysis.
behaviour) highlights some of the practical issues De Vaus (2001) elaborates on the logic of the
involved in evaluating survey research. methods of survey analysis and on concepts of
Denzin’s The Research Act (2009) provides a causality.

EXERCISES

1 Draw a variable by case data grid for six people and b Can use unstructured interviews,
five variables of your choice. Fill in characteristics for c Can use observation,
each case. d Can use content analysis,
2 Draw three variable by case data grids each using a e Uses a variable by case data grid,
different unit of analysis from the following list: countries, f Is based on ‘natural’ variation between cases,
years and people. In each grid use four cases and five g Researcher creates variation between cases,
variables appropriate to the selected unit of analysis. h Compares the ‘treatment’ with the ‘no treatment’
3 Different methods of research (survey, experiment group for analysis.
and case study) have different characteristics. Next to 4 Imagine that you believe being unemployed leads to
each characteristic below indicate the method having a loss of self-esteem. Briefly contrast how the case
the particular characteristic. study, the experiment and the survey would differ in
a Can use questionnaires. their basic procedure for testing this proposition.