Experimental Psychology: Chapter 15: Drawing Conclusions The Search For The Elusive Bottom Line

EXPERIMENTAL PSYCHOLOGY
Chapter 15: Drawing Conclusions

The Search for the Elusive
Bottom Line
 Evaluating the Experiment from the Inside:

Internal Validity
 Taking a Broader Perspective: The Problem of
External Validity
 Handling a Nonsignificant Outcome
What is internal validity?
An experiment is internally valid when the effects
on the dependent variable are due to the
independent variable.
An internally valid experiment is free of

confounding.
What is a manipulation check?
A manipulation check evaluates how well
the experimenter manipulated the experimental
situation.
A manipulation check determines whether subjects

followed directions and were appropriately affected
by our treatments.
What did Orne (1969) mean by a pact of
ignorance?
Subjects expect their data to be discarded if they
guess the experimental hypothesis, and don’t
volunteer this information to the experimenter.
Experimenters don’t want to test additional subjects

and may take subject reports at face value.
How can researchers overcome this
problem?
Debrief subjects after the experiment and convey
that you want to know if they guessed the
hypothesis.
Provide incentives for guessing the hypothesis.

Which mistakes could produce this threat
to internal validity?
Selecting the wrong statistical test
Using a t-test to analyze ordinal data.
Improperly using a statistical test

Calculating multiple t-tests.
Drawing the wrong conclusions from the test

Reporting p = .07 as a trend.
External Validity
An experiment is externally valid when its findings
can be extended to other situations and populations.
What two requirements must an
externally valid study satisfy?
1. The experiment must be internally valid.
2. The experimental findings can be replicated.

What does it mean to generalize across
subjects? Why is this important?
The findings can be extended to a larger group than
our sample.
Generalizing across subjects is critical to the

external validity and usefulness of experimental
findings.
Which problems prevent us from
generalizing across subjects?
The samples used in psychological research are
often biased and may not represent the larger
population.
The samples may not always represent even

college sophomores since we heavily depend on
volunteers.
Explain generalizing from procedures to
concepts.
Experimental variables like anger may have
multiple operational definitions.
When we generalize from our experimental
results, we move from discussing our specific
operational definition of anger to discussing the
concept of anger itself.
Why is this a problem in research?
It is dangerous to generalize from a single
experiment’s operational definition of anger.
We cannot be sure of the reliability or validity

of our procedures.
What is research significance?
A study achieves research significance when its
findings clarify or extend knowledge gained from
previous studies and raise implications for broader
theoretical issues.
When should we question novel findings?
We should question novel findings when they
contradict prior findings that have been successfully
replicated.
The burden of proof is on the experimenter who

claims novel findings to explain this discrepancy.
Why do we want to generalize beyond
the laboratory?
We want to generalize beyond the laboratory to
increase the external validity of our findings.
What is the problem in extending
laboratory findings?
Since extraneous variables are uncontrolled
in real world setting and operate in complex
combinations, they can modify the influence
of our individual variables.
What is the trade-off between laboratory
and field experiments?
The trade-off is between the laboratory’s more
precise control of extraneous variables and the field
experiment’s greater realism and external validity.
What did Hanson (1980) find about the
agreement between laboratory and field
studies?
Hanson (1980) found that more laboratory
than field studies reported a positive correlation
between reported attitudes and behavior.
When can we extend results from an
experiment to everyday life?
We can’t confirm external validity until additional
studies are completed in field settings.
Researchers can increase and verify the external

validity of laboratory findings using aggregation,
multivariate designs, nonreactive measurements,
field experiments, and naturalistic observation.
What is aggregation? What is its logic?
Aggregation is the grouping together and averaging
of data to increase external validity.
Combining the results of experiments with different

subjects and methodologies increases the generality
and external validity of our findings.
How is this logic shared by meta-analysis?
Meta-analysis uses statistical analysis to combine
and quantify data from many comparable
experiments to calculate an average effect size.
How does aggregation establish external
validity?
Aggregation establishes external validity by
combining the results of experiments performed
using different subjects, stimuli and/or situations,
trials or occasions, and measures.
What is a multivariate design?
A multivariate design studies multiple DVs.
For example, a study of repetitive strain places a

computer keyboard at different distances from the
subject IV) and measures the effect on three
different muscle groups (3 DVs).
What is the advantage of multivariate
designs?
Multivariate designs allow us to study the effect of an
independent variable on combinations of dependent
variables.
These designs better simulate the complexity

of the real world than univariate designs and provide
more detailed information.
How are multivariate experiments
analyzed?
We analyze multivariate experiments with a
multivariate analysis of variance (MANOVA).
How should researchers handle a
nonsignificant outcome?
Accept the outcome, don’t reframe your result as
“almost significant.”
Examine the experimental procedures for design

flaws.
If the design appears sound, decide whether the

hypothesis was reasonable.
How should we handle the possibility of
faulty procedures?
Check for possible causes of a nonsignificant
outcome like:
1. confounding
2. extraneous variables that increase within-
subjects variability
3. weak manipulation of the IV
4. inconsistent or flawed procedures
5. ceiling and floor effects
6. insufficient power
How should we handle the possibility of a
faulty hypothesis?
If previous studies supported the hypothesis and
ours did not, look for differences in experimental
design or sample.
If there was no previous support and our design and

execution were good, we may have to revise or
discard our hypothesis.
END
THANK YOU

Experimental Psychology: Chapter 15: Drawing Conclusions The Search For The Elusive Bottom Line

Uploaded by

Copyright:

Available Formats

Experimental Psychology: Chapter 15: Drawing Conclusions The Search For The Elusive Bottom Line

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Experimental Psychology: Chapter 15: Drawing Conclusions The Search For The Elusive Bottom Line

Uploaded by

Copyright:

Available Formats

EXPERIMENTAL PSYCHOLOGY

Chapter 15: Drawing Conclusions

 Evaluating the Experiment from the Inside:

An internally valid experiment is free of

A manipulation check determines whether subjects

Experimenters don’t want to test additional subjects

Provide incentives for guessing the hypothesis.

Improperly using a statistical test

Drawing the wrong conclusions from the test

2. The experimental findings can be replicated.

Generalizing across subjects is critical to the

The samples may not always represent even

We cannot be sure of the reliability or validity

The burden of proof is on the experimenter who

Researchers can increase and verify the external

Combining the results of experiments with different

For example, a study of repetitive strain places a

These designs better simulate the complexity

Examine the experimental procedures for design

If the design appears sound, decide whether the

If there was no previous support and our design and

You might also like