SHEG Evaluating Information Online
SHEG Evaluating Information Online
SHEG Evaluating Information Online
SHEG.STANFORD.EDU
4 EXECUTIVE SUMMARY
15 EVALUATING EVIDENCE
16 Task
17 Overview
18 Rubric
Between January 2015 and June 2016, we administered 56 tasks to students across 12
states. In total, we collected and analyzed 7,804 student responses. Our sites for field-
testing included under-resourced, inner-city schools in Los Angeles and well-resourced
schools in suburbs outside of Minneapolis. Our college assessments, which focused on
open web searches, were administered online at six different universities that ranged from
Stanford, an institution that rejects 94% of its applicants, to large state universities that
admit the majority of students who apply.
In what follows, we provide an overview of what we learned and sketch paths our
future work might take. We end by providing samples of our assessments of civic online
reasoning.
EXECUTIVE SUMMARY
Evaluating Information: The Cornerstone of Civic Online Reasoning
November 22, 2016
5
STANFORD HISTORY EDUCATION GROUP EXECUTIVE SUMMARY
measures using digital content. We used 4) News Search: Students distinguish between
screen shots of Slate’s landing page to assess a news article and an opinion column.
students’ ability to distinguish between a
news item and an ad. Similarly, we used 5) Home Page Analysis: Students identify
screen shots of tweets, Facebook posts, and advertisements on a news website.
a reproduction of CNN’s website in crafting
other exercises. We are mindful of the criticism
High School
of using paper-and-pencil measures to assess
students’ ability to judge online sources. At the 1) Argument Analysis: Students compare and
same time, there is evidence from the OECD evaluate two posts from a newspaper’s
that important abilities for evaluating online comment section.
sources can be measured offline.8 Even more
crucial in our decision, however, was the hope 2) News on Facebook: Students identify
that our assessments would be used in under- the blue checkmark that distinguishes a
resourced schools where online assessment verified Facebook account from a fake one.
often remains a remote possibility. Our middle
school assessments provide easy-to-use 3) Facebook Argument: Students consider the
measures that teachers and others can use relative strength of evidence that two users
to gauge students’ basic skills. At the high present in a Facebook exchange.
school level, we designed more complex tasks
that asked students to reason about multiple 4) Evaluating Evidence: Students decide
sources; at the college level, the exercises whether to trust a photograph posted on
were administered online. When students are a photo-sharing website.
working at advanced levels, there is nothing
to prevent the high school exercises from 5) Comparing Articles: Students determine
being used with middle school students, or the whether a news story or a sponsored post
college exercises from being used with high is more reliable.
school students.
College
Summaries of each of our exercises are below.
The exercises in bold appear in the following 1) Article Evaluation: In an open web search,
pages. students decide if a website can be trusted.
6
STANFORD HISTORY EDUCATION GROUP EXECUTIVE SUMMARY
demonstrate the link between digital literacy
NEXT STEPS
and citizenship. By drawing attention to this
We envision several next steps that build on connection, a series of videos could help to
what we have accomplished. These include: mobilize educators, policymakers, and others
to address this threat to democracy.
Assessment for Learning. Although our
tasks could be used in a variety of ways, we
think they are powerful tools for classroom
instruction. Rather than simply serving as
assessments of learning, they can also be
assessments for learning, or what are known
as “formative assessments.” Teachers can use
our tasks to track student understanding and
to adjust instruction accordingly. Similarly,
teachers can use these exercises as the
basis for broader lessons about the skills
these tasks measure. We also hope to create
accompanying materials that help teachers
incorporate these tasks into the flow of
classroom instruction.
7
STANFORD HISTORY EDUCATION GROUP EXECUTIVE SUMMARY
H O M E PAG E A NA LYSIS
1. This is / is not (circle one) an
Here is the home page of Slate.com. Some of the H STORY
advertisement because_________ EDUCATION GROUP
things that appear on Slate.com are news stories,
____________________________ and others are advertisements.
____________________________
____________________________
____________________________
____________________________
____________________________
In this assessment, students are presented with the home page of Slate magazine’s website,
which includes both news items and advertisements. The task assesses students’ ability
to distinguish between an article and an advertisement. Students must evaluate three
different sections of the web page—a traditional advertisement, a news story, and a native
advertisement—and determine the nature of each. Successful students understand the
different forms that advertising can take and identify both traditional and native advertising.
They are also able to explain the features that distinguish a news story from an ad.
We piloted several drafts of this task with 350 middle school students. We completed final
piloting with 203 middle school students. Results indicated that students were able to identify
traditional news stories and traditional advertisements: more than three-quarters of the
students correctly identified the traditional advertisement and the news story. Unfortunately,
native advertising proved vexing for the vast majority of students. More than 80% of students
believed that the native advertisement, identified by the words “sponsored content,” was
a real news story. Some students even mentioned that it was sponsored content but still
believed that it was a news article. This suggests that many students have no idea what
“sponsored content” means and that this is something that must be explicitly taught as early
as elementary school.
10
STANFORD HISTORY EDUCATION GROUP HOME PAGE ANALYSIS
RUBRIC
Student correctly identifies the item as an ad or non-ad and provides
MASTERY
coherent reasoning.
SAMPLE RESPONSES
M AST E RY
It has a coupon code , a big company logo , and has the words
” l i mi ted time offer.”
This student chooses an irrelevant factor: how “useful” the content seems as a reason it is an
advertisement.
M A ST E RY
Student correctly identifies this story as an article and identifies several features of the article that
helped her categorize it as an article.
EMERGING
Student identifies a feature that may or may not indicate its status as an ad.
12
STANFORD HISTORY EDUCATION GROUP HOME PAGE ANALYSIS
BE G I N N I N G
N AT I V E A D : W O M E N I N T E C H
M AST E RY
This student explains that the words “sponsored content” signify that the story is an advertisement.
EMERGING
This student correctly identifies the story but offers an inaccurate idea about what being
“sponsored” means.
13
STANFORD HISTORY EDUCATION GROUP HOME PAGE ANALYSIS
BE G I N N I N G
This student argues that this story must be an article because it lacks traditional features of an ad.
This student notices the words “sponsored content” but still argues that it is an article.
14
STANFORD HISTORY EDUCATION GROUP HOME PAGE ANALYSIS
EVALUATING EVIDENCE
H STORY
EDUCATION GROUP
On March 11, 2011, there was a large nuclear disaster at the Fukushima Daiichi
Nuclear Power Plant in Japan. This image was posted on Imgur, a photo
sharing website, in July 2015.
Not much more to say, this is what happens when flowers get nuclear birth defects
Does this post provide strong evidence about the conditions near the
Fukushima Daiichi Power Plant? Explain your reasoning.
OV ERV IEW
Given the vast amount of information available online, students need to be able to distinguish
between legitimate and dubious sources. Students need to ask a basic question: Where did this
document I’m looking at come from? This task assesses whether students will stop to ask this
question when confronted with a vivid photograph. Students are presented with a post from
Imgur, a photo sharing website, which includes a picture of daisies along with the claim that the
flowers have “nuclear birth defects” from Japan’s Fukushima Daiichi nuclear disaster.
Although the image is compelling and tempting to accept at face value, successful students will
argue that the photograph does not provide strong evidence about conditions near the nuclear
power plant. Students may question the source of the post, arguing that we know nothing
about the credentials of the person who posted this photo (especially since it appears on a site
where anyone can upload a photo). Alternatively, students may point out that the post provides
no proof that the picture was taken near the power plant or that nuclear radiation caused the
daisies’ unusual growth.
Various drafts of this task were piloted with 454 high school students. The final version was given
to 170 high school students. By and large, students across grade levels were captivated by the
photograph and relied on it to evaluate the trustworthiness of the post. They ignored key details,
such as the source of the photo. Less than 20% of students constructed “Mastery” responses, or
responses that questioned the source of the post or the source of the photo. On the other hand,
nearly 40% of students argued that the post provided strong evidence because it presented
pictorial evidence about conditions near the power plant. A quarter of the students argued that
the post did not provide strong evidence, but only because it showed flowers and not other
plants or animals that may have been affected by the nuclear radiation.
17
STANFORD HISTORY EDUCATION GROUP EVALUATING EVIDENCE
RUBRIC
Student argues the post does not provide strong evidence and questions
the source of the post (e.g., we don’t know anything about the author
MASTERY
of the post) and/or the source of the photograph (e.g., we don’t know
where the photo was taken).
Student argues that the post does not provide strong evidence, but the
EMERGING explanation does not consider the source of the post or the source of the
photograph, or the explanation is incomplete.
Student argues that the post provides strong evidence or uses incorrect
BEGINNING
or incoherent reasoning.
SAMPLE RESPONSES
MASTERY
This student questions the source of the photo, arguing that there is no way to know whether the
photo was actually taken near the plant or if the mutations were a result of nuclear radiation.
This student questions the source of the post, arguing that we know nothing about the poster’s
credentials or whether the evidence was doctored.
No, itdoes not really provide strong evidence . A photo posted by a stranger
on l ine has little cred ibility.This photo could very easily be Photoshopped
or stolen from another completelydifferent source; we have no dea i given
this information, which makes it an unreliable source.
18
STANFORD HISTORY EDUCATION GROUP EVALUATING EVIDENCE
E M ERGI N G
This student begins to question both the photo and the source of the post but does not fully explain
his thinking.
This student critiques the evidence by arguing that it could have been digitally altered but does not
offer any further explanation or critique of the evidence.
B EGI N N I N G
This student accepts the evidence at face value, arguing that it provides visual proof of the effects
of the nuclear disaster.
This post does provide strong evidence because it shows how the
small and beautiful things were affected greatly , that theylook and
grow completelydifferent than theyare supposed to . Additionally , it
suggests what such a disaster could do to humans .
Although this student argues that the post does not provide strong evidence, she still accepts the
photo as evidence and simply wants more evidence about other damage caused by the radiation.
No, this photo does not provide strong evidence because it only shows a
small portion of the damage and effects caused by the nuclear disaster.
19
STANFORD HISTORY EDUCATION GROUP EVALUATING EVIDENCE
CLAIMS ON SOCIAL MEDIA
H STORY
EDUCATION GROUP
The assessment directs students to this webpage:
OVERV IEW
Twitter is filled with individuals and groups seeking to further their agendas. In order to navigate
this sea of information, students need to be able to weigh the relative strengths and weaknesses
of tweets as sources of information. Specifically, students need to consider the sources of tweets
and the information contained in them.
This task presents students with a tweet from the liberal advocacy organization MoveOn.org that
reads: “New polling shows the @NRA is out of touch with gun owners and their own members.”
The tweet includes a graphic that asserts, “Two out of three gun owners say they would be more
likely to vote for a candidate who supported background checks.” The tweet contains a link to a
press release by the poll’s sponsor, the Center for American Progress, another liberal advocacy
organization. Both the news release and the tweet indicate that Public Policy Polling conducted
the poll in November 2015. Students are asked why this tweet might and might not be a useful
source of information. Strong responses will note that the tweet may provide useful information
given that it is based on a poll conducted by a professional polling firm. At the same time,
students must acknowledge how the political motivations of the Center for American Progress
and MoveOn.org, both of which support stronger gun control measures, may have shaped the
structure of the poll and how its results were publicized.
We piloted this task with 44 undergraduate students at three universities. Results indicated that
students struggled to evaluate tweets. Only a few students noted that the tweet was based on
a poll conducted by a professional polling firm and explained why this would make the tweet
a stronger source of information. Similarly, less than a third of students fully explained how
the political agendas of MoveOn.org and the Center for American Progress might influence the
content of the tweet. Many students made broad statements about the limitations of polling or
the dangers of social media content instead of investigating the particulars of the organizations
involved in this tweet.
23
STANFORD HISTORY EDUCATION GROUP CLAIMS ON SOCIAL MEDIA
An interesting trend that emerged from our think-aloud interviews was that more than half of
students failed to click on the link provided within the tweet. Some of these students did not click
on any links and simply scrolled up and down within the tweet. Other students tried to do outside
web searches. However, searches for “CAP” (the Center for American Progress’s acronym, which
is included in the tweet’s graphic) did not produce useful information. Together these results
suggest that students need further instruction in how best to navigate social media content,
particularly when that content comes from a source with a clear political agenda.
RUBRIC
Student fully explains that the tweet may be useful because it includes
MASTERY
data from a poll conducted by a polling firm.
Student addresses the polling data and/or the source of the polling
EMERGING data but does not fully explain how those elements may make the tweet
useful.
Student does not address the polling data or the source of the polling
BEGINNING
data as a reason the tweet may be useful.
24
STANFORD HISTORY EDUCATION GROUP CLAIMS ON SOCIAL MEDIA
SAMPLE RESPONSES
M ASTE RY
This student identifies the polling firm and provides evidence of the firm’s reliability.
The polling information which the tweet references was collected by Public Policy Polling,
which appears to have a fairly strong accuracy record, though with a Democratic bent (e.g.,
Wall Street Journal article: http://www.wsj.com/articles/SB122592455567202805)
EME RGI N G
This student references the poll but does not explain why that makes the tweet a useful source of
information.
The photo used in this tweet was compiled from a public policy polling survey.
BEGINNING
This student focuses on the tweet’s appearance rather than its content.
It could be useful because a graphic with a strong message can be enlightening or more likely
thought provoking.
25
STANFORD HISTORY EDUCATION GROUP CLAIMS ON SOCIAL MEDIA
Question 2: Why might this tweet not be a useful source?
Student addresses the source of the tweet or the source of the news release
EMERGING but does not fully explain how those elements may make the tweet less
useful.
Student does not address the source of the tweet or the source of the news
BEGINNING
release as reasons the tweet may be less useful.
SAMPLE RESPONSES
MASTERY
This student explains how MoveOn.org’s work as a political advocacy organization might influence
the tweet’s contents.
26
STANFORD HISTORY EDUCATION GROUP CLAIMS ON SOCIAL MEDIA
E M ERGI N G
The student suggests that the tweet is politically motivated but does not explain how this might
influence the content of the tweet.
Although MoveOn.org claims to be independent, they also were paid to work on Obama’s
campaign so are clearly Democrat-oriented, and the NRA members tend to be Republicans
(http://front.moveon.org/about/#.V0NYK5MrLBI).
B EGI N N I N G
This student focuses on the nature of Twitter rather than the source of the tweet.
Twitter is a social platform built for sharing opinions, and though there are plenty of news
organizations sharing facts on Twitter, I’d be more likely to trust an article than a tweet.
27
STANFORD HISTORY EDUCATION GROUP CLAIMS ON SOCIAL MEDIA