Creative Education, 2020, 11, 674-692

https://www.scirp.org/journal/ce
ISSN Online: 2151-4771
ISSN Print: 2151-4755

Digital Literacy for Secondary School Students:

Using Computer Technology to Educate about
Credibility of Content Online

Robin Cohen1, Alexandre Parmentier1, Glaucia Melo1, Gaurav Sahu1, Aswin Annamalai1,
Sheldon Chi1, Trevor Clokie1, Amir Farrag1, Abdul Naik1, Syed Naseem1, Shikhar Sakhuja1,
Jean Wang1, Rich Clausi2, Anita Santin2

David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

1

Ontario Secondary School Teachers Federation, ‎Toronto, Canada

2

How to cite this paper: Cohen, R., Par- Abstract

mentier, A., Melo, G., Sahu, G., Annamalai,
A., Chi, S., Clokie, T., Farrag, A., Naik, A., This paper presents an approach to educate secondary school students in the
Naseem, S., Sakhuja, S., Wang, J., Clausi, province of Ontario about the credibility of online content. The critical focus
R., & Santin, A. (2020). Digital Literacy for
here is on integrating computer technology into the teaching of the topic; how
Secondary School Students: Using Computer
Technology to Educate about Credibility of to introduce the material in classroom settings with respect to the current cur-
Content Online. Creative Education, 11, riculum is also outlined. Contrast with an existing proposal for digital literacy
674-692. developed by historians at Stanford University is provided at the outset. In all,
https://doi.org/10.4236/ce.2020.115050
the value of appealing to the current digital experiences of students, when re-
Received: March 6, 2020 vealing the potential for misinformation, is the critical message. Exploration of
Accepted: May 9, 2020 social media environments popular with youth and opportunities for game-based
Published: May 12, 2020
quizzes for interactive engagement are both advocated.
Copyright © 2020 by author(s) and
Scientific Research Publishing Inc. Keywords
This work is licensed under the Creative
Commons Attribution International Use of Internet and Social Media, Digital Technologies,
License (CC BY 4.0). Online Misinformation, Digital Literacy
http://creativecommons.org/licenses/by/4.0/
Open Access

1. Introduction
In this paper, we present novel approaches for educating secondary school stu-
dents about the origins of online misinformation, promoting greater scrutiny about
the credibility of sources. In particular, the proposals offered integrate techniques
for analyzing content, enabling engaging interaction and revealing key examples,

DOI: 10.4236/ce.2020.115050 May 12, 2020 674 Creative Education

 R. Cohen et al.

all constructed from computer science methodology. We contrast our approach

with one of the most prominent valuable efforts for teaching students about cre-
dibility of sources, that of Stanford’s History Education Group (SHEG) (Donald,
2016). As part of our reflection on the potential of our proposed solutions, we
integrate as well feedback received from current educators. This enables impor-
tant commentary as well about how the content that we imagine could be deli-
vered within secondary schools in the province of Ontario in Canada, as a co-
gent example.
The rest of this paper is organized as follows. Some background is introduced
first. This is followed by a section outlining some novel contexts of use (fake re-
views, hoaxes and sponsored ads) as valuable topics of discussion in class settings.
The subsequent section provides a detailed proposal for how to educate students
about misleading claims and evidence on websites. This section also puts for-
ward specific proposals for how to leverage technology in order to deliver the
lessons to the students. The section that follows then focuses on how to position
this content with respect to curriculum, outlining the role that educators play.
The final section steps back to emphasize the key messages of the paper and to
propose several important final recommendations for the future.

Background
Establishing digital literacy quickly has become an important concern for today’s
youth. According to a study by the Pew Research Center in the USA in 2018 (M.
Anderson & Jiang, 2018), 95% of teens now report that they have a smartphone
or access to one, and 45% say that they are online on a near-constant basis.
There has, as well, been a significant increase in teenage users of social media
(more than a 100% increase over the past four years). The preponderance of mi-
sinformation is also alarming. A study by Vosoughi, Roy, & Aral (2018), con-
ducted by analyzing diffusion of news stories on Twitter from 2006 to 2017,
suggests that false news spreads farther, faster, deeper and more broadly that the
truth, especially for political news. And yet a report released by the Stanford
History Education Group (SHEG) in 2015 (Wineburg, McGrew, Breakstone, &
Ortega, 2016) shows a dismaying inability by students to reason about informa-
tion that they see on the Internet (including an absence of effort to do fact check-
ing by visiting alternate sites (Wineburg & McGrew, 2016). Students, for exam-
ple, had a hard time distinguishing advertisements from news articles or iden-
tifying where information came from. The report mentions “...students may fo-
cus more on the content of social media posts than on their sources” and also
“...despite their fluency with social media, many students are unaware of basic
conventions for indicating verified digital information”.
This report chronicles the Stanford effort to engage middle school, high school
and college students in determining whether sources of information were credi-
ble or not. This includes both asking students to judge posts in social media en-
vironments such as Twitter and Facebook, and challenging them to identify wheth-

DOI: 10.4236/ce.2020.115050 675 Creative Education

R. Cohen et al.

er content on a home page was in fact an advertisement. Students were exposed

to photos and videos as well as texts, for the content, and were asked to fill out
worksheets to reveal both their conclusions and explanations for their answers.
This work then led to the creation of a resource called Civic Online Reasoning
(COR) (Wineburg et al., 2020), which aimed at providing for teachers sample
lessons and an overall curriculum.
We have noticed as well an increasing focus from youth on such new social
media as Instagram and Snapchat, environments where fake reviews may prevade
and where influencers may promote sponsored advertising. In the discussion that
follows, we design some strategies for educating secondary school students in
Ontario regarding these current concerns. We also are motivated to educate stu-
dents who may be making purchases online. The prevalence of teen use of social
media and their activity with online shopping has been drawn out in from the
Pew Research 2018 study (M. Anderson & Jiang, 2018); we include here some
figures (Figure 1 and Figure 2) to underscore the importance of these issues.

Figure 1. Teenagers prefer to shop online according to PEW research (M. Anderson &
Jiang, 2018).

Figure 2. Most commonly used social media according to PEW research (M. Anderson &
Jiang, 2018).

DOI: 10.4236/ce.2020.115050 676 Creative Education

 R. Cohen et al.

2. Contexts of Use and Suggested Content of Focus

In this section, we discuss a few specific online contexts to present to students, in
order to educate about sources of misinformation. We first of all delve into con-
cern over fake reviews, which may be prevalent in online shopping contexts. The
current tendency for youth to purchase online motivates our interest in highlight-
ing this specific concern with credibility. A second related issue is the presence
of sponsored advertising, and the need for students to discern when content is
being directed to users for reasons of financial gain. We also suggest presenting
examples in a social media context with heavy investment from teenagers, namely
Instagram. Finally, we highlight possible deception that may occur from hoaxes
perpetrated online in contexts such as Youtube, which is also very popular with
youth (M. Anderson & Jiang, 2018), offering one brief example that is especially
relevant to teens in Ontario.

2.1. Fake Reviews

The example presented in Figure 3 shows reviews of a product that is supposed
to enhance the growth of eyelashes, shown on the Amazon site1. In the review, the
customers mention shipping and other characteristics of the product, but not the
product goal. This can be considered problematic, as businesses can use frivol-
ous strategies to inflate product reviews.
A real case which occurred revealed that after buying a product on a popular
retail website, when the product was delivered, a piece of paper containing an
email address was also sent to the customer, with instructions to receive a gift
from the store. After submitting an email to retrieve the award, the store replied,
asking for the product review and a screenshot of the review posted. When the
screenshot was received, the customer either got another product or credit to make
further purchases. The email is presented in Figure 4. Therefore, this product
probably has reviews that are not real and are stimulated by the promised gift.
Examples such as these would be useful to present to students to highlight possi-
ble concerns with fake reviews. Note that the prevalence of fake reviews online
and the challenge in properly identifying them is a current topic of research by
data scientists and artificial intelligence experts (Mukherjee, Liu, & Glance, 2012;
Elmurngi & Gherbi, 2017; Talwar, Jurca, & Faltings, 2007). Included here are ef-
forts as well to incentivize peers to report honestly (Jurca & Faltings, 2009) and
yet there still are challenges due to the existence of bot-generated content (Picchi,
2019; Davis, Varol, Ferrara, Flammini, & Menczer, 2016).

2.2. Sponsored Advertising

The social media platform Instagram includes a profiting market for advertising
products, meaning that this image sharing social media is also using its space to
run advertisement campaigns. Figure 5 displays one such example. The words
“paid partnership” appear at the top. It would be valuable to make students
aware of this practice so that they can identify and distinguish paid partnerships.
https://www.amazon.com
1

DOI: 10.4236/ce.2020.115050 677 Creative Education

R. Cohen et al.

Figure 3. Product review.

Figure 4. Email asking for product review in exchange for prizes.

Figure 5. Paid partnership label on instagram post.

DOI: 10.4236/ce.2020.115050 678 Creative Education

 R. Cohen et al.

We propose including a number of these examples as well, as part of the curriculum

for increasing awareness of deception in online social networks. Instagram ads
should have the word “sponsored” next to the ad, but this may be hard to spot
because the font is small or the content is immersive with main content. Other
signs of ads are hashtags that say “sponsor” or “promotion”. Figure 6 shows
some types of Instagram ads, which we can present to students, to increase their
awareness of this kind of content. We challenge the reader to try to locate the
word “sponsored” in each of the three screenshots contained in Figure 6 (drawn
from Instagram Explore Search, Instagram story and Instagram timeline spon-
sored post); each time, the size of the word is pretty tiny.
It is important to note that some celebrities have achieved significant financial
benefit from sponsored advertisements in Instagram (for example Kylie Jenner
made $1,266,000 and Cristiano Ronaldo made $975,000 on Instagram in 2019
according to an Instagram Rich List cited by Marketwatch (Pesce, 2019)). Insta-
gram in fact offers support to people who wish to advertise products on their
pages by providing engagement statistics, a feature called Insights, an example of
which is presented in Figure 7.
The ability to see the reach of one’s products may thus in turn end up pro-
moting quite an active participation in sponsored advertising. These sponsored
ads may then also encourage false testimonials and endorsements, in order to
promote financial gain. This makes it all the more important to educate youth
about this kind of content on Instagram.

2.3. Youtube Hoaxes

There also are at times pure hoaxes perpetrated in online social networks. For
example, Ontario heartthrob Justin Bieber was shown eating a burrito sideways
on Yes Theory, a popular Youtube channel (5.21 million subscribed users in
February 2020) and yet this was simply a fake photo orchestrated to draw atten-
tion (Alexander, 2018; Global News, 2018). The ability to doctor photos and
soon video with the emergence of deepfakes is another area for possibly engag-
ing in discussion with students (Wong, 2019).

Figure 6. Three types of instagram ads.

DOI: 10.4236/ce.2020.115050 679 Creative Education

R. Cohen et al.

Figure 7. Instagram insights feature.

The problem of distorted videos provides another opportunity for education.

In particular, students can be challenged to determine whether deception has
occurred with the images and then also to reflect on what steps to take in order
to encourage action by the companies that host this misinformation. It is espe-
cially valuable to have examples that are specific to the province of Ontario and
which may appeal to youth (the display of teen idols, though likely this will ap-
peal to a somewhat younger age group than secondary school students).

3. Engaging Students to Enlighten about Misinformation

In this section, we first of all present a platform that engages students to reflect
on whether the content they are viewing as webpages is truthful or not. A gener-
al overview of our design is displayed in Figure 8. We focus on how to present
examples and most importantly on how students could be invited to reveal their
responses, in an engaging manner. We design as well some options for providing
additional feedback to students. We imagine that some of the judgment of misin-
formation may be done using intelligent software; in this case, additional com-
mentary on the source of the questionable information may be forthcoming.
We follow the presentation of this creative technology with some distinct sug-
gestions for assessing student comprehension of misinformation, namely in the
form of quizzes. Two distinct options for running game-like scenarios in order
to question students are proposed.

DOI: 10.4236/ce.2020.115050 680 Creative Education

 R. Cohen et al.

Figure 8. Flowchart of the complete game.

3.1. The LEGIT App: Encouraging Students to Question Credibility

The following tool is designed to be a web app which students can run and engage
with, in order to present their opinions on whether an article they have accessed
is a true or a false account. The approach involves labelling some sources from
the internet as blacklisted or whitelisted2, with students receiving scores in a
game-like setting based in part on whether they deem blacklisted sources reput-
able while responding. An interesting feature in the game is the optional “Help”
button, which when chosen, will highlight some chunks in the provided text that
should make the task of discerning the given post easier (see Figure 9). The goal
of the app is to teach students to focus on important aspects of a given fact, such
as its source, and specific chunks in the text, in order to make a decision about
whether the text contains true information based on their understanding. The
entire app can be run as well as a competition between students in the classroom
setting; we discuss that option in further detail in Section 3.2. Although a web
app, this game could also be run across other platforms such as tablets and smart-
phones. Teachers can opt to provide each student in the class with some hand-
held device for real-time participation in the game, if desired.
Identifying misinformation or fake news online is the first lesson that we aim
to teach the younger generation. In order to do so, we propose a game (quiz)
backed by a module in the web app running artificial intelligence natural lan-
guage processing (NLP), which will help students identify attributes of fake on-
line content. It will be a point-based system as shown in Figure 10 that works as
follows:
• Each student is shown some claims and articles from three potential evidence
links3. It is important to note that each evidence link belongs to either a
“blacklist” or a “whitelist,” where the blacklist contains convincing articles
but from untrustworthy sources and the whitelist contains articles from
trustworthy sources such as verified fact-checking websites. Both the blacklist
and the whitelist would be predetermined by us. For instance, a URL such as
http://Network-Channel-Wereblogs.com will be blacklisted as opposed to a
trusted url such as http://bbc.com.
2
Ideally, incorrect/correct would be a better choice to represent the black-/white-listed URL but we
are abusing the meaning slightly here.
3
An evidence link is a web page which contains description of an event that might help in assessing
the validity of a claim.

DOI: 10.4236/ce.2020.115050 681 Creative Education

R. Cohen et al.

Figure 9. Working of the “Help” function; this figure is best read in colour.

Figure 10. Scoring scheme for the game.

• Each student will be asked both to assess the truthfulness of a statement made
online and to reflect on the origin of their conclusions. Basically, given a claim
and the three evidence articles, the student now needs to classify the claim as
fake or true and select an evidence link from the provided list that they believe
is a valid evidence and can back their decision. To avoid confusion, there will
be only one whitelisted evidence link among the provided list of articles.
For each claim, there will be a “Help” checkbox (described in more detail in
the following subsection), which when chosen, will highlight important chunks
in the provided text as shown in Figure 9. A scoring system could be introduced
if running the quiz as a game. For instance: each correct answer scores +2 points,
whereas each incorrect answer scores 0; furthermore, they lose 1 point if they
select a link from the blacklist, and 0.5 point if they choose a link that is neither
in the blacklist nor the whitelist. A leaderboard will be maintained for the entire
class with the student with the most points on top.
A student is, therefore, forced to develop the required “fact-checking” skills
while competing with their classmates. We believe that gamification of the idea
will encourage students to really focus on the given task, which in turn, will help
them identify the attributes of fake content online. We now briefly discuss the
NLP-based backend for the proposed game.

Long Short Term Memory w/Attention

Our NLP-based backend is constructed in order to run the “Help” button, which
assists students in learning which parts of an article may be most relevant when
assessing the truthfulness of a claim. Basically, we are designing an algorithm
(see Figure 11) which both predicts whether a given source is truthful or not
and which also identifies the origins of that conclusion. This is achieved through
a machine learning process, via an approach which makes use of neural networks.

DOI: 10.4236/ce.2020.115050 682 Creative Education

 R. Cohen et al.

Figure 11. Algorithm for the “Help” feature.

As sample examples are fed to this network (training), the system is able to pro-
gressively determine the basis for deciding whether an article (test case) is true,
so that judgement can be passed on an entirely new example, later. Our backend
is made up of a Long Short Term Memory (LSTM) cells (Hochreiter & Schmid-
huber, 1996) coupled with an attention mechanism (Bahdanau, Cho, & Bengio,
2014). An LSTM is a special type of recurrent neural network (RNN) which can
learn a meaningful representation4 of a given piece of text (claims and articles, in
our case). Amongst the different types of RNNs available, we chose LSTMs, as
web articles can be quite long and LSTMs are known to capture long term con-
text in a given piece of text. However, there is still a limit to the range of text
they can cover. Therefore, we add an attention mechanism (attn.) to the network,
which provides a soft pointer and tells the model which part of text to focus on,
while making a prediction.
We train our network on the Leaders Prize dataset5, which is a collection of
articles published on various fact-checking websites, such as, Politifact, Snopes,
and The Weekly Standard. Each data sample contains the following fields: a claim,
a claimant, some evidence articles, and finally, a label (among 0, 1, and 2, where
0 denotes False, 2 denotes True and 1 denotes a Neutral claim) that were scraped
from the article’s webpage. In order to run our NLP-based backend, we feed the
claim and evidence articles as input to our network and the provided labels as
output. We perform some preprocessing on the text where we lowercase all words
4
Vectors, to be more precise.
5
Coauthor G.Sahu is involved in running the Leaders Prize competition through the AI Institute of
the University of Waterloo. Entrants are tasked with properly identifying whether a source is true
or false, on a large set of sample texts. See https://leadersprize.truenorthwaterloo.com/en/.

DOI: 10.4236/ce.2020.115050 683 Creative Education

R. Cohen et al.

and normalize them to remove any non-ascii characters. Our algorithm is inten-
tionally a simplistic baseline to predict truthfulness, but the inclusion of the at-
tention mechanism then enables us to highlight a subset of text to students, as
outlined below. We achieve an F1-score of 48.21% on the dataset6.
We use PyTorch (Paszke et al., 2019) to implement the LSTM + attn. network
and train our architecture on a TitanXp GPU for ~3 hours.
We can now use the trained LSTM + attn. network in the backend of our app.
To be more precise, we will use it in the implementation of the “Help” checkbox.
Note that we only use the attention module of the trained network even though
it can be used to predict the truth rating of a claim as well; we would already
know truth-ratings when we are choosing claims and articles for the game. We
only need an automated system that can identify important chunks of text in a
large body of text. The working algorithm of the “Help” box for a given claim
and a set of evidence articles is shown in Figure 9 and described in Figure 11.
The flow chart of the complete game is shown in Figure 8. Note that the final
response box here is the output provided by the student, both deciding true or
false and identifying the article that best drives this conclusion.
We can notice from the performance that our LSTM + attn. model does not
give us the best F1-score but it can still be used to get some indication about im-
portant chunks in a given piece of text. With a suitable user interface for the de-
scribed game, we can make the content more engaging and will have succeeded
in fitting current technology to the education system such that the young gener-
ation can leverage its benefits.

3.2. Quizzes and Games for Further Engagement of Students

In this section, we elaborate on how technology can be used as part of the educa-
tion of students about digital misinformation. More specifically, we propose us-
ing games in order to deliver some of the lessons to students, in a setting where
students are asked to respond to questions and educators are responsible for di-
recting class discussion, once the results from the games are revealed.

3.2.1. An interactive Competitive Game to Identify Sponsored Ads

In Section 2, we highlighted some specific challenges for young people with re-
spect to misinformation: fake reviews, sponsored ads and hoaxes. In order to
engage students in their learning to improve digital literacy, we propose using
online quiz services. Game-based learning is considered promising when applied
in education. Researchers have shown that game-based learning helps to pro-
mote in students both critical thinking and content review (Dellos, 2015). In ad-
dition to capturing the attention of students, this activity could also assist in
6
An F1-score reflects how well the algorithm is performing, but not in terms of the more standard
unit of accuracy but as the harmonic mean of precision and recall. A score of 100% represents per-
fect prediction. For the Leaders Prize, the top competitor had a score of ~54%. Machine Reading
Comprehension of lengthy content is still an active area of research (Das, Munkhdalai, Yuan,
Trischler, & McCallum, 2018; Nguyen et al., 2016).

DOI: 10.4236/ce.2020.115050 684 Creative Education

 R. Cohen et al.

putting into practice the learned theory and clarifying possible remaining doubts
regarding the content presented. We also believe implementing this game quiz
can spark students’ further interest in the topic. The idea is to present questions
regarding the correct identification of elements that characterize a paid adver-
tisement. At the end of the round of questions, there should be a ranking pre-
senting which user (student) scored the highest points. When presenting the
quiz in class, teachers should act as moderators, explaining what should be done
during the quiz, how much time students would have to answer each question,
how many points you get for selecting correct questions, and how to answer the
questions, i.e., select the correct answer. When there are a lot of incorrect an-
swers or considerable disagreement on the correct response, the teacher should
take more time to explain why that answer is incorrect and what the rationale
should be to answer that question, using the same example in the question or
others. Students should answer the questions as they appear. Altogether, this
would be a very valuable opportunity for discussion and peer-based learning, as
proposed in (Crouch & Mazur, 2001) and (Mazur & Hilborn, 1997). Students
could, for example, confer before selecting their individual choices.
Kahoot! (http://www.kahoot.com/) is a game-based learning platform where
users can create a quiz with customized questions, and respondents can interact
using an internet browser (e.g., Chrome, Firefox), allowing the interaction in
desktops, mobile phones, tablets or any device that supports internet browsers.
Kahoot! is an interesting option to be presented as an example and even used in
classrooms, as it can be easily adopted and maintained. Using Kahoot! as exam-
ple, an image of a question we would put to students is presented in Figure 12;
students would be asked to reflect on whether the content is a paid ad.

Figure 12. Question example preview.

DOI: 10.4236/ce.2020.115050 685 Creative Education

R. Cohen et al.

3.2.2. Spot the Fake News

The authors of this paper are part of the first generation that grew up with the
widespread use of internet; most of us are somewhat adept in identifying fake
and promoted content since we have seen the landscape evolve. As the more
mature knowledgeable generation in identifying fake contents and ads, it is our
responsibility to teach youth who are currently faced with a landslide of highly
masked and sophisticated fake news and ads. Identifying content as accurate or
not is just as important as understanding how to solve basic math questions and
we propose an idea of including this training in classroom settings through the
means of interactive quizzes and games. In our proposed app, students can be
sensitized to fake news as discussed in Section 3. A mock-up of the app is shown
in Figure 13 and we now describe the setup to run this game in a class.
The teacher first creates a digital classroom by registering on the web-app. For
simplicity, let’s say the teacher named the digital classroom “DCR-01.” We now
consider two settings:
• Students are allowed to use electronic devices such as smartphones/tablets/
laptops: In this setting, the students would first add themselves to the “DCR01”
class by entering their credentials. Once the students finish registration, the
teacher will now launch a game from their interface. Once launched, the game
would be available on the students’ interface and the students play the game
as described previously in Section 3.1.

Figure 13. Mock up of the app: The user is shown a fact on the screen followed by some
evidence articles. In the current mock-up, the user needs to scroll down the screen to see
the rest of the evidence articles. Note that only one of the provided evidence articles
comes from a trustworthy (whitelisted) source. Given this information, the user now
needs to select a source and tap on the red “X” if they think the shown fact is fake or they
choose the green heart otherwise.

DOI: 10.4236/ce.2020.115050 686 Creative Education

 R. Cohen et al.

• Students are not allowed to use electronic devices: In this setting, we can
use an iClicker-type device (Bruff, 2009) which has a unique ID that can be
used to “add” a student in the class and receive their responses. Once all the
students are added to “DCR-01,” the teacher now projects the facts one-by-one
onto a screen using a projector; the screen-layout in this case would be iden-
tical to the previous case, only the question is now being shown as a presen-
tation as opposed to in a game. Once the students go through the entire con-
tent of the screen, they need to register their two responses using the iClick-
er-type device.
Finally, when the game concludes, the teacher will have access to a leaderboard
corresponding to the game session that can be shown to the students. Addition-
ally, the teacher would also receive a detailed report which would contain every
student’s submitted response so that, in the future, they can help the students
improve by educating them about their mistakes.

4. The Placement of Our New Methods in Classroom Settings,

Curricula
In this section we reflect further on where our new proposals may be integrated
into current secondary school teaching structures, in Ontario. We begin by com-
menting on the possible roles for educators when students are introduced to the
content we propose above. We then provide specific ideas for the placement of
our educational modules.
Educators are expected to be involved in engaging the class in viewing the dif-
ferent examples that are presented, devoting class time for students to run the
various software in order to test their knowledge. Class discussion (small group
or the entire class at once) of the differing responses may lead to greater overall
insights.
Delivering the content proposed in this paper within a classroom setting would
enable important face to face networking and peer support. Educators can view
our proposals as a resource of use in their classrooms. There then remains the
question of which subjects within the Ontario Ministry of Education secondary
school curriculum (Ontario Ministry of Education, 2020) would offer the best
environment for this material. Our local teaching experts have suggested that
English or Social Sciences or History (under Canadian and World Studies) may
all be appropriate, as students are already learning how to select the best sources
to cite for their essays. Students could also be educated about the important issue
of bias in reporting, if this material is integrated into the digital literacy agenda.
Civics is thus another area of study where our proposals fit the mandate well.
This subtopic seems to be showcased considerably under the umbrella of Social
Sciences and Humanities, where at the secondary level students are urged to
“develop... their literacy skills... to challenge texts”. There is as well an explicit
listing of Civics within the curriculum for Canadian and World Studies, as well
as a very interesting, relevant Citizenship Education Framework. In fact, educa-

DOI: 10.4236/ce.2020.115050 687 Creative Education

R. Cohen et al.

tors at the national level have already begun to discuss an important agenda
within schools for the general topic of digital and media literacy, where good ci-
tizenship online may be promoted but where consumer awareness is also a cru-
cial consideration (MediaSmarts, n.d.). Educating about the credibility of infor-
mation would fit nicely here. A final obvious home for educating students about
digital misinformation within the Ontario curriculum would be as part of Tech-
nological Education and the Inherent Roles and Responsibilities which are out-
lined. The Computer Technology outline for Grade 11 and 12, for instance, lists
expectations surrounding Technology and Society including discussing draw-
backs for society7.
Another possible option for delivery is an online course dedicated to digital
literacy, one that could perhaps be mandatory for all students to complete as
part of their graduation requirements. There is already some precedent for ob-
liging all students to pass the Ontario Secondary School Literacy Test (OSSLT)
(Education Quality and Accountability Office, 2020) in Grade 10, so that expand-
ing the vision of literacy to the digital world is not unreasonable. Online delivery
of the topics proposed in this paper could also be achieved in a course that is op-
tional and that may be selected as part of the looming mandate of the Ontario
government that all secondary students complete at least 2 courses online (Miller,
2020). An e-learning course on cyber safety might actually appeal to students.
This may still be arranged as informational with online grading as each module
is completed and would not be meant as pre-requisites to other courses. Aiming
for an exposure level half-credit course to be completed by graduation so that stu-
dents will have contact with the essential basic information may in fact be ideal.
The course mandate could even expand to continue to educate students about
cybersecurity as well, a topic recently outlined in Blackberry’s 2020 Threat Re-
port (BlackBerry, 2020).
Several cautions apply for the suggestions of running competitive games in a
classroom setting. The focus should not be on whether a student has won or lost,
but instead retention of the primary lessons about misinformation and credibility
should be paramount. Whether gamification in education is a positive or a nega-
tive is still a topic of much discussion amongst educators (Walker, 2014; Zaino,
2013; Cantador & Conde, 2010). It is also important to note that most course
curricula are already quite packed, leaving little room for expansion of content.
In order for these additions to be contemplated, educators, school boards and
the Ontario Ministry of Education will need to examine the larger picture for
that subject’s curriculum.

5. Conclusion and Recommendations for the Future

In this section, we highlight the main contributions of the methods that we are
proposing for educating students. We comment as well on challenges and short-
comings, leading to proposals for next steps.
7
We are aware that actually approving changes to provincial curriculum documents is often slow to
happen.

DOI: 10.4236/ce.2020.115050 688 Creative Education

 R. Cohen et al.

The primary advantage of the methods we propose for educating Ontario Sec-
ondary School students about credibility of online information, is to cover a
breadth of contexts that are particularly relevant to this demographic, as well as
to reveal specific issues such as fake reviews and targetted advertising that go
beyond the concerns drawn out by the Stanford project (Wineburg et al., 2016,
2020). Our approach also introduces some gamification in order to engage the
students, and supports running this education on mobile devices.
It is important to note that the proposals presented in this paper were created
by a team of computer scientists and educators. One of the coauthors in fact is a
computer science graduate student who previously was a teacher in the Ontario
secondary school system. One very significant hallmark of our work is clarifying
how to leverage technical solutions towards the teaching of this very important
topic (which also has a specific technical bent). But we integrate as well reflec-
tion on best practices in education, when clarifying how we propose to deliver
each of the different student experiences that we are creating.
We cannot emphasize enough the need to educate young people today specif-
ically about the topic of digital misinformation. According to a Pew Research
Report in 2017 (J. Anderson & Rainie, 2017), experts may be divided about whether
false narratives online will increase or reduce. Some promise may even poten-
tially come from automated approaches such as using artificial intelligence trust
modeling methods to identify questionable sources through an analysis of con-
tent (Ohashi, Cohen, & Fu, 2017). But an entire subgroup of the prominent res-
pondents of the Pew survey believed that the information environment will im-
prove explicitly because people will become more adept at sorting through ma-
terial. They went on to explain that information literacy must be elevated to be a
primary goal at all levels of education. Outlining the steps that can be taken to
realize this important achievement is the primary topic of this paper.

Directions Moving Forward

One direction for the future is to continue to develop valuable content for the
lessons. At least one example that is more specific to the province of Ontario
(that of Justin Bieber (Alexander, 2018; Global News, 2018)) has been suggested
and would urge the designers of the systems being run to delve into Canadian
examples, beyond those introduced for the Stanford effort as well. One challenge
of maintaining an example-based teaching of digital literacy is the requirement
of some group of technical experts to continue to locate relevant examples, and to
refine the algorithms which are running behind the games that may be launched.
At the very least, this may be an additional cost for the education system to bear.
Continued refinement of the systems that are designed, in terms of their logic
and their interfaces, will be an ongoing issue as well.
We also acknowledge that the online experiences of youth will evolve over time.
For example, while Instagram was showcased as a social media environment with
significant current attraction, different networking platforms may come to be
more popular in the future. In fact, completely novel ways for acquiring informa-

DOI: 10.4236/ce.2020.115050 689 Creative Education

R. Cohen et al.

tion and for communicating may become the order of the day, tomorrow. While
the methods and approaches presented in this paper are designed to still be func-
tional in entirely new contexts, it will always be useful for educators and tech-
nology experts to be aware of the current attention of youth. Valuable opportuni-
ties to bring together researchers from different disciplines have occurred recently
(such as Joel Breakstone’s introduction to the Stanford History project at the
2017 Weblogs and Social Media conference (ICWSM 2017 Workshop on Digital
Misinformation, 2017; Ciampaglia, Mantzarlis, Maus, & Menczer, 2018)), a venue
with heavy participation from computer scientists) and efforts such as these
must continue.

Conflicts of Interest
The authors declare no conflicts of interest regarding the publication of this pa-
per.

References
Alexander, J. (2018). The Bizarre Justin Bieber Burrito Incident Reminds Us Not to Be-
lieve Everything Online. The Verge.
https://www.theverge.com/2018/10/29/18037402/justin-bieber-burrito-yes-theory-pran
k-youtube
Anderson, J., & Rainie, L. (2017). The Future of Truth and Misinformation Online. Pew
Research Center.
https://www.pewresearch.org/internet/2017/10/19/the-future-of-truth-and-misinforma
tion-online
Anderson, M., & Jiang, J. (2018). Teens, Social Media & Technology. Pew Research Center.
https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learn-
ing to Align and Translate.
BlackBerry, C. T. (2020). The Blackberry Cylance 2020 Threat Report.
https://www.cylance.com/en-us/resources/knowledge-center/2020-threat-report.html
Bruff, D. (2009). Teaching with Classroom Response Systems: Creating Active Learning
Environments. Hoboken, NJ: John Wiley & Sons.
Cantador, I., & Conde, J. M. (2010). Effects of Competition in Education: A Case Study in
an e-Learning Environment. In Proceedings of the 2010 IADIS International Confe-
rence on e-Learning (pp. 11-18). Freiburg, Germany.
Ciampaglia, G. L., Mantzarlis, A., Maus, G., & Menczer, F. (2018). Research Challenges of
Digital Misinformation: Toward a Trustworthy Web. AI Magazine, 39, 65-74.
https://doi.org/10.1609/aimag.v39i1.2783
Crouch, C. H., & Mazur, E. (2001). Peer Instruction: Ten Years of Experience and Re-
sults. American Journal of Physics, 69, 970-977. https://doi.org/10.1119/1.1374249
Das, R., Munkhdalai, T., Yuan, X., Trischler, A., & McCallum, A. (2018). Building Dy-
namic Knowledge Graphs from Text Using Machine Reading Comprehension.
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016). Botornot: A Sys-
tem to Evaluate Social Bots. In Proceedings of the 25th International Conference on
World Wide Web (pp. 273-274). Montreal, Canada. Geneva, Switzerland: International
World Wide Web Conferences Steering Committee.
https://doi.org/10.1145/2872518.2889302

DOI: 10.4236/ce.2020.115050 690 Creative Education

 R. Cohen et al.

Dellos, R. (2015). Kahoot! A Digital Game Resource for Learning. International Journal of
Instructional Technology and Distance Learning, 12, 49-52.
Donald, B. (2016). Stanford Researchers Find Students Have Trouble Judging the Credi-
bility of Information Online. Stanford Graduate School of Education: Research Stories.
https://ed.stanford.edu/news/stanford-researchers-find-students-have-trouble-judging-
credibility-information-online
Education Quality and Accountability Office (2020). The Ontario School Literacy Test.
https://www.eqao.com/en/assessments/osslt
Elmurngi, E., & Gherbi, A. (2017). Detecting Fake Reviews through Sentiment Analysis
Using Machine Learning Techniques. In IARIA 2017 Conference on Data Analytics
(pp. 65-72). Barcelona, Spain: IARIA XPS Press.
Global News (2018). Here’s How a Fake Photo of Justin Bieber Eating a Burrito Fooled
the World. https://globalnews.ca/news/4606909/justin-bieber-burrito
Hochreiter, S., & Schmidhuber, J. (1996). LSTM Can Solve Hard Long Time Lag Prob-
lems. In Proceedings of 1996 Neural Information Processing Systems (pp. 473-479).
Denver, United States: MIT Press.
ICWSM 2017 Workshop on Digital Misinformation (2017).
https://cnets.indiana.edu/blog/2016/12/29/icwsm-2017-misinformation-workshop
Jurca, R., & Faltings, B. (2009). Mechanisms for Making Crowds Truthful. Journal of Ar-
tificial Intelligence Research, 34, 209-253. https://doi.org/10.1613/jair.2621
Mazur, E., & Hilborn, R. C. (1997). Peer Instruction: A User’s Manual. Upper Saddle
River, NJ: Prentice Hall. https://doi.org/10.1063/1.881735
MediaSmarts (n.d.). Digital Literacy Fundamentals. Canada’s Centre for Digital and Me-
dia Literacy.
https://mediasmarts.ca/digital-media-literacy/general-information/digital-media-litera
cy-fundamentals/digital-literacy-fundamentals
Miller, J. (2020). Plan for Mandatory Online High School Courses Shrouded in Mystery.
Ottawa Citizen.
https://ottawacitizen.com/news/local-news/plan-for-mandatory-online-high-courses-s
hrouded-in-controversy-and-mystery
Mukherjee, A., Liu, B., & Glance, N. (2012). Spotting Fake Reviewer Groups in Consumer
Reviews. In Proceedings of the 21st International Conference on World Wide Web (pp.
191-200). Lyon, France: Association for Computer Machinery.
https://doi.org/10.1145/2187836.2187863
Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., & Deng, L.
(2016). Ms Marco: A Human-Generated Machine Reading Comprehension Dataset.
Ohashi, D., Cohen, R., & Fu, X. (2017). The Current State of Online Social Networking
for the Health Community: Where Trust Modeling Research May Be of Value. In Pro-
ceedings of the ACM 2017 International Conference on Digital Health (pp. 23-32).
London, UK: Association for Computer Machinery.
https://doi.org/10.1145/3079452.3079462
Ontario Ministry of Education (2020). The Ontario Curriculum: Secondary.
http://www.edu.gov.on.ca/eng/curriculum/secondary
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Chintala, S. et al.
(2019). Pytorch: An Imperative Style, High-Performance Deep Learning Library. In H.
Wallach, H. Larochelle, A. Beygelzimer, F. D. Alche’-Buc, E. Fox, & R. Garnett (Eds.),
Advances in Neural Information Processing Systems 32 (pp. 8024-8035). New York:
Curran Associates, Inc.

DOI: 10.4236/ce.2020.115050 691 Creative Education

R. Cohen et al.

Pesce, N. L. (2019). Kylie Jenner Can Make More Money in One Instagram Post than
Many People Earn in a Lifetime. MarketWatch.
https://www.marketwatch.com/story/kylie-jenner-can-make-more-money-in-one-insta
gram-post-than-many-people-earn-in-a-lifetime-2019-07-23
Picchi, A. (2019). Buyer Beware: Scourge of Fake Reviews Hitting Amazon, Walmart and
Other Major Retailers. CBS Interactive.
https://www.cbsnews.com/news/buyer-beware-a-scourge-of-fake-online-reviews-is-hit
ting-amazon-walmart-and-other-major-retailers
Talwar, A., Jurca, R., & Faltings, B. (2007). Understanding User Behavior in Online
Feedback Reporting. In Proceedings of the 8th ACM Conference on Electronic Com-
merce (pp. 134-142). San Diego, United States: Association for Computer Machinery.
https://doi.org/10.1145/1250910.1250931
Vosoughi, S., Roy, D., & Aral, S. (2018). The Spread of True and False News Online. Science,
359, 1146-1151. https://doi.org/10.1126/science.aap9559
Walker, T. (2014). Gamification in the Classroom: The Right or Wrong Way to Motivate
Students. NeaToday: News and Features from the National Education Association.
Wineburg, S., & McGrew, S. (2016). Why Students Can’t Google Their Way to the Truth.
Education Week, 36, 22-28.
Wineburg, S., Breakstone, J., Smith, M., McGrew, S., Ortega, T., & Kerr, D. (2020). COR:
Civic Online Reasoning Curriculum Resources.
https://cor.stanford.edu/?page=0main-content
Wineburg, S., McGrew, S., Breakstone, J., & Ortega, T. (2016). Evaluating Information:
The Cornerstone of Civic Online Reasoning. Stanford Digital Repository.
Wong, Q. (2019). Deepfakes Are Coming. Facebook, Twitter and Youtube Might Not Be
Ready. CNET.
https://www.cnet.com/news/facebook-twitter-and-youtube-grapple-with-altered-video
s-ahead-of-the-2020-election
Zaino, J. (2013). The Pros and Cons of Gamification in the Classroom. EdTech Magazine.
https://edtechmagazine.com/higher/article/2013/07/pros-and-cons-gamification-classroom

DOI: 10.4236/ce.2020.115050 692 Creative Education