GPT-3

Generative Pre-trained Transformer 3 (GPT-3) is a large

language model released by OpenAI in 2020. Like its predecessor Generative Pre-trained
GPT-2, it is a decoder-only transformer model of deep neural Transformer 3 (GPT-3)
network, which uses attention in place of previous recurrence- and Original author(s) OpenAI[1]
convolution-based architectures.[2] Attention mechanisms allow Initial release June 11, 2020
the model to selectively focus on segments of input text it predicts
(beta)
to be the most relevant.[3] It uses a 2048-tokens-long context and
then-unprecedented size of 175 billion parameters, requiring Repository github.com
800GB to store. The model demonstrated strong zero-shot and /openai/gpt-3
few-shot learning on many tasks.[4] (https://github.
com/openai/gp
Microsoft announced on September 22, 2020, that it had licensed t-3)
"exclusive" use of GPT-3; others can still use the public API to
receive output, but only Microsoft has access to GPT-3's Predecessor GPT-2
underlying model.[5] Successor GPT-3.5
Type Large
Background language
model
According to The Economist, improved algorithms, powerful
Generative
computers, and an increase in digitized data have fueled a
pre-trained
revolution in machine learning, with new techniques in the 2010s
resulting in "rapid improvements in tasks" including manipulating transformer
language.[6] Software models are trained to learn by using Foundation
thousands or millions of examples in a "structure ... loosely based model
on the neural architecture of the brain".[6] One architecture used in Website openai.com
natural language processing (NLP) is a neural network based on a
/blog/openai-
deep learning model that was first introduced in 2017—the
api (https://ope
transformer architecture.[7] There are a number of NLP systems
capable of processing, mining, organizing, connecting and nai.com/blog/o
contrasting textual input, as well as correctly answering penai-api)
questions.[8]

On June 11, 2018, OpenAI researchers and engineers posted their original paper introducing the first
generative pre-trained transformer (GPT)—a type of generative large language model that is pre-trained
with an enormous and diverse corpus of text via datasets, followed by discriminative fine-tuning to focus
on a specific task. GPT models are transformer-based deep learning neural network architectures. Up to
that point, the best-performing neural NLP models commonly employed supervised learning from large
amounts of manually-labeled data, which made it prohibitively expensive and time-consuming to train
extremely large language models.[4] That first GPT model is known as "GPT-1," and it was then followed
by "GPT-2" in February 2019. GPT-2 was created as a direct scale-up of GPT-1, with both its parameter
count and dataset size increased by a factor of 10. It had 1.5 billion parameters, and was trained on a dataset
of 8 million web pages.[9]
In February 2020, Microsoft introduced its Turing Natural Language Generation (T-NLG), which was
claimed to be the "largest language model ever published at 17 billion parameters."[10] It performed better
than any other language model at a variety of tasks which included summarizing texts and answering
questions.

Training and capabilities

On May 28, 2020, an arXiv preprint by a group of 31 A sample student essay about pedagogy
engineers and researchers at OpenAI described the written by GPT-3
achievement and development of GPT-3, a third- The construct of “learning styles” is
generation "state-of-the-art language model".[1][12] The problematic because it fails to account for
team increased the capacity of GPT-3 by over two the processes through which learning styles
orders of magnitude from that of its predecessor, GPT- are shaped. Some students might develop a
2,[13] making GPT-3 the largest non-sparse language particular learning style because they have
model to date.[1]: 1 4 [14] Because GPT-3 is structurally had particular experiences. Others might
similar to its predecessors,[1] its greater accuracy is develop a particular learning style by trying
attributed to its increased capacity and greater number of to accommodate to a learning environment
parameters.[15] GPT-3's capacity is ten times larger than that was not well suited to their learning
that of Microsoft's Turing NLG, the next largest NLP needs. Ultimately, we need to understand
model known at the time.[12] the interactions among learning styles and
environmental and personal factors, and
Lambdalabs estimated a hypothetical cost of around how these shape how we learn and the
$4.6 million US dollars and 355 years to train GPT-3 on kinds of learning we experience.
a single GPU in 2020,[16] with lower actual training
time by using more GPUs in parallel.
– Text generated by Mike Sharples[11]
Sixty percent of the weighted pre-training dataset for
GPT-3 comes from a filtered version of Common Crawl
consisting of 410 billion byte-pair-encoded tokens.[1]: 9  
Other sources are 19 billion tokens from WebText2 representing 22% of the weighted total, 12 billion
tokens from Books1 representing 8%, 55 billion tokens from Books2 representing 8%, and 3 billion tokens
from Wikipedia representing 3%.[1]: 9   GPT-3 was trained on hundreds of billions of words and is also
capable of coding in CSS, JSX, and Python, among others.[17]

GPT-3 training data[1]: 9 

Proportion
Dataset # tokens
within training
Common Crawl 410 billion 60%

WebText2 19 billion 22%

Books1 12 billion 8%
Books2 55 billion 8%

Wikipedia 3 billion 3%

Since GPT-3's training data was all-encompassing, it does not require further training for distinct language
tasks.[17] The training data contains occasional toxic language and GPT-3 occasionally generates toxic
language as a result of mimicking its training data. A study from the University of Washington found that
GPT-3 produced toxic language at a toxicity level comparable to the similar natural language processing
models of GPT-2 and CTRL. OpenAI has implemented several strategies to limit the amount of toxic
language generated by GPT-3. As a result, GPT-3 produced less toxic language compared to its
predecessor model, GPT-1, although it produced both more generations and a higher toxicity of toxic
language compared to CTRL Wiki, a language model trained entirely on Wikipedia data.[18]

On June 11, 2020, OpenAI announced that users could request access to its user-friendly GPT-3 API—a
"machine learning toolset"—to help OpenAI "explore the strengths and limits" of this new
technology.[19][20] The invitation described how this API had a general-purpose "text in, text out" interface
that can complete almost "any English language task", instead of the usual single use-case.[19] According to
one user, who had access to a private early release of the OpenAI GPT-3 API, GPT-3 was "eerily good" at
writing "amazingly coherent text" with only a few simple prompts.[21] In an initial experiment 80 US
subjects were asked to judge if short ~200 word articles were written by humans or GPT-3. The
participants judged correctly 52% of the time, doing only slightly better than random guessing.[1]

On November 18, 2021, OpenAI announced that enough safeguards had been implemented that access to
its API would be unrestricted.[22] OpenAI provided developers with a content moderation tool that helps
them abide by OpenAI's content policy.[23] On January 27, 2022, OpenAI announced that its newest GPT-
3 language models, collectively referred to as InstructGPT, was now the default language model used on
their API. According to OpenAI, InstructGPT produced content that was better aligned to user intentions
by following instructions better, generating fewer made-up facts, and producing somewhat less toxic
content.[24]

Because GPT-3 can "generate news articles which human evaluators have difficulty distinguishing from
articles written by humans,"[12] GPT-3 has the "potential to advance both the beneficial and harmful
applications of language models."[1]: 3 4  In their May 28, 2020 paper, the researchers described in detail the
potential "harmful effects of GPT-3"[12] which include "misinformation, spam, phishing, abuse of legal and
governmental processes, fraudulent academic essay writing and social engineering pretexting".[1] The
authors draw attention to these dangers to call for research on risk mitigation.[1]: 3 4 

GPT-3 is capable of performing zero-shot and few-shot learning (including one-shot).[1]

In June 2022, Almira Osmanovic Thunström wrote that GPT-3 was the primary author on an article on
itself, that they had submitted it for publication,[25] and that it had been pre-published while waiting for
completion of its review.[26]

InstructGPT
InstructGPT is a finetuned version of GPT-3. It has been trained on a dataset of human-written instructions.
This training allows InstructGPT to better understand what is being asked of it, and to generate more
accurate and relevant outputs.

InstructGPT can be used to follow instructions that are given in natural language.
InstructGPT can be used to answer questions that are asked in natural language.
InstructGPT is more accurate and relevant than GPT-3 when following instructions and
answering questions.
InstructGPT can be used in a variety of applications, such as customer service, education,
and automation.

GPT-3 models
There are many models in the GPT-3 family, some serving different purposes than others. In the initial
research paper published by OpenAI, they mentioned 8 different sizes of the main GPT-3 model:

Model name Parameters API name

GPT-3 Small 125 M n/a

GPT-3 Medium 350 M ada

GPT-3 Large 760 M n/a

GPT-3 XL 1.3 B babbage

GPT-3 2.7B 2.7 B n/a

GPT-3 6.7B 6.7 B curie

GPT-3 13B 13B n/a

GPT-3 175B 175B davinci

Half of the models are accessible through the API, namely GPT-3-small, GPT-3-xl, GPT-3-6.7B and GPT-
3-175b, which are referred to as ada, babbage, curie and davinci respectively.

Model Parameters Description Series

Capable of very simple tasks, usually the fastest model in the GPT-
ada 350 M Base GPT-3
3 series, and lowest cost.
babbage 1.3 B Capable of straightforward tasks, very fast, and lower cost. Base GPT-3

curie 6.7B Very capable, but faster and lower cost than Davinci. Base GPT-3

Most capable GPT-3 model. Can do any task the other models can
davinci 175 B Base GPT-3
do, often with higher quality.
Capable of very simple tasks, usually the fastest model in the GPT- InstructGPT-
text-ada 350 M
3 series, and lowest cost. 3

text- InstructGPT-
175B Capable of straightforward tasks, very fast, and lower cost.
babbage 3

InstructGPT-
text-curie 6.7B Very capable, faster and lower cost than Davinci.
3
text- Older version of the most capable model in the GPT-3 series. Can
InstructGPT-
davinci- 175B perform any task the other GPT-3 models can, often with less
3
001 context.

text-
Similar capabilities to text-davinci-003 but trained with
davinci- 175B GPT-3.5
supervised fine-tuning instead of reinforcement learning
002

text- Can do any language task with better quality, longer output, and
davinci- 175B consistent instruction-following than the curie, babbage, or ada GPT-3.5
003 models. Also supports inserting completions within text.

gpt-3.5- Most capable GPT-3.5 model and optimized for chat at 1/10th the
175B GPT-3.5
turbo cost of text-davinci-003.

GPT-3.5
Generative Pre-trained Transformer 3.5 (GPT-3.5) is a sub Generative Pre-trained
class of GPT-3 Models created by OpenAI in 2022. Transformer 3.5 (GPT-3.5)
On March 15, 2022, OpenAI made available new versions of Original author(s) OpenAI[1]
GPT-3 and Codex in its API with edit and insert capabilities under
Initial release March 15,
the names "text-davinci-002" and "code-davinci-002".[27] These
2022
models were described as more capable than previous versions and
were trained on data up to June 2021.[28] On November 28, 2022, Repository n/a
OpenAI introduced text-davinci-003.[29] On November 30, 2022, Predecessor GPT-3
OpenAI began referring to these models as belonging to the "GPT-
Successor GPT-4
3.5" series,[28] and released ChatGPT, which was fine-tuned from
a model in the GPT-3.5 series.[30] OpenAI does not include GPT- Type Large
3.5 in GPT-3.[31] language
model
Generative
Models
pre-trained
transformer
There are four models:[32]
Foundation
Chat model
gpt-3.5-turbo License Proprietary
Text completion Website n/a
text-davinci-003
text-davinci-002

GPT-3.5 with browsing

On April 10, 2023, OpenAI introduced a new variant of its GPT-3.5 series model, known as GPT-3.5 with
Browsing (ALPHA).[33] This updated model was described to build upon the capabilities of its
predecessors "text-davinci-002" and "code-davinci-002".[34] The GPT-3.5 with Browsing (ALPHA)
model incorporated the ability to access and browse online information. This has led to more accurate and
up-to-date responses to user queries.[33]

The GPT-3.5 with Browsing (ALPHA) model has been trained on data up to September 2021, giving it
more information compared to previous GPT-3.5 models, which were trained on data up until June 2021.
The model attempted to provide developers and users with an advanced natural language processing tool
that can effectively retrieve and synthesize online information.[33]

To enable browsing capabilities, OpenAI implemented a new API that allows the GPT-3.5 with Browsing
(ALPHA) model to access selected online resources during operation.[35] This feature allows users to ask
questions or request information with the expectation that the model will deliver updated, accurate, and
relevant answers based on the latest online sources available to it.
On April 27, 2023, OpenAI made the GPT-3.5 with Browsing (ALPHA) model publicly available to GPT
Plus users. This allowed more people to access to its new features.[35]

Reception

Applications
GPT-3, specifically the Codex model, is the basis for GitHub Copilot, a code completion and
generation software that can be used in various code editors and IDEs.[36][37]
GPT-3 is used in certain Microsoft products to translate conventional language into formal
computer code.[38][39]
GPT-3 has been used in CodexDB[40] to generate query-specific code for SQL processing.
GPT-3 has been used by Jason Rohrer in a retro-themed chatbot project named "Project
December", which is accessible online and allows users to converse with several AIs using
GPT-3 technology.[41]
GPT-3 was used by The Guardian to write an article about AI being harmless to human
beings. It was fed some ideas and produced eight different essays, which were ultimately
merged into one article.[42]
GPT-3 was used in AI Dungeon, which generates text-based adventure games. Later it was
replaced by a competing model after OpenAI changed their policy regarding generated
content.[43][44]
GPT-3 is used to aid in writing copy and other marketing materials.[45]
A 2022 study from Drexel University suggested that GPT-3-based systems could be used to
screen for early signs of Alzheimer's disease.[46][47]

Reviews
In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to
generate computer code, poetry, and prose is not just "amazing", "spooky", and "humbling",
but also "more than a little terrifying".[48]
Daily Nous presented a series of articles by nine philosophers on GPT-3.[49] Australian
philosopher David Chalmers described GPT-3 as "one of the most interesting and important
AI systems ever produced".[50]
A review in Wired said that GPT-3 was "provoking chills across Silicon Valley".[51]
The National Law Review said that GPT-3 is an "impressive step in the larger process", with
OpenAI and others finding "useful applications for all of this power" while continuing to
"work toward a more general intelligence".[52]
An article in the MIT Technology Review, co-written by Deep Learning critic Gary Marcus,[53]
stated that GPT-3's "comprehension of the world is often seriously off, which means you can
never really trust what it says."[54] According to the authors, GPT-3 models relationships
between words without having an understanding of the meaning behind each word.
Jerome Pesenti, head of the Facebook AI lab, said GPT-3 is "unsafe," pointing to the sexist,
racist and other biased and negative language generated by the system when it was asked
to discuss Jews, women, black people, and the Holocaust.[55]
Nabla, a French start-up specializing in healthcare technology, tested GPT-3 as a medical
chatbot, though OpenAI itself warned against such use. As expected, GPT-3 showed several
 limitations. For example, while testing GPT-3 responses about mental health issues, the AI
advised a simulated patient to commit suicide.[56]
Noam Chomsky expressed his skepticism about GPT-3's scientific value: "It's not a
language model. It works just as well for impossible languages as for actual languages. It is
therefore refuted, if intended as a language model, by normal scientific criteria. [...] Perhaps
it's useful for some purpose, but it seems to tell us nothing about language or cognition
generally."[57]
Luciano Floridi and Massimo Chiriatti highlighted the risk of "cheap production of good,
semantic artefacts".[58]
OpenAI's Sam Altman himself criticized what he called "GPT-3 hype", acknowledging GPT-3
"has serious weakness and sometimes makes very silly mistakes... AI is going to change the
world, but GPT-3 is just a very early glimpse."[59]

Criticism

GPT-3's builder, OpenAI, was initially founded as a non-profit in 2015.[60] In 2019, OpenAI broke from its
usual open-source standards by not publicly releasing GPT-3's predecessor model, citing concerns that the
model could facilitate the propagation of fake news. OpenAI eventually released a version of GPT-2 that
was 8% of the original model's size.[61] In the same year, OpenAI restructured to be a for-profit
company.[62] In 2020, Microsoft announced the company had exclusive licensing of GPT-3 for Microsoft's
products and services following a multi-billion dollar investment in OpenAI. The agreement permits
OpenAI to offer a public-facing API such that users can send text to GPT-3 to receive the model's output,
but only Microsoft will have access to GPT-3's source code.[5]

Large language models, such as GPT-3, have come under criticism from a few of Google's AI ethics
researchers for the environmental impact of training and storing the models, detailed in a paper co-authored
by Timnit Gebru and Emily M. Bender in 2021.[63]

The growing use of automated writing technologies based on GPT-3 and other language generators, has
raised concerns regarding academic integrity[64] and raised the stakes of how universities and schools will
gauge what constitutes academic misconduct such as plagiarism.[65]

OpenAI's GPT series was built with data from the Common Crawl dataset,[66] a conglomerate of
copyrighted articles, internet posts, web pages, and books scraped from 60 million domains over a period of
12 years. TechCrunch reports this training data includes copyrighted material from the BBC, The New York
Times, Reddit, the full text of online books, and more.[67] In its response to a 2019 Request for Comments
on Intellectual Property Protection for Artificial Intelligence Innovation from the United States Patent and
Trademark Office (USPTO), OpenAI argued that "Under current law, training AI systems [such as its GPT
models] constitutes fair use," but that "given the lack of case law on point, OpenAI and other AI developers
like us face substantial legal uncertainty and compliance costs."[68]

See also
BERT (language model)
Hallucination (artificial intelligence)
LaMDA
Wu Dao
 GPT-4

References
1. Brown, Tom B.; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal,
Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal,
Sandhini; Herbert-Voss, Ariel; Krueger, Gretchen; Henighan, Tom; Child, Rewon; Ramesh,
Aditya; Ziegler, Daniel M.; Wu, Jeffrey; Winter, Clemens; Hesse, Christopher; Chen, Mark;
Sigler, Eric; Litwin, Mateusz; Gray, Scott; Chess, Benjamin; Clark, Jack; Berner, Christopher;
McCandlish, Sam; Radford, Alec; Sutskever, Ilya; Amodei, Dario (May 28, 2020). "Language
Models are Few-Shot Learners". arXiv:2005.14165 (https://arxiv.org/abs/2005.14165).
2. Polosukhin, Illia; Kaiser, Lukasz; Gomez, Aidan N.; Jones, Llion; Uszkoreit, Jakob; Parmar,
Niki; Shazeer, Noam; Vaswani, Ashish (June 12, 2017). "Attention Is All You Need".
arXiv:1706.03762 (https://arxiv.org/abs/1706.03762) [cs.CL (https://arxiv.org/archive/cs.CL)].
3. Bahdanau, Dzmitry; Cho, Kyunghyun; Bengio, Yoshua (September 1, 2014). "Neural
Machine Translation by Jointly Learning to Align and Translate". arXiv:1409.0473 (https://arx
iv.org/abs/1409.0473) [cs.CL (https://arxiv.org/archive/cs.CL)].
4. Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (June 11, 2018).
"Improving Language Understanding by Generative Pre-Training" (https://cdn.openai.com/re
search-covers/language-unsupervised/language_understanding_paper.pdf) (PDF). p. 12.
Archived (https://web.archive.org/web/20210126024542/https://cdn.openai.com/research-co
vers/language-unsupervised/language_understanding_paper.pdf) (PDF) from the original on
January 26, 2021. Retrieved July 31, 2020.
5. Hao, Karen (September 23, 2020). "OpenAI is giving Microsoft exclusive access to its GPT-3
language model" (https://www.technologyreview.com/2020/09/23/1008729/openai-is-giving-
microsoft-exclusive-access-to-its-gpt-3-language-model/). MIT Technology Review. Archived
(https://web.archive.org/web/20210205121656/https://www.technologyreview.com/2020/09/2
3/1008729/openai-is-giving-microsoft-exclusive-access-to-its-gpt-3-language-model/) from
the original on February 5, 2021. Retrieved September 25, 2020. "The companies say
OpenAI will continue to offer its public-facing API, which allows chosen users to send text to
GPT-3 or OpenAI's other models and receive its output. Only Microsoft, however, will have
access to GPT-3's underlying code, allowing it to embed, repurpose, and modify the model
as it pleases."
6. "An understanding of AI's limitations is starting to sink in" (https://www.economist.com/techn
ology-quarterly/2020/06/11/an-understanding-of-ais-limitations-is-starting-to-sink-in). The
Economist. June 11, 2020. ISSN 0013-0613 (https://www.worldcat.org/issn/0013-0613).
Archived (https://web.archive.org/web/20200731060114/https://www.economist.com/technol
ogy-quarterly/2020/06/11/an-understanding-of-ais-limitations-is-starting-to-sink-in) from the
original on July 31, 2020. Retrieved July 31, 2020.
7. Polosukhin, Illia; Kaiser, Lukasz; Gomez, Aidan N.; Jones, Llion; Uszkoreit, Jakob; Parmar,
Niki; Shazeer, Noam; Vaswani, Ashish (June 12, 2017). "Attention Is All You Need".
arXiv:1706.03762 (https://arxiv.org/abs/1706.03762) [cs.CL (https://arxiv.org/archive/cs.CL)].
8. "Natural Language Processing" (https://www.thomsonreuters.com/en/artificial-intelligence/n
atural-language-processing.html). Archived (https://web.archive.org/web/20200822144104/h
ttps://www.thomsonreuters.com/en/artificial-intelligence/natural-language-processing.html)
from the original on August 22, 2020. Retrieved July 31, 2020.
9. "Archived copy" (https://cdn.openai.com/better-language-models/language_models_are_un
supervised_multitask_learners.pdf) (PDF). Archived (https://web.archive.org/web/20210206
183945/https://cdn.openai.com/better-language-models/language_models_are_unsupervise
d_multitask_learners.pdf) (PDF) from the original on February 6, 2021. Retrieved April 28,
2023.
10. Sterling, Bruce (February 13, 2020). "Web Semantics: Microsoft Project Turing introduces
Turing Natural Language Generation (T-NLG)" (https://www.wired.com/beyond-the-beyond/2
020/02/web-semantics-microsoft-project-turing-introduces-turing-natural-language-generatio
n-t-nlg/). Wired. ISSN 1059-1028 (https://www.worldcat.org/issn/1059-1028). Archived (http
s://web.archive.org/web/20201104163637/https://www.wired.com/beyond-the-beyond/2020/
02/web-semantics-microsoft-project-turing-introduces-turing-natural-language-generation-t-n
lg/) from the original on November 4, 2020. Retrieved July 31, 2020.
11. Marche, Stephen (December 6, 2022). "The College Essay Is Dead" (https://www.theatlanti
c.com/technology/archive/2022/12/chatgpt-ai-writing-college-student-essays/672371/). The
Atlantic. Archived (https://web.archive.org/web/20230124042209/https://www.theatlantic.co
m/technology/archive/2022/12/chatgpt-ai-writing-college-student-essays/672371/) from the
original on January 24, 2023. Retrieved December 8, 2022.
12. Sagar, Ram (June 3, 2020). "OpenAI Releases GPT-3, The Largest Model So Far" (https://a
nalyticsindiamag.com/open-ai-gpt-3-language-model/). Analytics India Magazine. Archived
(https://web.archive.org/web/20200804173452/https://analyticsindiamag.com/open-ai-gpt-3-l
anguage-model/) from the original on August 4, 2020. Retrieved July 31, 2020.
13. "Language Models are Unsupervised Multitask Learners" (https://d4mucfpksywv.cloudfront.n
et/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
(PDF). openai.com. Archived (https://web.archive.org/web/20191212223916/https://d4mucfp
ksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitas
k_learners.pdf) (PDF) from the original on December 12, 2019. Retrieved December 4,
2019. "GPT-2, is a 1.5B parameter Transformer"
14. Shead, Sam (July 23, 2020). "Why everyone is talking about the A.I. text generator released
by an Elon Musk-backed lab" (https://www.cnbc.com/2020/07/23/openai-gpt3-explainer.htm
l). CNBC. Archived (https://web.archive.org/web/20200730123130/https://www.cnbc.com/20
20/07/23/openai-gpt3-explainer.html) from the original on July 30, 2020. Retrieved July 31,
2020. Four preprints were released between May 28 and July 22, 2020.
15. Ray, Tiernan (June 1, 2020). "OpenAI's gigantic GPT-3 hints at the limits of language models
for AI" (https://www.zdnet.com/article/openais-gigantic-gpt-3-hints-at-the-limits-of-language-
models-for-ai/). ZDNet. Archived (https://web.archive.org/web/20200601081629/https://www.
zdnet.com/article/openais-gigantic-gpt-3-hints-at-the-limits-of-language-models-for-ai/) from
the original on June 1, 2020. Retrieved July 31, 2020.
16. Li, Chuan (June 3, 2020), OpenAI's GPT-3 Language Model: A Technical Overview (https://la
mbdalabs.com/blog/demystifying-gpt-3), archived (https://web.archive.org/web/2023032721
3811/https://lambdalabs.com/blog/demystifying-gpt-3) from the original on March 27, 2023,
retrieved March 27, 2023
17. Bussler, Frederik (July 21, 2020). "Will GPT-3 Kill Coding?" (https://towardsdatascience.co
m/will-gpt-3-kill-coding-630e4518c04d). Towards Data Science. Archived (https://web.archiv
e.org/web/20200819070430/https://towardsdatascience.com/will-gpt-3-kill-coding-630e4518
c04d) from the original on August 19, 2020. Retrieved August 1, 2020.
18. Gehman, Samuel; Gururangan, Suchin; Sap, Maarten; Choi, Yejin; Smith, Noah A.
(November 16–20, 2020), REALTOXICITYPROMPTS: Evaluating Neural Toxic
Degeneration in Language Models, Association for Computational Linguistics, pp. 3356–
3369, arXiv:2009.11462 (https://arxiv.org/abs/2009.11462)
19. "OpenAI API" (https://openai.com/blog/openai-api/). OpenAI. June 11, 2020. Archived (http
s://web.archive.org/web/20200611150951/https://openai.com/blog/openai-api/) from the
original on June 11, 2020. Retrieved July 31, 2020.
20. Coldewey, Devin (June 11, 2020). "OpenAI makes an all-purpose API for its text-based AI
capabilities" (https://techcrunch.com/2020/06/11/openai-makes-an-all-purpose-api-for-its-tex
t-based-ai-capabilities/). TechCrunch. Archived (https://web.archive.org/web/202110270000
59/https://techcrunch.com/2020/06/11/openai-makes-an-all-purpose-api-for-its-text-based-ai-
capabilities/) from the original on October 27, 2021. Retrieved July 31, 2020. "If you've ever
wanted to try out OpenAI's vaunted machine learning toolset, it just got a lot easier. The
company has released an API that lets developers call its AI tools in on "virtually any English
language task." "
21. Arram (July 9, 2020). "GPT-3: An AI that's eerily good at writing almost anything" (https://arr.a
m/2020/07/09/gpt-3-an-ai-thats-eerily-good-at-writing-almost-anything/). Arram Sabeti.
Archived (https://web.archive.org/web/20200720192137/https://arr.am/2020/07/09/gpt-3-an-a
i-thats-eerily-good-at-writing-almost-anything/) from the original on July 20, 2020. Retrieved
July 31, 2020.
22. "OpenAI's API Now Available with No Waitlist" (https://openai.com/blog/api-no-waitlist/).
OpenAI. November 18, 2021. Archived (https://web.archive.org/web/20221105195042/http
s://openai.com/blog/api-no-waitlist/) from the original on November 5, 2022. Retrieved
November 5, 2022.
23. "OpenAI API" (https://beta.openai.com/). beta.openai.com. Archived (https://web.archive.org/
web/20221223073027/https://beta.openai.com/) from the original on December 23, 2022.
Retrieved November 5, 2022.
24. "Aligning Language Models to Follow Instructions" (https://openai.com/blog/instruction-follo
wing/). OpenAI. January 27, 2022. Archived (https://web.archive.org/web/20221105195041/h
ttps://openai.com/blog/instruction-following/) from the original on November 5, 2022.
Retrieved November 5, 2022.
25. Thunström, Almira Osmanovic (June 30, 2022). "We Asked GPT-3 to Write an Academic
Paper about Itself – Then We Tried to Get It Published" (https://www.scientificamerican.com/
article/we-asked-gpt-3-to-write-an-academic-paper-about-itself-then-we-tried-to-get-it-publis
hed/). Scientific American. Archived (https://web.archive.org/web/20220630233635/https://w
ww.scientificamerican.com/article/we-asked-gpt-3-to-write-an-academic-paper-about-itself-th
en-we-tried-to-get-it-published/) from the original on June 30, 2022. Retrieved June 30,
2022.
26. Transformer, Gpt Generative Pretrained; Thunström, Almira Osmanovic; Steingrimsson,
Steinn (June 21, 2022). "Can GPT-3 write an academic paper on itself, with minimal human
input?" (https://hal.archives-ouvertes.fr/hal-03701250). Archive ouverte HAL (in French).
Archived (https://web.archive.org/web/20220630233635/https://hal.archives-ouvertes.fr/hal-0
3701250) from the original on June 30, 2022. Retrieved June 30, 2022.
27. "New GPT-3 Capabilities: Edit & Insert" (https://openai.com/blog/gpt-3-edit-insert/). OpenAI.
March 15, 2022. Archived (https://web.archive.org/web/20230113234402/https://openai.com/
blog/gpt-3-edit-insert/) from the original on January 13, 2023. Retrieved January 13, 2023.
28. "OpenAI API" (https://platform.openai.com/). platform.openai.com. Archived (https://web.archi
ve.org/web/20230320023933/https://platform.openai.com/) from the original on March 20,
2023. Retrieved March 15, 2023.
29. "Check out OpenAI's new text-davinci-003! Same underlying model as text-davinci-002 but
more aligned. Would love to hear feedback about it! / Twitter" (https://twitter.com/janleike/stat
us/1597355354433916928). Retrieved May 6, 2023.
30. "ChatGPT: Optimizing Language Models for Dialogue" (https://openai.com/blog/chatgpt/).
OpenAI. November 30, 2022. Archived (https://web.archive.org/web/20221130180912/http
s://openai.com/blog/chatgpt/) from the original on November 30, 2022. Retrieved January 13,
2023.
31. "OpenAI API" (https://platform.openai.com/docs/models). Retrieved May 6, 2023.
32. "OpenAI API" (https://platform.openai.com/docs/models/gpt-3-5). Retrieved May 6, 2023.
33. tingetici (April 10, 2023). "Default (GPT-3.5) with browsing ALPHA -- NEW Model showed up
just now" (https://www.reddit.com/r/OpenAI/comments/1300c2g/default_gpt35_with_browsin
g_alpha_new_model/). r/OpenAI. Archived (https://web.archive.org/web/20230427085505/ht
tps://www.reddit.com/r/OpenAI/comments/1300c2g/default_gpt35_with_browsing_alpha_ne
w_model/) from the original on April 27, 2023. Retrieved April 27, 2023.
34. "Introducing GPT-3.5 Series: text-davinci-002 and code-davinci-002 Models" (https://platfor
m.openai.com/). OPEN AI. March 15, 2022. Archived (https://web.archive.org/web/20230320
023933/https://platform.openai.com/) from the original on March 20, 2023. Retrieved April 27,
2023.
35. "GPT-3.5 with Browsing (ALPHA) Now Available for GPT Plus Users" (https://platform.open
ai.com/). OPEN AI. April 27, 2023. Archived (https://web.archive.org/web/20230320023933/h
ttps://platform.openai.com/) from the original on March 20, 2023. Retrieved April 27, 2023.
36. "OpenAI Codex" (https://openai.com/blog/openai-codex/). OpenAI. August 10, 2021.
Archived (https://web.archive.org/web/20230203201912/https://openai.com/blog/openai-cod
ex/) from the original on February 3, 2023. Retrieved December 23, 2022.
37. Thompson, Clive (March 15, 2022). "How an AI Became My Code-Writing Genie" (https://ww
w.wired.com/story/openai-copilot-autocomplete-for-code/). Wired. Archived (https://web.archi
ve.org/web/20221223183659/https://www.wired.com/story/openai-copilot-autocomplete-for-c
ode/) from the original on December 23, 2022. Retrieved December 23, 2022.
38. "Microsoft announced its first customer product features powered by GPT-3 and @Azure" (ht
tps://blogs.microsoft.com/ai/from-conversation-to-code-microsoft-introduces-its-first-product-f
eatures-powered-by-gpt-3/). The AI Blog. May 25, 2021. Archived (https://web.archive.org/we
b/20210526120530/https://blogs.microsoft.com/ai/from-conversation-to-code-microsoft-introd
uces-its-first-product-features-powered-by-gpt-3/) from the original on May 26, 2021.
Retrieved May 26, 2021.
39. Vincent, James (May 25, 2021). "Microsoft has built an AI-powered autocomplete for code
using GPT-3" (https://www.theverge.com/2021/5/25/22451144/microsoft-gpt-3-openai-codin
g-autocomplete-powerapps-power-fx). The Verge. Archived (https://web.archive.org/web/202
21223183700/https://www.theverge.com/2021/5/25/22451144/microsoft-gpt-3-openai-coding
-autocomplete-powerapps-power-fx) from the original on December 23, 2022. Retrieved
December 23, 2022.
40. "CodexDB - SQL Processing Powered by GPT-3" (https://itrummer.github.io/CodexDB/).
CodexDB - SQL Processing Powered by GPT-3. Archived (https://web.archive.org/web/2022
1207034506/https://itrummer.github.io/CodexDB/) from the original on December 7, 2022.
Retrieved December 7, 2022.
41. Fagone, Jason (July 23, 2021). "The Jessica Simulation: Love and loss in the age of A.I." (htt
ps://www.sfchronicle.com/projects/2021/jessica-simulation-artificial-intelligence/) San
Francisco Chronicle. Archived (https://web.archive.org/web/20210728170927/https://www.sf
chronicle.com/projects/2021/jessica-simulation-artificial-intelligence/) from the original on
July 28, 2021. Retrieved July 29, 2021.
42. GPT-3 (September 8, 2020). "A robot wrote this entire article. Are you scared yet, human? |
GPT-3" (https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gp
t-3). The Guardian. ISSN 0261-3077 (https://www.worldcat.org/issn/0261-3077). Archived (ht
tps://web.archive.org/web/20200908090812/https://www.theguardian.com/commentisfree/20
20/sep/08/robot-wrote-this-article-gpt-3) from the original on September 8, 2020. Retrieved
September 15, 2020.
43. "Update: Language Models and Dragon" (https://latitude.io/blog/update-language-models).
Latitude blog. December 8, 2021. Archived (https://web.archive.org/web/20220425034449/ht
tps://latitude.io/blog/update-language-models) from the original on April 25, 2022. Retrieved
March 22, 2022.
44. "This Mystical Book Was Co-Authored by a Disturbingly Realistic AI" (https://www.vice.com/
en/article/7kbjvb/this-magickal-grimoire-was-co-authored-by-a-disturbingly-realistic-ai).
www.vice.com. 2022. Archived (https://web.archive.org/web/20221223183700/https://www.vi
ce.com/en/article/7kbjvb/this-magickal-grimoire-was-co-authored-by-a-disturbingly-realistic-
ai) from the original on December 23, 2022. Retrieved December 23, 2022.
45. GPT-3 (February 24, 2023). "38 Prompt Examples in 10 Different Categories | GPT-3" (http
s://gipiti.chat/prompt-examples#prompts-for-language-use). GiPiTi Chat. Archived (https://we
b.archive.org/web/20230408154018/https://gipiti.chat/prompt-examples#prompts-for-langua
ge-use) from the original on April 8, 2023. Retrieved February 24, 2023.
46. "Can ChatGPT AI chatbot spot early stages of Alzheimer's? - study" (https://www.jpost.com/h
ealth-and-wellness/mind-and-spirit/article-725929). The Jerusalem Post. 2022. Archived (htt
ps://web.archive.org/web/20230210054139/https://www.jpost.com/health-and-wellness/mind
-and-spirit/article-725929) from the original on February 10, 2023. Retrieved February 10,
2023.
47. Agbavor, Felix; Liang, Hualou (December 22, 2022). "Predicting dementia from spontaneous
speech using large language models". PLOS Digital Health. 1 (12): e0000168.
doi:10.1371/journal.pdig.0000168 (https://doi.org/10.1371%2Fjournal.pdig.0000168).
PMID 36812634 (https://pubmed.ncbi.nlm.nih.gov/36812634). S2CID 255029590 (https://ap
i.semanticscholar.org/CorpusID:255029590).
48. Manjoo, Farhad (July 29, 2020). "How Do You Know a Human Wrote This?" (https://www.nyt
imes.com/2020/07/29/opinion/gpt-3-ai-automation.html). The New York Times. ISSN 0362-
4331 (https://www.worldcat.org/issn/0362-4331). Archived (https://web.archive.org/web/2020
1029161230/https://www.nytimes.com/2020/07/29/opinion/gpt-3-ai-automation.html) from the
original on October 29, 2020. Retrieved August 4, 2020.
49. Weinberg, Justin, ed. (July 30, 2020). "Philosophers On GPT-3 (updated with replies by
GPT-3)" (http://dailynous.com/2020/07/30/philosophers-gpt-3/). Daily Nous. Archived (https://
web.archive.org/web/20201030232410/http://dailynous.com/2020/07/30/philosophers-gpt-
3/) from the original on October 30, 2020. Retrieved July 31, 2020.
50. Chalmers, David (July 30, 2020). Weinberg, Justin (ed.). "GPT-3 and General Intelligence"
(https://dailynous.com/2020/07/30/philosophers-gpt-3/#chalmers). Daily Nous. Philosophers
On GPT-3 (updated with replies by GPT-3). Archived (https://web.archive.org/web/20200804
135420/http://dailynous.com/2020/07/30/philosophers-gpt-3/#chalmers) from the original on
August 4, 2020. Retrieved August 4, 2020.
51. Simonite, Tom (July 22, 2020). "Did a Person Write This Headline, or a Machine?" (https://w
ww.wired.com/story/ai-text-generator-gpt-3-learning-language-fitfully/). Wired. ISSN 1059-
1028 (https://www.worldcat.org/issn/1059-1028). Archived (https://web.archive.org/web/2020
1101124640/https://www.wired.com/story/ai-text-generator-gpt-3-learning-language-fitfully/)
from the original on November 1, 2020. Retrieved July 31, 2020.
52. Claypoole, Theodore (July 30, 2020). "New AI Tool GPT-3 Ascends to New Peaks, But
Proves How Far We Still Need to Travel" (https://www.natlawreview.com/article/new-ai-tool-
gpt-3-ascends-to-new-peaks-proves-how-far-we-still-need-to-travel). The National Law
Review. Archived (https://web.archive.org/web/20201030140406/https://www.natlawreview.c
om/article/new-ai-tool-gpt-3-ascends-to-new-peaks-proves-how-far-we-still-need-to-travel)
from the original on October 30, 2020. Retrieved August 4, 2020.
53. Marcus, Gary (December 1, 2018). "The deepest problem with deep learning" (https://mediu
m.com/@GaryMarcus/the-deepest-problem-with-deep-learning-91c5991f5695). Medium.
Archived (https://web.archive.org/web/20190801212321/https://medium.com/@GaryMarcus/t
he-deepest-problem-with-deep-learning-91c5991f5695) from the original on August 1, 2019.
Retrieved September 29, 2020.
54. Marcus, Gary; Davis, Ernest (August 22, 2020). "GPT-3, Bloviator: OpenAI's language
generator has no idea what it's talking about" (https://www.technologyreview.com/2020/08/2
2/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion). MIT
Technology Review. Archived (https://web.archive.org/web/20200823022409/https://www.te
chnologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intellige
nce-ai-opinion/) from the original on August 23, 2020. Retrieved August 23, 2020.
55. Metz, Cade (November 24, 2020). "Meet GPT-3. It Has Learned to Code (and Blog and
Argue)" (https://www.nytimes.com/2020/11/24/science/artificial-intelligence-ai-gpt3.html).
The New York Times. ISSN 0362-4331 (https://www.worldcat.org/issn/0362-4331). Archived
(https://web.archive.org/web/20201206112300/https://www.nytimes.com/2020/11/24/scienc
e/artificial-intelligence-ai-gpt3.html) from the original on December 6, 2020. Retrieved
November 24, 2020.
56. "Medical chatbot using OpenAI's GPT-3 told a fake patient to kill themselves" (https://artificial
intelligence-news.com/2020/10/28/medical-chatbot-openai-gpt3-patient-kill-themselves/). AI
News. October 28, 2020. Archived (https://web.archive.org/web/20210110145323/https://artif
icialintelligence-news.com/2020/10/28/medical-chatbot-openai-gpt3-patient-kill-themselve
s/) from the original on January 10, 2021. Retrieved January 8, 2021.
57. Chomsky on Terence McKenna, Sam Harris, GPT3, Cryptocurrencies, Kierkegaard,
Neuralink, & Hofstadter (https://www.youtube.com/watch?v=c6MU5zQwtT4). March 24,
2021. Event occurs at 1:11:44. Archived (https://web.archive.org/web/20210429153422/http
s://www.youtube.com/watch?v=c6MU5zQwtT4) from the original on April 29, 2021.
Retrieved April 29, 2021.
58. Floridi, Luciano; Chiriatti, Massimo (November 1, 2020). "GPT‑3: Its Nature, Scope, Limits,
and Consequences" (https://doi.org/10.1007%2Fs11023-020-09548-1). Minds and
Machines. 30 (4): 681–694. doi:10.1007/s11023-020-09548-1 (https://doi.org/10.1007%2Fs1
1023-020-09548-1). S2CID 228954221 (https://api.semanticscholar.org/CorpusID:22895422
1).
59. Vincent, James (July 30, 2020). "OpenAI's latest breakthrough is astonishingly powerful, but
still fighting its flaws" (https://www.theverge.com/21346343/gpt-3-explainer-openai-example
s-errors-agi-potential). The Verge. Archived (https://web.archive.org/web/20200730235924/h
ttps://www.theverge.com/21346343/gpt-3-explainer-openai-examples-errors-agi-potential)
from the original on July 30, 2020. Retrieved November 9, 2022.
60. Olanoff, Drew (December 11, 2015). "Artificial Intelligence Nonprofit OpenAI Launches With
Backing From Elon Musk And Sam Altman" (https://techcrunch.com/2015/12/11/non-profit-op
enai-launches-with-backing-from-elon-musk-and-sam-altman/). Tech Crunch. Archived (http
s://web.archive.org/web/20221020165718/https://techcrunch.com/2015/12/11/non-profit-ope
nai-launches-with-backing-from-elon-musk-and-sam-altman/) from the original on October
20, 2022. Retrieved May 31, 2021.
61. Hao, Karen (August 29, 2019). "OpenAI has released the largest version yet of its fake-
news-spewing AI" (https://www.technologyreview.com/2019/08/29/133218/openai-released-i
ts-fake-news-ai-gpt-2/). MIT Technology Review. Archived (https://web.archive.org/web/2021
0509013721/https://www.technologyreview.com/2019/08/29/133218/openai-released-its-fak
e-news-ai-gpt-2/) from the original on May 9, 2021. Retrieved May 31, 2021.
62. Coldewey, Devin (March 11, 2019). "OpenAI shifts from nonprofit to 'capped-profit' to attract
capital" (https://techcrunch.com/2019/03/11/openai-shifts-from-nonprofit-to-capped-profit-to-a
ttract-capital/). Tech Crunch. Archived (https://web.archive.org/web/20230104154138/https://t
echcrunch.com/2019/03/11/openai-shifts-from-nonprofit-to-capped-profit-to-attract-capital/)
from the original on January 4, 2023. Retrieved May 31, 2021.
63. Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Shmitchell, Shmargaret (March
3, 2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. FAccT
'21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and
Transparency. pp. 610–623. doi:10.1145/3442188.3445922 (https://doi.org/10.1145%2F344
2188.3445922).
64. Mindzak, Michael; Eaton, Sarah Elaine. "Artificial intelligence is getting better at writing, and
universities should worry about plagiarism" (https://theconversation.com/artificial-intelligenc
e-is-getting-better-at-writing-and-universities-should-worry-about-plagiarism-160481). The
Conversation. Archived (https://web.archive.org/web/20211107102635/https://theconversatio
n.com/artificial-intelligence-is-getting-better-at-writing-and-universities-should-worry-about-p
lagiarism-160481) from the original on November 7, 2021. Retrieved November 6, 2021.
65. Rogerson, Ann M.; McCarthy, Grace (December 2017). "Using Internet based paraphrasing
tools: Original work, patchwriting or facilitated plagiarism?" (https://doi.org/10.1007%2Fs409
79-016-0013-y). International Journal for Educational Integrity. 13 (1): 1–15.
doi:10.1007/s40979-016-0013-y (https://doi.org/10.1007%2Fs40979-016-0013-y).
ISSN 1833-2595 (https://www.worldcat.org/issn/1833-2595). S2CID 9473217 (https://api.se
manticscholar.org/CorpusID:9473217).
66. Ver Meer, Dave. "ChatGPT Statistics" (https://www.namepepper.com/chatgpt-users).
NamePepper. Retrieved June 21, 2023.
67. Here are a few ways GPT-3 can go wrong (https://techcrunch.com/2020/08/07/here-are-a-fe
w-ways-gpt-3-can-go-wrong/). TechCrunch. Archived (https://web.archive.org/web/20211126
192240/https://techcrunch.com/2020/08/07/here-are-a-few-ways-gpt-3-can-go-wrong/) from
the original on November 26, 2021. Retrieved November 26, 2021.
68. Comment Regarding Request for Comments on Intellectual Property Protection for Artificial
Intelligence Innovation (https://www.uspto.gov/sites/default/files/documents/OpenAI_RFC-84
-FR-58141.pdf) (PDF). USPTO. Archived (https://web.archive.org/web/20211016024654/http
s://www.uspto.gov/sites/default/files/documents/OpenAI_RFC-84-FR-58141.pdf) (PDF) from
the original on October 16, 2021. Retrieved November 30, 2021.

Retrieved from "https://en.wikipedia.org/w/index.php?title=GPT-3&oldid=1164676687"