(Slide) Neural Machine Translation
(Slide) Neural Machine Translation
(Slide) Neural Machine Translation
All-in-One Course
NLP Project
AI VIET NAM
Nguyen Quoc Thai
1
Year 2023
Outline
Ø Introduction
Ø NMT using Transformer
Ø NMT using Pre-trained LMs
2
Introduction
! Translate a sentence w(s) in a source language (input) to a sentence w(t) in the
target language (output)
3
Introduction
! Translate a sentence w(s) in a source language (input) to a sentence w(t) in the
target language (output)
q Syntax
q Semantics
q Phonology
q Pragmatics
q Morphology
4
Introduction
! Translate a sentence w(s) in a source language (input) to a sentence w(t) in the
target language (output)
5
Introduction
Ø Human judgement
q Given: machine translation output
q Given: source / reference translation
q Task: assess the quality of machine translation output
Ø Different translations of “A Vinay le gusta Python”
7
Introduction
! Evaluating translation quality
8
Introduction
! Evaluating translation quality
9
Introduction
! Evaluating Metrics
10
Introduction
! Evaluating Metrics
Ø Precision: Ø Recall:
correct 3 correct 3
= = 50% = = 43%
output − length 6 reference − length 7
Ø F-measure:
PxR 0.5 x 0.43
= = 46%
(P + R)/2 (0.5 + 0.43)/2
11
Introduction
! Evaluating Metrics
12
Introduction
! Evaluating Metrics
BLEU
v N-gram overlap between machine translation output and reference translation
v Compute precision for n-grams of size 1 to 4
v Add brevity penalty (for too short translations)
* )/*
output − length
BLEU = min 1, > precision)
reference − length
'()
v Typically computed over the entire corpus, not single sentences
13
Introduction
! Evaluating Metrics
BLEU 1-gram
System A A officials responsibility of airport safety
Reference A officials are responsible for airport security
System B airport security A officials are responsible
Metric System A System B
Precision (1 gram) 3/6 6/6
Precision (2 gram)
Precision (3 gram)
Precision (4 gram)
Brevity penalty
BLEU
14
Introduction
! Evaluating Metrics
BLEU
System A A officials responsibility of airport safety
Reference A officials are responsible for airport security
System B airport security A officials are responsible
BLEU
-
r
logBLEU = min 1 − , 0 + B w,logp,
c
,()
r: reference-length, c: output (candidate)-length
n: n-gram (1,2,3,4), wn: weight of n-gram
uniform weights wn=1/n
pn: precision n-gram
SacreBLEU (A Call for Clarity in Reporting BLEU)
16
Introduction
! Evaluating Metrics
17
Outline
Ø Introduction
Ø NMT using Transformer
Ø NMT using Pre-trained LMs
18
NMT using Transformer
! Sequence to Sequence
19
NMT using Transformer
! Transformer Model
20
NMT using Transformer
! Training
Target
I go to work <end>
Loss
Prediction I go _earn work <end>
t
_ôi
đi ENCODER DECODER
l
_àm
<start> I go to work
21
NMT using Transformer
! Training
ENCODER DECODER
23
Outline
Ø Introduction
Ø NMT using Transformer
Ø NMT using Pre-trained LMs
24
NMT using Pre-trained LMs
! Pre-trained LMs
25
NMT using Pre-trained LMs
! Pre-trained LMs
Source
26
NMT using Pre-trained LMs
! Pre-trained LMs
27
NMT using Pre-trained LMs
! Pre-trained LMs: BERT
28
NMT using Pre-trained LMs
! Pre-trained LMs: BERT
29
NMT using Pre-trained LMs
! Pre-trained LMs: GPT2
30
NMT using Pre-trained LMs
! Pre-trained LMs: GPT2
31
NMT using Pre-trained LMs
! Encoder-Decoder with BERT and GPT2
32
NMT using Pre-trained LMs
! BERT for Encoder
33
NMT using Pre-trained LMs
! BERT for Decoder
34
NMT using Pre-trained LMs
! GPT2 for Decoder
35
NMT using Pre-trained LMs
! Experiment
36
Thanks!
Any questions?
37