Index Compression
Index Compression
Index Compression
Index Compression
Introduction to Information Retrieval
Today
Reuters RCV1
symbol statistic value
N documents 800,000
L avg. # tokens per doc 200
M terms (= word types) ~400,000
avg. # bytes per token 6
(incl. spaces/punct.)
avg. # bytes per token 4.5
(without spaces/punct.)
avg. # bytes per term 7.5
non-positional postings 100,000,000
19 July 2017 CS F469 6
Introduction to Information Retrieval
DICTIONARY COMPRESSION
.systilesyzygeticsyzygialsyzygyszaibelyiteszczecinszomo.
Blocking
Store pointers to every kth term string.
Example below: k=4.
Need to store term lengths (1 extra byte)
.7systile9syzygetic8syzygial6syzygy11szaibelyite8szczecin9szomo.
Net
Example for block size k = 4
Where we used 3 bytes/pointer without blocking
3 x 4 = 12 bytes,
now we use 3 + 4 = 7 bytes.
Exercise
Estimate the space usage (and savings compared to
7.6 MB) with blocking, for block sizes of k = 8 and 16.
Assuming each
dictionary term equally
likely in query (not really
so in practice!), average
number of comparisons
= (1+22+43+4)/8 ~2.6
Front coding
Front-coding:
Sorted words commonly have long common prefix store
differences only
(for last k-1 in a block of k)
8automata8automate9automatic10automation
8automat*a1e2ic3ion
POSTINGS COMPRESSION
Postings compression
The postings file is much larger than the dictionary,
factor of at least 10.
Key desideratum: store each posting compactly.
A posting for our purposes is a docID.
For Reuters (800,000 documents), we would use 32
bits per docID when using 4-byte integers.
Alternatively, we can use log2 800,000 20 bits per
docID.
Our goal: use far fewer than 20 bits per docID.
Example
docIDs 824 829 215406
gaps 5 214577
VB code 00000110 10000101 00001101
10111000 00001100
10110001
Unary code
Represent n as n 1s with a final 0.
Unary code for 3 is 1110.
Unary code for 40 is
11111111111111111111111111111111111111110 .
Unary code for 80 is:
11111111111111111111111111111111111111111111
1111111111111111111111111111111111110
Gamma codes
We can compress better with bit-level codes
The Gamma code is the best known of these.
Represent a gap G as a pair length and offset
offset is G in binary, with the leading bit cut off
For example 13 1101 101
length is the length of offset
For 13 (offset 101), this is 3.
We encode length with unary code: 1110.
Gamma code of 13 is the concatenation of length and offset:
1110101.
(Note: Here we assume that G have no leading 0s. If there are any, they are removed
before deleting the leading 1.)
19 July 2017 CS F469 32
Introduction to Information Retrieval
Delta code
Gamma codes are relatively inefficient for large
numbers as they encode the length of the offset in
inefficient unary code. Delta codes differ from Gamma
codes in that they encode the first part of the
code(length) in Gamma code instead of unary code.
The encoding of offset is the same.
Example: Delta code of 7 is 10,0,11
10,0 is the Gamma code for length (2 in this case) and
the encoding of offset (11) is unchanged.
END