Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

PV211: Introduction to Information Retrieval

https://www.fi.muni.cz/~sojka/PV211

IIR 1: Boolean Retrieval

Handout version

Petr Sojka, Hinrich Schütze et al.

Faculty of Informatics, Masaryk University, Brno

Center for Information and Language Processing, University of Munich

2019-02-21

Sojka, IIR Group: PV211: Boolean Retrieval 1 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Take-away

Basic information about the course, teachers, evaluation,

exercises
Boolean Retrieval: Design and data structures of a simple
information retrieval system
What topics will be covered in this class (overview)?

Sojka, IIR Group: PV211: Boolean Retrieval 2 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Overview

1 Introduction

2 History of information retrieval

3 Boolean model

4 Inverted index

5 Processing queries

6 Query optimization

7 Course overview and agenda

Sojka, IIR Group: PV211: Boolean Retrieval 3 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Definition of Information Retrieval

Information retrieval (IR) is finding material (usually documents) of

an unstructured nature (usually text) that satisfies an information
need from within large collections (usually stored on computers).

Sojka, IIR Group: PV211: Boolean Retrieval 5 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Prerequisites

Curiosity about how Information Retrieval works.

But seriously:
Chapters 1–5 benefit from basic course on algorithms and
data structures.
Chapters 6–7 need in addition linear algebra, vectors and dot
products.
For Chapters 11–13 basic probability notions are needed.
Chapters 18–21 demand course in linear algebra, notions of
matrix rank, eigenvalues and eigenvectors.

Sojka, IIR Group: PV211: Boolean Retrieval 6 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Active learning features in PV211

Student activities explicitly welcomed and built as part of

classification system (10 pts).
Mentoring rather than ‘ex cathedra’ lectures: “The flipped
classroom is a pedagogical model in which the typical lecture
and homework elements of a course are reversed.”
Respect to individual learning speed and knowledge.
Questions on PV211 IS discussion forum is welcomed
especially before lectures.
Richness of materials available in advance: MOOC (Massive
open online course) becoming widespread, parts of
IIR Stanford courses being available, together with other freely
available teaching materials, including the whole IIR book.

Sojka, IIR Group: PV211: Boolean Retrieval 7 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Teachers

Petr Sojka, [email protected]

Consulting hours Spring 2019:
Thursday 13:00–14:00 and Friday 10:00–11:00
or by appointment by email.
Room C523 or C522 or A502, fifth floor, Botanická 68a.
Course web page:
https://www.fi.muni.cz/~sojka/PV211/
TA: Vít Novotný, [email protected],
Consulting hours: Fri 10:00–12:00 (A502) or by appointment
TA: Dávid Lupták, [email protected],
Consulting hours: Fri 10:00–12:00 (A502) or by appointment

Sojka, IIR Group: PV211: Boolean Retrieval 8 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Evaluation of students

Classification system is based on points achieved (100 pts max).

You could get 50 points during the term: 20 pts for each of
2 midterm tests, 10 pts for your activity during term (lectures or
discussion forums,. . . ) evaluated subjectively by teachers of the
course, and 50 pts for the final test. Final written exam will consist
of open exercises (30 pts, similar to midterm ones) and multiple
choice questions (20 pts). In addition, one can get additional
premium points based on activities during lectures, exercises (good
answers) or negotiated related projects. Classification scale
(adjustments based on ECTS suggestions) z/k[/E/D/C/B/A]
corresponds ≈ 50/57/[64/71/78/85/92] points.
Dates of [final] exams will be announced via IS.muni.cz (at least
three terms). There wiil be a possibility to make midterm tests on
the first exam term for those ill.
Questions?
Sojka, IIR Group: PV211: Boolean Retrieval 9 / 77
Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Can we proceed [Y/N]?

Questions?
Presentation style? Warm ups? Personal cards.
Erasmus? Bc. or Mgr.? Discussion forum in IS!

Sojka, IIR Group: PV211: Boolean Retrieval 10 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

History of information retrieval: gradual changes of

channels

Sojka, IIR Group: PV211: Boolean Retrieval 12 / 77

Gradual speedup of changes in IR
Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

1998: google.stanford.edu

‘flipped IS’, collaborative project with Stanford faculty

on collected disks
Google 1998 ‘Anatomy paper’ (Page, Brin)

Sojka, IIR Group: PV211: Boolean Retrieval 16 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Boolean retrieval

The Boolean model is arguably the simplest model to base an

information retrieval system on.
The search engine returns all documents that satisfy the
Boolean expression.

Does Google use the Boolean model?

Sojka, IIR Group: PV211: Boolean Retrieval 21 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Does Google use the Boolean model?

On Google, the default interpretation of a query [w1 w2

. . . wn ] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor text
page contains variant of wi (morphology, spelling correction,
synonym)
long queries (n large)
boolean expression generates very few hits
Simple Boolean vs. Ranking of result set
Simple Boolean retrieval returns matching documents in no
particular order.
Google (and most well designed Boolean engines) rank the
result set – they rank good hits (according to some estimator
of relevance) higher than bad hits.

Sojka, IIR Group: PV211: Boolean Retrieval 22 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Unstructured data in 1650: Shakespeare

Sojka, IIR Group: PV211: Boolean Retrieval 24 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Unstructured data in 1650

Which plays of Shakespeare contain the words Brutus and

Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus and
Caesar, then strip out lines containing Calpurnia.
Why is grep not the solution?
Slow (for large collections)
grep is line-oriented, IR is document-oriented
“not Calpurnia” is non-trivial
Other operations (e.g., find the word Romans near
countryman) not feasible
Ranked retrieval (best documents to return) – focus of later
lectures, but not this one

Sojka, IIR Group: PV211: Boolean Retrieval 25 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Term-document incidence matrix

Anthony Julius The Hamlet Othello Macbeth ...
and Caesar Tempest
Cleopatra
Anthony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
...
Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar.
Entry is 0 if term doesn’t occur. Example: Calpurnia doesn’t occur in The
tempest.

Sojka, IIR Group: PV211: Boolean Retrieval 26 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Incidence vectors

So we have a 0/1 vector for each term.

To answer the query Brutus and Caesar and not
Calpurnia:
Take the vectors for Brutus, Caesar, and Calpurnia
Complement the vector of Calpurnia
Do a (bitwise) and on the three vectors
110100 and 110111 and 101111 = 100100

Sojka, IIR Group: PV211: Boolean Retrieval 27 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

0/1 vector for Brutus

Anthony Julius The Hamlet Othello Macbeth ...
and Caesar Tempest
Cleopatra
Anthony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0
...
result: 1 0 0 1 0 0

Sojka, IIR Group: PV211: Boolean Retrieval 28 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Answers to query

Anthony and Cleopatra, Act III, Scene ii

Agrippa [Aside to Domitius Enobarbus]: Why, Enobarbus,
When Antony found Julius Caesar dead,
He cried almost to roaring; and he wept
When at Philippi he found Brutus slain.

Hamlet, Act III, Scene ii

Lord Polonius: I did enact Julius Caesar: I was killed i’ the
Capitol; Brutus killed me.

Sojka, IIR Group: PV211: Boolean Retrieval 29 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Bigger collections

Consider N = 106 documents, each with about 1000 tokens

⇒ total of 109 tokens
On average 6 bytes per token, including spaces and
punctuation ⇒ size of document collection is about 6 · 109 =
6 GB
Assume there are M = 500,000 distinct terms in the collection
(Notice that we are making a term/token distinction.)

Sojka, IIR Group: PV211: Boolean Retrieval 30 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Can’t build the incidence matrix

M = 500,000 × 106 = half a trillion 0s and 1s.

But the matrix has no more than one billion 1s.
Matrix is extremely sparse.
What is a better representations?
We only record the 1s: inverted index!

Sojka, IIR Group: PV211: Boolean Retrieval 31 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Inverted index

For each term t, we store a list of all documents that contain t.

Brutus −→ 1 2 4 11 31 45 173 174

Caesar −→ 1 2 4 5 6 16 57 132 ...

Calpurnia −→ 2 31 54 101

..
.
| {z } | {z }
dictionary postings

Sojka, IIR Group: PV211: Boolean Retrieval 32 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Inverted index construction

1 Collect the documents to be indexed:

Friends, Romans, countrymen. So let it be with Caesar . . .
2 Tokenize the text, turning each document into a list of tokens:
Friends Romans countrymen So . . .
3 Do linguistic preprocessing, producing a list of normalized
tokens, which are the indexing terms: friend roman
countryman so . . .
4 Index the documents that each term occurs in by creating an
inverted index, consisting of a dictionary and postings.

Sojka, IIR Group: PV211: Boolean Retrieval 33 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Tokenization and preprocessing

Doc 1. I did enact Julius Caesar: I

Doc 1. i did enact julius caesar i was
was killed i’ the Capitol; Brutus killed
killed i’ the capitol brutus killed me
me.
Doc 2. So let it be with Caesar. The
=⇒ Doc 2. so let it be with caesar the
noble brutus hath told you caesar was
noble Brutus hath told you Caesar
ambitious
was ambitious:

Sojka, IIR Group: PV211: Boolean Retrieval 34 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Generate postings
term docID
i 1
did 1
enact 1
julius 1
caesar 1
i 1
was 1
killed 1
i’ 1
the 1
capitol 1
brutus 1
Doc 1. i did enact julius caesar i was
killed 1
killed i’ the capitol brutus killed me
me 1
Doc 2. so let it be with caesar the =⇒ so 2
noble brutus hath told you caesar was
let 2
ambitious
it 2
be 2
with 2
caesar 2
the 2
noble 2
brutus 2
hath 2
told 2
you 2
caesar 2
was 2
ambitious 2

Sojka, IIR Group: PV211: Boolean Retrieval 35 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Sort postings
term docID term docID
i 1 ambitious 2
did 1 be 2
enact 1 brutus 1
julius 1 brutus 2
caesar 1 capitol 1
i 1 caesar 1
was 1 caesar 2
killed 1 caesar 2
i’ 1 did 1
the 1 enact 1
capitol 1 hath 1
brutus 1 i 1
killed 1 i 1
me 1 i’ 1
so 2
=⇒ it 2
let 2 julius 1
it 2 killed 1
be 2 killed 1
with 2 let 2
caesar 2 me 1
the 2 noble 2
noble 2 so 2
brutus 2 the 1
hath 2 the 2
told 2 told 2
you 2 you 2
caesar 2 was 1
was 2 was 2
ambitious 2 with 2

Sojka, IIR Group: PV211: Boolean Retrieval 36 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Create postings lists, determine document frequency

term docID
ambitious 2
be 2 term doc. freq. → postings lists
brutus 1
ambitious 1 → 2
brutus 2
be 1 → 2
capitol 1
caesar 1 brutus 2 → 1 → 2
caesar 2 capitol 1 → 1
caesar 2 caesar 2 → 1 → 2
did 1 did 1 → 1
enact 1 enact 1 → 1
hath 1 hath 1 → 2
i 1 i 1 → 1
i 1 i’ 1 → 1
i’ 1
it 2
=⇒ it 1 → 2
julius 1 → 1
julius 1
killed 1 killed 1 → 1
killed 1 let 1 → 2
let 2 me 1 → 1
me 1 noble 1 → 2
noble 2 so 1 → 2
so 2 the 2 → 1 → 2
the 1 told 1 → 2
the 2
you 1 → 2
told 2
was 2 → 1 → 2
you 2
was 1 with 1 → 2
was 2
with 2

Sojka, IIR Group: PV211: Boolean Retrieval 37 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Split the result into dictionary and postings file

Brutus −→ 1 2 4 11 31 45 173 174

Caesar −→ 1 2 4 5 6 16 57 132 ...

Calpurnia −→ 2 31 54 101

..
.
| {z } | {z }
dictionary postings file

Sojka, IIR Group: PV211: Boolean Retrieval 38 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Later in this course

Index construction: how can we create inverted indexes for

large collections?
How much space do we need for dictionary and index?
Index compression: how can we efficiently store and process
indexes for large collections?
Ranked retrieval: what does the inverted index look like when
we want the “best” answer?

Sojka, IIR Group: PV211: Boolean Retrieval 39 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Simple conjunctive query (two terms)

Consider the query: Brutus AND Calpurnia

To find all matching documents using inverted index:
1 Locate Brutus in the dictionary
2 Retrieve its postings list from the postings file
3 Locate Calpurnia in the dictionary
4 Retrieve its postings list from the postings file
5 Intersect the two postings lists
6 Return intersection to user

Sojka, IIR Group: PV211: Boolean Retrieval 41 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Intersecting two postings lists

Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174

Calpurnia −→ 2 → 31 → 54 → 101

Intersection =⇒ 2 → 31
This is linear in the length of the postings lists.
Note: This only works if postings lists are sorted.

Sojka, IIR Group: PV211: Boolean Retrieval 42 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Intersecting two postings lists

Intersect(p1 , p2 )
1 answer ← h i
2 while p1 6= nil and p2 6= nil
3 do if docID(p1 ) = docID(p2 )
4 then Add(answer , docID(p1 ))
5 p1 ← next(p1 )
6 p2 ← next(p2 )
7 else if docID(p1 ) < docID(p2 )
8 then p1 ← next(p1 )
9 else p2 ← next(p2 )
10 return answer

Sojka, IIR Group: PV211: Boolean Retrieval 43 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Query processing: Exercise

france −→ 1 → 2 → 3 → 4 → 5 → 7 → 8 → 9 → 11 → 12 → 13 → 14 → 15
paris −→ 2 → 6 → 10 → 12 → 14
lear −→ 12 → 15

Compute hit list for ((paris AND NOT france) OR lear)

Sojka, IIR Group: PV211: Boolean Retrieval 44 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Boolean queries

The Boolean retrieval model can answer any query that is a

Boolean expression.
Boolean queries are queries that use and, or and not to join
query terms.
Views each document as a set of terms.
Is precise: Document matches condition or not.
Primary commercial retrieval tool for 3 decades
Many professional searchers (e.g., lawyers) still like Boolean
queries.
You know exactly what you are getting.
Many search systems you use are also Boolean: spotlight,
email, intranet etc.

Sojka, IIR Group: PV211: Boolean Retrieval 45 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Commercially successful Boolean retrieval: Westlaw

Largest commercial legal search service in terms of the

number of paying subscribers
Over half a million subscribers performing millions of searches
a day over tens of terabytes of text data
The service was started in 1975.
In 2005, Boolean search (called “Terms and Connectors” by
Westlaw) was still the default, and used by a large percentage
of users . . .
. . . although ranked retrieval has been available since 1992.

Sojka, IIR Group: PV211: Boolean Retrieval 46 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Westlaw: Example queries

Information need: Information on the legal theories involved in

preventing the disclosure of trade secrets by employees formerly
employed by a competing company
Query: “trade secret” /s disclos! /s prevent /s employe!

Information need: Requirements for disabled people to be able to

access a workplace
Query: disab! /p access! /s work-site work-place (employment /3
place)

Information need: Cases about a host’s responsibility for drunk

guests
Query: host! /p (responsib! liab!) /p (intoxicat! drunk!) /p guest

Sojka, IIR Group: PV211: Boolean Retrieval 47 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Westlaw: Comments

Proximity operators: /3 = within 3 words, /s = within a

sentence, /p = within a paragraph
Space is disjunction, not conjunction! (This was the default in
search pre-Google.)
Long, precise queries: incrementally developed, not like web
search
Why professional searchers often like Boolean search:
precision, transparency, control
When are Boolean queries the best way of searching? Depends
on: information need, searcher, document collection,. . .

Sojka, IIR Group: PV211: Boolean Retrieval 48 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Query optimization

Consider a query that is an and of n terms, n > 2

For each of the terms, get its postings list, then and them
together
Example query: Brutus AND Calpurnia AND Caesar
What is the best order for processing this query?

Sojka, IIR Group: PV211: Boolean Retrieval 50 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Query optimization

Example query: Brutus AND Calpurnia AND Caesar

Simple and effective optimization: Process in order of
increasing frequency
Start with the shortest postings list, then keep cutting further
In this example, first Caesar, then Calpurnia, then
Brutus
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Caesar −→ 5 → 31

Sojka, IIR Group: PV211: Boolean Retrieval 51 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Optimized intersection algorithm for conjunctive queries

Intersect(ht1 , . . . , tn i)
1 terms ← SortByIncreasingFrequency(ht1 , . . . , tn i)
2 result ← postings(first(terms))
3 terms ← rest(terms)
4 while terms 6= nil and result 6= nil
5 do result ← Intersect(result, postings(first(terms)))
6 terms ← rest(terms)
7 return result

Sojka, IIR Group: PV211: Boolean Retrieval 52 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

More general optimization

Example query: (madding or crowd) and (ignoble or

strife)
Get frequencies for all terms
Estimate the size of each or by the sum of its frequencies
(conservative)
Process in increasing order of or sizes

Sojka, IIR Group: PV211: Boolean Retrieval 53 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Exercise

Recommend a query processing order for: (tangerine OR

trees) AND (marmalade OR skies) AND (kaleidoscope
OR eyes)

Sojka, IIR Group: PV211: Boolean Retrieval 54 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Course overview and agenda

We are done with Chapter 1 of IIR (IIR 01).

Plan for the rest of the semester: 16–18 of the 21 chapters of
IIR
In addition to experts from FI lectures by leading industry
experts from Facebook (Tomáš Mikolov on March 12th as
part of FI Informatics Colloquium), Seznam.cz (Vláďa Kadlec)
or RaRe Technologies (Radim Řehůřek).
In what follows: teasers for most chapters – to give you a
sense of what will be covered.
Last two or three lectures on IR topics researched in my
research group MIR.fi.muni.cz and on state-of-the art
achievements in the area (vector space embeddings etc.).

Sojka, IIR Group: PV211: Boolean Retrieval 56 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 02: The term vocabulary and postings lists

Phrase queries: “Stanford University”

Proximity queries: Gates near Microsoft
We need an index that captures position information for
phrase queries and proximity queries.

Sojka, IIR Group: PV211: Boolean Retrieval 57 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 03: Dictionaries and tolerant retrieval

bo ✲ aboard ✲ about ✲ boardroom ✲ border

or ✲ border ✲ lord ✲ morbid ✲ sordid

rd ✲ aboard ✲ ardent ✲ boardroom ✲ border

Sojka, IIR Group: PV211: Boolean Retrieval 58 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 04: Index construction

splits assign master assign

postings

parser a-f g-p q-z inve rter a-f

parser a-f g-p q-z g-p

inve rter

inve rter q-z

parser a-f g-p q-z

segment reduce
map files
phase phase

Sojka, IIR Group: PV211: Boolean Retrieval 59 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 05: Index compression

7
6
5
4
log10 cf

3
2
1
0

0 1 2 3 4 5 6 7

log10 rank

Zipf’s law

Sojka, IIR Group: PV211: Boolean Retrieval 60 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 06: Scoring, term weighting and the vector space

model
Ranking search results
Boolean queries only give inclusion or exclusion of documents.
For ranked retrieval, we measure the proximity between the query and
each document.
One formalism for doing this: the vector space model
Key challenge in ranked retrieval: evidence accumulation for a term in
a document
1 vs. 0 occurrence of a query term in the document
3 vs. 2 occurrences of a query term in the document
Usually: more is better
But by how much?
Need a scoring function that translates frequency into score or weight

Sojka, IIR Group: PV211: Boolean Retrieval 61 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 07: Scoring in a complete search system

Parsing user query

Linguistics
Results
Documents Free text query parser page

Document Indexers Spell correction Scoring and ranking

cache

Metadata in Inexact
Tiered inverted Scoring
zone and top K k-gram
positional index parameters training
field indexes retrieval
Indexes MLR set

Sojka, IIR Group: PV211: Boolean Retrieval 62 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 08: Evaluation and dynamic summaries

Sojka, IIR Group: PV211: Boolean Retrieval 63 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 09: Relevance feedback & query expansion

Sojka, IIR Group: PV211: Boolean Retrieval 64 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 12: Language models

w P(w |q1 ) w P(w |q1 )
STOP 0.2 toad 0.01
the 0.2 said 0.03
q1 a 0.1 likes 0.02
frog 0.01 that 0.04
... ...
This is a one-state probabilistic finite-state automaton – a unigram
language model – and the state emission distribution for its one
state q1 .
STOP is not a word, but a special symbol indicating that the
automaton stops.
frog said that toad likes frog STOP
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02 ·0.01 ·0.2
= 0.0000000000048
Sojka, IIR Group: PV211: Boolean Retrieval 65 / 77
Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 13: Text classification & Naive Bayes

Text classification = assigning documents automatically to

predefined classes
Examples:
Language (English vs. French)
Adult content
Region

Sojka, IIR Group: PV211: Boolean Retrieval 66 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 14: Vector classification, kNN search

X
X
X X
X

X
X
∗
X
X

X
X

Sojka, IIR Group: PV211: Boolean Retrieval 67 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 15: Support vector machines

Sojka, IIR Group: PV211: Boolean Retrieval 68 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 16: Flat clustering

Sojka, IIR Group: PV211: Boolean Retrieval 69 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 17: Hierarchical clustering

http://news.google.com

Sojka, IIR Group: PV211: Boolean Retrieval 70 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 18: Latent Semantic Indexing

Sojka, IIR Group: PV211: Boolean Retrieval 71 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 19: The web and its challenges

Unusual and diverse documents

Unusual and diverse users and information needs
Beyond terms and text: exploit link analysis, user data
How do web search engines work?
How can we make them better?

Sojka, IIR Group: PV211: Boolean Retrieval 72 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 20: Crawling

Doc URL
FP’s set
✛✲ DNS ✓✏ ✓✏
To
✒✑ other ✒✑
✻
❄ ✒✑ nodes ✒✑
✻
❄ ✻✻✻ ✻
❄
www ✛ ✲ ✲ ✲ ✲
Content URL Host ✲ Dup
✲Fetch ✲
Parse
Seen? Filter splitter ✲ URL
✲ Elim

From
other
✻ nodes
URL Frontier ✛

Sojka, IIR Group: PV211: Boolean Retrieval 73 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

IIR 21: Link analysis / PageRank

Sojka, IIR Group: PV211: Boolean Retrieval 74 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Invited lecture: Fulltext architecture in Seznam

Introduction to the Seznam.cz fulltext search architecture by

Seznam research team lead (Vladimír Kadlec).
Abstract: The talk covers all basic web search engine blocks:
crawling, indexing, query reformulation, relevance. Explanation of
inner parts of the user interface such as: auto completer, query
corrector, suggested searches. Real statistics from Seznam’s traffic.
As a bonus: Image/video search.

Sojka, IIR Group: PV211: Boolean Retrieval 75 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Take-away

Basic information about the course, teachers, evaluation,

exercises
Boolean Retrieval: Design and data structures of a simple
information retrieval system
What topics will be covered in this class (overview)?

Sojka, IIR Group: PV211: Boolean Retrieval 76 / 77

Introduction History Boolean model Inverted index Processing Boolean queries Query optimization Cours

Resources

Chapter 1 of IIR
Resources at https://www.fi.muni.cz/~sojka/PV211/
and http://cislmu.org, materials in MU IS and FI MU
library
course schedule and overview
information retrieval links
Shakespeare search engine
https://www.rhymezone.com/shakespeare/

Sojka, IIR Group: PV211: Boolean Retrieval 77 / 77