Netzob Documentation

Release 0.4.1
Frdric Guihry, Georges Bossert
January 31, 2013
CONTENTS
1 The big picture 3
1.1 Table of contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Indices and tables 21
Python Module Index 23
i
ii
Netzob Documentation, Release 0.4.1
Netzob simplies the work for security auditors by providing a complete framework for the reverse engineering of
communication protocols. It handles different types of protocols : text protocols (like HTTP and IRC), xed elds
protocols (like IP and TCP) and variable elds protocols (like ASN.1 based formats). Netzob is therefore suitable for
reversing network protocols, stuctured les and system and process ows (IPC and communication with drivers).
Netzob is provided with modules dedicated to capture data in multiple contexts (network, le, process and kernel data
acquisition).
CONTENTS 1
Netzob Documentation, Release 0.4.1
2 CONTENTS
CHAPTER
ONE
THE BIG PICTURE
1.1 Table of contents
1.1.1 Introduction
1.1.2 Import
Communication protocols can be found is every parts of a system, as shown on the following picture:
3
Netzob Documentation, Release 0.4.1
Netzob can handle multiple kinds of input data. Hence, you can analyze network trafc, IPC communications, les
structures, etc.
Import can either be done by using a dedicated captor or by providing already captured messages in a specic format.
Current accepted formats are:
PCAP les
Structured les
Netzob XML les (used by Netzob for its internal representation of messages)
Current supported captors are:
Network captor, based on the XXX library
Intra Process communication captor (API calls), based on API hooking
Inter Process Communication captor (pipes, shared memory and local sockets), based on system call hooking
Imported messages are manipulated by Netzob through specic Python objects which contains metadata that describes
contextual parameters (timestamp or even IP source/destination for example). All the Python object that describe
messages derived from an abstract object : AbstractMessage.
The next part of this section details the composition of each message object.
AbstractMessage
All the messages inherits from this denition and therefore has the following parameters :
a unique ID
a data eld represented with an array of hex
NetworkMessage
A network message is dened with the following parameters :
a timestamp
the ip source
4 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
the ip target
the protocol (TCP/UDP/ICMP...)
the layer 4 source port
the layer 4 target port
Denition of a NetworkMessage :
FileMessage
A le message is dened with the following parameters :
a lename
the line number in the le
the creation date of the le
the last modication date of the le
the owner of the le
the size of the le
Denition of a NetworkMessage :
class netzob.Common.Models.FileMessage.FileMessage(id, timestamp, data, lename, cre-
ationDate, modicationDate, owner,
size, lineNumber)
Denition of the factory for XML processing of a FileMessage :
class netzob.Common.Models.Factories.FileMessageFactory.FileMessageFactory
1.1.3 Modelization
Denition of a communication protocol
A communication protocol is as language. A language is dened through~:
its vocabulary (the set of valid words or, in our context, the set of valid messages) ;
its grammar (the set of valid sentences which, in our context, can be represented as a protocol state machine,
like the TCP state machine).
A word of the vocabular is called a symbol. A symbol represents an abstract view of a set of similar messages. Similar
messages refer to messages having the same semantic (for example, a TCP SYN message, a SMTP HELLO message,
an ICMP ECHO REQUEST message, etc.).
A symbol is structured following a format, which species a sequence of elds (like the IP format). A eld can be
splitted into sub-elds. For example, a payload is a eld of a TCP message. Therefore, by dening a layer as a kind of
payload (which is a specic eld), we can retrieve the so-called Ethernet, IP, TCP and HTTP layers from a raw packet
; each layer having its own vocabular and grammar.
Fields size can be xed or variable. Fields content can be static of dynamic. Fields content can be basic (a 32 bits
integer) or complex (an array). A eld has four attributes~:
the type denes its denition domain or set of valid values (16 bits integer, string, etc.) ;
the data description denes the structuration of the eld (ASN.1, TSN.1, EBML, etc.) ;
1.1. Table of contents 5
Netzob Documentation, Release 0.4.1
the data encoding denes ... (ASCII, little endian, big endian, XML, EBML, DER, XER, PER, etc.) ;
the semantic denes ... (IP address, port number, URL, email, checksum, etc.).
Fields content can be~:
static ;
dependant of another eld (or a set of elds) of the same message (intra-message dependency) ;
dependant of a eld (or a set of elds) of a previous message in the grammar (inter-message dependency) ;
dependant of the environment ;
dependant of the application behaviour (which could depend on the user behaviour) ;
random (the initial value of the TCP sequence number for example).
Modelization in Netzob
Netzob provides a framework for the semi-automated modelization (inference) of communication protocols, i.e. in-
ferring its vocabular and grammar.
[INCLURE GRAPH]
Vocabular inference
Message structure inference (based on sequence alignment)
Regoupment of similar message structures
Field type inference
Field dependencies from the same message and from the environment
Field semantic inference
Grammar inference
Identication of the automata of the protocol
Fields dependencies with messages of previous states
All the functionalities of the framework are detailled in this chapter.
Vocabular inference
Structure inference
Regoupment of similar structures
Options during alignment process
read-only process (do not require a participation in the communication).
Identify the xed and dynamic elds of all the messages.
Regroups equivalent messages depending of their elds structures.
Clustering (Regroups equivalent messages using) :
an UPGMA Algorithm to regroup similar messages
6 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
an openMP and MPI implementation
Sequencing, Alignment (Identication of elds in messages) :
Needleman & Wunsch Implementation
Needleman and Wunsch algorithm
Originaly a bio-informatic algorithm (sequencing DNA)
Align two messages and identify common patterns and eld structure
Computes an alignment score representing the efciency of the alignment
The following picture shows the sequence alignment of two messages.
UPGMA algorithm
Identify equivalent messages based on their alignment score.
Build a hierarchical organization of the messages with the UPGMA algorithm (Unweighted Pair Group Method
with Arithmetic Mean)
The following picture shows a regroupment of similar messages based on the result of the clustering process.
Abstraction of a set of message The abstraction is the process of substituting the dynamic elds with their repre-
sentation as a regex. An example of abstraction is shown on the follinw picture.
1.1. Table of contents 7
Netzob Documentation, Release 0.4.1
Analyses after alignment process aaa
Message contextual menu aaa
Group contextual menu aaa
Rene regexes aaa
Slick regexes aaa
Concatenate aaa
Split column aaa
Merge columns aaa
8 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
Delete message aaa
Field type inference
Visualization options aaa
Type structure contextual menu aaa
Messages distribution This function shows a graphical representation of the distribution of bytes per offset for
each message of the current group. This function helps to identify entropy variation of each elds. Entropy variation
combined with byte distribution help the user to infer the eld type.
[INCLUDE GRAPH]
Data typing
Primary types [binary, ascii, num, base64...]
Denition domain, unique elements and intervals
Data carving (tar gz, png, jpg, ...)
Semantic data identication (emails, IP ...)
Domain of denition aaa
Change type representation aaa
Field dependencies from the same message and from the environment
Fields dependancies identication
Length elds and associated payloads
Encapsulated messages identications
And from the environment...
Payload extraction The function Find Size Fields, as its name suggests, is dedicated to nd elds that contain
any length value as well as the associated payload. It does this on each group. Netzob supports different encoding of
the size eld : big and little endian binary values are supported through size of 1, 2 and 4 bytes. The algorithm used to
nd the size elds and their associated payloads is desribed in the table XXX.
[INCLUDE ALGORITHM]
The following picture represents the application of the function on a trace example. It shows the automated extraction
of the IP and UDP payloads from an Ethernet frame.
1.1. Table of contents 9
Netzob Documentation, Release 0.4.1
Field semantic inference
Data carving Data carving is the process of extracting semantic information from elds or messages. Netzob allows
the extraction of the following semantic information :
URL
email
IP address
[INCLUDE FIGURE]
Search aaa
Properties aaa
Grammar inference
Identication of the automata of the protocol
Fields dependencies with messages of previous states
10 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
1.1.4 Vocabular inference
Structure inference
Regoupment of similar structures
Options during alignment process
read-only process (do not require a participation in the communication).
Identify the xed and dynamic elds of all the messages.
Regroups equivalent messages depending of their elds structures.
Clustering (Regroups equivalent messages using) :
an UPGMA Algorithm to regroup similar messages
an openMP and MPI implementation
Sequencing, Alignment (Identication of elds in messages) :
Needleman & Wunsch Implementation
Needleman and Wunsch algorithm
Originaly a bio-informatic algorithm (sequencing DNA)
Align two messages and identify common patterns and eld structure
Computes an alignment score representing the efciency of the alignment
The following picture shows the sequence alignment of two messages.
UPGMA algorithm
Identify equivalent messages based on their alignment score.
Build a hierarchical organization of the messages with the UPGMA algorithm (Unweighted Pair Group Method
with Arithmetic Mean)
The following picture shows a regroupment of similar messages based on the result of the clustering process.
1.1. Table of contents 11
Netzob Documentation, Release 0.4.1
Abstraction of a set of message
The abstraction is the process of substituting the dynamic elds with their representation as a regex. An example of
abstraction is shown on the follinw picture.
Analyses after alignment process
aaa
Message contextual menu
aaa
Group contextual menu
aaa
12 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
Rene regexes
aaa
Slick regexes
aaa
Concatenate
aaa
Split column
aaa
Merge columns
aaa
Delete message
aaa
Field type inference
Visualization options
aaa
Type structure contextual menu
aaa
Messages distribution
This function shows a graphical representation of the distribution of bytes per offset for each message of the cur-
rent group. This function helps to identify entropy variation of each elds. Entropy variation combined with byte
distribution help the user to infer the eld type.
[INCLUDE GRAPH]
1.1. Table of contents 13
Netzob Documentation, Release 0.4.1
Data typing
Primary types [binary, ascii, num, base64...]
Denition domain, unique elements and intervals
Data carving (tar gz, png, jpg, ...)
Semantic data identication (emails, IP ...)
Domain of denition
aaa
Change type representation
aaa
Field dependencies from the same message and from the environment
Fields dependancies identication
Length elds and associated payloads
Encapsulated messages identications
And from the environment...
Payload extraction
The function Find Size Fields, as its name suggests, is dedicated to nd elds that contain any length value as well
as the associated payload. It does this on each group. Netzob supports different encoding of the size eld : big and
little endian binary values are supported through size of 1, 2 and 4 bytes. The algorithm used to nd the size elds and
their associated payloads is desribed in the table XXX.
[INCLUDE ALGORITHM]
The following picture represents the application of the function on a trace example. It shows the automated extraction
of the IP and UDP payloads from an Ethernet frame.
14 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
Field semantic inference
Data carving
Data carving is the process of extracting semantic information from elds or messages. Netzob allows the extraction
of the following semantic information :
URL
email
IP address
[INCLUDE FIGURE]
Search
aaa
Properties
aaa
1.1. Table of contents 15
Netzob Documentation, Release 0.4.1
1.1.5 Grammar inference
Identication of the automata of the protocol
Fields dependencies with messages of previous states
1.1.6 Export
Netzob supports the export of the modelization of the protocol in the following formats: * XML meta representation
* Dedicated Scapy Dissector * Dedicated Wireshark Dissector
Each export format in described in this chapter.
XML Export
Scapy Dissector
The Scapy documentation about building new dissectors is available at the following address:
http://www.secdev.org/projects/scapy/doc/build_dissect.html
Wireshark Dissector
The Wireshark documentation about building new dissectors is available at the following address:
http://www.wireshark.org/docs/wsdg_html_chunked/ChDissectAdd.html
1.1.7 Simulation
Todo
1.1.8 Fuzzing
Live instrumentation through a dedicated proxy
Possibilities of variations :
...
...
1.1.9 Annexes
GOT Poisoning
This idea has rstly been described by Ryan ONeill in its article Modern Day ELF Runtime infection via GOT
poisoning.
Netzob has the following needs :
inject between an application and its libraries;
runtime injection,
16 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
dump any value of the variables which are transfered from an application to a libs and vice et versa,
modify the value of any variables which are transfered from an application to a libs and vice et versa,
hidden approach,
Those needs are generated by the following functionalities :
detection of all the running processes ;
lter for dynamically linked processes ;
display the available libs of a process ;
display all the entry functions of a chosen lib ;
inject a proxy between multiple functions of a lib and the application ;
the injected proxy must be controllable (dump and modify variables).
As an example, Netzob must be able to :
fetch all the sent and received message in an HTTPS connection ;
modify for fuzzing experiments in an HTTPS connection.
Glossary
First a bit of vocabulary :
Term Description
GOT Global Offset Table
PLT Procedure linking table
How it works
This approach is composed of two object :
an injector which has the responsibility to inject the parasite into a running process;
a parasite which aims are to dump and or to modify the values of the variable transfered between a process and
its libs.
And the scenario of the hijacking algorithm is :
1. Locate binary of targeted process by parsing /proc/<pid>/maps
2. Parse PLT to get desired GOT address
3. Attach to the process
4. Find a place to inject the parasite loader shellcode
5. Inject new code and save original code we are overwriting
6. Modify EIP (save old EIP) to point to our code
7. Resume traced process so that it executes our parasite loader shellcode and load our parasite
8. Reset register, replace original code and allow process to resume
9. get base address of our parasite from %eax
10. nd address of parasite function within shared lib by scanning for its code sequence
1.1. Table of contents 17
Netzob Documentation, Release 0.4.1
11. Retrieve and save value stored in desired GOT address
12. Modify the return address of the parasite function with the original function address
13. Overwrite desired GOT address with new value
14. Detach from process and enjoy
Step1 : Locate binary of targeted process by parsing /proc/<pid>/maps
INPUT :
PID of the process
OUTPUT :
the base address of the application
the full path of the binary
OPERATIONS :
Parse the le /proc/<pid>/maps.
Step2 : Parse PLT to get desired GOT address
INPUT :
the full path of the binary
name of the function to hijack
OUTPUT :
the GOT address / offset of the function to hijack
OPERATIONS :
Parse the binary (like readelf -r does)
Step3 : Attach to the process
INPUT :
PID of the process
OUTPUT :
OPERATIONS :
Attach to the process using PTRACE_ATTACH
Step4 : Find a place to inject the parasite loader shellcode
INPUT :
base address of the text segment of the process
parasite loader shellcode
OUTPUT :
18 Chapter 1. The big picture
Netzob Documentation, Release 0.4.1
the [start-end] offset in the text segment for the injection of shellcode
OPERATIONS :
Computes the offset in the segment of the injected
Step5 : Inject new code and save original code we are overwriting
INPUT :
start offset of the injection in the text
parasite loader shellcode
OUTPUT :
the [start-end] offset in the text segment for the injection of shellcode
OPERATIONS :
backup the original code into memory using ptrace(PTRACE_PEEKTEXT ;
load shellcode into text starting at offset using ptrace(PTRACE_POKETEXT ;
Step6 : Modify EIP (save old EIP) to point to our code
INPUT :
start offset of the injection in the text
OUTPUT :
OPERATIONS :
save the registre using ptrace(PTRACE_GETREGS ;
modify reg.eip using ptrace(PTRACE_GETREGS to the start offset
Step7 : Resume traced process so that it executes our parasite loader shellcode and load our parasite
INPUT :
OUTPUT :
OPERATIONS :
resume execution using trace(PTRACE_CONT
Step8 : Reset register, replace original code and allow process to resume
INPUT :
OUTPUT :
OPERATIONS :
wait for the end of application using * wait(NULL);* ;
retrieves the values of the registers and reset them to default ones
detach from process using ptrace(PTRACE_DETACH
1.1. Table of contents 19
Netzob Documentation, Release 0.4.1
What is a workspace ?
A workspace contains all the resources les associated with the current analysis which includes : * the repository of
automaton, * the logging conguration, * the list of available prototypes of functions which can be hijacked, * the
repository of imported traces, * the global conguration le.
1.1.10 API
API description
class netzob.Common.Workspace.Workspace(name, creationDate, path, pathOfTraces, pathOfLog-
ging, pathOfPrototypes, lastProjectPath=None, im-
portedTraces={})
Class denition of a Workspace
class netzob.Common.Project.Project(id, name, creationDate, path)
Class denition of a Project
The API section has a complete list of all the classes, methods, attributes and functions of the netzob module,
together with short examples for many items.
A day Netzob will have a proper and efcient documentation. But this day, PYTHON will also have one ! :)
20 Chapter 1. The big picture
CHAPTER
TWO
INDICES AND TABLES
genindex
modindex
search
21
Netzob Documentation, Release 0.4.1
22 Chapter 2. Indices and tables
PYTHON MODULE INDEX
n
netzob.Common.Models.Factories.FileMessageFactory,
5
netzob.Common.Models.FileMessage, 5
netzob.Common.Project, 20
netzob.Common.Workspace, 20
23