%%
%% This is file `squelette-rr.tex',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% RR.dtx  (with options: `sample')
%% ********************************************************************
%% Copyright (C) 1997-1999 2004 2006-2011 INRIA/APICS/MARELLE by Jose' Grimm
%% This file may be distributed and/or modified under the
%% conditions of the LaTeX Project Public License, either version 1.3
%% of this license or (at your option) any later version.
%% The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.3 or later is part of all distributions of LaTeX
%% version 2003/12/01 or later.
%% An archive of the software can be found at
%%    ftp://ftp-sop.inria.fr/marelle/RR-INRIA

\documentclass[twoside]{article}
\usepackage[a4paper]{geometry}
\usepackage[latin1]{inputenc} % ou \usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc} % ou \usepackage[OT1]{fontenc}
\usepackage{RR, listings}
\usepackage{hyperref}
\lstset{%
	basicstyle=\footnotesize,
	language=Java,
	numbers=left,
	numbersep=5pt,
	breaklines=true,
	frame=single
}
%%\usepackage[frenchb]{babel} % optionnel
\RRNo{7003}
%% \RTNo{0703}
%%
%% date de publication du rapport
\RRdate{April 2013}
%%
%% Cas d'une version deux
%% \RRversion{2}
%% date de publication de la version 2
%% \RRdater{November 2008}
%%
\RRauthor{% les auteurs
 % Premier auteur, avec une note
Julien Vaubourg\thanks{TELECOM Nancy (last year internship in MADYNES)}}
%% Ceci apparait sur chaque page paire.
\authorhead{Vaubourg}
%% titre francais long
\RRtitle{Programmation de l'exporteur NefFlow v9 du projet Flowoid sur Android}
%% English title
\RRetitle{Programming the Flowoid NetFlow v9 Exporter on Android}
\titlehead{Flowoid NetFlow v9 Exporter on Android}
%%
%\RRnote{This is a note}
%\RRnote{This is a second note}
%%
\RRresume{Ce document explique comment programmer l'exporteur de NetFlow v9
sur Android~: gestion des pilotes, d\'efinition des templates, g\'en\'eration des
donn\'ees et filtrage des paquets.}
\RRabstract{
The RFC 3954 documents the NetFlow services export protocol Version 9. This project
- associated to the Flowoid Projet - allows an Android program to listen the network
traffic and send it to a remote NetFlow Collector using this standard.\\

Thanks to an highly scalable design, any sort of template can be implemented and any kind
of network can be exported as NetFlow. The data generator side uses the flexibility of Java
(with the Android API) whereas the sniffer side uses the power of C (native code).\\

This document explains how to configure the Exporter in order to export consistent
information to the remote collector.}
%%
\RRkeyword{netflow, flowoid, network traffic, monitoring, security, android}
\RRmotcle{netflow, flowoid, trafic r\'eseau, supervision, s\'ecurit\'e, android}
%%
\RRprojet{MADYNES}  % cas d'un seul projet
%%\RRprojets{Apics and Op\'era and Marelle}
%%
%% \URLorraine % pour ceux qui sont \`a l'est
%% \URRennes  % pour ceux qui sont \`a l'ouest
%% \URRhoneAlpes % pour ceux qui sont dans les montagnes
%% \URRocq % pour ceux qui sont au centre de la France
%% \URFuturs % pour ceux qui sont dans le virtuel
%% \URSophia % pour ceux qui sont au Sud.
%%
%% \RCBordeaux % centre de recherche Bordeaux - Sud Ouest
%% \RCLille % centre de recherche Lille Nord Europe
%% \RCParis % Paris Rocquencourt
%% \RCSaclay % Saclay \^Ile de France
%% \RCGrenoble % Grenoble - Rh\^one-Alpes
%% \RCNancy % Nancy - Grand Est
%% \RCRennes % Rennes - Bretagne Atlantique
\URLorraine

%%
\begin{document}
%%
%\makeRR   % cas d'un rapport de recherche
\makeRT % cas d'un rapport technique.
%% a partir d'ici, chacun fait comme il le souhaite
\section{Design}

NetFlows are traveling between two sides:

\begin{itemize}
\item \textbf{Exporter:} Catches all network packets through Drivers (directly on the Android device), resumes network flows with the NetFlow format, generates
some additional data to add into NetFlows and exports them to the remote Collector.
\item \textbf{Driver:} Listens the traffic network on the Exporter side and transmits packets headers in live to the Exporter.
\item \textbf{Collector:} On the other side (remote server), receives and treats NetFlows sent by the Exporter.
\end{itemize}

~\\
In order to transmit NetFlows, an Exporter send one or more Export Packets to a remote Collector:

\begin{itemize}
\item \textbf{Export Packet:} Contains one or more Data FlowSets, and sometimes a Template FlowSet.
\item \textbf{Data FlowSet:} Contains one or more Data Records. All of these are defined with the same Template Record and
the Data FlowSet ID corresponds to the ID of this Template Record.
\item \textbf{Data Record:} Contains one or more Data Fields. This kind of data corresponds to what is generally called a NetFlow.
\item \textbf{Data Field:} Contains only one value (e.g. IP address, TCP port, timestamp).
\end{itemize}

~\\
The Collector knows how parsing Export Packets because this one sends regularly a Template Flowset among the Data FlowSets:

\begin{itemize}
\item \textbf{Template FlowSet:} Contains one or more Template Records. Its FlowSet ID is inevitably zero.
\item \textbf{Template Record:} Contains one or more Template Fields ordered.
\item \textbf{Template Field:} Is defined with a Field Type Definition (i.e. an ID and a length in bytes).
\item \textbf{Field Type Definition:} Described in RFC 3954 section 8. A Data Field associated with a Template Field should be
interpreted by the remote Collector in the same manner as described in RFC. Some types have a fixed length.
\end{itemize}

~\\
The Collector side needs to know all of the Field Type Definitions used by the Exporter especially if this one is
out of standards.

\section{Environment and prerequisites}

Setting up your project:

\begin{enumerate}
\item The minimum SDK Version of your Android application must be at least 9.
\item \textbf{Your device must be rooted.}
\item Add \emph{madynes-flowoid-exporter.jar}\footnote{TODO: Add web address} to your Java Build Path.
If you want to use the IP Driver, you could also add \emph{netutils-parse-v1.jar}\footnote{TODO: Add web address}
for parsing headers.
\item Copy your Driver binary files\footnote{TODO: Add web address} in your \emph{res/raw/} directory.
\item Your \emph{AndroidManifest} file must contain at least these permissions:

\begin{lstlisting}
<uses-permission android:name="android.permission.ACCESS_SUPERUSER" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.READ_PHONE_STATE" />
\end{lstlisting}

\item Create a new class extending \texttt{ExporterService} (e.g. \emph{Exporter.java}):

\begin{lstlisting}
import madynes.flowoid.exporter.*;
import madynes.flowoid.exporter.TemplateRecord.*;
import madynes.flowoid.exporter.TemplateField.InadequateLengthException;
import madynes.flowoid.exporter.TemplateFlowSet.TemplateNotFoundException;

public class Exporter extends ExporterService {

	@Override
	public void registerTemplates() throws AlreadyUsedIDException, InadequateIDException, InadequateLengthException {

	}

	@Override
	public void processPacketFiltering(byte[] headers) throws TemplateNotFoundException {

	}

	@Override
	public void onStart() {

	}

	@Override
	public void onStop() {

	}
}
\end{lstlisting}

\item In your \emph{AndroidManifest} think to declare your Exporter Service (here \emph{your/package/Exporter.java}):

\begin{lstlisting}
<service android:name="your.package.Exporter" ></service>
\end{lstlisting}
\end{enumerate}

The next sections explain you how to implement your own Exporter.

\section{Creating templates}

All templates are defined inside of the \texttt{registerTemplates()} function.

\begin{enumerate}
\item Create all of the Template Fields you need, by creating \texttt{TemplateField} objects. The constructor needs an ID
among those available in \texttt{TemplateField.TypeDefinition.*}. If the associated length is zero, you need to use the
alternative constructor to specify a length in bytes.

\textbf{Example:}

\texttt{TemplateField tcpPortSrc = new TemplateField(TemplateField.TypeDefinition.L4\_SRC\_PORT)}.

\item Create a new Template Record with your own ID. This one is unique and must be between 256 and 65535.

\textbf{Example:}

\texttt{TemplateRecord ipv6TcpTemplate = new TemplateRecord(257);}

\item Add Template Fields to your Template Record with the \texttt{addTemplateField(TemplateField)} function. A same Template
Field instance can be reused for many different Template Records.

\textbf{Example:}

\texttt{ipv6TcpTemplate.addTemplateField(tcpPortSrc);}

\item Register your new Template Record with the \texttt{registerTemplateRecord(RecordTemplate)}.

\textbf{Example:}

\texttt{registerTemplateRecord(ipv6TcpTemplate);}
\end{enumerate}

\section{Defining how to fill your NetFlows}

Template Fields define how the data is represented and - for the standardized ones - what is it. Data Fields define how
the value can be retrieved from the System. So each Template Field should have a corresponding Data Field.\\

To create a Data Field, you have to add a new class extending \texttt{DataField}. Then you have to implement four functions:

\begin{enumerate}
\item \textbf{\texttt{onInit(byte[]):}} Called when a new NetFlow is created. The argument corresponds to the headers (depending of
the Driver used) of the first packet received for this NetFlow.
\item \textbf{\texttt{onUpdate(byte[]):}} Called when a packet is catching for an existent NetFlow. The argument corresponds to the new packet headers.
\item \textbf{\texttt{onExport(byte[]):}} Called just before the Export Packet is sent with this NetFlow (Data Record) inside. The argument corresponds
to the last packet received. But if the Data Record was flagged ended (\texttt{isEnded()} condition met) it's the end packet. E.g. in the case of a TCP
connection, if the record is ended it's the packet containing a FIN or RST flag. And not the least packet (final ACK). However in the case of an UDP
connection, if the record is ended it's the least packet before the final time out.
\item \textbf{\texttt{getValue():}} Called when the Exporter creates the Export Packet. Please use the \texttt{getByteBuffer()} function to
create a \texttt{ByteBuffer}. For instance, if this Data Field is associated to a Template Field with a length of 4 (bytes), you have to use the
\texttt{putInt(int)} function on your \texttt{ByteBuffer} before returning it.
\end{enumerate}

Example of implementation (\emph{FieldTCPPortSrc.java}):

\begin{lstlisting}
import java.nio.ByteBuffer;
import edu.huji.cs.netutils.parse.TCPPacket;

public class FieldTCPPortSrc extends DataField {
    private short value;

    @Override
    public void onInit(byte[] headers) {
        TCPPacket tcpPacket = new TCPPacket(headers);
        value = (short) tcpPacket.getSourcePort();
    }

    @Override
    public void onUpdate(byte[] headers) {}

    @Override
    public void onExport(byte[] headers) {}
	
    @Override
    public ByteBuffer getValue() {
        ByteBuffer bytes = getByteBuffer();

        bytes.putShort(value);

        return bytes;
    }
}
\end{lstlisting}

\section{Condition to cut NetFlows}

A new NetFlow is created when the combination of these discriminant fields is not yet known. This point will be explained later in
the section~\ref{associate} page~\pageref{associate}.

Conditions to stop NetFlows are evaluated for every new packet. If conditions are met, the NetFlow is moved from the current
NetFlows list to the Export Queue.

So, for each Template Record you register, you have to create a Data Record extending the \texttt{DataRecord} class and override
the \texttt{isEnded(byte[])} function. The argument corresponds to the new packet headers added to the NetFlow.\\

For instance (\emph{RecordTCP.java}):

\begin{lstlisting}
import madynes.flowoid.exporter.TemplateRecord;
import edu.huji.cs.netutils.parse.TCPPacket;

public class RecordTCP extends DataRecord {

	public RecordTCP(TemplateRecord template) {
		super(template);
	}

	@Override
	public boolean isEnded(byte[] headers) {
		TCPPacket tcpPacket = new TCPPacket(headers);
		
		return tcpPacket.isFin() || tcpPacket.isRst(); 
	}

}
\end{lstlisting}

\section{Associating data with templates}
\label{associate}

Your programming space for that is the \texttt{processPacketFiltering(byte[])} function.

\begin{enumerate}
\item First choose a previously registred Template Record with a Template Record ID.\\

\textbf{Example:}

\texttt{TemplateRecord template = getRegistredTemplateRecord(257);}

\item Then create a Data Record with one of your previoulsy created class.\\

\textbf{Examples:}

\texttt{record = new RecordTCP(template);}

\item Associate Template Fields of your Data Record with Data Fields one by one. You must associate all of them and in
the same order you added Template Fields in your Template Record. The boolean argument specify whether this Data Field 
is a part of the NetFlow Key (see below and the example section~\ref{filter} page~\pageref{filter}) or not.\\

\textbf{Example:}

\texttt{record.associateTemplateWithData(new FieldIPv6AddrSrc(), true);}

\item Finally, request the packet processing with your record.\\

\textbf{Example:}

\texttt{processPacket(record, headers);}
\end{enumerate}

Your record - in fact only fields declared as NetFlow Key part - will be used to identify the NetFlow/record associated to the current packet. If the associated NetFlow/record already exists,
the packet will be added to it. Otherwise your new record will be used as new NetFlow.

\section{Filtering packets}
\label{filter}

Every sniffed packet is sent to the \texttt{processPacketFiltering(byte[])} packet. So before creating records, you have to parse and check headers.\\

For instance:

\begin{lstlisting}
@Override
public boolean processPacketFiltering(byte[] headers) throws TemplateNotFoundException {
	LocationFinder locationFinder = LocationFinder.getInstance(this);
	DataRecord record = null;

	try {
		// TCP / IPv6
		if(EthernetFrame.statIsIpv6Packet(headers)
			&& IPFactory.isTCPPacket(headers)) {

			TemplateRecord template = getRegistredTemplateRecord("ipv6-tcp");
			record = new RecordTCP(template);

			// NetFlow Key Parts
			record.associateTemplateWithData(new FieldIPv6AddrSrc(), true);
			record.associateTemplateWithData(new FieldIPv6AddrDst(), true);
			record.associateTemplateWithData(new FieldTCPPortSrc(), true);
			record.associateTemplateWithData(new FieldTCPPortDst(), true);

			// Others
			record.associateTemplateWithData(new FieldIPv6Length(), false);
		}

		// Other kind of packet for other record type
		else if(...) {
			...
		}

		// Go! (or not)
		if(record != null) {
			processPacket(record, headers);
		}

	// Parser error
	} catch(...) {
		return false;
	}

	return true;
}
\end{lstlisting}

\section{New Driver}

Drivers are written in native code (C) in order to access to low functionnalities as sniffing network traffic.\\

Your only constraint is sending every packet headers to the Java Exporter through a TCP socket (see section~\ref{constants} page~\pageref{constants} to know the port)
and creating templates on the Exporter side. You will need a Java parser for your headers.\\

Put your sources in the \emph{jni/drivers/} directory and use the Makefile to build your Driver. This one will copy the binary in the \emph{res/raw/} directory.
Then just after the Exporter Service starting, call \texttt{Driver.useDriver} with your Driver name resource and the network interface to listen on.\\

For instance: \texttt{Driver.useDriver(this, R.raw.driver\_ip, "eth0");}.\\

Finaly just reinstall your Android application.

\section{Preferences}
\label{constants}

All of the below variables are available from your \emph{ExporterService} implementation through classical getters and setters
and you have to set them before using the Exporter Service:

\begin{itemize}
\item \texttt{prefCheckToExportInterval:} The Export Thread checks every \emph{N} seconds if it can create and send an Export Packet.
\item \texttt{prefCollectorAddr:} Collector address (IP address or hostname).
\item \texttt{prefCollectorPort:} UDP port to connect to for sending NetFlow to the remote Collector.
\item \texttt{prefDriversPort:} TCP port to listen on for receiving packets headers from drivers.
\item \texttt{prefCheckDriversInterval:} The Drivers Thread checks every \emph{N} seconds if all Drivers are still working and
restarts them if necessary.
\item \texttt{prefMaxHeadersSize:} Max headers size in bytes authorized for a sniffed packet transmitted by a Driver.
\item \texttt{prefMaxNetflowsToTrack:} Max NetFlows/records to track simultaneous.
\item \texttt{prefMinRecordsByFlowset:} Don't send a FlowSet if there are less than \emph{N} records into it.
\item \texttt{prefMaxExportPacketSize:} In order to avoid fragmentation, cut Export Packets nearly \emph{N} bytes.
\item \texttt{prefNetflowTimeout:} After \emph{N} seconds of life, a NetFlow/record is considered ended and is added to the Export Queue.
This is useful for non-connected protocols (e.g. UDP) and long term connections (e.g. TCP streaming).
\item \texttt{prefPacketTimeout:} Even though \texttt{prefMinRecordsByFlowset} is not reached, after \emph{N} seconds,
the Export Thread empties the Export Queue and sends an Export Packet.
\item \texttt{prefSendTemplatesEveryNPackets:} Send the Template FlowSet every \emph{N} Export Packet.
\item \texttt{prefTimeBeforeNetFlowEnd:} Returns the life time remaining when a NetFlow is detected as ended. This is useful
to catch the last ACK with a TCP three-way handshake avoiding to create a second NetFlow.
\end{itemize}

\section{Statistics}

All of the below getters are available from your \emph{ExporterService} implementation:

\begin{itemize}
\item \texttt{getStatActiveNetflows():} Number of currently active NetFlows/records.
\item \texttt{getStatConnectedDrivers():} Number of currently connected Drivers.
\item \texttt{getStatExportedNetFlows():} Number of NetFlows/records sent from the Exporter to the Collector.
\item \texttt{getStatExportedPackets():} Number of Export Packets sent from the Exporter to the Collector.
\item \texttt{getStatFailedSniffedPackets():} Number of sniffed packets with which a parser error has occurred at the filtering step.
\item \texttt{getStatNetflowsWaitingToBeExported():} Number of NetFlows/records currently waiting to be exported (Export Queue size).
\item \texttt{getStatRecognizedSniffedPackets():} Number of sniffed packets recognized by at least one parser at the filtering step.
\item \texttt{getStatRejectedNetFlows():} Number of NetFlows/record not created due to the\\ \texttt{prefMaxNetflowsToTrack} preference.
\item \texttt{getStatSniffedPackets():} Number of sniffed packets received by the Exporter from all Drivers.
\item \texttt{getStatUnrecognizedSniffedPackets():} Number of sniffed packets unrecognized by any parser (\texttt{statSniffedPackets} - 
\texttt{statRecognizedSniffedPackets} - \texttt{statFailedSniffedPackets}).
\end{itemize}

\section{Activity}

The minimal code for calling your Exporter is (with an IP driver, a network interface \emph{eth0} and your implementation \emph{Exporter.java}):

\begin{lstlisting}
import madynes.flowoid.exporter.ExporterService;

public class MainActivity extends Activity {

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);

		ExporterService.start(this, Exporter.class);
		Driver.use(this, R.raw.driver_ip, "eth0", "tcp or udp");
	}
}

The last empty argument corresponds to optional libcap filters.
\end{lstlisting}

\tableofcontents

\end{document}
\endinput
%%
%% End of file `squelette-rr.tex'.
