building a yaf+silk testing vm
I am currently in the process of developing a collection of tools to provide network security monitoring and traffic analysis capabilities on a Raspberry Pi 3 called berry sense. After speaking with my mentor about this, he suggested I take a look at SiLK, a set of traffic analysis tools. Before any real development could begin, I needed to create a testing environment for these tools. This VM would be used solely for testing YaF+SiLK and generating output files that could later be used as template files for understanding how to process the data.
The SiLK Suite is a set of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA).
From the main website:
The SiLK tool suite supports the efficient collection, storage, and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets.
The suite primarily consists of command-line tools for processing SiLK flow records generated by the SiLK Packing System. The packing system is comprised of daemon applications that collect flow data and convert them to a more space-efficient outputs binary format which can then be partitioned, sorted, and counted using the collection of tools provided. Flow data must be IPFIX flows generated by yaf or NetFlow v5 or v9 PDUs from a suitable router. The application we will be using is
Silk collects records from sensors, typically routers. Since I had no suitable routers available, I would have to use yaf to generate IPFIX flows from the raw network traffic.
YaF (Yet Another Flowmeter) “processes packet data from pcap dumpfiles as generated by tcpdump or via live capture from an interface using pcap into bidirectional flows, then exports those flows to IPFIX Collecting Processes or in an IPFIX-based file format”.
YaF will be used to process saved pcap files from scheduled captures or do live captures and send them to
rwflowpack so we can then further process the data using the other SiLK tools.
Testing Environment Setup
I decided to begin testing YaF and SiLK by installing both to a virtual machine.
- Virtualbox 5.1.14
- OS: Ubuntu Server 16.04.2
- RAM: 1GB
- Network Adapters
- enp0s3 - NAT adapter (192.168.56.0/24)
- enp0s8 - Host-only adapter (10.10.16.0/24)
After the OS was installed and configured, the next step was installing YaF and SiLK. Both packages are available for download from the following sites:
I began with the YaF installation first.
Before building YaF, there were a few dependencies that needed to be installed.
sudo apt-get install build-essential pkg-config libfixbuf3 libfixbuf3-dev libpcap0.8-dev
Once these dependencies were installed, I moved into the directory where I had extracted yaf and went through the standard build process.
./configure make sudo make install
This produced a successful build and installation.
Resource: YaF Documentation
Most of SiLK’s dependencies were covered by YaF, so all that wsa left to install was
sudo apt-get install python-dev
Before installing Silk, it was necessary to create a directory where it’s data would be stored. Following the documentation from the CERT site, I created
/data as the storage directory.
sudo mkdir /data
Next, I changed into the directory where I had extracted SiLK and went through the standard build process again, passing the
--with-python flag to
./configure to include Python support.
./configure --with-python make sudo make install
Installation completed successfully.
Per the documentation, to avoid exporting LD_LIBRARY_PATH each time SiLK is used, I added the following paths to ld.so.conf like so:
cat <<EOF >>silk.conf /usr/local/lib /usr/local/lib/silk EOF sudo mv silk.conf /etc/ld.so.conf.d/
Finally, I ran ldconfig to update changes:
First, I copied the default configuration from the directory where SiLK was extracted at
site/twoway/silk.conf to the
sudo cp site/twoway/silk.conf /data
Next, I created the
sensors.conf file with the following settings:
probe S0 ipfix listen-on-port 18001 # you may need to allow this port through the firewall so that yaf can talk to it protocol tcp listen-as-host 127.0.0.1 end probe group my-network ipblocks 192.168.56.0/24 # address of the host-only adapter (change this to the appropriate address) ipblocks 10.0.2.0/24 # address of the NAT adapter end group sensor S0 ipfix-probes S0 internal-ipblocks @my-network external-ipblocks remainder end sensor
And saved this file to
I first made a copy of the default configuration stored at
/usr/local/share/silk/etc/rwflowpack.conf and changed the these settings to the following values:
ENABLED=yes CREATE_DIRECTORIES=yes SENSOR_CONFIG=/data/sensors.conf SITE_CONFIG=/data/silk.conf LOG_TYPE=legacy LOG_DIR=/var/log/
And saved the file to
Finally, I copied the start up script that was installed by SiLK to
/etc/init.d, set it to start on boot, and started the service.
sudo cp /usr/local/share/silk/etc/init.d/rwflowpack /etc/init.d sudo sudo update-rc.d rwflowpack start 20 3 4 5 . sudo service rwflowpack start
Per default settings,
rwflowpack listens on port 18001.
I now have a working standalone flow collection and analysis test environment. I can now we use
rwfiler and other tools provided SiLK to generate files then can later be process and formatted further with code. A future post will show the results of running some tests on the local machine and generating records.
YaF+SiLK Usage Example
In order to begin capturing and generating records, YaF must be started. Here is an example:
sudo nohup yaf --silk --ipfix=tcp --live=pcap --out=127.0.0.1 --ipfix-port=18001 --in=eth0 --applabel --max-payload=384 &
eth0 to the appropriate interface we want to capture on.
Now we can generate some traffic using ping or some other method and wait for the records to be flushed. It can take between 10-15 for the first records to be flushed. Coffee time!