Skip to content

Detailed tutorial on installing and configuring IntelMQ with Shadowserver feeds from SCRATCH

(Author: David Ruefenacht. Thanks!!)

While the previous page described the basics on how you configure Shadowservers API key into an already pre-packaged workflow, there is potentially more to be adjusted. Should you be the more adventurious type and if you want to roll your own IntelMQ instance, this section is for you!

In this section, we go more into more details on how to configure the nitty gritty parts of IntelMQ to your liking. Note that, this section is optional.

Also, in addition, IntelMQ comes with pretty decent documentation where you can continue should you get stuck.

Requirements:

These installation instructions were tested on

Ubuntu 20.04 VM and Debian 11 Stable.

For a minimal system, these requirements should suffice:

  • 4 GB of RAM
  • 2 CPUs
  • 10 GB disk size

Depending on your data input, you will need the twentiethfold of the input data size as memory for processing.

Installation

Using Ubuntu 20.04 LTS as a base image we will be installing IntelMQ directly on the host. Install the Virtual Machine.

Using: https://intelmq.readthedocs.io/en/latest/user/installation.html

There are multiple ways to install intelmq: via the .deb package manager, or via pip or via docker... In this tutorial we chose the approach via Debian packages:

# For Debian 11:

echo 'deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/Debian_11/ /' | sudo tee /etc/apt/sources.list.d/home:sebix:intelmq.list
curl -fsSL https://download.opensuse.org/repositories/home:sebix:intelmq/Debian_11/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_sebix_intelmq.gpg > /dev/null
sudo apt update
sudo apt install intelmq

For Ubuntu:

echo 'deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/xUbuntu_20.04/ /' | sudo tee /etc/apt/sources.list.d/home:sebix:intelmq.list
curl -fsSL https://download.opensuse.org/repositories/home:sebix:intelmq/xUbuntu_20.04/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/home_sebix_intelmq.gpg > /dev/null
sudo apt update
sudo apt install intelmq

Dependencies

We're going to install the basic and optional dependencies:

Basic:

apt install python3-pip python3-dnspython python3-psutil python3-redis python3-requests python3-termstyle python3-tz python3-dateutil redis-server bash-completion jq

Optional:

apt install python3-geoip2 python3-psycopg2 python3-pyasn  python3-stomp

The initial install of IntelMQ should now be complete. You can verify that the install worked by typing

intelmqctl status

Managing Bots via commandline

We can have a look at the current bots in

/etc/intelmq/runtime.yaml

Further information concerning the bots can be found at https://intelmq.readthedocs.io/en/latest/user/bots.html

The initial installation will come with 10 bots with an initial configuration within the runtime.yaml that is located in:

/etc/intelmq/

You can find all the variables a bot needs for in the file runtime.yml

Example Bot: setting up a Shadowserver Foundation bot

Requirements: A Shadowserver API key (To get one with data concerning your external facing network sign up at: https://www.shadowserver.org/what-we-do/network-reporting/get-reports/)

Objective: we are aiming to get the data from Shadowserver, parse it and store it in a PostgreSQL database.

Basically we want to create a simple stream: Collector -> Parser -> Output We are currently using our intelmq user, your command prompt should look like this:

intelmq@$HOSTNAME:~$

We can backup the initial copy of `runtime.yaml` and start from scrap.

```bash
sudo mv /etc/intelmq/runtime.yaml /etc/intelmq/runtime.yaml.bak

Now let's create a new runtime.yaml :

touch /etc/intelmq/runtime.yaml
nano /etc/intelmq/runtime.yaml

Comment: use whatever editor you feel comfortable with.

1. Collector

The Shadowserver collector will need the following entries:

description : a short description of what the bot is doing enabled : false or true group : what group the bot is part of (collector, parser, output or expert) module : which intelmq module it uses name : a descriptive name parameters : this will depend on the parameters the bot needs destination_queues: this tell intelmq where this specific collector should send it's output. You can add multiple destination queues (is this valid for all modules actually) run_mode : this can be continuous or once

So for the Shadowserver collector the entry could look like this:

shadowserver-collector:
  description: Our bot responsible for getting reports from a Shadowserver
  enabled: true
  group: Collector
  module: intelmq.bots.collectors.shadowserver.collector_reports_api
  name: Shadowserver_Collector
  parameters:
    destination_queues:
      _default: [shadowserver-parser-queue]
    api_key: "$API_KEY_your_received_from_the_shadowserver_foundation"
    secret: "$SECRET_your_received_from_the_shadowserver_foundation"
  run_mode: continuous

TIP Keep on eye on indention as the file is indent sensitive.

EXPLAIN DESTINATION_QUEUES

We can make a test run to see if our collector is working. A first command is to use the check command:

intelmqctl check

FRAGE having followed these instructions, I get: Not logging to file: [Errno 13] Permission denied: '/opt/intelmq/var/log/intelmqctl.log Why?

If you get the following error, change back to the root user by using for exampleCTRL-D or exit. The execute the following command:

chown intelmq -R /var/intelmq/var

And return to the intelmq user su - intelmq. Rerun the intelmqctl check command to see if the issue has been resolved by not getting the previous [Errno 13]error.

Since all bot logs are written to the following path /opt/intelmq/var/log/ , we can check if our shadowserver collector functioned by issuing the command:

tail /var/log/intelmq/shadowserver-collector.log (or the name you gave to the collector)

TIP You can also follow the logs by opening another terminal on the VM and issuing:

tail -f /var/log/intelmq/shadowserver-collector.log

If all went well, you should now have a first collector up and running. But we need to prepare the information for processing for this we now want to configure the shadowserver-parser.

2. Parser

We need to add our parser to the /opt/intelmq/etc/runtime.yaml

description : a short description of what the bot is doing enabled : false or true group : what group the bot is part of (collector, parser, output or expert) module : which intelmq module it uses name : a descriptive name parameters : this will depend on the parameters the bot needs FRAGE : run_mode: this does not seem needed.

FRAGE Should possible errors be explained, mentioned?

Comment homepage is missing the abuse.ch/feodotracker bot