Securing connection to check_mk agent with stunnel

Check_mk is quite nice to monitor hosts in your own network, however if you have remote server that you would like to monitor it’s not so secure, because check_mk agent is sending all its data as clear text. Of course you can limit connection to only one remote ip with firewall, or even with xinetd, but what about monitoring hosts running on dynamic external IP or even scuring data transfer between hosts. You simply cannot put DNS name to firewall or xinetd, that’s why you can use stunnel to secure connection.

It will act in two ways:

  • securing data transfer through the internet
  • adding authentication layer in front of check_mk agent

Please follow this how-to, it will show you how to secure connection between check_mk server running CentOS 7 and check_mk agent running on CentOS 7. This how to is not describing the way to install cehck_mk nor check_mk agent on the hosts.

Both sites (remote site and check_mk site)

First of all install install stunnel form CentOS 7 base repo:

yum install stunnel

After installation was completed you have to create systemd unit file for this service in /etc/systemd/system/stunnel.service:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target network.target
 
[Service]
ExecStart=/usr/bin/stunnel
Type=forking
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target

Now you are ready to configure stunnel.

Server site (where check_mk server lives)

Now you are ready to configure stunnel (/etc/stunnel/stunnel.conf):

client = yes
[check_mk_remote]
cert = /etc/pki/tls/certs/[cert_name].pem
accept = 127.0.0.1:6557
connect = [remote_ip]:6556

The cert file that is mentioned in config file will be generated on the remote site, so just copy the cert after you’ve generated it and then you can enable and start service.

systemctl enable stunnel.service
systemctl start stunnel.service

After that we can set up remote site.

Remote site (host which you would like to monitor)

This is how the configuration of stunnel should look like:

cert = /etc/pki/tls/certs/stunnel.pem
sslVersion = TLSv1
setuid = nobody
setgid = nobody
pid = /tmp/stunnel.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
output = /var/log/stunnel.log
 
[check_mk_agent]
accept = [external_ip_of_remote_location]:6556
connect = localhost:6556 TIMEOUTclose = 0

As you can see above there is a cert that is used in this config. We should generate it:

openssl req -new -x509 -days 3650 -nodes -out /etc/pki/tls/certs/stunnel.pem -keyout /etc/pki/tls/certs/stunnel.pem
dd if=/dev/urandom count=2 | openssl dhparam -rand - 512 >> /etc/pki/tls/certs/stunnel.pem

During the generation process you will have to answer few questions. Second command will generate DH parameters and will append them to the cert file.

Of course there is one more thing to do: enable logging.

touch /var/log/stunnel.log
chown nobody:nobody /var/log/stunnel.log

Before starting services we should edit check_mk.socket systemd unit:

(add 127.0.0.1 to ListenStream in /etc/systemd/system/check_mk.socket)

# systemd socket definition file
[Unit]
Description=Check_MK Agent Socket
 
[Socket]
ListenStream=127.0.0.1:6556
Accept=true
 
[Install]
WantedBy=sockets.target

Now we are ready to enable and start the services:

systemctl enable check_mk.socket
systemctl start check_mk.socket
systemctl enable stunnel.service
systemctl start stunnel.service

Add host in check_mk

The last step is to add the host in the check_mk.

Basically you should add host as usual just configuring some additional parameters:

IPv4 Address should be changed to the localhost and you should create the rule for tcp port of agent for this host:

Adding another hosts this way

You can add as many hosts as you wish, all you need to do is just multiply section [check_mk_remote] on monitoring hosts (of course changing name and port for each section) and add more rules in check_mk.

You can even add another services this way. Just modify stunnel.conf file accordingly.

Have fun!

Tagged with: , , , , ,

Deleting tons of files in Linux using rsync

If You have a problem with deleting tons of files in linux (when for example You’ve came across this problem: /bin/rm: Argument list too long.) You can try find, perl scripts and so on (you can find some more info here), but if described there ways to do this are not successful You can try my way to cope with this problem using rsync.

First of all You need to tune Your system a little bit (put those commands as root in console):

sysctl -w vm.dirty_ratio=10
sysctl -w vm.dirty_background_ratio=5

(why we are doing this)

Now we are ready to delete those files:

Create empty directory:

mkdir /empty

Now use rsync to copy empty directory over Your directory (attention it will delete all the contents of destination directory):

rsync -a --delete /empty/ /todelete/

It can take a lot of time, and maybe You want to use ionice to be nice for other processes and services that are using disk during this operation. It is also good to set this command on screen.

After above command will end its job just do:

rmdir /todelete

Of course it can also be I/O consuming operation so You may also want to use ionice command.

The above way helped me to delete almost 100 millions of files in one directory where find, perl and other ways were not successful.

Tagged with: , , , ,

Zabbix – check if host is on dnsbl

Sometimes You need to monitor status of Your host in case it is blacklisted (especially mail servers).

I wrote simple script and template for Zabbix to do this.

Here it is:

zabbix_dnsbl

This is early version, so feel free to send bugs reports and feature request here.

Tagged with: , , ,

Restoring corrupted InnoDB MySQL databases

Recently my Zabbix MySQL database was corrupted. Unfortunately I’ve needed historical data (database backup was too old), so there was only one way: restore everything I can from corrupted database. On the other hand I had every table in the separate file (innodb_file_per_table=1 in my.cnf), which was very helpful.

There are three ways to restore corrupted InnoDB databases (you should decide which one to choose, sometimes You will need to use not only one):

  • manually importing files to newly created database
  • using Percona InnoDB recovery tools
  • using innodb_force_recovery

For above methods You will need to have files from Your datadir (for example: /var/lib/mysql), so copy it somwhere.

Manually importing files

For this method You need to have ibd files from MySQL’s datadir and You need to know how was the table created (whole create command).

First step is to create new database, so login to MySQL and create it:

create database corrupted;

Now create table:

use corrupted;
CREATE TABLE `maintenances` (
	`maintenanceid`          bigint unsigned                           NOT NULL,
	`name`                   varchar(128)    DEFAULT ''                NOT NULL,
	`maintenance_type`       integer         DEFAULT '0'               NOT NULL,
	`description`            text                                      NOT NULL,
	`active_since`           integer         DEFAULT '0'               NOT NULL,
	`active_till`            integer         DEFAULT '0'               NOT NULL,
	PRIMARY KEY (maintenanceid)
) ENGINE=InnoDB;

And here is a tricky part – You need to discard tablespace by invoking this command in MySQL:

use corrupted;
ALTER TABLE maintenances DISCARD TABLESPACE;

Next step is to copy old file to correct place (using OS shell, not MySQL):

cp /var/lib/mysql-old/zabbix/maintenances.ibd /var/lib/mysql/corrupted/

After that You need to login to MySQL again and import new tablespace:

use corrupted;
ALTER TABLE maintenances IMPORT TABLESPACE;

In same cases after above steps You will be able to dump this table using mysqldump tool, but it is very often that MySQL will produce this error:

ERROR 1030 (HY000): Got error -1 from storage engine

After that simple go to MySQL log file and see why it is happening. In my case it was:

InnoDB: Error: tablespace id in file './zabbix/maintenances.ibd' is 263, but in the InnoDB data dictionary it is 5.

If the above error occurred You need to start from the beginning but with another method.

Percona InnoDB recovery tools

First You need  those tools – simply visit percona site and download it, unpack it and build those tools (You will find more info how to do this inside this archive). After that You are ready to repair above MySQL error. To do this follow next steps:

Drop table from corrupted database, and create it again (the same way as it was created before).

Stop MySQL daemon! – it is necessary.

Copy table file (overwrite it):

cp /var/lib/mysql-old/zabbix/maintenances.ibd /var/lib/mysql/corrupted/

Use ibdconnect:

./ibdconnect -o /var/lib/mysql/ibdata1 -f /var/lib/mysql/corrupted/maintenances.ibd -d zabbix -t maintenances

There will be some output and on the end there should be:

SYS_INDEXES is updated successfully

Now we can repair ibdata1 file:

./innochecksum -f /var/lib/mysql/ibdata1

Repeat this step until there will be no output.

Now You can start MySQL daemon again and You should be able to dump this table, if not follow instructions to see the last method.

Use innodb_force_recovery

In this method we will just copy table file and power up MySQL with innodb_force_recovery parameter. Here are the steps:

Change MySQL configuration. In [mysqld] section set datadir to Your copy of MySQL files, and set innodb_force_recovery parameter to 6:

datadir=/var/lib/mysql
 
innodb_force_recovery=6

Restart MySQL and You should be able to dump all corrupted tables by mysqldump.

 

Hope this post will help You. If You have some questions please leave comment below.

Tagged with: , , ,

Quick Tip: Centos 6.5 and zabbix agent 2.2.1

If You are facing with one of the following problems after upgrading CentOS and Zabbix agent to the latest stable realese (CentOS 6.5 and zabbix agent 2.2.1):

  • You cannot get autodiscovered items data (for examle: network interfaces bandwidth)
  • zabbix cannot collect data from MySQL (using provided by zabbix authors MYSQL template
  • Just try to:

  • disable SELinux
  • set SELinux to permissive mode (setenforce 0)
  • try to update SELinux policies
  • Tagged with:

    Using OMSA and snmp on CentOS to monitor Dell servers in zabbix.

    Recently I’ve needed to configure more advanced monitoring of Dell servers with hardware Raid, becasue of disk failure in one of my servers. I’ve came across a lots of information how to configure OMSA in Centos, but every description and howto shows that You need to install every component from OMSA to run it properly, so I’ve decided to write my own note how to do it without installing unnecessary software from OMSA repository. Here are the steps to install OMSA snmpd support:

    First of all install OMSA repository on Your CentOS system. To do this follow instructions on this site:

    http://linux.dell.com/wiki/index.php/Repository/OMSA#Yum_setup

    Now You can install necessary software and run snmpd with omsa support:

    yum install srvadmin-megalib srvadmin-rac-components srvadmin-server-snmp srvadmin-deng-snmp srvadmin-base srvadmin-storelib-sysfs srvadmin-deng srvadmin-isvc srvadmin-smcommon srvadmin-sysfsutils srvadmin-storage-snmp srvadmin-omilcore srvadmin-hapi srvadmin-cm srvadmin-idrac-snmp srvadmin-storage srvadmin-storageservices-snmp srvadmin-omacs srvadmin-storelib srvadmin-ominst srvadmin-xmlsup srvadmin-isvc-snmp

    Of course You need to set up snmpd to run as You wish (You will find how to install and configure snmpd on google ;) ), and as dependency You need to configure snmpd to get values from OMSA – just edit this file:

    /etc/snmp/snmpd.conf

    And add this line on the end:

    smuxpeer .1.3.6.1.4.1.674.10892.1

    You are almost ready to read OMSA variabled from snmp, just fire up all needed services:

    /etc/init.d/dataeng start
    /etc/init.d/snmpd restart

    And add those services to start automatically during boot:

    chkconfig dataeng on
    chkconfig snmpd on

    Now You can check if everything works as You wish (this command is for default configuration of snmpd on CentOS, and of course You need to have net-snmp-utils package installed):

    snmpwalk -v 2c -c public 127.0.0.1 .1.3.6.1.4.1.674.10892.1

    If everything is working as we needed You should see a lot of lines with informations.

    Now it is time to configure zabbix to read this data. I’ve found nice zabbix template on zabbix forums. Just download it from here: https://www.zabbix.com/forum/showthread.php?t=22054 and import in zabbix.

    Last step to get it working in zabbix is to Link this template to servers You need to be monitored.

    Hope this post will be helpful for somebody.

    Tagged with: , , ,

    Running raspbian in QEMU on Fedora 19

    As I mentioned earlier (Zabbix templates for Raspberry PI), I want to develop Zabbix template for Raspebrry PI, but unfortunatelly my Raspberry PI could no be used to do such things now, so I decided to run raspbian on QEMU.

    Here are the steps to run raspbian in QEMU on Fedora 19.

    First of all You need install some packages and its dependencies:

    yum install qemu-system-arm

    Now we are ready to download things that we will need to run raspbian in QEMU.

    # Linux Kernel for QEMU
    wget http://xecdesign.com/downloads/linux-qemu/kernel-qemu
    # Raspbian Wheezy
    wget http://raspberry.mythic-beasts.com/raspberry/images/raspbian/2013-07-26-wheezy-raspbian/2013-07-26-wheezy-raspbian.zip

    Now we are redy to set up everything. It will take few simple steps:

    # unzip Raspbian Wheezy image
    unzip 2013-07-26-wheezy-raspbian.zip
    # put everything in one place:
    mkdir rpi-qemu
    cp kernel-qemu rpi-qemu/
    cp 2013-07-26-wheezy-raspbian.img rpi-qemu/
    cd rpi-qemu
    # before mountin image You need to figure some thins:
    file 2013-07-26-wheezy-raspbian.img
    # now get "startsector" value for parition 2 and multiply by 512
    mount ~/qemu_vms/2013-02-09-wheezy-raspbian.img -o offset=<multiplied_value> /mnt
    # modify some files
    cd /mnt/etc/
    # edit this file: ld.so.preload
    nano ld.so.preload
    # comment out the line that is there by putting "#" before it
    # save this file
    # umount image:
    umount /mnt

    After above steps we are ready to fire Raspbian up.

    qemu-system-arm -kernel rpi-qemu/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot --append "root=/dev/sda2 panic=1" -hda rpi-qemu/2013-07-26-wheezy-raspbian.img -redir tcp:2222::22 -daemonize

    After issuing above command You should see smth like that:
    QEMU running raspbian
    After some time booting QEMU will give You root shell to fsck corrupted filesystem, just enter those command:

    # check filesystem
    fsck -y /dev/sda2
    # reboot system
    shutdown -r now

    After those steps You are ready to power it up again and start using it almost like normal Raspberry PI.

    Default username is pi and password is raspberry.

    Of course there are some points that must be mentioned:

      • ping will not work, but networking will
      • You cannot use commands from /opt/vc – it is just not working (why?? – because all of them are hardware related commands)
      • You can login to Your Raspbian via ssh from localhost:
    ssh pi@127.0.0.1 -p 2222
    Tagged with: , ,

    Zabbix templates for Raspberry PI

    Hello,
    recently I was searching some Zabbix templates for Raspberry PI, but I didn’s succeed. I’ve decided I will do some Zabbix templates for Raspberry PI. Now I am preparing LAB for it. So stay tuned – I will post templates soon.

    Tagged with: , ,

    Compiling zabbix-agent on Raspberry Pi (xbian or debian)

    First of all You need to install some software for compilation:

    apt-get install build-essential

    Now, just get sources of zabbix (in this case the latest version is 2.0.6):

    wget "http://garr.dl.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Stable/2.0.6/zabbix-2.0.6.tar.gz"

    Unpack it, go to unpacked sources:

    tar zxfv zabbix-2.0.6.tar.gz
    cd zabbix-2.0.6

    Now You need to specify configuration options, mine was like:

    ./configure --enable-agent --prefix=

    And You are ready to compile and to install zabbix-agent on Your system:

    make
    make install

    Also if You need startup scripts follow next steps:

    cd misc/init.d/debian #in zabbix source's directory
    cp zabbix-agent /etc/init.d/
    which zabbix_agentd #outup will be needed to edit zabbix-agent startup script

    Open /etc/init.d/zabbix-agent file for edit end edit this line:

    DAEMON=/usr/local/sbin/${NAME} #put here output of previous command, i.e /sbin/${NAME}

    Last thing to do is to add zabbix user, because zabbix cannot run as root user:

    adduser --system zabbix

    Now You should be able to start zabbix-agent through init scripts, of course You need to configure zabbix-agent in /etc/zabbix_agent.conf and /etc/zabbix_agentd.conf

    After configuration is completed You should be able to start zabbix-agent:

    /etc/init.d/zabbix-agent start

    If You want to start zabbix-agent automaticaly during boot process You need to run this command:

    update-rc.d zabbix-agent defaults

    And that’s all.

    Tagged with: , , , ,

    Monitoring ip_conntrack on zabbix

    I needed some statistics and triggers on zabbix of ip_conntrack. Here is how to monitor it, of course You can expand it as You wish.

    First of all You need to add some configuration to Your zabbix agents. Put those lines at the end of zabbix config (in my case it was:  /etc/zabbix/zabbix_agentd.conf):

    UserParameter=ip_conntrack_count,cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
    UserParameter=ip_conntrack_max,cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max

    And then restart zabbix agent:

    /etc/init.d/zabbix-agent restart

    Now You can configure zabbix, so let’s login and import this file:

    Zabbix template

    From now on, You are able to monitor the state of ip_conntrack on Your system.

    Tagged with: , , ,
    Top