Deleting tons of files in Linux using rsync

If You have a problem with deleting tons of files in linux (when for example You’ve came across this problem: /bin/rm: Argument list too long.) You can try find, perl scripts and so on (you can find some more info here), but if described there ways to do this are not successful You can try my way to cope with this problem using rsync.

First of all You need to tune Your system a little bit (put those commands as root in console):

sysctl -w vm.dirty_ratio=10
sysctl -w vm.dirty_background_ratio=5

(why we are doing this)

Now we are ready to delete those files:

Create empty directory:

mkdir /empty

Now use rsync to copy empty directory over Your directory (attention it will delete all the contents of destination directory):

rsync -a --delete /empty/ /todelete/

It can take a lot of time, and maybe You want to use ionice to be nice for other processes and services that are using disk during this operation. It is also good to set this command on screen.

After above command will end its job just do:

rmdir /todelete

Of course it can also be I/O consuming operation so You may also want to use ionice command.

The above way helped me to delete almost 100 millions of files in one directory where find, perl and other ways were not successful.

Posted in Uncategorized Tagged with: , , , ,

Zabbix – check if host is on dnsbl

Sometimes You need to monitor status of Your host in case it is blacklisted (especially mail servers).

I wrote simple script and template for Zabbix to do this.

Here it is:


This is early version, so feel free to send bugs reports and feature request here.

Posted in Uncategorized Tagged with: , , ,

Restoring corrupted InnoDB MySQL databases

Recently my Zabbix MySQL database was corrupted. Unfortunately I’ve needed historical data (database backup was too old), so there was only one way: restore everything I can from corrupted database. On the other hand I had every table in the separate file (innodb_file_per_table=1 in my.cnf), which was very helpful.

There are three ways to restore corrupted InnoDB databases (you should decide which one to choose, sometimes You will need to use not only one):

  • manually importing files to newly created database
  • using Percona InnoDB recovery tools
  • using innodb_force_recovery

For above methods You will need to have files from Your datadir (for example: /var/lib/mysql), so copy it somwhere.

Manually importing files

For this method You need to have ibd files from MySQL’s datadir and You need to know how was the table created (whole create command).

First step is to create new database, so login to MySQL and create it:

create database corrupted;

Now create table:

use corrupted;
CREATE TABLE `maintenances` (
	`maintenanceid`          bigint unsigned                           NOT NULL,
	`name`                   varchar(128)    DEFAULT ''                NOT NULL,
	`maintenance_type`       integer         DEFAULT '0'               NOT NULL,
	`description`            text                                      NOT NULL,
	`active_since`           integer         DEFAULT '0'               NOT NULL,
	`active_till`            integer         DEFAULT '0'               NOT NULL,
	PRIMARY KEY (maintenanceid)

And here is a tricky part – You need to discard tablespace by invoking this command in MySQL:

use corrupted;

Next step is to copy old file to correct place (using OS shell, not MySQL):

cp /var/lib/mysql-old/zabbix/maintenances.ibd /var/lib/mysql/corrupted/

After that You need to login to MySQL again and import new tablespace:

use corrupted;

In same cases after above steps You will be able to dump this table using mysqldump tool, but it is very often that MySQL will produce this error:

ERROR 1030 (HY000): Got error -1 from storage engine

After that simple go to MySQL log file and see why it is happening. In my case it was:

InnoDB: Error: tablespace id in file './zabbix/maintenances.ibd' is 263, but in the InnoDB data dictionary it is 5.

If the above error occurred You need to start from the beginning but with another method.

Percona InnoDB recovery tools

First You need  those tools – simply visit percona site and download it, unpack it and build those tools (You will find more info how to do this inside this archive). After that You are ready to repair above MySQL error. To do this follow next steps:

Drop table from corrupted database, and create it again (the same way as it was created before).

Stop MySQL daemon! – it is necessary.

Copy table file (overwrite it):

cp /var/lib/mysql-old/zabbix/maintenances.ibd /var/lib/mysql/corrupted/

Use ibdconnect:

./ibdconnect -o /var/lib/mysql/ibdata1 -f /var/lib/mysql/corrupted/maintenances.ibd -d zabbix -t maintenances

There will be some output and on the end there should be:

SYS_INDEXES is updated successfully

Now we can repair ibdata1 file:

./innochecksum -f /var/lib/mysql/ibdata1

Repeat this step until there will be no output.

Now You can start MySQL daemon again and You should be able to dump this table, if not follow instructions to see the last method.

Use innodb_force_recovery

In this method we will just copy table file and power up MySQL with innodb_force_recovery parameter. Here are the steps:

Change MySQL configuration. In [mysqld] section set datadir to Your copy of MySQL files, and set innodb_force_recovery parameter to 6:


Restart MySQL and You should be able to dump all corrupted tables by mysqldump.


Hope this post will help You. If You have some questions please leave comment below.

Posted in Uncategorized Tagged with: , , ,

Quick Tip: Centos 6.5 and zabbix agent 2.2.1

If You are facing with one of the following problems after upgrading CentOS and Zabbix agent to the latest stable realese (CentOS 6.5 and zabbix agent 2.2.1):

  • You cannot get autodiscovered items data (for examle: network interfaces bandwidth)
  • zabbix cannot collect data from MySQL (using provided by zabbix authors MYSQL template
  • Just try to:

  • disable SELinux
  • set SELinux to permissive mode (setenforce 0)
  • try to update SELinux policies
  • Posted in Uncategorized Tagged with:

    Using OMSA and snmp on CentOS to monitor Dell servers in zabbix.

    Recently I’ve needed to configure more advanced monitoring of Dell servers with hardware Raid, becasue of disk failure in one of my servers. I’ve came across a lots of information how to configure OMSA in Centos, but every description and howto shows that You need to install every component from OMSA to run it properly, so I’ve decided to write my own note how to do it without installing unnecessary software from OMSA repository. Here are the steps to install OMSA snmpd support:

    First of all install OMSA repository on Your CentOS system. To do this follow instructions on this site:

    Now You can install necessary software and run snmpd with omsa support:

    yum install srvadmin-megalib srvadmin-rac-components srvadmin-server-snmp srvadmin-deng-snmp srvadmin-base srvadmin-storelib-sysfs srvadmin-deng srvadmin-isvc srvadmin-smcommon srvadmin-sysfsutils srvadmin-storage-snmp srvadmin-omilcore srvadmin-hapi srvadmin-cm srvadmin-idrac-snmp srvadmin-storage srvadmin-storageservices-snmp srvadmin-omacs srvadmin-storelib srvadmin-ominst srvadmin-xmlsup srvadmin-isvc-snmp

    Of course You need to set up snmpd to run as You wish (You will find how to install and configure snmpd on google ;) ), and as dependency You need to configure snmpd to get values from OMSA – just edit this file:


    And add this line on the end:

    smuxpeer .

    You are almost ready to read OMSA variabled from snmp, just fire up all needed services:

    /etc/init.d/dataeng start
    /etc/init.d/snmpd restart

    And add those services to start automatically during boot:

    chkconfig dataeng on
    chkconfig snmpd on

    Now You can check if everything works as You wish (this command is for default configuration of snmpd on CentOS, and of course You need to have net-snmp-utils package installed):

    snmpwalk -v 2c -c public .

    If everything is working as we needed You should see a lot of lines with informations.

    Now it is time to configure zabbix to read this data. I’ve found nice zabbix template on zabbix forums. Just download it from here: and import in zabbix.

    Last step to get it working in zabbix is to Link this template to servers You need to be monitored.

    Hope this post will be helpful for somebody.

    Posted in Uncategorized Tagged with: , , ,

    Running raspbian in QEMU on Fedora 19

    As I mentioned earlier (Zabbix templates for Raspberry PI), I want to develop Zabbix template for Raspebrry PI, but unfortunatelly my Raspberry PI could no be used to do such things now, so I decided to run raspbian on QEMU.

    Here are the steps to run raspbian in QEMU on Fedora 19.

    First of all You need install some packages and its dependencies:

    yum install qemu-system-arm

    Now we are ready to download things that we will need to run raspbian in QEMU.

    # Linux Kernel for QEMU
    # Raspbian Wheezy

    Now we are redy to set up everything. It will take few simple steps:

    # unzip Raspbian Wheezy image
    # put everything in one place:
    mkdir rpi-qemu
    cp kernel-qemu rpi-qemu/
    cp 2013-07-26-wheezy-raspbian.img rpi-qemu/
    cd rpi-qemu
    # before mountin image You need to figure some thins:
    file 2013-07-26-wheezy-raspbian.img
    # now get "startsector" value for parition 2 and multiply by 512
    mount ~/qemu_vms/2013-02-09-wheezy-raspbian.img -o offset=<multiplied_value> /mnt
    # modify some files
    cd /mnt/etc/
    # edit this file:
    # comment out the line that is there by putting "#" before it
    # save this file
    # umount image:
    umount /mnt

    After above steps we are ready to fire Raspbian up.

    qemu-system-arm -kernel rpi-qemu/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot --append "root=/dev/sda2 panic=1" -hda rpi-qemu/2013-07-26-wheezy-raspbian.img -redir tcp:2222::22 -daemonize

    After issuing above command You should see smth like that:
    QEMU running raspbian
    After some time booting QEMU will give You root shell to fsck corrupted filesystem, just enter those command:

    # check filesystem
    fsck -y /dev/sda2
    # reboot system
    shutdown -r now

    After those steps You are ready to power it up again and start using it almost like normal Raspberry PI.

    Default username is pi and password is raspberry.

    Of course there are some points that must be mentioned:

      • ping will not work, but networking will
      • You cannot use commands from /opt/vc – it is just not working (why?? – because all of them are hardware related commands)
      • You can login to Your Raspbian via ssh from localhost:
    ssh pi@ -p 2222
    Posted in Uncategorized Tagged with: , ,

    Zabbix templates for Raspberry PI

    recently I was searching some Zabbix templates for Raspberry PI, but I didn’s succeed. I’ve decided I will do some Zabbix templates for Raspberry PI. Now I am preparing LAB for it. So stay tuned – I will post templates soon.

    Posted in Uncategorized Tagged with: , ,

    Compiling zabbix-agent on Raspberry Pi (xbian or debian)

    First of all You need to install some software for compilation:

    apt-get install build-essential

    Now, just get sources of zabbix (in this case the latest version is 2.0.6):

    wget ""

    Unpack it, go to unpacked sources:

    tar zxfv zabbix-2.0.6.tar.gz
    cd zabbix-2.0.6

    Now You need to specify configuration options, mine was like:

    ./configure --enable-agent --prefix=

    And You are ready to compile and to install zabbix-agent on Your system:

    make install

    Also if You need startup scripts follow next steps:

    cd misc/init.d/debian #in zabbix source's directory
    cp zabbix-agent /etc/init.d/
    which zabbix_agentd #outup will be needed to edit zabbix-agent startup script

    Open /etc/init.d/zabbix-agent file for edit end edit this line:

    DAEMON=/usr/local/sbin/${NAME} #put here output of previous command, i.e /sbin/${NAME}

    Last thing to do is to add zabbix user, because zabbix cannot run as root user:

    adduser --system zabbix

    Now You should be able to start zabbix-agent through init scripts, of course You need to configure zabbix-agent in /etc/zabbix_agent.conf and /etc/zabbix_agentd.conf

    After configuration is completed You should be able to start zabbix-agent:

    /etc/init.d/zabbix-agent start

    If You want to start zabbix-agent automaticaly during boot process You need to run this command:

    update-rc.d zabbix-agent defaults

    And that’s all.

    Posted in Uncategorized Tagged with: , , , ,

    Monitoring ip_conntrack on zabbix

    I needed some statistics and triggers on zabbix of ip_conntrack. Here is how to monitor it, of course You can expand it as You wish.

    First of all You need to add some configuration to Your zabbix agents. Put those lines at the end of zabbix config (in my case it was:  /etc/zabbix/zabbix_agentd.conf):

    UserParameter=ip_conntrack_count,cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
    UserParameter=ip_conntrack_max,cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max

    And then restart zabbix agent:

    /etc/init.d/zabbix-agent restart

    Now You can configure zabbix, so let’s login and import this file:

    Zabbix template

    From now on, You are able to monitor the state of ip_conntrack on Your system.

    Posted in Uncategorized Tagged with: , , ,

    Fedora 19, openvpn and default route

    According to one of my latest post (Fedora 19 and override routes with openvpn) I realized that Network Manager applet is not showing the status of VPN connection – icon will not change to this one: network-manager-icon-vpn.

    I decide that I need better workaround, so I’ve dig a little bit more and I found out that this issue can be resolved in simple way. Here is another workaround to this issue.

    All the connection that are setted up by Network Manager or nm-connection-editor are stored in:


    So to set up connection to disable default route through VPN connection, or use VPN connection to resources only on remote network,  simply edit file that is responsible for Your VPN connection. For example: if You’ve created connection VPN1 just edit file VPN1, and put there those lines (of course in appropriate section, in my case it was in [ipv4]):


    Save this file and now You can start Your VPN connection through Network Manager applet, also it will change Your icon according to the status of connection.

    Posted in Uncategorized Tagged with: , ,