Choosing which ansible role to test with molecule using git commit description

My previous entry described how to test ansible roles with molecule and gitlab-ci however most of the times you would probably like to test just a role that you’ve edited. To help you with that I’ve edited my repo on github and I will show you how to make it working.

I’ve edited my .gitlab-ci config file in the way that it will test one ansible role with molecule if the commit description would contain name of the role to test, without commit description it will test all the molecule enabled roles and when there is no role named as a commit description it will fail.

The most important change in the file is shown below:

script:
- |
cd playbooks/roles/
if [[ ! -z "$CI_COMMIT_DESCRIPTION" ]]; then
cd $CI_COMMIT_DESCRIPTION/ && molecule test
else
for role in find . -type d -name "molecule" | cut -d'/' -f 2; do
cd $role && molecule test
cd ..
done
fi

As you might see I’ve just replaced basic script that was used to test just one role with this more complex one that will do the job.

How it works? Basically it will take variable passed to gitlab-runner and will make use of it.

How to use it? It is simple as just commiting to your repo, however when you would like to test just one particular role you have to add new line at the end of the commit message and in the new line add the name of the role, like in this example (just a test commit message and description):

This is just a test commit
basic

In the above example text in the first line is the commit message (This is just a test commit) and the second line is the commit description that tells gitlab to test only basic role.

Another example could be found here. So Testing new pipieline config with commit message is the commit message and basic is the commit description that tells gitlab to test just a basic role.

Hopefully it is somehow useful and someone could make use of this trick. Of course feel free to clone and play with repo and molecule.

Tagged with: , , , ,

Automated testing ansible playbooks using gitlab-ci and molecule

When you are developing ansible playbooks you will come to the point when you have to (or want to) automate testing of the playbooks/roles before getting them to the production. Or even you would like to setup CI/CD system that plays your playbook against your production environment automatically, but of course previously tested them. Just follow this blog entry and I will try to show you in couple of simple steps how to fully automate testing ansible playbooks using gitlab-ci pipeline and molecule.

Contents list:
What is molecule

From molecule’s webpage:

Molecule is designed to aid in the development and testing of Ansible roles.

Molecule provides support for testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks and testing scenarios.

Molecule encourages an approach that results in consistently developed roles that are well-written, easily understood and maintained.

Why to use molecule instead of just ansible-linter, ansible syntax checking and yamllinter? Because with molecule you can spin up a docker image with a given operating system and test playbooks against it. It will also do linting, syntax checking but it will also take care of checking if the playbooks/roles are idempotent. What is more you can write your own scripts so that they will be executed to test if the playbooks were run properly and if the system is configured in the way you wanted to. In my opinion just a basic tests that are run with molecule are enough for the beginning. I will try to include more complex tests in the next articles. So please stay tuned.

Installing gitlab-ci runner and docker components

In order to test ansible playbooks you will have to install gitlab-ci runner and docker components on the hosts that will be running all the tests. The simplest way to install and register gitlab runner is to follow the instructions on the gitlab’s webpage. Please be sure to use shell executor as other types of gitlab runners are not suitable to run docker containers (I know there is docker-machine and docker executor, but we will not be able to run docker images inside docker environment, so we have to run docker images on top of the operating system).

The next stop is to install docker components on your freshly installed and registered gitlab runner. To install docker please just visit docker’s webpage and follow instructions.

I am using CentOS 7 host as a gitlab runner and after installing above software you will not be able to run gitlab runner with molecule out of the box. To be able to tests you have to fix/install some dependencies.

First of all python-six package is quite out of date in the CentOS 7 installation so you have to update it manually:

pip install six --upgrade

The second thing is that you have to install docker python bindings so that molecule can spin up new docker containers:

pip install docker-py

After that you have fully working gitlab runner that is able to run molecule tests. Of course you can configure above things in the VM, which would be nicer than messing with a system’s python libriaries, or even use virtualenv to run tests (idea for next blog post?:) ).

Adding molecule tests to your roles

Hopefully you’ve followed official way to install molecule on the host (pip or in virtualenv), otherwise you will end up with old molecule installation (version 1.x) which is not compatible with newest version. All the setup and configurations steps that are presented here are done on the newest available version (at the time being 2.19.0).

To start testing we have to either init new role with molecule witihin your ansible setup or add molecule config to existing role.

Init new molecule role

If you are creating new role from scratch you can use:

molecule init --role my-new-role --driver docker

Where my-new-role is the name of the new role and docker is a driver that we will be using. It will also create directories for tasks/vars/handlers and it will set up basic config file for the molecule testing.

Add molecule config to existing role

If you already have a role and just want to add molecule testing for it please use the following command:

molecule init scenario --role my-role-name --driver docker

It will set up molecule testing files for your my-role-name with a docker driver (as we are going to use docker containers to test our playbooks/roles).

Configuring molecule

After we’ve added basic config files for the roles in the above section, we have to configure molecule. Whole configuration is of molecule tests are done in the molecule.yml file (for example in ansible/playbooks/roles/my-role-name/molecule/molecule.yml). Here is an example of configuration file (not original one, it’s modified by me):

---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: centos7
image: centos/systemd
privileged: true
command: /sbin/init
security_opts:
- seccomp=unconfigured
capabilities:
- SYS_ADMIN
volume_mounts:
- "/sys/fs/cgroup:/sys/fs/cgroup:ro"
provisioner:
name: ansible
lint:
name: ansible-lint
scenario:
name: default

The most important for sections are:

driver: as you can see the driver is set up to use docker for testing playbooks on

platorms: in this section we can set up as many platforms as we wish molecule to test on. 

In the above example I am setting up systemd enabled CentOS 7 docker image (that’s why there are a lot options under centos7 section) to test my roles against. Of course you can add another platforms like: CentOS6, Ubuntu (any version) or Debian. It all depends on your roles and against what would like to test them.

Basically with this setup we can run molecule manually to test our role:

cd ansible/playbooks/roles/my-role-name
molecule test

After invoking above command, molecule will do:

  1. linting all the *.yml files found in the given location/role using following techniques:
    1. yamllint 
    2. flake8
    3. ansible-lint
  2. try to destroy all orphaned docker images that could potentially interrupt with tests
  3. try to cope with all the dependencies using ansible-galaxy
  4. check syntax
  5. prepare docker images
  6. converge docker images using given roles
  7. test idempotence of the given role
  8. execute verification scripts from molecule/default/tests/
  9. destroy all created docker images

If all tests passed without any issues then your role is written correctly and can be considered as production ready. 

Configuring gitlab runner (.gitlab-ci.yml file)

To execute molecule tests automatically when new changes are pushed to the git repo on our gitlab server we have to configure gitlab runner. It is as simple as just putting one file in the main directory of your repository. The file should be named .gitlab-ci.yml.

Here is an example of the gitlab runner configuration file:

---
variables:
GIT_DEPTH: "1"
PIP_CACHE_DIR: "$CI_PROJECT_DIR/pip-cache"

cache:
paths:
- "$CI_PROJECT_DIR/pip-cache"
key: "$CI_PROJECT_ID"

stages:
- molecule

molecule:
stage: molecule
tags:
- shell
script:
- cd playbooks/roles/basic/ && molecule test

Most important setting/sections are:

stages: we are saying what stages we are going to have in our pipeline. In this case just testing using molecule, but we could add more stages to it (every stage is run after preceding one has been succesful, so stages can be used to play role against production servers)

molecule: this describes and configures the molecule stage. The most important subsection is script, the rest could be ommited. Script is just a set of commands used to run tests in this stage.

All other setting are not so important for our purpose and you can read about them here

Summary

After following all the configuration steps your gitlab pipeline should be configured properly so after pushing new changes to git repo you should see green tick in the web gui near your latest commit, or red cross if the tests didn’t pass.

passed

I’ve prepared small repository on github showing all the functionality described here. The repo can be found here

Please feel free to clone/comment/play with it. 

P.S. Hopefully this entry would be helpful. I have some ideas for the next blog entries so stay tuned. VY 73!

Tagged with: , , , ,

Securing connection to check_mk agent with stunnel

Check_mk is quite nice to monitor hosts in your own network, however if you have remote server that you would like to monitor it’s not so secure, because check_mk agent is sending all its data as clear text. Of course you can limit connection to only one remote ip with firewall, or even with xinetd, but what about monitoring hosts running on dynamic external IP or even scuring data transfer between hosts. You simply cannot put DNS name to firewall or xinetd, that’s why you can use stunnel to secure connection.

It will act in two ways:

  • securing data transfer through the internet
  • adding authentication layer in front of check_mk agent

Please follow this how-to, it will show you how to secure connection between check_mk server running CentOS 7 and check_mk agent running on CentOS 7. This how to is not describing the way to install cehck_mk nor check_mk agent on the hosts.

Both sites (remote site and check_mk site)

First of all install install stunnel form CentOS 7 base repo:

yum install stunnel

After installation was completed you have to create systemd unit file for this service in /etc/systemd/system/stunnel.service:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target network.target
 
[Service]
ExecStart=/usr/bin/stunnel
Type=forking
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target

Now you are ready to configure stunnel.

Server site (where check_mk server lives)

Now you are ready to configure stunnel (/etc/stunnel/stunnel.conf):

client = yes
[check_mk_remote]
cert = /etc/pki/tls/certs/[cert_name].pem
accept = 127.0.0.1:6557
connect = [remote_ip]:6556

The cert file that is mentioned in config file will be generated on the remote site, so just copy the cert after you’ve generated it and then you can enable and start service.

systemctl enable stunnel.service
systemctl start stunnel.service

After that we can set up remote site.

Remote site (host which you would like to monitor)

This is how the configuration of stunnel should look like:

cert = /etc/pki/tls/certs/stunnel.pem
sslVersion = TLSv1
setuid = nobody
setgid = nobody
pid = /tmp/stunnel.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
output = /var/log/stunnel.log
 
[check_mk_agent]
accept = [external_ip_of_remote_location]:6556
connect = localhost:6556 TIMEOUTclose = 0

As you can see above there is a cert that is used in this config. We should generate it:

openssl req -new -x509 -days 3650 -nodes -out /etc/pki/tls/certs/stunnel.pem -keyout /etc/pki/tls/certs/stunnel.pem
dd if=/dev/urandom count=2 | openssl dhparam -rand - 512 >> /etc/pki/tls/certs/stunnel.pem

During the generation process you will have to answer few questions. Second command will generate DH parameters and will append them to the cert file.

Of course there is one more thing to do: enable logging.

touch /var/log/stunnel.log
chown nobody:nobody /var/log/stunnel.log

Before starting services we should edit check_mk.socket systemd unit:

(add 127.0.0.1 to ListenStream in /etc/systemd/system/check_mk.socket)

# systemd socket definition file
[Unit]
Description=Check_MK Agent Socket
 
[Socket]
ListenStream=127.0.0.1:6556
Accept=true
 
[Install]
WantedBy=sockets.target

Now we are ready to enable and start the services:

systemctl enable check_mk.socket
systemctl start check_mk.socket
systemctl enable stunnel.service
systemctl start stunnel.service

Add host in check_mk

The last step is to add the host in the check_mk.

Basically you should add host as usual just configuring some additional parameters:

IPv4 Address should be changed to the localhost and you should create the rule for tcp port of agent for this host:

Adding another hosts this way

You can add as many hosts as you wish, all you need to do is just multiply section [check_mk_remote] on monitoring hosts (of course changing name and port for each section) and add more rules in check_mk.

You can even add another services this way. Just modify stunnel.conf file accordingly.

Have fun!

Tagged with: , , , , ,

Deleting tons of files in Linux using rsync

If You have a problem with deleting tons of files in linux (when for example You’ve came across this problem: /bin/rm: Argument list too long.) You can try find, perl scripts and so on (you can find some more info here), but if described there ways to do this are not successful You can try my way to cope with this problem using rsync.

First of all You need to tune Your system a little bit (put those commands as root in console):

sysctl -w vm.dirty_ratio=10
sysctl -w vm.dirty_background_ratio=5

(why we are doing this)

Now we are ready to delete those files:

Create empty directory:

mkdir /empty

Now use rsync to copy empty directory over Your directory (attention it will delete all the contents of destination directory):

rsync -a --delete /empty/ /todelete/

It can take a lot of time, and maybe You want to use ionice to be nice for other processes and services that are using disk during this operation. It is also good to set this command on screen.

After above command will end its job just do:

rmdir /todelete

Of course it can also be I/O consuming operation so You may also want to use ionice command.

The above way helped me to delete almost 100 millions of files in one directory where find, perl and other ways were not successful.

Tagged with: , , , ,

Zabbix – check if host is on dnsbl

Sometimes You need to monitor status of Your host in case it is blacklisted (especially mail servers).

I wrote simple script and template for Zabbix to do this.

Here it is:

zabbix_dnsbl

This is early version, so feel free to send bugs reports and feature request here.

Tagged with: , , ,

Restoring corrupted InnoDB MySQL databases

Recently my Zabbix MySQL database was corrupted. Unfortunately I’ve needed historical data (database backup was too old), so there was only one way: restore everything I can from corrupted database. On the other hand I had every table in the separate file (innodb_file_per_table=1 in my.cnf), which was very helpful.

There are three ways to restore corrupted InnoDB databases (you should decide which one to choose, sometimes You will need to use not only one):

  • manually importing files to newly created database
  • using Percona InnoDB recovery tools
  • using innodb_force_recovery

For above methods You will need to have files from Your datadir (for example: /var/lib/mysql), so copy it somwhere.

Manually importing files

For this method You need to have ibd files from MySQL’s datadir and You need to know how was the table created (whole create command).

First step is to create new database, so login to MySQL and create it:

create database corrupted;

Now create table:

use corrupted;
CREATE TABLE `maintenances` (
	`maintenanceid`          bigint unsigned                           NOT NULL,
	`name`                   varchar(128)    DEFAULT ''                NOT NULL,
	`maintenance_type`       integer         DEFAULT '0'               NOT NULL,
	`description`            text                                      NOT NULL,
	`active_since`           integer         DEFAULT '0'               NOT NULL,
	`active_till`            integer         DEFAULT '0'               NOT NULL,
	PRIMARY KEY (maintenanceid)
) ENGINE=InnoDB;

And here is a tricky part – You need to discard tablespace by invoking this command in MySQL:

use corrupted;
ALTER TABLE maintenances DISCARD TABLESPACE;

Next step is to copy old file to correct place (using OS shell, not MySQL):

cp /var/lib/mysql-old/zabbix/maintenances.ibd /var/lib/mysql/corrupted/

After that You need to login to MySQL again and import new tablespace:

use corrupted;
ALTER TABLE maintenances IMPORT TABLESPACE;

In same cases after above steps You will be able to dump this table using mysqldump tool, but it is very often that MySQL will produce this error:

ERROR 1030 (HY000): Got error -1 from storage engine

After that simple go to MySQL log file and see why it is happening. In my case it was:

InnoDB: Error: tablespace id in file './zabbix/maintenances.ibd' is 263, but in the InnoDB data dictionary it is 5.

If the above error occurred You need to start from the beginning but with another method.

Percona InnoDB recovery tools

First You need  those tools – simply visit percona site and download it, unpack it and build those tools (You will find more info how to do this inside this archive). After that You are ready to repair above MySQL error. To do this follow next steps:

Drop table from corrupted database, and create it again (the same way as it was created before).

Stop MySQL daemon! – it is necessary.

Copy table file (overwrite it):

cp /var/lib/mysql-old/zabbix/maintenances.ibd /var/lib/mysql/corrupted/

Use ibdconnect:

./ibdconnect -o /var/lib/mysql/ibdata1 -f /var/lib/mysql/corrupted/maintenances.ibd -d zabbix -t maintenances

There will be some output and on the end there should be:

SYS_INDEXES is updated successfully

Now we can repair ibdata1 file:

./innochecksum -f /var/lib/mysql/ibdata1

Repeat this step until there will be no output.

Now You can start MySQL daemon again and You should be able to dump this table, if not follow instructions to see the last method.

Use innodb_force_recovery

In this method we will just copy table file and power up MySQL with innodb_force_recovery parameter. Here are the steps:

Change MySQL configuration. In [mysqld] section set datadir to Your copy of MySQL files, and set innodb_force_recovery parameter to 6:

datadir=/var/lib/mysql
 
innodb_force_recovery=6

Restart MySQL and You should be able to dump all corrupted tables by mysqldump.

 

Hope this post will help You. If You have some questions please leave comment below.

Tagged with: , , ,

Quick Tip: Centos 6.5 and zabbix agent 2.2.1

If You are facing with one of the following problems after upgrading CentOS and Zabbix agent to the latest stable realese (CentOS 6.5 and zabbix agent 2.2.1):

  • You cannot get autodiscovered items data (for examle: network interfaces bandwidth)
  • zabbix cannot collect data from MySQL (using provided by zabbix authors MYSQL template
  • Just try to:

  • disable SELinux
  • set SELinux to permissive mode (setenforce 0)
  • try to update SELinux policies
  • Tagged with:

    Using OMSA and snmp on CentOS to monitor Dell servers in zabbix.

    Recently I’ve needed to configure more advanced monitoring of Dell servers with hardware Raid, becasue of disk failure in one of my servers. I’ve came across a lots of information how to configure OMSA in Centos, but every description and howto shows that You need to install every component from OMSA to run it properly, so I’ve decided to write my own note how to do it without installing unnecessary software from OMSA repository. Here are the steps to install OMSA snmpd support:

    First of all install OMSA repository on Your CentOS system. To do this follow instructions on this site:

    http://linux.dell.com/wiki/index.php/Repository/OMSA#Yum_setup

    Now You can install necessary software and run snmpd with omsa support:

    yum install srvadmin-megalib srvadmin-rac-components srvadmin-server-snmp srvadmin-deng-snmp srvadmin-base srvadmin-storelib-sysfs srvadmin-deng srvadmin-isvc srvadmin-smcommon srvadmin-sysfsutils srvadmin-storage-snmp srvadmin-omilcore srvadmin-hapi srvadmin-cm srvadmin-idrac-snmp srvadmin-storage srvadmin-storageservices-snmp srvadmin-omacs srvadmin-storelib srvadmin-ominst srvadmin-xmlsup srvadmin-isvc-snmp

    Of course You need to set up snmpd to run as You wish (You will find how to install and configure snmpd on google ;) ), and as dependency You need to configure snmpd to get values from OMSA – just edit this file:

    /etc/snmp/snmpd.conf

    And add this line on the end:

    smuxpeer .1.3.6.1.4.1.674.10892.1

    You are almost ready to read OMSA variabled from snmp, just fire up all needed services:

    /etc/init.d/dataeng start
    /etc/init.d/snmpd restart

    And add those services to start automatically during boot:

    chkconfig dataeng on
    chkconfig snmpd on

    Now You can check if everything works as You wish (this command is for default configuration of snmpd on CentOS, and of course You need to have net-snmp-utils package installed):

    snmpwalk -v 2c -c public 127.0.0.1 .1.3.6.1.4.1.674.10892.1

    If everything is working as we needed You should see a lot of lines with informations.

    Now it is time to configure zabbix to read this data. I’ve found nice zabbix template on zabbix forums. Just download it from here: https://www.zabbix.com/forum/showthread.php?t=22054 and import in zabbix.

    Last step to get it working in zabbix is to Link this template to servers You need to be monitored.

    Hope this post will be helpful for somebody.

    Tagged with: , , ,

    Running raspbian in QEMU on Fedora 19

    As I mentioned earlier (Zabbix templates for Raspberry PI), I want to develop Zabbix template for Raspebrry PI, but unfortunatelly my Raspberry PI could no be used to do such things now, so I decided to run raspbian on QEMU.

    Here are the steps to run raspbian in QEMU on Fedora 19.

    First of all You need install some packages and its dependencies:

    yum install qemu-system-arm

    Now we are ready to download things that we will need to run raspbian in QEMU.

    # Linux Kernel for QEMU
    wget http://xecdesign.com/downloads/linux-qemu/kernel-qemu
    # Raspbian Wheezy
    wget http://raspberry.mythic-beasts.com/raspberry/images/raspbian/2013-07-26-wheezy-raspbian/2013-07-26-wheezy-raspbian.zip

    Now we are redy to set up everything. It will take few simple steps:

    # unzip Raspbian Wheezy image
    unzip 2013-07-26-wheezy-raspbian.zip
    # put everything in one place:
    mkdir rpi-qemu
    cp kernel-qemu rpi-qemu/
    cp 2013-07-26-wheezy-raspbian.img rpi-qemu/
    cd rpi-qemu
    # before mountin image You need to figure some thins:
    file 2013-07-26-wheezy-raspbian.img
    # now get "startsector" value for parition 2 and multiply by 512
    mount ~/qemu_vms/2013-02-09-wheezy-raspbian.img -o offset=<multiplied_value> /mnt
    # modify some files
    cd /mnt/etc/
    # edit this file: ld.so.preload
    nano ld.so.preload
    # comment out the line that is there by putting "#" before it
    # save this file
    # umount image:
    umount /mnt

    After above steps we are ready to fire Raspbian up.

    qemu-system-arm -kernel rpi-qemu/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot --append "root=/dev/sda2 panic=1" -hda rpi-qemu/2013-07-26-wheezy-raspbian.img -redir tcp:2222::22 -daemonize

    After issuing above command You should see smth like that:
    QEMU running raspbian
    After some time booting QEMU will give You root shell to fsck corrupted filesystem, just enter those command:

    # check filesystem
    fsck -y /dev/sda2
    # reboot system
    shutdown -r now

    After those steps You are ready to power it up again and start using it almost like normal Raspberry PI.

    Default username is pi and password is raspberry.

    Of course there are some points that must be mentioned:

      • ping will not work, but networking will
      • You cannot use commands from /opt/vc – it is just not working (why?? – because all of them are hardware related commands)
      • You can login to Your Raspbian via ssh from localhost:
    ssh pi@127.0.0.1 -p 2222
    Tagged with: , ,

    Zabbix templates for Raspberry PI

    Hello,
    recently I was searching some Zabbix templates for Raspberry PI, but I didn’s succeed. I’ve decided I will do some Zabbix templates for Raspberry PI. Now I am preparing LAB for it. So stay tuned – I will post templates soon.

    Tagged with: , ,
    Top