Currently in one of our projects we’re building platform in the cloud. The whole service is based mostly on our microservices, but we’re also using 3rd party components like Iguana from iNTERFACEWARE Inc. 

Platform should be highly available, which is a requirement for any serious and publicly accessible project. 

 

What’s the problem in here?

Iguana is mainly used for medical data integration. It grabs data from one source (database, text file, TCP connection), transforms it info other form (that’s optional) and puts the result destination (database, text file, etc.).

Our integration specialists told me that they are installing it in active-passive mode, and there is a degree of manual intervention involved – this suits their requirements.  I wanted to see what would be involved in automating the failover – especially given that our underlying architecture is different and therefore other failover options might be available.

Brainstorm!

So bunch of us sat there and we tried to figure out best solution. After watching this (https://interfaceware.wistia.com/medias/5v308sg9mu) webinar from INTERFACEWARE Inc. about HA Considerations, we thought it’s gonna be piece of cake to implement this in AWS.

The idea was to set up cluster in active-passive mode, have shared storage for configuration, logs and queues. AWS allows you to attach External Block Storages to your instances, but the limitation is that it can only be attached to one instance at a time. Next idea was to try out AWS service called Elastic File System – which is basically a NFS share. After quick cost calculations we thought it might be worth to try this out.

PoC, attempt #1

After I was done with Terraform code for PoC to build necessary resources in AWS, I started to play with Iguana itself to configure it to use shared storage. So I had an Elastic Load Balancer, two EC2 instances, NFS share mounted on both servers and two Iguana instances installed on both of them.

I have figured out that Iguana is using workspaces to store it’s config, licence and other data. I’ve decided to keep workspace separated from logs and other data. Also, to not loose licence file I’ve attached EBS to instances.

Iguana keeps its configuration in workspace/IguanaConfigurationRepo directory. If it doesn’t exist, it clones it from bare git repository (workspace/IguanaMainRepo) at startup. This approach actually made my life much easier! Now I was able to move bare git repository to shared storage and symlink it to old place. I had to keep in mind that it’s necessary to remove workspace/IguanaConfigurationRepo before Iguana starts to force it to “pull” most recent configuration from *the-only-valid-source-of-truth*. This guarantees that both instances will always have the same current configuration.

I also had to rethink how to store Iguana data and figured out to this directory structure would be the best:

Mount Point

Purpose

/opt/iguana home directory for iguana dedicated user
/opt/iguana/iguana-${version} application binaries
/opt/iguana/iguana-workspace workspace data (licence etc)
/opt/iguana/iguana-data shared storage for cluster, NFS mount
/opt/iguana/iguana-data/IguanaMainRepo Main configuration
/opt/iguana/iguana-data/logs Iguana processed data

 

I grabbed trial licence and set up first instance. Iguana was configured to store queue data in /opt/iguana/iguana-data and created first testing channel on both instances. Quickly I’ve found out, that Iguana is indexing data and stores metadata in sqlite format. Red light appeared in my head, so I’ve checked Iguana log files on second instance. There was plenty of SQLITE_IOERR messages in log files….

 

Google said:

SQLITE USES READER/WRITER LOCKS TO CONTROL ACCESS TO THE DATABASE. BUT USE CAUTION: THIS LOCKING MECHANISM MIGHT NOT WORK CORRECTLY IF THE DATABASE FILE IS KEPT ON AN NFS FILESYSTEM. THIS IS BECAUSE FCNTL() FILE LOCKING IS BROKEN ON MANY NFS IF MULTIPLE PROCESSES MIGHT TRY TO ACCESS THE FILE AT THE SAME TIME.

So stage #1 of PoC was instantly finished.

PoC, attempt #2

After grabbing fresh air with some Kainos people, someone mentioned that DRBD (http://drbd.org) might solve our problems. In short DRBD is a mechanism to replicate data blocks over TCP between two or more nodes. It is used by OpenStack, so that must be solid tested piece of code!

I had to redesign my PoC a little. We agreed to not set up autoscaling for Iguana and limit PoC cluster to two nodes only. Another try was made with 1 Elastic Load Balancer and 2 EC2 instances (each with Elastic Block Storage attached). On each EBS I’ve set up two logical volumes (LVM2) – one for workspace and another one for data, which should be replicated over these two EBS volumes.

After that I’ve configured DRBD to replicate data volume over two EC2 instances, created filesystems and mounted everything (note: DRBD is marking devices as primary or secondary, so data volume still can’t be mounted on both amis at the same time).

Directory structure for Iguana was changed to look like this:

Mount Point

Purpose

/opt/iguana home directory for iguana dedicated user
/opt/iguana/iguana-${version} application binaries
/opt/iguana/iguana-workspace workspace data (licence etc)

EBS mount – /dev/iguana/workspace

/opt/iguana/iguana-data shared storage for cluster

DRBD mount – /dev/iguana/data

/opt/iguana/iguana-data/IguanaMainRepo Main configuration
/opt/iguana/iguana-data/logs Iguana processed data

 

So I launched first Iguana instance, configured channel for testing purposes (listen on random TCP port, don’t filter, put output in /opt/iguana/iguana-data/logs/test_channel_output) and did simple load test with hl7simulator software (attached to Iguana). After this was finished, I’ve stopped first instance, mounted iguana-data device on second instance and launched fresh Iguana service there.

I was surprised this went so well! On dashboard I could see my test channel, which was configured on first Iguana instance, all data in /opt/iguana/iguana-data were replicated and – what’s most important – no SQLITE_IOERR messages in log files!

Okey, manual failover wasn’t really what we’re looking for…

PoC, attempt #3

After another two minutes with Google, I found out that DRBD management (and much more) can be easily done with Pacemaker/Corosync clustering software.

After installing Pacemaker/Corosync RPMs on both AMIs, I had no troubles with it’s configuration. After setting proper resources to manage, I had to keep in mind about colocation and ordering as well. Finally I was able to use Pacemaker to do things in order:

  • set primary role for DRBD device
  • mount it under proper mountpoint
  • start iguana service

Failover test can be also done manually without reboots:

pcs cluster stop iguana1.devenv01.local && \
  pcs cluster start iguana1.devenv01.local

 

I was able to successfully “move” Iguana from server A to server B on the fly, without any noticeable downtime. We have done some initial data load tests and never fallen into split brain (but recovery was tested after we forced this to happen :)). This is still PoC and there are lots of ideas how to improve this concept (like route DRBD traffic through dedicated vNIC etc).

I strongly suggest anyone, who’s having troubles setting an application in HA to try out Pacemaker/Corosync with DRBD (if shared storage is required and NFS isn’t an option). There’s also Windows driver for DRBD – but I’m far from any M$ products.

Configuration examples

DRDB

/etc/drbd.d/iguana_data.res
resource iguana_data {
  protocol C;
  device /dev/drbd0;
  disk /dev/iguana/iguana_data;
  meta-disk internal;
  on iguana1.devenv01.local {
    address 192.168.59.123:7788;
  }
  on iguana2.devenv01.local {
    address 192.168.59.124:7788;
  }
}

PCS cluster setup

pcs cluster auth iguana1.devenv01.local \
  iguana2.devenv01.local -u hacluster
pcs cluster setup --name iguana_cluster \
  iguana1.devenv01.local iguana2.devenv01.local
pcs cluster start --all
pcs cluster enable --all

Example PCS configuration

pcs cluster cib iguana_config
pcs -f iguana_config resource create \
  IguanaData ocf:linbit:drbd \
  drbd_resource=iguana_data \
  op monitor interval=10s
pcs -f iguana_config resource master \
  IguanaDataClone IguanaData \
  master-max=1 master-node-max=1 \
  clone-max=2 clone-node-max=1 \
  notify=true
pcs -f iguana_config resource create \
  IguanaDataFS Filesystem \
  device="/dev/drbd0" \
  directory="/opt/iguana/iguana-data" \
  fstype="xfs"
pcs -f iguana_config constraint colocation add \
  IguanaDataFS with IguanaDataClone \
  INFINITY with-rsc-role=Master
pcs -f iguana_config constraint order promote \
  IguanaDataClone then \
  start IguanaDataFS
pcs -f iguana_config resource create \
  iguana_service ocf:heartbeat:anything \
  binfile="./iguana_start.sh" \
  workdir="/opt/iguana/iguana-6_0_5" \
  user="iguana" \
  logfile="/tmp/iguana.log" \
  op monitor interval=10s
pcs -f iguana_config constraint order \
  IguanaDataFS then start iguana_service
pcs -f iguana_config constraint colocation add \
  IguanaDataFS with iguana_service
pcs cluster cib-push iguana_config
rm -f iguana_config