As in most modern software development projects, the CI/CD tool is one of critical elements. It can serve multiple purposes, for example:

  • building software packages,
  • managing fleet of VMs or containers,
  • testing cases (stress, security, functionality).

Each of these tasks can be simplified and automated with the use of such a tool. Quite often such a tool is considered part of the management infrastructure – a part that is managing all environments and aspects of successful solution delivery.

Question: what happens if your tool breaks? If you have no backup / restore procedure, things can get ugly, fast!

In Kainos Evolve we are working on AWS Cloud Platform and decided to use Jenkins as our tool of choice. The story behind backing up our Jenkins configuration was simple – we wanted to enable restoring operations as smoothly as possible in case of major failure, be it Cloud provider or tool itself.

failure proofing jenkins
We’ve outlined three things to make Jenkins more failure proof:

  1.  Ensuring Jenkins is always available.
  2.  Backing up the Jenkins configuration.
  3.  Automating the Jenkins installation with plugins and restoration from backup.

 

Ensuring Jenkins is always available

To ensure Jenkins is always available, we’ve decided to host it as an EC2 instance under AWS Auto-Scaling Group. This required building Jenkins AMI defined under Launch Configuration. Since we’re using Terraform we’ve decided to follow solution suggested by Paul Hinze from HashiCorp: https://groups.google.com/d/msg/terraform-tool/7Gdhv1OAc80/iNQ93riiLwAJ

resource "aws_launch_configuration" "jenkins" {
 lifecycle { create_before_destroy = true }

 image_id             = "${var.ami}"
 instance_type        = "${var.instance_type}"
 security_groups      = ["${var.security_group}"]
 iam_instance_profile = "${var.jenkins-profile}"
}

resource "aws_autoscaling_group" "jenkins" {
 lifecycle { create_before_destroy = true }

 name                  = "jenkins - ${aws_launch_configuration.jenkins.name}"
 launch_configuration  = "${aws_launch_configuration.jenkins.name}"

 min_size              = 1
 max_size              = 1
 vpc_zone_identifier   = ["${split(",", var.subnet_ids)}"]
 load_balancers        = ["${aws_elb.jenkins.id}"]
 wait_for_elb_capacity = 1
}

The traffic can be routed via CNAME to AWS ELB.

 

Backing up the Jenkins configuration.

We initially started with thinBackup plugin for Jenkins but that was a preconfigured set of rules that didn’t fit our needs.

It was decided to make a back-up job with a schedule and manage who can run it on demand, exclusively when no other job was running. To do that:

To back-up everything we require, we wrote a custom shell script. Please note we are not backing up Jenkins plugins – this allows us to shorten the time required to make a backup.

failure proofing jenkins

If you want, you can include the plugins directory too, but consider the time required to pack up all plugins and corresponding size of the backup. Since we are working on AWS it was simple to have versioned backups by using AWS S3 with versioning enabled. You can see we use s3cmd to send the archive to an S3 bucket.

 

Automating the Jenkins installation with plugins and restoration from backup.

This makes it possible to install bare Jenkins in an automated way including plugins you use. I’ll not spend much time here as each one of you can use different configuration management tool so I’ll only provide a few tips:

  • [AWS] When creating your Jenkins AWS AMI, run the instance behind NAT with limited IAM role, preferably with read-only access to S3 backup bucket.
  • Install Jenkins using package from repository. You can find the repo for your OS on Jenkins website.
  • Initially Jenkins does not have any security enabled.
  • Use jenkins-cli.jar available with each Jenkins installation to install plugins from CLI. More details: https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI
  • Ensure Jenkins will start headless with disabled wizard. Edit /etc/sysconfig/jenkins:
# Options to pass to java when running Jenkins.
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"
  • After plugins are installed, stop Jenkins service and apply the backup.

 

Restoring Jenkins configuration from backup is simple. A compressed backup can be transferred to Jenkins and extracted inside the Jenkins home directory to restore whole configuration.

I highly recommend defining a default backup with no jobs defined and with basic security i.e. admin user with complex password known to Ops team. This is useful i.e. when starting fresh as your configuration management tool will restore from previously created backup or from default.

 

Summary

As always “your mileage may vary”. There are many other ways we could have done this e.g. Jenkins container with external shared storage, single job in Jenkinsfile to pull all the rest of configuration, VM snapshot. The solution outlined above provides flexibility on what and how you store it. It is also easy to extend it and adapt it to your needs.

Interesting in working for Kainos? Check out our WebOps positions at Kainos.pl/kariera