TL;DR - You can test if you want to, you can leave your bugs behind

So, you’re working on an application that wants to get included in the CODE-RADE repository, but for some reason, it’s failing in CI.Let’s take the R language for example. This project has a slew of dependencies, some of them optional, such as java , lapack , openblas , pcre2 , readline and xz

It is going to be needed soon for the CODATA/RDA DataScience School , but the project is currently So, you say “shucks, I can fix that !” and set about preparing your pull request to the repository

Before you do that though, you want to test your changes… but how to do that ?This post will shed some light on how to reproduce the testing environment on your laptop or local setup so that you can send pull requests or commits with greater confidence and speed up the CI/CD pipeline.

CODE-RADE targets

CODE-RADE builds are done in a matrix of environments , which define targets . These are constructed using 3 axes :

• SITE : a variables describing the site for which the application has been compiled, in some qualitative way. Usually set to generic
• ARCH : the CPU architecture of the target. Typically x86_64
• OS : the OS of the target. We test for :
  • Ubuntu-14.04
  • Ubuntu-16.10
  • CEntOS-6
  • CEntOS-7

Every application is built for a combination of the size of these dimensions - typically $1\times 1\times 4 = 4$ .If different toolchains are used, these add to the dimensions - e.g. different compiler versions, MPI implementations, or pythons - the size of the matrix can grow quite large ¹ .In fact, one of the main benefits of using Jenkins in CODE-RADE is to be able to automate the tests across a wide variety of targets, allowing the developer to focus on writing code instead of manually running checks.

Sometimes, however, you still need to run a few things manually locally, in order to check whether a proposed fix will actually do the trick.Perhaps you don’t have to run the checks for all targets , but just one or two which are representative, or just those nodes which are failing.In order to do this, however, you need to have access to the same environment as is provided on the CI service.

CODE-RADE containers

Luckily, it is now quite easy to deliver execution environments, by building Docker containers .The CODE-RADE containers are built by AnsibleContainer for the specific targets ² and pushed to quay.io ³ :

OS	Container
CentOS 6
CentOS 7
Ubuntu 14.04
Ubuntu 16.10

So, you can run these containers on your local setup and reproduce the CI environment !

Accessing Dependencies

Of course, the containers do not contain the build products they only reproduce the environment - so you need to find some way to include the dependencies in the container when you run the tests.

Luckily for you, you can just mount the CVMFS repository from within the container to have access to it.If it is already mounted on your local setup (of course it is ! why on earth wouldn’t it be !?), you can expose the mountpoint as a volume to the container :

                  docker run -ti -v /cvmfs:/cvmfs quay.io/aaroc/code-rade-build-containers-build-slave-ubuntu1404 /bin/bash
                

Of course, once you have access to the dependencies, you need to add the modules in order to have them set up in the shell :

                  
                    root@2a7757ac74d3:~# 
                    . /etc/profile.d/modules.sh
                    root@2a7757ac74d3:~# 
                    export 
                    SITE
                    =
                    generic 
                    OS
                    =
                    u1404 
                    ARCH
                    =
                    x86_64
                    root@2a7757ac74d3:~# 
                    module use  /cvmfs/code-rade.africa-grid.org/modules/libraries/
                    root@2a7757ac74d3:~# 
                    module use  /cvmfs/code-rade.africa-grid.org/modules/compilers
                    root@2a7757ac74d3:~# 
                    module avail gcc-------------------------------------------------------------- /cvmfs/code-rade.africa-grid.org/modules/compilers --------------------------------------------------------------gcc/4.9.2 gcc/4.9.4 gcc/5.4.0 gcc/6.3.0
                  
                

Boom, now you have a choice of compilers.

Back to R

As of at least build 21 R is failing. Let’s suppose we want to have a stab at fixing it.First, we get the repository locally :

                  git clone https://github.com/SouthAfricaDigitalScience/rlang-deploy.git
                

The scripts in the repository explain where to get the source code and how to configure the build.In the case of R, we have the following dependencies :

                  module add  gcc/5.4.0module add openblas/0.2.15-gcc-5.4.0module add lapack/3.6.0-gcc-5.4.0module add jdk/8u66module add ncursesmodule add readlinemodule add bzip2module  add xz
                

and the build is configured as :

                  
                    CFLAGS
                    =
                    "
                    ${
                    CFLAGS
                    }
                     -I
                    ${
                    BZLIB_DIR
                    }
                    /include -I
                    ${
                    XZ_DIR
                    }
                    /include"
                    \
                    LDFLAGS
                    =
                    "-L
                    ${
                    BZLIB_DIR
                    }
                    /lib -L
                    ${
                    XZ_DIR
                    }
                    /lib -llzma"
                    \
                    ../configure 
                    \
                    --build
                    =
                    x86_64-pc-linux-gnu 
                    \
                    --host
                    =
                    x86_64-pc-linux-gnu 
                    \
                    --target
                    =
                    x86_64-pc-linux-gnu 
                    \
                    --prefix
                    =
                    ${
                    SOFT_DIR
                    }
                    \
                    --enable-static 
                    \
                    --enable-shared 
                    \
                    --with-readline
                    =
                    no 
                    \
                    --with-x
                    =
                    no 
                    \
                    --with-blas 
                    \
                    --with-lapack 
                    \
                    --without-recommended-packages
                  
                

Taking a look at the issue or the console output , we can see that the problem is linking against PCRE ⁴ :

                  
                    checking whether PCRE support suffices... configure: error: pcre >
                    =
                     8.10 library and headers are required
                  
                

Seems pretty easy to fix - just include -L${PCRE_DIR}/lib in the LDFLAGS and -I${PCRE_DIR}/include in the CFLAGS .

You can try that first in the container.

Speeding up manual builds

Finally, we would like to be able to run these things with a single command.Currently, a few manual commands need to be done interactively in order to set up the modules, etc.This could be improved by having a build-local.sh script in the repo which when run just passed the local CVMFS mount as a volume to the container, as well as the local working directory where the -deploy repository was cloned.In this way, one could just run build-local.sh without having to do things manually.Similarly, one could pass environment variables and scripts to the docker run command in order to set up the environment properly…We’ll comment in this at a later stage maybe.

For the meantime, creating an environment in which to interactively test the build could be done with :

                  docker run -ti -v /cvmfs:/cvmfs -v pcre-deploy:/workspace quay.io/aaroc/code-rade-build-containers-build-slave-ubuntu1404 /bin/bash build-local.sh
                

Debugging just got a whole lot more responsive and more keyboard-slammier

Footnotes