Burn down

TL;DR: The CODE-RADE platform is now mostly configured, and minor problems overcome. Continuous integration is being tested with weekly burn downs, to see whether the full chain is working. Spoiler alert: it’s not . Also, you can help !

Here’s the thing :

Stuff breaks. The whole point of this project is to develop a system which will reliably, reproducibly, rapidly rebuild the repository of applications used at various sites. We therefore conduct these burn downs to see if this is actually the case (spoiler: it’s not yet), and highlight the issues which remain in the ashes.

The CODE-RADE platform is how we think we can reliably deliver applications to both sites and individual users and do it right . Right means :

• Continuous - New requests are automatically integrated using standard build and integration procedures. Most tasks are automated and little intervention is needed from site administrators.
• User-driven - Want an app that’s not in our repository ? Just send us a pull request.
• Site-Independent - We build the apps for execution
• Reproducible - When we integrate something, we want it to stay integrated. The last budgie standing after the nuclear holocaust should be able to rebuild everything just by gently nudging their cute little beak against the still-radioactive screen showing the CI build server web interface.

Integration flow

Integration is not a linear path. Of course, it can’t be cyclical , but to satisfy various end-user and site configuration desires, there has to be a level of paralellism involved. We do this with the Jenkins MatrixJob Plugin . This allows us to define axes for our job configurations, for example the target operating system or application version number.

There are also various phases to the integration flow, from initial request to final delivered product. It would be really nice to have a visual representation of the state of your application - to be able to answer, e.g. the question :

“Where is my application in the delivery chain ?”

Another issue which we’d like to address how to raise, escalate and resolve issues that may arise during the integration workflow. Now, we’ve taken the route that each application which we want built and integrated should have its own repository - we’ve now got around 30 repos in South Africa Digital Science . How do we keep track of what state the various applications are, cross-reference them with their build results and understand where the bottlenecks are ?

It just so turns out that this problem is not a new one, basically, we need a “scrum” or “kanban” board ¹ .To this end, we’ve created a board to contain all of the tickets opened in the various repos against the specific applications. The single board allows us to place to see the overall status of the CODE-RADE project. Unfortunately, ticket dependencies are managed “by hand” - there’s no way (yet) to specify which cards are blocked by which other ones, but we can take care of that just by manually ordering the cards on the board. Anyone who owns the repos (ie, in the right team of the owner organisations on Github) can interact with the board. Updating tickets information just involves a few clicks, and shifting issues along the pipeline is a drag-and-drop procedure.

Dependency Graph

One of the greatest (and thus hardest) things about CODE-RADE is the expression of dependencies between applications. Most open source software is not a monolithic structure, but rather delicate - some might say graceful - assembly of various independent components. These components may be in turn variously configured to produce different functional states, each of which can be considered nodes in a DAG ² . In fact, one of the main corrolaries of CODE-RADE, if we do things right, is to express a more-or-less fully-populated graph of the dependencies of all applications used in scientific research ³ (or at least the most popular ones).

This is for what we’re calling “Foundation Release 1” - IE , “ stuff you can use to build other stuff ”.

Build Failure Analysis

We’re using the Jenknis Build Failure Analysis plugin to identify various build failures. This is a fantastic plugin which scans the build logs and tries to figure out (via regexp) what went wrong. The build failure analysis helps us to estimate how much time fixing a build failure would take and spotting error patterns that are affecting more than one job.

Of course, failure types have to be spotted and identified by actual humans - so, if you’ve got some time on your hands and feel like reading build logs, why not help us isolate these common errors and implement fixes ?

Bottlenecks

So, with the dependency graph expressing the route from one component to the next, and the waffle board detailing where the actual issues are, we can plan work to address this and ensure that the pipeline burns right down. Here’s a summary of the current state of affairs:

• Total jobs in Foundation Release: 20
• Failing jobs: 5

So, we’re roughly 75 % of the way there (at least, using one definitely inappropriate metric ! ⁴ ).

Disk space - workaround

One problem was the issue of disk space. The jobs were being built entirely on the slaves - which meant that the build artifacts including the intermediate ones were all kept on thes fairly small machines (~ 100 GB). The CI server however has a nice big (~1TB) disk which we are now exporting portions of over NFS to the build slaves. Thus, the jobs are all reading from and writing to a single disk.

While this solves the size available problem, it does bring up some severe performance issues, especially in the case of wide matrix jobs such as gcc , which has 6 concurrent builds for the different axes. This can probably be alleviated by ensuring that the matrix builds are sequential instead of parallel as they are at the moment.

Numpy errors

The NumPy build scripts in their own repo - currently, we’re getting the following error when building NumPy

                  
                    'import site'
                    failed
                    ;
                    use
                    -
                    v
                    for
                    traceback
                    Traceback
                    (
                    most
                    recent
                    call
                    last
                    ):
                    File
                    "setup.py"
                    ,
                    line
                    16
                    ,
                    in
                    <
                    module
                    >
                    from
                    __future__
                    import
                    division
                    ,
                    print_function
                    ImportError
                    :
                    No
                    module
                    named
                    __future__
                    Build
                    step
                    'Execute shell'
                    marked
                    build
                    as
                    failure
                  
                

We have no idea what this means, so if you’re a python speaker, please take a look at the build.sh script and maybe send us a PR that fixes it.

OpenMPI

OpenMPI is failing only on the SL6 slaves. From the looks of things, the builds are passing, but functional tests are not. These are giving modulefile errors such as

                  
                    + ./check-build.shAbout to make the modules./check-build.sh: line 3: 
                    cd
                    : /var/lib/jenkins/workspace/OpenMPI/ARCH/x86_64/GCC_VERSION/5.2.0/OS/sl6/SITE/generic/VERSION/1.8.1/openmpi-1.8.1: No such file or directory
                  
                

and further down the check-build.sh script :

                  
                    ModuleCmd_Load.c
                    (
                    208
                    )
                    :ERROR:105: Unable to locate a modulefile 
                    for 
                    openmpi/1.8.1-gcc-5.2.0which: no mpirun 
                    in
                    (
                    /usr/local/bin:/bin:/usr/bin
                    )
                    Build step Conditional step 
                    (
                    single
                    )
                     marked build as failure
                  
                

The NumPy build failures are inhibiting the rest of the python chain including SciPy and others.

SitRep and TODO

So… this weekend’s burn down went ok, but not awesomely. We need to focus on testing the compilers out of the build environment now. (this would be the validation channel in the waffle board) and call in some help on the NumPy and SciPy issues. Also, some time needs to be spent on actually populating the board with the tickets for checking the application states…

The next community call is Wednesday 16/09 at midday. Be there or be ⁵

In the meantime :

References and Footnotes