We killed Jenkins

We didn’t mean to. Actually, we were just about to publish Foundation Release 1 . The plan was to make the release, upgrade Jenkins and move on… in that order . Of course, thinking that everything would be fine if we just upgraded Jenkins and then published the release, we went ahead and upgraded to 1.625 and then … bad things happened.

What went wrong

So, actually the service is up and running, but several - almost all - of our jobs were removed from the Jenkins configuration . This is pretty horrific, we’d spent months adding these jobs and painstakingly ensuring that their dependencies were correctly encoded… However, checking the config directory, we noticed that all of the job configs ¹ were still there. They just weren’t being accepted by Jenkins.

So, what went wrong ?

The first log entry we encountered was :

                  WARNING: Failed to scout com.tikal.jenkins.plugins.multijob.MultiJobBuildSelector java.lang.InstantiationException: java.lang.NoClassDefFoundError: hudson/plugins/copyartifact/BuildSelector

Now, most of our jobs are matrix jobs which allow us to test the various configurations of operating system, version number, compiler configuration, etc . The above error was the first hint that this was a plugin issue and not an actual Jenkins issue.

Then, we started getting these :

                  SEVERE: Could not load actions from hudson.plugins.jswidgets.JsProjectActionFactory@6ca0e1b6 for hudson.matrix.MatrixProject@73ac157[hdf5-deploy]

followed by

                  Oct 23, 2015 2:41:58 PM jenkins.InitReactorRunner$1 onTaskFailedSEVERE: Failed Loading job hdf5-deployjava.lang.NullPointerException

As you can see that is for a specific job hd5-deploy , which was using the MatrixProject configuration. It’s also using a bunch of Jenkins plugins which are how we configure these jobs to do the things we want them to do.

Job triggers ?

In particular, the job build triggers are using the Github and Github Pull Request plugins.

The lowest-lying job in the dependency graph, gmp-deploy was also not loaded. This is only triggered by Github events, and not other jobs ² and was throwing errors like this :

                  Oct 23, 2015 2:42:13 PM jenkins.InitReactorRunner$1 onTaskFailedSEVERE: Failed Loading job gmp-deployjava.lang.NullPointerExceptionat hudson.matrix.MatrixProject.createTransientActions(MatrixProject.java:442)at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:748)at hudson.matrix.MatrixProject.updateTransientActions(MatrixProject.java:456)at hudson.model.AbstractProject.save(AbstractProject.java:304)at org.jenkinsci.plugins.ghprb.GhprbTrigger.save(GhprbTrigger.java:171)at org.jenkinsci.plugins.ghprb.GhprbTrigger.start(GhprbTrigger.java:186)at org.jenkinsci.plugins.ghprb.GhprbTrigger.start(GhprbTrigger.java:52)at hudson.model.AbstractProject.onLoad(AbstractProject.java:326)at hudson.matrix.MatrixProject.onLoad(MatrixProject.java:497)at hudson.model.Items.load(Items.java:322)at jenkins.model.Jenkins$17.run(Jenkins.java:2655)  at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)

So, this points in the direction of the “GhprbTrigger” the trigger that says “build this job when a pull request is sent to the Github repo”. There are some suggestions that these are known bugs, from the Jenkins JIRA , but the solution that was suggested did not fix anything.

Hints from the job configurations

The gmp-deploy job configuration looks like this :

                  
                    
                    
                    plugin=
                    ""
                    >
                    
                    
                    
                    class=
                    "hudson.tasks.LogRotator"
                    >
                    
                    30
                    
                    
                    100
                    
                    
                    60
                    
                    
                    30
                    
                    
                    
                    false
                    
                    
                    
                    plugin=
                    ""
                    >
                    
                    false
                    
                    
                    
                    plugin=
                    ""
                    >
                    
                    https://github.com/SouthAfricaDigitalScience/gmp-deploy.git/
                    
                    
                    
                    plugin=
                    ""
                    >
                    
                    false
                    
                    
                    
                    plugin=
                    ""
                    >
                    
                    
                    human approved
                    
                    
                    build passes on sl6
                    
                    
                    build passes on u1404
                    
                    
                    push artifacts
                    
                    
                    
                    
                    
                    class=
                    "hudson.plugins.git.GitSCM"
                    plugin=
                    ""
                    >
                    
                    2
                    
                    
                    
                    
                    https://github.com/SouthAfricaDigitalScience/gmp-deploy
                    
                    
                    you wish
                    
                    
                    
                    
                    
                    
                    */master
                    
                    
                    
                    
                    false
                    
                    
                    class=
                    "list"
                    />
                    
                    
                    
                    
                    plugin=
                    ""
                    >
                    
                    
                    plugin=
                    ""
                    >
                    
                    
                    
                    plugin=
                    ""
                    >
                            ... stuff ...      
                    
                    
                    gmp-deploy
                    
                    <__project
                    class=
                    "matrix-project"
                    reference=
                    "../../.."
                    />
                    
                    you wish
                    
                    
                    
                    
                    
                    
                    
                    
                    
                    
                    
                    
                    false
                    
                    
                    
                    
                    VERSION
                    
                    
                    
                    5.1.3
                    
                    
                    
                    
                    
                    SITE
                    
                    
                    
                    generic
                    
                    
                    
                    
                    
                    OS
                    
                    
                    
                    sl6
                    
                    
                    u1404
                    
                    
                    
                    
                    
                    ARCH
                    
                    
                    
                    x86_64
                    
                    
                    
                    
                    
                    class=
                    "hudson.matrix.DefaultMatrixExecutionStrategyImpl"
                    >
                    
                    false

If anyone is out there reading this and can spot the error, please help us out !

We goofed - might as well publish a relase…

So - we goofed ! CODE-RADE won’t be building anything for a few days, while we get this sorted out. The important thing is to learn some lessons :

First publish a release, then upgrade your build server
Upgrade jenkins and the plugins separately
Upgrade plugins one by one
Maintain a roll-back strategy in case of Bad Things like this.

So, we’ll be publishing Foundation Release 1 as is . Warts and all. What’s in there will be in there, what’s missing will come in Foundation Release 2 , scheduled for November 13 2015 . There should also be a first Public Release on October 31 2015 .

Then, Github died

Ok, so builds weren’t triggered… we needed to fix this and fast ! But this morning, we got another surprise :

The well-known step function. Decidedly sub-optimal.

We're experiencing issues delivering webhooks to a small number of hosts as we mitigate a DDoS attack.
— GitHub Status ( @githubstatus )

Sigh. Your pain is our pain @github

hang in there - don't know what we'd do without you ! https://t.co/zWrPquMWUl
— SAGrid ( )

Footnotes and References

The job configurations are kept in xml format in $JENKINSHOME/jobs . These are kept in source control (actually a private Github Repository) - so it could not really have been such a huge disaster. ↩
Ok, sometimes we humans intervene and trigger jobs, but only if they’ve been very naughty ! ↩

Tagged with blog • CODE-RADE • sitrep • downtime • goof • doh