You can never have too many compilers
Well, you can, I guess, but the whole point of CODE-RADE is to make any possible combination of applications available to researchers anywhere. So, during the Summer Hackfest it came to our attention that a new version of GCC had been released - GCC-6 . It was time to add a few more ticks to the GCC-deploy matrix jobs
You can totally have too many compilers
Well, enthusiasm is great and all, but it turns out that disk space is limited at tiny build cluster, which made the last few jobs of GCC fail. This is only going to be exacerbated when we start extending the list of targets that we support - currently only one kind of CPU and two operating systems. It’s going to be time to distribute the load across clouds soon…
A shift in CODE-RADE architecture ?
The original Jenkins setup at UFS was conceived to be as simple as possible - two slaves connected to a master, which built applications into an integration environment locally, then published the build results to a CVMFS repository on site. Basically what happens during integration gets built into
/apprepo
, while what happens during deploy gets built into
/cvmfs/fastrepo.sagrid.ac.za
. Both of these are local directories on the slave’s hard disk.
This locality is clearly not scalable, which was indeed forseen by Fanie during initial discussions. He suggested building relocatable tarballs of the artifacts and keeping them in a central repository for use during integration, shifting them down to arbitrary build slaves as needed. I argued against this for simplicity reasons, suggesting instead that we mount the shared directories over NFS. This would have worked as long as
- The build slaves had infinite local disk storage
- The build slaves were on the same LAN as the other components (CI server and CVMFS repo)
Both of those conditions are now close to being hard to fulfill. One potential way around this, whilst not sacrificing too much efficiency, is to keep atomic tarballs of integration builds, similar to how Homebrew keeps bottles, or perhaps how Python keeps eggs. These could be pulled down to a build slave at execution time as dependencies and unpacked just before compilation of the requested application. One might figure what the heck, why not just fork Homebrew or EasyBuild entirely and use that to build everything… and that’s something we need to discuss at some point.
For now, let’s just see how we can distribute the builds and highlight where there are serious structural weaknesses in the architecture.