TL;DR

We started to run some real-world performance studies on the grid. This post details the design of the experiment, as well as infrastructure and services used.

• TL;DR
• Background
• Design of the experiment
• Platform and services
  • Computing
  • Data storage
  • Coming Soon.
• References and Footnotes

Background

A few weeks ago, we were contacted by a student at the University of the North West who was undertaking a study of the relative performance of computing tasks on various architectures, using a series of common applications and data sets. The distributed computing architecture under consideration was indeed our very own AfricaGrid, specifically a sub-set of sites which support the SAGrid Virtual Organisation .

Design of the experiment

The experiment we had in mind was to determine the time required to process a set of standard data sets ¹ , using two machine-learning applications :

1. The Hidden Markhov Toolkit ²
2. the SVM library ³

These applications were integrated into the CODE-RADE platform in builds 148 and 146 respectively. Sites which supported the SAGrid VO were contacted by the VO manager, in order to see that they correctly mounted the CVMFS repository fastrepo ⁴ . Once this was done, the applications were available to all submitted jobs.

The processing of the data consisted in several tasks including training models with training data sets and then using them to perform characterisation on other data sets using those models. The questions that we aimed to answer, amongst others were :

1. Can this processing be done efficiently in a distributed environment ?
2. If the processing can be parallelised, what is the level of parallelism which gives the best efficiency ?
3. What overhead is there due to the geographically-distrbuted nature of the platform ?

Typically, these questions were answered quantitatively, by measuring the average amount of time required to perform various tasks in the workflow. Initially these were identified as :

1. Time taken to stage data from online storage endpoints to the compute facility
2. Time taken to perform estimation of the SVM “C value” ⁵

Since this was conducted in a real-world scenario, where other applications, users and services are vying for the same resources (compute time, network capacity, etc ) as our work, we expect there to be fluctuations in these times. In particular the following aspects are expeccted to impact on the average and shape of the distribution of these timing values :

1. Whether the site has local online storage - if not, data would have to be staged large distances over SANREN to the compute endpoint.
2. The performance tuning of the online storage and compute cluster frontend
3. The make and model of CPU on the compute cluser worker nodes.

Platform and services

The platform used consisted of three sites on the South African National Compute Grid, a subset of the Africa-Arabia Regional Operations Centre. These sites where :

• ZA-UJ : University of Johannesburg
• ZA-WITS-CORE : University of the Witwatersrand .
• ZA-CHPC : Centre for High-Performance Computing

These were selected due to their current state of support for the SAGrid VO. They also are fully certified in the ROC, providing middleware and operational compliance ⁶ . The OLA stipilates that these sites should have a certain level of operation of a pre-defined set of services. In particular:

High-Throughput Computing Platform

• Grid compute - allows scientists to run computational tasks on high quality IT resources, accessible via a standard interface and supporting authentication/authorisation based on a membership within a virtual organisation.
• Grid storage - allows files to be stored in and retrieved from high quality IT resources, accessible via a standard interface and supporting authentication/authorisation based on a membership within a virtual organisation.

These services are published by the sites, via the site BDII in the GLUE Schema ⁷ . Using the default top-level information index for the ROC ⁸ , the expected resources computing resources are :

Computing

The computing services in use at the sites is the CREAM-CE ^{9 10} , which is part of the Unified Middleware Distribution ¹¹ . These provide endpoints for job requests to be submitted to. The job requests were written in the JDL language, based on the Condor ClassAds. Job descriptions were written, as well as scripts which would set up the environment and execute the applications in the desired way and were published in the “CODE-RADE LibSVM release” ¹² . Workflows were submitted directly to CREAM interfaces using the command line glite-ce-job-submit tool. Authentication and authorisation was done by personal x.509 certificate and VOMS-signed proxies of that certificate.

Data storage

Data storage used during the study refers to protected online storage provided to the VO by the sites. Due to the high usage of storage by the big WLCG VOs (ATLAS and ALICE), capacity for smaller user communities is limited in South Africa. The services providing the online storage capability are UMD Disk Pool Managers (DPM) ¹³ exposing various transfer interfaces, such as SRM and GridFTP. The capacity (total and available) of these endpoints as published by their sites is shown in table below:

In order to ensure that the files were readily available locally at sites where online storage was available, the files were replicated to the respective storage endpoints. However this would have created a complication for the user, since data locality is lost. We therefore used the Logical File Catalogue (LFC) to replicate the files with a single namespace, and passed to the JDL as data input. For e.g. the NCHLT_2K data set, we thus have a single logical directory :

                  lfn:/grid/sagrid/nwu-hlt/NCHLT/
                

containing all of the data sets, which have been replicated across storage elements.

                  
                    DataRequirements 
                    =
                    {
                    [
                                           InputData 
                    =
                    {
                    "lfn:/grid/sagrid/nwu-hlt/NCHLT/NCHLT_2K.tar.gz"
                    }
                    ;                       DataCatalogType 
                    =
                    "DLI"
                    ;                       DataCatalog 
                    =
                    "http://lfc.magrid.ma:8085"
                    ;                      
                    ]
                    }
                    ;
                  
                

As can be seen here ¹⁴ , we are using the catalogue provided by the Moroccan National Grid Initiative (MAGrid), which is a member of the Africa-Arabia Regional Operations Centre. By registering the data sets in a single namespace, we can easily select the “closest” one to the computing element and improve the efficency of data transfer at job submission time.

Coming Soon.

This has been the first post on our performance study. Coming in the next one, we will describe the tasks that were done and the parameter scans, as well as how we used the gLibrary-2.0 metadata service to keep records of the timing performance.

References and Footnotes