The PPD Batch System

Resources

Allocations

The system tries to "fairly" share out jobs starts between different users, it uses a number of different factors to try to do this. Amongst them are whether it is a "local" or grid job, the Grid VO, groups within the VOs and individual user accounts. These are compared to the usage over the last 14 days.

Grid submission to the SL5 Cluster is now switched off so 100% of the remaining resources are available for local Tier 3 jobs.

For the SL6 Condor service the current highest level Shares are:

There are currently no group shares in the Local partition, for the grid jobs the split below the Grid partition is as follows:

-- ChrisBrew - 2009-11-17

The PPD Batch System

The PPD Batch Systems are shared between the Grid Tier 2 and Local usage (often referred to as Tier 3). We now have two batch system, the legacy Scientific Linux 5 system uses the torque batch system (an evolution of PBS) and the Maui scheduler to allocate the job slots fairly. While the newer Scientific Linux 6 services use Condor.

The PPD Batch System

The PPD Batch Systems are shared between the Grid Tier 2 and Local usage (often referred to as Tier 3). We now have two batch system, the legacy Scientific Linux 5 system uses the torque batch system (an evolution of PBS) and the Maui scheduler to allocate the job slots fairly. While the newer Scientific Linux 6 services use Condor.

Condor

Condor works differently to many other batch systems so it is advised that you have a look at the User Manual. We are currently only supporting the "Vanilla" universe.

Submitting a Job

To submit a jobs to the condor batch system you first need to write a "submit description file" to describe the job to the system: A very simple file would look like this:

####################                                                    
# 
# Example 1                                                            
# Simple HTCondor submit description file                                    
#                                                                       
####################                                                    
                                                                        
Executable   = myexe                                                    
Log          = myexe.log                                                    
Input        = inputfile
Output       = outputfile
Queue

That runs myexe on the batch machine (after copying it and inputfile to a temporary directory on the machine) and copies back the standard output of the job to a file called outputfile

A more complex example submit description would look like:

####################                                                    
# 
# Example 2                                                            
# More Complex HTCondor submit description file                                    
#                                                                       
####################                                                    

Universe               = vanilla
Executable             = my_analysis.sh
Arguments              = input-$(process).txt result/output-$(process).txt
Log                    = log/my_analysis-$(Process).log
Input                  = input/input-$(process).txt
Output                 = output/my_analysis-$(Process).out
Error                  = output/my_analysis-$(Process).err
Request_memory         = 2 GB
Transfer_output_files  = result/output-$(process).txt
Transfer_output_remaps = "output-$(process).txt = results/output-$(process).txt"
Notification           = complete
Notify_user            = your.name@stfc.ac.uk
Getenv                 = True
Queue 20

This submit runs 20 copies (Queue 20) of my_analysis.sh input-$(process).txt result/output-$(process).txt where $(process) is replaced by a number 0 to 19. It will copy my_analysis.sh and input-$(process).txt to each of the worker nodes (taking the input file from the local input directory). The standard output and error from the job are copied back to the local output directory at the end of the job and the file result/output-$(process).txt is copied back to the local results directory. It copies over the local environment to the worker node (Getenv = True) and requests 2GB of memory to run in on the worker node. Finally it e-mails the user when each job completes.

Torque

Resources

The PPD Batch Cluster currently has a nominal capacity of 1584 job slots (one for each of the 1584 CPUs) on 240 nodes. The power of the CPUs are measured using a benchmark called HEPSPEC06, the individual CPUs range from a HEPSPEC06 rating of 6.80 for the oldest CPUs to 8.50 for the newest. The nominal total HEPSPEC06 rating of the cluster is 12,918.4

While almost all of the CPUs (1544) run 64bit Scientific Linux 5, there is a small 32bit SL4 service of 40 CPUs for groups that still need it - though that will be phased out as the need goes away.

Allocations

Then below that:

(Which actually makes slightly over 100%, but the system sorts it out!)

-- ChrisBrew - 2009-11-17

The PPD Batch System

The PPD Batch Systems are shared between the Grid Tier 2 and Local usage (often referred to as Tier 3). We now have two batch system, the legacy Scientific Linux 5 system uses the torque batch system (an evolution of PBS) and the Maui scheduler to allocate the job slots fairly. While the newer Scientific Linux 6 services use Condor.

Condor

Condor works differently to many other batch systems so it is advised that you have a look at the User Manual. We are currently only supporting the "Vanilla" universe.

Submitting a Job

To submit a jobs to the condor batch system you first need to write a "submit description file" to describe the job to the system: A very simple file would look like this:

####################                                                    
# 
# Example 1                                                            
# Simple HTCondor submit description file                                    
#                                                                       
####################                                                    
                                                                        
Executable   = myexe                                                    
Log          = myexe.log                                                    
Input        = inputfile
Output       = outputfile
Queue

That runs myexe on the batch machine (after copying it and inputfile to a temporary directory on the machine) and copies back the standard output of the job to a file called outputfile

A more complex example submit description would look like:

####################                                                    
# 

Universe = vanilla Executable = my_analysis.sh Arguments = input-$(process).txt result/output-$(process).txt Log = log/my_analysis-$(Process).log Input = input/input-$(process).txt Output = output/my_analysis-$(Process).out Error = output/my_analysis-$(Process).err Request_memory = 2 GB Transfer_output_files = result/output-$(process).txt Transfer_output_remaps = "output-$(process).txt = results/output-$(process).txt" Notification = complete Notify_user = your.name@stfc.ac.uk Getenv = True Queue 20

This submit runs 20 copies (Queue 20) of my_analysis.sh input-$(process).txt result/output-$(process).txt where $(process) is replaced by a number 0 to 19. It will copy my_analysis.sh and input-$(process).txt to each of the worker nodes (taking the input file from the local input directory). The standard output and error from the job are copied back to the local output directory at the end of the job and the file result/output-$(process).txt is copied back to the local results directory. It copies over the local environment to the worker node (Getenv = True) and requests 2GB of memory to run in on the worker node. Finally it e-mails the user when each job completes.

Torque

Resources

The PPD Batch Cluster currently has a nominal capacity of 1584 job slots (one for each of the 1584 CPUs) on 240 nodes. The power of the CPUs are measured using a benchmark called HEPSPEC06, the individual CPUs range from a HEPSPEC06 rating of 6.80 for the oldest CPUs to 8.50 for the newest. The nominal total HEPSPEC06 rating of the cluster is 12,918.4

While almost all of the CPUs (1544) run 64bit Scientific Linux 5, there is a small 32bit SL4 service of 40 CPUs for groups that still need it - though that will be phased out as the need goes away.

Allocations

The system uses the Maui scheduler to "fairly" share out jobs starts between different users, it uses a number of different factors totry to do this. Amongst them are whether it is a "local" or grid job, the Grid VO, groups within the VOs and individualuser accounts. These are compared to theusage over the last 14 days.

The current highest level Shares are:

Then below that:

(Which actually makes slightly over 100%, but the system sorts it out!)

-- ChrisBrew - 2009-11-17

The PPD Batch System

Resources

The PPD Batch Cluster currently has a nominal capacity of 1584 job slots (one for each of the 1584 CPUs) on 240 nodes. The power of the CPUs are measured using a benchmark called HEPSPEC06, the individual CPUs range from a HEPSPEC06 rating of 6.80 for the oldest CPUs to 8.50 for the newest. The nominal total HEPSPEC06 rating of the cluster is 12,918.4

While almost all of the CPUs (1544) run 64bit Scientific Linux 5, there is a small 32bit SL4 service of 40 CPUs for groups that still need it - though that will be phased out as the need goes away.

Allocations

The system uses the Maui scheduler to "fairly" share out jobs starts between different users, it uses a number of different factors totry to do this. Amongst them are whether it is a "local" or grid job, the Grid VO, groups within the VOs and individualuser accounts. These are compared to theusage over the last 14 days.

The current highest level Shares are:

Then below that:

(Which actually makes slightly over 100%, but the system sorts it out!)

The PPD Batch System

The PPD Batch System is shared between the Grid Tier 2 and Local usage (often referred to as Tier 3). It uses the torque batch system (an evolution of PBS) and the Maui scheduler to allocate the job slots fairly.

Submitting and monitoring jobs

Resources

The PPD Batch Cluster currently has a nominal capacity of 1584 job slots (one for each of the 1584 CPUs) on 240 nodes. The power of the CPUs are measured using a benchmark called HEPSPEC06, the individual CPUs range from a HEPSPEC06 rating of 6.80 for the oldest CPUs to 8.50 for the newest. The nominal total HEPSPEC06 rating of the cluster is 12,918.4

While almost all of the CPUs (1544) run 64bit Scientific Linux 5, there is a small 32bit SL4 service of 40 CPUs for groups that still need it - though that will be phased out as the need goes away.

Allocations

Queues

-- ChrisBrew - 2009-11-17

The PPD Batch System

The PPD Batch System is shared between the Grid Tier 2 and Local usage (often referred to as Tier 3). It uses the torque batch system (an evolution of PBS) and the Maui scheduler to allocate the job slots fairly.

Resources

The PPD Batch Cluster currently has a nominal capacity of 1584 job slots (one for each of the 1584 CPUs) on 240 nodes. The power of the CPUs are measured using a benchmark called HEPSPEC06, the individual CPUs range from a HEPSPEC06 rating of 6.80 for the oldest CPUs to 8.50 for the newest. The nominal total HEPSPEC06 rating of the cluster is 12,918.4

Allocations

The system uses the Maui scheduler to "fairly" share out jobs starts between different users, it uses a number of different factors totry to do this. Amongst them are whether it is a "local" or grid job, the Grid VO, groups within the VOs and individualuser accounts. These are compared to theusage over the last seven day.

The current highest level Shares are:

Then below that:

Queues

-- ChrisBrew - 2009-11-17

The PPD Batch System

The PPD Batch System is shared between the Grid Tier 2 and Local usage (often referred to as Tier 3). It uses the torque batch system (an evolution of PBS) and the Maui scheduler to allocate the job slots fairly.

Resources

The PPD Batch Cluster currently has a nominal capacity of 1584 job slots (one for each of the 1584 CPUs) on 240 nodes. The power of the CPUs are measured using a benchmark called HEPSPEC06, the individual CPUs range from a HEPSPEC06 rating of 6.80 for the oldest CPUs to 8.50 for the newest. The nominal total HEPSPEC06 rating of the cluster is 12,918.4

While almost all of the CPUs (1544) run 64bit Scientific Linux 5, there is a small 32bit SL4 service of 40 CPUs for groups that still need it - though that will be phased out as the need goes away.

Allocations

The system uses the Maui scheduler to "fairly" share out jobs starts between different users, it uses a number of different factors totry to do this. Amongst them are whether it is a "local" or grid job, the Grid VO, groups within the VOs and individualuser accounts. These are compared to theusage over the last seven day.

The current highest level Shares are:

Then below that:

Queues

-- ChrisBrew - 2009-11-17