(r2) HowToRunDaVinciAtRAL < LHCb

Tags: view all tags
---+!! How to run DaVinci at RAL

%TOC%

---++ Creating an ntuple

---+++ Real Data

to create an ntuple with real data you need
   * the DST files you wish to run on
   * a python file pointing to the data
   * the davinci python file

My python file pointing to my data looks like:
<verbatim>from Gaudi.Configuration import *

EventSelector().Input   = [
#  for i in /opt/ppd/lhcb/nraja/MC2010/MC_2010_428935_DST/* ; do echo "\"   DATAFILE='pfn:${i}' TYP='POOL_ROOTTREE' OPT='READ'\"," ; done
"   DATAFILE='pfn:/opt/ppd/lhcb/nraja/MC2010/MC_2010_428935_DST/00006639_00000001_1.dst' TYP='POOL_ROOTTREE' OPT='READ'",
"   DATAFILE='pfn:/opt/ppd/lhcb/nraja/MC2010/MC_2010_428935_DST/00006639_00000002_1.dst' TYP='POOL_ROOTTREE' OPT='READ'",
# ...
]</verbatim>

<br /> and can be mostly generated with the for command commented in the file.

<strong>Be careful </strong>when creating this file as <strong>symlinks </strong>have caused me issues and prevented DaVinci from reading the data. Since the files existed it caused no errors and it seemed to just skip the file it was looking at resulting in<strong> 0 events processed.</strong>

the davinci file needed to create the ntuple can be found here: [[%ATTACHURL%/DaVinciAndCandidatesUp.py.txt][DaVinciAndCandidatesUp.py.txt]], you might want to remove the .txt from the file name.

Several things will need to be regularly changed in this file for your specific needs.
   * The number of events to analyse, DV.EvtMax which takes a value of -1 for all or a number&gt;0 for that number of events.
   * If the input data is simulated or not change DV.Simulation to True for simulated
   * The place where the final root ntuple is saved NTupleSvc().Output I usually have it pointing to something like [ "FILE1 DATAFILE='/home/hep/uoh35620/stuff/batch-generate/data/up/all/B2Dpi_Up_6_7_2010.direct.root' TYP='ROOT' OPT='NEW'" ]

It is also worth noting that recently a new version of DaVinci seems to have come out where DV.Lumi is set to true by default, which it was not previously, this has the effect of putting all your data in lumi.root whether or not you want it there, since I don't know the point of this variable, I set it to false. Check the DaVinci.py file for this if not add it.

Before running this you need to setup your environment to find the LHCb programs like SetupProject, to do this add the following line to your .bashrc and restart your shell. Be careful, the dot at the start of the line is very important.

<verbatim>. /opt/ppd/lhcb/lhcb/scripts/lhcbsetup.sh</verbatim>

After restarting your shell all this can be run with the following commands.
<verbatim>SetupProject DaVinci
gaudirun.py DaVinciAndCandidatesUp.py data.py | tee output.txt</verbatim>

the output will be copied to stdout and a file called output.txt. This will create a .root file which contains the ntuple and is readable with the macros mentioned in the analysis section below.

---++++ Running on ppd batch

you can run these taks on the ppd batch system, since doing a large number of events can take many hours. below is a script I use to run these taks.

<verbatim>#!/bin/bash

stda="`date`"
echo $stda

. /afs/rl.ac.uk/lhcb/lhcb/LBSCRIPTS/LBSCRIPTS_v5r2/InstallArea/scripts/SetupProject.sh DaVinci

dir=/home/hep/uoh35620/stuff/batch-generate/data/up/all

gaudirun.py ${dir}/DaVinciAndCandidatesUp.py ${dir}/6_7_2010_up.py  | tee ${dir}/output.txt | tee /home/hep/uoh35620/stuff/batch-generate/test.txt

echo
echo $stda
date
echo
echo Done.</verbatim>

The dot at the beginning of the line ending in SetupProject.sh DaVinci is very important. If this is not there the paths will not be set up properly for your script.

When using this script there are some things you settings you might want to change:
   * The dir variable to the correct directory where the scripts are.
   * Make sure the filenames in the gaudirun command are correct and pointing to data that does not involve symlinks

Placing this in a file called say script and running the following will make it run on the batch system.

<verbatim>qsub -q prod -S /bin/bash -j oe -o /home/hep/uoh35620/job-out/ script</verbatim>

-q chooses the queue you wish to submit the job to while -j oe joins the stderr and stdout together into one file. -o puts the output file into a given directory and script is the above script to run the job.
---+++++ Monitoring

The following command will list your currently running jobs. you will need to replace uoh35620 with your user name.

<verbatim>qstat -u uoh35620</verbatim>

For looking at jobs in real time I like to use the following, where the file at the beginning is the output of one of the tee's in the script and uoh35620 will need to be changed to your user name. There are also other various paths after that which would alos need to be changed, good luck, but you could just delete the end it only monitors the file sizes of root files.

<verbatim>watch -n 10 'tail -n 20 /home/hep/uoh35620/stuff/batch-generate/test.txt ; echo ; echo ; qstat -u uoh35620 ; echo ; echo ; du -hs ~/stuff ; echo ; echo ; find ~/stuff -iname "*.root" -print0 | xargs --null du | sort -nr | cut -f 2 | tr "\\n" "\\0" | xargs --null du -h | sed "s#/home/hep/uoh35620/stuff/##"'</verbatim>

you might want to make alias of these commands in your ~/.bashrc file so the following in ~/.bashrc will make a new command called qstatme which will be like typing the full command.

<verbatim>alias qstatme="qstat -u uoh35620"</verbatim>

---+++++ PBS Batch Errors

I have gotten this email a few times and I'm not really sure of the cause, maybe an error on the bathch macine, but anyway, if you get this just keep re-submitting exactly the same job until it works. The most I've ever had to do it is about 5 times.

<verbatim>PBS Job Id: 4121274.heplnx208.pp.rl.ac.uk
Job Name:   script
Exec host:  heplnc308.pp.rl.ac.uk/0
An error has occurred processing your job, see below.
Post job file processing error; job 4121274.heplnx208.pp.rl.ac.uk on host heplnc308.pp.rl.ac.uk/0

Unable to copy file /var/spool/pbs/spool/4121274.heplnx208.pp.rl.ac.uk.OU to /home/hep/uoh35620/job-out/
>>> error from copy
/bin/cp: cannot stat `/var/spool/pbs/spool/4121274.heplnx208.pp.rl.ac.uk.OU': No such file or directory
>>> end error output</verbatim>

---+++++ Splitting the job with ganga on the ppd batch

Using ganga to run DaVinci can be advantageous since it is able to run it on CERN batch, PPD batch, the grid or locally very easily also it can split the job into many pieces for faster processing.

Ganga likes to control the input and output files of the various jobs and sub jobs so it is best not to use absolute output paths for the ntuples otherwise all the sub jobs overwrite one another's root files, so by changing the NTupleSvc().Output property in the DaVinci python file to something like [ "FILE1 DATAFILE=B2Dpi_Up_6_7_2010.split.direct.root' TYP='ROOT' OPT='NEW'" ], this can be prevented.

Fire up ganga with

<verbatim>SetupProject Ganga
ganga</verbatim>

then in ganga create a job template with the correct options like so, convert it into a job and submit it. You will want to change the directory where the python files are stored and also the filenames if the are differnt to the ones I have used. "6_7_2010_up.direct.py" is the name of my python file pointing to my data. Additionally the name of the file to merge from the subjobs will need to be changed to the one specified in the DaVinci python file. The name option which would be useful to keep track of the submitted jobs does not appear to work with the PBS system as ganga trys to pass qsub -N name option which qsub does not like so it wont submit the job when you set a name.

<verbatim>t = JobTemplate( application = DaVinci() )

dir= "/home/hep/uoh35620/stuff/batch-generate/data/up/all/"
t.application.optsfile = [dir+"DaVinciAndCandidatesUp.py", dir+"6_7_2010_up.direct.py"]

t.splitter = SplitByFiles()
t.splitter.filesPerJob = 2

t.backend = PBS()
t.backend.queue = "prod"

t.merger= RootMerger()
t.merger.files = ['Bu2DStarplusX.root']

j = Job(t)
j.submit()</verbatim>

you can use the qstat commands mentioned before to check that it has worked. also you will be able to look at the value of j.status and j.subjobs to see the progress of the job as a whole and also the subjobs.

---+++ MC Data

Creating the ntuple with monte carlo data is harder than with the actual data since you must first strip it before creating the ntuple, but can be done with a simalar set up as before. Again you will need
   * the DST files you wish to run on
   * a python file pointing to the data
   * the davinci python file
   * the stripping python file

The stripping python file can be found here [[%ATTACHURL%/StrippingB2DPiLoose.mod.py.txt][StrippingB2DPiLoose.mod.py.txt]], again you probably want to change the file extension.

The python file pointing to the MC data is the same format as before just with different filenames, see the Real Data section.

the DaVinci file that you need is this one here [[%ATTACHURL%/DaVinci.py.txt][DaVinci.py.txt]].

Again you might want to change a few things to make this work how you want it to.
   * The path to StrippingB2DPiLoose.mod.py will need to be changed.
   * The number of events to analyse, DV.EvtMax which takes a value of -1 for all or a number&gt;0 for that number of events.
   * If the input data is simulated or not change DV.Simulation to True for simulated
   * The place where the final root ntuple is saved NTupleSvc().Output I usually have it pointing to something like [ "FILE1 DATAFILE='/home/hep/uoh35620/stuff/batch-generate/data/up/all/B2Dpi_Up_6_7_2010.direct.root' TYP='ROOT' OPT='NEW'" ]

Then this can again be run like the real data, with the following command.
<verbatim>SetupProject DaVinci
gaudirun.py DaVinci.py data.py | tee output.txt</verbatim>

The output will be copied to output.txt. To run it on the ppd batch system use the same method and scripts as before but cahge the filenames where appropriate.

---++ List of Files

Below are all the files mentioned on this wiki

   * [[%ATTACHURL%/StrippingB2DPiLoose.mod.py.txt][StrippingB2DPiLoose.mod.py.txt]]: File needed to strip MC data
   * [[%ATTACHURL%/DaVinci.py.txt][DaVinci.py.txt]]: DaVinci file to run the stripping of MC data
   * [[%ATTACHURL%/DaVinciAndCandidatesUp.py.txt][DaVinciAndCandidatesUp.py.txt]]: File to generate ntuple for actual data

-- Main.ThomasBird - 2010-07-26
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
txt	DaVinci.py.txt	r1	manage	3.2 K	2010-07-26 - 11:18	ThomasBird	DaVinci file to run the stripping of MC data
txt	DaVinciAndCandidatesUp.py.txt	r1	manage	6.5 K	2010-07-26 - 11:19	ThomasBird	File to generate ntuple for actual data
txt	StrippingB2DPiLoose.mod.py.txt	r1	manage	9.1 K	2010-07-26 - 11:18	ThomasBird	File needed to strip MC data
Topic revision: r2 - 2010-08-06 - ThomasBird