Centre for Atmospheric Science
University of Cambridge Home


p-TOMCAT
A massively parallel tropospheric chemistry transport model

How to use p-TOMCAT on our local Dobson cluster

Hardware

The Dobson cluster consists of Sun Opteron based servers. Each machine has two dual-core processors which means p-TOMCAT (or MPI) sees 4 processors per machine. Each machine is connected via a high speed, low-latency Infiniband network which reduces communication delays compared to normal ethernet. Timings have shown the model runs >20% faster on Dobson than on HPCx.

You can log in to dobson (using rlogin,telnet or ssh) which is the front-end machine of the cluster but you cannot log into the other compute nodes. The /home and /data directories are mounted on all machines, so you can refer to files in these directories in your jobs.

An important point to note is that the cluster machines are based on Intel processors unlike the julius, caesar & rome servers which use sparc processors. Therefore you MUST recompile your programs before you can run them on dobson. Also, binary format on these machines is little-endian rather than big-endian on the other servers. If you write binary files from your Fortran programs on rome (say), you need to add the appropriate compiler option to read them correctly on dobson.

Setting up your environment

To access all the new software correctly and because dobson runs Solaris 10, you will need to add some directories to your PATH variable. The directories you need to add are:

/usr/sfw/bin : /opt/sfw/bin : /usr/local/bin : /opt/SUNWhpc/HPC7.0/bin

You also need to add the following directories to your MANPATH to correctly access the man pages:

/opt/n1/man : /usr/sfw/man : /usr/local/man : /opt/SUNWhpc/HPC7.0/man

Lastly, to access the batch queue commands and man pages, edit your .profile file and add this at the end of the file:

# N1 grid software startup
if [ -f /opt/n1/default/common/settings.sh ]; then
   . /opt/n1/default/common/settings.sh
fi

 

p-TOMCAT files

All the files required to run p-TOMCAT have been downloaded from HPCx and are in the directories under /home/tomcat/public/data. The sample dobson tom.run script has the correct directory names in the namelist for the model.

The forcing files are in /home/tomcat/public/data/ECMWF. We have all the forcing files from 1995- present day. I will keep the files up to date with the forcing files on HPCx. If anyone notices any missing or wants any earlier years please let me know.

 

Queues

There are two queues available on the cluster. Each queue support running jobs in parallel but they have different limits.
allnodes.q
This queue provides 16 processors and has a runtime limit of 2 days. It is a batch queue only.
4cpu.q
This queue provides 4 processors (all on dobson) and has a runtime limit of 12 hrs. It is both a batch and interactive queue and is intended for development or debugging runs.
To see the status of the queues you can use the commands:

qstat -g c
CLUSTER QUEUE CQLOAD USED AVAIL TOTAL aoACDS cdsuE
-------------------------------------------------------------------------------
4cpu.q 0.00 0 4 4 0 0
allnodes.q 0.00 0 16 16 0 0

or

qstat -f
queuename qtype used/tot. load_avg arch states
----------------------------------------------------------------------------
4cpu.q@dobson.atm.ch.cam.ac.uk BIP 0/4 0.01 sol-amd64
----------------------------------------------------------------------------
allnodes.q@node1.atm.ch.cam.ac BP 0/4 0.00 sol-amd64
----------------------------------------------------------------------------
allnodes.q@node2.atm.ch.cam.ac BP 0/4 0.00 sol-amd64
----------------------------------------------------------------------------
allnodes.q@node3.atm.ch.cam.ac BP 0/4 0.00 sol-amd64

 

Job control

In the same way as HPCx, the top of the tom.run script needs some job control instructions. The format for the dobson batch queue system is the same as NQS (for those who remember NQS batch systems).

In order to make your job run in the right queue you need to specify how many processors you want to use in the job (you can also specify the queue name as well).

Here's the top of the tom.run script for dobson with the suggested options:

#$ -N tomcat
#$ -pe orte 16 ( or #$ -pe devel 4 for the 4 cpu development queue)
#$ -M glenn@atm.ch.cam.ac.uk
#$ -m ea
#$ -cwd
#$ -j y
#$ -S /usr/bin/ksh

The options are:

-N
: name of the job. The job logfile will be created with this name and the job will appear in the queue with this name.
-pe orte 16 or -pe devel 4
: this specifies the parallel environment and should always be present. The name 'orte' means run on the main batch queue on the cluster which has a limit of 2 days runtime and 16 cpus. The name 'devel' means run on dobson on the 4 cpu queue which has a limit of 12hrs. The number following is the number of processes you are requesting, in this case 16 processes.
-M glenn@atm.ch.cam.ac.uk
-m ea
: These options control email to the user when the job finishes or aborts. The -M option gives the email address of the person who should be emailed. The -m ea option specifies the conditions when the user should be emailed: the 'e' of 'ea' means send an email when the job 'ends', the 'a' of 'ea' means send an email when the job aborts. There are other conditions that can trigger an email, for instance, '-m be' would mean send an email when the job begins and one when it ends. For more options see the man page for the qsub command.
-cwd
: this makes the job logfile go into the same directory where the job was submitted from. If you don't specify this the job logfile will go into your home directory.
-j y
: means merge the standard output and error messages from the job. If you don't specify this, they will be separated into two logfiles.
-S /usr/bin/ksh
: this specifies the shell that the job should be run under. If you don't specify this then /bin/csh (C shell) is the default. p-TOMCAT jobs require the K shell in order to run correctly.
(optional) -q queuename e.g. -q allnodes.q or -q 4cpu.q
: this specifies the queue to run under. You don't need this normally because if you specify the parallel environment option (-pe) this puts the job in the correct queue.

 

Submitting jobs

There are 2 batch queues available on dobson described above. As dobson is our local cluster the queues can be configured how we decide. If anyone thinks a different queue setup would be better let me know.

Use the tom.run.dobson script downloadable from the main p-TOMCAT website. This run script will have the job control options specified above. Change any you might need to (ie. email address, no. of processors).

Also, check the namelist variables and make sure the experiment directory is correct. The directories in the namelist will be different in the tom.run.dobson job and should be correct. But if you have any custom settings in your HPCx jobs you will need to change them here.

Once the run script is ready, you submit the job by the command:

qsub tom.run.dobson

or whatever your run script filename is called.

To check on the status of your job, use the 'qstat' command. e.g.

Dobson $ qstat
         job-ID prior name user state submit/start at queue  slots  ja-task-ID
         ---------------------------------------------------------------------------------------
         28 0.55500 tomcat glenn r 04/24/2007 21:57:04 batch@node2.atm.ch.cam.ac.uk          16

If your job is not running you can use 'qstat -j' followed by the job-id (e.g. qstat -j 28) which then gives more details and the reason why the job isn't running.

By default qstat will only list your jobs. If you want to see all the jobs in the batch queues then do: qstat -u "*".

To delete a queued or running job, use the 'qdel' command. e.g.

qdel 28

For more details of these commands, see their man pages.

NOTE! It is very important when your job is running that you don't attempt to look at the log file being generated by the model. For example, the job above would send it's output to the file: tomcat.o28. Due to the way parallel I/O is handled by the system, if you attempt look at this file in any way whilst the job is running, say by using 'more' or 'tail', no further output will be written to this file. Do NOT look at the contents of the file until the job has finished otherwise you will lose the rest of the output: the job will continue until finished by no further log output will be written.

 

Useful commands

Some other commands which might be useful:

qmon - brings up a X11 program which allows you to control jobs submitted. It also allows you to inspect the job queues and configurations. Some of the options on this program will be disabled for normal users.

qhost - gives various pieces of information regarding the cluster machines e.g.

Dobson $ qhost
HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-------------------------------------------------------------------------------
global                  -               -     -       -       -       -       -
dobson                  sol-amd64       4  0.00    3.4G  835.0M    7.0G     0.0
node1                   sol-amd64       4  0.00    3.4G  753.0M    7.0G     0.0
node2                   sol-amd64       4  0.00    3.4G  649.0M    7.0G     0.0
node3                   sol-amd64       4  0.00    3.4G  649.0M    5.0G     0.0
node4                   sol-amd64       4  0.00    4.0G  650.0M    7.0G     0.0