Centre for Atmospheric Science
University of Cambridge Home


p-TOMCAT
A massively parallel tropospheric chemistry transport model

How to use p-TOMCAT on the Cambridge University Darwin HPC computer

Darwin is the High Performance Computer (HPC) service provided by the University of Cambridge. It is free for all University members to use but priority is giving to those who pay for the service.

Further information about the machine can be found on the Darwin HPC website at http://www.hpc.cam.ac.uk/.

Note that at the time of writing, Darwin does not have any tape archiving facility. The best approach would be to sftp your output files to HPCx for storing on their tape system or sftp the output to your local Union Road account.

Registering for an account

To use the machine you first need to register for an account. To do this, go to: http://www.hpc.cam.ac.uk/applications.html and download and complete the application form. Note that the PI is John Pyle, leave the 'funding' section blank as we currently don't have money to pay for use of the machine. Your 'CRSid' is your username on the Central Cambridge services such as hermes. If you do not have a CRSid username you will need to apply for one from the Chemistry Dept. computing officers.

Using p-TOMCAT

In order to use p-TOMCAT you must make sure you have the directory: /home/gdc11/bin, in your PATH environment variable. This provides access to 'nupdate' needed for the tom.build script.

Full details on using the machine are provided at the Darwin HPC website (http://www.hpc.cam.ac.uk/). Only brief details are provided below.

To connect to the machine, use 'ssh username@login.hpc.cam.ac.uk'. Note that Darwin can only be accessed from machines in the 'cam.ac.uk' domain.

Note also that Darwin is based on Intel processors and is therefore a 'little-endian' machines, whereas p-TOMCAT binary input files are 'big-endian'. By default the Makefile for p-TOMCAT will specify a compiler option to ensure binary files are read in as 'big-endian'. If you are generating binary files for p-TOMCAT (or reading them) on Darwin remember to recompile your code and add in the '-convert big_endian' compiler option.

p-TOMCAT files

All p-TOMCAT files (data and nupdate libraries) are available in /home/gdc11/tomcat. Not all years of forcing files have been transferred to Darwin. If you find the year you need is missing please contact Glenn.

Job control

Job control is very similar to HPCx and previous HPC computer systems. Keywords are included as comments at the top of the tom.run scripts. Darwin uses a batch queue system called 'PBS'. The job control options are explained in the tom.run script and in more detail on the HPC website. Perhaps the only one that might need explaining is:

      #PBS -l nodes=8:ppn=4,walltime=02:00:00
      
Each 'node' on HPC consists of 4 compute cores which behave a 4 separate processors. To specify the TOTAL number of processes you need for the job (say, 32 processes) you need to specify the number of nodes and number of processes-per-node (ppn) which is always '4' if you want to use a complete node. So the above line would ask for 8 nodes each with 4 processors to give a total of 32 processors for the job.

To submit a job is 'qsub tom.run'. You can use the 'qstat' command to examine the jobs on the queue but a better command is 'showq'. A useful command which tells you when your job is likely to start is the 'showstart' command. To delete a job use the 'qdel' command.

Debugging

The totalview debugger is not available on Darwin but the DDD visual debugger (which is also on HPCx) is available. To use this you will need to run p-TOMCAT on the login nodes. There are no interactive batch queues available at the time of writing. To run on the login nodes, reduce the no. of processes for p-TOMCAT to 4 or 2 by setting nproci and nprock and run the DDD debugger. If you need help with DDD please ask Glenn.