Centre for Atmospheric Science
University of Cambridge Home


p-TOMCAT
A massively parallel tropospheric chemistry transport model

How to use p-TOMCAT on HECToR

Hardware

HECToR is a Cray XT4 massively parallel system. The processors are about the same performance as the Dobson cluster machines. There is a very fast network - faster than dobson or HPCx. Tests show that p-TOMCAT does not suffer as much slowdown at high resolutions due to the improved network on HECToR.

You can log in to HECToR (using ssh only) at login.hector.ac.uk, which is the front-end machine of the cluster: but you cannot log into the other compute nodes. The /home directory is only mounted on the login node, it cannot be seen on the compute/batch nodes. Any data that needs to be accessed by the batch jobs must be copied onto your /work directory.

An important point to note is that the cluster machines are based on Intel processors. Therefore you MUST recompile your programs before you can run them on Hector. Also, binary format on these machines is little-endian rather than big-endian on the other servers. If you write binary files from your Fortran programs on rome (say), you need to add the appropriate compiler option to read them correctly on HECToR.

Setting up your environment

In order to run p-TOMCAT and compile it correctly you need to set up your environment. In your .profile file in your home directory, add the following lines at the bottom:

# make sure we load the right modules
module unload netcdf
module unload PrgEnv-pgi
module load   PrgEnv-pathscale
module load   xtpe-quadcore
# adjust paths to get my pathscale version of netcdf
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/n02/n02/emgdc/netcdf
export LD_LIBRARY_PATH
PATH=$PATH:/home/n02/n02/emgdc/bin; export PATH

p-TOMCAT doesn't work correctly with the default PGI compiller so the module commands above switch to the PathScale compiler. Also, the netcdf module supplied on the system does not work with p-TOMCAT as it uses the PGI compiler. A version compiled with the PathScale compiler is available in my home directory and the above sets access to it.

 

p-TOMCAT files

All the files required to run p-TOMCAT have been transferred to HECToR and are in the directories under /work/n02/n02/emgdc/tomcat/data. The sample HECToR tom.run script has the correct directory names in the namelist for the model.

The forcing files are in /work/n02/n02/emgdc/data/ECMWF. We have all the forcing files from 1995- present day. I will keep the files up to date. If anyone notices any missing or wants any earlier years please let me know.

 

Submitting jobs

Jobs are submitted on HECToR using the qsub command (as on Dobson). qstat is used to check the status of jobs.

 

More information

For more information, check the user guide on the HECToR website.