User environment

Chemfarm is a multiuser machine used for different kinds of scientific computations. For this reason, it has various general and dedicated software available for use. Different programs, however, require different software components called libraries (think of them as building blocks). In addition, our cluster allows you not only run jobs but also compile code from your collaborators and colleagues, and even develop your own code. In order to make efficient use of all the software and tools, a user must have some grasp of a concept commonly referred to as "user environment".

A "user environment" consists of several layers/parts. We will be focusing on a few that are crucial. The user is encouraged to seek out more of these. The first and closest to the user is shell configuration. Another very similar part which is not found on desktop user environments is environmental modules. Also, we can consider the physical storage as belonging to the user environment. On Chemfarm we have two storage spaces - each of them with different properties and limitations. Understanding these three parts may greatly assist you in your work. In addition, there is a configuration for SSH (Secure Shell) which is used for sending jobs and logging into compute nodes, but normally only minimal interaction with that part will be needed.

NOTE: Changing your environment without deep understanding of what you are doing may completely destroy it, prevent your access to Chemfarm, and cause data loss!

Shell configuration

As you probably know, work in Linux is usually performed from the command line. This command line interface itself is program called a "shell". Within the scientific computations community, there are two popular shells: bash (Bourne Again SHell) and tcsh (a Unix shell based on and compatible with the C shell). Similar to many Linux programs any shell has configuration files. Bash uses two hidden files: ".bash_profile" and ".bashrc" (Note: a hidden file name starts from "."symbol, and to see it you must use 'ls -a' command). Tcsh configuration is kept in ".cshrc" and ".login" files.

When you actually login to any Linux machine, either ".bashrc" or ".login" files are sourced. When you connect to a Linux machine without logging in, e.g. when your job starts running, either ".bash_profile" or ".cshrc" are sourced. Note: bash users - it is standard practice to load the .bashrc configurations from the .bash_profile so that the two shell configurations are as close as possible.

For every new user, these files contain minimal configuration required for a normal work of the shells. You can edit and add more functionality to your shell, such as aliases or constantly used modules, but remember two important points:

  1. Make backups of your current configuration files so that changes can be reverted.
  2. Always keep an extra console window open with a working shell, since shell may become unusable by adding erroneous lines to its configuration. This extra shell can be used to revert using the backups you prepared in (1).

Environment Modules

The most appropriate and safe way of changing the environment is using modules. Simply put, these are pre-packaged and easily revertible changes to the user shells that can be modularly added and removed. We use the TACC Lmod program for modules. Modules provided by Chemfarm are tested and will not destroy your login settings. Moreover, the dependencies between modules prevent from using incompatible software and libraries. There is a set of modules loaded by default on the login node and on all the compute nodes. You can change this list by either loading the modules in the shell configuration file or in the job script.

To display list of currently loaded modules type:

module list

in the command prompt. To see all the options just type

module avail

Certain research groups provide their own modules, ask your peers if such are available.

Storage

User accessible storage on Chemfarm is located on GPFS (General Parallel File System).

The storage system on Chemfarm is parallel, designed to allow the compute nodes to all have access to the same files. Naturally, it is very different from a simple hard-drive and the user is not expected to need to configure it. However, the configuration that is currently set-up is important to understand:

Our file system is divided to two areas. These areas differ by backup policy and by quota.

  1. /home – The home partition contains the folder to which you log in. We purposely enforce a rather small personal quota (about 100GB) on your "home" folder. This area is dedicated for keeping your important data, such as scripts, programs, and processed results. A daily backup of the "home" area is performed. Thus, if you accidently 'rm' something that you cannot re-create, there still might be a possibility to retrieve it from backup if it has been a day since it was created. Note: that this process takes a while so do not rely on it, use it only for emergencies.
  2. /work – this is a large space dedicated for the daily work. The quota applied here is enforced on a group basis, meaning that it is shared between all group members. The group quotas are pretty large (several TB per group). This partition was designed so you can use it for raw data and storage of intermediate results. ATTENTION: There is no backup for "work". The user is responsible of moving important results to the "home" folder or to other storage services.

Note that there is a quota not only for the space your files take, but also for the number of files you are allowed to keep. To find the amount of space and files you use in all the areas use the following commands:

mmlsquota -u $USER

to see your personal usage. Or:

mmlsquota -g `id -gn`

to display usage of your group.

In addition to the central storage we have ~300GB "/scratch" partition on every compute node. There is no quota on "/scratch", however the data in scratch in absolutely unprotected. It should be used only for temporary files during the job execution of I/O demanding jobs. If your software uses the scratch, please make sure to delete whatever it created there before the job ends.

SSH

SSH is probably the most convenient way to connect to a Linux machine remotely. Usually, connecting via SSH requires a username and password. However, since Chemfarm is a cluster (and as we already mentioned a cluster is a cost-effective alternative to a "real supercomputer") a passwordless SSH between all the nodes is already preconfigured for all users. This configuration is placed in ".ssh" folder in your home directory. Some users prefer to also configure passwordless access from their own Linux or Mac machine. Take care to do it without damaging the existing configuration.

NOTE: Deleting or damaging files in the ".ssh" directory will prevent you from running jobs! Never modify this directory without deep understanding of what you are doing.