Installing a Basic dCache Instance

Installing a Basic dCache Instance
Prev	Chapter 2. Installing dCache	Next

In the following the installation of a central admin node of a dCache instance and of an arbitrary number of dCache nodes will be described. These nodes may each contain several dCache pools and optionally one SRM, one GridFTP door, and/or one GSIdCap door. On the admin node, a pnfs server and several central dCache components are installed. The pnfs server, some central components, and each SRM need an PostgreSQL server installed locally on the node. The first section describes the configuration of a PostgreSQL server. After that the installation of the pnfs server and of the dCache components will follow. During the whole installation process root access is required.

[return to top]

Prerequisites

In order to install dCache the following requirements must be met:

An RPM-based Linux distribution is required.
dCache 1.8 requires Java 1.5 or 1.6 SDK. Previous releases of dCache can use Java 1.4.2. It is recommended to use the newest Java release available within the release series used.
PostgreSQL must be installed and running. See the section called “Installing a PostgreSQL Server” for more details. It is strongly recommended to use version 8 or higher.

[return to top]

Installation of the dCache Software

The RPM packages may be installed right away on each node, for example using the command:

[root] # rpm -ivh dcache-server-<version>-<release>.i386.rpm
[root] # rpm -ivh dcache-client-<version>-<release>.i386.rpm

The pnfs server software on the admin node can be installed with the command:

[root] # rpm -ivh pnfs-postgresql-<version>-<release>.i386.rpm

[return to top]

Readying the PostgreSQL server

You must configure PostgreSQL for use by dCache and create the necessary PostgreSQL user accounts and database structure. This section describes how to do this.

[return to top]

Configuring the PostgreSQL server

Using a PostgreSQL server with dCache places a number of requirements on the database. This section describes what configuration is necessary to ensure PostgreSQL operates so dCache can use it.

Restarting PostgreSQL

If you have edited PostgreSQL configuration files, you must restart PostgreSQL for those changes to take effect. On many systems, this can be done with the following command:

[root] # /etc/init.d/postgresql restart

[return to top]

Enabling `TCP` connections

When connecting to PostgreSQL, dCache will always use TCP connections. So, for dCache to use PostgreSQL, support for TCP sockets must be enabled.

In contrast to dCache, the PostgreSQL stand-alone client application psql can connect using either a TCP socket or via the local filesystem (a “UNIX” socket). Because of this, it is common for PostgreSQL to disable TCP sockets by default, requiring the admin to explicitly configure PostgreSQL so connecting via a TCP socket is supported.

To enable TCP sockets, edit the PostgreSQL configuration file postgresql.conf. This is often found in the /var/lib/pgsql/data, but may be located elsewhere. You should ensure that the line tcpip_socket is set to true; for example:

tcpip_socket = true

[return to top]

Enabling local trust

Perhaps the simplest configuration is to allow password-less access to the database and the following documentation assumes this is so.

To allow local users to access PostgreSQL without requiring a password, ensure the file pg_hba.conf, usually located in /var/lib/pgsql/data, contains the following lines.

local   all         all                        trust
host    all         all         127.0.0.1/32   trust
host    all         all         ::1/128        trust

Note

Please note it is also possible to run dCache with all PostgreSQL accounts requiring passwords.

[return to top]

Creating data users and structure

Prepare the PostgreSQL users and databases as they are needed for the components of dCache and/or the pnfs server: The pnfs server only needs a database user. We suggest to call it pnfsserver. Create it with:

[root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt pnfsserver

Several databases will be created by this user. At initial installation, as described below, two databases will be created: admin and data1. These databases will contain the information about the namespace of the pnfs filesystem. If the information in these databases is lost, the whole data in the dCache instance is not accessible anymore. Therefore, make sure these databases are backed up regularly and also stored on appropriately reliable hardware. Further advice may be found in Chapter 21, PostgreSQL and dCache.

The dCache components will access the database server with the user srmdcache which can be created with the createuser; for example:

[root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt srmdcache

Several central components running on the admin node as well as each SRM will use the database dcache for state information:

[root] # createdb -U srmdcache dcache

There might be several of these on several hosts. Each is used by the dCache components running on the respective host.

The pnfs companion uses the database companion to store the pools all files are located on. On the admin node create and initialize it with the command:

[root] # createdb -U srmdcache companion
[root] # psql -U srmdcache companion -f /opt/d-cache/etc/psql_install_companion.sql

(It has to be located on the same host as the pnfs server.)

If the resilience feature provided by the replica manager is used, the database “replicas” has to be prepared on the admin node with the command:

[root] # createdb -U srmdcache replicas
[root] # psql -U srmdcache replicas -f /opt/d-cache/etc/psql_install_replicas.sql

Note

Note that the disk space will at least be cut in half if the replica manager is used.

If the billing information should also be stored in a database (in addition to files) the database billing has to be created:

[root] # createdb -U srmdcache billing

However, we strongly advise against using the same database server for the pnfs server and the billing information. For how to configure the billing cell to write into this database, see below.

[return to top]

Installing the `pnfs` Server

The pnfs server software is installed in the directory /opt/pnfs/. For the installation copy the file /opt/pnfs/etc/pnfs_config.template to /opt/pnfs/etc/pnfs_config. The default should be suitable for most installations. It contains:

PNFS_INSTALL_DIR = /opt/pnfs
PNFS_ROOT = /pnfs
PNFS_DB = /opt/pnfsdb
PNFS_LOG = /var/log
PNFS_OVERWRITE = no
PNFS_PSQL_USER = pnfsserver

Next run /opt/pnfs/install/pnfs-install.sh. This will write the central configuration file /usr/etc/pnfsSetup and initialize the databases in the PostgreSQL server as well as configuration information below /opt/pnfsdb/ (as configured by PNFS_DB in /opt/pnfs/etc/pnfs_config). For example:

[root] # /opt/pnfs/install/pnfs-install.sh
PNFS_PSQL_USER = pnfsserver
 Checking nfs servers : Ok
      Preparing setup : Ok

 Creating database admin
 Creating database data1

 Starting pnfs server   ... Ok
 Trying to talk to dbserver 0 [1122] ... Ok
 Trying to talk to dbserver 1 [1122] ... Ok
             Trying to mount 'pnfs' : Ok
        Correcting pnfs permissions : Ok
 Detecting wormhole target (config) : 0000000000000000000010E0
                  Digging wormholes :  dig-0-ok dig-1-ok Done
             Creating database link : Ok
 Setting mount permissions to world : Ok

 Remarks :
   ii) Any host may now mount this pnfs server
         mount -o intr,rw,noac,hard  <thisServerName>:/pnfs /<mountdir>

Installation of PNFS completed - stop PNFS
 Stopping Heartbeat ....  Ready
 Killing pnfsd  Done
 Killing pmountd  Done
 Killing dbserver . Done
 Removing 8 Clients  0+ 1+ 2+ 3+ 4+ 5+ 6+ 7+
 Removing 8 Servers  0+ 1+ 2+ 3+ 4+ 5+ 6+ 7+
 Removing main switchboard ... O.K.

The pnfs server may now be started with /opt/pnfs/bin/pnfs start, for example:

[root] # /opt/pnfs/bin/pnfs start
Starting dcache services:  Shmcom : Installed 8 Clients and 8 Servers
Starting database server for admin (/opt/pnfsdb/pnfs/databases/admin) ... O.K.
Starting database server for data1 (/opt/pnfsdb/pnfs/databases/data1) ... O.K.
Waiting for dbservers to register ... Ready
Starting Mountd : pmountd
Starting nfsd : pnfsd

This script may be linked from /etc/init.d/. Please do not copy the init script as it may change between releases. On Red Hat derived Linux distributions the pnfs may be configured to start at boot time with

[root] # chkconfig --add pnfs
[root] # chkconfig pnfs on

It is advisable to create a basic directory structure in the pnfs namespace where each directory uses a separate pnfs database. This is still true for the PostgreSQL version of pnfs since the pnfs server uses global locks on each database. the section called “The Databases of pnfs” describes how it is done. WLCG sites should use at least two databases for each VO they support: One for the directory /pnfs/<domainName>/data/< voName>/ and one for the generated files, i.e. for /pnfs/<domainName>/data/< voName>/generated/.

[return to top]

Installing dCache Components

Use the templates of the configuration files found in /opt/d-cache/etc/ to create the following files.

The central configuration file of a dCache instance is /opt/d-cache/config/dCacheSetup. For most installations it is only necessary to set the variable java to the binary of the Java VM and the variable serviceLocatorHost to the hostname of the admin node. Note that the file has to go into the subdirectory config/ even though the template is found in etc/.

The installation and start-up scripts use the information in /opt/d-cache/etc/node_config. The variable NODE_TYPE controls whether the admin node should be installed or just pools and/or doors. Accordingly set it to “admin” or “pool” (for doors, as well). All other variables may be left at their default value.

For authorization of grid users the file /opt/d-cache/etc/dcache.kpwd is needed. Note that it may be generated from the standard /etc/grid-security/grid-mapfile with the tool grid-mapfile2dcache-kpwd which is distributed with the WLCG software.

How to proceed from here depends on whether the release to be installed is older than 1.8.0-14 or not. In particular init scripts and pool creation procedures have changed in dCache 1.8.0-14. Both procedures are described in the following sections.

Once dCache is installed, the section called “Using dCache as an LCG Storage Element” may be consulted about how to activate the info provider included since version 1.6.6-2 based on an earlier WLCG installation.

[return to top]

Prior to 1.8.0-14

Whether and how many pools should be installed on the current node is configured by /opt/d-cache/etc/pool_path. Each line in this file describes one pool. The format is as follows:

<poolDataDirectory> <poolSizeInGB> <reinstallPoolYesNo>

where <poolDataDirectory> is the full path to the directory which will contain the data files as well as some of the configuration of the pool, <poolSizeInGB> is the size of the pool. Make sure that there is always enough space under <poolDataDirectory>. Be aware that only pure data content is counted by dCache. Leave enough room for configuration files and filesystem overhead.

When all configuration files are prepared, configure the system with

[root] # /opt/d-cache/install/install.sh

[INFO]  No 'SERVER_ID' set in 'node_config'. Using SERVER_ID=<your.domain>.

[INFO]  Moving /opt/d-cache/bin/dcache-opt out of the way, because it is obsolete.

[INFO]  Creating link /pnfs/ftpBase --> /pnfs/fs which is used by the GridFTP door.
[INFO]  Creating link /pnfs/<your.domain> --> /pnfs/fs/usr/


[INFO]  Checking on a possibly existing dCache/PNFS configuration ...

[INFO]  Configuring pnfs export '/pnfsdoors' (needed from version 1.6.6 on)
        mountable by world.
[INFO]  You may restrict access to this export to the GridFTP doors which
        are not on the admin node. See the documentation.

[INFO]  Generating ssh keys:
Generating public/private rsa1 key pair.
Your identification has been saved in ./server_key.
Your public key has been saved in ./server_key.pub.
The key fingerprint is:
e3:4a:13:d5:33:45:e0:cd:69:a3:fb:d7:a8:64:df:73 root@grid-se3.desy.de


[INFO]  Creating Pool <hostname>_1
[INFO]  Creating Pool <hostname>_2

and start the central components (only on the admin node) with

[root] # /opt/d-cache/bin/dcache-core start
Starting dcache services:
Starting lmDomain  6 5 4 3 2 1 0 Done (pid=6802)
Starting dCacheDomain  6 5 4 3 2 1 0 Done (pid=6875)
Starting dirDomain  6 5 4 3 2 1 0 Done (pid=6973)
Starting doorDomain  6 5 4 3 2 1 0 Done (pid=7058)
Starting adminDoorDomain  6 5 4 3 2 1 0 Done (pid=7144)
Starting httpdDomain  6 5 4 3 2 1 0 Done (pid=7234)
Starting utilityDomain  6 5 4 3 2 1 0 Done (pid=7330)
Starting pnfsDomain  6 5 4 3 2 1 0 Done (pid=7436)
Starting gridftp-clintonDomain  6 5 4 3 2 1 0 Done (pid=7569)
Starting gsidcap-clintonDomain  6 5 4 3 2 1 0 Done (pid=7672)
Starting srm-clintonDomain  6 5 4 3 2 1 0 Done (pid=7777)

the configured pools are started with

[root] # /opt/d-cache/bin/dcache-pool start
Starting dcache pool: Starting clintonDomain  6 5 4 3 2 1 0 Done (pid=7990)

These scripts may be linked from /etc/init.d/. Please do not copy the init scripts as they may change between releases. On Red Hat derived Linux distributions dCache may be configured to start at boot-time using chkconfig, for example:

[root] # chkconfig --add dcache-core
[root] # chkconfig --add dcache-pool
[root] # chkconfig dcache-core on
[root] # chkconfig dcache-pool on

[return to top]

Since 1.8.0-14

A new init script was introduced in release 1.8.0-14. In addition to being able to start and stop dCache, it provides commands for creating and configuring pools. Thus pool creation is no longer part of the install script and pools can be created after the install script has been executed. We therefore proceed by finalising the initial configuration by executing /opt/d-cache/install/install.sh, for example:

[root] # /opt/d-cache/install/install.sh
INFO:Skipping ssh key generation

 Checking MasterSetup  ./config/dCacheSetup O.k.

   Sanning dCache batch files

    Processing adminDoor
    Processing chimera
    Processing dCache
    Processing dir
    Processing door
    Processing gPlazma
    Processing gridftpdoor
    Processing gsidcapdoor
    Processing httpd
    Processing info
    Processing infoProvider
    Processing lm
    Processing maintenance
    Processing pnfs
    Processing pool
    Processing replica
    Processing srm
    Processing statistics
    Processing utility
    Processing xrootdDoor


 Checking Users database .... Ok
 Checking Security       .... Ok
 Checking JVM ........ Ok
 Checking Cells ...... Ok
 dCacheVersion ....... Version production-1-8-0-14

No pools have been created on the node yet. Adding pools to a node is a two step process:

The directory layout of the pool is created and filled with a skeleton configuration using dcache pool create <poolSize> <poolDirectory>, where <poolDirectory> is the full path to the directory which will contain the data files as well as some of the configuration of the pool, and <poolSize> is the size of the pool, specified in bytes or with a M, G, or T suffix (for mibibytes, gibibytes and tibibytes, respectively).
Make sure that there is always enough space under <poolDirectory>. Be aware that only pure data content is counted by dCache. Leave enough room for configuration files and filesystem overhead.
Creating a pool does not modify the dCache configuration.
The pool is given a unique name and added to the dCache configuration using dcache pool add <poolName> <poolDirectory>, where <poolDirectory> is the directory in which the pool was created and <poolName> is a name for the pool. The name must be unique throughout the whole dCache installation, not just on the node.
Adding a pool to a configuration does not modify the pool or the data in it and can thus safely be undone or repeated.

An example may help to clarify the use of these commands:

[root] # /opt/d-cache/bin/dcache pool create 500G /q/pool1
Created a 500 GiB pool in /q/pool1. The pool cannot be used until it has
been added to a domain. Use 'pool add' to do so.

Please note that this script does not set the owner of the pool directory.
You may need to adjust it.
[root] # /opt/d-cache/bin/dcache pool add myFirstPool /q/pool1/

Added pool myFirstPool in /q/pool1 to dcache-vmDomain.

The pool will not be operational until the domain has been started. Use
'start dcache-vmDomain' to start the pool domain.
[user] $ /opt/d-cache/bin/dcache pool ls
Pool        Domain                       Size   Free Path
myFirstPool dcache-vmDomain               500    550 /q/pool1
Disk space is measured in GiB.

All configured components can now be starting with dcache start, for example:

[root] # /opt/d-cache/bin/dcache start
Starting lmDomain  Done (pid=7514)
Starting dCacheDomain  Done (pid=7574)
Starting pnfsDomain  Done (pid=7647)
Starting dirDomain  Done (pid=7709)
Starting adminDomain  Done (pid=7791)
Starting httpdDomain  Done (pid=7849)
Starting utilityDomain  Done (pid=7925)
Starting gPlazma-dcache-vmDomain  Done (pid=8002)
Starting infoProviderDomain  Done (pid=8081)
Starting dcap-dcache-vmDomain  Done (pid=8154)
Starting gridftp-dcache-vmDomain  Done (pid=8221)
Starting gsidcap-dcache-vmDomain  Done (pid=8296)
Starting dcache-vmDomain  Done (pid=8369)