6th dCache Workshop - Installing dCache

Aus Gkswiki

This article illustrates how to install a basic dCache system on one single machine. We will concentrate on dCache of version 2.2, as it will be the anticipated next (the third) Golden Release, that is supposed to be supported for at least whole 2013. The current Golden Release is 1.9.12, which will be supported until end of 2012. There are only minor differences regarding the installation procedure and configuration mechanics, but an accurate installation guide for dCache 1.9.12 can be found as part of the GridKa School 2011: Installing dCache 1.9.12.

Furthermore does this article not go into details about terminology or concepts. Please read the Introduction to dCache to learn about them.

Prerequisites

Operating system

The procedure described in this manual is based on Scientific Linux 6.2, but should be applicabel to any Linux derivative.

Virtual Machines

For this workshop we will utilize virtual machines on a cluster farm in Zeuthen. Everyone of you should have received personal connection instructions at registration.

Java

dCache is completely written in Java. In order to run dCache, all machines must have installed the Java Development Kit in version 1.6 or higher.

Using a non-Orcale-JRE should be possible, but it has not been widely tested.

Database software

For few services dCache uses databases – PNFS/Chimera and SRM are the most important ones – for some others databases are only optional. dCache.org suggests to use the open source software PostgreSQL - here version 8.3 or higher is recommended.

Preparations

At this point this manual assumes you have a machine that meets all prerequisites mentioned before.

From now on we will edit flat files several times. Therefore we advise to use the editor vi. Start it by stating

vi <file to edit>

on the command line. If a file is not existing, it will be created, otherwise opened. To toggle between vi's read and edit mode press 'i' or 'ESC' respectively. You can find a word quickly by typing '/' directly followed by the string to search for (e.g. "/SpaceManager" will match srmSpaceManagerEnabled) - the capitalization matters! After you have edited the file you can save the changes with ':w' and quit with ':q' or both in one go with ':x' (you must be in read-mode for this!).

PostgreSQL

PostgreSQL must be installed on every server, that hosts a dCache service with database backend (although there are possibilities to set up the database apart of the dCache domain, that uses it). At least the following rpms of PostgreSQL are needed:

• postgresql
• postgresql-server
• postgresql-libs

After one has installed these rpms (participants of the dCache administration workshop received an ready-made image with preinstalled PostgreSQL), the database has to be initialized, configured and started, before dCache can be installed.

1. Initialize postgres with

   service postgresql initdb

2. Open /var/lib/pgsql/data/pg_hba.conf and change following lines near the end of the file

   vi /var/lib/pgsql/data/pg_hba.conf

   # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all  ::1/128 trust

3. Since we will be running all dCache services on one node, it is very likely, that we will reach the limit for maximum number of connections to the Postgres databases. Therefore, we will have to increase the default value.

   vi /var/lib/pgsql/data/postgresql.conf

   (…) max_connections = 1000 # (change requires restart) (…)

4. Start the postgresql server

   service postgresql start

5. One might find it usefull to let postgres start automatically on reboots (which is needed definitely, if you want to make dCache start on boot-up, too).

   chkconfig postgresql on

Installation

According to the purpose each node has to fulfill, not every dCache component has to be installed on each node. The installation packages can be found at dCache.org, in some cases you can also use yum repositories – specifically in the case of dcap client packages. However, during this dCache workshop we will install the whole dCache software on a sole (virtual) machine. If you are working on the offered ready-made VMs you can find all required rpm-packages in the directory /root/data and the dcap-client packages are referenced by the epel.repo.

dCache Server

This package includes all the dCache components that dCache consists of.

yum localinstall /tmp/dcache-2.2.0-1.noarch.rpm

dCache Clients

The client packages are needed on all nodes that are meant to offer access to dCache setups.

Storage Resource Manager (SRM)

yum install dcache-srmclient

dCache's nativ transfer protocol (dcap)

yum install dcap dcap-libs

Configuration

PostgreSQL databases

Users chimera and srmdcache

After PostgreSQL is installed, the databases are owned by the newly added user postgres. But dCache will connect to the databases as yet another user which depends on the service. As these users need other rights for handling PostgreSQL databases than a normal unix user, we have to create them with the id of the user postgres with PostgreSQL tools (the option -U allows to execute commands with a different user account).

createuser -U postgres --no-superuser --no-createrole --createdb chimera

Chimera

The first database we need to create is for Chimera, the namespace provider of our choice in this workshop.

createdb -U postgres chimera
psql -f /usr/share/dcache/chimera/sql/create.sql -U chimera chimera
createlang -U postgres plpgsql chimera
psql -f /usr/share/dcache/chimera/sql/pgsql-procedures.sql -U chimera chimera

SRM

We will stay with the historical default name for the PostgreSQL database that is used by the (optional) SRM service and create the database called dcache. Per default dCache will connect to it as user srmdcache.

createuser -U postgres --no-superuser --no-createrole --createdb srmdcache
createdb -U srmdcache dcache

Configuration files

We make use of flat files, to define and change the settings for the layout and the behaviour of dCache on different levels (per host, per domain or per cell). There are three main places for the configuration files:

• /usr/share/dcache/defaults

This directory is filled with files defining the default settings for all dCache services, as they are shipped by dCache.org. Do not modify these files, as they will be replaced by subsequent updates!

• /etc/dcache/dcache.conf

The central configuration file, that ideally should be nearly identical on all nodes of the dCache setup. Maybe it is possible to have only one difference among all nodes at all: the parameter pointing to the layout configuration files. To get to know what settings can be made in dcache.conf you can look through dcache.properties in the defaults directory.

• /etc/dcache/layouts

Layout files are the place to define the actual topology for the dCache services/domains on this node. So typically no layout file of one node is identical to any other layout file of another node in the setup.

dCache.org provides us with premade layout files that state a possible distribution of services over domains: head.conf, pool.conf and single.conf. Right now you could start dCache and it would use the empty dcache.conf file and the fallback layout file single.conf. With this the most important core services will be configured to run in one single domain with default settings. Alternatively, head.conf has predefined the mandatory services in a decent number of domains to be run on the headnode of your setup. Of course, you will need at least one other node using pool.conf to provide some disk space to dCache.

dCache layout

We create our own layout file mylayout.conf with a reasonable distribution of services over some domains. First we have to instruct dCache that it should use our own layout file by means of the dcache.conf file.

vi /etc/dcache/dcache.conf

dcache.layout = mylayout

Create the layoutfile. Note that the file name has the suffix .conf, which is not stated in dcache.conf!

touch /etc/dcache/layouts/mylayout.conf

Services and domains

Secondly, we define our own system for dCache domains and what services they are to host.

vi /etc/dcache/layouts/mylayout.conf

# Define one domain called "dCacheDomain".
[dCacheDomain]
# It will run these cells.
[dCacheDomain/poolmanager]
[dCacheDomain/broadcast]
[dCacheDomain/loginbroker]
[dCacheDomain/topo]

# Likewise for some other domains.
[namespaceDomain]
[namespaceDomain/pnfsmanager]
[namespaceDomain/cleaner]
[namespaceDomain/dir]

[httpdDomain]
[httpdDomain/billing]
[httpdDomain/srm-loginbroker]
[httpdDomain/httpd]
[httpdDomain/info]

[utilityDomain]
[utilityDomain/pinmanager]
[utilityDomain/statistics]

[doorsDomain]
[doorsDomain/dcap]
[doorsDomain/gsidcap]
[doorsDomain/gridftp]

# [nfsDomain]
# Use nfsv3 if your kernel is not ready to support nfsv41.
# [nfsDomain/nfsv3]
# Replaces nfsv3 - it is not possible to run both at the same time!
# [nfsDomain/nfsv41]

# [srmDomain]
# [srmDomain/srm]
# [srmDomain/gplazma]
# [srmDomain/spacemanager]

# [statisticsDomain]
# [statisticsDomain/statistics]

# [adminDoorDomain]
# [adminDoorDomain/admin]

# [webadminDomain]
# [webadminDomain/webadmin]

# [replicaDomain]
# [replicaDomain/replica]

# [transfermanagersDomain]
# [transfermanagersDomain/transfermanagers]

# example for doors ran in their own domain
# [dcap-${host.name}Domain]
# [dcap-${host.name}Domain/dcap]

# [xrootd-${host.name}Domain]
# [xrootd-${host.name}Domain/xrootd]

# [gridftp-${host.name}Domain]
# [gridftp-${host.name}Domain/gridftp]

# [webdav-${host.name}Domain]
# [webdav-${host.name}Domain/webdav]

# [gsidcap-${host.name}Domain]
# [gsidcap-${host.name}Domain/gsidcap]

With this we have defined five domains each with a set of services. Administrators are completely free for decisions about the number (and the name) of domains (except that there must be a dCacheDomain on any dCache node, even if it does not host any services) and which services they manage. Documentation about what services do exist and what further settings are available respectively can be looked up in the defaults directory mentioned before.

Pools

All data managed by dCache is located in pools. Since pools behave like all other dCache services, they are defined in the same way in the layout file.

vi /etc/dcache/layouts/mylayout.conf

(…)
# Beginn a new domain for pools, whereas the hostname is used for
# constructing it's name.
[${host.name}Domain]
[${host.name}Domain/pool]
  # name and path are the only required properties.
  name = pool1
  # path must be existing, it will not be created by dCache!
  path = /dcache/${name}
  # dCache will delay the start of this pool until the data directory
  # is accessable.
  waitForFiles = ${path}/data
  # dCache will not store more than maxDiskSpace of data in this pool.
  maxDiskSpace = 5G
# Note that we define the second pool with identical settings, except
# for the name.
[${host.name}Domain/pool]
  name = pool2
  path = /dcache/${name}
  waitForFiles = ${path}/data
  maxDiskSpace = 5G

We configured two pools, hosted by the same domain and made some specific settings per pool. Note the way we defined the name for the domain using the hostname. This way we will have a specialised result for each host without modifying the layout file. A similar feature applies for definition of the name for each pool and its path. Make use of this feature whenever possible in order to keep the layout file less error prone.

dCache expects the pool's directories to exist; it will not create them, when they are missing. Additionally, we have to keep in mind, that dCache will run under user dcache instead of root. Thus first we create the directories and make them over to user dcache afterwards.

mkdir -p /dcache/pool{1,2}/data
chown -R dcache:dcache /dcache

Java home

dCache uses Java to parse the configuration files and will search for Java on the system path first; if it is found there, no further action is needed. If Java is not on the system path, the environment variable JAVA_HOME defines the location of the Java installation directory. Alternatively, the environment variable JAVA can be used to point to the Java executable directly. There are, however, situations that require a different solution, e.g. when one cannot or does not want to include JAVA_HOME/JAVA in the global environment variables in the operating system (for example if several versions of Java are installed and the overall configuration should not get broken by this), then they can be defined in either /etc/default/dcache or /etc/dcache.env. These two files are sourced by the init script and allow JAVA_HOME, JAVA and DCACHE_HOME to be defined.

vi /etc/dcache.env

JAVA_HOME=/usr/java/latest
JAVA=/usr/java/latest/bin/java

Note

The Java installation that comes with Scientific Linux is configured in the alternatives management and is pretty outdated (java version "1.4.2"). In our virtual machines, we already updated Java and the alternatives settings, so /etc/dcache.env in principle is not needed.

Java heap size

Technically, each domain is an instanciated Java virtual machine. Running too many domains with default settings for Java will quickly consume all the available memory. Therefore we may set limits for the memory usage by Java. Thanks to the hierarchical structure of the layout file, we have fine grained control over which domain may use how much memory.

vi /etc/dcache/layouts/mylayout.conf

# Default settings applying to all domains on this host.
dcache.java.memory.heap = 256m
dcache.java.memory.direct = 128m

# Define one domain called "dCacheDomain".
[dCacheDomain]
  # Grant the dCacheDomain more maximal memory.
  dcache.java.memory.heap = 512m
# It will run these cells.
[dCacheDomain/poolmanager]
[dCacheDomain/broadcast]
[dCacheDomain/loginbroker]
[dCacheDomain/topo]
(…)

Some properties may even be used for services, so one level deeper than domain-level.

(Finally) Starting dCache

Check configuration

Before starting dCache, we will first check if the configuration is syntactically correct.

dcache check-config

Start dCache

We have performed all necessary steps to install dCache now. You may start it anytime with

dcache start

Validation

Export of namespace file system

The namespace service of dCache Chimera internally manages a global file system hierarchy that includes all files of dCache. We as a human beeing like to see and interact with this file system with (almost all) standard, well-known POSIX tools. For this we need to run a dCache service, that manages an internal NFS server, who's namespace than can get mounted in the root file system hierarchy. dCache offers two different services for this task: nfsv3, following the NFS3 standards, or nfsv41, implementing the modern NFS4.1 protocol.

Once again, edit the layout file.

vi /etc/dcache/layouts/mylayout.conf

(…)
[nfsDomain]
# Use nfsv3 if your kernel is not ready to support nfsv41.
# [nfsDomain/nfsv3]
# Replaces nfsv3 - it is not possible to run both at the same time!
[nfsDomain/nfsv41]
(…)

Edit /etc/exports, which defines which hosts are allowed to mount the namespace.

vi /etc/exports

/ localhost(rw)
/pnfs localhost(rw)

Now localhost is allowed to mount the entire namespace tree and all hosts from the domain dcache.org may mount the /pnfs branch of it.

As any RPC service Chimera NFS requires rpcbind service to run on the host. Nevertheless rpcbind has to be configured to accept requests from Chimera NFS.

vi /etc/sysconfig/rpcbind

RPCBIND_ARGS="-i"

service rpcbind restart
sleep 20

Restart the Domain hosting the NFS service and try to mount the namespace locally.

dcache start nfsDomain
mount -o nfsvers=4,minorversion=1 localhost:/ /mnt
# If using nfsv3:
# mount -o nfsvers=3 localhost:/ /mnt
mkdir -p /mnt/pnfs/dcache.org/data
umount /mnt
mkdir /pnfs
mount -o nfsvers=4,minorversion=1 localhost:/pnfs /pnfs
# If using nfsv3:
# mount -o nfsvers=3 localhost:/pnfs /pnfs

File transfer into dCache

There are several clients useable to write into or read from dCache. In this lecture we only demonstrate the native dCache file transfer protocol dcap. As access with this protocol is completely unauthenticated (all accesses will be mapped to the user nobody) and our dCache setup has no well defined authentication and authorization yet – more about access control later – we have to set a directory world-write- and readable:

mkdir -m 777 /pnfs/dcache.org/data/worldwritable

Of course normally not root is writing to dCache but any other user. Switch back to your regular user account and try to write a file to dCache with dccp.

dccp /bin/bash dcap://$(hostname -f):22125/pnfs/dcache.org/data/worldwritable/dccp.tst

We just gave dccp a destination URL as target definition. Note the same structure like with any url:

<protocol>://<host name>.<domain name>:<port number>/<path to your file>

The port number for DCAP has been standardised by IANA and needs not to be specified if it is left at the default (22125).

This way we can also read our just introduced test file:

dccp dcap://$(hostname -f)/pnfs/dcache.org/data/worldwritable/dccp.tst /tmp/dccp.tst

Make sure the original file and the file read from dCache are not differing (no response on screen is to be expected):

diff -q /bin/bash /tmp/dccp.tst

It may prove interesting to see what happens in background when using dccp. Repeat the read commandline supplemented with the debug option "-d n", where n specifies the debug level.

dccp -d 10 dcap://$(hostname -f)/pnfs/dcache.org/data/worldwritable/dccp.tst /tmp/dccpdebug.tst 2>&1 | less -S

The Admin Interface

The core mechanism for administrating a dCache instance is the admin interface. This is a service you may connect to using a ssh client. Using the admin interface, you may communicate with the various components making up dCache, query their status and update their behavior. Although dCache is a distributed system, you only ever connect to a single node; dCache will route your messages internally.

After you have switched back to the superuser, we need to configure dCache to run the admin service.

vi /etc/dcache/layouts/mylayout.conf

(…)
[utilityDomain]
[utilityDomain/admin]
[utilityDomain/pinmanager]
(…)

Since we use ssh as communication tool, some keys need to be generated.

ssh-keygen -t dsa -f /etc/dcache/admin/ssh_host_dsa_key -N ""

Restart the utilityDomain.

dcache restart utilityDomain

Once dCache finished starting the service, we can connect to the administration interface with following command.

ssh -1 -c blowfish -p 22223 admin@localhost
  <passphrase is "dickerelch">

Every running dCache service has its own cell, which can be visited in the admin interface. Now we want to see the complete repository of our dCache pool that we used for our dccp transfer:

(local) admin > cd pool1
(pool1) admin >

Now you could view all status information with the command "info" or view all known commands with "help". For our purpose we just execute "rep ls" for the pool's repository:

(pool1) admin > rep ls

Up till now dCache has no guidelines on where to store which kind of data (this will be explained to you later), so our testfile may have been written in both pools we created so far. Thus if you do not get any output from "rep ls", try it again on the cell for the other pool. But before you can change to the other cell, you first have to leave the cell of pool1 by typing in "..":

(pool1) admin > ..
(local) admin > cd pool2
(pool2) admin > rep ls

In order to log out you have to return to "local" (with "..") and then you can exit the admin interface with "logoff". Alternatively you can log out directly by pressing CTRL+d.

(pool1) admin > ..
(local) admin > logoff

dCache's webservice

One of the quickest methods of obtaining simple information about a dCache instance is through the web interface. It provides read-only access to the dCache setup and offers information such as the pool occupancy or which components are running.

The web interface is provided on port 2288. To see the output, point your favourite web browser towards "http://<FQDN of your headnode>:2288/". In our case, using the virtual machine, you may use the adress http://<your vm name>:2288.

Statistics module

On the web interface you might notice the red text "Module not installed" next to the "statistics" header. In order to activate it, one must first add the service statistics to the layout file. After the service has been started the statistics for the dCache setup will be collected. But to make them available via the web interface we need to modify the website's source file. Open /usr/share/dcache/httpd/static/index.html for writing and modify the following passages.

vi /usr/share/dcache/httpd/static/index.html

<h1>Statistics</h1> <span style="color:red;">Module not installed"</span>

<h1>Statistics</h1>

Now dCache needs some time to build the respective webpages, which is done in predefined intervals. Normally, the statistics are available under the web interface for the first time after two days, when dCache has completed a full cycle for generating the html files. Consequently, you will not be able to see the statistics of dCache in this workshop. :-)

Back to the timetable.