Release Notes : dCache 1.6.5

This improvement covers the cases, where the host of the gridftp door is equipped with 2 network interfaces. One connected to the network, the clients are connecting from, and the other interface is attached to the network of the pool nodes. Both networks may or may not be connected. One of the grid ftp transfer modes is using a so called door transfer adapter, running on the door node, assuming that the client and pool data channels connect to this adapter. In case of the client, the door sends its hostname attached to the network interface which has been used by the clients control connection. With this improved version of the door, the pool can be instructed to connect to a configurable interface which will be the interface of the subnet the pool nodes are located.

The option specifying the internal interface is

       -ftp-adapter-internal-interface=<Ip Address Of Internal Interface>
  

and has to be give in the batch 'create' command of the Ftp door. If the option is omitted or pointing to an empty string, the internal interface will be the interface attached to the hostname.

Example

   create dmg.cells.services.login.LoginManager GFTP \
               "2811 \
                diskCacheV111.doors.GsiFtpDoorV1 \
                -export \
                -prot=raw \
                -clientDataPortRange=${clientDataPortRange} \
                -ftp-adapter-internal-interface=192.168.100.4 \
                -root=${ftpBase} \
                -kpwd-file=${kpwdFile} \
                -maxLogin=100 \
                -brokerUpdateTime=5 \
                -protocolFamily=gsiftp \
                -loginBroker=LoginBroker \
                -poolManagerTimeout=5400 \
                -pnfsTimeout=120 \
                -maxRetries=80 \
                -maxStreamsPerClient=10"

   

With this improved version, the dCache ftp server may run on hosts with more than one interface connected to client subnets. From the raw ftp protocol view, no configuration would be necessary. In the case of GridFtp using host certificates, the situation is slightly different. Because each interface will have a different DNS name, the related host certificates will be different as well. So the administrator has to specify the path to the actual certificates, belonging to the related network interfaces.

Example

Door node with two interfaces, bush.desy.de(131.169.40.100) and clinton.desy.de(131.169.41.101) and host certs/keys in /etc/grid-security/bush.desy.de-cert.pem(-key.pem) and same for clinton.

  
create dmg.cells.services.login.LoginManager GFTP-clinton \
            "2811 \
             diskCacheV111.doors.GsiFtpDoorV1 \
             -export \
             -prot=raw \
             -listen=131.169.41.101   \
             -service-cert=/etc/grid-security/clinton.desy.de-cert.pem \
             -service-key=/etc/grid-security/clinton.desy.de-key.pem \
             -clientDataPortRange=${clientDataPortRange} \
             -kpwd-file=${kpwdFile} \
                  [OTHER OPTION]"
                  
create dmg.cells.services.login.LoginManager GFTP-bush \
            "2811 \
             diskCacheV111.doors.GsiFtpDoorV1 \
             -export \
             -prot=raw \
             -listen=131.169.40.100   \
             -service-cert=/etc/grid-security/bush.desy.de-cert.pem \
             -service-key=/etc/grid-security/bush.desy.de-key.pem \
             -clientDataPortRange=${clientDataPortRange} \
             -kpwd-file=${kpwdFile} \
                  [OTHER OPTION]"
                  
  

So, two different doors are listening on port 2811 but one is attached to the clinton.desy.de interface while the other is connected to the bush.desy.de interface. Both have the path to the corresponding cert/key file specified.

Ftp transfer are now visible on the Transfer web page as well. No additional configuration is needed.

Example

       create dmg.cells.services.login.LoginManager GFTP \
                   "2811 \
                    -export \
                    diskCacheV111.doors.GsiFtpDoorV1 \
                    -prot=raw \
                    -clientDataPortRange=${clientDataPortRange} \
                    -root=${ftpBase} \
                    -kpwd-file=${kpwdFile} \
                    -maxLogin=100 \
                    -brokerUpdateTime=5 \
                    -protocolFamily=gsiftp \
                    -loginBroker=LoginBroker \
                    -poolManagerTimeout=5400 \
                    -pnfsTimeout=120 \
                    -maxRetries=80 \
                    -maxStreamsPerClient=10"
  

Ftp server stucks when attempting to write files exceeding 2 GBytes in ftp block mode. Problem is fixed. There is a misconfiguration which looks nearly the same : make sure the PnfsManager is configured to store meta data of large files.

PnfsManager2 with (p)nfs2

         -enableLargeFileSimulation
      

PnfsManagerV3 with (p)nfs2

         -enableLargeFileSimulation
      

PnfsManagerV3 with (p)nfs3 or Chimera

         -storeFilesize
      

Eventually, pools started to complain about invalid syntax in remove filespec.. This was mainly observed after pools have been very busy. Although, with this fix, the error message doesn't show up, the basic problem still exists : If a pool is too busy to reply on a 'removeFile' message in time, the Cleaner assumes that the file has not been removed. This is insofar not a big deal, that the cleaner will retry after the 'recover' period.

For systems where pools are always very busy, it is now possible to set the timeout for the pool reply on Cleaner startup. The option is -poolTimeout=<poolTimeout_in_seconds>.

In cases where the RemoteGsiftpTransferProtocol_1 mover failed or had been interrupted the pool space module started to miscalcuate the amount of available space. Fixed now.

In cases where the RemoteGsiftpTransferProtocol_1 mover failed or had been interrupted files stayed LOCKED and in mode : Receiving from client. Bug is fixed. For all Exceptions, the file will be unlocked and will become precious.

HSM activities can be excluded from cost calculation. In some cases this becomes necessary to have movers equally distributed despite of high and unbalanced HSM activities.

This somehow solves the following problem : The dCache doesn't know which of the external processes, it lauched, are busy and which are just waiting. This is especially problematic with HSM copy commands like encp/osmcp/hpsscp. Most of the time those programs don't do anything but waiting for a drive (= no cpu), after they got their drive assigned, they us as much bandwidth resp. CPU as they can get. The suppress cost solution works fine so far, because most sites don't have too many tape drives, so that not too many hsmcp commands are actually active. Most of them are in wait for drive state.

Syntax

pool suppress hsmload on | off

What's the magic flag ?

The Central Cost Module, located in the PoolManager, holds information about the space and cpu cost of each individual dCache pool. This information is locally cached and updated every 30 to 120 seconds by the pools themselfs. During that interval the cached cost information is becoming more and more incorrect. As a consequence, requests, coming-in in bunches, are basically not distrubuted equally among the pools. To overcome this deficiency, the cost cache, within the CostModule, monitors the requests to the PoolManager, asking for an appropriate pool. As soon as the PoolManager replies with a pool candidate, the cache is updated with this information. This is insofar not correct because it does not necessarily mean that the request is really going to the requested pool. But up to now, all modules, making this request, subsequently send the request to the suggested pool. Even if they wouldn't, the cache information is updated after 30 - 120 seconds by the real values. Whether or not this 'predictive' feature is enabled is determined by the cm set magic ... command of the CostModule within the PoolManager.

What changed

Up to now, the default value of the magic flag was off. In addition, and this can be regarded as a bug, the magic flag value was NOT stored in the PoolManager.conf file on save. Consequently the setup got lost after a pool restarted. This has now changed :

• The default value for the magic flag is on.
• The state of the magic flag is displayed with cm info
• The state of the magic flag is stored into the PoolManager.conf on 'save'.

Syntax (PoolManager)

cm set magic on | off # default : on