1.9.12 Release Notes

dCache 1.9.12 focuses on:

Table of contents

Upgrade Instructions

Incompatibilities

Please consider the following changes when upgrading from a version before 1.9.12-1:

Compatibility

It is safe to mix pools of releases 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9.10, 1.9.11 and 1.9.12. Head nodes and doors must be upgraded to 1.9.12 together and cannot be mixed with head nodes or doors of releases before 1.9.12. Components of different 1.9.12 releases can be mixed freely.

Compatibility Matrix

We distinguish between pool components and head nodes. Any component which is not a pool is considered a head node, including doors. The following table shows compatibility between different releases.

1.9.12-3 Head1.9.12-3 Pool
Head 1.9.5nono
1.9.7nono
1.9.8nono
1.9.9nono
1.9.10nono
1.9.11nono
1.9.12yesyes
Pool 1.9.5yesyes
1.9.7yesyes
1.9.8yesyes
1.9.9yesyes
1.9.10yesyes
1.9.11yesyes
1.9.12yesyes

1.9.12-3

Service: info

The info service lists the network interfaces of each door in dCache. The DNS name of the interface is used as a key in the list. This causes problems when several interfaces share the same name and would often result in the host name being absent in the data published by the info provider. This problem has now been fixed by giving each network interface a unique name.

Service: httpd

Fixed a number of image files that were corrupted in the 1.9.12-2 release.

Service: pool

The lifetime of sticky bits are now reported in human readable format. The output of the 'rep sticky ls' command is affected.

Service: dcap

Minor logging related cleanups.

Service: nfsv3

Improved server side readdir caching.

Service: srm

Fixed processing of escape codes in SURLs and TURLs.

Out of memory handling

dCache triggers an automatic restart when a domain runs out of memory. We have observed that he restart sometimes fails, presumably because further out of memory errors are triggered during shutdown. A number of changes have been made to make the shutdown sequence more robust.

Since 1.9.12-1 dCache automatically generates heap dump files upon out of memory errors. A check has now been added to the startup script to check for the existing of these files and issue a warning.

Libraries

Several third party libraries have been upgraded. The upgrades represent minor releases that fix bugs.

RPM packaging

Ownership of the billing directory is now preserved accross upgrades.

Changelog 1.9.12-2 to 1.9.12-3

1.9.12-2

Service: srm

The following properties can now be specified separately for ls, put, get, bringonline and reserve requests: srmDatabaseEnabled, srmCleanPendingRequestsOnRestart, srmKeepRequestHistoryPeriod, srmExpiredRequestRemovalPeriod, srmRequestHistoryDatabaseEnabled, and srmStoreCompletedRequestsOnly. These properties replace the now deprecated properties srmJdbcMonitoringLogEnabled, srmNumberOfDaysInDatabaseHistory, srmOldRequestRemovalPeriodSeconds, srmJdbcSaveCompletedRequestsOnly, srmJdbcEnabled, and srmCleanPendingRequestsOnRestart.

Service: pool

Fix the processing of the waitForFiles property. The property did not have the inteded effect.

Migration script

Fix a typo that caused the script to fail in 1.9.12-1.

Added translation of SRM properties deprecated in 1.9.12-2.

Added support for moving admin authorization files to new location.

dCache now refuses to start if it detects a 1.9.5 configuration. This prevents that dCache is accidentally started before the configuration has been migrated.

Service: gplazma

Propagate the read-only flag back to doors. In recent versions the flag was ignored.

Several gPlazma 2 related changes were made, but gPlazma 2 is still under development and should not be used in this version. We expect gPlazma 2 to be production ready in 1.9.12-3.

Service: pnfsmanager

Reintroduce check that prevents existing checksums from being altered. The check was present in 1.9.5 but was lost in later versions.

Service: admin

SSH keys have moved from /opt/d-cache/config/ to /opt/d-cache/etc/. The migrate-from-1.9.5.sh script has been updated to move the files. However when upgrading from a version newer than 1.9.5 the SSH keys have to be moved by hand.

Location manager

Fixed a problem in which dCacheDomain would fail to start if /opt/d-cache/etc/lm.config could not be created. This would typically happen if dCacheDomain is started as a user different from root.

The location of lm.config is now configurable.

Service: httpd

Static content used by the monitoring service has been moved to /opt/d-cache/share/httpd/static/.

Service: webdav

WebDAV doors now report the DN to billing when files are deleted.

Changelog 1.9.12-1 to 1.9.12-2

1.9.12-1

Service: pool

The /opt/d-cache/bin/dcache pool commands have been updated to provide an ls subcommand to list configured pools. The create subcommand has been extended to add the new pool directly to the layout file. The command no longer creates the pool setup file and instead adds the pool specific configuration to the layout file. The command thus provides the functionallity of both the create and add commands found in some earlier releases.

The migration module was extended to reduce interference between multiple migration jobs running on the same pool. The migration module now respects the pool mode on the target pool.

Service: srm

The SRM used to have the ability to periodically submit vacuum commands to the database. This command is PostgreSQL specific and thus tied the SRM to PostgreSQL. PostgreSQL has for quite a while supported auto vacuuming, and manually executing the vacuum command is no longer necessary. We have therfore removed the submission of vacuum commands from dCache. The configuration parameters srmVacuum and srmVacuumPeriod are obsolete.

Some SRM operations may be processed synchronously or asynchronously, at the server's discretion. Since 1.9.10 dCache can start to process such requests synchronously and, if this is taking too long, reply asynchronously and continue to work on the operation in the background. In dCache 1.9.12 this behaviour is enabled by default for get, put, bring online, and list operations. The default switch over time is 1 second. The old behaviour can be restored by setting the srm*SwitchToAsynchronousModeDelay parameters.

The RemoteHttpTransferManager cell has been removed. The functionality has been merged into the RemoteGsiftpTransferManager cell, which has in turn been renamed to RemoteTransferManager. The SRM was extended to support third party srmCopy transfers from HTTP resources. Third party upload to HTTP resources and support for HTTPS is not available yet in the SRM.

Service: dcap, gsidcap, kerberosdcap

DCAP supports prestaging from tape. The DCAP door used to rely on a stager service to perform the staging. In dCache 1.9.12 the DCAP door instead uses the PinManager to trigger staging. The dummy-prestager service is obsolete and can be deleted from the layout.

Service: pnfsmanager

The PNFS manager has been restructured. One consquence is that the output of the info command in the cell admin interface has changed slightly: It is now grouped into sections in line with how it is done in pool manager and pools. Third party scripts may need to be adjusted.

Service: webdav

WebDAV doors now submit remove entries to billing when files are deleted.

The HTML rendering of a directory listing now includes a column containing the file size. The HTML is customizable through a template file. As a consequence the properties webdav.images.logo, webdav.images.directory, webdav.images.file, webdav.style.css, webdavLogoPath, webdavDirIconPath, webdavFileIconPath, webdavCssPath, dirIconPath, fileIconPath, and cssPath are obsolete. Please read the document embedded in the default template file /opt/d-cache/share/webdav/templates/list.stg for details on customizing the look and feel.

The property webdavAnonymousListing was introduced to prevent unauthorized users from listing directories.

The property webdavProtocol was extended with a third option, https-jglobus, to enable HTTPS support based on the JGlobus library. The advantages compared to the https option are that host and CA certificates are read and refreshed directly from /etc/grid-security/ and that proxy certificates (including VOMS certificates) can be used for client authentication.

Info Provider

The info provider is an XSLT script that transforms the output of the info service into GLUE compliant LDIF. The XSLT script used to be stored in /opt/d-cache/etc/glue-1.3.xml and used to contain both the transformation logic and local configuration parameters.

In dCache 1.9.12 the info provider has been extended to support GLUE 2.0 in addition to GLUE 1.3, and it has been restructured to separate configuration parameters from the transformation logic.

The old info provider file, /opt/d-cache/etc/glue-1.3.xml, is obsolete. All site specific configuration is done within the file /opt/d-cache/etc/info-provider.xml. The configuration parameters httpHost, xsltProcessor, xylophoneConfigurationDir, xylophoneConfigurationFile, xylophoneXSLTDir, and saxonDir are deprecated and have been replaced by other parameters. The configuration parameter httpPort is obsolete. The parameter info-provider.publish specifies whether to publish GLUE 1.3, 2.0, or both. See /opt/d-cache/share/defaults/info-provider.properties for details.

Service: pinmanager

The pin manager is responsible for staging and pinning files on behalf of SRM and DCAP doors. In this release the pin manager has been reimplemented from scratch. The design has been significantly simplified to increase performance and robustness of this critical component.

The database schema is separate from the old implementation's schema. The new schema is created the first time the pin manager is started. Existing pins are automatically imported if the same database is used. We strongly recommend using the same database. The tables of the old pin manager are not deleted, however downgrading is not possible without loosing the pins that were created with the new pin manager.

Notice that the migration from the old pin manager requires at least PostgreSQL 8.3. The migration will fail when using older versions.

The command line interface of the pin manager has changed slightly and third party scripts may have to be updated.

Service: poolmanager

Pool manager has seen a fair amount of cleaning in this release, however user visible changes should be small. The most significant change is that pool manager no longer verifies the correctness of location information stored in the name space, that is, on which pools a file is stored. Instead pool manager expects doors to verify the information and deal with any stale locations. This change eliminates a round trip to the pool and thus reduces the latency of opening files.

Pool mode changes are now logged at warning level by pool manager.

Doors and DNS lookups

Most doors and movers have been updated to avoid DNS lookups and reverse lookups that slow down transfers. Instead IP addresses are passed around. This may break existing deployments on multi-homed hosts. When this happens doors may need to be bound explicitly to particular network interfaces.

DCAP, FTP, SSH and admin doors now support the listen property to bind the door to a particular IP address.

Service: gplazma

The get mapping cell command was added. The command was present in earlier releases, but was dropped when the gPlazma cell was reimplemented in 1.9.9.

Database Support

The /opt/d-cache/bin/dcache script has been updated with several new commands related to databases used by various services.

The subcommand database ls lists the databases used by services on this host and is supported by all services that use an external database.

Other database subcommands only apply to services using the LiquiBase schema management library. Currently pin manager is the only service that uses this library. Please consult the man page for details about these commands.

Scripts

Besides changes to the pool and database commands, /opt/d-cache/bin/dcache now supports the services command. The command is similar to the services command found in dCache 1.9.5: It lists all configured services and the domains they belong to. The status command was extended to provide additional information. The new check-config command provides diagnostic information about obsolete and deprecated properties used.

The migration script was moved and renamed to /opt/d-cache/libexec/migrate-from-1.9.5.sh. It was updated to follow best practice by using the host name as the name of the layout. The new version does its best to remove lines copied verbatime from the original template into dCacheSetup. Many deprecated properties are automatically renamed to the new names. At the end of the migration the script invokes the check-config command to report about any deprecated or obsolete properties being used.

The /opt/d-cache/install/install.sh script is no longer needed and has been removed. The remaining two scripts in /opt/d-cache/install/ have been moved to /opt/d-cache/libexec/.

Configuration Properties

Lots of minor issues have been fixed in the configuration system. Many properties that were already deprecated or obsoleted in earlier releases are now marked as such and generate errors or warnings. Other properties that were removed during the transition to the new configiration system in 1.9.7 have been reintroduced as deprecated properties. Default declarations of most services have been moved from batch files and into properties files in /opt/d-cache/share/defaults/.

Changelog 1.9.11-1 to 1.9.12-1