1.9.12 Release Notes

dCache 1.9.12 focuses on:

Table of contents

Upgrade Instructions

Incompatibilities

Please consider the following changes when upgrading from a version before 1.9.12-1:

Compatibility

It is safe to mix pools of releases 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9.10, 1.9.11 and 1.9.12. Head nodes and doors must be upgraded to 1.9.12 together and cannot be mixed with head nodes or doors of releases before 1.9.12. Components of different 1.9.12 releases can be mixed freely.

Compatibility Matrix

We distinguish between pool components and head nodes. Any component which is not a pool is considered a head node, including doors. The following table shows compatibility between different releases.

1.9.12 Head1.9.12 Pool
Head 1.9.5nono
1.9.7nono
1.9.8nono
1.9.9nono
1.9.10nono
1.9.11nono
1.9.12yesyes
Pool 1.9.5yesyes
1.9.7yesyes
1.9.8yesyes
1.9.9yesyes
1.9.10yesyes
1.9.11yesyes
1.9.12yesyes

1.9.12-13

Service: pool

Fix another HTTP compliance issues related to range request. This time it is about using the correct reply code (416) when the client requests a range beyond the end of the file. This fix resolves a compatibility issue with ARC.

Fix 'dcache pool ls' to show the pool size stored in the pool's setup file, in case such a pool size exists.

Service: xrootd, webdav, ftp, gridftp, kerberosftp

Fix a harmless but noisy null pointer exception. The problem could affect FTP and WebDAV doors too, but was first observed in the xrootd door.

Service: webdav

Fix a null pointer exception being thrown while generating the error page.

Service: xrootd

Fix regression in Alice token authorization plugin. The plugin failed to load in 1.9.12-12.

Changelog 1.9.12-12 to 1.9.12-13

1.9.12-12

Service: pool

Fix several race conditions that could lead to stale hidden movers, TCP connection leaks and link count inconsistencies.

Service: admin

Add detection of terminal size. Some ssh clients do not transmit terminal resize messages. For those clients resizing the terminal after login to the admin service will break line editing.

Service: ftp, gridftp, kerberosftp

Make timeout of write mover creation configurable. The timeout is shared with read mover creation. The timeout can be configured by adjusting the gsiftpPoolTimeout property. A side effect is that the default timeout for write mover creation increases from 10 seconds to 10 minutes.

Service: xrootd

Refactored authorization plugin support to make plugins truly pluggable. These are now loaded through the Java ServiceLoader facility. This allows third parties to develop authorization and path mapping plugins the dCache xrootd implementation. Existing third party xrootd authorization plugins will have to be updated.

Miscellaneous

Fix well known cell name resolving when using JMS messaging.

Added a plugin directory to dCache. The following directories are scanned for plugins: /usr/share/dcache/plugins, /usr/local/share/dcache/plugins, /opt/d-cache/share/plugins. Plugins are represented as sub-directories to any of these directories. jar files are included on the Java classpath, properties files are parsed and added as default configuration values, and batch files are made recognized as service definition scripts.

Changelog 1.9.12-11 to 1.9.12-12

1.9.12-11

Service: dcap, gsidcap, kerberosdcap

Resolve an issue when the dcap client is in active mode (ie when the -A option is used) and the server cannot resolve the host name sent by the client (eg when the client is behind NAT). Instead the door now uses the remote address of the control channel when selecting an appropriate pool.

Improve logging in case of errors during GSI or Kerberos sessions.

Fix race condition that could occur during shutdown.

Service: pool

Ensure forward compatibility of the HTTP mover.

Fix several HTTP compliance issues related to range request (ie partial reads).

Service: ftp, gridftp, kerberosftp

Add glob support to the NLST command. This allows mget to work in some clients.

Fix race condition that could occur during shutdown.

Service: webdav

Improve logging on fatal errors. This change may help debug the source of the errors.

Service: poolmanager

Fix a race condition.

Service: admin

Fix several bugs related to line editing. Add history search (press Ctrl-R).

An unfortunate side effect of these fixes is that line editing capabilities need to know the terminal size. The admin door currently does not support determining the terminal size and thus always assumes that the terminal is 80 characters wide. Line editing will misbehave with other terminal sizes.

Fix listing of admin service in dcache services.

Service: pnfsmanager, nfsv3, nvfsv41

Fix name space corruption when moving directories (chimera only).

Fix RFC 5661 compliance of ACL delete permission check. A file or directory can now be deleted if the subject has the DELETE_CHILD permission on the parent directory, or the DELETE permission on the entry being deleted, or in case neither are specified if the subject has the ADD_FILE permission on the parent directory.

Service: srm

Improve robustness against ill formed srmLs requests.

Service: info

Fix logging of several erroneous warning and error messages.

Info provider

Fix GLUE2 compliance. Be sure you have at least v2.0.8 of glue-schema RPM installed on the node running the info provider.

Miscellaneous

Improve documentation in configuration defaults.

Improve white space processing around property values.

Fix race condition in getcellinfos command.

Add service definition for hopping manager. The name of the new service is hopping.

Changelog 1.9.12-10 to 1.9.12-11

1.9.12-10

Service: pool

Fixed bug in which failure to stage would leave behind a repository entry in state D. The only way to get rid of the entry was by restarting the poo.

Service: pinmanager

Improved robustness of pinmanager. In particular an out-of-memory problem has been fixed.

Changelog 1.9.12-9 to 1.9.12-10

1.9.12-9

Service: poolmanager

Fixed a race condition that caused a null pointer exception in rebalancer.

Fixed persistence of the slope setting.

Updated format of PoolManager.conf such that settings are persistet using the partition manager commands introduced in dCache 1.7. The legacy commands are still supported, but will be removed in a future version. Starting with version 1.9.12-9 the pool manager save command save in a format that will be compatible with future versions.

Services: ftp,gridftp,kerberosftp,xrootd,webdav

Use correct transaction duration for billing records.

Services: pool

Fixed the 'dcache pool reconstruct' on Solaris. Without the fix the command would fail with the error "File name too long".

Fixed URI generation in hsmcp.rb. The pool now also logs incorrect URIs returned by the HSM script.

Fixed HTTP range support. Added support for multiple ranges. Fixed path check. These fixes resolve compatibility issues with ARC and ROOT.

Service: billing

Fixed a race condition related to formating of timestamps.

Service: pnfsmanager

Fixed a bug in the PNFS backend that prevented listing through SRM.

Service: xrootd

Improved robustness of transfer timeout logic.

Changelog 1.9.12-8 to 1.9.12-9

1.9.12-8

Service: pnfsmanager

Fixed service instantiation with PNFS backend.

Service: pool

Avoid verbose logging of NoSuchElementException when listing movers. The fix possible solves similar problems for other operations and other services.

Avoid verbose logging of non-fatal DCAP client failures.

Service: ftp, gridftp, kerberosftp

Fixed a race condition that would cause the door to become unresponsive to client commands.

Fixed a potential denial of service problem.

Fixed a null pointer exception triggered by running the info command before the client was authenticated.

Avoid verbose logging when clients do not disconnect cleanly.

Service: spacemanager

Fixed pool selection cost model for systems with space manager enabled. Without the fix a system with space manager would perform pool selection with outdated cost information.

dcache script

Fixed file system size check for the dcache pool create command. It didn't actually check the file system size and in some cases it caused the operation to fail, leaving an incomplete pool behind.

Changelog 1.9.12-7 to 1.9.12-8

1.9.12-7

Service: gplazma

The configuration property gPlazmaNumberOfSimutaneousRequests was deprecated and replaced by gPlazmaNumberOfSimultaneousRequests.

Service: webdav

Generate themable HTML for 404 errors.

Service: nfsv41

Improved compatibility with early versions of rhel5 and MacOS.

Service: transfermanagers

Fixed deletion of the target file in case of transfer failures. Previous releases would fail with a stack trace logged.

Service: dcap

Reduced logging noise triggerd by file upload.

Service: ftp, gridftp, kerberosftp

Added a workaround for a bug in Globus Online.

Service: srm, webdav, webadmin

Downgraded to Jetty 7.2-2. Jetty 7.4.2 introduced in dCache 1.9.12-6 was the source of a regression in socket error handling.

Init script

Ensure that dcache start starts all domains even when some are already running.

Changelog 1.9.12-6 to 1.9.12-7

1.9.12-6

Service: nfsv3

Fixed a regression introduced in 1.9.12-5.

Fixed portmap registeration: Previous releases registered TCP even when only UDP was enabled.

Fixed attribute caching.

Service: acl

Make acladmin cell a well known cell (like it was in 1.9.5).

Service: dcap

Enable TCP_NODELAY on the control and data channels. Should reduce latency.

Service: ftp

Enable TCP_NODELAY on the control channel. Should reduce latency.

Service: admin

Enable TCP_NODELAY. Should reduce latency.

Service: webdav

Upgraded to Jetty 7.4.2.

Fixed error reporting. Previous versions reported all errors as "server error". Now the proper HTTP error code is reported.

Added HTML formatting and CSS styling of error messages. The template file has been replaced. The property webdav.templates.list is deprecated and has been replaced by webdav.templates.html.

Service: webadmin

Upgraded to Jetty 7.4.2.

Service: gplazma (version 2)

The kpwd plugin now uses defaults from /opt/d-cache/share/defaults/gplazma.properties.

Now supports ${...} substitution for properties defined in dcache.conf or the layout file.

Fixed reloading of the kpwd file.

Service: xrootd

Fixed recovery logic from internal message loss. Ensures that transfers are removed from the active transfers page of the httpd service.

Info provider

Fixed path to XSLT file in FHS packages.

1.9.5 migration script

Updated list of properties to rewrite during upgrade.

Changelog 1.9.12-5 to 1.9.12-6

1.9.12-5

Service: nfsv41

Fix a null pointer exception.

Adds optional support for NFS 3 in the NFS 4 door. Eventually the nfsv3 and nfsv41 doors will be merged into one.

Service: ftp

Support UTF-8 encoded file names independent of the local of the host system.

Output permission information by default in MLSD and MLST commands.

Fixed encoding of the UNIX.mode fact introduced in 1.9.12-4. The mode field is now encoded as an octal. This ensure compatibility with UberFTP and other clients.

Added authorization checks for various attributes accessed with MLSD and MLST commands.

Surpress the size fact for MLSD and MLST commands with the entity is a directory. This improves RFC 3659 compliance.

Publish GridFTP protocol version to the "Active Transfers" monitoring page.

Service: admin

Fix null pointer exception.

Service: pool

Improve error handling in case file deletion fails.

Chimera client

Fix readtag and writetag commands and added writefile command.

Configuration

Flags the properties httpdEnablePoolCollector and removeUnexistingEntriesOnFlush as obsolete. The properties have not been used since dCache 1.9.5.

1.9.5 migration script

Fixed translation of the dCacheSetup file. In previous releases only the first line of dCacheSetup was translated.

Logging

dCache configuration properties are now available as ${...} placeholders in logback.xml.

The log files /tmp/events.out and /tmp/trace.out have been replaced by per domain log files. This solves a permission problem when domains are run as different users.

Changelog 1.9.12-4 to 1.9.12-5

1.9.12-4

Service: nfsv41

Fixed attribute caching.

Service: xrootd

Kill mover if client or door timed out before the mover was started. Should reduce the number of mover timeouts.

Service: ftp

Implemented support for RFC 3659 UNIX facts (these are not specified by the RFC, but several clients implement these facts).

Improved compatibility with the UberFTP client (recursive directory listing now works).

Service: srm

Fixed a bug that would cause the message "can not find handler with id=nnnn" to be logged by the transfer manager after srmCopy requests successfully completed.

Service: webadmin

Fixed authorization.

Service: pnfsmanager

Fixed batch file for use with PNFS.

Service: gplazma

Fixed banning in gplazmalite-vorole-mapping plugin. Adding the entry:

"DN" -

will now ban the DN no matter whether it is used in combination with an FQAN or not. Note that banning in the gplazmalite-vorole-mapping plugin doesn't prevent other plugins from authorizing the request.

Fixed a bug which would cause a missing entry in storage-authzdb to authorize a user rather than reject the request. The problem would only occur if grid-vorolemap mapped a user to a user name that isn't mapped in storage-authzdb.

Service: pool

Fixed a conflict between the HTTP and xrootd mover that caused them to use the same TCP port when they shouldn't.

Fixed logging context handling. The bug could cause session IDs from transfers to erroneously be assigned other transfers and could even bleed through to other components.

The pool now no longer updates the last access time of a file during checksum scanning.

Fixed TCP buffer size handling in DCAP movers. The bug in some cases caused the requested buffer size to be incorrectly applied.

Enabled TCP keep alive in DCAP movers. This will ensure that DCAP movers eventually fail if the client abruptly disappears (eg due to a operating system crash or a network failure).

Service: pinmanager

Avoid unintended pool manager cost adjustment when pinning file. Also requires update of poolmanager.

Service: poolmanager

Avoid unintended pool manager cost adjustment when pinning file. Also requires update of pinmanager.

Service: info

Add context information to log messages.

Fixed bug that triggered "StateUpdate with zero updates encountered" log messages.

Removed the GlueSAAccessControlBaseRule work-around that publishes legacy values for broken clients. Any problems with lcg-infosites should be solved by upgrading to the latest version of the UI.

Service: admin

Fixed a couple of error messages.

Fixed an issue that could cause socket to be leaked.

Packaging

Fixed a number of RPM packaging errors in the EMI packages.

Other

Fixed System cell CLI help output.

Changelog 1.9.12-3 to 1.9.12-4

1.9.12-3

Service: info

The info service lists the network interfaces of each door in dCache. The DNS name of the interface is used as a key in the list. This causes problems when several interfaces share the same name and would often result in the host name being absent in the data published by the info provider. This problem has now been fixed by giving each network interface a unique name.

Service: httpd

Fixed a number of image files that were corrupted in the 1.9.12-2 release.

Service: pool

The lifetime of sticky bits are now reported in human readable format. The output of the 'rep sticky ls' command is affected.

Service: dcap

Minor logging related cleanups.

Service: nfsv3

Improved server side readdir caching.

Service: srm

Fixed processing of escape codes in SURLs and TURLs.

Out of memory handling

dCache triggers an automatic restart when a domain runs out of memory. We have observed that he restart sometimes fails, presumably because further out of memory errors are triggered during shutdown. A number of changes have been made to make the shutdown sequence more robust.

Since 1.9.12-1 dCache automatically generates heap dump files upon out of memory errors. A check has now been added to the startup script to check for the existing of these files and issue a warning.

Libraries

Several third party libraries have been upgraded. The upgrades represent minor releases that fix bugs.

RPM packaging

Ownership of the billing directory is now preserved accross upgrades.

Changelog 1.9.12-2 to 1.9.12-3

1.9.12-2

Service: srm

The following properties can now be specified separately for ls, put, get, bringonline and reserve requests: srmDatabaseEnabled, srmCleanPendingRequestsOnRestart, srmKeepRequestHistoryPeriod, srmExpiredRequestRemovalPeriod, srmRequestHistoryDatabaseEnabled, and srmStoreCompletedRequestsOnly. These properties replace the now deprecated properties srmJdbcMonitoringLogEnabled, srmNumberOfDaysInDatabaseHistory, srmOldRequestRemovalPeriodSeconds, srmJdbcSaveCompletedRequestsOnly, srmJdbcEnabled, and srmCleanPendingRequestsOnRestart.

Service: pool

Fix the processing of the waitForFiles property. The property did not have the inteded effect.

Migration script

Fix a typo that caused the script to fail in 1.9.12-1.

Added translation of SRM properties deprecated in 1.9.12-2.

Added support for moving admin authorization files to new location.

dCache now refuses to start if it detects a 1.9.5 configuration. This prevents that dCache is accidentally started before the configuration has been migrated.

Service: gplazma

Propagate the read-only flag back to doors. In recent versions the flag was ignored.

Several gPlazma 2 related changes were made, but gPlazma 2 is still under development and should not be used in this version. We expect gPlazma 2 to be production ready in 1.9.12-3.

Service: pnfsmanager

Reintroduce check that prevents existing checksums from being altered. The check was present in 1.9.5 but was lost in later versions.

Service: admin

SSH keys have moved from /opt/d-cache/config/ to /opt/d-cache/etc/. The migrate-from-1.9.5.sh script has been updated to move the files. However when upgrading from a version newer than 1.9.5 the SSH keys have to be moved by hand.

Location manager

Fixed a problem in which dCacheDomain would fail to start if /opt/d-cache/etc/lm.config could not be created. This would typically happen if dCacheDomain is started as a user different from root.

The location of lm.config is now configurable.

Service: httpd

Static content used by the monitoring service has been moved to /opt/d-cache/share/httpd/static/.

Service: webdav

WebDAV doors now report the DN to billing when files are deleted.

Changelog 1.9.12-1 to 1.9.12-2

1.9.12-1

Service: pool

The /opt/d-cache/bin/dcache pool commands have been updated to provide an ls subcommand to list configured pools. The create subcommand has been extended to add the new pool directly to the layout file. The command no longer creates the pool setup file and instead adds the pool specific configuration to the layout file. The command thus provides the functionallity of both the create and add commands found in some earlier releases.

The migration module was extended to reduce interference between multiple migration jobs running on the same pool. The migration module now respects the pool mode on the target pool.

Service: srm

The SRM used to have the ability to periodically submit vacuum commands to the database. This command is PostgreSQL specific and thus tied the SRM to PostgreSQL. PostgreSQL has for quite a while supported auto vacuuming, and manually executing the vacuum command is no longer necessary. We have therfore removed the submission of vacuum commands from dCache. The configuration parameters srmVacuum and srmVacuumPeriod are obsolete.

Some SRM operations may be processed synchronously or asynchronously, at the server's discretion. Since 1.9.10 dCache can start to process such requests synchronously and, if this is taking too long, reply asynchronously and continue to work on the operation in the background. In dCache 1.9.12 this behaviour is enabled by default for get, put, bring online, and list operations. The default switch over time is 1 second. The old behaviour can be restored by setting the srm*SwitchToAsynchronousModeDelay parameters.

The RemoteHttpTransferManager cell has been removed. The functionality has been merged into the RemoteGsiftpTransferManager cell, which has in turn been renamed to RemoteTransferManager. The SRM was extended to support third party srmCopy transfers from HTTP resources. Third party upload to HTTP resources and support for HTTPS is not available yet in the SRM.

Service: dcap, gsidcap, kerberosdcap

DCAP supports prestaging from tape. The DCAP door used to rely on a stager service to perform the staging. In dCache 1.9.12 the DCAP door instead uses the PinManager to trigger staging. The dummy-prestager service is obsolete and can be deleted from the layout.

Service: pnfsmanager

The PNFS manager has been restructured. One consquence is that the output of the info command in the cell admin interface has changed slightly: It is now grouped into sections in line with how it is done in pool manager and pools. Third party scripts may need to be adjusted.

Service: webdav

WebDAV doors now submit remove entries to billing when files are deleted.

The HTML rendering of a directory listing now includes a column containing the file size. The HTML is customizable through a template file. As a consequence the properties webdav.images.logo, webdav.images.directory, webdav.images.file, webdav.style.css, webdavLogoPath, webdavDirIconPath, webdavFileIconPath, webdavCssPath, dirIconPath, fileIconPath, and cssPath are obsolete. Please read the document embedded in the default template file /opt/d-cache/share/webdav/templates/list.stg for details on customizing the look and feel.

The property webdavAnonymousListing was introduced to prevent unauthorized users from listing directories.

The property webdavProtocol was extended with a third option, https-jglobus, to enable HTTPS support based on the JGlobus library. The advantages compared to the https option are that host and CA certificates are read and refreshed directly from /etc/grid-security/ and that proxy certificates (including VOMS certificates) can be used for client authentication.

Info Provider

The info provider is an XSLT script that transforms the output of the info service into GLUE compliant LDIF. The XSLT script used to be stored in /opt/d-cache/etc/glue-1.3.xml and used to contain both the transformation logic and local configuration parameters.

In dCache 1.9.12 the info provider has been extended to support GLUE 2.0 in addition to GLUE 1.3, and it has been restructured to separate configuration parameters from the transformation logic.

The old info provider file, /opt/d-cache/etc/glue-1.3.xml, is obsolete. All site specific configuration is done within the file /opt/d-cache/etc/info-provider.xml. The configuration parameters httpHost, xsltProcessor, xylophoneConfigurationDir, xylophoneConfigurationFile, xylophoneXSLTDir, and saxonDir are deprecated and have been replaced by other parameters. The configuration parameter httpPort is obsolete. The parameter info-provider.publish specifies whether to publish GLUE 1.3, 2.0, or both. See /opt/d-cache/share/defaults/info-provider.properties for details.

Service: pinmanager

The pin manager is responsible for staging and pinning files on behalf of SRM and DCAP doors. In this release the pin manager has been reimplemented from scratch. The design has been significantly simplified to increase performance and robustness of this critical component.

The database schema is separate from the old implementation's schema. The new schema is created the first time the pin manager is started. Existing pins are automatically imported if the same database is used. We strongly recommend using the same database. The tables of the old pin manager are not deleted, however downgrading is not possible without loosing the pins that were created with the new pin manager.

Notice that the migration from the old pin manager requires at least PostgreSQL 8.3. The migration will fail when using older versions.

The command line interface of the pin manager has changed slightly and third party scripts may have to be updated.

Service: poolmanager

Pool manager has seen a fair amount of cleaning in this release, however user visible changes should be small. The most significant change is that pool manager no longer verifies the correctness of location information stored in the name space, that is, on which pools a file is stored. Instead pool manager expects doors to verify the information and deal with any stale locations. This change eliminates a round trip to the pool and thus reduces the latency of opening files.

Pool mode changes are now logged at warning level by pool manager.

Doors and DNS lookups

Most doors and movers have been updated to avoid DNS lookups and reverse lookups that slow down transfers. Instead IP addresses are passed around. This may break existing deployments on multi-homed hosts. When this happens doors may need to be bound explicitly to particular network interfaces.

DCAP, FTP, SSH and admin doors now support the listen property to bind the door to a particular IP address.

Service: gplazma

The get mapping cell command was added. The command was present in earlier releases, but was dropped when the gPlazma cell was reimplemented in 1.9.9.

Database Support

The /opt/d-cache/bin/dcache script has been updated with several new commands related to databases used by various services.

The subcommand database ls lists the databases used by services on this host and is supported by all services that use an external database.

Other database subcommands only apply to services using the LiquiBase schema management library. Currently pin manager is the only service that uses this library. Please consult the man page for details about these commands.

Scripts

Besides changes to the pool and database commands, /opt/d-cache/bin/dcache now supports the services command. The command is similar to the services command found in dCache 1.9.5: It lists all configured services and the domains they belong to. The status command was extended to provide additional information. The new check-config command provides diagnostic information about obsolete and deprecated properties used.

The migration script was moved and renamed to /opt/d-cache/libexec/migrate-from-1.9.5.sh. It was updated to follow best practice by using the host name as the name of the layout. The new version does its best to remove lines copied verbatime from the original template into dCacheSetup. Many deprecated properties are automatically renamed to the new names. At the end of the migration the script invokes the check-config command to report about any deprecated or obsolete properties being used.

The /opt/d-cache/install/install.sh script is no longer needed and has been removed. The remaining two scripts in /opt/d-cache/install/ have been moved to /opt/d-cache/libexec/.

Configuration Properties

Lots of minor issues have been fixed in the configuration system. Many properties that were already deprecated or obsoleted in earlier releases are now marked as such and generate errors or warnings. Other properties that were removed during the transition to the new configiration system in 1.9.7 have been reintroduced as deprecated properties. Default declarations of most services have been moved from batch files and into properties files in /opt/d-cache/share/defaults/.

Changelog 1.9.11-1 to 1.9.12-1