Highlights

The 5.1 series of dCache focuses on new data transfer features.

It introduces 3rd-party copy capabilities to the xrootd protocol.

FTP doors can now support anonymous-FTP logins, and explicit TLS encrypted control channel (FTPS).

gPlazma has gained support for SciToken authentication as well as improved OpenID Connect support.

Pools can now send the full set of inotify events through SSE.

Pool manager can create dynamic pool groups based on defined tags.

Incompatibilities

  • The Kafka configuration was moved from dcache.properties to a dedicated kafka.properties file.

Acknowledgments

Once again, we are grateful for code and documentation contributions from contributors outside of our core team.

Onno Zweers provided many helpful corrections to the documentation.

Lea Morschel also provided documentation and code refinements.

Other contributions originated from a lecture at HTW Berlin; many thanks to Jonas Grabber, “tabea”, “Co” and “xpinkyx” for their contributions.

Release 5.1.20

frontend

The current release removed unnecessary login requirement on restores and transfers.

The current release fixed a bug in the frontend if the inotify events are used.

skel

The current release repaired erroneous batch directives before cell creation.

Now it is fixed and domain is not left in zombie state after a fatal error, but restarts, as it should.

srm

Now host IP is used for comparison when determining if SURL is local.

Changelog 5.1.19..5.1.20

462bd7c
[maven-release-plugin] prepare release 5.1.20
e049838
skel: repair erroneous batch directives before cell creation
eb3ec8e
dcache-frontend: remove unnecessary login requirement on restores and transfers
2483270
srm: use host IP for comparison when determining if SURL is local
ba8781b
frontend: events inotify fix deadlock
156099d
dcache,frontend: release dcache-view version 1.5.6
87278e8
[maven-release-plugin] prepare for next development iteration

Release 5.1.19

cell

Curator client was not able to restore the connection to ZK server after network partitioning. The is now fixed.

skel

The current relase fixed tape-reserved size calculation.

webdav

The current release fixed, where the WebDAV door failed to follow RFC 4918. This make some clients reject dCache WebDAV door as a valid WebDAV endpoint.

Changelog 5.1.18..5.1.19

44d7c94
[maven-release-plugin] prepare release 5.1.19
06467af
Fix tape-reserved size calculation
d940db9
webdav: include DAV header in OPTIONS requests.
34b2beb
cells: do not re-define zookeeper watcher
5f83ba7
[maven-release-plugin] prepare for next development iteration

Release 5.1.18

canl

The current release updated lib version to 2.5.1.

gplazma

The current release fixed URL-prefix SciToken parsing and error handling if JWT contains malformed SciToken scopes.

Changelog 5.1.17..5.1.18

ae9f749
[maven-release-plugin] prepare release 5.1.18
761528a
gplazma: scitoken add unit tests and fix SciTokenScope
0a1b9aa
canl: update to version 2.5.1
448265c
[maven-release-plugin] prepare for next development iteration

Release 5.1.17

config

A typo in the dcap config file was fixed correcting dcacp.enable.kafka to dcap.enable.kafka.

dcache

Pool compatibility with Xrootd–2 and Xrootd–4 versions is now allowed.

gplazma

The SciToken plugin will now reject any JWT where there is none of the expected scopes defined. This allows dCache to support both OpenID-Connect and SciTokens.

webdav

The current release fixed an issue of transfers through dCacheView when the webdav door is configured with empty webdav.allowed.client.origins value, which is the default value.

Changelog 5.1.16..5.1.17

f5e71b8
[maven-release-plugin] prepare release 5.1.17
d204071
dcache: add null check to pool info collector util
1a9be8f
config: fix typo in property name
f27d03d
gplazma: scitoken fix two issues with SciToken plugin
dc948d0
webdav: fix CORS when all clients are allowed to connect
99e2106
dcache: allow pool compatibility with Xrootd–2 and Xrootd–4 versions
0b06dfd
[maven-release-plugin] prepare for next development iteration

Release 5.1.16

srm

The current release fixed a problem resulting in high CPU use in SrmManager if clients are attempting to pin a file and PinManager is unavailable.

A regression fixed where SrmManager will reject all QUEUED jobs and INPROGRESS BringOnline requests on restart, if there are no SRM doors running when SrmManager starts.

Changelog 5.1.15..5.1.16

521fe2e
[maven-release-plugin] prepare release 5.1.16
73cc5c4
SrmManager: fix handling of saved requests on start-up
ca2bd3d
SrmManager: avoid spamming if PinManager is down
a761455
[maven-release-plugin] prepare for next development iteration

Release 5.1.15

doors

The current release fixed a bug where running the lb set tags admin command without any arguments triggers a NullPointerException.

pool

The current release improved error messages about jobs cancellation.

scripts

The dcache-storage-descriptor command no longer requires a URL argument.

Changelog 5.1.14..5.1.15

06f58c2
[maven-release-plugin] prepare release 5.1.15
78a2168
doors: fix “lb set tags” command with no arguments
8bbfcb0
pool: improve messages when migration job is cancelled.
77f4323
scripts: fix variable ordering in dcache-storage-descriptor
9944724
docs: TheBook add chapter on SRR
4432077
[maven-release-plugin] prepare for next development iteration

Release 5.1.14

gplazma

The SciToken gplazma plugin now supports the audience (aud) claim where the claim’s value is an array. This allows dCache to support SciTokens with multiple audience values.

pool

Pool health-check log messages now include the pool’s name.

webdav

On an unsuccessful HTTP-TPC pull request, dCache will delete the file. If this deletion did not work then an error was logged. This is fixed now and failures to delete the incomplete file from a failed HTTP-TPC pull request, where the incomplete file has been deleted by some other means are now logged at DEBUG level, rather than WARN level.

xrootd

The current release refited checksum handling after xrootd4j bug fix.

Changelog 5.1.13..5.1.14

09f44ba
[maven-release-plugin] prepare release 5.1.14
bbe0dc2
dcache-xrootd: refit checksum handling after xrootd4j bug fix
81af14e
webdav: avoid logging non-error as an error
f30b95b
pool: include pool name in health-check reports
a68785d
gplazma: scitoken add support for multiple audience claims
25a4d2d
[maven-release-plugin] prepare for next development iteration

Release 5.1.13

frontend

The current release fixed QoS pin semantics.

A bug is fixed in frontend that results in a NullPointerException for billing queries where no limit is specified.

Changelog 5.1.12..5.1.13

1eb7e32
[maven-release-plugin] prepare release 5.1.13
c83c585
frontend: fix NPE if limit is not specified
9ed3404
dcache-frontend: fix QoS pin semantics
a931d26
[maven-release-plugin] prepare for next development iteration

Release 5.1.12

Changes affecting multiple services

The Apache Commons Compress library used in dCache was updated to version 1.19.

A rare deadlock situation in the Chimera database was eliminated. In cases where, within the same directory, concurrent mkdir and rmdir events happened, transactions within the database could deadlock. This would be indicated by the message

ERROR: deadlock detected

in the logs.

pool

There were reports of extraordinarily high CPU usage on pool nodes with a large number of cached files. Through an optimization of the sweeper, CPU usage was reduced significantly.

xrootd

This release fixes a vulnerability in dCache’s XRootD protocol implementation. We recommend that all sites update their XRootD doors. Details will be made available through EGI Security and, in a week’s time, through an update to these release notes.

Changelog 5.1.11..5.1.12

fe857de571
[maven-release-plugin] prepare release 5.1.12
b9895148f9
dcache-xrootd: honor read paths when listing directories
5022e47257
resilience: don’t compare Integer objects by refference
f0d9e124dd
sweeper: use in-memory map instead of repository for histogram data
7de9af5be9
dcache-xrootd: replace constants for version number
00b2df8fbd
dcache-xrootd: update protocol version numbers
522a84d482
libs: update apache.commons:commons-compress to 1.19
99f1932666
chimera: fix ABBA db deadlock when mkdir and rmdir run concurrently
21dbec4a6b
[maven-release-plugin] prepare for next development iteration

Release 5.1.11

dcap

dcap door could not handle out-of-date errors. This is now fixed.

gplazma

The current release fixed thread leak by explicitly close NamingEnumeration

httpd

The current release fixed escape status field in HttpPoolMgrEngineV3.

Changelog 5.1.10..5.1.11

da80e6a
[maven-release-plugin] prepare release 5.1.11
5626e54
dcap: restart pool selection on OUT-OF-DATE error
508a34f
gplazma-ldap: avoid thread leak by explicitly close NamingEnumeration
93de29b
httpd: escape status field in HttpPoolMgrEngineV3
e8c663f
[maven-release-plugin] prepare for next development iteration

Release 5.1.10

srm

A new user community requires the srm tools to be able to handle very large file listings. During preliminary tests, OutOfMemory errors from the srmls tool were observed. This is now fixed and srm can now support operations on very large file lists without running out of memory.

webdav

The current release added allow header to list of response headers for OPTION method request.

Changelog 5.1.9..5.1.10

2b5ef20
[maven-release-plugin] prepare release 5.1.10
51bbef8
webdav: add allow header to OPTION method request
281da6d
srm: Remove JVM memory limits
cf14b7f
[maven-release-plugin] prepare for next development iteration

Release 5.1.9

common

The current release fixed formatting of error message in Checksum.

frontend

Admins may now configure frontend to specify in which country (or countries) data may be stored. This information is visible through dCacheView.

gplazma

The current release fixe remote reading of JSON with UTF–8 CharSet and dCache can now work with OPs that use utf-8 charset.

Changelog 5.1.8..5.1.9

8e8b2b9
[maven-release-plugin] prepare release 5.1.9
6948f87
frontend: make geographic placement configurable
b5e245d
common: fix formatting of error message in Checksum
1662c2c
scitoken: fix remote reading of JSON with UTF–8 CharSet
176f4eb
[maven-release-plugin] prepare for next development iteration

Release 5.1.8

nfs

NPE on “show transfers” command is now fixed.

webdav

The current release fixed CORS for WebDAV doors that do not allow anonymous access; in particular, to support dCacheView uploading and downloading files with such authentication-required WebDAV doors.

Changelog 5.1.7..5.1.8

29688cf
[maven-release-plugin] prepare release 5.1.8
2bfc913
nfs: fix NPE on “show transfers” command
c07272b
docs: Update UserGuide to use guide-specific navigation header
173b4ae
webdav: fix cross origin resources sharing issue
11c5ce0
[maven-release-plugin] prepare for next development iteration

Release 5.1.7

chimera

The shell infrastructure supports commands being given interactively, on the commandline (e.g., ‘chimera mkdir /path/to/dir’) and from stdin (e.g., ‘echo “mkdir /path/to/dir” | chimera’). chimera now supports the latter case and properly shows command output when invoked in that fashion.

frontend

This release updates dCache View to 1.5.5.

nfs

The NFS door now correctly handles situations where newly created read-only files could occasionally not be written into:

f = os.open('test.txt', os.O_WRONLY|os.O_CREAT, 0400)
os.write(f,"Hello pNFS!")
os.fsync(f)
os.close(f)

will now succeed.

webdav

A client may issue a PUT request that targets an existing collection resource; i.e., attempt to write a file as a path that is a directory. dCache, until now, responded with an incorrect status code of 500. This release changes the status code for this operation to 405 (Method not allowed), thus keeping closer to RFC 4918.

xrootd

This release improves compatibility with the xrdcp client in versions >4.9 by responding correctly to query strings requesting a specific checksum type.

Changelog 5.1.6..5.1.7

06521c8fb8
[maven-release-plugin] prepare release 5.1.7
c994d6a39f
dcache, frontend: release dcache-view version 1.5.5
7db20d5f7f
nfs: introduce workaround ‘permission deny’ on layout commit
65d8c86ac1
chimera: chimera shell should show output when commands come from stdin.
9bf1cbc775
webdav: return 405 status code for PUT requests targeting collections
fb4576d6f6
dcache-xrootd: add checksum cgi handling to door query
7850889dac
[maven-release-plugin] prepare for next development iteration

Release 5.1.6

many

The dcache pool ls command now provides correct output even if the pool is defined with a single-digit number of bytes.

Changelog 5.1.5..5.1.6

8f7d147636
[maven-release-plugin] prepare release 5.1.6
72aad72efd
scripts: avoid copy-n-paste error when calculating pool size
f2fb97af41
[maven-release-plugin] prepare for next development iteration

Release 5.1.5

frontend

This release updates the dCache View web GUI to version 1.5.4.

ftp

HAProxy can probe endpoints to discover if they are still alive.

The FTP door has an optimisation that detects such probes and does not create the FTP command interpreter, since the FTP client (the HA-Proxy instance) is calling on behalf of itself, and will not issue any FTP requests.

This release fixes a regression that would cause erroneous NullPointerExceptions when FTP doors were probed by HAProxy.

pool

The default value for the xrootd Third-Party Copying server response timeout, pool.mover.xrootd.tpc-server-response-timeout, was increased from 2 to 30 seconds to provide more robust behaviour in the face of high loads and network congestion.

transfermanager

Error messages like the WebDAV door’s

Failed to fetch information for progress marker: failed to query pool: (0) Job not found : Job-1

where the TransferManager is unable to discover the current status of the pool mover now include the pool’s name, which should make debugging easier.

Changelog 5.1.4..5.1.5

9efd766039
[maven-release-plugin] prepare release 5.1.5
60ea0aa2d3
dcache, frontend: release dcache-view version 1.5.4
8e420a6883
pools: make the xrootd tpc response timeout less aggressive
22148b8a3d
transfermanager: include pool name in error for ‘mover ls’ failures
0d0093540a
ftp: avoid NPE on HA-Proxy probes
8eb19681b6
core: fix pool selection in killAll command of TransferManager
ca26b686e7
[maven-release-plugin] prepare for next development iteration

Release 5.1.4

Changes affecting multiple services

This release includes an updated Jetty library, with the update adressing CVE–2019–10247.

This release includes an updated Jetty library, with the update adressing CVE–2019–10247.

dcap

The Kafka messaging implementation in the dcap service has been made more robust, fixing issue [#4831](https://github.com/dCache/dcache/issues/4831).

frontend

Periodic activity associated with the frontend door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

nfs

Periodic activity associated with the NFS door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

pool

Attempting to start a full checksum scan (with csm check *) while an existing scan is still running is no longer reported as a bug.

Pool start-up logging now includes the corresponding pool cell name.

An internal timing check was updated, which should result in more robust pool behaviour. There should be no user-visible impact.

webdav

Periodic activity associated with the WebDAV door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

xrootd

A new configuration property, pool.mover.xrootd.tpc-server-response-timeout, allows setting a timeout for xrootd 3rd party copy operations. This can also be changed through the new admin command xrootd set server response timeout.

Changelog 5.1.3..5.1.4

8bc74e3f13
[maven-release-plugin] prepare release 5.1.4
c9ed0d87e3
pool: avoid IllegalStateException in ‘csm check *’ command
8cdf34d790
dcap: fix premature close of kafka sender
d88ba9bef4
sweeper: compute now after the values have been fetched
8aaabb6316
libs: use jetty 9.4.18.v20190429
03ad080bed
dcache-xrootd: add ability to override default timeout for server response (TPC)
906ab5ac03
[maven-release-plugin] prepare for next development iteration
d8f0ee3954
frontend: include CDC in scheduled activity
6007c39a82
nfs: include CDC in scheduled activity
8ba362e928
webdav: include CDC in scheduled activity
a5337a705a
pool: ensure initialisation thread has correct CDC information
0e7fcabfd8
jetty: make CanlContextFactory subclass of jetty.ssl.SslContextFactory.Server
6e566d3d9f
pom: use jetty 9.4.17.v20190418

Release 5.1.2

alarms

To ease troubleshooting, the POOL_DEAD alarm message now includes the pool name.

pinmanager

A bug was fixed where PinManager’s bulk ls admin command yielded a NullPointerException if the optional argument was omitted.

A typo prevented the error message “Remote connection failure while unpinning…” from appearing completely and correctly in the logs. The error message string now contains the message string of the underlying Exception, hopefully providing helpful details for troubleshooting.

pool

A regression that prevented a replica’s last access time from being updated was fixed.

A regression that prevented a replica’s position in the LRU queue for garbage collection from being updated was fixed.

inotify’s IN_CLOSE_WRITE event was sent at a time when it could not be guaranteed that the file triggering the event could immediately be opened. This race condition was fixed, and consumers can start using a file immediately after receiving this event.

webdav

Users asserting the “admin” role would occasionally receive NullPointerExceptions when trying to transfer files through WebDAV. This release fixes that issue.

Changelog 5.1.1..5.1.2

d8b2275279
[maven-release-plugin] prepare release 5.1.2
6f23091cff
UnpinProcessor: fix assumed typo {)
438a96a1d9
webdav: allow transfers as user with role ‘admin’
f914ddcf08
pinmanager: avoid NPE if no argument given for ‘bulk ls’ command
71cb623a8d
alarms: add pool name to POOL_DEAD alarm
033bd83c8c
pool: fix reordering of removable replicas on access
84ed9d20c6
pool: fix storage of replica last access time
a98a1ce5ad
[maven-release-plugin] prepare for next development iteration
ad9852c91a
pool: inotify avoid race in IN_CLOSE_WRITE event

Release 5.1.1

Changes affecting multiple services

Stage request from unknown locations resulted in NPE in dcap and pinmanager services. this is now fixed and using dccp to stage a file should work even if the location is unknown. ‘–’

frontend

A client that disconnects and quickly reconnects could had triggered the following NPE, this is now fixed.

resilience

The current release fixed race condition on replica state and no inaccessible file errors occures for a newly written file.

Changelog 5.1.0..5.1.1

adba768
[maven-release-plugin] prepare release 5.1.1
7d7c4c5
dcap/pinmanager: stage request for unknown location results in NPE
ef3a977
dcap batch : fix handling of dcap.kafka.topic variable
fc88f10
dcache-resilience (stable branches): fix race condition on replica state
0e979ce
frontend: fix race on client reconnecting
40307e9
[maven-release-plugin] prepare for next development iteration

Release 5.1.0

Alarms

So that they can be given different priority levels, a new alarm, POOL_DEAD, was created to distinguish a pool which is completely unreachable, from POOL_DOWN or POOL_DISABLED.

Alarms was also modified in order to be able to support a more modern version of the logback library.

Billing

The database connection was wrapped so as to catch fatal exceptions and report them as alarms (this is already done for other databases such as chimera).

When using the Kafka messaging system, the “billing” topic was hardcoded until now. This release introduces a new configuration variable, dcache.kafka.topic=billing to configure this.

When a pool reports a replica was removed, it includes information on why the replica was deleted. By default, this information is logged at the end of the line in double-quotes, where previously an empty string was logged.

dCacheView

Support macaroons for sharing files and directories. There is now an easy way for a user to generate a pre-authorised URL by requesting a macaroon from dCache.

Gravatar is an externally run service that maps an email address to an icon. The goal is that many independent services may use the same icon for the same user. At sites’ request, this behaviour is now optional.

Frontend

A bug preventing the proper filtering of active transfers according to uid was fixed. A reminder as to how this works: (a) admins always see everything; (b) authenticated users always see their own transfers; (c) if frontend.authz.unlimited-operation-visibility is set to true, authenticated users can see all transfers. Non-authenticated users can only see anonymous transfers.

The inotify support now includes sending IN_OPEN, IN_ACCESS, IN_MODIFY, IN_CLOSE_WRITE, IN_CLOSE_NOWRITE events on file activity. Uploading or writing a file will send IN_OPEN, IN_MODIFY, IN_CLOSE_WRITE. Downloading or reading a file will send the IN_OPEN, IN_ACCESS, IN_CLOSE_NOWRITE events. Support in the pool for sending these events is configurable pool.inotify-generation.enable (enabled by default). Some fine-tuning is possible using pool.inotify-generation.backlog.initial, pool.inotify-generation.backlog.per-door and pool.inotify-generation.message-batch-size where these have the same meaning as the similarly named pnfsmanager. properties.

Sending IN_MODIFY and IN_ACCESS events is rate-limited by the pool, so that multiple IO operations of the same type generate at most one event every pool.inotify-generation.io-suppression pool.inotify-generation.io-suppression.unit (by default 10 SECONDS).

FTP

Add support for anonymous FTP. Anonymous FTP is when the client authenticates with a place-holder username (e.g., anonymous or ftp) and something (perhaps their email address) as the password. When enabled, dCache will grant limited (read-only) access to that user. Many aspects may be configured, including the username (anonymous by default) and whether the password must look like an email address (by default not enforced). A specific root directory may be configured for anonymous users (by default, dCache’s root), further limiting what data is exposed. The Globus transfer service uses anonymous access for “archival” access.

Add support for FTPS doors. This is explicit FTPS sometimes written as FTP(E)S. Implicit FTPS is not widely used and is not supported in dCache. This new FTP door is represented by a new possible value for ftp.authn.protocol. Both FTPS and gsiftp encrypt the control channel and neither support encrypted data transfers. FTPS doors are advertised (in info-provider, frontend, etc) with URLs that start ftps://. Currently, only username and password authentication is supported.

gplazma

The ban plugin can now ban users by their uid or gid.

The restrictions infrastructure now supports a mult-targeted restriction: MultiTargetedRestriction. This allows a plugin to allow users to have different permissions for different parts of the namespace.

OpenID-Connect

The oidc plugin will introspect JWT to learn issuer. The access token, issued by the OpenID-Connect server is opaque; however, a JWT is a common choice for this token. If the token is a JWT then the oidc plugin can discover which OIDC server issued the token. If multiple OIDC servers are configured, knowing which one issued the token avoids having to contact each OIDC server, resulting in a faster response.

The oidc plugin now accepts the gplazma.oidc.http.slow-threshold and gplazma.oidc.http.slow-threshold.unit configuration properties, with 2 SECONDS as default values. If looking up a user takes longer than this threshold then the plugin will log the slow query.

The oidc plugin will cache the OIDC discover document. This document contains slowly changing information, so it makes sense to cache the information. The cache lifetime is controlled by the configuration properties gplazma.oidc.discovery-cache and gplazma.oidc.discovery-cache.unit, with default values of 1 HOUR. Fetching an up-to-date version of the discovery document when the cache has expired is done asynchronously: login attempts will use the old data until the fresh document is received.

The oidc plugin will now query OIDC servers in parallel, if more than one OIDC server is configured.

The oidc plugin will try to keep the TCP network connections with OIDC services (used to query a user’s identity) alive between subsequent requests.

The oidc plugin caches a user’s identity, as this lookup operation may be slow for some OIDC servers. The maximum number of identities cached is gplazma.oidc.access-token-cache.size (by default 1000). Cached information is refreshed, controlled by gplazma.oidc.access-token-cache.refresh and gplazma.oidc.access-token-cache.refresh.unit (by default 100 SECONDS). This refresh happens asynchronously: existing login activity uses the existing data until new information is returned. Users are automatically removed from the cache if they are idle. The expiry time is controlled by gplazma.oidc.access-token-cache.expire and gplazma.oidc.access-token-cache.expire.unit (by default 120 SECONDS).

SciToken

Add scitoken plugin to support SciTokens. A SciToken is a bearer token from some OAuth2 server that uses the JWT format. The SciToken includes a description of which activities the bearer to entitled. The scitoken plugin maps all tokens from the same OAuth2 server to a single account. Paths within the SciToken are mapped to dCache paths by adding a prefix. The plugin optionally supports t

The jti claim is unique for each SciToken. dCache can prevent replay attacks by remembering the previous gplazma.scitoken.token-history values and rejecting the token if already seen. If the configuration value is zero (the default) then this feature is disabled.

The aud claim describes which service the SciToken was issued. The gplazma.scitoken.audience-targets configuration property is a space-separated list of targets that dCache should consider for itself. If the configuration property is non-empty then dCache will reject any SciToken with an aud claim that does not match one of the listed claims.

NFS

Improve performance of file system locks.

Pool

For HTTP transfers, the pool now honours the pool.mover.http.chunk-size property. Some storage backends (in particular, CEPH) suffer from very poor performance if IO operations are small. Setting this property to a larger value will result in the pool caching incoming data (in memory) so that writes are the configured size. When sending data, the pool will try to read this amount of data before sending it to the client.

When accepting data, the pool may know the file’s ultimate size before the transfer has started; for example, if an HTTP client specifies the ‘Content-Length’ HTTP request header in a PUT or an HTTP TPC request, or if an FTP upload is orchestrated using SRM. Under these circumstances, the the pool will now allocate all of that capacity in the storage backend before initiating the transfer. This avoids allocating more capacity with each client write. Some storage backends (CEPH in particular) suffer from poor performance if capacity is allocated incrementally.

Improve error reporting for HTTP Third-party copy. In particular, the error message describes any redirection that took place while removing redundant information.

When making an HTTP third-party PUSH copy, the pool makes an HTTP PUT request and then verifies that the uploaded file was transferred correctly by requesting the uploaded file’s checksum. Previously, the pool always asked for the same set of checksum algorithms, irrespective of which algorithm values dCache knows for this file. dCache now builds the list of desired checksum algorithms based on the known checksums for the file.

When making an HTTP third-party PUSH copy, the pool checks the HTTP response status code from the server. The server may response 204 and 205 to indicate a successful upload where the content is identical, a status code that some systems (including OneData) use. dCache now considers 204 and 205 as successful HTTP response codes.

A successful HTTP PUT request for an HTTP third-party PUSH copy does not guarantee that a subsequent HTTP HEAD request will succeed. In particular, a pre-authorised URL (from an S3 endpoint) may allow the PUT request but not the HEAD request. Previously, dCache always considered this to be a failure and failed the request. Now, if the transfer is not required to verify checksums then the HEAD request failure is ignored.

The NFS mover now tells the client that if it wishes to update file attributes then such requests must go to the door (not the pool). A client calling fsync on an open file will now update the file’s mtime and size.

Support pools with multiple tags. The pool.tags configuration property was meant as a comma-separated list of key-value pairs; e.g., pool.tags = tag1=value1, tag2=value2, .... Due to a bug, a pool had only one tag, with the remainder tags included in that tag’s value (e.g., tag1 value1,tag2=value2). Now pools will accept multiple tags.

Pool Manager

Added option -dynamic to psu create pool command to create a pool group that will dynamically add all pools that have a specific attribute. Currently, only pool tag, defined by pool.tags property is supported.

Example: > psu create pgroup -dynamic -tags=zone=A zone-A-pools

Will create pool group zone-A-pools and any existing pool as well as any new pool that configured with tag zone=A will be automatically included into that pool group.

Add the ability to disable anonymous staging. Without enabling explicit stage protection, dCache allows anyone who can read a file to stage it. Various doors allow anonymous access, where any world-readable files may be read. If such a file is stored on tape then an anonymous user could trigger staging of those files. This version of dCache has the ability to disallow staging if the user is not authenticated.

The psu admin commands are inconsistent. Several commands that support modifying associations have the form psu addto and psu removefrom to add and remove an association (respectively). However, the commands for managing the associations between pools or poolgroups and links do not follow this form; instead, the commands are psu add link and psu unlink. This version of dCache fixes this inconsistency by also supporting the psu addto link and psu removefrom link commands.

The rc failed admin command documentation is updated, so it no longer wrongly described the argument as a PNFS-ID.

Resilience

An important bug (from a race condition) which made resilience susceptible to causing data loss during removal of excess replicas was fixed.

Several smaller bug fixes were also applied:

1) The restart command

\s Resilience pool ctrl start

now properly reinitializes the pool state on the pool operations.

2) The retry command

\s Resilience retry errors

no longer throws a string index exception.

  1. After excluding a pool, it cannot be scanned under any circumstances; it must be explicitly included again before scanning can be activated on it; i.e.,
\s Resilience pool include [pool expression]

Logging of messages has also been added to resilience in order to be able to trace its communication with other dCache components. These records appear at INFO level in the resilience.resilience log.

For clarity, the final state of a file operation which ends up being unnecessary is no longer reported as ‘VOID’, but as ‘NOOP’, when you do:

\s Resilience file ls

WebDAV

The HTTP status code returned after successfully initiating an HTTP third-party-copy is now 202 (Accepted), rather than 200 (OK). The HTTP response also includes a Content-Type header describing the response as text/perf-marker-stream.

HTTP third-party copy now supports RFC–3230 ‘Want-Digest’ headers on COPY requests. For HTTP PUSH requests, this just acts like an HTTP HEAD request with the ‘Want-Digest’ header: the response includes the file’s checksum. For HTTP PULL requests, dCache will ensure that the most appropriate checksum algorithm is calculated. If the client advertises it suppers HTTP trailers then the checksum value is returned after the transfer has completed.

XRootD

The dCache xrootd door/server now supports security level configuration and signed hash verification. See dcache.properties under “Xrootd: support for verifying signed hashes (by server/door)” for information.

We have also added support for mapping clients with valid credentials to NOBODY; this also enables the current version of third-party-copy (which lacks the full proxy delegation that will be available in the next Golden Release [5.2]) to allow destination servers to log in anonymously to dCache as source during third-party-copy; in this latter case (only), dCache also allows the server to see read-protected files.

A bug involving access logging when the door is configured with HAproxy was fixed, and trace-level logging in general was converted to debug-level, so that the former may be reserved for handshake packet dumps only.

Changes affecting multiple services

Kafka topics can now be set independently for several services using the variables

dcache.kafka.topic = billing

(where “billing” is the default value) and the more specific

dcap.kafka.topic = ${dcache.kafka.topic}
ftp.kafka.topic = ${dcache.kafka.topic}
nfs.kafka.topic = ${dcache.kafka.topic}
pool.kafka.topic = ${dcache.kafka.topic}
webdav.kafka.topic = ${dcache.kafka.topic}
xrootd.kafka.topic = ${dcache.kafka.topic}

Macaroons now include an Issue ID (iid) caveat, which dCache ensures is the first caveat in each new macaroon. The iid caveat ensures that the ID is always unique, even when generated with the same secret, but that derived macaroons (ones with additional caveats) may still be identified.

By default, the WebDAV and frontend doors now reject any macaroon sent on an unencrypted connection. Although these doors used to accept macaroons on unencrypted connections and this behaviour is configurable, we recommend sites accept this change and have dCache reject unencrypted macaroons.

This version of dCache configures systemd limits in a single place, avoiding that ineffective configuration is included.

Changelog from 5.0.0 to 5.1.0

40307e91f2
[maven-release-plugin] prepare for next development iteration
5234c07f2a
[maven-release-plugin] prepare release 5.1.0
30b6209ad8
Revert “docker: Add a way to create docker image”
92975e7a54
resilience: adjust synchronization of file operation removal from map
7c212b24d6
pool: avoid throwing a RuntimeException for non-bugs
0863f49066
docs: TheBook UserGuide fix location of edit-me-on-github ribbon
4b2d9381de
docs: TheBook update heading importance to be more consistent
f8365ec8a7
transfermanager: do not retry starting mover if transfer is not supported
aac0eb6e52
pool: avoid log-and-throw anti-pattern
ec453280d2
frontend: ensure client is disconnected when shutting down channel
a56116e565
frontend: avoid race on cancelling channel garbage-collect task
bc68734a3c
transfermanager: avoid NPE on shutdown
886f36acb3
pool: throw exception with meaningful error message
0fc21c0217
[maven-release-plugin] prepare branch 5.1
68acbbf689
docs: add rest API active transfers description
99f049d6ee
docs: UserGuide REST API add description of space reservations
3f3ed31715
docs/UserGuide add initial frontend QoS description
4a5c1836bf
Remove copy-n-paste error
eddd8b0670
docs: add inital information about frontend REST API
82c0bbc4a3
pool: register size of successful zero-length uploads
136cbe1cde
Core: make dCache buildable on non-English locales
8ac85b095f
docs: add UserGuide
0ffdd34f46
libs: use slf4j–1.7.26
ef39e437f2
Core: make dCache buildable on non-English locales
3313ca8b8a
docs: TheBook remove reference to non-existing service
e3c4223424
gplazma: oidc better handling if JWT is unknown or invalid
f0f1379ecc
resilience: do not allow pool scans of excluded pools, period
5733f5e60c
alarms: move to logback 1.1.4 and fix handling of MDC
bab35350a6
src: remove dead code (artifacts of ReplicaManager)
f894624ee0
dcache-resilience: fix indexing on retry errors command
b2dc1245d0
pools: JSON mover info timeInSeconds should be timeInMilliseconds
0d9963becd
pool: update StateChangeListener classes with change explanation
b51ca78b8a
pool: include change explanation in replica monitoring events
84f3e15e35
pool: add explanation field when updating a replica
3cf5967975
pool: describe what triggered replica state changes
fb97274611
ftp: add support for (explicit) FTPS doors
ec34f0c332
resilience: update state on pool operations when restarted from admin command
07772db56f
libs: use nfs4j–0.18.2
aaecade7e4
chimera-shell: fix class cast of extractor in constructor
eddda93d9e
libs: update jetty version to 9.4.12.v20180830
00d89b7244
nfs: fix missing CDC initialization
18dc14c270
dcache: adding more configurations to KafkaProducer
7e8993fa45
gplazma: add SciToken support
d688b16846
common: add multi-targeted restriction
f7ac12a41a
gplazma: ban add ability to select based on uid and gid
8161f08b81
resilience: do simple existence check of replica on pool to avoid dark removes
7069ed2183
systemtest: remove ancient replica pools
4651736dd5
pool: add inotify support for open/read/write/close events
1e5e355150
docs: mongodb as pool metadata backend
b66a891fd0
docs: Fixed lists in config-PoolManager
7f3fbf0261
docs: RT#9533 and 9494: IPv6
e5a7041156
docs: RT#9534: Ceph pool backend config
cd826a7d25
gplazma: oidc cache user-info lookup result
08656e96b5
TheBook: some hints on using a banfile (#4740)
311c7182a1
Fixing example of ban.conf that breaks dCache.
424f968388
gplazma: oidc fix broken commit
4a1da8a569
gplazma: oidc optimise network usage
09d3757069
resilience: add ability to log resilience activity
18a79e7768
pool: grow file prior HTTP TPC
884dcc9e8d
pom: use rados4j–0.0.4
549b22cd8b
resilience: add ability to log resilience activity (incoming)
87aa817764
resilience: remove the ‘VOID’ operation state
cad57b6561
Add documentation to accompany patch 806de4076b3ea7c0ccad3c28abf0bf8430ffacf0
806de4076b
kafka: add ability to specify Kafka topic name
83c8b1bcd6
pool-ceph: Failure when initiating a macaroon authz HTTP PUSH, authn with X.509/macaroon to target.
1ac0ac77fd
Changes to the dCache documentation (TheBook): broken links, typos, formatting, wording, …
4c124a11d7
gplazma: oidc add discovery cache configuration properties
2abff00b2e
gplazma: oidc warn slow user-info queries
69669e6e3f
webdav: fix resource name for door root
53b95fbdb2
Updated dependency list for building on Debian
5cafe39aad
ftp: add support for anonymous FTP access
f87e4a070c
util:add new setter to ConfigurationMapFactoryBean
ddf9009b52
dcache-xrootd: use debug level instead of trace
2a2455f11c
xrootd: upgrade to xrootd4j v3.4.1
ac5419d132
gplazma: oidc introspect JWT to learn issuer
0f2a60e276
transfermanagers: recover from non-fatal error starting mover
277f773dd0
pnfsmanager: network issues triggered a NPE
65b8e499d1
Changed .gitignore to include .reviewboardrc file
9566a165bb
cells: Improving readability of cells communication code
223992f630
pool: fix CDC for repository listener notification
186a407d7f
ftp: store calculated checksum using root privileges
fc8de4575a
docs: add initial information about the ftp door
915d66e454
util:add unit test to ConfigurationMapFactoryBean
094f99fc01
dcache-xrootd: implement support for security level and signed hashes
77b0b33708
pnfsmanager: fix regression when querying attributes of the root directory
9853191a15
systemtest: fix OpenSSL DN format change
099d7e2ffc
core: describe missing FileAttribute in PoolManager message
2e3c02acc2
systemtest: remove reference to access-log plugin
968926535c
pnfsmanager: include link information in storageinfo
949936a21e
poolmanager: introduce dynamic pool groups based on pool tags
33d957a899
webdav: fix path-to-caveat for macaroon minting endpoint
59145459e2
cells: changed readme (#4634)
a4acb0ec79
gplazma gridmap plugin: compare DNs ignoring letter case for attribute names
2b64c115c5
pool: fix Guava to Java-API migration introduced in 3de5f34179
478049fbd4
poolmanager: update ‘rc failed’ command
cbd3ceba9f
pool: fix tag parsing regression introduces in #7f9a32beec
dbe93282ce
dcache: Expand star import in Indexer
70e1b12d6a
docs: describe poolmanager configuration location in zookeeper
7f9a32beec
pool: fix parsing of pool tags
6c33d339a2
webdav: fix NPE when Kafka notified file deletion
3de5f34179
various: replace guava Files usages JDK Files
aa2883f25a
doc: fix typo
0d8df30064
pool: report correct replica creation time to sweeper
b5e076fd68
psu: remove dead code
c26df9f1d4
libs: update to hazelcast–3.11
137d2ed565
xrootd: fix access logging when xrootd door is configured with HAproxy
bca2869abf
webdav/frontend: disabling basic authn should not disable macaroons
0dc483bdcc
srm: do not log a stack-trace on expected Exception errors
ea3cdd48cb
webdav/frontend: update default to reject macaroons on unencrypted channels
3bb180b849
transfermanager: fail third-party copy if the file is still being uploaded
9cb63c2142
webdav: fail COPY early if file is currently being uploaded
800c979d74
transfermanager: abort transfer if there is a bug
8550267c56
Fix typo
224574aced
chimera: README to README.md, clean contents
690d7f9145
webdav/frontend: add switch to reject macaroons sent unencrypted
b3d0f026f0
pool: introduce clone method for FileAttributes
9fde33d6d8
pom: update nfs4j to version 0.18.1
a95763ecce
gplazma: JAAS plugin logs a stack-trace on misconfiguration
01c9a760b1
httpd: write requests on admin webpage to .access-log
7991e8ed61
pool: nfs mover always returns DATA_SYNC4 as stable write indication
f400fcd0a3
packaging: remove redundant systemd configuration, add note to limits file
eee88ce105
transfermanager: include pool address in the mover start failure message
6946e79390
pool: update error messages to make them distinct
b0839feca7
poolmanager: update some psu commands to be consistent
a16ef71546
pool: avoid using the same error message in multiple places
131d0ecb6d
admin: update dependency on mina-sshd library
1483ee5d12
admin: update dependency on mina-sshd library
c48f8ef30a
libs: use nfs4j–0.18.0
7cc4963ad6
alarms: add pool dead alarm
b65acf64af
webdav.properties: note about redirects to HTTPS
a1cadfe017
pool: fix lookup for canonical hostname for IPv6 addresses
273821b44e
doc: add paragraph in direct command execution in admin section
22270ff344
admin : add direct command execution capability
17f01abd5e
pool: grow file prior FTP upload
4bb48bed23
pool: don’t update atime on flush
e4e841630f
pnfsmanager: do not emit IN_ATTRIB event on atime updates
3e6e7cf782
scripts: fix ‘dcache pool yaml’ command
63b864039e
webdav: 401 for unauthenticated requests; message in status line
b2d7393ba3
dcache, frontend: release dcache-view version 1.5.3
7a9ac72b11
door: fix issue 4551 (wring storage)
63438af62d
Update config-chimera.md
b508529b83
Update config-write-token.md
99ddbf5050
ftp: fix MLSC command for non-small directories
37c3a825c1
pom: update maven-git-commit-id-plugin to version 2.2.6
d936ee8bf4
dcache-xrootd: remove mv request hack
31e4f9c818
dcache-frontend: add documentation concerning restores
71367f26e0
dcache-xrootd: Allow ROOT read access by destination server when dCache is the source in a third-party transfer.
4e2e589eee
pom.xml: update xrootd4j dependency to 3.3.4
8a7ced4cc5
dcache-frontend: undefined suid parameter on transfers should be NULL not “null”
5f82d2710e
dcache: update kafka-client lib version to 2.1.0
cecd828583
macaroon: add an issue ID caveat
1e9353e8de
webdav/pool: add support for RFC3230 Want-Digest on COPY command
24bf16cbf6
pool: http-TPC do not fail if HEAD fails for non-verified transfer
dd66bf0ea3
pool: http-TPC consider more PUT response codes as successful
ad17e63ac0
pool: remove desired checksum from ChecksumMover
606c8236b7
nfs: do not filter device’s IP addresses based on site locality
f29ddeab04
dcache: wrap billing data source with AlarmEnabledDataSource
1f8a3c8322
pool: HTTP TPC create ‘Want-Digest’ value based on known checksums
6c17311370
common: fix random data generation in TimeseriesHistogram unit test
1c17e70bf8
docs: clarify requirement gplazma.ldap.try-uid-mapping option
a7bc5ef250
webdav: work-around Milton racy API for creating collections
3f5d8f8255
poolmanager: add ability to disable anonymous staging
a42022b6c1
webdav: fix name of root
4560d3f4ba
webdav: add Content-Type and fix status
4f29cba8a0
pool: move ChecksumChannel creation out of AbstractMover
32e1cbb7ee
pom: use rados4j–0.0.3 with ARM64 support
220e2f19ef
Removing version reference
57f1e6b460
Remove version reference in new paragraph
73ffa3dff3
Update cookbook-transport-security.md
563094fb82
pom: use nfs4j–0.17.10
73a107892f
pool: improve error logging for failed HTTP third-party copies
b7d3b663ac
gplazma voms plugin: add trust anchor refresh paramater
9c526832cf
srm: include TLS/SSL port in ‘dcache ports’ command
41c32a8c14
dcache, frontend: release dcache-view version 1.5.1
653bb697bb
pool: fix HTTP chunked upload
806e8bc3d1
xrootd: support mapping clients with valid credentials to NOBODY
b36842c800
libs: use nfs4j–0.17.9
4fd18c5dbe
pool: grow file prior HTTP upload
4d91d68ca4
Update BUILDING.md
a0a53eef1f
pool: let http mover respect pool.mover.http.chunk-size
169e6579ea
[maven-release-plugin] prepare for next development iteration