dCache 5.0 Release Notes

Highlights

dCache now supports Java 11 as its platform
Documentation, especially the dCache Book, received a major revision and will remain in focus
HTTP 3rd-party-copying has matured to a feature-rich, well tested state
Pinboard includes timestamps
updated external dependencies

Incompatibilities

This release breaks compatibility with pools running dcache version 3.2 or earlier.

Acknowledgments

We thank HTW Berlin students Marcel Munce, Ferdinand Wolff and MaKrHTW (???) as well as Onno Zweers from surfSARA for their contributions.

Release 5.0.37

dcache-xrootd

The version is upgrated to xrootd4j 3.5.7 with lossen username validation, so that

usernames like foo.1234:56 are no longer rejected.

The current release fixed java.lang.IllegalStateException: ChecksumChannel must not be written to after getChecksums.

TPC client first will be shut down when the pool netty channel goes inactive.

Changelog 5.0.36..5.0.37

c57beae: [maven-release-plugin] prepare release 5.0.37
43b7357: dcache-xrootd: bump to xrootd4j 3.4.7
79c7b05: dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool
ba3e17c: [maven-release-plugin] prepare for next development iteration

Release 5.0.36

pool

The unix xrootd tpc security plugin was included in order to enable the dCache TPC client to use a dCache pool as source when signed hash verification is on. However, this is now fixed and no special configuration necessary for organizations (like Tier 1) needing to communicate with EOS.

Changelog 5.0.35..5.0.36

ebd370f: [maven-release-plugin] prepare release 5.0.36
2df3093: pool (xrootd): make tpc security plugin default unix
71828df: [maven-release-plugin] prepare for next development iteration

Release 5.0.35

frontend

When frontend is run in a core domain and there is no history service reachable, the retry on no route to cell ends up spamming the message queues. This is now fixed.

xrootd

The current release fixed compatible level security for sigver.

The xrootd client has a command-line option, --path, which tells the server to create missing directories. This option is included in two-party copy, but setting it for TPC has no effect. TPCs which wish to write to dCache (as destination) to a non-existent subdirectory fail. This is now fixed and dCache no longer fails in these cases.

Changelog 5.0.34..5.0.35

ff35a45: [maven-release-plugin] prepare release 5.0.35
b2221af: dcache-xrootd: always create missing directories on write
ae0c458: dcache-xrootd: fix compatible level security for sigver
58754d4: dcache-frontend: remove retry flag on sendAndWait to history service
43ce3d1: [maven-release-plugin] prepare for next development iteration

Release 5.0.34

dcache

QOS migration policy engine was raising JVM error when no tape pool found. This is now fixed and the normal behavior is that HTTP error code reported back with No HSM pool found.

frontend

The current release improved error responses and they are more specific now.

The current release fixed the NPE stack trace arised because the pool data could be sent with a default sweeper data object.

Changelog 5.0.33..5.0.34

6b9bd86: [maven-release-plugin] prepare release 5.0.34
d978392: dcache-frontend: make ErrorResponseProvider return the more specific error message
f60c4f6: dcache: qos migration policy engine should not raise JVM error when no tape pool foundMotivation:
10bdda8: dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
462af62: [maven-release-plugin] prepare for next development iteration

Release 5.0.33

frontend

From RESTful admin API, the POST to pools/{name}/usage/mode -d {"rdonly":true} failed but {"rdonly": "true"} succeeded.

This is fixed now and both boolean and string work.

The current release fixed stack trace in logs.

Changelog 5.0.32..5.0.33

7c5f528: [maven-release-plugin] prepare release 5.0.33
7d16cda: dcache-frontend: allow pool enable/disable to use boolean JSON value
7e5fc48: dcache-frontend,history: protect against missing highest bin in histogram data
6cb80d6: [maven-release-plugin] prepare for next development iteration

Release 5.0.32

dcacheview

dcache-view version 1.5.7 is released

Release 5.0.32

dcacheview

dcache-view version 1.5.7 is released

Changelog 5.0.31..5.0.32

bd15b47: [maven-release-plugin] prepare release 5.0.32
3a61be3: dcache, frontend: release dcache-view version 1.5.7
07762f7: [maven-release-plugin] prepare for next development iteration

Release 5.0.32

frontend

The current release added support for OIDC names and Client-IDs with spaces.

Changelog 5.0.31..5.0.32

bd15b47: [maven-release-plugin] prepare release 5.0.32
3a61be3: dcache, frontend: release dcache-view version 1.5.7
07762f7: [maven-release-plugin] prepare for next development iteration

Release 5.0.31

frontend

The current release removed unnecessary login requirement on restores and transfers.

The current release fixed a bug in the frontend if the inotify events are used.

skel

The current release repaired erroneous batch directives before cell creation.

Now it is fixed and domain is not left in zombie state after a fatal error, but restarts, as it should.

srm

Now host IP is used for comparison when determining if SURL is local.

Changelog 5.0.30..5.0.31

9273c5f: [maven-release-plugin] prepare release 5.0.31
cfecbda: skel: repair erroneous batch directives before cell creation
90f5227: dcache-frontend: remove unnecessary login requirement on restores and transfers
a805d4b: srm: use host IP for comparison when determining if SURL is local
64b4f40: frontend: events inotify fix deadlock
683e8e9: dcache,frontend: release dcache-view version 1.5.6
4dc3213: [maven-release-plugin] prepare for next development iteration

Release 5.0.30

cell

Curator client was not able to restore the connection to ZK server after network partitioning. The is now fixed.

skel

The current relase fixed tape-reserved size calculation.

webdav

The current release fixed, where the WebDAV door failed to follow RFC 4918. This make some clients reject dCache WebDAV door as a valid WebDAV endpoint.

Changelog 5.0.29..5.0.30

7ec7ca4: [maven-release-plugin] prepare release 5.0.30
b353b4b: Fix tape-reserved size calculation
aa43c70: webdav: include DAV header in OPTIONS requests.
815f749: cells: do not re-define zookeeper watcher
0905ddc: [maven-release-plugin] prepare for next development iteration

Release 5.0.29

canl

The current release updated lib version to 2.5.1.

Changelog 5.0.28..5.0.29

035e1c5: [maven-release-plugin] prepare release 5.0.29
f980837: canl: update to version 2.5.1
51fb90a: [maven-release-plugin] prepare for next development iteration

Release 5.0.28

config

A typo in the dcap config file was fixed correcting dcacp.enable.kafka to dcap.enable.kafka.

dcache

Pool compatibility with Xrootd–2 and Xrootd–4 versions is now allowed.

webdav

The current release fixed an issue of transfers through dCacheView when the webdav door is configured with empty webdav.allowed.client.origins value, which is the default value.

Changelog 5.0.27..5.0.28

a74929d: [maven-release-plugin] prepare release 5.0.28
f96d231: dcache: add null check to pool info collector util
7d98792: config: fix typo in property name
f94e80e: webdav: fix CORS when all clients are allowed to connect
22a2d63: dcache: allow pool compatibility with Xrootd–2 and Xrootd–4 versions
2cd1b02: [maven-release-plugin] prepare for next development iteration

Release 5.0.27

srm

The current release fixed a problem resulting in high CPU use in SrmManager if clients are attempting to pin a file and PinManager is unavailable.

A regression fixed where SrmManager will reject all QUEUED jobs and INPROGRESS BringOnline requests on restart, if there are no SRM doors running when SrmManager starts.

Changelog 5.0.26..5.0.27

d9098bf: [maven-release-plugin] prepare release 5.0.27
a5f60cc: SrmManager: fix handling of saved requests on start-up
158795b: SrmManager: avoid spamming if PinManager is down
e58343a: [maven-release-plugin] prepare for next development iteration

Release 5.0.26

doors

The current release fixed a bug where running the lb set tags admin command without any arguments triggers a NullPointerException.

pool

The current release improved error messages about jobs cancellation.

scripts

The dcache-storage-descriptor command no longer requires a URL argument.

Changelog 5.0.25..5.0.26

e16683b: [maven-release-plugin] prepare release 5.0.26
432aa60: doors: fix “lb set tags” command with no arguments
2633541: pool: improve messages when migration job is cancelled.
88cce36: scripts: fix variable ordering in dcache-storage-descriptor
d164159: docs: TheBook add chapter on SRR
f52ee67: [maven-release-plugin] prepare for next development iteration

Release 5.0.25

pool

Pool health-check log messages now include the pool’s name.

webdav

On an unsuccessful HTTP-TPC pull request, dCache will delete the file. If this deletion did not work then an error was logged. This is fixed now and failures to delete the incomplete file from a failed HTTP-TPC pull request, where the incomplete file has been deleted by some other means are now logged at DEBUG level, rather than WARN level.

xrootd

The current release refited checksum handling after xrootd4j bug fix.

Changelog 5.0.24..5.0.25

bc977bd: [maven-release-plugin] prepare release 5.0.25
0ab2c68: dcache-xrootd: refit checksum handling after xrootd4j bug fix
3cedbaa: webdav: avoid logging non-error as an error
659b63f: pool: include pool name in health-check reports
3cf1302: [maven-release-plugin] prepare for next development iteration

Release 5.0.24

frontend

The current release fixed QoS pin semantics.

A bug is fixed in frontend that results in a NullPointerException for billing queries where no limit is specified.

Changelog 5.0.23..5.0.24

fcc7919: [maven-release-plugin] prepare release 5.0.24
ba65ccd: frontend: fix NPE if limit is not specified
2a20915: dcache-frontend: fix QoS pin semantics
44d9b74: [maven-release-plugin] prepare for next development iteration

Release 5.0.23

Changes affecting multiple services

The Apache Commons Compress library used in dCache was updated to version 1.19.

A rare deadlock situation in the Chimera database was eliminated. In cases where, within the same directory, concurrent mkdir and rmdir events happened, transactions within the database could deadlock. This would be indicated by the message

ERROR: deadlock detected

in the logs.

pool

There were reports of extraordinarily high CPU usage on pool nodes with a large number of cached files. Through an optimization of the sweeper, CPU usage was reduced significantly.

xrootd

This release fixes a vulnerability in dCache’s XRootD protocol implementation. We recommend that all sites update their XRootD doors. Details will be made available through EGI Security and, in a week’s time, through an update to these release notes.

Changelog 5.0.22..5.0.23

f16f4828dd: [maven-release-plugin] prepare release 5.0.23
efaa3c6089: dcache-xrootd: honor read paths when listing directories
d32187f7c3: resilience: don’t compare Integer objects by refference
c3775c8d57: sweeper: use in-memory map instead of repository for histogram data
de14b4877f: dcache-xrootd: replace constants for version number
20db76731a: dcache-xrootd: update protocol version numbers
9af1f67ed9: libs: update apache.commons:commons-compress to 1.19
e9ff3b259f: chimera: fix ABBA db deadlock when mkdir and rmdir run concurrently
fe723c8563: [maven-release-plugin] prepare for next development iteration

Release 5.0.22

dcap

dcap door could not handle out-of-date errors. This is now fixed.

httpd

The current release fixed escape status field in HttpPoolMgrEngineV3.

Changelog 5.0.21..5.0.22

d15ec87: [maven-release-plugin] prepare release 5.0.22
3b69ad0: dcap: restart pool selection on OUT-OF-DATE error
0cdb5fa: httpd: escape status field in HttpPoolMgrEngineV3
336f472: [maven-release-plugin] prepare for next development iteration

Release 5.0.21

srm

A new user community requires the srm tools to be able to handle very large file listings. During preliminary tests, OutOfMemory errors from the srmls tool were observed. This is now fixed and srm can now support operations on very large file lists without running out of memory.

webdav

The current release added allow header to list of response headers for OPTION method request.

Changelog 5.0.20..5.0.21

783dd61: [maven-release-plugin] prepare release 5.0.21
79baec2: webdav: add allow header to OPTION method request
0a97928: srm: Remove JVM memory limits
1c8c08c: [maven-release-plugin] prepare for next development iteration

Release 5.0.20

common

The current release fixed formatting of error message in Checksum.

frontend

Admins may now configure frontend to specify in which country (or countries) data may be stored. This information is visible through dCacheView.

Changelog 5.0.19..5.0.20

7d77a9c: [maven-release-plugin] prepare release 5.0.20
1687106: frontend: make geographic placement configurable
93a253e: common: fix formatting of error message in Checksum
7883369: [maven-release-plugin] prepare for next development iteration

Release 5.0.19

nfs

NPE on “show transfers” command is now fixed.

webdav

The current release fixed CORS for WebDAV doors that do not allow anonymous access; in particular, to support dCacheView uploading and downloading files with such authentication-required WebDAV doors.

Changelog 5.0.18..5.0.19

3d5c25b: [maven-release-plugin] prepare release 5.0.19
63a2357: nfs: fix NPE on “show transfers” command
5b3067f: webdav: fix cross origin resources sharing issue
49df7bb: [maven-release-plugin] prepare for next development iteration

Release 5.0.18

chimera

The shell infrastructure supports commands being given interactively, on the commandline (e.g., ‘chimera mkdir /path/to/dir’) and from stdin (e.g., ‘echo “mkdir /path/to/dir” | chimera’). chimera now supports the latter case and properly shows command output when invoked in that fashion.

frontend

This release updates dCache View to 1.5.5.

webdav

A client may issue a PUT request that targets an existing collection resource; i.e., attempt to write a file as a path that is a directory. dCache, until now, responded with an incorrect status code of 500. This release changes the status code for this operation to 405 (Method not allowed), thus keeping closer to RFC 4918.

xrootd

This release improves compatibility with the xrdcp client in versions >4.9 by responding correctly to query strings requesting a specific checksum type.

Changelog 5.0.17..5.0.18

d1b275f790: [maven-release-plugin] prepare release 5.0.18
8b8f32315c: dcache, frontend: release dcache-view version 1.5.5
9171aee1a4: chimera: chimera shell should show output when commands come from stdin.
0859aa2792: webdav: return 405 status code for PUT requests targeting collections
c7dca2d4de: dcache-xrootd: add checksum cgi handling to door query
cd0a6f0f66: [maven-release-plugin] prepare for next development iteration

Release 5.0.17

many

The dcache pool ls command now provides correct output even if the pool is defined with a single-digit number of bytes.

Changelog 5.0.16..5.0.17

719fb8a506: [maven-release-plugin] prepare release 5.0.17
639c8e204c: scripts: avoid copy-n-paste error when calculating pool size
40b7849ade: [maven-release-plugin] prepare for next development iteration

Release 5.0.16

frontend

This release updates the dCache View web GUI to version 1.5.4.

ftp

HAProxy can probe endpoints to discover if they are still alive.

The FTP door has an optimisation that detects such probes and does not create the FTP command interpreter, since the FTP client (the HA-Proxy instance) is calling on behalf of itself, and will not issue any FTP requests.

This release fixes a regression that would cause erroneous NullPointerExceptions when FTP doors were probed by HAProxy.

pool

The default value for the xrootd Third-Party Copying server response timeout, pool.mover.xrootd.tpc-server-response-timeout, was increased from 2 to 30 seconds to provide more robust behaviour in the face of high loads and network congestion.

transfermanager

Error messages like the WebDAV door’s

Failed to fetch information for progress marker: failed to query pool: (0) Job not found : Job-1

where the TransferManager is unable to discover the current status of the pool mover now include the pool’s name, which should make debugging easier.

Changelog 5.0.15..5.0.16

437a8768b0: [maven-release-plugin] prepare release 5.0.16
010d456aad: dcache, frontend: release dcache-view version 1.5.4
214cb68c6f: pools: make the xrootd tpc response timeout less aggressive
c0a95a2f71: transfermanager: include pool name in error for ‘mover ls’ failures
2db61d1da5: ftp: avoid NPE on HA-Proxy probes
843a15b9a5: core: fix pool selection in killAll command of TransferManager
1c96bb4b77: [maven-release-plugin] prepare for next development iteration

Release 5.0.15

Changes affecting multiple services

This release includes an updated Jetty library, with the update adressing CVE–2019–10247.

dcap

The Kafka messaging implementation in the dcap service has been made more robust, fixing issue [#4831](https://github.com/dCache/dcache/issues/4831).

frontend

Periodic activity associated with the frontend door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

nfs

Periodic activity associated with the NFS door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

pool

Attempting to start a full checksum scan (with csm check *) while an existing scan is still running is no longer reported as a bug.

Pool start-up logging now includes the corresponding pool cell name.

An internal timing check was updated, which should result in more robust pool behaviour. There should be no user-visible impact.

webdav

Periodic activity associated with the WebDAV door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

xrootd

A new configuration property, pool.mover.xrootd.tpc-server-response-timeout, allows setting a timeout for xrootd 3rd party copy operations. This can also be changed through the new admin command xrootd set server response timeout.

Changelog 5.0.14..5.0.15

61228154c6: [maven-release-plugin] prepare release 5.0.15
df878d862d: pool: avoid IllegalStateException in ‘csm check *’ command
76575b0771: Update config-SRM.md
9ab6197b73: Update config-SRM.md
462cf31a93: dcap: fix premature close of kafka sender
df99ce0710: sweeper: compute now after the values have been fetched
fa071d90cc: libs: use jetty 9.4.18.v20190429
cda8868232: dcache-xrootd: add ability to override default timeout for server response (TPC)
3b9c7f7674: [maven-release-plugin] prepare for next development iteration
6ec0c14175: frontend: include CDC in scheduled activity
99c377c2f6: nfs: include CDC in scheduled activity
6b2298ec51: webdav: include CDC in scheduled activity
4e146e7e02: pool: ensure initialisation thread has correct CDC information
ec475d6d07: jetty: make CanlContextFactory subclass of jetty.ssl.SslContextFactory.Server
1acd5f4f3d: pom: use jetty 9.4.17.v20190418

Release 5.0.13

alarms

To ease troubleshooting, the POOL_DEAD alarm message now includes the pool name.

pinmanager

A bug was fixed where PinManager’s bulk ls admin command yielded a NullPointerException if the optional argument was omitted.

A typo prevented the error message “Remote connection failure while unpinning…” from appearing completely and correctly in the logs. The error message string now contains the message string of the underlying Exception, hopefully providing helpful details for troubleshooting.

pool

A regression that prevented a replica’s last access time from being updated was fixed.

A regression that prevented a replica’s position in the LRU queue for garbage collection from being updated was fixed.

webdav

Users asserting the “admin” role would occasionally receive NullPointerExceptions when trying to transfer files through WebDAV. This release fixes that issue.

Changelog 5.0.12..5.0.13

64e2063b2c: [maven-release-plugin] prepare release 5.0.13
6b5e05f0ea: UnpinProcessor: fix assumed typo {)
07f6f5b58d: webdav: allow transfers as user with role ‘admin’
2e531f6134: pinmanager: avoid NPE if no argument given for ‘bulk ls’ command
249c895912: alarms: add pool name to POOL_DEAD alarm
51982824c3: pool: fix reordering of removable replicas on access
421cec3adf: pool: fix storage of replica last access time
554a40de9c: [maven-release-plugin] prepare for next development iteration

Release 5.0.12

Changes affecting multiple services

Stage request from unknown locations resulted in NPE in dcap and pinmanager services. this is now fixed and using dccp to stage a file should work even if the location is unknown. ‘–’

frontend

A client that disconnects and quickly reconnects could had triggered the following NPE, this is now fixed.

resilience

The current release fixed race condition on replica state and no inaccessible file errors occures for a newly written file.

Changelog 5.0.11..5.0.12

33d6d90: [maven-release-plugin] prepare release 5.0.12
0d78953: dcap/pinmanager: stage request for unknown location results in NPE
8c0666d: dcap batch : fix handling of dcap.kafka.topic variable
4a8a6dd: dcache-resilience (stable branches): fix race condition on replica state
e63f6b0: frontend: fix race on client reconnecting
01e3ed8: [maven-release-plugin] prepare for next development iteration
702f8d7: resilience: adjust synchronization of file operation removal from map

Release 5.0.11

frontend

The destroy sequence for a channel failed to obtain the Channel monitor. This could result in concurrent changes to the Channel’s state and incomplete Channel shutdown. This is now fixed and the shutdown sequence of a channel is more robust.

pool

An unhelpful error message “Parameter directory is not a directory” is replaced with one that provides information on which directory is missing.

The pool no longer logs configuration or deployment problems that prevent the pool from creating a mover as if that problem was a bug.

The current release fixed certain error cases, where a pool is unable to create a mover are no longer logged as a bug in dCache.

transfermanager

By default, the transfermanager will retry starting the mover ten times before giving up on the pool. If we know that a pool doesn’t support this transfer type (for whatever reason), then this makes no sense. This is now fixed and a pool that does not support a particular transfer is not immediately retried.

The current release fixed a NPE if transfer was cancelled.

Changelog 5.0.10..5.0.11

9f2d38e: [maven-release-plugin] prepare release 5.0.11
7511623: pool: avoid throwing a RuntimeException for non-bugs
f2c015b: docs: TheBook fix location of edit-me-on-github ribbon
2adec24: docs: TheBook update heading importance to be more consistent
ebb9071: transfermanager: do not retry starting mover if transfer is not supported
1c22df2: pool: avoid log-and-throw anti-pattern
412876a: frontend: ensure client is disconnected when shutting down channel
71d2244: frontend: avoid race on cancelling channel garbage-collect task
a1c02ea: transfermanager: avoid NPE on shutdown
dc9c6b9: pool: throw exception with meaningful error message
af5daa0: [maven-release-plugin] prepare for next development iteration

Release 5.0.10

pool

The current release improved time formatting for Json mover info.

resilience

Faulty parsing of the error string on the retry errors command results in operation failure, with message IllegalArgumentException: Illegal pnfsid string length. This is now fixed.

Changelog 5.0.9..5.0.10

8099bb7: [maven-release-plugin] prepare release 5.0.10
232341f: dcache-resilience: fix indexing on retry errors command
e01cc02: pools: JSON mover info timeInSeconds should be timeInMilliseconds
ba60436: [maven-release-plugin] prepare for next development iteration

Release 5.0.9

dcache

Jetty version is updated to 9.4.12.v20180830.

nfs

nfs4j version is changed to 0.17.11 with fixed export table evaluation order.

resilience

It is now possible to record resilience activity (on the receiving end), which may prove useful in understanding behaviour.

In rare circumstances dark removes can result in data loss by removing of all replicas for a given file. The current release fixed the issue.

Pool operations now can successfully be restarted from the command line after they have been shutdown, without restarting resilience.

Changelog 5.0.8..5.0.9

c3af74b: [maven-release-plugin] prepare release 5.0.9
0829208: resilience: update state on pool operations when restarted from admin command
31e70b9: libs: use nfs4j–0.17.11
2e2418b: chimera-shell: fix class cast of extractor in constructor
a120777: libs: update jetty version to 9.4.12.v20180830
73950c3: nfs: fix missing CDC initialization
4edc62c: resilience: do simple existence check of replica on pool to avoid dark removes
4828d68: systemtest: remove ancient replica pools
1f345c6: Fixing example of ban.conf that breaks dCache.
87410e2: [maven-release-plugin] prepare for next development iteration
147e11b: resilience: add ability to log resilience activity

Release 5.0.8

dcache

rados4j version is updated which contains bugfix addressing data corruption on write with HTTP.

kafka

The current release added ability to specify Kafka receiving topic name: dcache.kafka.topic=billing.

pool

The current release added performance boost for ceph pool.

pool-ceph

A NPE is fixed when initiating a macaroon authz HTTP PUSH, authn with X.509/macaroon to target.

resilience

It is now possible to record resilience activity (on the receiving end).

webdav

The current release fixed resource name for door root error.

Changelog 5.0.7..5.0.8

c284e8f: [maven-release-plugin] prepare release 5.0.8
9093601: pool: grow file prior HTTP TPC
0f065aa: pom: use rados4j–0.0.4
a2c97ac: resilience: add ability to log resilience activity (incoming)
1e89f34: kafka: add ability to specify Kafka topic name
a2a6064: pool-ceph: Failure when initiating a macaroon authz HTTP PUSH, authn with X.509/macaroon to target.
157580e: webdav: fix resource name for door root
cc14e94: [maven-release-plugin] prepare for next development iteration

Release 5.0.7

ftp

Now clients can request the checksum value of a file not owned by that user and where dCache does not already know the checksum value.

pnfsmanager

The current release fixed a NPE triggered by network issues.

pool

The current release fixed some logging on the pool where messages were recorded against an arbitrary context (i.e., the bit in square brackets), resulting in misleading information.

transfermanagers

TransferManager failed a transfer if the pool reports any problem when requesting the transfer was started. This is now fixed and third-party transfer is more robust against non-fatal errors that occur normally on a busy system.

xrootd

Xrootd4j is updated to version , which fixed regression in supporting TPC pre–4.9.

Changelog 5.0.6..5.0.7

19bd3ac: [maven-release-plugin] prepare release 5.0.7
2118353: dcache-xrootd: use debug level instead of trace
e3a1f18: xrootd: upgrade to xrootd4j v3.4.1
f73404c: transfermanagers: recover from non-fatal error starting mover
969561f: pnfsmanager: network issues triggered a NPE
a663aad: pool: fix CDC for repository listener notification
106f51c: ftp: store calculated checksum using root privileges
d2488fa: [maven-release-plugin] prepare for next development iteration

Release 5.0.6

xrootd

With this release, dCache has gained full signature verification support for xrootd.

Changelog 5.0.5..5.0.6

0367fc0348: [maven-release-plugin] prepare release 5.0.6
0aa6f55584: dcache-xrootd: implement support for security level and signed hashes
d883379f12: [maven-release-plugin] prepare for next development iteration

Release 5.0.5

gplazma

Due to a formatting issue, certificates issued by InCommon CA were not accepted because of case mismatches between the supplied “postalCode” and “street” and the expected “PostalCode” and “STREET” field descriptors.

Since this release, dCache handles those certificates as expected despite the unusual naming convention.

pool

There was an issue with the timestamps output by the sweeper ls command. The command lists replica creation time and the time of the last access. The output for replica creation time was incorrect, as it showed the startup timestamp for the replica’s pool if that was more recent than the actual creation time of the replica.

This issue was fixed, and the timestamps are showing the correct values now.

scripts

The system-test build target provides a self-contained dCache deployment for development or evaluation purposes. Due to an oversight, its GSI xrootd door tried instantiating the access-log plugin redundantly. This was fixed with this release.

webdav

When users request a macaroon via an HTTP POST request targeting a specific path, a caveat is created that restricts the macaroon to that path (requests to / result in a non-limited macaroon).

Commit 99c726e3 resulted in users getting back a non-limited macaroon for every request. This issue was fixed with this release.

Changelog 5.0.4..5.0.5

b23feaa86f: [maven-release-plugin] prepare release 5.0.5
019e4d1339: systemtest: fix OpenSSL DN format change
6ca716bcba: systemtest: remove reference to access-log plugin
8c8ab4a072: webdav: fix path-to-caveat for macaroon minting endpoint
b604835f59: gplazma gridmap plugin: compare DNs ignoring letter case for attribute names
32399dd349: webdav: fix NPE when Kafka notified file deletion
4a595d7a54: [maven-release-plugin] prepare for next development iteration
6cc5119357: pool: report correct replica creation time to sweeper

Release 5.0.4

admin

he current support added direct command execution capability and it works with support for semicolon (;) separated list of commands like so: ssh -p <port> user@example.com "command1; command2; command3".

alarms

Pool errors involving a fatal repository fault, for instance, can be sent now as an email alarm without having to send all pool disabled alarms.

billing

Better formating for storageInfo key, when events send with kafka producer.

dcache-view

Troubles when using firefox and/or safari to browse dcache-view were reported. This is now fixed.

gplazma

The JAAS gplazma plugin no longer logs a stacktrace on bad configuration.

pool

The current release improved error messages making them clearer by avoid using the same error message in multiple places.

The current release boosts performance for ceph pool.

The current release fixed lookup for canonical hostname for IPv6 addresses and now secure HTTP transfers work over IPv6 and problems are easier to diagnose.

The current release improved error message (previously “Could not create mover”) to provide more information about why the mover could not be created.

srm

A stack-traces fixed for concurrent updates in pin-manager and similar (expected) failures.

transfermanager

When transfermanager returns an error to the caller (e.g., WebDAV) if there was a problem starting the mover. This message did not include any details describing on which pool this failure occured. This is now fixed and HTTP TPC failures in which the pool does not start the mover now include the pool’s address in the error message. This allows admins to investigate further.

Now Third-party transfers fail if the client is requesting to copy a file from dCache that has not fully been uploaded.

webdav

Disabling basic authn should not now disable macaroons. The current release fixed webdav.authn.basic and frontend.authn.basic so that setting these configuration properties to false no longer blocks macaroons from being accepted in the HTTP Authorization header.

The current release impoved error messaging for unauthenticated request.

IllegalArgumentException exception is fixed now and attempts by a client to copy a file that has not fully been uploaded results in a clear error response.

The current release added switch to reject macacroons sent unencrypted and now following security recommendations sites may configure dCache to reject any macaroons send over an unencrypted channel. The default behaviour is to continue accepting macaroons sent over an unencrypted channel to avoid breaking existing deployment.

xrootd

The current release fixed access logging when xrootd door is configured with HAproxy.

Changelog 5.0.3..5.0.4

4a0c3d0: [maven-release-plugin] prepare release 5.0.4
e033d6f: webdav/frontend: add switch to reject macaroons sent unencrypted
35e0d9a: xrootd: fix access logging when xrootd door is configured with HAproxy
ac195b1: webdav/frontend: disabling basic authn should not disable macaroons
7ca1116: srm: do not log a stack-trace on expected Exception errors
ba8079f: transfermanager: fail third-party copy if the file is still being uploaded
325caee: webdav: fail COPY early if file is currently being uploaded
0b580ee: transfermanager: abort transfer if there is a bug
086a416: gplazma: JAAS plugin logs a stack-trace on misconfiguration
869ca7e: transfermanager: include pool address in the mover start failure message
ecdad0d: pool: update error messages to make them distinct
6557695: pool: avoid using the same error message in multiple places
3a91693: alarms: add pool dead alarm
49af685: pool: fix lookup for canonical hostname for IPv6 addresses
dc6fd90: admin : add direct command execution capability
cefff6c: pool: grow file prior FTP upload
3ffc516: [maven-release-plugin] prepare for next development iteration
78146ac: pool: don’t update atime on flush
ee4ac8a: pnfsmanager: do not emit IN_ATTRIB event on atime updates
edb882d: scripts: fix ‘dcache pool yaml’ command
48b54c8: webdav: 401 for unauthenticated requests; message in status line
345d322: dcache, frontend: release dcache-view version 1.5.3
4200c68: door: fix issue 4551 (wring storage)
57515f1: dcache: update kafka-client lib version to 2.1.0

Release 5.0.3

dcache-frontend

The current release added documentation concerning restores.

ftp

The current release fixed MLSC command for non-small directories and Globus is now able to list directories with > 100 directories.

xrootd4j

The current release updated the xrootd4j including the following fixes and improvements: add ERROR status to tpc info, change the protocol version to int, prevent NPE when constructing error response, fix path handling in move request, correctly handle multiple authn protocols as indicated by server, handle correctly IO/Security exceptions on credential loading and distinguish correctly between kXR_wait and kXR_waitresp.

Changelog 5.0.2..5.0.3

359b183: [maven-release-plugin] prepare release 5.0.3
ca48ef5: ftp: fix MLSC command for non-small directories
d113cb5: dcache-xrootd: remove mv request hack
76e846c: dcache-frontend: add documentation concerning restores
ce48a8c: pom.xml: update xrootd4j dependency to 3.3.4
b3c20ee: dcache-frontend: undefined suid parameter on transfers should be NULL not “null”
72338e6: [maven-release-plugin] prepare for next development iteration

Release 5.0.2

billing

Database connection loss now is reported for billing.

docs

Improve documentation by clarifying requirement gplazma.ldap.try-uid-mapping option in cookbook-transport-security.md.

nfs

The NFS door assumed that routable IP address, like 130.199.49.35, in general can’t access private subnet, like 10.1.1.1. This assumption was not always true for all sites and ended up with non functional pNFS deployment. This is now fixed.

pool

The current release improved performance for CEPH based pools and fixed regression affected chunked uploads.

The IO chunk size can be configured now for read as well as for write.

webdav

The current release fixed the problem where all but one requests fail, if multiple concurrent PUT requests have directories in the path that do not already exist.

Changelog 5.0.1..5.0.2

2d1f0e7: [maven-release-plugin] prepare release 5.0.2
bd9ac6d: nfs: do not filter device’s IP addresses based on site locality
b3102cd: dcache: wrap billing data source with AlarmEnabledDataSource
8499f47: common: fix random data generation in TimeseriesHistogram unit test
0ebb667: docs: clarify requirement gplazma.ldap.try-uid-mapping option
09ac758: webdav: work-around Milton racy API for creating collections
829a0d6: webdav: fix name of root
c3a548d: pom: use rados4j–0.0.3 with ARM64 support
b730cf4: [maven-release-plugin] prepare for next development iteration
e14cef6: pool: let http mover respect pool.mover.http.chunk-size
0f7763e: pool: fix HTTP chunked upload
5940b45: pool: grow file prior HTTP upload

Release 5.0.1

dcache-view

The following new functionalities have been added for dCache View: Now it is possible to use macaroon for file sharing. Files can be shared by sending the generated link, QR code or macaroon for the files to the person you want to give access to your files. Gravatar request is now make optional and how the images are stored are now more efficient to reduce the number of request made.

gplazma

Since update to newer BC and voms-java-api libraries sites report VOMS certificate validation errors like This is now fixed.

nfs

The current release fixed the bug introduced by ByteBuffer#limit, which is used instead of Buffer#limit.

srm

The dcache ports command now includes the srm’s TLS/SSL interface.

Changelog 5.0.0..5.0.1

f6d2455: [maven-release-plugin] prepare release 5.0.1
b870a14: pom: use nfs4j–0.17.10
a3c86b1: gplazma voms plugin: add trust anchor refresh paramater
cb15ef8: srm: include TLS/SSL port in ‘dcache ports’ command
5eb104c: [maven-release-plugin] prepare for next development iteration
b770e51: dcache, frontend: release dcache-view version 1.5.1

Release 5.0.0

Admin

A new property in the frontend frontend.authz.unlimited-operation-visibility now controls visibility of operations exposing file metadata. The default is false, meaning non-admin users can only see file operations for files which they own or which are anonymous. Setting it to true allows everyone access. (267d937c79).

Monitoring information exposed through the HTTP GET method is now available to all users and not only admin role users. (32597dc77a).

The admin data fields like the lists of pools, groups, units, etc., are now sorted by default for the admin REST API. (4928eff71d).

The dCache admin ssh server now supports kerberos as an authentication mechanism (along with password and publickey).

cbab40a841 added the following property to configure admin ssh server authentication:

(any-of?kerberos|password|publickey)admin.ssh.authn.enabled = password,publickey

The keytab’s location can be set under

admin.ssh.authn.kerberos.keytab-file = /etc/krb5.keytab

Alarms

A bug impeding reception of email alarms when the XML database is used has been fixed.

DCAP

Improved features: when using dcap URL to create a file or a directory, they are created with dcap get desired file permissions.

Frontend

dCache now supports more scientific file formats: HDF4 and 5 files as well as ROOT files are now identified and treated as such.

The new configuration property (one-of?true|false)dcache.enable.authn.anonymous-fallback-on-failed-login = true allows modifying the behaviour of the frontend in case of failed logins: dCache has a hard-coded “feature” where a user providing bad authentication (e.g., wrong password, expired OIDC access-token or macaroon) is treated as the anonymous user. This has proved counter-intuitive, as wrong/expired credentials often appear to succeed for some operations (e.g., directory listing), while failing others (upload/download). Providing the new property allows to set a fail-fast behaviour in those cases, providing a quicker response to users.

To support inotify events, a new plugin for SSE is introduced. Clients can discover changes in dCache namespace using an interface modelled after the inotify(7) API (See dCache book for detail).

dCache View is updated to a new version (v1.5), see dcache-view repository for new feature details.

FTP

Bug which have been fixed:

The leaking server sockets issue , when a client aborts a proxied transfers with kafka ebnabled is now fixed. No further server sockets leaked when a proxy is being used, Kafka notification is enabled, and the client aborts the transfer.

Improved features: Improve date value formatting when sending billing events via Kafka.

gplazma

The credential information (e.g., distinguished name) is now logged for x509 certification chain validation and FQAN extraction failures. (9c39e149e0).

Large numerical value gids may be used to define roles fro groupid (gid). (11b34011ae).

Wildcard match of FQANS is possible for the VO group (vo-group.json) gplazma plugin. 173dca3a96).

A new role, “observer”, is defined and available for according read-only access to system or file information. (4aa440ab2a).

The Storage AuthzDB file format is updated to accept an optional ‘max-upload=<value>’ element after the ‘read-write’ or ‘read-only’ value. The label is optional. If present, the value describes the maximum file size the user can upload. (e3dce67083)

As some newer authentication mechanisms embed usage limitations; i.e., a user may authenticate in a way that limits what that user can do (E.g. SciTokens) New authentication plugins have the possibility to specify a Restriction as part of the authentication process. Existing authentication plugins are supported as before. (204024b9e8).

A new configuration option has been introduced to capture all information about an OpenID-Connect provider, which is some external service that dCache users can authenticate against.

This configuration property is a map. Each entry of the map associates a nickname with information about that provider. The nickname is used when logging problems with the provider. The information is the URI of the issuer endpoint. This must be a valid URL that starts ‘https://’.

The following example associates the nickname ‘google’ with Google’s issuer endpoint.

gplazma.oidc.provider!google = https://accounts.google.com/

(bab4e635ac).

History

Error handling in the history service was improved.

Info

The info service now publishes the time that information was collected along with the actual data. The timestamp is available via the last-updated attribute.

Info clients (such as info-provider and storage-report) are now informed of the number of files stored in a space reservation.

NFS

When pNFS client uses flex_file layout IO errors with pool (data server) are reported to NFS door. The erros can be interpreted as:

NFS4ERR_NXIO: The client was unable to establish any communication with the storage device.

NFS4ERR_*: The client was able to establish communication with the storage device and is returning one of the allowed error codes.

PNFS Manager

A user with a macaroon that authorises them to upload data into a particular directory will now also be able to create parent directories to achieve uploading the data.

A bug that prevented get file checksum from working in some cases was fixed.

Pool

Fixed pool repository space accounting leak on failed restores from tape (815ce3eb6a).

Added Cross-Origin Resource Sharing (CORS) support for HTTP requests (049c87a814) required by dCacheView.

Fixed HTTPS redirected transfers by returning pool canonical hostname in the redirected URLs. (7f81b8e79d).

Fixed stopwath error to ensure that IO-statistics collecting is more robust, avoiding stack-traces with the message ‘This stopwatch is already stopped’ (86ede8a240).

Better handling of HTTP 3-rd party transfers - improved logging of exceptions (a98d667c16), increased socket timeout for GET requests (845cfe0bda). Improved error logging in billing by using exception calss name if exception has null message (24de520285).

Removed stack-trace logging of checked exceptions on P2P failures (7a570355fa). Fixed pool runtime configured size regression (f5ba0103ea).

Updated HTTP 3-rd party copy to support retrying GET and HEAD requests for better ineroperability with DPM (d0a621c775).

Updated FTP mover to log additional information if it detects partial transfers (e725f7b9e7). Dropped subject from StorageInfoMessage (0e60cdcaaa). Fixed regression when restoring files from tape (7cdcf4e0a7). Fixed NullPointerException on flush when using Kafka to collect billing records (4e396b9234). Fixed protocol movers to handle out of disk/out of capacity errors.

Eliminated stack trace generated by bad input for the following admin commands:

queue activate
queue activate class
queue remove class
queue suspend class
queue resume class
queue remove pnfsid

(0faa607806).

Added support for Content-MD5 request header (4d954e6b5f).

Updated HTTP mover to report errors as HTTP status message phrase so that clients that log the status line now provide their users with more detailed information about what caused a transfer to fail (6fcaeca34c).

Fixed regression that broke dcache pool convert command (80461b2f9a and 80461b2f9a).

Introduced a retry loop to retry file attributes update in timeout to pnfs manager (8c60877527.

Pool Manager

Select Read Pool requests for which the user does not have enough permissions now do not affect other requests any more.

Several smaller bugfixes for Pool Manager also went into this release.

Resilience

Bugs which have been fixed: (1) an error is no longer reported when trying to handle a broken file which has already been unlinked; (2) the entire pool scan no longer fails when one file in the list is not resilient or has no locations; (3) filters referencing invalid pool names no longer cause scan cancel to fail.

Improved features: (1) command retry errors immediately reprocesses the most recent failed file operations; (2) the command pool ls now displays the number of file operation errors encountered during a given scan; (3) the list of pools is now sorted by STATE (RUNNING, WAITING, IDLE) and then by pool name in ascending lexicographic order; (4) the inaccessible command now has options to check the status of the job, to display the current contents of the ‘inaccessible list’ file for that pool, and to clean up/delete that file; (5) ‘referring pool’ has been added to the inaccessible alarm to enable grep’ing the resilience log for a given scanned pool.

SRM / SRM Manager

Fixes in gridsite delegation storage handling - fixed querying validity of delegated credential stored on the gridsite end-point allowing clients like davix-cp to work (839604e45f) with dCache; fixed handling of delegated credential with VOMS AC that expires before the X.509 (54658383d1); imporved error reporting (41976be12d); added add gridsite delegation interface access-log (5392271fcf).

SRM client has been updated to support X509_CERT_DIR environmental variable (ed8b86e604).

Fixed handling of duplicate SURLs by SRM client (36b9e0c7d6).

WebDAV

A lot of work has gone into making 3rd party copying functionality more robust and scalable.

XRootD

Third-party copy was introduced in 4.2, and continues to be improved. For further information on configuration, please refer to the documentation in The Book (5.0).

Bug fixes and improvements: (1) the correct error (kXR_NoSpace) is now returned to the client when there is no more disk space; (2) xrootd now fails fast if the MaxUploadSize is supplied, and the file is too large; (3) the xrootd door spring configuration no longer fails to load when kafka is not activated; (4) the ‘stat’ request now supports both open file handles as well as paths, enabling use of the --zip option; (5) dCache no longer logs a stack trace when a client requests a file be created, the parent directory does not exist, and the make parent option is omitted; (6) a source path containing a query part on a mv request no longer causes the request to fail; (7) a potential race condition preventing directory listing now is correctly handled; (8) support for the ‘tpc’ query on the pools has been added in order to comply with the newer (4.9) XrootD clients; (9) it is now no longer necessary nor correct to list ‘access-log’ among the xrootd plugins; this log handler is added automatically as it is for other doors; (10) file handles and query strings are now included in the access log information; (11) logging of failed authentication is improved to include more useful information, like the DN; (12) it is now possible to identify all entries in the access-log from the same TCP connection via a session identifier.

Zookeeper

The internal Zookeeper version was updated to 3.4.13.

Changelog from 4.2.0 to 5.0.0

4631292f51: Revert “docker: Add a way to create docker image”
d411179bb7: [maven-release-plugin] prepare branch 5.0
48868f886e: docs: include CSS for profiled output
c65c5a39d6: dCap -> DCap
5ecd79a82a: pool: add session to xrootd activity
3046573a15: xrootd: fix logging of failed authentication
b13dd2b29c: xrootd/pool: add extra information to xrootd access log file
12e2cdd75d: pool: stop using deprecated API from netty
1d454088cb: utils: remove historic code from diskCacheV111.util.Adler32
cfaefaaba5: docs: TheBook. Also modify child selector pre>code to have auto overflow (scroll).
a9eadd51a6: docs: Minor improvements to install.md
ebaf8076f4: change pre code overflow to auto so that extra long lines are horizontally scrolled.
2c6c4a15f9: dcache-xrootd: hard-code order of access log handler in pipeline
c388d550e1: webdav: fix proxied partial (vector-read) GET requests
0ba973a103: nfs: overcome the 16 group limit of AUTH_SYS
4a9c7206e7: docs: restructure README.md to promote ‘The Book’
63a037ed2e: docs: removed outdated maven archetype option
815ce3eb6a: pool: fix pool space accounting on failed restores
abeaa757e5: docs: Merged OpenID Connect info into gPlazma page
ce15eeaca0: docs: fix archive filenames when generating without profile
65107ac647: docs: move assembly directory to correct location
8975179b97: resilience: fix NPE if file unlinked when resilience processes a broken file
0c8a33e3e4: dcap: fix another potential restriction by-pass
9d4354a9a0: dcap: use path when requesting changes in with PnfsManager
1873fa41bf: docs: add basic navigation
196c5af6ce: ldap: search user by uidNumber attribute if only UidPrincipal is provided
e8d8dbac46: docs: added link to admin service
8cb9eb8fd3: docs: fix text in introduction. Added srm. srmmanager ans spacemanager all linked to config-SRM.md
a02c60a809: docs: stage protection Issue: https://github.com/dCache/dcache/issues/4477
0375c56c15: ftp/webdav: fix bypass of restrictions
e43ad0ebc0: docs: fix classic SE migration guide
0f253b2491: docs: fix reference to admin ‘cd’ command
f61890e942: docs: unify reference to config-message-passing.md
2e69719b3f: docs: merge config-cellpackage.md into config-message-passing.md
af46b787af: Update config-hsm.md
9841279a6e: docs: remove obsolete references to dcache.broker.domain
fa51c31fcd: common: fix email principal string representation
f6d3d95bb3: docs: fix typo in postgres service example
7a795e2c79: dcache-xrootd: add missing query support for tpc on pools
725a30a944: alarms: fix persistence.xml configuration
0ee97b3185: pom.xml : revert back to 3.3.3 version of xrootd4j
553056da1e: dcap: fix permission propagation with DCAP
1440a469fd: docs: drop “The ….”, copy admin to config-admin.md
9ba80a6ce3: docs: Imported HSM plugin page from Github wiki
308edcaf58: dcache-xrootd: handle possible race condition in directory listing
d74ea0c250: docs: remove the HARDWRAPS generation option
b56133e9db: docs: handle ‘detached head’ checkouts when building github link
1674cb743b: docs: fix reference to hoppingmanager
427e74a661: docs: added missing-files service description
8ce1bf0edc: docs: replace cd with \c
0aedbeae8b: docs: replace cd with \c
7068c0c639: docs: Solved Github markdown rendering issue on readme.md
d96bbcb4e1: docs: edited roles section of gplazma
6f441c95a1: docs: Updated documentation of standard ports
a40054c52c: nfs41: update reset pool command to issue cb device notify
08a706ca95: documentation (TheBook): edit frontend, history, gplazma
67f398dbdd: docs: add an ‘Edit Me’ ribbon to dCache pages
72ef6933f6: libs: use nfs4j–0.17.8
83ff307b29: docs: describe dCache pools with CEPH as a backend
c1f651827a: test: add test to validate pool’s runtime configure limit
09d4b376ab: docs: TheBook remove all trailing white-space
8ab972ff0f: docs: Add assembly stage to produce output archive
3b97acc1b6: libs: use postgres-jdbc 42.2.5
c9d4b927ae: docs: add dCache.org profile
9b800c317c: docs: Removed outdated central flushing page
6d38fa2a1f: documentation : fix markdown
10848b4fe6: documentation: some more work on admin section
52cdd9aff8: docs: Link fixes, renamed config-resilience to lowercase
249554a345: docs: add custom header/footer and CSS to TheBook
d8fed7ecbc: docs: add build task to create HTML version of The Book
0008d573bc: docs: Some refinements in the Preface
cc092b14be: the book : add to admin and SRM sections
e7c138fffe: spellcheck
d2c56741f7: the book: remove references to ssh1, rename Securiting to Securing and change “Getting in Touch” to “Getting to know” :)
f4e648b7fc: nfs: use server builder when door is constructed
ab9be7546d: The Book: fix typo
c5185b6f6d: made minor changes to intro and added description of kerberos authentication mechanism in admin ssh server
d6f58b4075: pom: use official release of jacoco
1623ebebdc: pom: use jacoco with java11 support
8f045ec447: dcache-frontend: optimize delivery of aggregate sweeper histograms
a8f838941c: common: fix histogram metadata merge
c246cab1c5: dcache-frontend,dcache-history: revisit NPE fix
851cc59dc9: nfs4: only block pool selection on the first attempt
eb662a97df: restful: Return not Found/404 for non existing pool.
dd2bf55dd2: dcache: release dcache-view version 1.5.1
a941746bbc: nfs: reset pool selection task if pool disabled before redirect
9c39e149e0: gplazma2: Log credential information on x509 cert. chain validation and FQAN extraction failures.
df931cde9a: nfs: ignore JdbcFs errors when constructing acceess log entries
8c9f0ad857: frontend: filter out any IP addresses in door information
bdd96052fb: libs: update nfs4j to version 0.17.7
d1baecfe29: nfs: handle chimera exception on remove of a missing file
049c87a814: pool: add CORS support for HTTP requests
7f81b8e79d: pool: use hostname in HTTPS redirection URL
c63b52bf76: libs: use java–11 compatible aspectj–1.9.2
640796b795: dcache: release dcache-view version 1.5.0
64fbd38cef: dcap: fix NullPointerException:
b319876458: restful: Rename operations to have unique operationId.
9071d4d9f5: ftp: avoid kafka bug, make shutdown more robust
80c7260883: dcache-history,dcache-frontend: check for serialized error when handling pool data request messages
86ede8a240: pool: fix stopwatch error
d9447453ce: common: fix bug in CountingHistogram index computation
c04714455b: libs: update to nfs4j–0.17.6
d67f348e48: cookbook transport security: fix markup and language
e9a1dfc5f6: transport security: CAA records
839604e45f: srm: gridsite fix querying validity of delegated credential
91244c8c74: chimera: support origin tag discovery
1316bfd5a3: Revert “chimera: fix postgres 95 optimization regression”
a9abaa7863: chimera: fix postgres 95 optimization regression
61f765d41f: Documentation: TheBook/config-xrootd.md
6c37e7a294: nfs: increase request retry delay when selecting/starting pool or mover
267d937c79: dcache-frontend: provide switch to control visibility of file operations for non-admin users
bb7649fd73: webdav: adjust minimum validity after requesting delegation
54658383d1: srmmanager/webdav: consider VOMS AC validity of delegated credential
4ac0b589d0: ftp: make performance marker task robust.
333d09a28f: ftp: avoid NullPointerException if adapter is not connected
fca9f993d1: webdav/frontend: add mime types for more scientific file formats
dc624af2ce: cookbook-transport-security: fix typo
a98d667c16: pool: HTTP TPC rework exception logging
845cfe0bda: pool: increase TPC socket timeout for GET requests
f0ecdacd75: scripts: Avoid findbugs memory errors
41976be12d: srm: fix credential store logging
24de520285: pool: update log status using exception class name if no message
25c2c80c24: docs: How to use kafka with dCache
9085ad3ef0: storagedescriptor: update information based on WLCG feedback
a2c4f93487: xrootd: strip off query part from kXR_mv source
895e88c792: webdav: fail TPC request early on unknown hostname
97579c0697: nearline-provider: do not propagate thread interrupt flag
b169df497a: poolmanager: fix NPE on unknown host
1e94e9bb4f: webdav: improve logging of TPC requests
54369c76ea: docs: describe gplazma mutator plugin
65f692867a: docs: describe gplazma-jass plugin
02aa1b424a: libs: upgrade to bouncycastle 1.54, CANL to 2.5.0 and voms-api-java to 3.3.0
75b9a4de20: libs: update aspectj to java11 friendly version 1.9.1
c839a3f381: zookeeper: remove ZooKeeperConnectionExceptionAspect
52be70e5a0: libs: use zookeeper–3.4.13
d48a8b6950: xrootd: fix broken configuration property
3a4573d3e5: Revert “packaging: use private BC 1.50 release that provides JSSE compatible handling of key agreement secret generation.”
2ddfabeb18: src: move away from ListenableFuture
4422bf0466: dcache: use getAddress for uniform client IPs in Transfer info
3ec201ee41: docs: clean up ‘cd’ (and adding ‘\c’)
684f4f37e4: docs: clean up ‘cd’ (and adding ‘\c’)
16f8b1014a: docs: clean up ‘cd’ (and adding ‘\c’)
e755639780: libs: update to commons-compress–1.18
7826c26528: docs: minimal postgres version is 9.2
65505cc8d8: docs: remove reference to Solaris packages
d5cdb1720e: Markdown & typo fix
8fe4e1aea2: packages: remove solaris packages
cf896feae7: Create cookbook-transport-security.md
042ec5e59d: book: fronted, history, xrootd TPC, dot commands, and various indexing changes
bf0d2c623c: book: cleanup alarms, billing, resilience
ea100ecc7f: doc:clean up command line
11b34011ae: gplazma: support large gid values for roles
7a570355fa: pool: P2P failures trigger stack-trace
e655d926a5: webdav: obtain FQAN from X.509 credential for gridsite
0ee7f96d57: transfermanager: fix missing path
5b90514c15: doc:corrections
f5ba0103ea: pool: fix pool’s runtime configured size regression (b70b0d9)
aea18ac9f3: httpd: add path to context/transfers.json
0fcb1926a1: ftp: java.lang.IllegalStateException: Cannot send after the producer is closed.
5b65b41923: core: avoid sending bad macaroons to gplazma
60cb2ab08b: webdav: update access log to record macaroon request details
50ab46b5c2: macaroons: include macaroon id in error message
f797e2f7bc: core: provide better feedback and logging if a macaroon is rejected
e77c317092: Added individual entries for each service in index.
290b91a4e9: docs: Remove info docs about pre–1.9 config
7c72a06a3f: Added individual entries for each service in index.
002c55d569: docs: document all_root nfs export option
07d23e3e74: updating intro and intouch chapters
1230bea81f: dcache-frontend: fix array out of bounds exception in cell info service
a9cf7c6a13: Update intro.md
7a3efaf13d: Update intro.md
4d77e6ec17: clean up links
62332877a2: Update intro.md
803d66c451: Update intro.md
f3702019d9: Update intro.md
45efd38270: changes for book
f97b47179d: pool: fix numerical out-of-range error
5392271fcf: srm: add gridsite delegation interface access-log
052e7a1d67: macaroons: fix logged id
cc2079aa1e: core: avoid stacktrace on arbitrary CacheException
d0a621c775: pool: update HTTP TPC to support retrying GET and HEAD requests for DPM
ececd7871d: pnfsmanager: allow restricted user with UPLOAD to create parent directories
32a166949b: xrootd: remove spurious stack-trace
f0c2ed02fb: xrootd: add support for kXR_stat on open files
192ceaee1c: xrootd: update to xrootd4j dependency to 3.3.3
b6660d2af9: packaging: use private BC 1.50 release that provides JSSE compatible handling of key agreement secret generation.
3d56d28b59: dcache-frontend: fix error message for IdResource
b473eaf6ff: ftp: better address selection for cross-family passive proxied transfers
00fa700c83: webdav/frontend: make anonymous fallback on bad login optional
9a5efb73d2: info/space-manager: monitor number of files in reservation
2dbac4d573: info: display the timestamps when metrics were collected
9488cf474b: poolmanager: do not squash request if state is not allowed
1d08b62dba: libs: use nfs4j–0.17.5
6e90136c12: dcache-xrootd: add missing kafka property
1f1113fa5d: dcache-frontend: add path filter to transfers
d6337e2f12: dcache: add path to transfer information
6c67943842: vehicles: fix serialization regression in FileCorruptedCacheException
869f682936: dcache: Adjust Date formating of Timestamp value for date key for kafka producer
978d9f645b: libs: update to use spring–5.1 release
173dca3a96: gplazma2-fermi plugin (vogroup plugin): allow for wildcard match of fqans
5119c2d283: Update README.md
bfe6f03460: Update preface.md
a41b71a788: Update and rename readme.md to README.md
439b0fe75a: reformat readme.md to book
ff790ee07d: Update readme.md
1fb1e12be3: add readme.md to book
32597dc77a: dcache-frontend: remove admin restrictions on GET and filter transfers on uid if not admin
05d2ecc493: pom.xml: update to xrootd4j dependency to 3.3.1
d07e79e7ca: Revert “dcache-dcap: add uid/gid to transfer info for plain/anonymous dcap”
e725f7b9e7: pool: instrument ftp mover to show partial transfers
0e60cdcaaa: pool: do not include subject with StorageInfoMessage
932e53bdfa: ftp: fix regression in unit-tests
88022876e9: libs: update to nfs4j–0.17.4
9ef883a988: dcache: Creating multiple KafkaProducer instances results in ‘Too many open files’
5b0a1e05fb: webdav: use TLS credential directly for gridsite
3ce0c39393: dcache: add configuration for the Kafka producer timeout
e149c0e305: webdav+transfermanagers: support TPC pull with targeted macaroons
e44fef2e06: Revert “dcache: Creating multiple KafkaProducer instances results in ‘Too many open files’”
93186ef6af: dcache: Creating multiple KafkaProducer instances results in ‘Too many open files’
e07ed8bac9: TransferManager: remove state history class and corresponding table responsible for storing request state changes. It is not used, but may grow rapidly in database.
4aa440ab2a: gplazma-role: add observer role
270b7827c0: dcache-dcap: add uid/gid to transfer info for plain/anonymous dcap
17b5f57e25: systemd: Add /etc/security/limits.d/92-dcache.conf in the dcache systemd unit and generator.
eab3977d89: frontend: add events.html an events test client for webbrowsers
ce1ba5c64d: frontend: add SSE plugin to support inotify events
dae3cf0f09: frontend: fix broken directory qos reporting
e3dce67083: gplazma: support max-upload in storage-authzdb files.
f08a82720f: webdav: avoid throwing any exception when listing a directory for PROPFIND
da1b6259f2: nfs: Instrument NFS door to support inotify events
705a69fc5f: pnfsmanager: add instrumentation to support sending events
b3e803c421: core: add an EventReceiver that sends inotify events
c0ec01c269: frontend: update SSE selection support
6b05aafc30: xrootd: Add support for MaxUploadSize LoginAttribute
884b98910e: dcap: add support for MaxUploadSize LoginAttribute
2d0c003d25: ftp: add support for MaxUploadSize LoginAttribute
594948859c: webdav: add support for MaxUploadSize LoginAttribute
0f8e7ad13f: macaroons: add max-upload caveat
e18020821b: common: add standard byte-size parser
99c726e32c: webdav/macaroon: Fix macaroon creation with multiple path restrictions.
204024b9e8: gplazma: add support for authentication plugins registering restrictions
6ba3c3366d: webdav: add cdmi and dcache-view tags to default properties
1f32e50971: frontend: expose list of available doors
6ed6883e8b: ftp: add ability to log client-aborted transfers
2e81036724: nfs: make timeout of pnfshandler configurable
7cdcf4e0a7: poolmanager: fix staging files from tape
779b458844: dcache: release dcache-view version 1.4.5
6e9d76e22c: dcache-xrootd: add necessary gsi properties for tpc credentials
5069488e79: nearline-provides: do not interrupt processing thread on cancel
5d4515d87a: nfs41: invalidate open-state on layoutget if file is removed
fba83d52ae: docs: markdownify glossary
439834e244: docs: update acl chapter
4e396b9234: pool: fix NPE on flush
a2e18c0d0f: webdav: always respond to OPTIONS request
b7d0836d48: dcache: release dcache-view version 1.4.4
f964703ab9: docs: fix format of index.md
95478a3056: core: ensure pool/poolmanager communication receives errors
af9238bff2: frontend: add targetQoS for not-yet-flushed tape files
8c579ab237: pool: update xrootd handler to support OutOfDisk exception
c8afca7be0: pool: update dcap mover to handle OutOfDiskException
299b435d35: pool: ftp mover suppresses including the IP address when out-of-storage
0a87fa7850: pool: update http mover to return 507 status code on out-of-capacity
5eb12297d2: authn: add MaxUploadSize LoginAttribute
c51663e73a: vehicles: allow doors to specify a maximum file size
706da8f5be: pool: support maximum size when creating a replica
4857f8c813: dcache: release dcache-view version 1.4.3
36b9e0c7d6: srm-client : fix handling of duplicate SURLs
21cdebdb98: docs: add export section to nfs chapter
88c2ebdfb2: pool: fix NullPointerException
81249c9ba7: docs: describe DNS TXT records for nfs id mapping
845700d0c5: docs: revert zookeeper config chapther update
0221e455a8: poolmanager: fix NullPointerException when staging files and reporting hits
b0170e5746: libs: update hazelcast to version 3.10.4
bd49fc1976: gplazma: oidc fix FullNamePrincipal creation
04c4c230fe: docs: Imported HA / replicable services from wiki
aa56d42000: docs: Imported message passing from wiki
75adff92f3: docs: Imported Zookeeper page from wiki
32bd7ac998: docs: Imported Resilience page from wiki
6359c9d951: docs: Recreated main index
ee3eaf4331: libs: update jetty to version 9.4.11
0faa607806: pool: ‘queue’ admin commands not the log stack-trace on bad arguments
c3868cc593: dcache-xrootd: fix third-party billing records
926dfa34b0: docs: remove obsolete chapter about x86_64 deployments
4d8ea3ef60: docs: chimera: clarify PnfsManager, Chimera and db relationship
e9d1615c69: docs: markdonify cookbook ToC
69ff42a865: docs: chimera: remove redundant warning on update of default files
be5308cbeb: docs: remove obsolete multiple-pnfsmanagers chapter
020fc3f17a: docs: config-nfs: describe exports.d directory
5dc9dd3007: docs: update and markdonify config-nfs chapter
00ce43ab30: docs: markdownify config-chimera.md
e287245f87: docs: remove docbook build instruction
977bbc8744: docs: markdonify config chapter’s ToC
2d2e0625e4: docs: fix zookeeper configuration link
bd93447ac8: docs: first pass-through introduction and install guide.
7ec4483204: docs: add zookeeper configuration chapter
90825470f0: docs: remove obsolete lagacy pnfs configuration
6693faf417: docs: move dcahce book into dcache source tree
91d81b79a2: ftp: fix NullPointerException
8e02472d63: ftp: fix scope of used pool stub
bab4e635ac: gplazma: oidc support OIDC providers with non-root issuer URLs.
20de92c346: chimera: do not treat CockroachDB as PostgreSQL.
c93949d506: pool: retry allocation on OutOfDiskException from Allocator
41dcc64d85: pool: refactor Allocator interface and implementations
13fa2bef38: webdav: try fetching HTTP status code when pool disconnects an upload
476c23038f: webdav: report pool response when relaying data
48fbb4be1d: webdav: update default credential delegation for third-party copy
ed8b86e604: srmclient: update delegation client to support X509_CERT_DIR en.var.
7d28721696: srm-server: unified logger Code Convention
7e1324958d: Revert “utils: ensure that pool and poolmanager subs set in Transfer.class”
4aea7281f1: dcache-xrootd: repair handling of delayed sync errors to client
cb3410af35: Revert “util: update Transfer should use poolStub when communicating with a pool”
f237447353: vehicles: remove unused field in infoMessage class
3a46e66a55: dcache-history: handle Gson syntax errors explicitly
c6c850f7d1: vehicles: remove backward compatibility with 2.1 and 2.16 in InfoMessage
5b8235fab4: cells: add handling of RemoteProxyFailureException nested InterruptedException to UncaughtException handler
a79e6b18ab: scripts: allow new java versions as 9, 10 and 11
9904302e9f: dcap: clean code changes
97191a6d42: ftp: add kafka to push messages
8c03fb8c54: dcache: remove unused Spring JDO class
fa8e7dc982: dcache: remove unused jndi initializers from httpd
1fcf3a53b9: core: Removed NotFilter.
791159188f: acl-vehicles: unified logger dcache codestyle
000531916d: core: replaced collection of Predicates by predicate-chain. Removed NotStickyOwnerFilter.
11a9aeb384: core: replaced CacheEntryFilter by native java.util.function.Predicate
a358b50418: docs: fix markdown formatting in README.md
48dd00d072: dcache-resilience: report file check errors instead of fail fast
db2bed33cb: dcache-resilience: add command to retry failed files
66ea9aca4d: dcap: enable possibility to push transfer events to Kafka
590cff1d7c: dcache-xrootd: add support for redirect handling during third-party-copy
c357d030e9: dcache-xrootd: add third-party support to pool (dcache as destination)
ccff945673: dcache-xrootd: add support for third-party copy to door
c147358c5f: poolmanager: fix broke commit 6df58e4d
6de13c8fc7: poolmanager: fix broke commit 6df58e4d
6df58e4dfe: src: remove historical quota manager code
2c13454f3b: nfs: add inotify monitoring wrapper
e6b3e60f31: core: add inotify monitoring wrapper NameSpaceProvider
09a45c33be: pom: use nfs4j–0.17.3 bugfix version
9171a6a7a6: core: split ConfigurationProperties into separate classes
4f63f15433: core: add wrapping namespace
50b8be9ad8: webdav: pass on status message phrase to client
52e13b9a10: ftp: update SocketAdapter to use data transfer direction enum
857c2ce5a0: ftp: describe data transfer mode consistently within the door
2b6113a1cd: webdav: note usage of response element
4d954e6b5f: pool: add support for Content-MD5 request header
73e86d9e9e: webdav: add partial support for Content-MD5 header in uploads
6fcaeca34c: pool: update HTTP mover to report errors as HTTP status message phrase
14fb9ebf59: admin: fix regression in startup
654611041f: dcache-resilience: propagate file op error to pool op and display
9914d17687: pnfsmanager: fix digest name handling in get file checksum command
c3f1b224a6: pom: update package version to 5.0.0-SNAPSHOT
fe96ba2b8f: dcache-resilience: display pool operation list in sorted order
cbaf8faba8: dcache-resilience: improve inaccessible file accounting
0028308a63: dcache-resilience: skip invalid cancel filters
303f5e70a4: nfs: log IO errors report by clients on layout return
d843d6f491: nfs: move layout driver initialization into NFSv41Door#init method
72d6279134: nfs: include pool name into PoolDS object
6e20c58284: utils: ensure that pool and poolmanager subs set in Transfer.class
bae20c30d8: nfs: propagate layoutreturn data to the layout driver
a1b09acf79: frontend.authn.protocol by default https
80461b2f9a: pool: fix ‘dcache pool convert’ command
8f76fe401b: scripts: update reference to configuration property
a2122bff9c: ftp: fix scope of used pool stub
011b3b2439: pool: fix metadata migration tool to use Path
9aed07097f: BUILDING.md: fix typos
7e17e4841d: BUILDING.md - installing a new Maven
64875efd9b: cells: remove historical MemoryWatch
3be6c54171: cells: remove historical dead code
46552da5f6: cells: do not enforce timeformat in pinboard
44fa96e66d: util: update Transfer should use poolStub when communicating with a pool
8167c60f35: ftp: always close proxied data connection if client closes their half
66d304f439: vehicles: fail-fast on invalid path
c117cdc951: util: fix invalid number of args for logging in RunSystem
8c60877527: pool: retry request to pnfs manager if timed out
81a0a97ee1: dcache-frontend: invalidate transfer when killed mover not found
78787fe8ea: dcache-frontend: add “Requires admin role” to alarms methods (Swagger)
429e8ab220: gplazma.properties: hint to enable roles
60c2c3c69a: README.md: add link to packages
cbab40a841: admin: add kerberos authentication support to admin ssh server
4928eff71d: dcache-frontend: provide default sorting on RESTful admin data fields
2b7af29035: doors: support advertising multiple addresses in LoginBroker
05335bdaf1: [maven-release-plugin] prepare for next development iteration

What’s new in dCache 5.0 Release notes

Highlights

Incompatibilities

Acknowledgments

Release 5.0.37

dcache-xrootd

Changelog 5.0.36..5.0.37

Release 5.0.36

pool

Changelog 5.0.35..5.0.36

Release 5.0.35

frontend

xrootd

Changelog 5.0.34..5.0.35

Release 5.0.34

dcache

frontend

Changelog 5.0.33..5.0.34

Release 5.0.33

frontend

Changelog 5.0.32..5.0.33

Release 5.0.32

dcacheview

Release 5.0.32

dcacheview

Changelog 5.0.31..5.0.32

Release 5.0.32

frontend

Changelog 5.0.31..5.0.32

Release 5.0.31

frontend

skel

srm

Changelog 5.0.30..5.0.31

Release 5.0.30

cell

skel

webdav

Changelog 5.0.29..5.0.30

Release 5.0.29

canl

Changelog 5.0.28..5.0.29

Release 5.0.28

config

dcache

webdav

Changelog 5.0.27..5.0.28

Release 5.0.27

srm

Changelog 5.0.26..5.0.27

Release 5.0.26

doors

pool

scripts

Changelog 5.0.25..5.0.26

Release 5.0.25

pool

webdav

xrootd

Changelog 5.0.24..5.0.25

Release 5.0.24

frontend

Changelog 5.0.23..5.0.24

Release 5.0.23

Changes affecting multiple services

pool

xrootd

Changelog 5.0.22..5.0.23

Release 5.0.22

dcap

httpd

Changelog 5.0.21..5.0.22

Release 5.0.21

srm

webdav

Changelog 5.0.20..5.0.21

Release 5.0.20

common

frontend

Changelog 5.0.19..5.0.20

What’s new in dCache 5.0
Release notes