Highlights

  • better nfs v4.x performance of state validation in highly concurrent environment

Incompatibilities

  • the mountd service is started only if NFS v3 is enabled
  • removed historic code of HSM flush manager

Acknowledgments

(tbd, this section uses markdown formatting)

Release 8.0.16

nearline-storage

A NPE on error path has been fixed.

The race condition when flushing removed file is fixed now.

Changelog 8.0.15..8.0.16

aa9bf6d434
[maven-release-plugin] prepare release 8.0.16
749bb7ccb8
nearline-storage: fix race in flushing removed file
9b9d6e2929
nearline-storage: fix NPE on error path
d522b5d708
[maven-release-plugin] prepare for next development iteration

Release 8.0.15

frontend

The current release fixed type default of RESTful pool/nearline/queues resource

Changelog 8.0.14..8.0.15

da56544ded
[maven-release-plugin] prepare release 8.0.15
f04f03b9a6
dcache-frontend: fix type default of RESTful pool/nearline/queues resource
70e662242a
[maven-release-plugin] prepare for next development iteration

Release 8.0.14

chimera

Pools that are no longer actively being cleaned were previously falsely still listed as active in the admin interface when the cleaning attempt ended due to an exception. This has been fixed.

core

If a pin is unpinned while a pin request for that same pin is still ongoing in pinmanager, pinning will not be retried.

Changelog 8.0.13..8.0.14

74db9a46fc
[maven-release-plugin] prepare release 8.0.14
57dec67417
dcache-core: let pinmanager stop retrying pinning upon unpin state
bf9ee600d4
dcache-chimera: disk-cleaner, error handling in parallel pool deletes
0cd908caac
[maven-release-plugin] prepare for next development iteration

Release 8.0.13

gplazma2-grid

An empty requires field in IGTF policy is now accepted. Files with invalid or unexpected policies are ignored.

webdav

A bug was fixed that caused a NullPointerException while sending Kafka messages.

jquery was updated to version 3.6.1 and bootstrap was updated to version 5.2.1

Changelog 8.0.12..8.0.13

530f6ed109
[maven-release-plugin] prepare release 8.0.13
58629d1bbf
dcache: NPE on removal via WebDAV and token
14815d279c
webdav: update 3rd party js libraries
7d17c92cc8
doc: add recipe to cookbook for migration/draining of resilient pools
2a4f47be9d
gplazma2-grid: invalid IGTF policy file should fail with ParserException
888fecc199
[maven-release-plugin] prepare for next development iteration

Release 8.0.12

common

dCache now reports RemoteTransferManager as being replicable.

dcache-xroot

A possible thread starvation and inability to allow more clients to connect hase been fixed.

webdav

Fix a bug where the WebDAV door will continue attempting to kill a cancelled HTTP-TPC transfer indefinitely if the corresponding RemoteTransferManager cell has been switched off.

The current release fixed a NullPointerException when reporting on active transfers if the client has disconnected but the door has not yet cancelled the transfer.

xroot

Documentation now offers some explicit guidance as to pool configuration for xrootd reqarding memory.

Changelog 8.0.11..8.0.12

2c4134828e
[maven-release-plugin] prepare release 8.0.12
6c1b92bd5b
dcache-xroot: interrupt doOnOpen thread in the event of channel inactive
4122a3b15e
webdav: http-tpc do not keep retrying to cancel transfer
48ad0f7042
webdav: fix NPE if client aborts transfer
9a4b63f79e
dcache-xroot: add Book section on direct memory requirements for pools*
b028c0ae2a
pom.xml: update to latest xrootd4j bugfix
d7892ccb7f
[maven-release-plugin] prepare for next development iteration
141a314453
properties: report that remote transfer manager is replicable

Release 8.0.11

dcache

A clarification on authorized keys format has been added.

Changelog 8.0.10..8.0.11

d7dba3390a
[maven-release-plugin] prepare release 8.0.11
f03030ce4d
Add clarification on authorized keys format
843aa5336c
[maven-release-plugin] prepare for next development iteration

Release 8.0.10

dcache-bulk

Runtime exceptions do not kill the queue thread now and do not leave the request in a stalled state.

webdav

Existing monitoring information does not show HTTP-TPC transfers, which may result in admins being unaware of dCache activity.

This is now fixed and The Active Transfers page (from httpd) and the transfers resource (from frontend) now include HTTP-TPC transfers.

WebDAV HTTP-TPC now supports multiple RemoteTransferManager services, which allows for high-availability deployments. The WebDAV door and all RemoteTransferManager services must be upgraded to support this new feature.

Changelog 8.0.9..8.0.10

1e11319d30
[maven-release-plugin] prepare release 8.0.10
f8020c82cf
dcache-bulk: (version 1) handle unexpected exceptions in queue
1fca124200
webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
b35d3e847a
webdav/transfermanager: support multiple RemoteTransferManager
c02d6ece3c
[maven-release-plugin] prepare for next development iteration

Release 8.0.10

dcache-bulk

Runtime exceptions do not kill the queue thread now and do not leave the request in a stalled state.

webdav

Existing monitoring information does not show HTTP-TPC transfers, which may result in admins being unaware of dCache activity.

This is now fixed and The Active Transfers page (from httpd) and the transfers resource (from frontend) now include HTTP-TPC transfers.

WebDAV HTTP-TPC now supports multiple RemoteTransferManager services, which allows for high-availability deployments. The WebDAV door and all RemoteTransferManager services must be upgraded to support this new feature.

Changelog 8.0.9..8.0.10

1e11319d30
[maven-release-plugin] prepare release 8.0.10
f8020c82cf
dcache-bulk: (version 1) handle unexpected exceptions in queue
1fca124200
webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
b35d3e847a
webdav/transfermanager: support multiple RemoteTransferManager
c02d6ece3c
[maven-release-plugin] prepare for next development iteration

Release 8.0.9

frontend, bulk

Both numeric and string values for PIN lifetime are valid now. Also, skipDirs for DELETE can be both boolean and string. All other current arguments are required to be strings, as before.

Changelog 8.0.8..8.0.9

7887ef8a2d
[maven-release-plugin] prepare release 8.0.9
30ec4454d5
dcache-frontend,bulk: fix typing of argument values
baf0d890f8
[maven-release-plugin] prepare for next development iteration

Release 8.0.8

frontend

Anonymous access receives a 401 error and doesn’t trigger a stack-trace when unlimitedVisibility is set to false.

clearOnSuccess, clearOnFailure and cancelOnFailure can have boolean and string values now.

pool

A bug is fixed where the checksum calculation would fail for empty files.

Changelog 8.0.7..8.0.8

1a412d30c9
[maven-release-plugin] prepare release 8.0.8
9fae9e8a75
dcache-frontend: protect against RuntimeError in case of denied anonymous access
551fc779da
restore inadvertently removed ‘target’ extraction in toBulkRequest()
faffec45bf
dcache-frontend: bulk-requests POST, allow either string or boolean for boolean values
e449c33482
pool: don’t treat an empty file as a sparse file
ea3e74ef2a
[maven-release-plugin] prepare for next development iteration

Release 8.0.7

ChimeraVfs

This release restores the ability to modify file levels.

info

The delay between messages sent by info was increased to reduce log messages of “DGA {…} triggering too quickly”.

pool

If the scrubber state file is empty, scrubber will now log a more helpful error message and proceed scrubbing the pool.

Changelog 8.0.6..8.0.7

5781884976
[maven-release-plugin] prepare release 8.0.7
4a2eb74fab
pool: improve handling empty saved state by scrubber
b90c4fbec2
dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
bf11ceecbb
dcache-xroot: flesh out channel inactive and exception caught
42f94b841e
info: increase delay between messages
f0d69b8c0f
ChimeraVfs: Check file level in addition to inode type when disallowing file size change
ce556c9015
[maven-release-plugin] prepare for next development iteration

Release 8.0.6

dcache

Improved documentation for kafka Producer properties.

Improved error message for inexistent statistics path.

dcache-xroot

The current relase Updated master xrootd4j to 4.3.0.

frontend

NPE is fixed when SRR when space information is not available yet.

nfs

Closing on open-for-read file now does not trigger extra layout recall.

xrootd

Rendezvous TPC without requiring a JWT token to be passed by the third-party client is possible (again).

Changelog 8.0.5..8.0.6

a92fa0cf2d
[maven-release-plugin] prepare release 8.0.6
d488867ee8
dcache:improve documentation for kafka properties
4389adec33
nfs: re-send kill only for write movers
cc04c4f5dd
dcache-xrootd: fix TPC rendezvous to work with token authorization
675302e989
dcache-core: improve error message for inexistent statistics path
2fe8b1446f
frontend: fix NPE in SRR when space information is not available yet.
b01c233ed6
dcache-xroot: update master xrootd4j to 4.3.0, and stable branches to next minor version
c5b07458ac
[maven-release-plugin] prepare for next development iteration

Release 8.0.5

script

Direct deprecation warnings are now redirected to stderr.

Changelog 8.0.4..8.0.5

2901d045e5
[maven-release-plugin] prepare release 8.0.5
b9e835c028
info: fix broken unit-test, remove redundant one
1eb775efa2
dcache script: redirect deprecation warning to stderr
bf93e96c83
[maven-release-plugin] prepare for next development iteration

Release 8.0.4

pnfsmanager

Doors no longer allow a user to discover whether or not a file or directory exists within directories they cannot access.

pom

Updated spring libs to version 5.2.20, CVE–2022–22965

pool

The pool now provides more information when a migration job was cancelled.

Changelog 8.0.3..8.0.4

5d7043b9e6
[maven-release-plugin] prepare release 8.0.4
6654c88269
pom: update spring libs to 5.2.20
3506bcd74f
pool: describe why migration job was cancelled
9f0d88d370
pnfsmanager: avoid leaking whether or not file exists
476efc5c11
[maven-release-plugin] prepare for next development iteration

Release 8.0.3

dcache-xroot

A ZTN token can now be given without further downstream tokens expressed, and full authorization of the subject will take place.

Changelog 8.0.2..8.0.3

e7fc0dc8e9
[maven-release-plugin] prepare release 8.0.3
40681c2702
dcache-xroot: Allow ZTN authentication to function as fallback authorization
fa567e2bc2
[maven-release-plugin] prepare for next development iteration

Release 8.0.2

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 8.0.1..8.0.2

f5ff9dcb2a
[maven-release-plugin] prepare release 8.0.2
ee5c3329de
srr: don’t publish link local addresses (v2)
1e386a1be7
util: treat broken file as unrecoverable error in Trnasfer class
bd8fefa54a
nearline-storage: fix request count on error path in AbstractRequest#failed
4ababb6902
nearline-storage: fix queued count when request initialization falied
3e492aef9f
pool: add bunch of unit tests for NearlineStorageHandlerTest
ee191ff429
UderGuide: describe supported checksums, add example including sha
c052e902da
frontend: fix problem with making SRR resources public
80a277d033
dcache-chimera: add sha checksum commands to chimera shell
fdbd2cbe2c
dcache-core,common: fix user having to request sha-1, not sha
7e189ce96e
nfs: enforce subject propagation on RPC level
0a0b7f6c14
dcache-core: prevent ocasional message ttl test failure
f05ff3b670
Fix typo
c2b795c275
dcache-chimera: fix cleaner batch delete exception
8280d097a4
[maven-release-plugin] prepare for next development iteration

Release 8.0.2

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 8.0.1..8.0.2

f5ff9dcb2a
[maven-release-plugin] prepare release 8.0.2
ee5c3329de
srr: don’t publish link local addresses (v2)
1e386a1be7
util: treat broken file as unrecoverable error in Trnasfer class
bd8fefa54a
nearline-storage: fix request count on error path in AbstractRequest#failed
4ababb6902
nearline-storage: fix queued count when request initialization falied
3e492aef9f
pool: add bunch of unit tests for NearlineStorageHandlerTest
ee191ff429
UderGuide: describe supported checksums, add example including sha
c052e902da
frontend: fix problem with making SRR resources public
80a277d033
dcache-chimera: add sha checksum commands to chimera shell
fdbd2cbe2c
dcache-core,common: fix user having to request sha-1, not sha
7e189ce96e
nfs: enforce subject propagation on RPC level
0a0b7f6c14
dcache-core: prevent ocasional message ttl test failure
f05ff3b670
Fix typo
c2b795c275
dcache-chimera: fix cleaner batch delete exception
8280d097a4
[maven-release-plugin] prepare for next development iteration

Release 8.0.1

Changes affecting multiple services

The OIDC ‘sub’ (subject) and ‘jti’ (JWT ID) claims are logged to access log file for WebDav, frontend and SRM doors if OIDC is used.

admin-shell

Fix the line termination printed to console. It’s \r\n now.

bulk

Request-specific pinning is enabled now, but generic unpinning is still possible.

dCache now returns 404 instead of 403 if a submitted id in GET or PATCH bulk-request is not recognized by the service.

Jobs which originally got stuck in STARTED state now complete; their failure information contains the reason for premature completion.

frontend

The property frontend.srr.public=true|false is added to frontend to control SRR resource restriction. The default value is false.

ftp

The FTP cell is killed asynchronously now after the session terminates so the door will remain responsive when many clients terminate their FTP session concurrently.

pool

HTTP-TPC transfers now prefer IPv6 address, if both endpoints support it.

If a HSM script survives SIGTERM it will now be stopped with SIGKILL and a warning is logged. This reduces expired or canceled requests staying in CANCELED state.

If a NFS mover is killed twice it’s now logged without a stack trace. This provides more stable NFS error recovery, too.

webdav

Fix handling of ‘webdav.authz.allowed-paths’ so it no longer triggers a NullPointerException if a request targets a path that is not in allowed-paths.

xroot

Change default TLS to OPTIONAL on pools so defaults for xroot TLS are uniform on pools and doors.

Changelog 8.0.0..8.0.1

df6fcc1f9a
[maven-release-plugin] prepare release 8.0.1
fc1d423b3d
frontend: make srr resources public
8fee90cce9
webdav/frontend/srm/gplazma: log OIDC ‘sub’ and ‘jti’ claims
ca3dbb82f4
ftp: kill cell asynchronously
57bceb1b2f
pool: http-tpc prefer IPv6 address
d8fe9f420c
pool: handle multiple shutdowns of a nfs mover
3ef40e0f64
dcache-bulk: aborted request gets stuck in the STARTED state
bb223954b7
dcache-bulk: pin by request id and unpin optionally by request
2b4c8a148c
pool: try to forcefully kill HSM process if needed
a4b8babd1a
admin shell: fix line termination in SshOutputStream
cdbe5ad0f2
dcache-xroot: set default TLS to OPTIONAL for pools
edf2a83165
bulk, frontend-rest: return 404, not 403, when request id does not exist
b241a4c150
webdav: fix NPE if ‘webdav.authz.allowed-paths’ disallows a request
e0b3b623d3
chimera: fix error when adding new label to a file
a8cb617f44
[maven-release-plugin] prepare for next development iteration

Release 8.0.0

Bulk

The handling of pin requests was fixed to be fully asynchronous and not time out. There was also a regression fix for aborted requests which would leave the request in the STARTED state instead of completing.

Chimera-shell

The chimera shell now supports creation symbolic and hard links.

Frontend

Transfer rate was changed to a double value so that very fast transfers do not get rounded down to 0. ‘pin’ and ‘unpin’ were added to the namespace resource POST method for individual files. In both REST and NFS (dot command), pin and unpin was modified to use the uid as request id. Bulk requests were fixed to support any combination of attribute styles (camel-, snake- and kebab-case) for a given request.

Clearer error message which does not denote a bug when CA certs directory is missing and HTTPS is configured.

The “key” field of alarm log entries is now build as a colon-separated list of keywords.

FTP

GSIFTP was reverted to use the Java SSL library instead of OpenSSL (API incompatibilities).

gplazma

It is now possible to configure a group name using the simple group prefix in the banfile and scitokens gPlazma plugins. This prefix will also work for the “test login” and “explain login” admin commands.

The oidc plugin is now able to accept a JWT access token without querying the user-info endpoint. If the token is not a JWT then the oidc plugin will call the user-info endpoint to discover the necessary information about the user.

The oidc gPlazma plugin now supports the ‘aud’ claim. The list of allowed audience values is configured via the ‘gplazma.oidc.audience-targets’ configuration property.

NFS

The mountd server is started only if NFSv3 is enabled.

Exposed files PNFSID, checksum and locality as read-only extended attribyutes. $ attr -l file1.txt Attribute "dcache.id" has a 36 byte value for file1.txt Attribute "dcache.locality" has a 8 byte value for file1.txt Attribute "dcache.checksum" has a 10 byte value for file1.txt

NOTE: this functionality is in BETA state and should be used with cation and client side caching might produce wrong results

Update NFS4J library to 0.23.x version with various fixes and improvements.

In version 8.0.0 we have introduced a workaround for linux NFSv4.1/pNFS client race condition that under high write IO workload the client might close a file, despite the fact that not all VM dirty pages are flushed and putting the client into unresponsive state.

Pool

Removed support of unused centralized flush manager.

When transferring files with HTTP-TPC, the pool will try to reuse TCP connections. Idle connections are closed only if they are idle for too long or if a maximum number of connections is reached. The exact behaviour is configurable. This is helpful when transferring many files with the same remote endpoint, particularly if the files are small.

Previously, when a client downloads a file via HTTP, the pool considers the transfer complete only after the client disconnects from the pool (or the pool disconnects the client due to inactivity). Now, if the client requests the entire file’s contents then the pool will consider the transfer complete once all the data has been sent. A request for partial file contents will continue to follow the existing behaviour.

The NearlineStorage interface is updated to support stateful plugins that can fail during initialisation.

Resilience

There was a fix to a runtime exception triggered as a side effect of logging.

Telemetry

Added a command, print data, that prints data that is send to the collector.

WebDAV

Clearer error message which does not denote a bug when CA certs directory is missing and HTTPS is configured.

XRootD

Since 7.2 there have been a number of upgrades and fixes for important regressions. These have been noted in the time-releases. The most significant were a fix to handle clients writing to a closed checksum channel, to support delete requests on the pool, and to provide a workaround for older xroot clients using prime values in the GSI handshake shorter than the recommended length.

Changelog from 7.2.0 to 8.0.0

a8cb617f44
[maven-release-plugin] prepare for next development iteration
a8b9c63d61
[maven-release-plugin] prepare release 8.0.0
5b44b8e61e
common: provide reasonable error message if path is not absolute
b0812a7572
pool: fix NPE on attempt by client to read from a broken replica
8323f0172d
common-security: re-read server credentials on context re-creation
d2ed77eabd
rest api: add checksum paramater to namespace resource
8b28862cda
frontend: update dCacheView to v2.0.2
0a0a095821
TransferManager: log bugs with corresponding stack-trace
ff4077fa10
CopyManager: reply success when the target transfer completes.
d4c5943b94
pool: only send new URLs when flushing file
33f2937057
[maven-release-plugin] prepare branch 8.0
ca1e3c7ce8
nfs4.1: limit layout-recall workaround to flex_files layout only
c0c77a2828
skel: minor improvements to example configuration for needrestart
a6230571ed
common: add support for “group” principals in various places
143802520a
qos-verifier: (2 of 8) pool selection utilities
4f561f9914
dcache-xroot: set default TLS to OPTIONAL
6f415a50e9
dcache-core: fix admin command to query pins by state
b2cdfac20a
util: drop Transfer#get/getOnlineFilesOnly
1746a22e62
gplazma: oidc rewrite OidcAuthPluginTest
7582bf9434
gplazma: oidc add support for profiles
40603217d9
gplazma: oidc fix minor/typo errors in IdentityProviderTests
2671c9ffae
srm-server: log trs tape queue states when tape requests are added
86cbd0cddc
chimera: remove unused constructor in ChimeraDirectoryEntry
5a001af333
nfs4.1: recall write layout on close
d232f34c85
libs: use nfs4j–0.23.0
a85e84ba8a
book: Add chapter about telemetry cell to index
0b15b16b53
book: Add chapter about telemetry cell
ece6360996
gplazma: oidc add support for offline JWT validation
6b99107559
nfs4: expose pnfsid, checksum, locality as extended attribytes
da1fcdc883
telemetry: Add command to display data
8baabae881
libs: update ZooKeeper version
b8639d1f80
nfs: use local pool monitor for reads
d38d5e940e
dcache-chimera: avoid conversion id->path->id on getLocality
0191f0b40e
common: move Jdbc base classes from qos
c7711c7ab0
skel: bump logback to 1.2.10
ac93764537
book: fix typo
3223a9faad
qos-verifier: (1 of 8) data store interfaces and implementation (jdbc)
8a4a02cd02
gplazma: oidc support chaining of TokenProcessor implementations
1f63fbbeab
pnfsmanager: add support for resetting gauge and counter statistics
1947b9386f
libs: use jline2–2.14.6 with small bugfix
30828d4930
loginbroker: add support for multiple protocol families