Highlights

  • support for nested groups in the PoolManager

Incompatibilities

  • pool group names can’t start with @ symbol
  • hard links on directories is not allowed
  • GET api/v1/bulk-requests has a new return value (see below)

Acknowledgments

(tbd, this section uses markdown formatting)

Release 8.2.5

bulk

The garbled request policy command output is fixed now

There was a wrong option for target type (REGULAR, not FILE). This is now fixed and admin commands are now consistent.

nearline-storage

A NPE on error path has been fixed.

The race condition when flushing removed file is fixed now.

poolmanager

Motivation: When a new entry is added to the RequestContainer the file pnfsid and netunit is used as an unique identity. However, despite the fact that two different units are a members of the same unit group, we might get two requests created, thus two stages are triggered. For instance:

0.0.0.0/0.0.0.0 & ::/0 => world-net

00007BBC36AB359E47A383A75CFA2AB6CDEF@0.0.0.0/0.0.0.0-/ m=1 r=1339 [pool_A] [Waiting for stage: pool_A 10.05 16:41:55] {0,} 00007BBC36AB359E47A383A75CFA2AB6CDEF@::/0 m=1 r=1339 [pool_B] [Waiting for stage: pool_B 10.05 16:43:23] {0,}

now it should work with:

00007BBC36AB359E47A383A75CFA2AB6CDEF@world-net-/ m=1 r=1339 [pool_A] [Waiting for stage: pool_A 10.05 17:11:37] {0,}

Changelog 8.2.4..8.2.5

c9b209d41c
[maven-release-plugin] prepare release 8.2.5
0c4e620bfb
dcache-bulk: fix garbled request policy command output
34a183b4cd
poolmanager: use net unit groups to identify the request
560a641bf6
nearline-storage: fix race in flushing removed file
1d7e7f374c
dcache-bulk: fix message thread invocation of database query for statistics
15868b0e17
dcache-bulk: fix admin command options (uppercase)
b663601d0f
nearline-storage: fix NPE on error path
4c432f2b34
[maven-release-plugin] prepare for next development iteration

Release 8.2.4

dcache-qos

A minor regressions in handling of pool info diffs has been fixed.

Full scan of pool info diff behavior is available.

dcache-rest

A new optional query parameter to retriveve a wealth of information about a file has been added.

frontend

The current release fixed type default of RESTful pool/nearline/queues resource

pool

The pools with disabled HSM operations are not visible to hsm cleaner.

Changelog 8.2.3..8.2.4

c09832c5a1
[maven-release-plugin] prepare release 8.2.4
e7f4a31e3c
dcache.properties: Fyx synax erorr
202ff9aa1a
dcache-rest namespace resource: add optional query parameter to retrieve a wealth of information about a file
f83316996a
dcache-qos: fix minor regressions in handling of pool info diffs
0dad875fd0
pool: don’t publish attached HSM if lfs mode is precious
d516bc3e5e
dcache-frontend: fix type default of RESTful pool/nearline/queues resource
29da88f7b3
pom.xml: bump xrootd4j to 4.5.2
0df499cc65
dcache-qos: fix two more regressions in the PoolOpChangeHandler
d0c3b0dc00
dcache-qos: add missing pool tag comparison in PoolOpChangeHandler
fb82a9f366
[maven-release-plugin] prepare for next development iteration

Release 8.2.3

bulk

The command \s bulk request submit -activity=STAGE now overrides the -prestore option to true. This makes it consistent with the RESTful API.

Cancellation of PIN requests via bulk module which were not marked with -prestore=true no longer fail to complete.

chimera

Pools that are no longer actively being cleaned were previously falsely still listed as active in the admin interface when the cleaning attempt ended due to an exception. This has been fixed.

core

If a pin is unpinned while a pin request for that same pin is still ongoing in pinmanager, pinning will not be retried.

pnfsmanager

On massive uploads (create entries) to a single directory we observed performance degradation leading to all available threads being busy, denying other users from accessing the namespace. This has been alleviated.

pool

The performance of downloading over plain http is increased by using zero-copy mode.

Less CPU load on pools in case of client accessing a file with invalid/expired state id.

xroot

A warning was added to the documentation about the default TLS option, which changed. OFF is the only option for TLS which does not require the presence of the host cert/key.

Changelog 8.2.2..8.2.3

d108a4d82d
[maven-release-plugin] prepare release 8.2.3
d590276b20
pool: use zero-copy send for plaintext http transfers
dccb9ccddb
pool: avoid exception based flow in case of nfs bad stateid error
a5e1a082fc
dcache-xroot: add note about TLS and host cert/key to documentation
b0b922a22e
dcache-bulk: force admin command STAGE activity to prestore=true, as with REST API
4c52046162
dcache-bulk: fix missing pnfsid on cancel of PIN/STAGE
f7a7c6c038
dcache-core: let pinmanager stop retrying pinning upon unpin state
adb206be77
pnfsmanager: dispatch create entry messages to threads associated with parent entries
4c5ad228ed
dcache-chimera: disk-cleaner, error handling in parallel pool deletes
1e773a6731
[maven-release-plugin] prepare for next development iteration

Release 8.2.2

spacemanager

Fixes NullPointerException that occurs on startup of space manager when spacemanager.authz.link-group-file-name is not set.

Changelog 8.2.1..8.2.2

364638a55a
[maven-release-plugin] prepare release 8.2.2
db004422fd
dcache-spacemanager: fix NPE when link-group-file-name is not set
04bdcef478
[maven-release-plugin] prepare for next development iteration

Release 8.2.1

Changes affecting multiple services

The commands psu match and psux match can now be used with pnfsid or path.

core

ac_psu(x)_match commands has been improved to handle null/globbed values correctly in psu match.

frontend

It is now possible to filter and sort the results of the RESTful API call GET /restores by path, owner and group.

NOTE: This resource is protected by the frontend.authz.unlimited-operation-visibility property, which means that proprietary information can be optionally protected by limiting it to those with admin privileges.

gplazma2-grid

An empty requires field in IGTF policy is now accepted. Files with invalid or unexpected policies are ignored.

webdav

A bug was fixed that caused a NullPointerException while sending Kafka messages.

jquery was updated to version 3.6.1 and bootstrap was updated to version 5.2.1

xroot

The proxy is now bind to a local address that respects the network of the client and is likely reachable by it.

Changelog 8.2.0..8.2.1

6d2670057e
[maven-release-plugin] prepare release 8.2.1
6d46d60d8c
dcache-core,dcache-frontend: add pnfsid and path options to ac psu match and RESTful resource
dbae6e778c
dcache: NPE on removal via WebDAV and token
eff5548052
webdav: update 3rd party js libraries
a10630d44c
dcache-frontend: add filtering and sorting on owner, group and path to restore info API
ca6486cd98
dcache-core: handle null/globbed values correctly in psu match
c2694d4200
dcache-xroot: explictly close the proxy’s bound port channel
a6696f9193
dcache-xroot: select proxy door address on basis of reachability for client
a0de98f35c
doc: add recipe to cookbook for migration/draining of resilient pools
9a09718207
[maven-release-plugin] prepare for next development iteration
c25d4246ef
qos-engine: prevent NPE if pool monitor not available
9c37633ce2
gplazma2-grid: invalid IGTF policy file should fail with ParserException

Release 8.2.0

Bulk

The bulk service has undergone a major revision. There have been numerous internal changes and enhancements, but the most important have to do with persistence and scalability.

The bulk service now requires a database which holds request information (instead of raw files, as in version 1). Bulk requests are serviced concurrently, but each request runs in its own container in order to optimize threading and I/O.

The UPDATE_QOS activity communicates with (and thus requires the presence of) the new QoS Engine, but the other bulk activities can be run without QoS being present.

Aside from being accessed through frontend REST APIs, the bulk service has a full set of admin shell commands which are useful for spot-checking its operational state and the status of requests (along with their individual targets).

A fuller explanation is available in The Book chapter on the bulk service.

Cleaner

An admin command was added to the cleaner cell that allows forgetting a pool, meaning removing any corresponding delete target entries from the cleaner’s trash table database. If the requested pool is not currently blacklisted, the admin is required to explicitly enforce the operation.

Frontend

For the bulk-requests API: the typing of primitive parameters in the JSON bulk request has been loosened so that booleans and numerics can also be specified as strings.

The RESTful options for QoS (for instance, in api/v1/namespace) as of version 8.1 communicate with the new QoS Engine.

The JSON returned from GET api/v1/bulk-requests is no longer a simple array of URLs, but an array of objects with fuller info concerning each request (hence this return value now requires parsing; see the SWAGGER page for fuller information). All matching requests are now returned (rather than those owned only by the user), though it is possible to filter the result by a comma-delimited list of user/owner ids.

A new TAPE API has been made available for the purposes of bulk staging (api/v1/tape); it implements the WLCG TAPE specification, and thus includes api/v1/tape/archiveinfo [=locality for dCache], api/v1/tape/stage and api/v1/tape/release [=unpin for dCache].
See the SWAGGER pages for fuller info. There is also a section in the User Guide (Frontend/BulkRequests) which details the APIs for bulk-requests and tape along with their differences.

PNFS Manager

Historically, the dCache namespace was allowing hard links on directories. This violates POSIX restrictions and is a source of file system cyclic loops. With dCache release 8.2 creation of hard links on directories is not allowed any more. The existing hard links will work as before; however, it is recommended to replace them with soft links.

To allow pool, nearline storage and other components to make scheduling decisions based on file creation time, PnfsManager will include the CREATION_TIME attribute whenever STORAGEINFO is requested.

Pool

Added rm kill command that canceles a pending file remove request from HSM.

When a pool decides which storage class to flush it uses the last submission time. However, submission time is re-set when a pool fails to flush the file and re-submit the file. Starting with 8.2, a pool will sort the flushing based on the time when files are created on the pool.

Poolmanager

PoolManager has been updated to support pool groups that contain other pool groups. For example:

psu create pgroup -dynamic -tags=hw-class=IO io-pools
psu create pgroup -dynamic -tags=hw-class=replica replica-pools
psu create pgroup all-pools
psu addto pgroup all-pools @io-pools @replica-pools

The values prefixed by @ in the psu addto pgroup command are treated as groups.

Resilience / QoS Engine

The fully refactored QoS Engine first became available in 8.1. Among other things, its components are intended to replace Resilience. While Resilience is still available in 8.2, it will eventually be deprecated, so it is advisable to begin switching to QoS with this Golden Release.

There is a chapter on the QoS service components in The Book, which includes instructions on how to set it up to emulate Resilience (the differences are minor).

Note that a major change between Resilience and QoS is that the QoS Verifier uses a database whereas none was used by Resilience.

XRootD

Several improvements have been made so that the door endpoint address is consistently reported.

An important oversight in the door code has been corrected. A client disconnect now also interrupts any thread waiting on a response from PoolManager; otherwise, all available threads could be tied up, potentially blocking further client connections.

The xroot door is now capable of running in “proxy” mode; when this switch is turned on, the client connects through the door instead of directly to the pool in order to read or write data (allowing for outside communication with pools protected on an internal network, for instance). An internal address (subnet) can also be given which will be used for the Pool Manager pool selection. Details are available in the xroot chaper of The Book.

Zookeeper

The Zookeeper version has been updated to the latest stable release 5.7.1

Changelog from 8.1.0 to 8.2.0

9a09718207
[maven-release-plugin] prepare for next development iteration
826c56fe3d
[maven-release-plugin] prepare release 8.2.0
283c8b75c3
nfs4: return EACCESS when stage is not allowed
312ea69bdf
gplazma: vo-group plugin
3d04e98d26
[maven-release-plugin] prepare branch 8.2
e04424facf
gplazma: update gplazma config loader to use ParsableFile
7f1cdae61b
gplazma: fix year in copyright statement
3829bcd5e9
common: add hashCode and equals methods for Result
317f9a6832
pool: order storage classes by oldest entry in the queue
e929f9c0e9
util,namespace: ensure that files’ creation time is always populated
e0ebf992f3
Revert “common: remove orphaned test file”
70f97b6766
common: remove orphaned test file
89dbaec931
gplazma: switch configuration parser to use LineBasedParser
31aa21f2f9
gplazma: refactor ConfigurationParserFactory layer
ac18e03164
gplazma: remove dead code
5a3b0909aa
libs: update to the latest stable zk version 3.7.1
689f76c38b
dcache-bulk,dcache-frontend: modify GET for bulk-requests to be somewhat more useful
13e497d7b9
dcache-xroot: add configurable internal address for proxying
1624bee957
common: have ParsableFile return Result
32f3dbaeca
nearline: add rm kill command to cancel pending removes from HSM
ad67ecbee2
src: add benchmark module
c01b89d6e3
dcache-xroot: add Book section on direct memory requirements for pools
1bcc5ecebb
dcache-xroot: add optional max write buffer size to decoder
2bacd47d70
webdav: fix NPE if client aborts transfer
c16d516d4f
webdav: http-tpc do not keep retrying to cancel transfer
e1d0d7114e
src: don’t use repackaged guava
8fc9957358
Revert “src: drop dependency on jython”
59fd0d8da8
gplazma: oidc drop support for ‘gplazma.oidc.hostname’
6b5fa34f0c
src: drop dependency on jython
c429647bb5
build(deps): bump jetty-webapp from 9.4.43.v20210629 to 9.4.44.v20210927
e769dbc7bc
dcache-xroot: interrupt doOnOpen thread in the event of channel inactive
0acb2433ea
spacemanager: switch linkgroupauthz file to use ParsableFile
7bd4abc6c0
common: add support for discovering file path
6310d9c9fc
spacemanager: remove dead code
7a734e3196
commons: move generic configuration file support
2b799c6093
properties: report that remote transfer manager is replicable
9873e45b30
cleaner: add better cleaner logging
0dcd5e4fd2
dcache-xroot: add section to The Book on proxying through the door
858ea9529c
dcache-xroot: use door address only on PoolManager protocol info
16376625c2
dcache-xroot: add proxying capability to door
7cc887929e
pom.xml: upgrade xrootd4j to 4.4.0
651a4e06a8
Switch to using grep for the java version check.
e5b49d81a4
chimera: disallow hard links on directories
1222d3d4ee
pool: switch from Iterable to Collection
fa9bbd5059
docs: update SRR documentation to address WLCG complains
126560b18c
dcache-bulk: (version 2) handle unexpected exceptions in container manager
4b9fef7833
build(deps): bump spring-core from 5.2.20.RELEASE to 5.2.22.RELEASE
e0bd9ce990
Revert “webdav,dcache: update deprecated netty http usages”
8b68eeecdb
dcache-frontend: move wlcg tape rest resources under common ‘tape’ path
7fc973c448
webdav,dcache: update deprecated netty http usages
388c7c3a78
pool: minor refactor of ReplicaDescriptor
4f70939350
UPDATE_METADATA is needed for staging
77f0369431
pool: finalise on-the-fly checksums on channel close
3e9190e04e
core: update FileAttributes to allow adding extra checksum values.
16247d7450
webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
62de9e9759
9a09718207
[maven-release-plugin] prepare for next development iteration
826c56fe3d
[maven-release-plugin] prepare release 8.2.0
283c8b75c3
nfs4: return EACCESS when stage is not allowed
312ea69bdf
gplazma: vo-group plugin
3d04e98d26
[maven-release-plugin] prepare branch 8.2
e04424facf
gplazma: update gplazma config loader to use ParsableFile
7f1cdae61b
gplazma: fix year in copyright statement
3829bcd5e9
common: add hashCode and equals methods for Result
317f9a6832
pool: order storage classes by oldest entry in the queue
e929f9c0e9
util,namespace: ensure that files’ creation time is always populated
e0ebf992f3
Revert “common: remove orphaned test file”
70f97b6766
common: remove orphaned test file
89dbaec931
gplazma: switch configuration parser to use LineBasedParser
31aa21f2f9
gplazma: refactor ConfigurationParserFactory layer
ac18e03164
gplazma: remove dead code
5a3b0909aa
libs: update to the latest stable zk version 3.7.1
689f76c38b
dcache-bulk,dcache-frontend: modify GET for bulk-requests to be somewhat more useful
13e497d7b9
dcache-xroot: add configurable internal address for proxying
1624bee957
common: have ParsableFile return Result
32f3dbaeca
nearline: add rm kill command to cancel pending removes from HSM
ad67ecbee2
src: add benchmark module
c01b89d6e3
dcache-xroot: add Book section on direct memory requirements for pools
1bcc5ecebb
dcache-xroot: add optional max write buffer size to decoder
2bacd47d70
webdav: fix NPE if client aborts transfer
c16d516d4f
webdav: http-tpc do not keep retrying to cancel transfer
e1d0d7114e
src: don’t use repackaged guava
8fc9957358
Revert “src: drop dependency on jython”
59fd0d8da8
gplazma: oidc drop support for ‘gplazma.oidc.hostname’
6b5fa34f0c
src: drop dependency on jython
c429647bb5
build(deps): bump jetty-webapp from 9.4.43.v20210629 to 9.4.44.v20210927
e769dbc7bc
dcache-xroot: interrupt doOnOpen thread in the event of channel inactive
0acb2433ea
spacemanager: switch linkgroupauthz file to use ParsableFile
7bd4abc6c0
common: add support for discovering file path
6310d9c9fc
spacemanager: remove dead code
7a734e3196
commons: move generic configuration file support
2b799c6093
properties: report that remote transfer manager is replicable
9873e45b30
cleaner: add better cleaner logging
0dcd5e4fd2
dcache-xroot: add section to The Book on proxying through the door
858ea9529c
dcache-xroot: use door address only on PoolManager protocol info
16376625c2
dcache-xroot: add proxying capability to door
7cc887929e
pom.xml: upgrade xrootd4j to 4.4.0
651a4e06a8
Switch to using grep for the java version check.
e5b49d81a4
chimera: disallow hard links on directories
1222d3d4ee
pool: switch from Iterable to Collection
fa9bbd5059
docs: update SRR documentation to address WLCG complains
126560b18c
dcache-bulk: (version 2) handle unexpected exceptions in container manager
4b9fef7833
build(deps): bump spring-core from 5.2.20.RELEASE to 5.2.22.RELEASE
e0bd9ce990
Revert “webdav,dcache: update deprecated netty http usages”
8b68eeecdb
dcache-frontend: move wlcg tape rest resources under common ‘tape’ path
7fc973c448
webdav,dcache: update deprecated netty http usages
388c7c3a78
pool: minor refactor of ReplicaDescriptor
4f70939350
UPDATE_METADATA is needed for staging
77f0369431
pool: finalise on-the-fly checksums on channel close
3e9190e04e
core: update FileAttributes to allow adding extra checksum values.
16247d7450
webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
62de9e9759
webdav/transfermanager: support multiple RemoteTransferManager
2d59373489
dcache-bulk: (version 2) ensure container cancellation under RuntimeException
e240dda600
build(deps): bump spring-expression
0c11b64c5b
build(deps): bump opensaml from 2.4.1 to 2.6.5
f48da3140c
chimera,srm: remove deprecation warning propagateIfInstanceOf
97dd8dfe82
srm,poolmanager: replace buggy murmur3_32 with fixed implementation
69b6a3d698
dcache-frontend,bulk: fix typing of argument values
f6035f9f39
dcache-frontend: protect against RuntimeError in case of denied anonymous access
7cc8367c9c
dcache-frontend: bulk-requests POST, allow either string or boolean for boolean values
fd2b56f637
poolmanager: add possibility to create nested pool groups.
7d79a2210a
xrootd/pool: Use login token to identify door endpoint
22b1a8bcf0
xrootd: update door to send the pool the advertised endpoint
511adea3a2
core: refactor LoginBrokerProvider
6bb0b357b1
pool: fix enforcecrc checksum policy, always verify expected checksums
3d657b140b
pool: don’t treat an empty file as a sparse file
c916141a1e
dcache-core: do not report InterruptedException as bug
107c03e87c
pool-monitor: throttle and limit the number of fetches after start
6a593e33f0
srm-client: fix leftovers from guava migration
b7d864501d
libs: use up-to-date guava–31
8370c107d6
dcache-bulk: fix target cancellation
83659da6cd
dcache-bulk: add checks for file type and RP to PIN and STAGE activities
208e458211
dcache-bulk,dcache-frontend,dcache-vehicles: add activity check for WLCG
6247095f69
dcache-bulk: do not return the (internal) root request path with file info
9f84650e73
pool: improve handling empty saved state by scrubber
5499f80e18
src: stop using deprecated CheckedFuture class
031b679439
dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
98228466d1
dcache-qos,skel: parse out qos properties by component and correct the db properties for visibility
ab0ee3c77e
system-test,sbin: fix build of tests for bulk and update of database
8c3be49c59
bulk,qos documentation: Add missing image files for TheBook and UserGuide
763e2dfbb4
dcache-bulk,dcache-frontend,documentation: add Bulk version 2 with frontend APIs and documentation
319029827a
Revert “dcache-xroot,pom.xml: bump xrootd4j to 4.3.1”
ed3794cc85
dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
566800bd2a
srm-server: don’t use deprecated guava’s CharMatcher#JAVA_LETTER_OR_DIGIT
abde13cee1
dcache-xroot: modernize door admin shell commands
802f090cad
dcache-xroot: flesh out channel inactive and exception caught
07f21394c0
info: increase delay between messages
e84d189d55
src: explicitly specify directExecutor when using guava async
3e54167c3d
dcache-info: clean code, improve readability and logged values
6d348ec29c
build(deps-dev): bump gson from 2.8.0 to 2.8.9
50aff71517
Updated instructions for ssh2 keys
7e5a17e0e8
ChimeraVfs: Check file level in addition to inode type when disallowing file size change
d7e83bf391
dcache-chimera: add cleaner command to forget a pool
4e7abc9ac6
dcache-chimera: clean code, remove likelihood of NPEs
676ac65262
dcache: updating kafka libs part 2
b4bd4b90e4
pool: http-tpc fix NPE if ‘Content-MD5’ response header is absent
08d6e791d8
gplazma: introduce support for observers
52820415d1
pool: http-tpc add more sources of checksum information
b7f7595b25
pool: http-tpc refactor remote checksum handling
718e3af4df
pool: switch off IGTF namespace checking (by default) for clients
213f9206ca
dcache:improve documentation for kafka properties
5811373f40
nfs: re-send kill only for write movers
f703b2661b
dcache: updating kafka libs
1fd2e3b7a0
frontend: fix NPE in SRR when space information is not available yet.
5633b31af0
[maven-release-plugin] prepare for next development iteration
f520e9147a
Fix java version check.

(tbd as soon as the branching is done)