What's new in dCache 12.0
Highlights
Zone-aware pool selection (PoC) – dCache now supports directing clients to pools closest to them geographically or topologically, useful in multi-site deployments
OIDC Authorization Code Flow support in dCache View (PoC)
Incompatibilities
- Removed SRM and SRM server functionality.
- Removed NIS support.
- Removed the
gridmap-filegPlazma plugin. Sites must migrate to themultimapplugin.
Acknowledgments
We want to thank Shawn McKee and Niklas Edmundsson for the contributions.
Release 12.0.1
Billing
Cells
The cell command route find, in addition to the regular cell routing output, also displays TOPIC routes.
Frontend
Update REST API to allow space reservations:
curl -X POST \
'https://dcache-frontent-endpoint:3881/api/v1/space/tokens?accessLatency=online&retentionPolicy=replica&minSize=8192&description=foobar' \
-d ''
{"sizeInBytes":8192,"description":"foobar","id":8}
Respond with the Retry-After HTTP header when the client hits the rate limit.
gPlazma
Removed the NIS plugin as sites are (hopefully) not using it anymore.
The gridmap-file plugin has been removed. It was redundant with the multimap plugin, which covers all the same use cases. A conversion script to migrate existing gridmap files to multimap format is provided in the documentation.
NFS
Pool
If the pool identifies that the underlying repository operates in READ-ONLY mode, then all running checksum scrubbing processes will be stopped.
Scripts
Added dcache pool benchmark command that executes a predefined fio benchmark for all pools on the host.
Pool Manager
dCache now supports zone-aware pool selection. Zones are useful in multi-site deployments to direct traffic to pools that are geographically or topologically closest to the client.
All protocol doors (WebDAV, NFS, FTP, xrootd, etc.) propagate their zone tag through transfer messages to the pool manager. The pool manager then applies best-effort (“soft”) filtering — preferring pools in the same zone as the requesting door, with automatic fallback to the full pool list if no eligible pool in the correct zone is found.
Zone-aware routing applies to:
- Read and write selection
- Pool-to-pool (p2p) transfers — destination pools are filtered by zone in all partition types
- Staging — files recalled from tape land in a pool in the client’s zone where possible; if the selected stage pool is not a read pool, a p2p copy to the client’s zone is triggered automatically
To enable zones, assign a zone tag to pools and doors in the pool manager configuration:
psu set pool <poolname> -zone=<zonename>
And in the layout file for doors:
<door>.zone=<zonename>
SRM / SRM Client
The SRM server and SRM client are excluded from the project build and test process.
WebDAV
The browser now handles downloads natively via a temporary anchor element, showing its own progress indicator immediately upon initiation. Users can select multiple files and download them all at once from the context menu. Short-lived Macaroon tokens are generated automatically for authenticated downloads where HTTP Basic credentials cannot be passed to the browser. Certificate-based sessions are unaffected. The shared-files view, including subdirectories, now uses the same download logic as the main namespace view.
Respond with the Retry-After HTTP header when the client hits the rate limit.
dCache now supports RFC 9530 for HTTP content digest negotiation. RFC 9530 supersedes RFC 3230 and introduces Want-Repr-Digest / Repr-Digest headers. WLCG transfer tools such as FTS are adopting RFC 9530, so this change is required for compatibility.
- Clients sending
Want-Repr-Digestreceive aRepr-Digestresponse header from both the WebDAV door and pool HTTP movers. - TPC Pull: outgoing GET and HEAD requests include both
Want-Repr-DigestandWant-Digest. - TPC Push: the
Repr-Digest(orDigestas fallback) from incoming COPY requests is forwarded to the destination PUT. - TPC Passive:
Repr-Digestvalues on incoming PUT requests are verified against the pool checksum.
dCache-view
Downloads are now handled by the browser directly instead of dCache View, giving users real-time progress feedback and reliably supporting large files. Multiple files can now be downloaded at once, using short-lived Macaroons for authentication where needed.
This release adds OIDC Authorization Code Flow support to the dcache-view web interface.
Users can now log in to dcache-view using their institutional identity via for example Keycloak (Helmholtz AAI).
Configuration Changes
1. etc/dcache.conf — Frontend OIDC Settings
Add or verify the following properties:
# OIDC client credentials
frontend.authn.oidc.client-id=1234
frontend.authn.oidc.client-secret=OSwvvbbF26vioqGkeKVXy4IV7QHFbmUv
frontend.authn.oidc.token-url=https://keycloak.desy.de/auth/realms/dcache-dev/protocol/openid-connect/token
# dcache-view UI settings
frontend.static!dcache-view.endpoints.webdav=https://localhost:2881/
frontend.static!dcache-view.oidc-authz-endpoint-list=https://keycloak.desy.de/auth/realms/dcache-dev/protocol/openid-connect/auth
frontend.static!dcache-view.oidc-client-id-list=1234
frontend.static!dcache-view.oidc-provider-name-list=Helmholtz%20AAI(Keycloak)
frontend.static!dcache-view.oidc-authz-redirect-url=https://localhost:3881/api/v1/auth/callback
frontend.static!dcache-view.oidc-authz-endpoint-extra=-
2. etc/gplazma.conf — gPlazma Authentication Stack
Ensure the following plugins are configured in the correct stages:
# Auth stage — verify OIDC token
auth optional oidc
# Map stage — resolve identity to local user
map optional multimap gplazma.multimap.file=/pathtodcache/dcache/packages/system-test/target/dcache/etc/multimap-id-to-username.conf
3. etc/multimap-id-to-username.conf — Identity Mappings
Map OIDC subject identifiers to local usernames and Unix attributes:
# Keycloak identity
oidc:47055fee-1@keycloak username:usernameoidc uid:1000 gid:1000,true
---
### 4. `etc/grid-security/storage-authzdb` — Authorization
Verify the file starts with `version 2.1` and contains the following entries:
version 2.1
authorize username:usernameoidc read-write 1000 1000 / /
---
### Xrootd
### Zookeeper
Updated the apache-curator library, which is used to implement zookeeper-related functionality, such as leader election.
TLS encryption is now supported for connections between dCache and ZooKeeper. The feature uses the standard dCache PEM certificate files (`hostcert.pem`, `hostkey.pem`, `ca-certificates.pem`) already used by other services, rather than separate JKS or PKCS12 keystores. The CRL and OCSP modes are shared with the rest of dCache via the existing `dcache.authn.crl-mode` and `dcache.authn.ocsp-mode` properties.
To enable TLS for ZooKeeper, add the following to your layout or `dcache.conf`:
dcache.zookeeper.tls.enabled = true
The connection will use the host certificate and trust store already configured for the domain. See the dCache Book (config-zookeeper) for full details.
Changelog from 11.2.0 to 12.0.0
- 3281c7787b
- [maven-release-plugin] prepare for next development iteration
- c28772832a
- [maven-release-plugin] prepare release 12.0.1
- dd38dab26d
- Revert “clean for tagging 12.0.0 again”
- c74e35d1b1
- Revert “clean for tagging 12.0.0 againin all pom”
- 4410ac739d
- Revert “[maven-release-plugin] prepare release 12.0.0”
- 72b9197b12
- Revert “[maven-release-plugin] prepare for next development iteration”
- aa10998a7c
- [maven-release-plugin] prepare for next development iteration
- 2cbaa88b1f
- [maven-release-plugin] prepare release 12.0.0
- abdc782dd5
- clean for tagging 12.0.0 againin all pom
- 0708d0e582
- clean for tagging 12.0.0 again
- d85d1d9b0d
- ci: fix upload_container job
- 17dc34d0ee
- [maven-release-plugin] prepare for next development iteration
- efe0874f91
- [maven-release-plugin] prepare release 12.0.0
- be2582453b
- [maven-release-plugin] prepare branch @{releaseLabel}
- a1eb1de879
- pom: change project dependency for dcache-view Fix the following issue with dcache-view version
- 7960bb578e
- front: first version of working draft for code flow, works for download and delete
- 9a9162f836
- complete HttpExtHeader conversion
- 46e184d527
- webdav: use util HttpExtHeader record instead of hardcoding Digest header values
- b20368c27b
- webdav: add support for rfc9530 Http Headers
- bd0cde7971
- httpd: restore some of the legacy color scheme and table layout
- b6e2a9fe7f
- REST API: add domains resource.
- 5ff074d3a0
- webdav/frontend: update rate limiter to respond with Retry-After header
- 43e2222153
- pom: use bugfix release oncrpc4j-3.4.3
- d6c7a7af41
- build(deps): bump io.netty:netty-codec-haproxy
- e6e297035f
- build(deps): bump io.netty:netty-handler
- 3248b35b58
- frontend: delete the option heather for CORS
- 5740466181
- poolmanager: filter staging pools by zone
- 77fa2a89ab
- poolmanager: zone based p2p destination selection in partitions
- c9e701b457
- poolmanager: select pool by zone logic in PoolMonitorV5
- 18b370737b
- Revert “Merge branch ‘zones’”
- bcaf8604d4
- poolmanager: filter staging pools by zone
- 32cd45a444
- poolmanager: filter staging pools by zone
- ef4cd77f93
- poolmanager: zone based p2p destination selection in partitions
- fd99f2c0cc
- poolmanager: select pool by zone logic in PoolMonitorV5
- 2de6db1531
- webdav: remove custom workaround of milton error handling
- 6064bb7895
- pool: use java14 record type to implement QueueStat
- 4f080d5b92
- dcacheview: nodejs downgrade to previous version
- 47c9c355c7
- dcacheview:upgrade node,js version
- b61c738810
- clean unused comments
- 54c05d1303
- door: fix npe in setZone
- dc0e8d2812
- correct the limmit
- 1329f83f54
- dcacheview: moving dcache-view into dcache
- 76d7be5083
- Motivation: The recent implementation of zone based pool selection needs documentation
- f1c6da0eca
- Motivation: Zones are useful in multi site deployments to easily direct reads and writes to pools that are closest to the client. The current implementation of Zones only made pools aware of their zone. Doors, messages and transfers also need to carry the information of the zone they originate from to enable zoneaware reads and writes
- 6f3d0f99a6
- packages: drop dependency on gplazma2-nis (fixes commit 0d6166a07b)
- 0d6166a07b
- gplazma: drop nis support
- be579f9f43
- build(deps): bump io.netty:netty-transport-native-epoll
- aec5a72e94
- build(deps): bump org.postgresql:postgresql from 42.3.9 to 42.7.11
- 3045a52717
- build(deps): bump io.netty:netty-codec-http
- 0505cdb46a
- test: fix unit test broken by commit 6e8e1e3448
- 6e8e1e3448
- cells: add custom DelegatingCuratorFramework
- 8c80adbf7d
- Motivation: The Gridmap file plugin is redundant because the Multimap Plugin does the same as per Issue #7659 “drop gridmap-file support”.
- dfad263a21
- pool: stop automatic scrubber on repository read-only event
- c670b570b9
- Motivation: The recently added imlementation of TLS support for ZooKeeper used jks/p12 files while the rest of dCache uses .pem
- 69702638ee
- chimera: applying attributes to newly created file should skip permission check
- bd588fdc95
- ci: use custom k8s deployment for minio
- 34c4cedc59
- ci: skip git clone it sources are not needed
- 3850e3654c
- cells: update
route findcommand to support topics - 2452055701
- code coverage: replace strict JaCoCo COPY with conditional staging
- 6682edadd9
- chimera: fix reading of empty tags
- 2d72aa23ad
- pom: update aspect4j to java25 capable 1.9.25
- cbd374aad9
- bulk: report 413 Entity Too Large is limits on bulk requests are exceeded
- 3d892cc638
- Motivation: The configuration of TLS encryption for domain-domain communication is not well documented. This has led to confusion among users.
- b394a1aecb
- domain: Add TLS support for ZooKeeper connections
- c093b7772c
- gplazma2-oidc: Add and test support for
storage.pollWLCG claim - fd91c54812
- tape rest API: remove AccessMask.READ_DATA requirement when querying for file attributes for file locality
- 59aeed7b3e
- alarms: use HardenedLoggingEventInputStream to address possible RCE when deseriaizing log messages
- 1ec7074c7e
- scripts: fix double rw definition in FIO benchmark script
- 2a0ada323e
- code coverage: add integrational coverage
- 8819c24523
- pool: Address Copilot review comments on
hotfile show - 670fc3b051
- ci: move keycloack-deployment.yaml into .ci/k8s directory
- 07fb84092b
- pool: emit firefly onStart marker from RemoteHttpDataTransferProtocol
- b656f9a53d
- libs: update apache-curator to 5.9.0
- 8528738964
- pool: New
hotfile showcommand, and show enablement status withinfo -a - 46aae42c27
- pom: upgrade of libraries
- e846f1a9e3
- docs/dev: add historical plans and work summaries for future reference.
- fea6122834
- pool: Fix properties tests to use the correct property
- ab9db28eff
- pool: Improve logging for hot file replication
- 787d08edb9
- qos-policy: update to QoS policy API
- 5fe6ec98bd
- rpm: make package require java17 or java21
- ec61a1912f
- build(deps): bump io.netty:netty-codec-http
- 1c651355a6
- httpd: fix generated pool group usage urls
- e98233ec86
- pom: drop dependency on patch plugin
- f4cad99e7c
- docs: frontend chapter updates
- 8d870cfbe3
- pool: fix failing unit tests
- 9094d5997f
- pool: enhance migration pool choice logic
- 2e0e589347
- dev: Instructions for GitHub Copilot CLI, plugins, and agents
- d0be76403d
- firefly: address latest PR 8044 inline feedback
- e726a98198
- Revert “pool: enhance migration pool choice logic”
- 358360c278
- pool: enhance migration pool choice logic
- 9195703028
- pool: exclude p2p requests from numberOfRequestsFor count
- db228398b8
- firefly: address PR 8044 review fixes
- 40b7775e2c
- util: remove unused code
- 5cef4f7080
- xrootd: fix logSciTagsRequest called before cell address initialization
- cfa848240d
- Fix SciTag WAN TPC marker generation and validation
- 3ef41d6070
- src: skip SRM when building and testing dCache
- a0b6cda1bc
- frontend: add space reservations functionality
- 88a9be1857
- Add WAN SciTag support for remote HTTP(S) TPC movers
- f093502a02
- Update CITATION.cff with affiliation as authors
- 9cde193aa1
- Firefly: address PR review comments
- 68d51cb1e5
- Firefly: send additional copy to configured collector
- 76dee8f81e
- docs: QoS chapter inconsistencies fix
- 1a16d61c36
- Address PR review feedback for SciTag parsing and VO mapping
- 5ac31f6d4e
- Add SciTag debug logging for WebDAV and xrootd doors
- 5038ab473c
- Require both endpoints to match firefly excludes
- 9f883f77c8
- Fix firefly experiment-id resolution for SciTag and FQAN fallback
- 2e61583af7
- script: add
dcache pool benchmarkcommand - 6537358ef6
- build(deps): bump org.apache.zookeeper:zookeeper from 3.8.4 to 3.8.6
- 5acd486b27
- namespace: ensure that access latency and retention policy are always defined
- 45b9e77923
- common-security: remove historic garbage
- 80c12dae9e
- pom: update nfs4j to 0.27.2
- 6f00b7bf91
- build(deps): bump com.fasterxml.jackson.core:jackson-core
- 488e648faf
- ci: move junit test report into run_ut_with_coverage job
- a106a79559
- ci: use version 9.2.40 for compatibility test
- b82ea31b1c
- pool: send firefly for https transfers
- dd8db3ef25
- rest api: add support for Java8 Optional data type
- 91de69d0f8
- dcache cli: fix issue with printing pool size when max diskspace is set in percentage
- a93e7cd714
- docs: update installation requirements and instructions
- 13ee819efd
- src: disable SRM-client build in CI pipeline
- 324abc2203
- lib: update mina-sshd to 2.17.1
- 1731eae972
- http/tpc: add support for ArchiveMetadata header
- 4997c2f55a
- code coverage: centralize JaCoCo version
- 148ab3f75e
- code coverage: implement JaCoCo report generation in CI
- bebaeee564
- [maven-release-plugin] prepare for next development iteration