Highlights

In terms of security and access management, dCache 3.2.0 offers several significant new features. This release introduces TLS encryption for domain communication, which will greatly facilitate setting up large distributed instances with WAN interconnections. On the client-visible side, macaroons can now be used as a means for detailed access control. And on a more admin-oriented level, the newly-introduced concept of roles makes privilege management easier and facilitates the delegation of tasks to trusted users.

From an administrative perspective, perhaps the most outstanding change is the switch to systemd-compatible scripts on Debian systems.

Incompatibilities

  • Classic replica manager is no longer supported.
  • Uses systemd only on Debian-derived systems if available.
  • The admin door limits key-based login to the usernames listed in the authorized_keys2 file.
  • The output of the ps admin command has changed. External scripts must be updated.
  • The dcache.broker.port property is deprecated.
  • The srmmanager.net.port and srmmanager.net.local-hosts are no longer used.
  • Upgrading nodes running frontend, webdav or httpd to version 3.2.7 (or newer) requires upgrading nodes running poolmanager at least to 3.2.7 (or newer) version.

Acknowledgments

Once again, we are pleased to have received contributions by several people who are not members of our core team.

We would like to thank Ivan Kadochnikov for his patches to xrootd.

For the first time, dCache was used in teaching a software development course at a university. We would like to thank the students of HTW Berlin who participated, and especially Fritz Heiden, Vuong Luu Minh, Stefan Moll, Lotta Rüger, Robin Wenzel, Yannick Vahldieck, Alena Schemmert, Marisa Nest, Martin Bürger, Sarah Schulte, Hasan Jahid and Max Patzelt, whose code made it into 3.2.0.

Differences from dCache v3.1

The notes for release 3.2.0 detail the differences from dCache 3.1. Please read this section very carefully when upgrading from this version.

Release 3.2.42

Changes affecting multiple services

If a client specifies a checksum value with either a WebDAV or FTP upload, a Restriction check by-passed due to missing path warning was logged occasionally. This was fixed now, ensuring that restrictions are always applied.

pool

Space reservations on pools that are connected to tape showed a problem with failing restore requests: If a restore failed, the space that was reserved to hold the file that was supposed to come in from tape was not freed again but kept in the ‘sticky’ state. This resulted in lots of unusable space on pools that could only be reclaimed through a restart.

With the current release, this issue is fixed and space is freed as soon as possible after a failed restore request.

resilience

A very rare race-condition is fixed where a failed upload results in resilience recording a stack-trace.

webdav

An issue with the Milton WebDAV library prevented Partial (or vector-read) GET requests from succeeding. This was fixed now through both an update of the dependency and a local patch while we wait for the proposed fix to be included upstream.

Changelog 3.2.41..3.2.42

e35f2a0252
[maven-release-plugin] prepare release 3.2.42
ecbb4843f9
fix compilation
0c4c34bc5a
webdav: fix proxied partial (vector-read) GET requests
f9b7c732de
pool: fix pool space accounting on failed restores
e77e0164c0
resilience: fix NPE if file unlinked when resilience processes a broken file
9b707a36a2
ftp/webdav: fix bypass of restrictions
5e98600a74
[maven-release-plugin] prepare for next development iteration

Release 3.2.41

alarms

An internal issue with the alarms configuration was fixed, which should prevent a rare NullPointerException from occuring.

dcap

Creating a file or directory using the DCAP protocol with a URL as parameter, the file permissions were not set correctly.

With the current release, this was corrected, and such files use the client-supplied file permissions. If none are provided, the default modes 0700 (for directories) and 0600 (for files) are used.

xrootd

An uncaught exception in xrootd doors was fixed.

Changelog 3.2.40..3.2.41

33df8badde
[maven-release-plugin] prepare release 3.2.41
ca929f7d3c
alarms: fix persistence.xml configuration
e44c049717
dcap: fix permission propagation with DCAP
4783b17d3e
dcache-xrootd: handle possible race condition in directory listing
a84feebbc9
[maven-release-plugin] prepare for next development iteration

Release 3.2.40

statistics

Metadata merge was using max when it should had used min, this is now fixed.

Changelog 3.2.39..3.2.40

61d4b76
[maven-release-plugin] prepare release 3.2.40
4e1c2aa
common: fix histogram metadata merge
788b788
[maven-release-plugin] prepare for next development iteration

Release 3.2.39

ftp

The current release provides better protection against leaking proxy/data TCP sockets if client aborts a proxied transfer.

srm

Clients that use the gridsite protocol, such as davix, can now delegate their credential.

Changelog 3.2.38..3.2.39

db287b9
[maven-release-plugin] prepare release 3.2.39
06d5bfe
ftp: make shutdown more robust
1f83d3b
common: fix bug in CountingHistogram index computation
3dc7e59
[maven-release-plugin] prepare for next development iteration
e4d22d3
srm: gridsite fix querying validity of delegated credential

Release 3.2.38

ftp

The performance markers that dCache sends back to the client in FTP transfers are now more robust against bugs.

nfs

When transient errors in pools cause NFS transfers to have to wait and retry, the system’s behaviour is now more robust and no StackOverflowErrors should be logged any more.

scripts

Maven’s findbugs plugin is now granted more working memory in order to make builds, especially on our continuous integration system, more robust.

srm

Certificate lifetime considerations for VOMS proxy certificates are improved in this release: if a client delegates a credential where the VOMS AC expires before the X.509 proxies, dCache now will not use the credential beyond the AC expiry time. This avoids unnecessary authentication errors.

webdav

When the WebDAV door is considering an HTTP third-party-copy request that uses grid-site delegation, there is a minimum 20 minute validity that any existing delegated credential must satisfy. If this is not satisfied then dCache will request a fresh delegated credential.

Until now, if the client failed to delegate a fresh certificate then the subsequent COPY request was rejected. This release changes that behaviour and enables such transfers.

Changelog 3.2.37..3.2.38

8d0835b43e
[maven-release-plugin] prepare release 3.2.38
b64009c878
scripts: Avoid findbugs memory errors
8cd739b36d
nfs: increase request retry delay when selecting/starting pool or mover
8e4c745633
webdav: adjust minimum validity after requesting delegation
6a026804e2
srmmanager/webdav: consider VOMS AC validity of delegated credential
8c15015f63
ftp: make performance marker task robust.
8d877aec3b
[maven-release-plugin] prepare for next development iteration

Release 3.2.37

pool

Diagnostic logging for failed HTTP third-party transfers was improved.

Billing records for failed transfers now show more detailed information.

The handling of cancelled flush requests for nearline media was rewritten to be more efficient. This resolves issues where pools report “Flush of 0000… failed with: CacheException” followed by “Pool restart required: Internal repository error”.

Compatibility with DPM was improved by increasing HTTP GET requests’ timeouts. This should allow more transfers to succeed.

poolmanager

Supplying poolmanager with an unresolvable hostname as the target will now result in an UnknownHostException instead of the previous behaviour where an (unnecessary) NullPointerException was thrown.

srm

Logging of errors in the SRM credential store was improved.

webdav

If a non-resolvable host name is given as the source or destination of a third-party copy request, WebDAV will now fail the transfer immediately instead of waiting for a Poolmanager timeout.

Diagnostic logging for failed HTTP third-party transfers was improved.

xrootd

dCache allows xrootd clients to specify a query/opaque string in a kXR_mv request’s source path.

Changelog 3.2.36..3.2.37

2015e44697
[maven-release-plugin] prepare release 3.2.37
da040a2395
pool: HTTP TPC rework exception logging
f2f359d250
pool: increase TPC socket timeout for GET requests
a3995fe5b7
srm: fix credential store logging
1a474ac8f7
pool: update log status using exception class name if no message
08e79346ef
xrootd: strip off query part from kXR_mv source
fd0187db6c
webdav: fail TPC request early on unknown hostname
1986a52bfe
nearline-provider: do not propagate thread interrupt flag
f1cde32f35
poolmanager: fix NPE on unknown host
632bd934dd
webdav: improve logging of TPC requests
4c835638fa
[maven-release-plugin] prepare for next development iteration

Release 3.2.36

Changes affecting multiple services

In order to more easily identify a rejected macaroon in the logs, its ID is now included in the log message.

An irrelevant stacktrace was logged on unexpected CacheExceptions. This was removed, leading to less clutter in the logs.

Different macaroons that were issued against the same secret are now discernible in the logs.

Users now get more information about the reasons why an invalid macaroon was rejected: HTTP requests that are made with an invalid macaroon have a 401 HTTP response with the status-line explanation phrase that describes why the macaroon is invalid.

The access log file also logs why a macaroon was rejected.

core

A library dependency was updated to avoid CVE–2018–11771. This patch introduces no user-visible changes.

gplazma

Invalid macaroon logins no longer “spam” gPlazma.

pnfsmanager

When creating a macaroon to allow uploading of data, the desired path may not already exist. Without restrictions, WebDAV will auto-create parent directory items that are missing, or the client can create these directory elements explicitly with MKCOL.

With restrictions (such as from a macaroon) such directory creation currently requires the MANAGE activity, which allows other actions beyond the scope of this scenario. With this release, the behaviour was changed so that a user with a macaroon that authorises them to upload data into a particular directory will be able to create parent directories to achieve uploading the data.

pool

A regression caused pools that had their size only specified in a layout file to report a size of 8 Exabytes. This issue was fixed.

dCache now supports a DPM-specific HTTP extension that indicates the checksum calculation is not yet complete, avoiding potential data corruption with third-party copies: If DPM is calculating a checksum, then any RFC 3230 (i.e., with a ‘Want-Digest’ header) GET or HEAD request returns ‘202 Accepted’ respond status line and an HTML page as the response entity. Since dCache considers any 2xx response as success, the HTML page was previously accepted as the file’s contents, resulting in data corruption.

dCache pools no longer log a stack-trace for non-bug P2P failures.

srm

The domain ‘.access’ log file now contains log information for grid-site delegation activity, which facilitates debugging of http third-party-copying issues.

transfermanagers

The “restriction check by-passed” warning for each WebDAV-initiated third-party transfer is fixed.

webdav

A user may request a macaroon by making an HTTP POST request to the WebDAV door. This log entry was augmented by the ID and type of macaroon used.

A previous patch needed a bit of an update to ensure that X.509-with-FQAN authenticated third-party transfers with macaroons work under all circumstances. This is now ensured.

xrootd

The --zip option of xrootd clients is now supported.

Changelog 3.2.35..3.2.36

4166b4b29b
[maven-release-plugin] prepare release 3.2.36
acd769fba0
xrootd: add support for kXR_stat on open files
9556512fa0
pool: P2P failures trigger stack-trace
80dfa61625
webdav: obtain FQAN from X.509 credential for gridsite
c1b19f6443
core: avoid sending bad macaroons to gplazma
4b26086b58
webdav: update access log to record macaroon request details
5cea320aff
transfermanager: fix missing path
7ff01148cc
libs: update to commons-compress–1.18
1e84c57b12
macaroons: include macaroon id in error message
635dee7b22
pool: fix pool’s runtime configured size regression (b70b0d9)
e3e03a50dd
core: provide better feedback and logging if a macaroon is rejected
eba086bd1a
pool: update HTTP TPC to support retrying GET and HEAD requests for DPM
1ddd05ec66
srm: add gridsite delegation interface access-log
06f235a099
macaroons: fix logged id
5bcd4d03a8
core: avoid stacktrace on arbitrary CacheException
c5830a6522
[maven-release-plugin] prepare for next development iteration
cdc0d5eb18
pnfsmanager: allow restricted user with UPLOAD to create parent directories

Release 3.2.35

poolmanager

This release increases responsiveness for users that are not allowed to stage files, and for NFS users who access offline files. In cases where such a user issued a read request at the same time that Pool Manager handled a staging request, the first request would block for the duration of the staging – potentially quite a while. From now on, users that are not allowed to stage receive appropriate error messages as soon as possible, without having to wait for anyone else.

xrootd

Support for xrootd mkdir was improved.

Changelog 3.2.34..3.2.35

154c5445b1
[maven-release-plugin] prepare release 3.2.35
22aaf03765
xrootd: update to xrootd4j dependency to 3.2.3
1a1691e3fa
poolmanager: do not squash request if state is not allowed
25815d60f0
[maven-release-plugin] prepare for next development iteration

Release 3.2.34

sysytemd

Systemd did not inherite the system-wide limits and was completely ignoring /etc/security/limits.d/92-dcache.conf. This is now fixed and the limits successfully loaded and enabled as expected.

Changelog 3.2.33..3.2.34

fa184d2
[maven-release-plugin] prepare release 3.2.34
e465689
[maven-release-plugin] prepare for next development iteration
529a5a2
systemd: Add /etc/security/limits.d/92-dcache.conf in the dcache systemd unit and generator.

Release 3.2.33

Changes affecting multiple services

This rlease fixes an issue with WebDAV 3rd-party-copy requests that are authorized using a macaroon that is only valid for writing a specific file.

NOTE: both the webdav door and transfermanagers must be updated before the fix is effective.

pool

In order to help with debugging issues with partial FTP transfers, dCache pools now are able to log considerable information about failed FTP transfers.

This is controlled by the new property pool.mover.ftp.enable.log-aborted-transfers.

webdav

dCache can now transfer data with a remote site, authenticating with that remote site using a delegated X.509 credential, but authenticating locally with a macaroon.

xrootd

This release updates xrootd4j, which should help fix occasional “pad block corrupted” issues with older clients.

Changelog 3.2.32..3.2.33

d390a16982
[maven-release-plugin] prepare release 3.2.33
f26c3650c1
pom.xml: update xrootd4j dependency to 3.2.3
0c6f51c5e4
webdav: use TLS credential directly for gridsite
bd13c21bc2
pool: instrument ftp mover to show partial transfers
9663d40ae5
webdav+transfermanagers: support TPC pull with targeted macaroons
c5a6d0af64
[maven-release-plugin] prepare for next development iteration

Release 3.2.32

frontend

The current release fixed broken directory QoS reporting and now frontend now more accurately describes the QoS of directories; i.e., the QoS that newly written files will receive when written into this directory, assuming none of the targeted pools are volatile.

webdav

the macaroon creation with multiple path restrictions failed with a http error 500 and the error message. This is now fixed and the macaroon creation succeeds when multiple path restrictions are defined.

The current release improved error handling for PROPFIND request.

Changelog 3.2.31..3.2.32

117b68a
[maven-release-plugin] prepare release 3.2.32
40e9387
frontend: fix broken directory qos reporting
c34cba3
webdav: avoid throwing any exception when listing a directory for PROPFIND
e370a41
webdav/macaroon: Fix macaroon creation with multiple path restrictions.
07924ed
[maven-release-plugin] prepare for next development iteration

Release 3.2.31

ftp

dCache now has the ability to log the current status of a transfer at the point the client decided to abort an FTP transfer. This should support a post mortem investigation on why a transfer was cancelled.

nfs

With the current release the timeout of pnfshandler is configurable and nfs door quicker recovers from situations, when a PnfsManager is not available.

Changelog 3.2.30..3.2.31

ae85848
[maven-release-plugin] prepare release 3.2.31
bd2e07b
ftp: add ability to log client-aborted transfers
8e491e4
nfs: make timeout of pnfshandler configurable
1dffc9c
dcache: release dcache-view version 1.3.3
c65318a
[maven-release-plugin] prepare for next development iteration

Release 3.2.30

NFS

When two clients A and B operate on a file in quick succession, A opening the file and B deleting it before LAYOUTGET is called, dCache puts the transfer into the list of active transfers and returned NFS4ERR_NOENT. If a client tries to optimize the corresponding CLOSE call away, as some do, the entries are never removed from the list, effectively creating a leak.

This problem was fixed. Clients now receive an NFS4ERR_STALE message in those cases.

core

Certain transfer failures, such as attempting to use a space-reservation that has insufficient capacity, resulted in the door eventually reporting a time-out problem to the client.

A typical error message would resemble

Request to [>SpaceManager@local ... ] timed out.

This problem was traced to an internal misconfiguration of a messaging component and is fixed from this release onwards.

frontend

The reporting of a file’s QoS status in frontend was improved. Files that are being scheduled for moving to tape are now reported as ‘tape’ instead of ‘disk’.

pool

A bug was fixed that occasionally caused problems with the pools’ Berkeley DB. This could, for example, be triggered by removing files which were in a flush queue.

A typical error message was, e.g.

27 Aug 2018 12:09:33 (cat2_lhcbtape) [Frontend-dcacheview PoolDataRequest] Fault occurred in repository: Internal repository error. Pool restart required: : CacheExcept
ion(rc=204;msg=Meta data lookup failed and a pool restart is required: (JE 7.3.7) Environment must be closed, caused by: com.sleepycat.je.ThreadInterruptedException: En
vironment invalid because of previous exception: (JE 7.3.7) /space/lhcb/tape/pool/meta java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException may cause
incorrect internal state, unable to continue. Environment is invalid and must be closed.)
27 Aug 2018 12:09:33 (cat2_lhcbtape) [Frontend-dcacheview PoolDataRequest] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server,dead): Pool restart req
uired: Internal repository error

webdav

Web clients (such as web-browsers) make OPTIONS pre-flight requests to discover what they are allowed to do, according to the CORS standard.

Unfortunately, some web-browsers make the OPTIONS request without presenting any credentials. If the resource is within a protected directory then dCache currently fails the OPTIONS request.

This release introduces a new behaviour where such requests will always succeed, so that browser pre-flight requests are not hampered.

Changelog 3.2.29..3.2.30

a47eea0e8f
[maven-release-plugin] prepare release 3.2.30
9bc218ab2a
nearline-provides: do not interrupt processing thread on cancel
303de641f9
nfs41: invalidate open-state on layoutget if file is removed
73a5f72db9
webdav: always respond to OPTIONS request
129188e8ff
core: ensure pool/poolmanager communication receives errors
bb53e518f3
frontend: add targetQoS for not-yet-flushed tape files
10d95ca99a
[maven-release-plugin] prepare for next development iteration
4cbee39946
dcache: release dcache-view version 1.3.2

Release 3.2.29

gplazma

The OidcAuthPlugin plugin was updated so that users whos op does not claim name, and does not claim given_name nor family_name can use dCache.

pool

This release fixed the log stack-trace for queue admin commands and now bad admin input for the following admin commands no longer results in a stack-trace being logged:

  • queue activate
  • queue activate class
  • queue remove class
  • queue suspend class
  • queue resume class
  • queue remove pnfsid

poolmanager

NPE is fixed when staging files back from tape and poolmanager.enable.cache-hit-message is true.

webdav

The current release updated default credential delegation for third-party copy so that now requesting a third-party copy using a macaroon does not trigger a failed attempt to OpenID-Connect delegation.

Changelog 3.2.28..3.2.29

608c97c
[maven-release-plugin] prepare release 3.2.29
63e6a6b
poolmanager: fix NullPointerException when staging files and reporting hits
65a8d62
gplazma: oidc fix FullNamePrincipal creation
7fb4034
libs: update jetty to version 9.4.11
013c846
pool: ‘queue’ admin commands not the log stack-trace on bad arguments
7711b70
webdav: update default credential delegation for third-party copy
28b9c34
[maven-release-plugin] prepare for next development iteration

Release 3.2.28

history

This release fixes a bug that could cause startup errors in the history service in the face of network errors.

many

Remote pool monitor would occasionally log stack traces from exceptions
when a domain shut down due to an interrupt. This has been fixed, reducing the number of irrelevant log entries in such situations.

Changelog 3.2.27..3.2.28

ba9e256593
[maven-release-plugin] prepare release 3.2.28
11c703f059
dcache-history: handle Gson syntax errors explicitly
5088923788
cells: add handling of RemoteProxyFailureException nested InterruptedException to UncaughtException handler
7e1e2a4bf9
[maven-release-plugin] prepare for next development iteration

Release 3.2.27

nfs

dCache 4.0 and 3.2 now use nfs4j version 0.15.4, which includes bugfixes for rarely observed deadlocks and incomplete directory listing over nfs.

Changelog 3.2.26..3.2.27

406a375f8f
[maven-release-plugin] prepare release 3.2.27
a638309921
pom: update nfs4j–0.15.4 bugfix version
79a88a12e4
[maven-release-plugin] prepare for next development iteration

Release 3.2.26

pool

HTTP responses now contain more meaningful messages along with the HTTP response codes, instead of only just showing stock messages like “400 Bad request”.

Changelog 3.2.25..3.2.26

86f0c693ff
[maven-release-plugin] prepare release 3.2.26
d77154fa4a
pool: update HTTP mover to report errors as HTTP status message phrase
6c809c0f4f
[maven-release-plugin] prepare for next development iteration

Release 3.2.25

resilience

Resilience suffered from a bug that would lead to a NoSuchElementException when a pool name no longer mapped to a location known to the Resilience service. This issue has been fixed.

When multiple pools go offline it is possible that all replicas for a given resilient file become unreadable. If the file is not CUSTODIAL, and thus cannot be restored from tape, the discovery of such a file during scanning will generate an error in the ‘history errors’ listing, in the resilience domain .resilience log, and will also raise a general alarm concerning the pool.

There currently exists a command, ‘inaccessible’, which generates a listing of the pnfsids on a given pool which in the current state of dCache have no readable replicas. However, this command takes a while to complete (asynchronously), and the output is written to a file which must be viewed by logging in.

This release introduces ‘refering pool’ information to the error output so that grepping the resilience log for a given pool becomes easier, and adds options to the command to check further details.

Changelog 3.2.24..3.2.25

6f85108e66
[maven-release-plugin] prepare release 3.2.25
5419c0a8a3
dcache-resilience: improve inaccessible file accounting
bc803be5d0
dcache-resilience: skip invalid cancel filters
50ce7254b7
[maven-release-plugin] prepare for next development iteration

Release 3.2.24

scripts

A regression in the dcache pool convert command was fixed; the command works again.

scripts

The instructions that are printed out once dcache pool convert completes successfully now correctly point to the property that needs to be updated, namely pool.plugins.meta.

Changelog 3.2.23..3.2.24

427f08e306
[maven-release-plugin] prepare release 3.2.24
c06f09b136
pool: fix ‘dcache pool convert’ command
90a41ca479
scripts: update reference to configuration property
ce0430201e
pool: fix metadata migration tool to use Path
c7772ea933
[maven-release-plugin] prepare for next development iteration

Release 3.2.23

pool

This release improves dCache’s robustness against network errors: In case registering a file with PNFS manager fails due to a timeout, the request is retried transparently.

Changelog 3.2.22..3.2.23

bb4d51dd22
[maven-release-plugin] prepare release 3.2.23
a7739d65b9
vehicles: fail-fast on invalid path
11ed4c7cea
pool: retry request to pnfs manager if timed out
b76775f728
[maven-release-plugin] prepare for next development iteration

Release 3.2.22

door

The current release added support for a door advertising multiple hostnames or IP addresses. dCache doors can now advertise multiple interfaces, including DNS aliases.

webdav

Milton’s OPTIONS handler was returning a 404 error if an OPTIONS request targets an entity that did not exist. This behavior deviated from Apache httpd server and was resulting in failed uploads for dcache-view. The current release fixed.

Changelog 3.2.21..3.2.22

5b8e32b
[maven-release-plugin] prepare release 3.2.22
2aafba0
gplazma.properties: hint to enable roles
7fe48f1
doors: support advertising multiple addresses in LoginBroker
7ded7cf
webdav: do not return 404 for OPTIONS request targeting absent entity
64cf055
[maven-release-plugin] prepare for next development iteration

Release 3.2.21

dcache-resilience

There was a small regression in the way resilience computes the number of operations necessary to adjust copies when a storage unit definition changes.

The current rellease fixed computation of operation count when storage requirements change.

ftp

In order to aid diagnosing problems when FTP response being lost, now dcache logs failures to wrap/encrypt responses.

webdav

The current release improved error handling for client authentication with OpenID-Connect. A more complete set of information is now logged if OIDC delegation fails, supporting the ability to discover why the delegation failed.

Changelog 3.2.20..3.2.21

e10182e
[maven-release-plugin] prepare release 3.2.21
163d4bb
scripts: add support for parsing ZooKeeper transaction logs
38ae2b7
ftp: log failures to wrap/encrypt responses
5b2ba4b
dcache-resilience: fix computation of operation count when storage requirements change
4ef97b8
webdav: log errors if OIDC delegation fails
7e86358
[maven-release-plugin] prepare for next development iteration

Release 3.2.20

ftp

Error reporting in the FTP service has been improved: in some mixed IPv4 / IPv6 scenarios, only unclear error messages were reported.

Changelog 3.2.19..3.2.20

ecd42ce507
[maven-release-plugin] prepare release 3.2.20
7827470210
ftp: returned error is too vague for meaningful investigation
77c9337778
[maven-release-plugin] prepare for next development iteration

Release 3.2.19

nfs

Situations, where selection process was incomplete could not be manually recovered When selection process incomplete, due to PoolManager restart, there was no way to trigger a new selection. The current release added two new commands to nfs door: transfer retry and transfer forget.

The first command manually re-activates existing transfer by re-trying selection process. The second one should be used to completely ‘forget’ the stale transfer and let client to trigger a fresh selection process.

Changelog 3.2.18..3.2.19

d580e01
[maven-release-plugin] prepare release 3.2.19
df58f42
nfs: add commands to reactivate stale transfers
69d822a
[maven-release-plugin] prepare for next development iteration

Release 3.2.18

gplazma

gplazma now supports a Fermilab-specific authorization data source in JSON format.

pool

During active ftp transfers, connection problems would lead to the rather unhelpful error message “451 General problem”. This error reporting was refactored, so that diagnosis of the cause is now greatly facilitated.

poolmanager

A potential NullPointerException (that was not observed in real-world usage until now) was fixed in Pool Manager.

resilience

A correction to resilience’s error handling results in no more reports on non-resilient (but corrupted) files.

spacemanager

In order to facilitate debugging, Space Manager now logs link-group related content in greater detail.

Changelog 3.2.17..3.2.18

3914a3e46b
[maven-release-plugin] prepare release 3.2.18
9d01c5a804
poolmanager: fix migration command if named pool is removed
ed541dbd38
dcache-resilience: repair over-aggressive handling of broken file messages
bce6a346d6
pool: fix error message for failed active FTP transfers
1e73d5f6a4
fix the project version in pom.xml
e58062e696
gplazma-fermi: fix last modified check in junit test
fef97c5dd1
spacemanager: add remote pool monitor debug logging
abf7548dd2
gplazma-fermi: add mapping plugin to support VO group and username from file
8b228e192a
[maven-release-plugin] prepare for next development iteration

Release 3.2.17

Changes affecting multiple services

This version removes the (by now unused) directory /var/lib/cell-info from a default installation. The directory was previously used to store cell info data. With the introduction of the history service in dCache 3.2.1, this became obsolete.

A small bug-fix addresses wrong directory permissions on tar or Debian packages, where the directory /var/lib/dcache/pool-history had the wrong permissions.

The deprecated properties for configuring alarms have now finally been made unavailable.

chimera

An internal update enables chimera to use PostgreSQL 10.

nfs

A modification in IP address handling greatly increases the speed of NFS client access for Linux clients in mixed IPv4/IPv6 environments.

Prior to Linux 4.12, Linux clients with only an IPv4 address would wait for (timeout * retry) seconds when connecting to pools with both a v4 and v6 address. This was fixed upstream in Linux 4.12, but that fix was not backported to e.g. RHEL 7 yet.

Changelog 3.2.16..3.2.17

3c3d887c68
[maven-release-plugin] prepare release 3.2.17
ab5d6e6e97
nfs: filter out IPv6 DS addresses if client connected with v4
278d7e5a33
chimera: adjust postgres driver provider to new version schema
1f18b011be
skel: remove extraneous cell-info dir
2872ce17ba
packaging: add missing chown and chmod on pool-history
742a3666fc
skel: make deprecated alarms properties forbidden
24939dc5ac
[maven-release-plugin] prepare for next development iteration

Release 3.2.16

resilience

Logging for cases where file replication was fatally aborted was improved. Previously, alarms messages pertained to the PNFSID of the affected files. In rare cases, like when facing network congestion, many hundreds of alarms could be created. With this change, alarms messages refer to the storage unit, and a suffix based on an hourly timestamp is added to the alarm message. The alarm will thus be incremented during the hour but a new alarm will be created (only) hourly; in this way, those receiving email alerts will receive them once an hour.

Changelog 3.2.15..3.2.16

b2195c4e2f
[maven-release-plugin] prepare release 3.2.16
6c98c2b867
alarms: fix broken path
5ec525255d
[maven-release-plugin] prepare for next development iteration
84f6af6218
substituted Calendar for Instance which was failing.
529c10c14d
dcache-resilience: avoid spamming alarms with abort messages

Release 3.2.15

ftp

A bug (that was not observed in real-world settings yet) that might have caused NullPointerExceptions was fixed in the ftp service.

httpd

dCache will no longer log a stack-trace if HTTP requests are made asking for information from the info service when the info service is not running.

poolmanager

An earlier change in PoolManager introduced regular broadcasting of the stage request queue to various internal listeners. In some cases, this could lead to erroneous NoRouteToHost error messages being logged. These error messages are now being suppressed.

dCache 3.0 introduced a regression where a dCache domain does not start up if it hosts a poolmanager with poolmanager.conf containing either the “rc set sameHostCopy” or the “rc set sameHostRetry” command. This regression was fixed.

resilience

Error handling within the Resilience service was improved.

Changelog 3.2.14..3.2.15

6e95e936af
[maven-release-plugin] prepare release 3.2.15
525705459f
dcache-resilience: handle properly RuntimeExceptions from tasks
be6abaf5c7
ftp: ensure adapter is closed
465d2c04b1
ftp: remove rare NullPointerException when proxying data
40b57484c1
httpd: do not log an exception if info cell not running
8cea3b4568
poolmanager: silence NoRouteToCell for stage queue topic
71a1728ed2
poolmanager: fix poolmanager startup with certain poolmanager.conf content
741864a1f6
[maven-release-plugin] prepare for next development iteration

Release 3.2.14

dcache-resilience

When a checksum or broken file message/error is generated, Resilience makes a best effort to (a) remove the broken copy and (b) make another replica. This, of course, is not always possible, particularly if the broken file is the only accessible copy. This resulted in faulty behavior particularly the thrashing noted in the case of a restaging operation which results in a checksum error. This is now fixed.

The current release improved error handling for resilience. It fixed unnecessary Migration Task exceptions resulting from source pools with no replica in the repository.

Now it should be possible for Resilience to use pools blocked only for writes from doors.

packaging

Upgrading to dCache v3.2 (or newer) was resulting in a broken dCache installation due to a missing services.sh file. This is now fixed and upgrade to dCache v3.2 (or newer) from dCache v3.1 (or older) no longer breaks dCache by removing /usr/share/dcache/lib/services.sh.

Changelog 3.2.13..3.2.14

a7c71df
[maven-release-plugin] prepare release 3.2.14
d455f9d
bad commit put DOWN twice
d848910
dcache-resilience: define non-writable pool to mean p2p-client is disabled
8ce3c06
dcache: fix remote pool monitor wait bug
dac5e17
dcache-resilience: repair handling of broken files*
f2cb660
packaging: check ‘services.sh’ after old rpm removed
0876aa8
[maven-release-plugin] prepare for next development iteration
e7d5e16
dcache-resilience: fix bug in source handling with Clear Cache Location messages

Release 3.2.13

cells

The current release added explicit ZooKeeper/Curator monitoring. Events generated by ZooKeeper and Curator are now logged in a new, which may help diagnose problems that are suspected to come from bad ZooKeeper interaction.

frontend

The current release improved the error handling to work with Jackson exceptions.

Changelog 3.2.12..3.2.13

353bd42
[maven-release-plugin] prepare release 3.2.13
80d60ae
dcache-resilience: fix wrong assumption about error type in Message
673c067
cells: add explicit ZooKeeper/Curator monitoring
1fa42cd
frontend: Map requests with bad JSON to HTTP 400 Bad Request status code
8b24220
[maven-release-plugin] prepare for next development iteration

Release 3.2.12

nfs

NFS door has been updated to return NFS4ERR_LAYOUTUNAVAILABLE for DOT files.

star

The current release improved documentation to help dCache admins to have a better understanding of how to generate StAR record.

The current release fixed fix printing exception error message for dcache-star script if there’s a problem when run with newer versions of Python.

Changelog 3.2.11..3.2.12

5ea66f8
[maven-release-plugin] prepare release 3.2.12
8c1e09d
nfs: return LAYOUTUNAVAILABLE for DOT files
78c57d8
star: fix printing exception error message
4f62d5a
star: update documentation to provide better description of script
554bcc0
[maven-release-plugin] prepare for next development iteration

Release 3.2.11

info

The info service collects information about who is allowed to reserve space. Since some of this information, like VOs, usernames and gids, may be considered sensitive information, this update allows admins to control whether or not to publish them. The default behaviour is unchanged from the previous behaviour, i.e. info publishes everything. If a site admin wants to change this, the info.limits.show-only-vo-authz property can be set to true.

nfs

Accessing a nonexisting file on recent NFS implementations could cause a FileNotFoundChimeraFsException, which is now caught and properly handled.

pool

In rare circumstances, running info on a pool could cause a NullPointerException. This issue has been fixed.

scripts

The dcache script and manpage still refered explicitely to Java 6. This patch changes the phrasing of the respective text.

Changelog 3.2.10..3.2.11

4791b79689
[maven-release-plugin] prepare release 3.2.11
0dae9f19f4
nfs: fix ServerFault on FileNotFoundHimeraFsException
952fd1d2c0
[maven-release-plugin] prepare for next development iteration
b68757dd48
scripts: update reference to JDK to avoid mentioning specific java version
f1c860a74b
info: allow admin to control whether non-VO / non-FQAN identities are shown
64ac107458
pools: fix NPE from info command at startup

Release 3.2.10

chimera

A database deadlock was observed in some rare situations with the latest 3.2 releases. This patch resolves the issue, ensuring trouble-free chimera operation.

Changelog 3.2.9..3.2.10

f50d7d22c6
[maven-release-plugin] prepare release 3.2.10
70c1ba902f
chimera: fix deadlock in Postgres driver
89bb1f959d
[maven-release-plugin] prepare for next development iteration

Release 3.2.9

chimera

The current release fixed previously introduced issues for lost+found directory permissions. Now, the lost+found directory permissions is updated without causing problems if that directory has been removed or permissions have been modified.

Changelog 3.2.8..3.2.9

44a6ec8
[maven-release-plugin] prepare release 3.2.9
91c30e6
chimera: correct previous attempt to fix ‘lost+found’ directory permission
0c61067
[maven-release-plugin] prepare for next development iteration

Release 3.2.8

nfs

The current release fixes transfer leak, if the door failed to start a mover.

pnfsmanager

The current release improves documentation for set log slow threshold admin command help.

spacemanager

dCache now allows an SRM client to specifying from which linkgroup a reservation should be made.

When trying to upload into dCache using a space-token where there is no selectable link for this operation then the user was presented with a generic error message; for example,

No write links configured for [net=131.169.71.98,protocol=GFtp/2,store=dot:user@osm,cache=,linkgroup=].

This behavior is changed now and an improved error message is returned to the user if they attempt an upload data into dCache using a space-reservation in a way where poolmanager configuration prevents the upload.

srmclient

The srm-reserve-space command now supports a user choosing from which linkgroup a reservation should be made, provided the corresponding dCache also supports this.

webdav

The current release improved error handling when dCache is full.

Changelog 3.2.7..3.2.8

558f21e
[maven-release-plugin] prepare release 3.2.8
1781cee
systemtest: work with new OpenSSL DN format
815a561
spacemanager: allow SRM clients to specify linkgroup in reserve requests
086d559
srmclient: add support for specifying linkgroup when reserving space
636fdce
spacemanager: provide space-specific error message on bad upload
f758653
webdav: return 507 Insufficient Storage when dCache is full
606d7ed
pnfsmanager: update slow logging admin command help
bc15f25
nfs: fix transfer leak, if the door failed to start a mover
2490982
[maven-release-plugin] prepare for next development iteration

Release 3.2.7

cells

dCache no longer logs stack-traces when running multiple cells with the same name.

frontend, webdav, httpd

With high availability, it is now possible to run redundant services. In the case of Pool Manager, restore requests are distributed to the separate instances, so as to avoid staging the same file twice. This means, however, that the full list of current restore requests is partitioned among the pool manager instances. To receive a full listing, it is no longer possible to query for them on the named PoolManager queue, since this means the response will be from the first responder only. The current issue fixed this issue and all current http services report all restore requests. The current release fixed this issue.

It is important to note that, upgrading nodes running frontend, webdav or httpd to version 3.2.7 (or newer) requires upgrading nodes running poolmanager at least to 3.2.7 (or newer) version.

pool

For certain failures,the pool was logging transfer failures twice. This is now fixed.

rpm

dCache ensures now that user ‘dcache’ is a member of group ‘dcache’.

srmclient

The same error has been logged multiple times resulting in stack-trace. This current release fixed both issues.

srmmanager

Support tickets indicated that for some services it was unclear how to fix a configuration that still has assignments for either srmmanager.net.port or srmmanager.net.local-hosts. The current release updated the documentation describing how to fix their configuration after upgrade.

star

The current release Introduced new property star.db.*, which makes possible now to run PostgreSQL on non-standard ports can use STAR.

Changelog 3.2.6..3.2.7

ff339ca
[maven-release-plugin] prepare release 3.2.7
93cbaf0
httpd, dcache-frontend: support requests for restore listing when there are multiple pool managers
58b0338
[maven-release-plugin] prepare for next development iteration
8fd4ba4
pool: fix double logging on remote FTP transfer error
a1c9b6e
srmclient: avoid stack-trace and repeated logging
7deae9d
srmmanager: provide better hints on obsolete properties
29394bd
pool: Fix how certain bugs are logged
3d8e83d
star: support PostgreSQL running on non-standard TCP ports
64b97ef
rpm: don’t assume existing dcache user is member of dcache group
bbf75cb
cells: don’t log stack-trace on starting cell with same name as running cell

Release 3.2.6

admin

The admin interface reported an attempt to connect to an absent cell as a bug. The current release fixed the issue.

httpd

Requests to httpd targeting an unknown resource was returning 200 OK response code. Nevertheless the 404 NOT FOUND response would be closer fit. This is now fixed.

maven

The global dcache.service was missing from the built packages. The current release fixed this problem and dcache.service is now included in Debian packages.

nfs

The current release corrects inaccurate documentation of nfs.enable.pnfsmanager-query-on-move.

pool

Closing dcap mover connection no longer logs a stack trace.

statistics

Timeout in contacting PoolManager no longer results in a stack-trace being logged.

Changelog 3.2.5..3.2.6

6956c9b
[maven-release-plugin] prepare release 3.2.6
b695c88
statistics: avoid stack-trace on internal timeout
b1201b6
nfs: fix documentation of nfs.enable.pnfsmanager-query-on-move
d327b87
admin: do not report attempts to connect to missing cell as a bug
556e2d4
maven: include dcache.service in Debian packages
d830799
pool: fix stack-trace when closing dcap mover connection
44b5541
httpd: return 404 status code on an unknown page
995b5bf
[maven-release-plugin] prepare for next development iteration

Release 3.2.5

Changes affecting multiple services

This release addresses several issues with systemd support and packaging on Debian systems.

The rsyslog configuration has been updated from using language version 7 to version 8.

There is a new systemd service unit dcache.service that can be used to have all dCache domains started with only one startup call.

An installation bug with the Debian package was fixed that prevented a successful installation because of a missing cell-info directory.

Under systemd, log files are again back in their usual location under /var/log/dcache/$DOMAIN.log.

Changelog 3.2.4..3.2.5

bc7d6088db
[maven-release-plugin] prepare release 3.2.5
9ce9bc9ad4
switch to rsyslog v8 configuration language
83a779d06f
move logfiles back to /var/log/dcache
d19373ae34
systemd: adding a global dcache.service which pulls in dcache@*.service
0dff5cd0e0
packaging: include empty var directory: ‘cell-info’
d6696bcbd1
[maven-release-plugin] prepare for next development iteration

Release 3.2.4

Changes affecting multiple services

dCache no longer logs stack-traces if a Java VirtualMachineError occurs. This is unnecessary as dCache was (presumably) working fine until Java discovered a problem.

chimera

Sites updating to dCache 2.15 or later might observe that a lost+found directory with incorrect permissions was created during the update. This patch ensures correct permissions. Since we cannot know if the current permissions in lost+found are intended, this patch does not modify any existing lost+found directory permissions.

frontend

An irrelevant stack trace could occasionally be logged by the frontend. This patch corrects that issue.

During service interruptions, timeouts have been logged at WARN level until now. The logging level has been changed to INFO with this release.

An irrelevant stack trace was occasionally logged by the frontend service. This release corrects that.

history

Currently, the history service will block dCache startup for history.service.poolmanager.timeout (2 minutes) if the service is started while PoolManager is not running. This blocking was removed, so that system starts are quicker and more reliable.

pool

The sweeper freecommand no longer logs a stack trace if it is started with incorrect input information.

An irrelevant stack trace was logged by the pool. This release corrects that.

Changelog 3.2.3..3.2.4

4f25e874fe
[maven-release-plugin] prepare release 3.2.4
a760f18d72
chimera: update schema migration when creating ‘lost+found’ directory.
fc9f218337
pool: fix stack-trace on bad command input
e17c58e654
pool: fix stacktrace on FaultEvent logging
3926ead39c
system: Don’t log stack-trace on fatal JVM error
a924c2175f
dcache-frontend: fix ConcurrentModificationException in ReadWriteData
37e229cffe
srmclient: refactor ‘srm’ helper script, enforcing environment variables
bfda91c5cc
history: do not block on startup if PoolManager is not running
732c5e2b9c
dcache-frontend: adjust level of timeout logging
91b3d01342
dcache-frontend: adjust REST API for Pool Info Resources
50652e65ea
[maven-release-plugin] prepare for next development iteration

Release 3.2.3

alarms

Until now, the sorting order of alarms did not provide a correct ordering for all types of alarms. With this release, alarms are now implicitly ordered by at least their latest modification timestamp.

frontend

The cause of a stack-trace during system shutdown has been fixed.

nfs

The handling of directories with hard links in them has been improved, providing NFS clients with a way to correctly list them in all cases.

resilience

One of the features of resilience is the enforcement of file partioning on pools according to pool tags. The pool tag restrictions are observed whenever a file is copied. In addition, it is rechecked when a storage unit is updated, in order to make sure the files are distributed correctly according to the new requirements. This is done by removing the offending copies and recopying them in a new location.

Should files get redistributed, however, by rebalancing or a migration job, it is possible that the partitioning will be violated, since only resilience observes it.

The resilience service now verifies that files are distributed according to the requirements specified by pool tags while doing periodic scans (or scans initiated through the admin command).

statistics

A possible race condition was removed from the implementation of the create stat admin command.

Changelog 3.2.2..3.2.3

f4a96d52d6
[maven-release-plugin] prepare release 3.2.3
a95f3a56d2
statistics: fix race in “create stat” admin command
17d15a2bf9
srm: fix stacktrace on database failure
d89fd3086c
dcache-frontend: fix shutdown not to cause stack trace in collection services
245e40df10
alarms: revert LogEntry.compareTo() to throw NPE on null object
5bec23331b
nfs: change the way how directory cookies are generated
ed2b1a7138
resilience: force tag partition checking on scans from admin command and periodic checks
fcfe6f0437
alarms: fix natural order comparator to use timestamp first
c33a097997
[maven-release-plugin] prepare for next development iteration

Release 3.2.2

alarms

Shutting down dCache using dcache stop is now faster.

dcap

A pool that has gone offline and comes back up again may become very slow to respond due to a large amount of superfluous error messages to dcap clients that disconnected in the meantime. This patch ensures a more responsive reaction to these cases by introducing a time-to-live value for such messages.

frontend

In cases where very many alarms need to be processed by the frontend, they are now fetched in batches, ensuring better responsiveness of the system.

The way in which services send large amounts of data to the frontend, as for example the active transfers or staging requests lists, has been made far more efficient and performant.

pool

Error reporting was improved for cases of IO errors in pools.

Changelog 3.2.1..3.2.2

983e3644bb
[maven-release-plugin] prepare release 3.2.2
132d8fa428
alarms: fix shutdown timeout
c1d1324c07
pool: avoid ‘null’ and other nondescript error messages
520bcdf633
dcache-restful-api: change the way in which transfers and restores support paging
a775365ef1
[maven-release-plugin] prepare for next development iteration
3236047d4d
dcap: add TTL information to dcap messages
0e9dd30126
alarms: remove default value for LogEntry received
b4676a9295
dcache-restful-api: add offset and limit to fetch of alarms

Release 3.2.1

Changes affecting multiple services

Many dCache components use RemotePoolMonitor to provide fast access to the information that PoolManager has about pools. In order to facilitate system diagnosis, the ‘info’ admin command was augmented by information about the current status of the RemotePoolMonitor.

frontend

Nearline movers now display the elapsed time they ran for in the dCache frontend.

The RESTful API has received support for disabling and enabling of pools as well as killing movers.

Minor bugs in the history service were fixed.

The timeouts for the collectors run by the frontend service were reduced to 1 minute (2 for the history service, which needs to ping all pools in an instance) in order to provide more immediate change information in the frontend.

The RESTful API now offers access to the data about PoolSelectionUnit that was previously available on the webadmin pages.

history

In making the pool info service stateless, collection of timeseries information for queue status and file lifetime on the pools was moved into a separate service (history).

When pools are unavailable to the history service, as is possible during restarts, it is possible that history data is lost. This patch corrects that issue, so that history is preserved under all circumstances.

pool

A bug in gfal2 results in FTP transfers being aborted some 50 ms after being initiated. This results in the door killing the mover shortly after the pool received the PoolDeliverFile message. If the mover is not queued, but not yet fully started, this may lead to the pool disabling itself. This patch corrects that problem, ensuring that the pool continues to run despite any aborted transfers.

system-test

The system-test admin interface configuration was updated to allow admin logins after ssh user restrictions are now significant.

xrootd

In the 4.7 releases, the xrootd client started enforcing protocol requirements for kXR_login which, unfortunately, broke access to dCache. The xrootd client expects an answer with a 16-character session ID from the door and then the pool after the redirection. Without this ID, the client would retry (without success) repeatedly and appear to hang.

dCache’s xrootd implementation has been augmented with the session ID, enabling it to work with xrootd clients of version 4.7 and up.

Changelog 3.2.0..3.2.1

bd6004d84f
[maven-release-plugin] prepare release 3.2.1
caeda8c243
many: add diagnostic information about remote pool monitor
702caa2af1
pool: dont disable pool if mover cancelled before open
4b58778caa
systemtest: authorise developer to login as ‘admin’ via ssh
7a81aa71f0
dcache-history: fix bug which forces overwrite of data when pool unavailable
f34e87ccd9
dcache-restful-api: fix aggregation issues for pool info and move aggregation to history service
8a012e6576
dcache: add elapsed time to NearlineData JSON
78493a3d86
dcache-restful-api: fix configuration bug in billing collection utils
996a1e2310
[maven-release-plugin] prepare for next development iteration
871182acec
dcache: fix NPE bug in TransferInfo.toFormattedString
614a805535
dcache-restful-api: avoid NPE in PoolDataRequestProcessor
41767ab043
dcache-restful-api: lower collector timeout defaults
32336bacbb
dcache: fix bug in PoolSelectionUnitV2 match()
4e8edebf38
dcache-xrootd: Fix login handshake to support xrootd clients (> 4.7.0)
52eaf075c5
dcache-restful-api: Add POST for enable/disable or kill mover to pool info service
34720e0c43
dcache-restful-api: add selection resource and providers
31930df123
dcache-xrootd: (WIP) Add support to the xrootd (kxr)posc flag in (kXR)open.

Release 3.2.0

Authentication

We’ve added support for macaroons in the webdav door.

Macaroons are a new idea that comes from research by Google. They are bearer tokens that allow the bearer to do something, without requiring the users to identify who they are.

A macaroon may be limited by caveats which only allow the bearer to do certain things. These limitations could be nearly anything. A caveat could make the macaroon time-limited (e.g., only good for five minutes), location-limited (e.g., only from a particular IP address), or anything else. You can learn more about macaroons from an Air Mozilla talk called Macaroons: Cookies with Contextual Caveats for Decentralized Authorization in the Cloud.

Macaroons are a building-block technology, so have many potential uses. They may be used by a web-portal to allow its clients (of which dCache knows nothing) direct access to files stored in dCache. They may be used to authorise third-party transfers without requiring X.509 credential delegation. They may be used to create a link that allows sharing of resources. There are likely many other uses for them.

Requesting a macaroon is as simple as making a POST request with a Content-Type of application/macaroon-request. This returns a time-limited (but otherwise unlimited) macaroon. Additional caveats may be added by dCache (included as JSON in the POST request) or subsequently by anyone else. dCache supports a rich set of possible restrictions, including namespace (e.g., only a particular file or directory), time (e.g., only the next 5 minutes), location (e.g., only from a particular IP address or subnet), and activity (e.g., only downloads).

Creating a macaroon takes very little CPU and no storage in dCache. Therefore it is safe to request many macaroons; for example, a web portal could create a seperate macaroon for every file its user wishes to view.

A macaroon is a bearer token. To use a macaroon in an HTTP request, just add it to the Authorization header value with a prefix bearer, just like with an OpenID-Connect access-token.

Unfortunately, not all HTTP clients allow a user to add a bearer token to the Authorization HTTP header; for example, a user cannot easily include a bearer token in their web-browser. To support these clients, dCache accepts URLs that include the ‘authz’ query parameter, with the value used as a bearer token (e.g., https://dcache.example.org/path/to/file?authz=<token>). This works for both OpenID-Connect access-token or macaroon.

The webdav directory browsing has been updated so that any query parameters in the original request are included in all the generated page’s navigation and download links. This makes it easy to navigate through the webdav interface using a macaroon or OpenID-Connect access-token: just add ?authz=<token> to the URL.

Roles

It may be desirable for dCache to behave differently for the same user between different login sessions. For example, an administrative user may wish sometimes to interact with dCache as a normal user (for testing purposes or to reduce the risk of mistakes), and other times to interact with dCache with effectively root-like privileges.

This functionality has been enabled by introducing the concept of user roles in dCache. A user may log in requesting zero or more roles and the login process chooses which roles (if any) to enact.

For the HTTP-based services (webdav and RESTful), roles can be requested by appending to the username a ‘#’ followed by a comma-separated list; for example, user ‘fred’ wishing to adopt ‘role-a’ and ‘role-b’ would authenticate with the username ‘fred#role-a,role-b’.

This release introduces limited support for the ‘admin’ role.

The ‘admin’ role is intended to give users root-like permissions without the user knowing the root credentials and with the user’s identity still being bound to that user. The result is that the user with the ‘admin’ role is able to do everything, but files created by that user are still owned by that user and log entries show which user made any changes.

An ‘admin’ role user has a root directory of / and no restrictions.

The functionality is enabled by adding a new gPlazma session plugin, ‘roles’, to the gplazma.conf file, like this:

session  required    roles

A new property gplazma.roles.admin-gid = 0 has been added to gplazma.properties. A user must be a member of the above specified group to be authorized to obtain the admin role.

External networking

The handling of external connections has changed for some protocols.

The SRM network configuration was simplified, and dCache can now work with multiple SRM services on different ports.

The deprecated configuration properties srmmanager.net.port and srmmanager.net.local-hosts are no longer needed.

admin ssh public keys

In previous versions of dCache, any user who has their ssh public key in the admin.paths.authorized-keys file (/etc/dcache/admin/authorized_keys2 by default) can log into the admin interface as any user.

With this version of dCache, a public key is limited to authenticating as a single user. The standard ssh public key format does not have any way to describe for which user the line allows authentication; therefore, dCache uses the comment field to provide this information.

The comment field appears after the base64-encoded text of the public key. dCache requires the comment contains the desired username followed by an @ symbol; without this, the line is ignored. The text after the @ does not matter. The username describes for which user the corresponding private key may be used to authenticate.

The following (truncated) line shows an example authorized_keys2 file that allows the use of a public key to authentication as user admin:

ssh-rsa AAAAB...fPQ== admin@localhost

Internal networking

Network communication between core and satellite domains has, until now, been unencrypted. dCache 3.2 adds the possibility to use TLS encryption between domains, which should be extremely helpful for building larger, distributed setups.

A number of new configuration properties control encrypted cell communication.

Set dcache.broker.core.channel.security to none in order to listen only for plain-text communication on dcache.broker.plain.port, to tls in order to listen only for TLS encrypted communication on dcache.broker.tls.port or to none,tls in order to listen for both plain and encrypted communication.

dcache.broker.core.client.channel.security and dcache.broker.satellite.channel.security can be used to enable encryption between core and satellite domains.

In order to ease managing encrypted communications, the LocationManager cell, lm, now has three new commands available: ls, get core-config and set core-config, which can be used to query and set the operating mode for domains.

Regarding backwards compatibility, the introduction of TLS encryption deprecates the dcache.broker.port property. Satellite domains which do not receive a configuration update can still connect to updated core domains which listen to plain communication, and core domains which are not updated can still connect to updated core domains.

Admin improvements

The ps command of the System cell has been overhauled. In particular, the output of the -f option has been cleaned up and extended.

The admin cell now generates access log events containing connection and disconnection, as well as authentication, events.

Key-based authentication has been improved by enforcing that a particular key can only log in with the user name specified in the authorized_keys2 file. Upon upgrade, admins should review this file and ensure that keys are mapped to the correct user name.

Namespace

Tags are now reference-counted like any other filesystem object. This avoids an expensive database query upon file deletion that in some cases has led to bottlenecks, in particular when PostgreSQL statistics were inaccurate. Upon upgrade, the reference count for existing tags has to be populated, which may take a little while on large databases.

Checksum handling

Two improvements have been made. The first has to do with what happens when a client provides a checksum. In this case, dCache now verifies and stores the provided checksum as well as computes and stores the checksum of the type configured by the admin in the pool setup.

The second has to do with interoperability between clients requesting different checksum types (this mainly derives from the Globus MD5 requirement). The FTP door has been modified to compute on the fly a missing checksum required by the client. This way, files written by a protocol requiring ADLER32 can be read by clients requiring MD5, and vice-versa. This fix requires updating only the FTP door.

Finally, an admin command

    get file checksums <pnfsid>

has been added to PnfsManager to display multiple checksums.

RESTful api

With this release, information that has traditionally been made available via the legacy httpd and webadmin pages will also be accessible via REST apis.

A full description of the paths and parameters for each RESTful service, along with example JSON output, will be published separately on the dCache GitHub Wiki.

Included in this release are the apis and supporting services for alarms, billing, cell info, pool and pool group info, active transfers and tape restores/stages.

An illustrative example:

    curl -k -u arossi#admin:xxxxx 'https://fndcatemp1.fnal.gov:3880/api/v1/pools/dmsdca22-7?info=true'

The request is for the basic information concerning a pool; the JSON object returned includes cell information, pool configuration/setup information, pool cost and request statistics, space statistics, sweeper statistics, etc. There are additional parameters for requesting histogram data on pool requests, space usage, and file lifetime on the pool, as well as listings of movers, stores and restores.

Notice the user name in the curl command. Use of these services requires admin privileges. Here, it is presumed that the user has been accorded the admin role; the ‘#’ after the login name indicates a request to express that role for this session. See the section on Roles above for a fuller explanation.

In addition to the new admin services, a small bug has been fixed in the handling of QoS updates on the namespace service, so that the request correctly considers the current locality of the file.

gPlazma

The X.509 plugin has been updated to extract email addresses from the Subject Alternative Name. This extracted information is now part of the principals identifying the user, which is available to all dCache components like webdav, frontend etc.

Frontend and WebDAV

dCache-view, which is part of Frontend services, has the following new features and fixes:

  • display of user profile information
  • added support for user roles
  • drag and drop for moving files and directories
  • user login with open-ID connect
  • upload of files and directories using drag and drop
  • a customised context menu replacing the hover-context
  • users stay in their current path after successful authentication.

Since the gPlazma plugin has been updated to extract the user’s email, Frontend clients can now discover a user’s email addresses, if any are known.

Unnecessary Frontend backward-compatible configuration data, consumed mainly by dCache-view, were dropped.

For WebDAV and Frontend doors non-/ root has been fixed, ensuring that users with non-/ root directory will see the same files and directories under WebDAV/Frontend as with other protocols. Also, Frontend ‘mkdir’ and ‘mv’ operations are updated to honour door and user roots.

Obsolete services

dCache 2.16 introduced the next generation replication service called the resilience manager. At that time, we announced that the replica manager would eventually be removed. We try to keep what we promise, so now the old service is gone. If you haven’t migrated yet, you should do so before upgrading to dCache 3.2.

systemd support on Debian-based systems

Traditionally, we have shipped our own scripts to daemonize dCache. This made it easier to support many different distributions as well as the multi-process architecture of dCache (in other words, it was more fun to implement our own than to study how each distribution did it). For better or worse, the mainstream Linux distributions have all moved to systemd, so it now becomes hard to justify why dCache shouldn’t make use of the functionality offered.

As a first step towards systemd, this release recognizes systemd on Debian based systems. If detected during installation, part of the dCache runtime management scripts are replaced by callouts to systemd. A systemd generator scans the dCache configuration and creates a systemd unit for every dCache domain. systemd fully and directly manages each dCache process. This means:

  • No custom wrapper scripts; there is now only one process per dCache domain.

  • systemd monitors the Java process and restarts it if it quits directly. There is no separate ‘restart file’ to suppress the dCache auto-restart mechanism.

  • No PID files, as systemd tracks the dCache processes directly.

  • systemd captures the stdout/stderr output of the process and directs it to journald. The default journald setup passes the log on to syslog.

  • systemd drops privileges of the Java process during startup. systemd mounts /etc, /usr, /boot, and /home read-only for the dCache process, preventing dCache from writing to any of these directories. Note: This is important to remember if you happen to use /home for pool data or tape integration.

Starting and stopping dCache domains

Whenever the list of dCache domains changes, the dCache units need to be regenerated. dCache will do this automatically whenever its dcache script is invoked as root, but one may also do this manually by running systemctl daemon-reload. This also fixes a long standing issue with dCache loosing track of running domains that are removed from its configuration.

The dcache script has been updated to call out to systemctl to start and stop domains. One can continue to use these commands to manually start and stop domains. Since the systemd support replaces the classic SysV init script, auto-startup during boot has changed: the generated systemd units are not flagged as enabled automatically. Although one can start and stop these domains, they do not start automatically. Use the systemd enable command on every domain that should start automatically. In contrast to before, one can select exactly which domains should start automatically.

Logging

Since dCache by default logs to stdout, and since systemd redirects stdout to journald and thus to syslog, dCache logs now end up in syslog. An rsyslog configuration is included to seperate the dCache log output from other syslog messages. Each domain logs to /var/log/dcache@DOMAIN.log where DOMAIN is the dCache domain. The file cannot be placed in /var/log/dcache due to permission requirements enforced by rsyslog. The default logrotate setup is adjusted to rotate the new files. It is no longer necessary to use the copytruncate option, which makes logrotation more efficient, uses less disk space, and avoids the risk of losing log entries.

Since syslog includes timestamps automatically, the default log format is modified upon upgrade to not include the timestamp in the dCache output. If the log format has been customized, it must be adjusted. Since the log format changes anyway, we also adjusted how the NDC is logged.

If you really liked the old log format and placement, you may reconfigure the logging in /etc/dcache/logback.xml to log directly to /var/log/dcache rather than stdout. If you do, you should ideally use the logback logrotation rather than rely on logrotated.

RedHat

Nobody has bothered to upgrade the RedHat packaging with systemd support yet. Volunteers are welcome.

FTP

The FTP door was updated to allow pipelining of commands. Since the FTP protocol expects some commands to have immediate effect – i.e., before the previous commands have finished – this is not as trivial as it sounds. We believe we have nailed it, both fixing bugs in the existing implementation and avoiding the bugs other services have in their attempt to support pipelining.

The immediate effect is better compatibility with Globus Online.

Space usage

Continuing the WLCG quest to reimplement the features of SRM in other protocols, dCache now exposes space reservation stats through both FTP and WebDAV.

For FTP, the SITE USAGE command is implemented, supplying information from SpaceManager using reservations with a description that matches the TOKEN argument (if a TOKEN was given) or that are bound to the supplied path (if TOKEN was omitted).

For WebDAV, reservations are exposed as RFC 4331 quotas and can be queried as such.

NFS

Updated handling of directory listings. This should avoid situations when the client receives a BAD_COOKIE error caused by server-side cache invalidation. The stage and p2p operations are handled the same way and are more client-friendly. The ‘show transfer’ admin command supports filtering based on client ip, pnfsid and pool name.

Xrootd

The xrootdfs FUSE driver immediately closes a file on creation, then reopens it to write. The behavior of the xrootd door in dCache has been modified to allow new empty files to be overwritten, thus enabling file copies from xrootdfs FUSE into a dCache mounted filesystem.

Pools

Pools can use mongoDB to store metadata. This can be enabled by the

pool.plugins.meta=org.dcache.pool.repository.meta.mongo.MongoDbMetadataRepository

configuration option. Additional properties control mongoDB server location, database name and collecton name:

pool.plugins.meta.mongo.url=mongodb://localhost:27017    
pool.plugins.meta.mongo.db=pdm
pool.plugins.meta.mongo.collection=poolMetadata

A single shared mongoDB instance can be used for all pools.

NOTICE: in production, mongodb must run in a cluster in order to provide high performance and availability.

srmmanager

Most SRM operations are only allowed on local SURLs. Only third-party copying allows non-local SURLs; however, there at least one of source/destination SURL-pairs to be local.

In previous versions of dCache, the properties srmmanager.net.port and srmmanager.net.local-hosts allow the srmmanager to decide which SURLs are local. In many cases, this information is redundant, as srm services already publish this information within dCache. Therefore, with this version of dCache, the srmmanager will consider SURLs local if there is an srm door that advertises it listens on that host and port.

Sites may have a DNS alias or have some proxy service to which SRM clients connect. Under these circumstances, the client will not connect to the FQDN of the machine hosting the srm service, but some other address (that of the DNS alias or the proxy service). To support this, the srm.loginbroker.address and srm.loginbroker.port properties must be configured correctly so that at least one srm service advertises the hostname clients use when connecting to dCache.

Changelog from 3.1.1 to 3.2

636b169abb
webdav: fix regression in OPTIONS response
60ddde1934
dcache (collection service): handle execution exceptions correctly
ad7fdddfc0
libs: update nfs4j to version 0.15.2
129398d24e
Update FileOperationHandler.java
8951fd2b77
pool: fix data integrity regression for 3rd-party GridFTP pull transfers
c205175281
pool: fix regression in GridFTP OPTS CKSM command
d7c5d2d63b
common: fix time computation in TimeseriesHistogramTest
c6bba6d32d
resilience: handle file deletion during scan correctly
7826205a32
pool: ceph: ignore file-not-found on remove
16f408d5a4
dcache-restful-api: return incomplete info instead of throwing NoSuchElementException
17721ed564
resilience: add pool operation logging
2351b6779d
resilience: handle storage unit NoSuchElement failure
fd502cd927
srmclient: parameterise shell path of srmclient utilities
3a4c2ead2d
nfs: shutdown callback ScheduledExecutorService on shutdown
7ee44eea7e
libs: update to nfs4j–0.15.1
53068de346
webdav: adjust header parsing to be case insensitive
1d8239fade
cells,dcap,ftp: Support for accepting connections from an allowed list of subnets and IP addresses
dd671b22b9
resilience: handle all cases where no locations for file may be discovered
04e32327d7
resilience: distinguish correctly between file not in repository and file not found
ffb85de53c
resilience: fix bug in formatting and handling of cache exception types
26d939d611
dcache-restful-api: extract the locations from storage info and add to JSON attributes
c824efb49d
common: fix bug in histogram max index computation
12b9a8ada0
srmclient: remove non-functioning script with BASH dependencies
c972cdcc12
[maven-release-plugin] prepare branch 3.2
366fb071da
dcache-restful-api: add missing aggregated cost data to JSON
d59342ab7e
srm-server: refactoring slf4j logging messages
39f10cda45
srm-common: refactoring slf4j logging messages
345dbfbbd5
nfsv41door: add filter method for show transfers command
db17e99d25
cells: refactoring slf4j logging messages
46b799062e
dcache-chimera: refactoring slf4j logging messages
a7497f3b57
ssh: add username check to pubkey authentication
266caa6228
ftp: update exception logging to include context
728e32a9d8
gplazma2-grid: refactoring slf4j logging messages
2626ed9619
gplazma2: refactoring slf4j logging messages
aaa7f3cc40
dcache-xrootd: refactoring slf4j logging messages
a705d04296
dcache-webdav: refactoring slf4j logging messages
7bca95e137
dcache-webadmin: refactoring slf4j logging messages
6fe5ff78b5
corrected requested typo
4972298c18
dcache-spacemanager: refactoring slf4j logging messages
21a077d341
dcache-info: refactoring slf4j logging messages
a337212e36
dcache-ftp: refactoring slf4j logging messages
c68428c074
dcache-dcap: refactoring slf4j logging messages
e32bf805f4
dcache: update Subnet utility class to have a isValid method
7f5fe17d6c
srmmanager: reduce network configuration
e0867e15f0
pool: fix minor errors in PoolInfoRequestHandler
7fb9b61d44
fix RPM package building and dangling reference in text
b2cbf3016b
core: Add Remote-Host-Restriction capability to Ssh2Admin PublicKeyAu… (#3431)
a768abfae7
dcache-core: refactoring slf4j logging messages
17154424c2
dcache: added a decorator for RepositoryChannels to get IO statistics (#3430)
9eb63b13b6
acl: refactoring slf4j logging messages
2761aca19d
skel: remove legacy Berkeley DB jar and corresponding preupgrade-script
536dbdf652
inserted again requested changes
154709991e
Motivation: Prior to Java 1.5 enums did not exist, thus integer constants were used. These constants can now be replaced by enums.
38c1144452
ssh: add logging to domain access log file (#3427)
5b1f65b0ce
dcache: introduce history service with pool timeseries component
0f2188aab9
dcache-restful-api: pool info service implementation
fa3d4eb0f7
dcache-restful-api: move admin collector service abstractions to dcache module
213893df48
dcache-common: refactoring slf4j logging messages and logger variable name
311f09acfa
chimera: fix broken commit e064f5577b
e064f5577b
libs: switch to nfs4j–0.15
e88c1e31c0
acl: refactoring slf4j logging messages Motivation: with normal string concatenations in log-messages strings are always build, regardless if log-level is activated or not. with parameterized log-messages the strings only become build, when the log-level is activated;
625ff42cfb
acl-vehicles: refactoring slf4j logging messages
a10786c84a
chimerashell: use chimera.db.* options as defaults
cf9154131b
cells: better handling of rogue domains with badly formatted dCache versions
79161ee475
configuration: update zookeeper configuration with hints
a1a2e50d4b
pool: suppress unecessary ‘jtm set timeout’ in setup
d5adeb363e
pool: fix loading ‘setup’ that requires queues created by ‘pool.queues’
38d4249f98
pool: consolidate error-handling in pool IoQueueManager admin commands
13074ca921
fix minor tyo
df1bc6d682
various: miscellaneous minor adjustments from restful commits
858fbfa6c1
webdav: avoid stack-trace on bad user requests
f106f7a8df
systemd: start service after ZooKeeper
3f30893ac2
Martin (#3403)
e7946a7130
Vuong-Test (#3401)
7e29c99ec6
Signed-off-by: local local@lp-hrz-d209-linux.wh.f4.htw-berlin.de
506e612582
dcache : added reedme.md
27d0d5a837
gPlazma2-voms: Add README for Module Info
bd8265e0bf
webdav: add README file to gplazma-nis module
d5897c1cae
srm-common: add README file
feb41dec5c
Motivation: Test-Readme for gplazma2-roles
c83e3dd7d4
cells: readme file for testing
6d1833b204
Signed-off-by: Lotta Rüger l.r.@Lottas-MacBook-Pro.local
df1ac51260
common: HTW-Berlin Big Data Test Commit
2a8da23880
alarms: guard against NPEs on LogEntry getters
af3d12aa4d
admin: Fix Inconsistent ACL enforcement, RT 9207
28e093eec0
nfs: add a possibility to specify offered layout types
68f49618a9
system-test: add list of allowed client origins
770082b527
systemtest: update systemtest to point to a reasonable WebDAV door
6ee63306d6
pools: add support for requesting live data for histories
f75794c68c
common: fixes potential NPEs in histogram metadata
56bc72919e
webdav: fix more regressions with CredentialSource.NONE
05026a1511
pool: fix regression in HTTP third-party transfer with redirection
355ff88135
systemtest: fix populate script for when systemtest already exists
13bb5878c5
webdav: fix error recovery for macaroon users without DELETE
aabac213aa
macaroons: add implicit authorisation of READ_METADATA
fbae70d684
dcache-restful-api: simplify alarms api and service
7e5e81e173
webdav: fix regression in third-party copy with no delegation
52a409c136
dcache: release dcache-view version 1.3.1
b70b0d946a
dcache (pools): Add messaging support for frontend/restful pool info service
891060e863
dcache-restful-api: removing disk-based caching from CellInfoService
27841575f4
configuration: update description for replicable
0b21d9d5e4
dcache-restful-api: fix handling of no route to cell in alarms collector
c5199298a6
debian: Adjust how NDC gets logged
c668b85cbc
gplazma-oidc: improve code-style for oidc plugin in accordance with the dCache code style guideline
96e14cece5
srm/srmmanager: fix srmPing confusion
56fd242379
dcache-restful-api: add api and implementation for alarms service
bdd188b536
dcache-restful-api: fix misnamed restores command
dfe1b89c38
dcache-restful-api: fix restore service initialization
306c0d06ab
dcache-restful-api: add service implementation for collecting staging/restore info
af7136e45d
ftp: convert timestamps to GMT (to follow RFC 3659)
7d5e0fef1e
Revert “pool: handle initial space allocation for existing files”
22365dcfb6
pool: fix regression in accounting during file upload
feddfcc542
pool: use Throwables.getRootCause(e) inspecting RuntimeException
691ad40190
packaging: tighten permissions in var directories
f23954a64a
billing: update documentation to describe CellAddress
c5db7fa1cf
srm/srmmanager: update documentation about root path
4ca756f32a
srm,srmmanager: add configuration property to allow easy modification of srm root
ade17bb00e
pool: handle initial space allocation for existing files
f7dae69511
gplazma: extract email from x.509 certificates
e5e80f491e
frontend: expose users email address
85d4801aae
logback: make socket appender construction depend on log level
4139d89bcb
frontend: fix NPE when billing is disabled.
3e596ed3ba
resilience: remove reference to pnfsmanager property
3b7013bfd6
frontend: fix regression in configuration properties
d60f09b0c9
config: improved description of port numbers
6ea03d7155
config: add obsolete|forbidden annotation for dropped properties
79fde8079e
resilience: remove stray conflict marker
bce7615108
resilience: make namespace provider properties immutable
3b372e2b51
dcache: release dcache-view 1.3.0 for dcache current master
ba946f112e
frontend: remove backwards compatible config data
2a53e1f2e3
config: add obsolete|forbidden annotation for dropped properties
745149fb99
config: add obsolete|forbidden annotation for dropped properties
728db53ef9
pnfsmanager: remove obsolete comments from properties file
c9f38a3b40
clarified documentation of gplazma.authz.upload-directory
6b64e86ab0
improved description of upload-directories
7a9dd29913
added hint that pnfsmanagers must use the same DB
670584f7a0
fixed several typos in the documentation
400b4fc8be
use correct terminology
b0abb286b6
cells: fix TLS support to work with embedded zookeeper
47afd23844
systemtest: fix regression introduced with systemd integration
5b081219eb
srmclient: give version of srmclient
e86a92a893
packages: Fix build of RPM
43253ec27e
debian: Add rsyslog config for dCache
6374ddb965
debian: Systemd integration
7b7369a60a
frontend: avoid sending messages before cell is registered
c977392bb0
cells: add tls based encrypted channels for cell communication
4d27373d98
zookeeper: work-around race-condition in zookeeper server shutdown
387fbbcef4
common,system-test: fix minor annoyances with roles in system-test
dd5fa1ccc3
webadmin: use new role-based login and support active/deactive roles
e29757959f
authentication: add support for macaroons
a18ba8f063
libs: update jetty to latest 9.4.6.v20170531 release
3392b5dc0a
nfs: merge p2p and stage handling
4d1e5e66ae
dcache-restful-api: restore creationTime on JSONAttributes
732e938ca4
packages: missing rpm server spec line for cell-info dir
815cf82abe
ssh: fix handling of ssh idle timeout
96188c7a14
dcache-restful-api: add cell service API and implementation
11392b9ae8
dcache-restful-api: add RESTful billing service and support
50f32f5768
dcache (alarms): add support for RESTful frontend messages to alarms service
ced8d9fe75
authz: return a list of allowed but unasserted roles
3ada58ab04
common: add javadoc to AccessLatency and RetentionPolicy
4ccd181edc
common: make AccessLatency and RetentionPolicy more enum like
4f8b39f7e4
system-test: add series of functional test for frontend service
8d9c3bf7e3
authz: add initial support for Roles
3cbc1f11da
cells: improve System’s ps output
f31bb95a16
chimera: remove redundant code
1fc6082dc6
security: drop dead code
789dbcaabc
webdav: add support for RFC 4331 for reporting space usage
32db31a0bd
chimera: keep track of tags usage
7fc9aaf7d0
common: repair some minor issues with HistogramModel
5e286d5382
common: use null/Optional instead of Double.MAX_VALUE and Double.MIN_VALUE for histogram stats
4761c1c73a
frontend: refactor static (configuration) data
f264c0fb96
spacemanager: dont try to release expired spaces
c94fc2e3c6
srmmanager: use path to support srmSetPermission operations
4694030f9d
common: use round instead of floor to bin counting histogram values
893846e231
dcache-restful-api: fix pnsfHandler reference in IdResource
9b9879837e
Revert “admin: Fix Inconsistent ACL enforcement, RT 9207”
4032dfbf1a
frontend: fix Restriction usage
5b96a948ce
pool: do not throw InterruptedException on Repository#openEntry
1668b1a66d
dcache-restful-api: add API to get file attributes from pnfsid
8f8c79b3f9
dcache-restful-api: add pnfsid and nlink to all attribute requests
ad720c2725
admin: Fix Inconsistent ACL enforcement, RT 9207
c42a9dd8ef
Fix typo in broken commit
30c3ec4e28
poolmanager: fix unit-test to avoid race-condition
6f8da800c6
zookeeper: work-around SessionTracker racy initialisation on startup
ce61e28cf8
srmmanager: fix NPE when querying spaces
80c082c258
httpd: Fixed table headers in usageInfo
45a40c3e20
ftp: add support for SITE USAGE command
b864b313ff
pool: use StndardOpenOption.CREATE instead of OpenFlags.CREATEFILE
273c83e048
pool: use Set<OpenOption> instead of Set<Repository.OpenFlags>
dca973f5b1
pool: update Repository.OpenFlags to implement OpenOption
b40a81e622
pool: use OpenOption instead of StandardOpenOption
554b91b4b8
zookeeper: work-around racy startup
4003ec7b35
dache-restful-api: add log.debug to trace Qos transitions
a7b3d077ad
zookeeper: silence ZK server errors
7e210d8d98
dependencies: remove log4j jar
0e07ff7a77
replica: remove final references to replica manager
268af787ea
pool: refactor pool to use StandardOpenOption instead of IoMode
28c15abb5a
ssh: do not cast timeout to int
c727a0628f
frontend: fix “Attribute is not defined: SIZE” bug
315c25d312
pool: fix error message on timeout
130e160f5c
pool: add support for mongodb as a backend for medatada
a912e8393b
info: avoid sending messages too early.
4ac977ab43
frontend,webdav: add supress-wwwauthenticate to allow headers
f723e96b0c
dcache: remove old Replica Manager
840ba3fb35
authentication: suppress WWW-Authenticate if requested
6c37ffbc51
pool: log why a transfer was forcefully aborted
5a369e8455
webdav: improve logging on transfer failures
cddc697a38
webdav: make Milton work-around more robust
773bc0df99
script-nearline-spi: fix space leak when polling script is used
0bc04f6b71
http: fix non-/ root for WebDAV and frontend doors
554fd2a969
authentication: support embedding a bearer token in HTTP URLs
aa9cdbca04
webdav: Fix restriction check when downloading a file
13b2d7716c
common: factor out PathMapper as common feature
d7db259922
httpd: Fix incomplete restore list in HA setup
f2392e4cdd
httpd, admin: Fix some hard-coded cell names
cec2f1fd4b
frontend: expose open-id connect to dcache-view
83ba36f475
rest-api: include username to the user attributes
a35a988042
systemtest: update Globus clients to use generated trust store
21c0b25341
systemtest: add transfer tests for UberFTP
437f24c85c
frontend: fix problem introduce by jetty update
db29ebc60c
Add cascadeConstraints=“true” to liquibase dropTable action on pinsv3 table
3961e557c6
nfs: bind vfs cache invalidation with file’s layout
04f929cf1c
ceph: map RadosException to corresponding IOException
a10376ae03
dache-restful-api: add transition state from DISK to TAPE to QoS description
a899b42c34
dcache-restful-api:bug fix: current locality of the file should be considered
f6175a325d
dcache: fix NPEs in TransferInfo time string methods
0e573b9685
system-test: update test script so curl uses system-test’s trust-store
f77ebaa6c6
nfs: show transfer status when displayed
a19f27589e
system-test: update credential-generating script
ce1deb63a9
system-test: update disposable-CA generated credentials
0ebcceaff4
nfs: fix loosing movers due to short timeout
03436a0da5
pinmanager: fix query for getting the number of pins for a file.
976b783d46
Fix class path generation
ff52847c4f
Add workaround for Liquibase bug CORE–3001
9ff9eb91d0
Update various dependencies
3cabb610cb
Upgrade base libraries to Java 8
caccb7719d
dcache: add timeElapsedSinceWaiting formatted string to TransferInfo
3d67d4c2ed
xrootd: allow xrootd to overwrite new empty files. Motivation: Xrootdfs FUSE driver immediately closes a file on creation, then reopens it to write. The file cannot be opened as it already exists. This patch allows the dcache xrootd door to tolerate this behavior.
5c17c8f6f0
ftp: fix (unreleased) regressions in certain FTP commands
73f60deb2a
ceph: log repository IO error
6c9e3a1a2a
writing to existing file is kXR_NotAuthorized, not kXR_Unsupported
faf5e142df
srm: remove file-level timeout
90f38a4cec
ftp: ensure server replies even if there are bugs in dCache
b97bd275a3
Update code to use ByteUnit
19ec5b3078
chimera: do not update inode generation on atime only update
c960d46a47
ftp: refactor error handling
2af2ae8c01
nfs: use v4.1 for flex file layout
7dbbe2b0a2
nfs: do not add Origin to read-only subject
23ab37a11e
common: add general-purpose histogram and histogram models, with unit tests
502e09e561
dcache-restful-api: migrating file to a an appropriate pool
68fc74c810
common: move billing TimeFrame class to the common histograms package
7b0567fb46
parent/dcache: add mail jar to deployment
23edcbd45e
movers: fix NPE caused on upload of a zero length file
0b5141e849
dcache-restful-api: add unit tests for transfer service
d7440e22aa
dcache-restful-api: add transfer service implementation
26104eac59
dcache-restful-api: add transfer collection utilities
317030ed1f
dcache-restful-api: Add RESTful resource for active transfers
03d529cd68
dcache-restful-api: Add common abstractions for services supporting RESTful admin resources
a06c2ec57e
restful-api: add file mime-type to file attribute
4da5635cac
dcache: added command to display multiple checksums for a file
ebceda3297
dcache,movers: Storing multiple checksums for a file when the client provided checksum is of different type than that of the pool setup.
dd1b563498
pool: handle CEPH exceptions
da55011f64
pool: fix double close on p2p
39fbb93c9e
srm-client: improve handling of checksum options
ee81623519
nfs: recall file layout on pool disable
de31f61417
libs: update to bug fix release nfs4j–0.14.2
086c1e0136
common, ftp-client, srm-client: remove 1.7 source and target restrictions on common, ftp and srm client modules
6c75e48889
chimera: fixed database query for storing multiple checksums for a file.
dd2b4bf1ea
srmclient: fix handling of checksum options
75358b3283
ftp: fix broken commit cfa765bb
7e2fa7bb9c
ftp: add support for dynamic checksum calculation
cfa765bb17
ftp: add support for command pipelining
3d21e6e1b6
ftp: prevent execution of most commands when unwrapped
f49a1455c2
srmclient: fix compatibility with castor
60b4224372
chimera : handle empty paths elements path2inumber stored procedure
641dc750ac
pom: Fix dCache version ready for 3.2.0
62be2a65c7
[maven-release-plugin] prepare branch 3.1
5a333bcd3e
[maven-release-plugin] prepare for next development iteration
3d91506a31
xrootd : use lower case for checksum algorithm names when replying to checksum queries.