SciTags Flow Marking (Firefly) in dCache
This document explains how SciTags (Firefly flow markers) are implemented in dCache. It is written for two audiences:
- Site operators and advanced users who want to understand configuration and behavior.
- Developers who want to debug, validate, or extend the implementation.
Scope and terminology
Within dCache, SciTags classification data is carried as a transfer tag string (transferTag) in door-to-pool protocol info objects. Pool-side code then derives experiment-id and activity-id and emits a Firefly marker over UDP.
In this document:
- “SciTag value” means the integer encoded by clients.
- “transferTag” means the dCache protocol field that carries that value as text.
- “Firefly marker” means the JSON payload emitted via syslog-formatted UDP packet.
Fast operator summary
The behavior is:
- The door extracts SciTags data from protocol-specific request metadata.
- For HTTP-TPC, the active pool forwards the tag to the remote (passive) party, so both endpoints classify the transfer under the same SciTag.
- The pool validates and decodes the value (or falls back to VO-to-experiment mapping via FQAN).
- dCache sends Firefly markers to:
- the data-flow destination host on UDP 10514 (default), and
- optionally, an additional configured collector.
Supported marker protocols in current pool logic:
xrootdhttphttpsremotehttpdatatransferremotehttpsdatatransfer
SciTag encoding follows:
$$ scitag = (experiment_id << 6) ;|; activity_id $$
$$ experiment_id = scitag >> 6 $$
$$ activity_id = scitag & 0x3F $$
Accepted SciTag value range in current implementation is 64..65535 (inclusive).
End-user and operator configuration
Primary settings are in pool configuration defaults:
pool.enable.fireflypool.firefly.destinationpool.firefly.excludespool.firefly.vo-mappingpool.firefly.storage-statistics
Reference defaults: skel/share/defaults/pool.properties.
Typical site configuration in /etc/dcache/dcache.conf:
pool.enable.firefly=true
# Optional additional collector (host or host:port).
pool.firefly.destination=us.scitags.org:10333
# Optional list of subnets to exclude from marker emission when BOTH endpoints are local.
pool.firefly.excludes=10.0.0.0/8,192.168.0.0/16
# Fallback mapping when transferTag is missing (keys are lower-cased VO names).
pool.firefly.vo-mapping=atlas:2,cms:3,lhcb:4,alice:5
# Include storage statistics in end markers.
pool.firefly.storage-statistics=true
Notes:
- If
pool.firefly.destinationis unset, markers are sent only to the peer host at UDP 10514. - If set, dCache sends an additional copy to that collector (unless it equals the primary destination).
- Exclusion requires both source and destination addresses to match configured local subnets.
Request-side extraction by protocol
WebDAV / HTTP door path
For regular WebDAV requests, SciTag extraction is in modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java.
Order of precedence is:
SciTagheader (case-insensitive lookup)TransferHeaderSciTagheader (case-insensitive lookup)scitag.flowquery parameter
Blank values are ignored and treated as not present.
The resulting string is copied into HttpProtocolInfo.setTransferTag(...) and propagated to the pool.
WebDAV Third-Party Copy (TPC)
For COPY/TPC, capture and forwarding are in modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/CopyFilter.java and modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java.
Behavior:
CopyFiltercapturesSciTagandTransferHeaderSciTagrequest headers.- Missing or blank header values are normalized to sentinel
-for request attributes. - Pool transferTag precedence is:
SciTagTransferHeaderSciTag- empty string
RemoteTransferHandlercopies this intoRemoteHttpDataTransferProtocolInfo/RemoteHttpsDataTransferProtocolInfoviasetTransferTag(...).
Propagation to the remote (passive) party
The active pool sends the transfer tag to the remote endpoint of an HTTP-TPC transfer, so that the passive side can classify the flow and record the same transferTag in its billing records (for example, the Http/Https mover records in the Kafka stream).
Implementation: modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java. RemoteHttpDataTransferProtocol#addHeadersToRequest(...) adds an outbound SciTag request header carrying RemoteHttpDataTransferProtocolInfo.getTransferTag() to every request the pool issues to the remote party (GET, PUT, HEAD and DELETE, including retries and redirects).
Behavior:
- The header is only added when a transfer tag is set (empty or missing tags are not sent).
- If the client already supplied an explicit
SciTagheader viaTransferHeaderSciTag, the original header is forwarded unchanged and no extra tag is added. - Because all requests for a transfer are built through the same method, the remote party sees the same tag for the whole transfer.
This is an automatic, per-transfer behavior with no site configuration. It has no effect on transfers that do not carry a tag.
XRootD door path
XRootD extraction is in modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java.
Behavior:
- transferTag is read from opaque parameter
scitag.flow. - Missing parameter yields empty string.
- Value is copied into
XrootdProtocolInfo.setTransferTag(...).
Door call-site: XrootdDoor.createTransfer(...) constructs XrootdTransfer, and XrootdTransfer handles SciTag extraction and propagation into protocol info.
Important validation boundary
Door code primarily captures and propagates tag values; SciTag numeric validation is pool-side.
Client examples
Compute a SciTag value
Example for experiment-id=2 and activity-id=1:
$$ scitag = (2 << 6) | 1 = 129 $$
WebDAV / HTTP request
curl -H "SciTag: 129" "https://dcache.example.org/pnfs/example.org/data/file.dat"
WebDAV query fallback
curl "https://dcache.example.org/pnfs/example.org/data/file.dat?scitag.flow=129"
XRootD request
xrdcp "root://dcache.example.org//pnfs/example.org/data/file.dat?scitag.flow=129" /tmp/file.dat
Pool-side lifecycle and marker emission
Core implementation:
modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java
The pool emits markers on transfer start/end with guard checks:
- Firefly enabled.
- Not excluded by local subnet rules.
- Protocol supported.
- Experiment ID resolvable (from transferTag or VO fallback).
If any check fails, marker emission is skipped.
Start marker invocation path
Current invocation is from:
modules/dcache/src/main/java/org/dcache/pool/movers/NettyTransferService.java
NettyTransferService.executeMover(...) calls transferLifeCycle.onStart(...) with:
- source = remote endpoint from
IpProtocolInfo.getSocketAddress() - destination = local pool endpoint chosen for mover
End marker invocation path
Current invocation is from:
modules/dcache/src/main/java/org/dcache/pool/classic/DefaultPostTransferService.java
DefaultPostTransferService.sendFinished(...):
- uses
((IpProtocolInfo) mover.getProtocolInfo()).getSocketAddress()as source, - uses
mover.getLocalEndpoint()as destination when available, - otherwise tries
deriveLocalEndpoint(remoteEndpoint)viaNetworkUtils.getLocalAddress(...), - calls
transferLifeCycle.onEnd(...)when either direct or derived local endpoint is available, - logs structured skip reasons when end-marker invocation cannot proceed.
Developer note on service paths
The only direct call to TransferLifeCycle.onStart(...) is currently in NettyTransferService. TransferLifeCycle.onEnd(...) is invoked from DefaultPostTransferService when a local endpoint is available from the mover or can be derived from the remote endpoint.
Classification logic
Classification is implemented in TransferLifeCycle as follows.
transferTag path (preferred)
- Parse
protocolInfo.getTransferTag()as integer. - Reject values outside
64..65535. - Decode:
- experiment-id =
transferTag >> 6 - activity-id =
transferTag & 0x3F
FQAN fallback path
If transferTag is empty/missing:
- read primary FQAN from subject,
- get group path,
- lowercase,
- split on first
/boundary, - map first segment (VO name) via
pool.firefly.vo-mapping.
Fallback activity-id defaults to 1.
Exclusion logic
pool.firefly.excludes builds an OR-combined subnet predicate. Transfer is excluded only when both source and destination addresses match excluded subnets.
Firefly payload details
Payload builder:
modules/dcache/src/main/java/org/dcache/net/FlowMarker.java
Marker wire format:
- syslog header prefix:
<134>1 ... dCache - firefly-json - - followed by JSON body
Main JSON sections:
versionflow-lifecyclewithstate,start-time, optionalend-time,current-timeflow-idwith AFI, IPs, ports, and protocolcontextwithexperiment-id,activity-id, and application/protocol nameusage(received,sent)- optional
storage-statswhenpool.firefly.storage-statistics=true
For end markers, start-time is reconstructed from mover submit time and end-time is derived as start-time + max(connectionTime, 0).
Test schema file:
modules/dcache/src/test/resources/org/dcache/net/firefly.schema.json
Logging and debugging
SciTags debug logging uses dedicated logger org.dcache.scitags with structured key/value messages. This includes request capture at doors, start-abort diagnostics, end-path skip/invoke decisions, and marker send/skip events.
Relevant class loggers are:
org.dcache.scitags(DEBUG): structured SciTags lifecycle/request/marker diagnostics.org.dcache.pool.movers.TransferLifeCycle(WARN): invalid transferTag values, invalid VO mapping entries, and marker send failures.
Enabling debug output for SciTags-related classes
Because dCache uses CellThresholdFilter, adding a logger alone is insufficient. You typically need both:
- logger entry
- matching threshold entry inside
CellThresholdFilter
Template example for /etc/dcache/logback.xml:
<logger name="org.dcache.scitags" level="DEBUG"/>
<turboFilter class="dmg.util.logback.CellThresholdFilter">
...
<threshold>
<logger>org.dcache.scitags</logger>
<level>debug</level>
</threshold>
...
</turboFilter>
Reference template with CellThresholdFilter: skel/etc/logback.xml.
Useful structured message patterns
Look for:
scitags event=request ... tagSource=... transferTag=...scitags lifecycle=start abort reason=interrupted ...scitags lifecycle=start abort reason=execution-failed ...scitags lifecycle=end skip reason=...scitags lifecycle=end invoke ...scitags event=marker-skip reason=...scitags event=marker state=start ...scitags event=marker state=end ... bytesRead=... bytesWritten=...
Marker skip conditions now carry explicit reason= codes in structured debug messages.
Packet-level verification
On the pool host, verify UDP emission (example):
tcpdump -nn -i any udp port 10514
If pool.firefly.destination is configured, also verify traffic to the collector host/port.
Developer extension guide
Add a new protocol to SciTags
Minimum steps:
- Door-side: extract and set
transferTagin that protocol’sProtocolInfo. - Pool-side: ensure
TransferLifeCycle.needMarker(...)recognizes the protocol name. - Lifecycle hooks: confirm start and end invocation paths are wired for that mover service.
- Tests: add/adjust unit and integration tests.
Change SciTag validation or encoding
Edit in TransferLifeCycle:
MIN_VALID_TRANSFER_TAGMAX_VALID_TRANSFER_TAG- bit decode constants (
EXPERIMENT_ID_BIT_SHIFT,ACTIVITY_ID_MASK)
Then update/extend tests in:
modules/dcache/src/test/java/org/dcache/pool/movers/TransferLifeCycleTest.java
Change header precedence or WebDAV behavior
Edit:
DcacheResourceFactory.HttpTransfer.readTransferTag(...)CopyFilter.transferTagForPool(...)
And update:
- tests under
modules/dcache-webdav/src/test/java/org/dcache/webdav/ - tests under
modules/dcache-webdav/src/test/java/org/dcache/webdav/transfer/
Extend payload content
Edit:
FlowMarker.FlowMarkerBuilder
Then update schema and tests accordingly.
Existing tests relevant to SciTags
TransferLifeCycleTest- accepts minimum valid tag (
64) - rejects below-range values (
63) - verifies slash-prefixed FQAN mapping
- verifies exclusion behavior
WebDAV unit coverage includes case-insensitive SciTag header lookup in:
modules/dcache-webdav/src/test/java/org/dcache/webdav/DcacheResourceFactoryTest.java
There is no dedicated unit test that exercises full HttpTransfer.readTransferTag(...) precedence end-to-end.
Source map (quick navigation)
- Pool marker logic:
modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java - Marker start hook:
modules/dcache/src/main/java/org/dcache/pool/movers/NettyTransferService.java - Marker end hook:
modules/dcache/src/main/java/org/dcache/pool/classic/DefaultPostTransferService.java - HTTP/WebDAV extraction:
modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java - WebDAV TPC forwarding:
modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/CopyFilter.java - Remote HTTP TPC propagation:
modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java - Remote HTTP TPC outbound SciTag header:
modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java - XRootD extraction:
modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java - XRootD transfer creation call-site:
modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdDoor.java - Flow marker payload builder:
modules/dcache/src/main/java/org/dcache/net/FlowMarker.java - Pool defaults:
skel/share/defaults/pool.properties - Lifecycle bean wiring:
modules/dcache/src/main/resources/org/dcache/pool/classic/pool.xml - Logback template with threshold filter:
skel/etc/logback.xml