dCache as NFSv4.1 Server
This chapter explains how to configure dCache in order to access it via the NFSv4.1 protocol, allowing clients to mount dCache and perform POSIX IO using standard NFSv4.1 clients.
Important
The
pNFSmentioned in this chapter is the protocolNFSv4.1/pNFSand not the namespace pnfs.
- Setting up
- Pool IP addresses for pNFS redirection
- Concurrent write access
- Reporting capacity to clients
- Configuring NFSv4.1 door with GSS-API support
- Configuring NFSv4.1 with RPC-over-TLS
- Configuring principal-id mapping for NFS access
- Managing group ids
- Remote Quota Management
- Accessing directory tags
Setting up
To allow file transfers in and out of dCache using NFSv4.1/pNFS, a new NFSv4.1 door must be started. This door acts then as the mount point for NFS clients.
To enable the NFSv4.1 door, you have to change the layout file corresponding to your dCache-instance. Enable the NFS within the domain that you want to run it by adding the following line
[<domainName>/nfs]
nfs.version = 4.1
Example: You can just add the following lines to the layout file:
[nfs-${host.name}Domain]
[nfs-${host.name}Domain/nfs]
nfs.version = 4.1
Exporting file system
In addition to run an NFSv4.1 door you need to add exports to the exports file. The location of exports file is controlled by nfs.export.file property and defaults to /etc/exports.
After reading exports file dCache will read the content of the directory with additional export tables. The location of directory defined by nfs.export.dir property and default to /etc/exports.d. Only files ending with .exports are considered. Files starting with a dot are ignored. The format of the export tables is similar to the one which is provided by Linux:
#
<path> [host [(options)]]
Where
-
ro matching clients can access this export only in read-only mode. This is the default value.
-
rw matching clients can access this export only in read-write mode
-
noacl dCache ACLs will be ignored; only posix access permissions will be considered. This is the default value.
-
acl dCache ACLs will be respected; if present, they override posix permissions. To view the ACLs at the NFS client side, use
nfs4_getfaclwhich is in EL7 packagenfs4-acl-tools. -
sec=sys matching clients must access NFS using RPCSEC_SYS authentication, where client’s local uid and gid is used. This is the default value.
-
sec=krb5 matching clients must access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is NONE, e.g., the data is neither encrypted nor signed when sent over the network. Nevertheless the RPC packets header still protected by checksum.
-
sec=krb5i matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is INTEGRITY. The RPC requests and response are protected by checksum.
-
sec=krb5p matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is PRIVACY. The RPC requests and response are protected by encryption.
-
all_squash Map all uis and gids to the anonymous user.
-
no_root_squash Do not map requests from the root to anonymous user. This is useful for clients with administrative rights.
-
all_root the requests from matching clients will be handles as from user root. Additionally, when a new file or directory is created, then parent directory’s ownership is inherited.
-
anonuid and anongid specify which uid and gid should be used for anonymous user.
-
dcap Expose the fact, that nfs server is a dCache instance through special dot file. This option is useful when dcap client are used over mounted file system and preferred IO protocol is DCAP. This is the default value.
-
no_dcap The opposite of dcap option. Hide the fact, that nfs server is a dCache instance. This option is useful when dcap client are used over mounted file system but nfs protocol still have to be used for IO.
-
pnfs Enable NFSv4.1/pNFS to redirect pnfs capable clients to the dCache pools. This is the default value.
-
no_pnfs and nopnfs Opposite of pnfs option. Enforce clients to send all IO requests to NFS door event independent from client’s pNFS capabilities.
-
secure Specifying whether clients are required to use a privileged port (< 1024). This option typically is used to disallow user-space NFS clients, as they might spoof request credentials.
-
insecure The opposite of secure option. The use of privileged port by a client is not required. This is the default value.
-
lt= A colon separated ordered list of pnfs layouts that offered to the client. The valid values are:
-
flex_files
- nfsv4_1_files
For modern clients (linux kernel >= 4.10 and RHEL7.4) flex_files is recommended. Default is nfsv4_1_files.
For example:
/pnfs/dcache.org/data *.dcache.org (rw,sec=krb5i)
Notice, that security flavour used at mount time will be used for client - pool communication as well.
Multiple specifications can be declared like this:
/pnfs/dcache.org/data *.dcache.org(rw) externalhost.example.org(ro)
In this example, hosts in the dcache.org may read and write, while host externalhost.example.org may only read.
If there are multiple path specifications, the shortest matching path wins. If there are multiple host/subnet specifications, the most precise specification wins.
Pool IP addresses for pNFS redirection
With pNFS the NFS door redirects clients to the pool that holds the data, so each pool must know the IP addresses it advertises to clients. By default a pool advertises the addresses of all of the host’s non-loopback network interfaces that are up, and rediscovers them while running.
The advertised addresses can be restricted with the pool.mover.nfs.multipath property, which takes a comma separated list of IP addresses:
pool.mover.nfs.multipath=192.168.1.1,fe80::2e2e:50de:7aa5:395
When pool.mover.nfs.multipath is empty, the pool advertises all local addresses, unless the org.dcache.net.localaddresses Java property is set.
If a pool cannot identify any IP addresses — for example because pool.mover.nfs.multipath lists no valid addresses, or because no non-loopback network interface is up — then the pool fails to start instead of running without the addresses needed for pNFS redirection. In that case check that the host’s network interfaces are up and correctly configured, and that pool.mover.nfs.multipath (if set) contains valid addresses.
Concurrent write access
A file that is currently open for writing is exclusively locked: while such an open file exists, the NFS door denies any other attempt to open the same file, regardless of whether that attempt requests read or write access. Any such attempt fails with Permission denied (EACCES).
When the writing client closes the file, the lock is released and further opens succeed. This prevents two concurrent writers from creating competing replicas of the same file (with the same pnfsid but different contents) and reflects the fact that dCache treats file content as write-once.
Reporting capacity to clients
Clients that mount dCache with NFSv4.1 use the file system statistics (the values shown by df) to see how much space is available. By default the NFS door reports a fixed size of 1 EiB with all of it free, so df is of little use for monitoring or for clients that decide whether they can upload based on the reported free space.
The nfs.capacity.source property controls the capacity the door reports. It defaults to the value of dcache.capacity.source, which is namespace unless changed:
nfs.capacity.source=namespace
The supported values are:
-
namespace: a fixed size of 1 EiB, all of it free. This is the default and keeps the previous behaviour. The number of files and (with this source) the used space are taken from the namespace.
-
pools: the total and free space of the online pools, counting removable (cached) replicas as free, just as the frontend does for storage resource reporting.
Because NFS does not tell the door which file a statistics request is for, the NFS door always reports the space of all online pools. It cannot narrow the reported pools to those that a file in a particular directory would be written to, so the reported capacity does not reflect the routing or space reservation configuration for a specific path.
- reservation: the size and available space of a space reservation, as the FTP door reports for
SITE USAGE.
As the NFS door does not know the queried path, it cannot read the WriteToken tag itself. With this source you must set the space token of the reservation to report:
nfs.capacity.source=reservation
nfs.capacity.reservation=10000
The door fails to start if reservation is selected without nfs.capacity.reservation. The value must be the numeric id of a space reservation.
For pools and reservation, the door keeps the file counts from the namespace and falls back to the namespace statistics if the capacity cannot be determined (for example, while the spacemanager or poolmanager is unavailable). A failure is logged once and reported again once the capacity can be determined.
Configuring NFSv4.1 door with GSS-API support
Adding sec=krb5 into /etc/exports is not sufficient to get kerberos authentication to work.
All clients, pool nodes and node running DOOR-NFS4 must have a valid kerberos configuration. Each clients, pool node and node running DOOR-NFS4 must have a /etc/krb5.keytab with nfs service principal:
nfs/host.domain@<YOUR.REALM>
The /etc/dcache/dcache.conf on pool nodes and node running NFSv4.1 door must enable kerberos and RPCSEC_GSS:
nfs.rpcsec_gss=true
dcache.authn.kerberos.realm=<YOUR.REALM>
dcache.authn.jaas.config=/etc/dcache/gss.conf
dcache.authn.kerberos.key-distribution-center-list=your.kdc.server
The /etc/dcache/gss.conf on pool nodes and node running NFSv4.1 door must configure Java’s security module:
com.sun.security.jgss.accept {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useKeyTab=true
keyTab="${/}etc${/}krb5.keytab"
debug=false
storeKey=true
principal="nfs/host.domain@<YOUR.REALM>";
};
Now your NFS client can securely access dCache.
Configuring NFSv4.1 with RPC-over-TLS
dCache NFS door supports RPC-over-TLS protocol extension for data-in-transit protection. To work with pNFS, host certificates must be installed on NFS doors and pools. As pNFS uses multipath addresses for client redirection, the host certificates must include the IPv4/IPv6 records in Subject Alternative Name.
To enable, the RPC-over-TLS capability on pools and nfs doors the following properties must be enabled:
pool.mover.nfs.enable.rpc-over-tls=true
nfs.enable.rpc-over-tls=true
or
dcache.enable.rpc-over-tls=true
NOTE: the TLS layer is only used for in-flight data protection and have no influence on user mapping or authentication.
Client side configuration
To support certificate handling by the Linux kernel the tlshd service must be installed and running (provided by ktls-utils package on RHEL and clones). If dCache certificates are not signed by OS-wide trusted CA, then /etc/tlshd.conf should be updated such, that x509.truststore points server the trust certificate chain in PEM format.
The mount option xprtsec controls the transport security for NFS:
mount -o xprtsec=tls nfs-door:/data /mnt/
Configuring principal-id mapping for NFS access
The NFSv4.1 uses utf8 based strings to represent user and group names:
This is the case even for non-kerberos based accesses. Nevertheless UNIX based clients as well as dCache internally use numbers to represent uid and gids. A special service, called idmapd, takes care for principal-id mapping. On the client nodes the file /etc/idmapd.conf is usually responsible for consistent mapping on the client side. On the server side, in case of dCache mapping done through gplazma2. The identity type of plug-in required by id-mapping service. Please refer to Chapter 10, Authorization in dCache for instructions about how to configure `gPlazma.
For correct user id mapping nfs4 requires that server and client use the same naming scope, called nfs4domain. This implies a consistent configuration on both sides. To reduce deployment overhead a special auto-discovery mechanism was introduced by SUN Microsystems ( now Oracle) - a DNS TXT record. dCache supports this discovery mechanism. When nfs.domain property is set, it gets used. If it’s left unset, then DNS TXT record for _nfsv4idmapdomain is taken or the default localdomain is used when DNS record is absent.
To avoid big latencies and avoiding multiple queries for the same information, like ownership of a files in a big directory, the results from gPlazma are cached within NFSv4.1 door. The default values for cache size and life time are good enough for typical installation. Nevertheless they can be overriden in dcache.conf or layoutfile:
# maximal number of entries in the cache
nfs.idmap.cache.size = 512
# cache entry maximal lifetime
nfs.idmap.cache.timeout = 30
# time unit used for timeout. Valid values are:
# SECONDS, MINUTES, HOURS and DAYS
nfs.idmap.cache.timeout.unit = SECONDS
Managing group ids
The remote procedure call v2 (RPC) that used by nfs protocol defines different mechanisms for user authentication. One of them is AUTH_SYS, which majority of deployments are using. Dispire it’s popularity, the one downside of using AUTH_SYS is the sixteen groups limit. To overcome this limitation dCache nfs server can query gplazma to discover additional groups that the client might have. To enable it the nfs.idmap.manage-gids property can be used.
nfs.idmap.manage-gids = true
As enabling it might have negative effects on the performance, the default value it false.
NOTICE: currently only the ldap plugin supports group mapping by uid when
gplazma.ldap.try-uid-mappingoption is enabled.
Remote Quota Management
The NFSv4.1 door supports remote quota protocol, which allows to query user and group quotas. Only files with the retention policy REPLICA are used to report disk usage (See: quota system). The same value is reported for both the soft and hard quotas.
Example output:
$ quota -s -f /mnt
Disk quotas for user test (uid 1000):
Filesystem space quota limit grace files quota limit grace
dcache-lab:/
0K 100T 100T 2148m* 2148m 2148m 00:00
The remote quota service uses UDP protocol and listens on port specified by nfs.net.rquota.port property.
Accessing directory tags
There are two ways to access dCache directory tags over NFS mount: the legacy one as via magic files and as extended attributes.
Directory tags as magic files
Each directory tag can be accessed via special file name, for example to read or write a tag with the name foo, the magic file is .(tag)(foo):
echo bar > '.(tag)(foo)'
or
cat '.(tag)(foo)'
|bar
To list all existing file use .(tags)() magic file:
cat '.(tags)()'
|.(tag)(OSMTemplate)
|.(tag)(foo)
|.(tag)(sGroup)
NOTICE: the directory tags can’t be remove via magic files
Directory tags as extended attributes
The dCache’s NFSv4 door implements extended attributes over NFS which is specified in RFC8276. With compatible NFS client (for the moment only Linux kernel 5.10 and later), the directory tags can be access as extended attributes prefixed with dcache.tag.. For example:
attr -l .
|Attribute "dcache.tag.OSMTemplate" has a 15 byte value for .
|Attribute "dcache.tag.sGroup" has a 8 byte value for .
or
getfattr .
|# file: .
|user.dcache.tag.OSMTemplate
|user.dcache.tag.sGroup
NOTICE: the
getfattrandsetfattrcommands adds an extrauser.prefix.