Edit me on GitHub

dCache as NFSv4.1 Server

This chapter explains how to configure dCache in order to access it via the NFSv4.1 protocol, allowing clients to mount dCache and perform POSIX IO using standard NFSv4.1 clients.

Important

The pNFS mentioned in this chapter is the protocol NFSv4.1/pNFS and not the namespace pnfs.



Setting up

To allow file transfers in and out of dCache using NFSv4.1/pNFS, a new NFSv4.1 door must be started. This door acts then as the mount point for NFS clients.

To enable the NFSv4.1 door, you have to change the layout file corresponding to your dCache-instance. Enable the NFS within the domain that you want to run it by adding the following line

[<domainName>/nfs]
nfs.version = 4.1

Example: You can just add the following lines to the layout file:

[nfs-${host.name}Domain]
[nfs-${host.name}Domain/nfs]
nfs.version = 4.1

Exporting file system

In addition to run an NFSv4.1 door you need to add exports to the exports file. The location of exports file is controlled by nfs.export.file property and defaults to /etc/exports.

After reading exports file dCache will read the content of the directory with additional export tables. The location of directory defined by nfs.export.dir property and default to /etc/exports.d. Only files ending with .exports are considered. Files starting with a dot are ignored. The format of the export tables is similar to the one which is provided by Linux:

#
<path> [host [(options)]]

Where is a comma separated combination of:

  • ro matching clients can access this export only in read-only mode. This is the default value.

  • rw matching clients can access this export only in read-write mode

  • noacl dCache ACLs will be ignored; only posix access permissions will be considered. This is the default value.

  • acl dCache ACLs will be respected; if present, they override posix permissions. To view the ACLs at the NFS client side, use nfs4_getfacl which is in EL7 package nfs4-acl-tools.

  • sec=sys matching clients must access NFS using RPCSEC_SYS authentication, where client’s local uid and gid is used. This is the default value.

  • sec=krb5 matching clients must access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is NONE, e.g., the data is neither encrypted nor signed when sent over the network. Nevertheless the RPC packets header still protected by checksum.

  • sec=krb5i matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is INTEGRITY. The RPC requests and response are protected by checksum.

  • sec=krb5p matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is PRIVACY. The RPC requests and response are protected by encryption.

  • all_squash Map all uis and gids to the anonymous user.

  • no_root_squash Do not map requests from the root to anonymous user. This is useful for clients with administrative rights.

  • all_root the requests from matching clients will be handles as from user root. Additionally, when a new file or directory is created, then parent directory’s ownership is inherited.

  • anonuid and anongid specify which uid and gid should be used for anonymous user.

  • dcap Expose the fact, that nfs server is a dCache instance through special dot file. This option is useful when dcap client are used over mounted file system and preferred IO protocol is DCAP. This is the default value.

  • no_dcap The opposite of dcap option. Hide the fact, that nfs server is a dCache instance. This option is useful when dcap client are used over mounted file system but nfs protocol still have to be used for IO.

  • pnfs Enable NFSv4.1/pNFS to redirect pnfs capable clients to the dCache pools. This is the default value.

  • no_pnfs and nopnfs Opposite of pnfs option. Enforce clients to send all IO requests to NFS door event independent from client’s pNFS capabilities.

  • secure Specifying whether clients are required to use a privileged port (< 1024). This option typically is used to disallow user-space NFS clients, as they might spoof request credentials.

  • insecure The opposite of secure option. The use of privileged port by a client is not required. This is the default value.

  • lt= A colon separated ordered list of pnfs layouts that offered to the client. The valid values are:

  • flex_files

  • nfsv4_1_files

For modern clients (linux kernel >= 4.10 and RHEL7.4) flex_files is recommended. Default is nfsv4_1_files.

For example:

/pnfs/dcache.org/data *.dcache.org (rw,sec=krb5i)

Notice, that security flavour used at mount time will be used for client - pool communication as well.

Multiple specifications can be declared like this:

/pnfs/dcache.org/data *.dcache.org(rw) externalhost.example.org(ro)

In this example, hosts in the dcache.org may read and write, while host externalhost.example.org may only read.

If there are multiple path specifications, the shortest matching path wins. If there are multiple host/subnet specifications, the most precise specification wins.

Pool IP addresses for pNFS redirection

With pNFS the NFS door redirects clients to the pool that holds the data, so each pool must know the IP addresses it advertises to clients. By default a pool advertises the addresses of all of the host’s non-loopback network interfaces that are up, and rediscovers them while running.

The advertised addresses can be restricted with the pool.mover.nfs.multipath property, which takes a comma separated list of IP addresses:

pool.mover.nfs.multipath=192.168.1.1,fe80::2e2e:50de:7aa5:395

When pool.mover.nfs.multipath is empty, the pool advertises all local addresses, unless the org.dcache.net.localaddresses Java property is set.

If a pool cannot identify any IP addresses — for example because pool.mover.nfs.multipath lists no valid addresses, or because no non-loopback network interface is up — then the pool fails to start instead of running without the addresses needed for pNFS redirection. In that case check that the host’s network interfaces are up and correctly configured, and that pool.mover.nfs.multipath (if set) contains valid addresses.

Concurrent write access

A file that is currently open for writing is exclusively locked: while such an open file exists, the NFS door denies any other attempt to open the same file, regardless of whether that attempt requests read or write access. Any such attempt fails with Permission denied (EACCES).

When the writing client closes the file, the lock is released and further opens succeed. This prevents two concurrent writers from creating competing replicas of the same file (with the same pnfsid but different contents) and reflects the fact that dCache treats file content as write-once.

Reporting capacity to clients

Clients that mount dCache with NFSv4.1 use the file system statistics (the values shown by df) to see how much space is available. By default the NFS door reports a fixed size of 1 EiB with all of it free, so df is of little use for monitoring or for clients that decide whether they can upload based on the reported free space.

The nfs.capacity.source property controls the capacity the door reports. It defaults to the value of dcache.capacity.source, which is namespace unless changed:

nfs.capacity.source=namespace

The supported values are:

  • namespace: a fixed size of 1 EiB, all of it free. This is the default and keeps the previous behaviour. The number of files and (with this source) the used space are taken from the namespace.

  • pools: the total and free space of the online pools, counting removable (cached) replicas as free, just as the frontend does for storage resource reporting.

Because NFS does not tell the door which file a statistics request is for, the NFS door always reports the space of all online pools. It cannot narrow the reported pools to those that a file in a particular directory would be written to, so the reported capacity does not reflect the routing or space reservation configuration for a specific path.

  • reservation: the size and available space of a space reservation, as the FTP door reports for SITE USAGE.

As the NFS door does not know the queried path, it cannot read the WriteToken tag itself. With this source you must set the space token of the reservation to report:

nfs.capacity.source=reservation
nfs.capacity.reservation=10000

The door fails to start if reservation is selected without nfs.capacity.reservation. The value must be the numeric id of a space reservation.

For pools and reservation, the door keeps the file counts from the namespace and falls back to the namespace statistics if the capacity cannot be determined (for example, while the spacemanager or poolmanager is unavailable). A failure is logged once and reported again once the capacity can be determined.

Configuring NFSv4.1 door with GSS-API support

Adding sec=krb5 into /etc/exports is not sufficient to get kerberos authentication to work.

All clients, pool nodes and node running DOOR-NFS4 must have a valid kerberos configuration. Each clients, pool node and node running DOOR-NFS4 must have a /etc/krb5.keytab with nfs service principal:

nfs/host.domain@<YOUR.REALM>

The /etc/dcache/dcache.conf on pool nodes and node running NFSv4.1 door must enable kerberos and RPCSEC_GSS:

nfs.rpcsec_gss=true
dcache.authn.kerberos.realm=<YOUR.REALM>
dcache.authn.jaas.config=/etc/dcache/gss.conf
dcache.authn.kerberos.key-distribution-center-list=your.kdc.server

The /etc/dcache/gss.conf on pool nodes and node running NFSv4.1 door must configure Java’s security module:

com.sun.security.jgss.accept {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useKeyTab=true
keyTab="${/}etc${/}krb5.keytab"
debug=false
storeKey=true
principal="nfs/host.domain@<YOUR.REALM>";
};

Now your NFS client can securely access dCache.

Configuring NFSv4.1 with RPC-over-TLS

dCache NFS door supports RPC-over-TLS protocol extension for data-in-transit protection. To work with pNFS, host certificates must be installed on NFS doors and pools. As pNFS uses multipath addresses for client redirection, the host certificates must include the IPv4/IPv6 records in Subject Alternative Name.

To enable, the RPC-over-TLS capability on pools and nfs doors the following properties must be enabled:

pool.mover.nfs.enable.rpc-over-tls=true
nfs.enable.rpc-over-tls=true

or

dcache.enable.rpc-over-tls=true

NOTE: the TLS layer is only used for in-flight data protection and have no influence on user mapping or authentication.

Client side configuration

To support certificate handling by the Linux kernel the tlshd service must be installed and running (provided by ktls-utils package on RHEL and clones). If dCache certificates are not signed by OS-wide trusted CA, then /etc/tlshd.conf should be updated such, that x509.truststore points server the trust certificate chain in PEM format.

The mount option xprtsec controls the transport security for NFS:

mount -o xprtsec=tls nfs-door:/data /mnt/

Configuring principal-id mapping for NFS access

The NFSv4.1 uses utf8 based strings to represent user and group names:

user@DOMAIN.NAME

This is the case even for non-kerberos based accesses. Nevertheless UNIX based clients as well as dCache internally use numbers to represent uid and gids. A special service, called idmapd, takes care for principal-id mapping. On the client nodes the file /etc/idmapd.conf is usually responsible for consistent mapping on the client side. On the server side, in case of dCache mapping done through gplazma2. The identity type of plug-in required by id-mapping service. Please refer to Chapter 10, Authorization in dCache for instructions about how to configure `gPlazma.

For correct user id mapping nfs4 requires that server and client use the same naming scope, called nfs4domain. This implies a consistent configuration on both sides. To reduce deployment overhead a special auto-discovery mechanism was introduced by SUN Microsystems ( now Oracle) - a DNS TXT record. dCache supports this discovery mechanism. When nfs.domain property is set, it gets used. If it’s left unset, then DNS TXT record for _nfsv4idmapdomain is taken or the default localdomain is used when DNS record is absent.

To avoid big latencies and avoiding multiple queries for the same information, like ownership of a files in a big directory, the results from gPlazma are cached within NFSv4.1 door. The default values for cache size and life time are good enough for typical installation. Nevertheless they can be overriden in dcache.conf or layoutfile:

# maximal number of entries in the cache
nfs.idmap.cache.size = 512

# cache entry maximal lifetime
nfs.idmap.cache.timeout = 30

# time unit used for timeout. Valid values are:
# SECONDS, MINUTES, HOURS and DAYS
nfs.idmap.cache.timeout.unit = SECONDS

Managing group ids

The remote procedure call v2 (RPC) that used by nfs protocol defines different mechanisms for user authentication. One of them is AUTH_SYS, which majority of deployments are using. Dispire it’s popularity, the one downside of using AUTH_SYS is the sixteen groups limit. To overcome this limitation dCache nfs server can query gplazma to discover additional groups that the client might have. To enable it the nfs.idmap.manage-gids property can be used.

nfs.idmap.manage-gids = true

As enabling it might have negative effects on the performance, the default value it false.

NOTICE: currently only the ldap plugin supports group mapping by uid when gplazma.ldap.try-uid-mapping option is enabled.

Remote Quota Management

The NFSv4.1 door supports remote quota protocol, which allows to query user and group quotas. Only files with the retention policy REPLICA are used to report disk usage (See: quota system). The same value is reported for both the soft and hard quotas.

Example output:

$ quota -s -f /mnt
    Disk quotas for user test (uid 1000):
         Filesystem   space   quota   limit   grace   files   quota   limit   grace
    dcache-lab:/
                         0K    100T    100T           2148m*  2148m   2148m   00:00

The remote quota service uses UDP protocol and listens on port specified by nfs.net.rquota.port property.

Accessing directory tags

There are two ways to access dCache directory tags over NFS mount: the legacy one as via magic files and as extended attributes.

Directory tags as magic files

Each directory tag can be accessed via special file name, for example to read or write a tag with the name foo, the magic file is .(tag)(foo):

echo bar > '.(tag)(foo)'

or

cat '.(tag)(foo)'
|bar

To list all existing file use .(tags)() magic file:

 cat '.(tags)()'
|.(tag)(OSMTemplate)
|.(tag)(foo)
|.(tag)(sGroup)

NOTICE: the directory tags can’t be remove via magic files

Directory tags as extended attributes

The dCache’s NFSv4 door implements extended attributes over NFS which is specified in RFC8276. With compatible NFS client (for the moment only Linux kernel 5.10 and later), the directory tags can be access as extended attributes prefixed with dcache.tag.. For example:

 attr -l .
|Attribute "dcache.tag.OSMTemplate" has a 15 byte value for .
|Attribute "dcache.tag.sGroup" has a 8 byte value for .

or

getfattr  .
|# file: .
|user.dcache.tag.OSMTemplate
|user.dcache.tag.sGroup

NOTICE: the getfattr and setfattr commands adds an extra user. prefix.