Internal Copy Manager

Purpose

Limitations

• Beta version, not yet tested properly.
• The destination pool must have suffient (empty or removable) space to hold the content of the source pool.
• A single CoypManager can only manage a single source destination pool pair at a time. But more than one CopyManager can be started to manage other pool copy pairs.

Preparing the copy manager

Create a batch file with the following content within the dCache config directory (preferrable on the headnode). The file should be called copy.batch.

#
set printout default 3
set printout CellGlue none
onerror shutdown
#
check -strong setupFile
#
copy file:${setupFile} context:setupContext
#
#  import the variables into our $context.
#  don't overwrite already existing variables.
#
import context -c setupContext
#
#   Make sure we got what we need.
#
check -strong serviceLocatorHost serviceLocatorPort
#
create dmg.cells.services.RoutingManager  RoutingMgr
#
#   The LocationManager Part
#
create dmg.cells.services.LocationManager lm \
       "${serviceLocatorHost} ${serviceLocatorPort} "
#
#

#
create diskCacheV111.replicaManager.CopyManager copy0 \
       "default -export"
#

Change the cwd to jobs and run

./initPackage.sh

(and ignore possible error message).

Start the copy Domain. (from the jobs directory)

./copy start

Everything up to here only has to be done once. The follow procedure has to be done for each full pool copy.

Copy manager commands

Login to the admin interface.

ssh -l admin -c blowfish -p 22223 <headnode>
....
#
#  change the pnfs timeout of the destination pool
#  
#
cd <DestinationPool>
pp set pnfs timeout 300
save
#
#  define the maximum number of pool-to-pool transfers of the
#  source pool.
#
..
cd <SourcePool>

p2p set max active 10
#
#
#  make sure, all incoming trafic stopped.
#
mover ls
pp ls
#
#  change to copy0 and start the transfers
#
..
cd copy0
copy  <SourcePool> <DestinationPool> -max=5
#
#  check progress
#
info
#

The max options allows to specify the maximum number of active transfers at a time. The number in the p2p set max active command must be larger than the -max number.

Check the progress by using the info command in the copy cell. A progess bar should grow from right to left until the full content has been copied.

It might happen, that the info command times out when used immediately after the copy had been started. This is because the copy cell is rather busy while setting up the transfers. After awhile info will become responsive again.

Using the -precious option restricts the copy process to files which are precious.

copy  <SourcePool> <DestinationPool> -max=5 -precious

One may modify the copy.batch file to start more than one CopyManager. In that case make sure all of them have distingued names. (copy0, coyp1, copy2).

create diskCacheV111.replicaManager.CopyManager copy0 \
       "default -export"
#
create diskCacheV111.replicaManager.CopyManager copy1 \
       "default -export"
#
create diskCacheV111.replicaManager.CopyManager copy2 \
       "default -export"
#

If the destionation pool already holds a subset of the content of the source pool, those files are not copied again but are shown in the copy cell info command.

Precautions and recovery from problems

Because the software is not yet suffiently tested it would be good to do a

rep ls -s

on the source and destination pools before the copy process started and the same on the destination pool after the process finished. Make sure the numbers are consistent.

If for whatever reason, the copy process messes up, its no problem to restart the copy domain.

#
#  in the jobs directory
#
./copy stop
./copy start

One only needs to restart the copy command to proceed. Files already copied will not be copied again.

Bugs and problems

Please report all problems to support@dCache.ORG.