In this post, we document how we installed and configured BBCP on pulpo-dtn.

• Installation
• Firewall
• Testing

Installation

For simplicity, We simply downloaded a precompiled bbcp binary executable and placed it in /usr/local/bin:

# cd /usr/local/bin/
# wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp
# chmod +x bbcp

We note in passing that although the binary executable was built for 64-bit RHEL 6, it works without issue on RHEL/CentOS 7.

Create account for Jeffrey LeFevre on pulpo-dtn:

# ldapsearch -x -H ldap://ldap-blue.ucsc.edu -LLL 'uid=jlefevre'
dn: uid=jlefevre,ou=people,dc=ucsc,dc=edu
gidNumber: 100000
uidNumber: 28981

# groupadd -g 28981 jlefevre
# useradd -u 28981 -g jlefevre -c "Jeffrey LeFevre" -m -d /mnt/pulpos/jlefevre jlefevre

Firewall

Append the following 2 lines to /etc/services:

bbcpfirst       60000/tcp               # bbcp
bbcplast        60015/tcp               # bbcp

Open inbound TCP ports 60000 - 60015 from any using FirewallD:

# firewall-cmd --permanent --zone=public --add-port=60000-60015/tcp
success

# firewall-cmd --reload
success

Testing

Similarly install BBCP on on the 4-GPU workstation Hydra.

Transfer a 4GB file from hydra to pulpo-dtn:

$ bbcp -P 1 -F 4GB.dat jlefevre@pulpo-dtn.ucsc.edu:/mnt/pulpos/jlefevre/4GB.dat

NOTE:

1). We must use -F option, which forces the copy by not checking if there is enough free space on the target host, in order work around a bug in Ceph Luminous; otherwise we’ll get the following error:

bbcp: Insufficient space to copy all the files from hydra.soe.ucsc.edu

2). BBCP doesn’t honor ssh_config. If I place the following stanza in ~/.ssh/config on hydra:

Host pd
        HostName pulpo-dtn.ucsc.edu
        User jlefevre
        IdentityFile ~/.ssh/pulpos

and attempt a transfer using the following command (pd instead of jlefevre@pulpo-dtn.ucsc.edu):

$ bbcp -P 1 -F 4GB.dat pd:/mnt/pulpos/jlefevre/4GB.dat

the command will fail:

bbcp: Unable to connect to  (null) ;  Name or service not known
bbcp: Unable to allocate more than 0 of 4 data streams.
bbcp: Accept timed out on port 60000
bbcp: Unable to allocate more than 0 of 4 data streams.

3). If we use a private SSH key that is not the default ~/.ssh/identity, ~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 or ~/.ssh/id_rsaid_rsa, we can use the -i option to specify the key. For example:

$ bbcp -i ~/.ssh/pulpos -P 1 -F 4GB.dat jlefevre@pulpo-dtn.ucsc.edu:/mnt/pulpos/jlefevre/4GB.dat