CLOCKSS: Box Operations

From CLOCKSS Trusted Digital Repository Documents
Revision as of 22:09, 6 April 2014 by Dshr (Talk | contribs)

Jump to: navigation, search

Contents

CLOCKSS: Box Operations

Requirements for CLOCKSS host sites

A CLOCKSS host site signs an agreement with the CLOCKSS board under which they commit to providing:

  • Rack space in a physically secure location accessible only to authorized personnel.
  • Power.
  • Cooling.
  • Network bandwidth.

And to have an in-place disaster recovery plans if these resources fail. The CLOCKSS Executive Director has copies of these agreements.

The CLOCKSS Network Administrator is responsible for all processes involving production and ingest CLOCKSS box hardware and software infrastructure.

Hardware Bringup

Hardware Vendors

All new systems are currently purchased from iXsystems based in San Jose, CA. The LOCKSS team's contact there is sales representative Kevin Lee (kle@ixsystems.com).

The LOCKSS team previously worked with two hardware vendors, Iron Systems, located in Fremont, CA and eRacks in Orange, CA. Most of the systems currently in production were purchased from Iron Systems. Business contacts are Kawaljit Nagi (kawal@ironsystems.com) and James King (jamesk@eracks.com), respectively. The LOCKSS team also has a relationship with Joesph Wolff (joe@eracks.com), the owner of eRacks.

Hardware Purchase Process

Purchases start as a discussion among LOCKSS Engineering Staff, the CLOCKSS Network Administrator and the CLOCKSS Executive Director about what is needed. A specification is drawn by the engineers and sent to our hardware vendors for an early quote. If the quote looks reasonable, it is sent to the CLOCKSS Executive Director, who negotiates a final quote and sends a purchase order. The CLOCKSS Executive Director is advised to release payment once the machine has been built, tested, verified working by the LOCKSS team, preconfigured with software (see below) and shipped.

Hardware Warranty

Machines purchased from Iron Systems and eRacks come with 1 - 2 years of limited hardware warranty, depending on hardware vendor. The warranty on most of the CLOCKSS boxes has since expired but it proved useful for exchanging failed hardware components. Details about the warranty are in the purchase orders that have been submitted as well as the invoices subsequently received.

Recommended Hardware

CLOCKSS hardware or virtual machines meet or exceed the following specifications:

Item Ingest Production Triggered (VMware)
Chassis Supermicro 2U Supermicro 4U N/A
Disk Bays 12 24 N/A
Processor AMD Operton 6128 (eight cores) Dual Xeon E5504 (eight cores) Dual-core CPU
RAM 16GB ECC 24GB ECC 4GB
Disk 8 x 3TB SATA 6Gbps 7200RPM 12 x 2TB SATA 3Gbps 40GB disk
RAID LSI Megaraid Software (Linux mdadm) None (Underlying RAID array)
Network Onboard dual Gbe Onboard dual Gbe 10/100Mb NIC
Remote Access IPMI IPMI VMware vSphere Client
Power Supply 800W redundant 1600W redundant N/A

CLOCKSS Virtual Machines

CLOCKSS virtual machines run on servers running the VMWare vSphere ESXi 5 hypervisor.

Hardware Service Life

The service life of a CLOCKSS box is five to seven years. This is dictated by budget restrictions, software needs as well as cost and performance efficiency. Please see the section on #Hardware Replacement later in this document.

Building and testing

Our hardware vendors are responsible for sourcing components and building the machines once a purchase order is submitted by CLOCKSS. After a machine is built, both hardware vendors have an extensive and rigorous process in place to ensure a machine will function correctly.

Remote Access via IPMI

Every CLOCKSS box is equipped with an Intelligent Platform Management Interface (IPMI) module that provides remote "side-band" and "out-of-band" access. Although IPMI is invaluable in the event we need to access or recover a CLOCKSS box outside the scope of its operating system, there are known security vulnerabilities and so it is disabled by default on all machines. This is done by disabling it in BIOS and unplugging the network cable from the IPMI module.

In the event we need to use IPMI, the following precautions should be taken:

  1. Update the firmware, if a newer version exists.
  2. Do not use the default username and password.
  3. Make IPMI accessible only through a VPN or other secure connection.

RAID Configuration

The CLOCKSS Ingest machines have LSI MegaRAID based hardware RAID controllers and eight 3TB disks. These disks were partitioned into groups of four and put into a RAID5 configuration. Each RAID array then yields approximately 8.1TB of usable space after RAID and EXT4 filesystem overhead. Since the Ingest machines have an extra disk dedicated for the OS (called a "system disk"), the full space of the RAID arrays is dedicated to the LOCKSS daemon. This configuration also isolates problems on the system disk from content storage arrays.

The CLOCKSS Production machines utilize the Linux kernel's mdadm module to implement software RAID. Similar to the CLOCKSS Ingest machines, the Production machines also have several RAID5 arrays consisting of three or four disks depending on the number of disks in the machine. In contrast to the Ingest machines, these RAID5 arrays are used by both the operating system and the LOCKSS daemon.

RAID Health Monitoring

The most common component failures are disks and it is expected that all CLOCKSS boxes will experience a few disk failures over the course of their service life. The RAID5 arrays employed in CLOCKSS boxes are only able to tolerate one disk failure each. It is thus important that any failed disks be found and replaced as quickly as possible; a double disk failure requires repair from another box via the LOCKSS: Polling and Repair Protocol, which is much slower than RAID rebuild.

The RAID array health on all CLOCKSS Production boxes are monitored by Nagios via the Nagios Remote Plugin Executor (NRPE). Each machine runs an NRPE daemon and has a copy of a custom plugin we wrote to monitor mdadm RAID arrays. NRPE executes the plugin on behalf of Nagios and returns the result. If Nagios does not receive an OK, it sends an alert to CLOCKSS engineers.

Software Bringup

All the software used for the CLOCKSS infrastructure is freely available and open source.

CentOS Installation and Configuration

A base install of CentOS 6.x with Java options is performed on the partition or disk designated for the operating system then the following steps are taken to configure it as a CLOCKSS box:

  1. The system accounts, lcap and lockss are created using useradd -r -m or equivalent. Then the LOCKSS LCAP public SSH keys are then added to their authorized_keys file, taking care to ensure the permissions are setup correctly.
  2. The mount points are created for the LOCKSS daemon repositories:
    • The preferred naming scheme is /cacheN/gamma where N is an integer starting from 0, (e.g. /cache0/gamma, /cache1/gamma and so forth).
    • The mount point ownerships are changed to the lockss user and group and permissions for other are stripped.
    • noexec is added to the mount point parameters.
    • Finally, the mount points should be added to updatedb's PRUNEPATHS variable to prevent the system from indexing the LOCKSS daemon repositories.
  3. Then the firewall (iptables rules) are set up:
    • The LOCKSS daemon requires the LCAP port (port 9729), used to communicate with other LOCKSS daemons, to be open to all machines: This is accomplished under CentOS with this iptables rule: 
      -A INPUT -m state --state NEW -m tcp -p tcp --dport 9729 -j ACCEPT
      Note: Although we have a list of IP addresses that should be granted access to this machine's LCAP port, we opt not to restrict access because it is protected by SSL certificate checks (see #CLOCKSS PLN Configuration).
    • Additionally, a CLOCKSS box is setup so that the administrative ports 22 (OpenSSH) and 8080 through 8083 are accessible to the remote site's administrative subnet(s) and the LOCKSS subnet at Stanford (171.66.236.0/24, previously 171.66.236.0/26). The following iptables should be repeated for each subnet, taking care to replace SUBNET with the subnet in CIDR form:
      -A INPUT -m state --state NEW -m tcp -p tcp -s SUBNET --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s SUBNET --dport 8080 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s SUBNET --dport 8081 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s SUBNET --dport 8082 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s SUBNET --dport 8083 -j ACCEPT
      These rules should be reflected in any external firewalls at the site.
  4. Setting up the CLOCKSS repository is a two step process:
    1. The first is to create a new file, lockss.repo under /etc/yum.repos.d containing the following lines:
      [lockss]
name = LOCKSS Daemon Repository
baseurl=http://www.lockss.org/clockss-repo/
gpgcheck=1
    2. Then to install the LOCKSS RPM GPG key:
      rpm --import http://www.lockss.org/LOCKSS-GPG-RPM-KEY
  5. The LOCKSS daemon can now be installed by invoking:
    yum install lockss-daemon
  6. The LOCKSS daemon installs a logrotate configuration file to /etc/logrotate.d/lockss. By default, the size is limited to 2M but on CLOCKSS boxes, we set the limit to 20M: 
    /var/log/lockss/daemon {
    size 20M
    rotate 5
    compress
    delaycompress
    create
    notifempty
    missingok
}

/var/log/lockss/stdout {
    size 10k
    rotate 5
    compress
    copytruncate
    notifempty
    missingok
}
  7. To keep the time on CLOCKSS boxes synchronized, we use NTP (or preferably, OpenNTPd where available). To install ntpd:
    yum install ntp
  8. To automatically install package updates, we use yum-cron. To install and configure it: 
    yum install yum-cron
    Then ensure it will start automatically on runlevels 3, 4 and 5: 
    chkconfig --add yum-cron
  9. Some extra, optional packages we've found to be useful are screen, wget, vim, emacs, lynx. Their installation is highly recommended to ease troubleshooting.
    yum install screen wget vim emacs lynx

Java Runtime Environment (JRE)

Early CLOCKSS boxes were configured to use the Sun Java JRE 6 (now Oracle Java JRE) but we now recommend using the OpenJDK JRE 6 freely available through the CentOS repository. All CLOCKSS boxes have a 64-bit JRE to take advantage of the large amount of memory available on these machines.

LOCKSS Daemon Configuration

Once the CentOS environment has been configured and verified working by a CLOCKSS engineer, the next step is to configure the LOCKSS daemon as part of the CLOCKSS network. This is done using the hostconfig utility packaged with the LOCKSS daemon. For a CLOCKSS box, the following settings are recommended:

Parameter Description
Fully qualified hostname (FQDN) of this machine Provided by CLOCKSS. We schema we use is clockss-site.clockss.org where site can uniquely identify the host institution of the machine.
IP address of this machine A static public IP address provided by host institution.
Initial subnet for admin UI access The subnet (in CIDR or X.Y.Z.* notation) that will be granted access to the web-based LOCKSS Administrative UI. The localhost is implicitly allowed. Additional subnets can be added through the LOCKSS Administrative UI.
LCAP V3 protocol port The TCP port the daemon will listen on for LCAP communication from peers. Remote sites should ensure both internal and external firewalls allow all connections to this port.
Mail relay for this machine This should be the DNS name of an SMTP relay that will accept and relay mail from this machine. The script will also prompt for a username and password if the mail relay requires them. It can also be set to localhost if the machine is capable of handling email.
E-mail address for administrator Occasional alerts will be sent to this address by the LOCKSS daemon.
Path to java The full path to a JRE. The default should suffice in most cases.
Java switches Java switches to be passed to the JRE. It should be left blank in most cases.
Configuration URL
Preservation group(s)
  • CLOCKSS Production: clockss
  • CLOCKSS Ingest: clockssingest
  • CLOCKSS Triggered: clockss-triggered
Content storage directories A semi-colon delineated list of paths to the LOCKSS repositories for use by this LOCKSS daemon.
Temporary storage directory /cache0/gamma/tmp
Password for web UI administration user admin Set this to a strong password, known to the host site administrator but to no-one else.

CLOCKSS PLN Configuration

The production CLOCKSS network is configured via the property server to:

  • Prevent access to the content by disabling both the proxy and content server functions of the LOCKSS daemon.
  • Prevent interception of communication between boxes in the network by the use of SSL for the LOCKSS: Polling and Repair Protocol.
  • Prevent communication via the LOCKSS: Polling and Repair Protocol except from other boxes in the same network by requiring each end of a newly established connection to verify the certificate of the other end against the corresponding public key in a keystore. During setup, or when the set of boxes in the network changes, the appropriate keystore is uploaded to a machine via scp. We also provide a small script for the CLOCKSS site administrator to run that installs the keystore into the correct location with restrictive permissions.

Monitoring

Nagios

The CLOCKSS infrastructure is monitored by the Nagios network monitoring system. The CLOCKSS boxes, servers and services that are monitored by Nagios include the CLOCKSS Production boxes, Ingest boxes, Triggered Content boxes and Property Server as well as the CLOCKSS HTTP, SMTP and FTP servers.

The CLOCKSS Network Administrator is responsible for Nagios monitoring.

Nagios plugins to monitor LOCKSS

Custom Nagios plugins were written to enable Nagios to monitor the following LOCKSS daemon services:

  • LOCKSS Daemon Version
  • LOCKSS Daemon Uptime
  • LOCKSS Repository Spaces (to monitor disk usage)
  • LOCKSS Web Administrative UI accessibility
  • LCAP Accessibility

Plugins shipped with Nagios allow us to monitor, where applicable:

  • OpenSSH accessibility
  • HTTP/HTTPS accessibility

Nagios Alerts

Nagios allows plugins to return four return codes depending on the nature and severity of an issue (or lack thereof). For any return other than OK, Nagios has been configured to notify CLOCKSS engineers through email alerts (see CLOCKSS: Logging and Records).

Access control to Nagios instance

Access to the CLOCKSS Nagios instance is restricted by an Apache Virtual Host definition to the LOCKSS subnet at Stanford (171.66.236.0/24) and by username and password.

Nagios Redundancy

CLOCKSS and LOCKSS have a second Nagios instance running in an Amacon EC2 instance located in West Virginia. The second Nagios is not able to monitor CLOCKSS boxes directly. This is by design: We do not want a machine outside of the Stanford network to have the type of access necessary to monitor internal CLOCKSS boxes and processes. Due to this restriction, it is configured only to monitor the CLOCKSS website and the Nagios instance running at Stanford. The second Nagios instance will alert CLOCKSS engineers should any problems occur at Stanford.

Hardware Upgrade

Currently, all CLOCKSS boxes have at least four spare hotswap disk bays. A natural way to incrementally upgrade the disk capacity of deployed CLOCKSS boxes is to fill these disk bays with the largest disk available (at the moment, 4TB), create a new RAID array and finally to move content from an existing array. The disks comprising the old array can then be removed and the process can be repeated for the remaining RAID arrays.

The other components in a CLOCKSS box are not expected to need upgrading during its service life.

Hardware Replacement

Component Failure

The components in all CLOCKSS boxes are readily available, cheap, off-the-shelf components and replacement components can be sourced from any well-stocked computer retailer. The most common component failures are disks. As mentioned earlier, the RAID5 arrays employed in CLOCKSS boxes are only able to tolerate one disk failure each so it is important that any failed disks be found and replaced as quickly as possible.

If a disk fails, we contact the CLOCKSS box's hardware vendor if the CLOCKSS box is still under warranty. If the box is no longer under warranty, we prefer to purchase a disk and ship it expedited to the remote site. However, if the remote site is not within the United States, an alternative option is to ask the site administrator to source replacement parts within their country and send an invoice to CLOCKSS for a reimbursement.

Other component failures are handled similarly. If a machine cannot be repaired, it is replaced.

Planned CLOCKSS Box Replacement

Despite the upgradability of existing hardware, we recognize that, eventually, it will no longer be cost effective or practical to continue to run old hardware due to increases in performance per watt efficiency, disk bandwidth, disk capacity and other measures. Software and content volume is also expected to push hardware towards obsolescence.

Replacing an entire machine is a fairly straightforward process. After purchasing and configuring the machine as described elsewhere in this document, it is brought up alongside the machine it is to replace. To minimize downtime, content is copied in two passes, using rsync. The first pass copies the bulk of the data to the new machine. Once the first pass is complete, the LOCKSS daemon on the old machine is taken offline and its filesystems are placed into read-only mode. The second pass then copies any changes to the content that occurred during the first pass, and verifies the integrity of content from the first pass. If no errors occurred during the two passes, the old machine is taken offline and the new machine takes its place. Subsequent LOCKSS: Polling and Repair Protocol will also verify the integrity of the copy.

Software Updates

Re-locating Content

Relocating an AU to another LOCKSS repository filesystem, for example if a filesystem has filled up, requires the following steps:

  1. Shutdown the LOCKSS daemon to prevent it from modifying the AU. For extra precaution, temporarily remount the filesystem as read-only.
  2. Copy the AU to the new LOCKSS repository. When the copy completes successfully, delete the the AU from the source filesystem.
  3. If the LOCKSS repository has not yet been made available to the LOCKSS daemon, edit /etc/lockss/config.dat and append the new LOCKSS repository path to LOCKSS_DISK_PATHS.
  4. The AU needs its lockss_repo added or modified to reflect the new path. This is done by editing the AU's lockss_repo line in /cache0/gamma/config/au.txt. For bulk relocation, a LOCKSS tool called auconf.pl can scan the LOCKSS repositories for AUs and update au.txt as necessary.
  5. If the source filesystem was remounted read-only, remount it as read-write.
  6. Finally, start the LOCKSS daemon and check that the AUs repository path points to the new location.

If a large amount of content is being moved and the daemon requires minimal disruption, a modified procedure is followed:

  1. With the LOCKSS daemon running and the filesystem mounted read-write, use rsync to copy the bulk of the content to the new filesystem.
  2. When the first pass completes, stop the LOCKSS daemon and remount the source filesystem as read-only.
  3. Run rsync again to copy any changes to the AUs since the last rsync was started.
  4. Update /etc/lockss/config.dat and /cache0/gamma/config/au.txt as necessary.
  5. Remount the filesystem as read-write. Bring up the LOCKSS daemon and check that the all the AUs repository paths have been updated.

LOCKSS Daemon Updates

All remote sites are encouraged to setup automatic updates for the LOCKSS daemon package so that updates are installed automatically within a few hours of their release to the CLOCKSS RPM repository. Additionally, all systems administrators receive announcements of newly released daemons through the CLOCKSS Administrators mailing list; systems administrators opting to install updates manually are encouraged to use the announcements as a cue to install updates. We use Nagios to monitor the LOCKSS daemon version of all CLOCKSS boxes. If a CLOCKSS box has not been updated with the latest LOCKSS daemon within a day, we notify its systems administrator(s).

System Package Updates

Most CLOCKSS boxes are configured for unattended upgrade, so new upgrades will be installed automatically. All CLOCKSS box systems administrators are encouraged to join the CentOS Announce mailing list to receive real-time annoucements about security updates. If the box in question is not configured for unattended updates:

  • Security updates should be downloaded and installed as soon as they are announced.
  • All other non-critical updates should be installed at least once a quarter.

Anyone can subscribe to the CentOS Announce mailing list by visiting the CentOS-announce Mailing List home page.

System Updates

During the lifetime of the CLOCKSS program, the CentOS project will release new major versions of their CentOS operating system (e.g., CentOS 5.x to 6.x). All production CLOCKSS boxes were configured so that its operating system filesystem is independent from its content filesystems. The CentOS upgrade path is then relatively simple: Save the LOCKSS daemon configuration, user accounts and OpenSSH keys, unmount and take offline all content filesystems then effectively reinstall CentOS and the LOCKSS daemon environment and restore the CLOCKSS box's configuration.

The end of life dates for the CentOS versions currently installed on CLOCKSS boxes are:

Version Release Date Full Updates Maintenance Updates
CentOS 5.x 2007-04-12 Q1 2014 2017-03-31
CentOS 6.x 22011-07-10 Q2 2017 2020-11-30

Change Process

Changes to this document require:

  • Review by:
    • CLOCKSS Network Administrator
    • LOCKSS Engineering Staff
  • Approval by CLOCKSS Technical Lead
Retrieved from "http://documents.clockss.org/index.php?title=CLOCKSS:_Box_Operations&oldid=285"