CLOCKSS: Box Operations - Revision history

Ntay: restoring post-2016 edits

2019-08-12T21:31:45Z

restoring post-2016 edits

Show changes

Dshr: /* LOCKSS Daemon Updates */

2014-04-06T22:10:37Z

‎LOCKSS Daemon Updates

Dshr: /* Re-locating Content */

2014-04-06T22:09:46Z

‎Re-locating Content

Dshr: /* LOCKSS Daemon Configuration */

2014-04-06T22:08:27Z

‎LOCKSS Daemon Configuration

Dshr: /* CentOS Installation and Configuration */

2014-04-06T22:06:54Z

‎CentOS Installation and Configuration

Dshr: /* Nagios plugins to monitor LOCKSS */

2014-04-06T22:01:45Z

‎Nagios plugins to monitor LOCKSS

Dshr: /* CLOCKSS PLN Configuration */

2014-04-06T22:00:41Z

‎CLOCKSS PLN Configuration

Dshr: /* RAID Configuration */

2014-04-06T21:58:56Z

‎RAID Configuration

Dshr: /* Change Process */

2013-12-07T22:46:49Z

‎Change Process

Dshr: /* CLOCKSS PLN Configuration */

2013-09-30T14:08:30Z

‎CLOCKSS PLN Configuration

@@ Line 280: / Line 280: @@
 === LOCKSS Daemon Updates ===
-All remote sites are encouraged to setup automatic updates for the LOCKSS daemon package so that updates are installed automatically within a few hours of their release to the CLOCKSS RPM repository.  Additionally, all systems administrators receive announcements of newly released daemons through the CLOCKSS Administrators mailing list; systems administrators opting to install updates manually are encouraged to use the announcements as a cue to install updates. We use Nagios to monitor the LOCKSS daemon version of all CLOCKSS boxes. If a CLOCKSS box has not been updated with the latest LOCKSS daemon within a day, we notify its systems administrator(s).
+All remote sites are encouraged to setup automatic updates for the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]] package so that updates are installed automatically within a few hours of their release to the CLOCKSS RPM repository.  Additionally, all systems administrators receive announcements of newly released daemons through the CLOCKSS Administrators mailing list; systems administrators opting to install updates manually are encouraged to use the announcements as a cue to install updates. We use Nagios to monitor the LOCKSS daemon version of all CLOCKSS boxes. If a CLOCKSS box has not been updated with the latest LOCKSS daemon within a day, we notify its systems administrator(s).
 === System Package Updates ===

@@ Line 263: / Line 263: @@
 Relocating an AU to another LOCKSS repository filesystem, for example if a filesystem has filled up, requires the following steps:
-# Shutdown the LOCKSS daemon to prevent it from modifying the AU. For extra precaution, temporarily remount the filesystem as read-only.
+# Shutdown the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]] to prevent it from modifying the AU. For extra precaution, temporarily remount the filesystem as read-only.
 # Copy the AU to the new LOCKSS repository. When the copy completes successfully, delete the the AU from the source filesystem.
 # If the LOCKSS repository has not yet been made available to the LOCKSS daemon, edit <tt>/etc/lockss/config.dat</tt> and append the new LOCKSS repository path to <tt>LOCKSS_DISK_PATHS</tt>.

@@ Line 148: / Line 148: @@
 === LOCKSS Daemon Configuration ===
-Once the CentOS environment has been configured and verified working by a CLOCKSS engineer, the next step is to configure the LOCKSS daemon as part of the CLOCKSS network. This is done using the <tt>hostconfig</tt> utility packaged with the LOCKSS daemon. For a CLOCKSS box, the following settings are recommended:
+Once the CentOS environment has been configured and verified working by a CLOCKSS engineer, the next step is to configure the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]] as part of the CLOCKSS network. This is done using the <tt>hostconfig</tt> utility packaged with the LOCKSS daemon. For a CLOCKSS box, the following settings are recommended:
 {| class="wikitable"

@@ Line 125: / Line 125: @@
 the following steps are taken to configure it as a CLOCKSS box:
 # The system accounts, <tt>lcap</tt> and <tt>lockss</tt> are created using <tt>useradd -r -m</tt> or equivalent. Then the LOCKSS LCAP public SSH keys are then added to their <tt>authorized_keys</tt> file, taking care to ensure the permissions are setup correctly.
-# The mount points are created for the LOCKSS daemon repositories:
+# The mount points are created for the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]] repositories:
 #* The preferred naming scheme is <tt>/cacheN/gamma</tt> where <tt>N</tt> is an integer starting from 0, (e.g. <tt>/cache0/gamma</tt>, <tt>/cache1/gamma</tt> and so forth).
 #* The mount point ownerships are changed to the <tt>lockss</tt> user and group and permissions for <tt>other</tt> are stripped.

@@ Line 213: / Line 213: @@
 === Nagios plugins to monitor LOCKSS ===
-Custom Nagios plugins were written to enable Nagios to monitor the following LOCKSS daemon services:
+Custom Nagios plugins were written to enable Nagios to monitor the following [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]] services:
 * LOCKSS Daemon Version
 * LOCKSS Daemon Uptime

@@ Line 199: / Line 199: @@
 The production CLOCKSS network is configured via the [[LOCKSS: Property Server Operations|property server]] to:
-* Prevent access to the content by disabling both the proxy and content server functions of the LOCKSS daemon.
+* Prevent access to the content by disabling both the proxy and content server functions of the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]].
 * Prevent interception of communication between boxes in the network by the use of SSL for the [[LOCKSS: Polling and Repair Protocol]].
 * Prevent communication via the [[LOCKSS: Polling and Repair Protocol]] except from other boxes in the same network by requiring each end of a newly established connection to verify the certificate of the other end against the corresponding public key in a keystore. During setup, or when the set of boxes in the network changes, the appropriate keystore is uploaded to a machine via <tt>scp</tt>. We also provide a small script for the CLOCKSS site administrator to run that installs the keystore into the correct location with restrictive permissions.

@@ Line 106: / Line 106: @@
 === RAID Configuration ===
-The CLOCKSS Ingest machines have LSI MegaRAID based hardware RAID controllers and eight 3TB disks. These disks were partitioned into groups of four and put into a RAID5 configuration. Each RAID array then yields approximately 8.1TB of usable space after RAID and EXT4 filesystem overhead. Since the Ingest machines have an extra disk dedicated for the OS (called a "system disk"), the full space of the RAID arrays is dedicated to the LOCKSS daemon. This configuration also isolates problems on the system disk from content storage arrays.
+The CLOCKSS Ingest machines have LSI MegaRAID based hardware RAID controllers and eight 3TB disks. These disks were partitioned into groups of four and put into a RAID5 configuration. Each RAID array then yields approximately 8.1TB of usable space after RAID and EXT4 filesystem overhead. Since the Ingest machines have an extra disk dedicated for the OS (called a "system disk"), the full space of the RAID arrays is dedicated to the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]]. This configuration also isolates problems on the system disk from content storage arrays.
 The CLOCKSS Production machines utilize the Linux kernel's <tt>mdadm</tt> module to implement software RAID. Similar to the CLOCKSS Ingest machines, the Production machines also have several RAID5 arrays consisting of three or four disks depending on the number of disks in the machine. In contrast to the Ingest machines, these RAID5 arrays are used by both the operating system and the LOCKSS daemon.

@@ Line 321: / Line 321: @@
 ** CLOCKSS Network Administrator
 ** LOCKSS Engineering Staff
-* Approval by LOCKSS Technical Lead
+* Approval by CLOCKSS Technical Lead