CLOCKSS: Threats and Mitigations - Revision history

Ntay: restoring post-2016 edits

2019-08-14T23:23:10Z

restoring post-2016 edits

Dshr: /* Operator Error */

2014-04-06T22:18:27Z

‎Operator Error

Dshr: /* Internal Attack */

2013-12-05T17:55:29Z

‎Internal Attack

Dshr: /* Internal Attack */

2013-10-05T14:12:53Z

‎Internal Attack

Dshr at 18:36, 30 September 2013

2013-09-30T18:36:38Z

Dshr: Initial version

2013-09-28T15:21:05Z

Initial version

Show changes

@@ Line 53: / Line 53: @@
 Senior technical staff of the LOCKSS Program attend and speak at many international digital preservation conferences (see [http://www.lockss.org/news-media/talks/ LOCKSS Talks page]). They conduct leading-edge research and publish extensively (see [http://www.lockss.org/news-media/publications/ LOCKSS Publications page] and [http://blog.dshr.org Dr. David S. H. Rosenthal's blog]) in digital preservation. They have done so consistently since 2000. This in-house expertise acts in place of a subscription to a technology watch service as regards digital preservation technologies.
-Risk: Senior engineers are in demand in Silicon Valley, and these key team members could be recruited away. This risk is mitigated because there are four of them, and the fact that they are all older. Older engineers are less in demand by industry, and find Stanford's excellent benefits attractive in comparison to industry's higher salaries but benefit structures more aimed at youngsters. Staffing the LOCKSS program is a continuous process. Position descriptions and classifications adhere to Stanford University's standards..
+Risk: Senior engineers are in demand in Silicon Valley, and these key team members could be recruited away. This risk is mitigated because there are four of them, and the fact that they are all older. Older engineers are less in demand by industry, and find Stanford's excellent benefits attractive in comparison to industry's higher salaries but benefit structures more aimed at youngsters. Staffing the LOCKSS program is a continuous process. Position descriptions and classifications adhere to Stanford University's standards.
 === Operational Awareness ===
@@ Line 85: / Line 85: @@
 CLOCKSS boxes can be down for extended periods without impairing the function of the network as a whole, as the box in Tokyo was in the aftermath of the Fukushima disaster.
-The CLOCKSS archive does not maintain service contracts for its hardware. It owns the hardware even though it is located at remote sites. The architecture of the CLOCKSS network means that rapid response to outages at individual sites is not required. All CLOCKSS hardware shipped to remote sites is equipped with redundant power supplies, and undergoes extended burn-in before shipment. Experience shows that failures of hardware components other than disks are rare. CLOCKSS boxes are equipped with warm spare disks to cover for disk failures. Non-disk failures are typically handled by exchanging a complete server with one from the LOCKSS team. Thus service contracts are not economically justified.
+The CLOCKSS Archive does not maintain service contracts for its hardware. It owns the hardware even though it is located at remote sites. The architecture of the CLOCKSS network means that rapid response to outages at individual sites is not required. All CLOCKSS hardware shipped to remote sites is equipped with redundant power supplies, and undergoes extended burn-in before shipment. Experience shows that failures of hardware components other than disks are rare. CLOCKSS boxes are equipped with warm spare disks to cover for disk failures. Non-disk failures are typically handled by exchanging a complete server with one from the LOCKSS team. Thus service contracts are not economically justified.
 Risk: There is a risk that delays in repairing hardware failures would result in enough CLOCKSS boxes being down simultaneously to impair the function of the network. This risk is mitigated by [[CLOCKSS: Logging and Records|monitoring of box operations]] and treating hardware repair as urgent. The purchase of spare hardware that could immediately be shipped from Stanford to reduce the delay in repair is being investigated.
@@ Line 141: / Line 141: @@
 === Software Obsolescence ===
-All [[CLOCKSS: Hardware and Software Inventory#Software|software in use by the CLOCKSS archive]] is either:
+All [[CLOCKSS: Hardware and Software Inventory#Software|software in use by the CLOCKSS Archive]] is either:
 * free, open-source, industry standard software such as Linux and Java, or
 * internally developed free, open-source software (the LOCKSS daemon), or
 * internally developed tools used for content testing, and diagnosis of the CLOCKSS network's performance.
-The LOCKSS daemon used to preserve the CLOCKSS archive's content [[LOCKSS: Software Development Process#Dependencies|depends upon]]:
+The LOCKSS daemon used to preserve the CLOCKSS Archive's content [[LOCKSS: Software Development Process#Dependencies|depends upon]]:
 * A POSIX file system.
 * A Java virtual machine, level 6 or above.
@@ Line 151: / Line 151: @@
 Changes which prevent the Linux environment satisfying these requirements are considered unlikely in the foreseeable future, and if they were to be envisaged by the Linux community it would only be after open discussion of which the LOCKSS team would be aware (see [[CLOCKSS: Threats and Mitigations#Awareness|Awareness above]]). The LOCKSS software is maintained by the LOCKSS team using processes defined in [[LOCKSS: Software Development Process]]. LOCKSS Program technical staff monitor the evolution of the open source ecosystem and, when indicated, routinely migrate the LOCKSS software (for example, from one Java library to another deemed more suitable). The threat of obsolescence is also monitored by the testing processes described in [[LOCKSS: Software Development Process]] and [[CLOCKSS: Ingest Pipeline]]. Loss of key team members could impact the effectiveness of this; for mitigation see [[CLOCKSS: Threats and Mitigations#Awareness|Awareness above]]. The rest of the stack is maintained by the Linux, Apache and other open source communities. Since all the software is free and open-source, no financial provision other than the normal [[LOCKSS: Software Development Process]] funding need be made for its replacement or upgrade.
-One consequence of software obsolescence might be format obsolescence. The CLOCKSS archive [[LOCKSS: Format Migration|implements format migration on access]]. Doing so depends on the eventual availability of format converters, a topic discussed [[LOCKSS: Format Migration#Availability of Format Converters|here]].
+One consequence of software obsolescence might be format obsolescence. The CLOCKSS Archive [[LOCKSS: Format Migration|implements format migration on access]]. Doing so depends on the eventual availability of format converters, a topic discussed [[LOCKSS: Format Migration#Availability of Format Converters|here]].
 Risk: The risk of the open source community being unable to sustain the dependencies on the Java virtual machine, some Java libraries, and the availability of a POSIX file system, or the Apache web server, is assessed as low. These basic dependencies have been stable since the LOCKSS prototype nearly 15 years ago. The requirements development process described in [[LOCKSS: Software Development Process]] might fail to detect the need for a change from the LOCKSS community or the content being preserved. Changes in the rest of the software stack might trigger a failure of one or more dependencies of the LOCKSS daemon. The unit and functional testing processes described in [[LOCKSS: Software Development Process]] are designed to detect this. CLOCKSS Archive income from libraries and publishers might not be adequate for the work needed to adapt to new publishers and conform to the evolution of existing publishers, leading to a backlog of content to be ingested.
@@ Line 185: / Line 185: @@
 All CLOCKSS documents are preserved in the CLOCKSS Archive, and thus in each of the CLOCKSS boxes.
-It appears that the use of the <tt>ServeContent</tt> servlet to serve most of the triggered content is preventing the [http://web.archive.org/web/ Internet Archive's WayBack Machine] preserving it. We plan to investigate possible ways around this issue so that the Internet Archive would also be a re-publishing server for triggered content.
+It appears that the use of the <tt>ServeContent</tt> servlet to serve most of the triggered content is preventing the [http://web.archive.org/web/ Internet Archive's Wayback Machine] preserving it. We plan to investigate possible ways around this issue so that the Internet Archive would also be a re-publishing server for triggered content.
 Risk: Given the high risk of a natural disaster in the Bay Area futher attention is needed to maintaining critical data outside the area, not merely off-site.
@@ Line 219: / Line 219: @@
 ** read-only data collection (see [[CLOCKSS: Box Operations]])
 ** changes to the LOCKSS daemon configuration (see [[LOCKSS: Property Server Operations]] and discussion of [[CLOCKSS: Threats and Mitigations#Operator Error|Operator Error]] above)
-** changes to the LOCKSS daemon software (see [[LOCKSS: Software Development Process]]), which could introduce malicious code into the network. This risk is mitigated by the use of SourceForge's source code control system, which allows code changes to be traced to their authorized committers, and easily rescinded, code signing (CLOCKSS boxes verify the signature on all software, whether from the LOCKSS team or from the CentOS repositories,  before installing it), and the staged release process.
+** changes to the LOCKSS daemon software (see [[LOCKSS: Software Development Process]]), which could introduce malicious code into the network. This risk is mitigated by the use of GitHub's source code control system, which allows code changes to be traced to their authorized committers, and easily rescinded, code signing (CLOCKSS boxes verify the signature on all software, whether from the LOCKSS team or from the CentOS repositories,  before installing it), and the staged release process.
 With the exception of the Stanford CLOCKSS box, staff at the host institution of each CLOCKSS box have access only to their box, not to any of the others. Their role in changing the system is limited to maintaining the operating system of their box current with the requirements of the network. LOCKSS staff have read-only access to all boxes for monitoring and data collection purposes.
@@ Line 249: / Line 249: @@
 Changes to this document require:
 * Review by LOCKSS Engineering Staff
-* Approval by LOCKSS Chief Scientist
+* Approval by LOCKSS Program Manager
 == Relevant Documents ==
@@ Line 255: / Line 255: @@
 # [http://dx.doi.org/10.1145/1047915.1047917 ''LOCKSS: A Peer-to-Peer Digital Preservation System'']
 # [https://www.usenix.org/conference/2005-usenix-annual-technical-conference/attrition-defenses-peer-peer-digital-preservation ''Attrition Defenses for a Peer-to-Peer Digital Preservation System'']
-# [http://www.lockss.org/news-media/talks/ LOCKSS Talks page]
+# [https://www.lockss.org/about/resources LOCKSS papers and presentations]
-# [http://www.lockss.org/news-media/publications/ LOCKSS Publications page]
+# [https://blog.dshr.org Dr. David S. H. Rosenthal's blog]
-# [http://blog.dshr.org Dr. David S. H. Rosenthal's blog]
 # [[CLOCKSS: Box Operations]]
 # [[LOCKSS: Polling and Repair Protocol]]

@@ Line 165: / Line 165: @@
 * Distributing a syntactically malformed property file, which will be detected by the boxes and treated as a service interruption.
 * Distributing a syntactically correct property file that sets unsuitable property values. The LOCKSS daemon software is skeptical of property values. Critical properties have range checks and the code takes other defensive measures to ensure that erroneous property values can at worst cause daemon activities such as polling to stop; they cannot cause loss of or damage to content.
-An error by a software developer, or the daemon or plugin build master, could affect the entire network but the [[LOCKSS: Software Development Process|testing and release process]] is as automated as possible and designed to catch such errors before they get to the network.
+An error by a [[LOCKSS: Basic Concepts#LOCKSS Daemon|daemon]] or [[LOCKSS: Basic Concepts#Plugins|plugin]] developer, or the daemon or plugin build master, could affect the entire network but the [[LOCKSS: Software Development Process|testing and release process]] is as automated as possible and designed to catch such errors before they get to the network.
 Other precautions taken include:

@@ Line 225: / Line 225: @@
 Members of the LOCKSS staff have delineated roles, responsibilities and authorizations regarding making changes to the system as follows:
 * LOCKSS technical staff can check changes in to the daemon source code repository, but these changes are not deployed to the CLOCKSS boxes until they have been tested, included in a release candidate built and signed by the LOCKSS build master, and approved for release by the LOCKSS technical lead. The LOCKSS source code control system identifies the author of each change to the system. See [[LOCKSS: Software Development Process]].
-* LOCKSS content staff can check changes in to the plugin source, but these changes are not deployed to the CLOCKSS boxes until they have been tested, included in a plugin release built and signed by the plugin build master, and approved for release by the content lead. See [[LOCKSS: Software Development Process]].
+* LOCKSS content staff can check changes in to the plugin source, but these changes are not deployed to the CLOCKSS boxes until they have been tested, included in a plugin release built and signed by the plugin build master, and approved for release by the plugin lead. See [[LOCKSS: Software Development Process]].
 * Access to the server room containing the Stanford CLOCKSS box, the property server and other critical systems is restricted to the LOCKSS sysadmin and the LOCKSS senior engineers. Similar secure physical locations are required of the other CLOCKSS boxes, see [[CLOCKSS: Box Operations#Requirements for CLOCKSS host sites|CLOCKSS: Box Operations]].

@@ Line 216: / Line 216: @@
 Internal attack could take one of two forms:
 * Insider abuse at the CLOCKSS host institutions is limited to affecting a single box, not the preserved content as a whole. This is because each of the (currently 12) CLOCKSS boxes is independently administered; insiders at the host institution have access only to their box. The CLOCKSS boxes do not trust each other, only the consensus of the boxes as a whole.
-* Insider abuse by the LOCKSS team. The policy is that when a new CLOCKSS box is bought up, the LOCKSS staff managing the network have write and administrative access to it via <tt>sudo</tt>. All such accesses are logged. Once confidence is achieved in the working relationship with staff at the host institution, this access is terminated. This stage has been achieved with XXX/11 remote CLOCKSS boxes. Eventually, the LOCKSS staff will have such access only to the box at Stanford; their access to the other boxes is limited to:
+* Insider abuse by the LOCKSS team. The policy is that when a new CLOCKSS box is bought up, the LOCKSS staff managing the network have write and administrative access to it via <tt>sudo</tt>. All such accesses are logged. Once confidence is achieved in the working relationship with staff at the host institution, this access is terminated. This stage has been achieved with 7/11 remote CLOCKSS boxes. Eventually, the LOCKSS staff will have such access only to the box at Stanford; their access to the other boxes is limited to:
 ** read-only data collection (see [[CLOCKSS: Box Operations]])
 ** changes to the LOCKSS daemon configuration (see [[LOCKSS: Property Server Operations]] and discussion of [[CLOCKSS: Threats and Mitigations#Operator Error|Operator Error]] above)

@@ Line 124: / Line 124: @@
 * Easy to monitor for obsolescence, since that would be an industry-wide event.
 * Easily replaced with newer physical or virtual resources when necessary.
-The LOCKSS team monitors the state of the [[CLOCKSS: Hardware and Software Inventory#Hardware|hardware inventory]] as documented in [[CLOCKSS: Logging and Records]]. The CLOCKSS archive has [[CLOCKSS: Box Operations#Hardware Replacement|technical]] and [[CLOCKSS; Budget and Planning Process|financial]] plans in place to replace failed or life-expired hardware.
+The LOCKSS team monitors the state of the [[CLOCKSS: Hardware and Software Inventory#Hardware|hardware inventory]] as documented in [[CLOCKSS: Logging and Records]]. The CLOCKSS archive has [[CLOCKSS: Box Operations#Hardware Replacement|technical]] and [[CLOCKSS: Budget and Planning Process|financial]] plans in place to replace failed or life-expired hardware.
 There are three reasons why ingest machines or CLOCKSS boxes might need to be replaced: