LOCKSS: Polling and Repair Protocol - Revision history

Ntay: restoring post-2016 edits

2019-08-14T23:42:53Z

restoring post-2016 edits

Dshr: Reviewed TAL, approved DSHR

2015-07-21T05:30:19Z

Reviewed TAL, approved DSHR

Dshr: Update for subversion

2015-07-21T05:27:51Z

Update for subversion

Dshr: /* Demonstration */ Add Demo instructions - Approved by DSHR

2014-08-13T14:37:14Z

‎Demonstration: Add Demo instructions - Approved by DSHR

Show changes

Dshr: /* Enhancements */ Add Demonstration section - approved by DSHR

2014-07-21T17:40:53Z

‎Enhancements: Add Demonstration section - approved by DSHR

Dshr: Changes after auditor visit: approved by David Rosenthal

2014-04-10T03:30:19Z

Changes after auditor visit: approved by David Rosenthal

Dshr: Initial version

2013-09-26T23:32:00Z

Initial version

Dshr: Initial version

2013-09-26T23:24:54Z

Initial version

New page

= LOCKSS: Polling and Repair Protocol =

As described in [[CLOCKSS: Box Operations]] the CLOCKSS boxes are configured to form a Private LOCKSS Network (PLN). The boxes run the LOCKSS polling and repair protocol as described in [http://dx.doi.org/10.1145/1047915.1047917 our ''ACM Transactions on Computer Systems'' paper] with the following modifications:
* Because the CLOCKSS PLN is closed network secured by SSL certificate checks at both ends of all connections, the defenses against sybil attacks, which involve the adversary creating new peer identities, are not necessary and are not implemented.
* The efficiency enhancements described below are being deployed to the CLOCKSS PLN.
The LOCKSS polling and repair protocol performs regular integrity checks on each AU at each CLOCKSS box (the ''poller'') by:
* Selecting a random sample of the other CLOCKSS boxes (the <i>voters</i>).
* Inviting the voters to participate in a <i>poll</i> on the AU.
* The poll involves the voters voting, using a procedure based on nonced cryptographic hashes, on the content of each URL in their copy of the AU.
* In tallying the votes, the poller may detect that:
** A URL it has does not match the consensus of the voters, or
** A URL that the consensus of the voters says should be present in the AU is missing from the poller's AU, or
** A URL it has does not match the checksum generated when it was stored.
* If so, it repairs the problem by:
** requesting a new copy from one of the voters that agreed with the consensus,
** then verifying that the new copy does agree with the consensus.

== Enhancements ==

The LOCKSS team's internal monitoring and evaluation processes identified some areas in which the efficiency of the polling process could be improved in the context of the Global LOCKSS Network (GLN). The [http://www.lockss.org/news-media/news/lockss-program-receives-andrew-w-mellon-foundation-grant/ Andrew W. Mellon Foundation funded work to implement and evaluate improvements] in these areas. This is expected to be complete by March 2014. Although these improvements will be deployed to the CLOCKSS network, because there are many fewer boxes in the CLOCKSS network than the GLN the areas of inefficiency are less relevant to the CLOCKSS network. Thus the improvements are not expected to make a substantial difference to the performance of the CLOCKSS network.

The Mellon-funded work included development of improved instrumentation and analysis software, which polls the administrative Web UI of each LOCKSS box in a network to collect vast amounts of data about the operations of each box. These tools were used on the CLOCKSS network for an initial 59-day period, collecting over 18M data items. The data collected has yet to be fully analyzed but initial analysis shows that the polling process among CLOCKSS boxes continues to operate satisfactorily. Some examples of the graphs generated follow.

[[File:Sample Graph 1.png|200px|thumb|center]] This graph shows the number of AU instances in CLOCKSS boxes which have reached agreement with N other CLOCKSS boxes, showing the progress AUs make after ingest as the [[LOCKSS: Polling and Repair Protocol]] identifies matching AU instances at other boxes. It will be seen that there are few AU instances in the sample with few boxes with whom they have reached agreement, and that the majority of AU instances have reached agreement with AU instances at the majority of other CLOCKSS boxes.

[[File:Sample Graph 2.png|200px|thumb|center]] This graph shows the extent of agreement among the over 40,000 completed polls in the sample. As can be seen, the overwhelming majority of the polls showed complete agreement. Polls with less than complete agreement are likely to have been caused by polling among AU instances that were still collecting content, so had different sub-sets of the URLs in an AU.

== Change Process ==

Changes to this document require:
* Review by LOCKSS Engineering Staff
* Approval by LOCKSS Chief Scientist

== Relevant Documents ==
# [[CLOCKSS: Box Operations]]
# Petros Maniatis, Mema Roussopoulos, TJ Giuli, David S.H. Rosenthal, Mary Baker, and Yanto Muliadi. “LOCKSS: A Peer-to-Peer Digital Preservation System”, ACM Transactions on Computer Systems vol. 23, no. 1, February 2005, pp. 2-50. http://dx.doi.org/10.1145/1047915.1047917 accessed 2013.8.7

@@ Line 76: / Line 76: @@
 foo@bar:~$ svn checkout svn://svn.code.sf.net/p/lockss/svn/lockss-daemon/tags/last_released_daemon lockss-daemon
 ...
 foo@bar:~$
 </pre>
 The next step is to build the LOCKSS daemon. This takes a while, there's a lot of code to build. It generates Java warnings that you should be able to ignore, but no errors. Just to be sure that everything is OK, we run the unit and functional tests on the daemon that gets built. This takes much longer, especially on the little netbook I'm using to test the instructions:

@@ Line 57: / Line 57: @@
 <pre>
 foo@bar:~$ cd
-foo@bar:~$ sudo apt-get install default-jdk ant  cvs libxml2-utils
+foo@bar:~$ sudo apt-get install default-jdk ant subversion libxml2-utils
 [sudo] password for foo:
 Reading package lists... 0%
@@ Line 74: / Line 74: @@
 The next step is to check the latest release of the LOCKSS daemon out from SourceForge:
 <pre>
-foo@bar:~$ export CVSROOT=:pserver:anonymous@lockss.cvs.sourceforge.net:/cvsroot/lockss
+foo@bar:~$ svn checkout svn://svn.code.sf.net/p/lockss/svn/lockss-daemon/tags/last_released_daemon lockss-daemon
-foo@bar:~$ cvs get -P -rlast_released_daemon lockss-daemon
-cvs checkout: Updating lockss-daemon
 ...
 foo@bar:~$
 </pre>

@@ Line 267: / Line 267: @@
 Changes to this document require:
 * Review by LOCKSS Engineering Staff
-* Approval by LOCKSS Chief Scientist
+* Approval by LOCKSS Technical Manager
 == Relevant Documents ==
 # [[CLOCKSS: Box Operations]]
 # Petros Maniatis, Mema Roussopoulos, TJ Giuli, David S.H. Rosenthal, Mary Baker, and Yanto Muliadi. “LOCKSS: A Peer-to-Peer Digital Preservation System”, ACM Transactions on Computer Systems vol. 23, no. 1, February 2005, pp. 2-50. http://dx.doi.org/10.1145/1047915.1047917 accessed 2013.8.7

@@ Line 38: / Line 38: @@
 == Enhancements ==
-The LOCKSS team's internal monitoring and evaluation processes identified some areas in which the efficiency of the polling process could be improved in the context of the Global LOCKSS Network (GLN). The [http://www.lockss.org/news-media/news/lockss-program-receives-andrew-w-mellon-foundation-grant/ Andrew W. Mellon Foundation funded work to implement and evaluate improvements] in these areas. This is expected to be complete by March 2014. Although these improvements will be deployed to the CLOCKSS network, because there are many fewer boxes in the CLOCKSS network than the GLN the areas of inefficiency are less relevant to the CLOCKSS network. Thus the improvements are not expected to make a substantial difference to the performance of the CLOCKSS network.
+The LOCKSS team's internal monitoring and evaluation processes identified some areas in which the efficiency of the polling process could be improved in the context of the Global LOCKSS Network (GLN). The [http://www.lockss.org/news-media/news/lockss-program-receives-andrew-w-mellon-foundation-grant/ Andrew W. Mellon Foundation funded work to implement and evaluate improvements] in these areas; the grant period extends through March 2015. Although these improvements will be deployed to the CLOCKSS network, because there are many fewer boxes in the CLOCKSS network than the GLN the areas of inefficiency are less relevant to the CLOCKSS network. Thus the improvements are not expected to make a substantial difference to the performance of the CLOCKSS network.
 The Mellon-funded work included development of improved instrumentation and analysis software, which polls the administrative Web UI of each LOCKSS box in a network to collect vast amounts of data about the operations of each box. These tools were used on the CLOCKSS network for an initial 59-day period, collecting over 18M data items. The data collected has yet to be fully analyzed but initial analysis shows that the polling process among CLOCKSS boxes continues to operate satisfactorily. Some examples of the graphs generated follow.
@@ Line 45: / Line 45: @@
 [[File:Sample Graph 2.png|200px|thumb|center]] This graph shows the extent of agreement among the over 40,000 successfully completed polls in the sample. As can be seen, the overwhelming majority of the polls showed complete agreement. Polls with less than complete agreement are likely to have been caused by polling among AU instances that were still collecting content, so had different sub-sets of the URLs in an AU.
 == Change Process ==

@@ Line 3: / Line 3: @@
 == Overview ==
-As described in [[CLOCKSS: Box Operations]] the CLOCKSS boxes are configured to form a Private LOCKSS Network (PLN). The boxes run the LOCKSS polling and repair protocol as described in [http://dx.doi.org/10.1145/1047915.1047917 our ''ACM Transactions on Computer Systems'' paper] with the following modifications:
+LOCKSS boxes run the LOCKSS polling and repair protocol as described in [http://dx.doi.org/10.1145/1047915.1047917 our ''ACM Transactions on Computer Systems'' paper]. The paper describes the polling mechanism as applying to a single file; the [[LOCKSS: Basic Concepts#LOCKSS Daemon|LOCKSS daemon]] applies it to an entire [[LOCKSS: Basic Concepts#Archival Unit|Archival Unit (AU)]] of content. Each LOCKSS daemon chooses at random the next AU upon which it will use the LOCKSS polling and repair protocol to perform integrity checks. It acts as the ''poller'' to call a poll on that AU by:
-* Because the CLOCKSS PLN is closed network secured by SSL certificate checks at both ends of all connections, the defenses against sybil attacks, which involve the adversary creating new peer identities, are not necessary and are not implemented.
-* The efficiency enhancements described below are being deployed to the CLOCKSS PLN.
-The LOCKSS polling and repair protocol performs regular integrity checks on each AU at each CLOCKSS box (the ''poller'') by:
 * Selecting a random sample of the other CLOCKSS boxes (the <i>voters</i>).
-* Inviting the voters to participate in a <i>poll</i> on the AU.
+* Inviting the voters to participate in a <i>poll</i> on the AU, and sending each of them a freshly-generated random nonce ''Np''.
-* The poll involves the voters voting, using a procedure based on nonced cryptographic hashes, on the content of each URL in their copy of the AU.
+* The poll involves the voters voting by:
 * In tallying the votes, the poller may detect that:
 ** A URL it has does not match the consensus of the voters, or
@@ Line 17: / Line 25: @@
 ** requesting a new copy from one of the voters that agreed with the consensus,
 ** then verifying that the new copy does agree with the consensus.
 == Enhancements ==
@@ Line 24: / Line 42: @@
 The Mellon-funded work included development of improved instrumentation and analysis software, which polls the administrative Web UI of each LOCKSS box in a network to collect vast amounts of data about the operations of each box. These tools were used on the CLOCKSS network for an initial 59-day period, collecting over 18M data items. The data collected has yet to be fully analyzed but initial analysis shows that the polling process among CLOCKSS boxes continues to operate satisfactorily. Some examples of the graphs generated follow.
-[[File:Sample Graph 1.png|200px|thumb|center]] This graph shows the number of AU instances in CLOCKSS boxes which have reached agreement with N other CLOCKSS boxes, showing the progress AUs make after ingest as the LOCKSS: Polling and Repair Protocol identifies matching AU instances at other boxes. It will be seen that there are few AU instances in the sample with few boxes with whom they have reached agreement, and that the majority of AU instances have reached agreement with AU instances at the majority of other CLOCKSS boxes.
+[[File:hist_pr_auid_count27.png|200px|thumb|center]] This graph shows the number of AU instances in CLOCKSS boxes which have reached agreement with N other CLOCKSS boxes, showing the progress AUs make after ingest as the LOCKSS: Polling and Repair Protocol identifies matching AU instances at other boxes. It will be seen that there are few AU instances in the sample with few boxes with whom they have reached agreement, and that the majority of AU instances have reached agreement with AU instances at the majority of other CLOCKSS boxes.
-[[File:Sample Graph 2.png|200px|thumb|center]] This graph shows the extent of agreement among the over 40,000 completed polls in the sample. As can be seen, the overwhelming majority of the polls showed complete agreement. Polls with less than complete agreement are likely to have been caused by polling among AU instances that were still collecting content, so had different sub-sets of the URLs in an AU.
+[[File:Sample Graph 2.png|200px|thumb|center]] This graph shows the extent of agreement among the over 40,000 successfully completed polls in the sample. As can be seen, the overwhelming majority of the polls showed complete agreement. Polls with less than complete agreement are likely to have been caused by polling among AU instances that were still collecting content, so had different sub-sets of the URLs in an AU.
 == Change Process ==