CLOCKSS: Logging and Records - Revision history

Ntay: restoring post-2016 edits

2019-08-14T22:04:21Z

restoring post-2016 edits

Dshr: Changes after auditor visit: approved by Tom Lipkis

2014-04-10T03:33:36Z

Changes after auditor visit: approved by Tom Lipkis

Show changes

Dshr: /* Retention Policy */ fix format glitch

2014-03-12T04:30:03Z

‎Retention Policy: fix format glitch

Dshr: /* External Reports */ caveat about KBART for CLOCKSS

2014-01-09T22:43:41Z

‎External Reports: caveat about KBART for CLOCKSS

Dshr: /* Change Process */

2013-12-07T22:44:19Z

‎Change Process

Dshr: /* Alert Monitoring */

2013-12-07T22:43:53Z

‎Alert Monitoring

Dshr: /* Log Monitoring */

2013-12-07T22:43:24Z

‎Log Monitoring

Dshr: Initial version

2013-09-30T02:44:16Z

Initial version

New page

= CLOCKSS: Logging and Records =

The CLOCKSS system uses three types of record:
* '''Logs:''' detailed logs, at an extensively customizable level of the operations of the LOCKSS daemon written to log files on the host machine. The purpose of these logs is to enable diagnosis of problems that arise. Logs are retained on the machine that generated them in <tt>/var/log</tt>.
* '''Alerts:''' messages sent off-machine by the LOCKSS daemon when significant events occur. The purpose of Alerts is to draw attention to potential problems that may need diagnosis. Alerts are sent via e-mail to the <<tt>clockss-alerts@clockss.org</tt>> mail alias, and added to the log files on the host machine via the <tt>syslog</tt> mechanism.
* '''Records:''' statistical summaries and business records of the operation of the system as a whole, not of individual boxes. They are provided to the CLOCKSS board and CLOCKSS member organizations, and electronic copies are being stored in a system run by the Executive Director.

== Retention Policy ==

Although the LOCKSS daemon can generate extremely detailed logs, doing so routinely is counter-productive. It buries the signal in the noise. The goal of the logging and record policy, in the absence of a specific problem to diagnose, is to:
* Generate Logs adequate to, and retain them long enough to, enable simple diagnosis.
* Generate Alerts on any condition that the daemon determines is anomalous, and on other significant events, with sufficient detail to draw the system administrator's attention to problems requiring diagnosis, and to retain them indefinitely.
* Generate the Records needed for business and governance, and for monitoring of the CLOCKSS network's overall performance, and to retain them indefinitely.
Specific log retention policies for each CLOCKSS box are specified in <tt>/etc/logrotate.conf</tt> and the files in <tt>/etc/logrotate.d/</tt>. On each CLOCKSS Box:
** System logs are retained for a month.
** At least the most recent 20MB of LOCKSS daemon log data is retained.

== Ingest Alerts ==

An Alert is generated at the end of each crawl of a [[Definition of AIP#Creating AIPs from SIPs|SIP]] that meets certain criteria recording the final status of the crawl, and the number of HTTP 200 results obtained (this is equivalent to the number of new URLs which were found, plus the number of existing URLs that were found to have modified content). An example of such an alert:
<pre>
Date: Sat 19 Feb 2011 04:17:24 PST
From: LOCKSS box ingest2.clockss.org <clockss-alert@lockss.org>
Subject: [lockss-alert] LOCKSS box info: CrawlEnd

LOCKSS box 'ingest2.clockss.org' raised an alert at Sat Feb 19 04:12:24 PST 2011

Name: CrawlEnd
Severity: info
AU: Nature Reviews Genetics Volume 11
Explanation: Crawl ended successfully: 2276 new files

Crawl ended successfully, 2276 new files, 4 warnings.
</pre>

== Preservation Alerts ==

An Alert is generated at the end of each [[LOCKSS: Polling and Repair Protocol|poll]] that detects an integrity problem:
* If there were a non-zero number of URLs for which:
** A repair was needed because the content failed to match the consensus.
** Repair content was fetched.
** The repair content matched the consensus.
* If there were a non-zero number of URL version newly flagged as suspect because their content failed to match the locally stored hash.
An example of such an Alert:
<pre>
Date: Sat Jul 20 2013 04:17:24 PST
From: LOCKSS box ingest2.clockss.org <clockss-alert@lockss.org>
Subject: [lockss-alert] LOCKSS box info: PollEnd

LOCKSS box 'ingest2.clockss.org' raised an alert at Sat Jul 20 04:12:24 PST 2013

Name: PollEnd
Severity: info
AU: Nature Reviews Genetics Volume 11
Explanation: Poll ended successfully: 99.89% agreement

Poll ended successfully, 2866 URLs, 99.89% agreement, 3 suspect files found.
</pre>

== Dissemination Alerts ==

The CLOCKSS archive is a dark archive; access to the content is permitted only at the direction of the CLOCKSS board. Thus, as described in [[CLOCKSS: Box Operations]], the content access mechanisms of the LOCKSS daemon are disabled, and packet filters are used to further prevent access. Nevertheless, Alerts are generated on any access to the content in order that they may be treated as [[CLOCKSS: Logging and Records#Administrative and Security Alerts|Security Alerts]].

== Administrative and Security Alerts ==

Alerts are generated on the following administrative actions:
* Changes to the configuration files.
* Changes to the access control permissions.
* Adding or de-activating an AU.
* Enabling or disabling the content servers.
* User account added or removed or password changed.

== External Communications ==

=== Engagement ===

Engagement with harvest content publishers before ingestion is described in [[CLOCKSS: Ingest Pipeline#Harvest Publisher Engagement|CLOCKSS; Ingest Pipeline]].

Engagement with file transfer content publishers before ingestion is described in [[CLOCKSS: Ingest Pipeline#File Transfer Publisher Engagement|CLOCKSS; Ingest Pipeline]].

In all cases interactions with the publisher take place through the RT ticketing system, so they are recorded permanently.

=== External Reports ===

The technology for generating reports is being revised; the earlier technology became too inefficient as the number of articles on each box grew because it generated reports on each box from the [[LOCKSS: Metadata Database]] then merged them. The new technology is a centralized database with a row for each article, a column for each of the production and ingest boxes, and the cell containing the ingest timestamp of the article on that box, obtained by a regular polling process that asks each box for the articles ingested since the last time it was asked.

The following reports are generated for external consumption:
* Monthly reports of the state of preservation of all serials committed to preservation in the CLOCKSS archive are delivered to the CLOCKSS board, the Keepers Registry and posted [http://www.clockss.org/keepers/ on the Web].
* KBART reports, used to update link resolver knowledge bases, are generated monthly and posted [http://www.clockss.org/kbart/ on the Web].
* The CLOCKSS Executive Director is sent an e-mail report of the article counts in the CLOCKSS archive weekly. These reports are preserved in Stanford's backup system.
* The CLOCKSS archive charges publishers a small fee for each current article ingested, billed quarterly. Thus a quarterly report is generated showing for each publisher the number of their articles ingested in that quarter for each publication year. The report is submitted to the CLOCKSS Executive Director for onward transmission to the publishers. Significant discrepancies between this and the publisher's own article counts will result (and have resulted) in investigation and corrective action. To aid in this process more detailed reports, down to the article level, can be generated on request.

The CLOCKSS Metadata Lead is responsible for the production and dissemination of these reports.

== Monitoring ==

=== Log Monitoring ===

* Ingest boxes: The CLOCKSS Content Lead is responsible for monitoring logs on the ingest boxes
* Production boxes: The LOCKSS Technical Lead is responsible for monitoring logs on production boxes when needed.
* Web servers: The CLOCKSS Network Administrator is responsible for monitoring web server logs.

=== Alert Monitoring ===

The LOCKSS Technical Lead is responsible for monitoring the Alerts generated by CLOCKSS boxes.

== Nagios ==

The state of the CLOCKSS infrastructure, including the CLOCKSS boxes and the ingest machines, is monitored by Nagios as described in [[CLOCKSS: Box Operations#Nagios|CLOCKSS: Box Operations]].

The CLOCKSS Network Administrator is responsible for monitoring via Nagios.

== Network Diagnostics ==

The LOCKSS team's internal monitoring and evaluation processes identified some areas in which the efficiency of the polling process could be improved in the context of the Global LOCKSS Network (GLN). The Andrew W. Mellon Foundation funded work to implement and evaluate improvements in these areas. This is expected to be complete by March 2014. Although these improvements will be deployed to the CLOCKSS network, because there are many fewer boxes in the CLOCKSS network than the GLN the areas of inefficiency are not relevant to the CLOCKSS network. Thus the improvements are not expected to make a substantial difference to the performance of the CLOCKSS network.

The Mellon-funded work included development of improved instrumentation and analysis software, which polls the administrative Web UI of each LOCKSS box in a network to collect vast amounts of data about the operations of each box. For examples of the use of this software, see [[LOCKSS: Polling and Repair Protocol#Enhancements|LOCKSS: Polling and Repair Protocol]].

The CLOCKSS Network Administrator is responsible for collecting and analyzing this data.

== Change Process ==

Changes to this document require:
* Review by:
** LOCKSS Engineering Staff
** CLOCKSS Network Administrator
* Approval by LOCKSS Technical Lead

== Relevant Documents ==
# [[CLOCKSS: Box Operations]]
# [[CLOCKSS: Ingest Pipeline]]
# [[LOCKSS: Polling and Repair Protocol]]
# [[Definition of AIP]]

@@ Line 173: / Line 173: @@
 === External Reports ===
-The technology for generating reports is being revised; the earlier technology became too inefficient as the number of articles on each box grew because it generated reports on each box from the [[LOCKSS: Metadata Database]] then merged them. The new technology is a centralized database with a row for each article, a column for each of the production and ingest boxes, and the cell containing the ingest timestamp of the article on that box, obtained by a regular polling process that asks each box for the articles ingested since the last time it was asked.
+The technology for generating reports is being revised; the current technology is becoming too inefficient as the number of articles on each box grows because it generates reports on each box from the [[LOCKSS: Metadata Database]] then merges them. The new technology is a composite database with data synchronized from one or more preservation boxes' metadata databases. In addition to tracking individual article metadata, the consolidated database will track a per-machine ingest timestamp of the article on that box and will support de-duplication based on explicitly-defined rules.
 The following reports are generated for external consumption:
@@ Line 179: / Line 179: @@
 * KBART reports are generated monthly and posted [http://www.clockss.org/kbart/ on the Web]. For the Global LOCKSS Network, these reports are used to update link resolver knowledge bases so that libraries can provide their readers access to the content of their LOCKSS box. Because the CLOCKSS archive is a dark archive, these reports cannot be used to update link resolvers. However, several analysis tools use KBART as an input format, so the KBART reports for CLOCKSS are made public.
 * The CLOCKSS Executive Director is sent an e-mail report of the article counts in the CLOCKSS archive weekly. These reports are preserved in Stanford's backup system.
-* The CLOCKSS archive charges publishers a small fee for each current article ingested, billed quarterly. Thus a quarterly report is generated showing for each publisher the number of their articles ingested in that quarter for each publication year. The report is submitted to the CLOCKSS Executive Director for onward transmission to the publishers. Significant discrepancies between this and the publisher's own article counts will result (and have resulted) in investigation and corrective action. To aid in this process more detailed reports, down to the article level, can be generated on request.
+* The CLOCKSS Archive charges publishers a small fee for each current article ingested, billed quarterly. Thus a semi-annual report is generated showing for each publisher the number of their articles ingested in that quarter for each publication year. The report is submitted to the CLOCKSS Executive Director for onward transmission to the publishers. Significant discrepancies between this and the publisher's own article counts will result (and have resulted) in investigation and corrective action. To aid in this process more detailed reports, down to the article level, can be generated on request.
 The CLOCKSS Metadata Lead is responsible for the production and dissemination of these reports.
@@ Line 187: / Line 187: @@
 === Log Monitoring ===
-* Ingest boxes: The CLOCKSS Content Lead is responsible for monitoring logs on the ingest boxes
+* Ingest boxes: The CLOCKSS Technical Lead is responsible for monitoring logs on the ingest boxes
 * Production boxes: The CLOCKSS Technical Lead is responsible for monitoring logs on production boxes when needed.
 * Web servers: The CLOCKSS Network Administrator is responsible for monitoring web server logs.
@@ Line 214: / Line 214: @@
 * Review by:
 ** LOCKSS Engineering Staff
-** CLOCKSS Network Administrator
+** CLOCKSS Technical Lead
-* Approval by CLOCKSS Technical Lead
+* Approval by CLOCKSS Network Administrator
 == Relevant Documents ==

@@ Line 13: / Line 13: @@
 * Generate the Records needed for business and governance, and for monitoring of the CLOCKSS network's overall performance, and to retain them indefinitely.
 Specific log retention policies for each CLOCKSS box are specified in <tt>/etc/logrotate.conf</tt> and the files in <tt>/etc/logrotate.d/</tt>. On each CLOCKSS Box:
-** System logs are retained for a month.
+* System logs are retained for a month.
-** At least the most recent 20MB of LOCKSS daemon log data is retained.
+* At least the most recent 20MB of LOCKSS daemon log data is retained.
 == Ingest Alerts ==

@@ Line 103: / Line 103: @@
 === Alert Monitoring ===
-The LOCKSS Technical Lead is responsible for monitoring the Alerts generated by CLOCKSS boxes.
+The CLOCKSS Technical Lead is responsible for monitoring the Alerts generated by CLOCKSS boxes.
 == Nagios ==