CLOCKSS: Ingest Pipeline - Revision history

Ntay: restoring post-2016 edits

2019-08-14T17:13:48Z

restoring post-2016 edits

Dshr: Changes after auditor visit: approved by Tom Lipkis

2014-04-10T03:45:05Z

Changes after auditor visit: approved by Tom Lipkis

Show changes

Dshr: /* Completeness of Harvest Content */ fix typos

2014-02-28T21:40:51Z

‎Completeness of Harvest Content: fix typos

Dshr: /* Harvest Content Processing */ Response to Site VIsit Schedule

2014-02-19T22:11:39Z

‎Harvest Content Processing: Response to Site VIsit Schedule

Dshr: /* Harvest Content Processing */ Response to Site Visit Schedule

2014-02-19T21:46:10Z

‎Harvest Content Processing: Response to Site Visit Schedule

Dshr: /* e-Book Processing */ responsibility

2013-12-16T05:50:43Z

‎e-Book Processing: responsibility

Dshr: /* Annual Harvest Content Cycle */ responsibility

2013-12-16T05:49:59Z

‎Annual Harvest Content Cycle: responsibility

Dshr: /* Completeness of Harvest Content */ responsibility

2013-12-16T05:48:59Z

‎Completeness of Harvest Content: responsibility

Dshr: /* Harvest Process */ responsibility

2013-12-16T05:47:51Z

‎Harvest Process: responsibility

Dshr: /* Change Process */

2013-12-07T22:40:49Z

‎Change Process

@@ Line 4: / Line 4: @@
 * Harvest Content - the contents of a publisher's Web site, obtained by crawling it.
 * File Transfer content - the form of the content the publisher uses to create their Web site, obtained from the publisher typically via FTP.
 == Harvest Content Pipeline ==
@@ Line 60: / Line 61: @@
 Completeness in this context means that the content in the CLOCKSS archive faithfully reflects the content on the publisher's web site. The CLOCKSS ingest pipeline includes multiple ingest CLOCKSS boxes that individually harvest content from the publisher's website. Once collected, the ingest CLOCKSS boxes poll among themselves and repair content objects from the publisher if there is disagreement. The content testing process requires complete agreement among the ingest boxes before the content is released to the product CLOCKSS network.  As a result the content described by the CLOCKSS plugin and parameters for the [[Definition of SIP|Submission Information Package]] is preserved in the CLOCKSS PLN.
 To account for the possibility of errors in the rules and procedures of the CLOCKSS plugin, a second layer of testing is done by visually inspecting content submitted by a publisher to ensure that it functions as it did on the publisher's website, and that all content available from the publisher's website is also preserved in the CLOCKSS ingest boxes before being released to the CLOCKSS production network.
@@ Line 122: / Line 125: @@
 ** FTP from an FTP server that the publisher maintains to a CLOCKSS ingest machine.
 ** FTP by the publisher to an FTP server on a CLOCKSS ingest machine.
-** <tt>rsync</tt> between a publisher machine and a CLOCKSS ingest machine
+** <tt>rsync</tt> between a publisher machine and a CLOCKSS ingest machine.
 * The format of the file transfer content, in particular:
 ** How the ingest scripts can verify that the content received is correct, for example by checking manifests and checksums.
@@ Line 232: / Line 236: @@
 # [[Definition of SIP]]
 # [[Definition of AIP]]
-# [https://www.clockss.org/clocksswiki/files/CLOCKSS_Participating_Publisher_Agreement.pdf CLOCKSS: Publisher Agreement]
+# [https://clockss.org/wp-content/uploads/2018/10/2019-CLOCKSS-Participating-Publisher-Agreement.pdf CLOCKSS: Publisher Agreement]

@@ Line 59: / Line 59: @@
 ==== Completeness of Harvest Content ====
-Completeness in this context means that the content in the CLOCKSS archive faithfully reflects the content on the publisher's web site. The CLOCKSS ingest pipeline includes multiple ingest CLOCKSS boxes that individually harvest content from the publisher's website. Once collected, the ingest CLOCKSS boxes poll among themselves and repair content objects from the publiser if there is disagreement. The content testing process requires complete agreement among the ingest boxes before the content is released to the product CLOCKSS network.  As a result the content described by the CLOCKSS plugin and parameters for the [[Definition of SIP|Submission Information Package]] is preserved in the CLOCKSS PLN.
+Completeness in this context means that the content in the CLOCKSS archive faithfully reflects the content on the publisher's web site. The CLOCKSS ingest pipeline includes multiple ingest CLOCKSS boxes that individually harvest content from the publisher's website. Once collected, the ingest CLOCKSS boxes poll among themselves and repair content objects from the publisher if there is disagreement. The content testing process requires complete agreement among the ingest boxes before the content is released to the product CLOCKSS network.  As a result the content described by the CLOCKSS plugin and parameters for the [[Definition of SIP|Submission Information Package]] is preserved in the CLOCKSS PLN.
-To account for the possibility of errors in the rules and procedures of the CLOCKSS plugin, a second layer of testing is done by visually inspecting content submitted by a publisher to ensure that it functions as it did on the publisher's website, and all that content available from the publisher's website is also preserved in the CLOCKSS ingest boxes before being released to the CLOCKSS production network.
+To account for the possibility of errors in the rules and procedures of the CLOCKSS plugin, a second layer of testing is done by visually inspecting content submitted by a publisher to ensure that it functions as it did on the publisher's website, and that all content available from the publisher's website is also preserved in the CLOCKSS ingest boxes before being released to the CLOCKSS production network.
 The CLOCKSS Content Lead is responsible for this process.

@@ Line 36: / Line 36: @@
 * AUTest compares the results to ensure correct collection and correct operation of hash filters.
 * Metadata is extracted and checked for sanity.
-* A sample of the content is browsed by a human tester to ensure that:
+* A sample of the content is browsed by a human tester to verify the correctness of the plugin by ensuring that:
 ** all the types of files that should be collected are,
-** and that files that shouldn't be collected are properly excluded.
+** and that files that shouldn't be collected, such as advertisements or articles from previous years, are properly excluded.
 Any problems detected are addressed by modifying and testing the plugin in a development environment, or by contacting the publisher if necessary to resolve systemic site errors, then the tests are repeated on the content-testing boxes.

@@ Line 43: / Line 43: @@
 The process above is repeated on a substantial sample of new content as it becomes available (in following years/volumes, or new publications from the same publisher), in order to detect changes to the publisher's site, or the format of new titles, which require changes to the plugin.
-Once an AU has been successfully tested on the content-testing boxes, it is configured for collection on all ingest boxes.  If the plugin is new or changed it is also released to the ingest network.  The ingest boxes collect the content and run polls to detect and resolve transient collection errors.  When all the copies of an AU have come into full agreement on the ingest boxes, they are then configured on the CLOCKSS production boxes.
+Once an AU has been successfully tested on the content-testing boxes, it is configured for collection on all boxes in the ingest network.  If the plugin is new or changed it is also released to the ingest network.  Each ingest box collects the content and the network runs polls to detect and resolve transient collection errors.  When all the copies of an AU have come into full agreement on the ingest boxes, they are then configured on the network of CLOCKSS production boxes.
 Depending on the complexity and diversity of the publisher's content the "substantial sample" can be anything from quite small to the publisher's entire content. Note that the need for a "substantial sample" of content for testing means that there is a delay between the time the publisher starts adding content to the bibliographic unit represented by the AU, and the time the ingest network starts collecting it.

@@ Line 92: / Line 92: @@
 * When the plugin is deemed ready, all the publisher's books from the previous interval are configured on the ingest machines and allowed to crawl and poll. Results are monitored and corrective action is taken if necessary.
 * When a book is deemed fully processed on the ingest machines, it is configured on the production machines. The crawl and poll results are then monitored on the production machines until the book is deemed fully processed.
 == File Transfer Content Pipeline ==

← Older revision		Revision as of 05:49, 16 December 2013
Line 83:		Line 83:

	As regards subsequent errata and corrections, we assume that publishers follow the NLM ''best practice'' guidelines and at least refer and link to them in a subsequent issue. If the publisher does, they will be collected with that issue.		As regards subsequent errata and corrections, we assume that publishers follow the NLM ''best practice'' guidelines and at least refer and link to them in a subsequent issue. If the publisher does, they will be collected with that issue.
		+
		+	The CLOCKSS Content Lead is responsible for this process.

	==== e-Book Processing ====		==== e-Book Processing ====

← Older revision		Revision as of 05:48, 16 December 2013
Line 62:		Line 62:

	To account for the possibility of errors in the rules and procedures of the CLOCKSS plugin, a second layer of testing is done by visually inspecting content submitted by a publisher to ensure that it functions as it did on the publisher's website, and all that content available from the publisher's website is also preserved in the CLOCKSS ingest boxes before being released to the CLOCKSS production network.		To account for the possibility of errors in the rules and procedures of the CLOCKSS plugin, a second layer of testing is done by visually inspecting content submitted by a publisher to ensure that it functions as it did on the publisher's website, and all that content available from the publisher's website is also preserved in the CLOCKSS ingest boxes before being released to the CLOCKSS production network.
		+
		+	The CLOCKSS Content Lead is responsible for this process.

	==== Correctness of Harvest Content ====		==== Correctness of Harvest Content ====

← Older revision		Revision as of 05:47, 16 December 2013
Line 54:		Line 54:

	A harvest content AU is considered to be preserved when it has been ingested by all the production CLOCKSS boxes and at least one poll on it has been successful (see [[LOCKSS: Polling and Repair Protocol]]). The AU can then be removed from the ingest boxes. A review of the process for doing so deemed it inadequate and too manual; a replacement process is under development that will integrate with the improvements being made to [[CLOCKSS: Logging and Records#External Reports\|external report generation]].		A harvest content AU is considered to be preserved when it has been ingested by all the production CLOCKSS boxes and at least one poll on it has been successful (see [[LOCKSS: Polling and Repair Protocol]]). The AU can then be removed from the ingest boxes. A review of the process for doing so deemed it inadequate and too manual; a replacement process is under development that will integrate with the improvements being made to [[CLOCKSS: Logging and Records#External Reports\|external report generation]].
		+
		+	The CLOCKSS Content Lead is responsible for both the current and replacement processes.

	==== Completeness of Harvest Content ====		==== Completeness of Harvest Content ====

@@ Line 164: / Line 164: @@
 ** CLOCKSS Plugin Lead
 ** CLOCKSS Content Lead
-* Approval by LOCKSS Technical Lead
+* Approval by CLOCKSS Technical Lead
 == Relevant Documents ==