Definition of SIP - Revision history

Ntay: restoring post-2016 edits

2019-08-14T23:34:56Z

restoring post-2016 edits

Dshr: /* Creating AIPs from SIPs */

2014-04-05T19:30:07Z

‎Creating AIPs from SIPs

Dshr: Initial version

2013-09-28T03:40:06Z

Initial version

New page

= CLOCKSS Definition of SIP =

== OAIS Submission Information Package (SIP) ==

The OAIS definition of a SIP is:
<blockquote>Submission Information Package (SIP): An Information Package that is delivered by the
Producer to the OAIS for use in the construction or update of one or more AIPs and/or the
associated Descriptive Information.</blockquote>
Other relevant definitions are:
<blockquote>Data Submission Session: A delivery of media or a single telecommunications session that
provides Data to an OAIS. The Data Submission Session format/contents is based on a data
model negotiated between the OAIS and the Producer in the Submission Agreement. This
data model identifies the logical constructs used by the Producer and how they are
represented on each media delivery or in the telecommunication session.</blockquote>
<blockquote>Submission Agreement: The agreement reached between an OAIS and the Producer that
specifies a data model, and any other arrangements needed, for the Data Submission Session.
This data model identifies format/contents and the logical constructs used by the Producer
and how they are represented on each media delivery or in a telecommunication session.</blockquote>
<blockquote>Ingest Functional Entity: The OAIS functional entity that contains the services and
functions that accept Submission Information Packages from Producers, prepares Archival
Information Packages for storage, and ensures that Archival Information Packages and their
supporting Descriptive Information become established within the OAIS.</blockquote>
The discussion of SIPs in OAIS is:
<blockquote>The Submission Information Package (SIP) is that package that is sent to an OAIS by a
Producer. Its form and detailed content are typically negotiated between the Producer and
the OAIS (see related standards in 1.5). Most SIPs will have some Content Information and
some PDI.</blockquote>
<blockquote>The relationships between SIPs and AIPs can be complex; as well as a simple one-to-one
relationship in which one SIP produces one AIP, other possibilities include: one AIP being
produced from multiple SIPs produced at different times by one Producer or by many
Producers; one SIP resulting in a number of AIPs; and many SIPs from one or more sources
being unbundled and recombined in different ways to produce many AIPs. Even in the first
case, the OAIS may have to perform a number of transformations on the SIP. The Packaging
Information will always be present in some form.</blockquote>

== CLOCKSS Submission Information Package (SIP) ==

The majority of the content the CLOCKSS archive is chartered to preserve is electronic journals, which are serials. Serials publishers (Producers in OAIS terminology) emit a continuous stream of articles through time, together in most cases with metadata that organizes the articles into a logical structure of issues, volumes and journals. Decisions to trigger content from the CLOCKSS archive will normally be taken on at a journal, or perhaps at a range of volumes, granularity. It is thus important to preserve this logical structure; the CLOCKSS archive does so by organizing content into Archival Units (AUs) which generally correspond to a volume or a year of a journal, but in some cases may correspond to a year of multiple journals. An AU, together with some related information, forms the CLOCKSS archive's AIP.

Publishers submitting content to the CLOCKSS archive choose one of two ways to do so:
* Harvest, in which the CLOCKSS archive collects the content the publisher supplies to readers from their web site shortly after it is published (See [[CLOCKSS: Ingest Pipeline#Harvest Content Pipeline|CLOCKSS; Ingest Pipeline]]).
* File transfer, in which the publisher packages up content and metadata in a form they define and arranges for the packages to be transferred to the CLOCKSS archive at a time of the publisher's choosing.
Comparing these with OAIS' SIP definitions above, some important aspects are evident:
* SIPs are a structure that the archive imposes on what from the publisher's point of view is a continuous stream of content. SIP is not a useful concept in communicating with publishers.
* In neither case is the form of content as it is transferred to the archive defined by the archive. It is defined by the publisher; the archive has very limited ability to affect this definition during negotiation of the Submission Agreement (see [https://www.clockss.org/clocksswiki/files/CLOCKSS_Participating_Publisher_Agreement.pdf CLOCKSS: Publisher Agreement]).
* In both cases the content is constructed by the publisher.
* In neither case is the content transferred a completed logical unit of content. What is transferred is the set of articles published since the last transfer. This will in almost all cases form only part of a logical unit, for example a volume of a journal, and will in some cases form part of multiple logical units. Examples are transfers that span the end of one volume and the start of its successor, and file transfers from large publishers which typically include articles from many journals. Thus CLOCKSS SIPs are "for use in the construction or update of one or more AIPs".
* In neither case is a completed logical unit of content, or an entire SIP, transferred at a single point in time:
** For harvested content, the timing of the transfer is under the control of the archive. The CLOCKSS archive cannot wait until the successor of a volume has started then ingest the articles the older volume contains as a completed unit because:
*** Doing so places content published early in a volume at additional risk.
*** Publishers are not constrained to cease publishing content in an earlier volume once they have started publishing in its successor. Both "publish ahead of print" and corrections are examples in which this happens.
** For file transfer content, the timing of the transfer is under the control of the publisher. In general, publishers arrange for the transfer to happen shortly after publication, but they are not constrained to do so, nor are they constrained to identify the point in time at which a volume is complete.
The CLOCKSS archive is thus an example of how "The relationships between SIPs and AIPs can be complex".

Because the CLOCKSS archive provides two different ways publishers can choose to provide content for preservation, it supports two different types of SIP, for harvest and for file transfer.

=== Harvest SIP ===

Publishers choosing to have the CLOCKSS archive harvest their content normally demonstrate its logical structure by providing volume and issue table-of-contents (ToC) pages. The volume ToC acquires links to the issues as they are published; the issue ToC acquires links to the articles in that issue as they are published. The CLOCKSS archive requires two things of each publisher choosing harvest that, together with the content as it accumulates, form the SIP for that logical unit of content from that publisher:
* A volume ToC page, or some page playing an equivalent role, via whose links the content that is to form the AU can be found. This page is termed the "manifest page".
* A permission statement on the manifest page, or on a page with a known relation to the manifest page, containing either a Creative Commons license, or a statement granting CLOCKSS permission to collect and preserve the content pointed to by the manifest page.

=== File Transfer SIP ===

Publishers choosing to supply content via file transfer do so in some format of their choosing, subject to agreement by the CLOCKSS archive that the format is adequately specified and contains adequate metadata. The transferred files containing content and metadata form the SIP for that content from that publisher.

== Creating AIPs from SIPs ==

In both harvest and file transfer cases the [[CLOCKSS: Ingest Pipeline]] uses additional, internal publisher-specific information to create AUs (AIPs) from the SIPs. For details, see [[Definition of AIP]]. As described in [[CLOCKSS: Logging and Records], the CLOCKSS Executive Director receives regular reports on the progress of this process, upon which publisher billing is based.

== Content Information and Information Properties ==

=== OAIS Content Information and Information Properties ===

In the OAIS model, the SIP is a means to transfer Content Information containing Information Properties to the archive. The definitions of these terms are:
<blockquote>Content Information: A set of information that is the original target of preservation or that
includes part or all of that information. It is an Information Object composed of its Content Data
Object and its Representation Information.</blockquote>
<blockquote>Information Property: That part of the Content Information as described by the Information
Property Description. The detailed expression, or value, of that part of the information content
is conveyed by the appropriate parts of the Content Data Object and its Representation
Information.</blockquote>
Other relevant definitions are:
<blockquote>Content Data Object: The Data Object, that together with associated Representation
Information, comprises the Content Information.</blockquote>
<blockquote>Information Property Description: The description of the Information Property. It is a
description of a part of the information content of a Content Information object that is
highlighted for a particular purpose.</blockquote>
<blockquote>Representation Information: The information that maps a Data Object into more
meaningful concepts. An example of Representation Information for a bit sequence which is
a FITS file might consist of the FITS standard which defines the format plus a dictionary
which defines the meaning in the file of keywords which are not part of the standard.
Another example is JPEG software which is used to render a JPEG file; rendering the JPEG
file as bits is not very meaningful to humans but the software, which embodies an
understanding of the JPEG standard, maps the bits into pixels which can then be rendered as
an image for human viewing.</blockquote>
The discussion of Information Properties includes:
<blockquote>The
Producer may provide, or the Archive may itself define, as part of the Provenance
Information, Information Property Descriptions of Information Properties which should
be maintained over time, and indeed may provide Information Property Descriptions of
Information Properties which do not need to be maintained over time. An Information
Property is that part of the Content Information as described by the Information Property
Description. An Information Property Description is a description of a part of the information
content of a Content Information object that is highlighted for a particular purpose. The detailed
expression, or value, of that part of the information content is conveyed by the appropriate parts
of the Content Data Object and its Representation Information. For example, consider a simple
digital book which when rendered appears as pages with margins, title, chapter headings,
paragraphs, and text lines composed of words and punctuation. Information Property
Descriptions for Information Properties that must be preserved could be expressed as
‘paragraph identification’ and ‘characters expressing words and punctuation’. The
Information Properties would consist of all the book’s paragraph identifications, words, and
punctuation as expressed by the Content Data Object and its Representation Information.
This means that all formatting other than the recognition of paragraphs and readable text
could be altered while still maintaining required preservation. The Archive may express an
evaluation of the Authenticity of its holdings, based on community practice and
recommendations (including best practices, guidelines, standards, and legal requirements).
For example scientific Archives may have less stringent evaluation criteria than State
Archives; however, the Consumer may make his/her own judgment of the Authenticity
starting with the evidence obtained from PDI.</blockquote>

=== CLOCKSS Content Information and Information Properties ===

The OAIS definition of Information Properties is, in effect, properties of the Content Information that are, or are not, the same before and after a transformation of the content as part of a preservation operation. An example would be paragraph indentifications, which might or might not be preserved by a format migration of the preserved content.

The CLOCKSS archive does not perform content transformations as part of <i>preservation</i> operations, only as part of <i>dissemination</i> operations. The archive preserves the content it ingests in its original format and as its original bits. This content includes bibliographic and representational metadata. As described in [[LOCKSS: Extracting Bibliographic Metadata]] subsequent to ingestion the bibliographic metadata will be extracted and added to the [[LOCKSS: Metadata Database]] for ease of access and reporting. The metadata database is merely a cache of information from the preserved content. The database is not itself preserved; it can be reconstructed from the preserved content, which is unaffected by metadata extraction.

If content is ever triggered, as part of the process of generating a [[Definition of DIP|DIP]] containing it, the triggered content may be transformed from its original form to a form that is understandable by applying the Knowledge Base of the eventual Consumers. [[CLOCKSS: Extracting Triggered Content#Preparing File Transfer Content for Dissemination|CLOCKSS: Extracting Triggered Content]] describes this type of transformation as applied to file transfer content currently. The reasons for this policy are:
* The CLOCKSS archive is a dark archive. It is intended that the vast majority of its content will never be triggered. Devoting resources to transforming content that will never be triggered would be wasteful.
* [[CLOCKSS: Designated Community]] specifies that the Knowledge Base of the eventual Consumers of any triggered content includes Web browsers, or their equivalent in some successor technology to the Web. The goal is that the eventual Consumers of any triggered content are as able to understand it as readers of the publisher's Web site were at the time of publication. To ensure this, two conditions must hold:
** The original Content Information must include everything that would have been provided to a reader of the publisher's Web site at the time of publication, see [[Definition of AIP#Creating AIPs from SIPs|Definition of AIP]].
** The DIP containing the triggered Content Information must include it in a format that Web browsers, or their equivalent in some successor technology to the Web, can render successfully. If this is not the original format, a format migration will be performed during generation of the DIP, see [[LOCKSS: Format Migration#Format Migration in the CLOCKSS Archive]], but this will not affect the preserved AIP.
Thus the Information Properties that the CLOCKSS archive preserves are the entire set of Information Properties of the Content Information. No operations are performed on the preserved content that would affect those properties. It is not necessary for the CLOCKSS archive to specify individual Information Properties that are, or are not, to be preserved since all Information Properties are preserved. If a transformation is performed as part of generating a DIP, the properties of the Content Information may not be the same before and after the transformation, but this does not mean that the Information Properties have not been preserved. A later, different DIP generation process might result in the properties of the Content Information being the same before and after the transformation.

== Change Process ==

Changes to this document require:
* Review by LOCKSS Engineering Staff
* Approval by LOCKSS Chief Scientist

== Relevant Documents ==
# OAIS (2012) CCSDS 650.0-M-2: Reference Model for an Open Archival Information System (OAIS). Magenta Book. Issue 1. June 2012 (ISO 14721:2003) http://public.ccsds.org/publications/archive/650x0m2.pdf accessed 2013.08.31
# [[LOCKSS: Format Migration]]
# [[Definition of AIP]]
# [[Definition of DIP]]
# [[CLOCKSS: Designated Community]]
# [[CLOCKSS: Extracting Triggered Content]]
# [[LOCKSS: Extracting Bibliographic Metadata]]
# [[LOCKSS: Metadata Database]]
# [https://www.clockss.org/clocksswiki/files/CLOCKSS_Participating_Publisher_Agreement.pdf CLOCKSS: Publisher Agreement]

@@ Line 44: / Line 44: @@
 * SIPs are a structure that the archive imposes on what from the publisher's point of view is a continuous stream of content. SIP is not a useful concept in communicating with publishers.
 * In neither case is the form of content as it is transferred to the archive defined by the archive. It is defined by the publisher; the archive has very limited ability to affect this definition during negotiation of the Submission Agreement (see [https://www.clockss.org/clocksswiki/files/CLOCKSS_Participating_Publisher_Agreement.pdf CLOCKSS: Publisher Agreement]).
-* In both cases the content is constructed by the publisher.
+* In both cases, the content is constructed by the publisher.
 * In neither case is the content transferred a completed logical unit of content. What is transferred is the set of articles published since the last transfer. This will in almost all cases form only part of a logical unit, for example a volume of a journal, and will in some cases form part of multiple logical units. Examples are transfers that span the end of one volume and the start of its successor, and file transfers from large publishers which typically include articles from many journals. Thus CLOCKSS SIPs are "for use in the construction or update of one or more AIPs".
 * In neither case is a completed logical unit of content, or an entire SIP, transferred at a single point in time:
@@ Line 57: / Line 57: @@
 === Harvest SIP ===
-Publishers choosing to have the CLOCKSS archive harvest their content normally demonstrate its logical structure by providing volume and issue table-of-contents (ToC) pages. The volume ToC acquires links to the issues as they are published; the issue ToC acquires links to the articles in that issue as they are published. The CLOCKSS archive requires two things of each publisher choosing harvest that, together with the content as it accumulates, form the SIP for that logical unit of content from that publisher:
+Publishers choosing to have the CLOCKSS Archive harvest their content normally demonstrate its logical structure by providing volume and issue table-of-contents (ToC) pages. The volume ToC acquires links to the issues as they are published; the issue ToC acquires links to the articles in that issue as they are published. The CLOCKSS Archive requires two things of each publisher choosing harvest that, together with the content as it accumulates, form the SIP for that logical unit of content from that publisher:
 * A volume ToC page, or some page playing an equivalent role, via whose links the content that is to form the AU can be found. This page is termed the "manifest page".
 * A permission statement on the manifest page, or on a page with a known relation to the manifest page, containing either a Creative Commons license, or a statement granting CLOCKSS permission to collect and preserve the content pointed to by the manifest page.
@@ Line 124: / Line 124: @@
 The OAIS definition of Information Properties is, in effect, properties of the Content Information that are, or are not, the same before and after a transformation of the content as part of a preservation operation. An example would be paragraph indentifications, which might or might not be preserved by a format migration of the preserved content.
-The CLOCKSS archive does not perform content transformations as part of <i>preservation</i> operations, only as part of <i>dissemination</i> operations. The archive preserves the content it ingests in its original format and as its original bits. This content includes bibliographic and representational metadata. As described in [[LOCKSS: Extracting Bibliographic Metadata]] subsequent to ingestion the bibliographic metadata will be extracted and added to the [[LOCKSS: Metadata Database]] for ease of access and reporting. The metadata database is merely a cache of information from the preserved content. The database is not itself preserved; it can be reconstructed from the preserved content, which is unaffected by metadata extraction.
+The CLOCKSS Archive does not perform content transformations as part of <i>preservation</i> operations, only as part of <i>dissemination</i> operations. The archive preserves the content it ingests in its original format and as its original bits. This content includes bibliographic and representational metadata. As described in [[LOCKSS: Extracting Bibliographic Metadata]] subsequent to ingestion the bibliographic metadata will be extracted and added to the [[LOCKSS: Metadata Database]] for ease of access and reporting. The metadata database is merely a cache of information from the preserved content. The database is not itself preserved; it can be reconstructed from the preserved content, which is unaffected by metadata extraction.
 If content is ever triggered, as part of the process of generating a [[Definition of DIP|DIP]] containing it, the triggered content may be transformed from its original form to a form that is understandable by applying the Knowledge Base of the eventual Consumers. [[CLOCKSS: Extracting Triggered Content#Preparing File Transfer Content for Dissemination|CLOCKSS: Extracting Triggered Content]] describes this type of transformation as applied to file transfer content currently. The reasons for this policy are:
-* The CLOCKSS archive is a dark archive. It is intended that the vast majority of its content will never be triggered. Devoting resources to transforming content that will never be triggered would be wasteful.
+* The CLOCKSS Archive is a dark archive. It is intended that the vast majority of its content will never be triggered. Devoting resources to transforming content that will never be triggered would be wasteful.
 * [[CLOCKSS: Designated Community]] specifies that the Knowledge Base of the eventual Consumers of any triggered content includes Web browsers, or their equivalent in some successor technology to the Web. The goal is that the eventual Consumers of any triggered content are as able to understand it as readers of the publisher's Web site were at the time of publication. To ensure this, two conditions must hold:
 ** The original Content Information must include everything that would have been provided to a reader of the publisher's Web site at the time of publication, see [[Definition of AIP#Creating AIPs from SIPs|Definition of AIP]].
 ** The DIP containing the triggered Content Information must include it in a format that Web browsers, or their equivalent in some successor technology to the Web, can render successfully. If this is not the original format, a format migration will be performed during generation of the DIP, see [[LOCKSS: Format Migration#Format Migration in the CLOCKSS Archive]], but this will not affect the preserved AIP.
-Thus the Information Properties that the CLOCKSS archive preserves are the entire set of Information Properties of the Content Information. No operations are performed on the preserved content that would affect those properties. It is not necessary for the CLOCKSS archive to specify individual Information Properties that are, or are not, to be preserved since all Information Properties are preserved. If a transformation is performed as part of generating a DIP, the properties of the Content Information may not be the same before and after the transformation, but this does not mean that the Information Properties have not been preserved. A later, different DIP generation process might result in the properties of the Content Information being the same before and after the transformation.
+Thus the Information Properties that the CLOCKSS Archive preserves are the entire set of Information Properties of the Content Information. No operations are performed on the preserved content that would affect those properties. It is not necessary for the CLOCKSS Archive to specify individual Information Properties that are, or are not, to be preserved since all Information Properties are preserved. If a transformation is performed as part of generating a DIP, the properties of the Content Information may not be the same before and after the transformation, but this does not mean that the Information Properties have not been preserved. A later, different DIP generation process might result in the properties of the Content Information being the same before and after the transformation.
 == Change Process ==
@@ Line 137: / Line 137: @@
 Changes to this document require:
 * Review by LOCKSS Engineering Staff
-* Approval by LOCKSS Chief Scientist
+* Approval by LOCKSS Technical Manager
 == Relevant Documents ==
-# OAIS (2012) CCSDS 650.0-M-2: Reference Model for an Open Archival Information System (OAIS). Magenta Book. Issue 1. June 2012 (ISO 14721:2003) http://public.ccsds.org/publications/archive/650x0m2.pdf accessed 2013.08.31
+# OAIS (2012) CCSDS 650.0-M-2: Reference Model for an Open Archival Information System (OAIS). Magenta Book. Issue 1. June 2012 (ISO 14721:2003) http://public.ccsds.org/publications/archive/650x0m2.pdf accessed 31 August 2013
 # [[LOCKSS: Format Migration]]
 # [[Definition of AIP]]
@@ Line 148: / Line 148: @@
 # [[LOCKSS: Extracting Bibliographic Metadata]]
 # [[LOCKSS: Metadata Database]]
-# [https://www.clockss.org/clocksswiki/files/CLOCKSS_Participating_Publisher_Agreement.pdf CLOCKSS: Publisher Agreement]
+# [https://clockss.org/wp-content/uploads/2018/10/2019-CLOCKSS-Participating-Publisher-Agreement.pdf CLOCKSS: Publisher Agreement]

@@ Line 67: / Line 67: @@
 == Creating AIPs from SIPs ==
-In both harvest and file transfer cases the [[CLOCKSS: Ingest Pipeline]] uses additional, internal publisher-specific information to create AUs (AIPs) from the SIPs. For details, see [[Definition of AIP]]. As described in [[CLOCKSS: Logging and Records], the CLOCKSS Executive Director receives regular reports on the progress of this process, upon which publisher billing is based.
+In both harvest and file transfer cases the [[CLOCKSS: Ingest Pipeline]] uses additional, internal publisher-specific information to create AUs (AIPs) from the SIPs. For details, see [[Definition of AIP]]. As described in [[CLOCKSS: Logging and Records]], the CLOCKSS Executive Director receives regular reports on the progress of this process, upon which publisher billing is based.
 == Content Information and Information Properties ==