File-based storage of Digital Objects and constituent datastreams: XMLtapes and Internet Archive ARC files

TL;DR

This paper presents a file-based storage method for Digital Objects using XMLtapes and ARC files, enabling efficient access and management of complex digital content through protocol-based mechanisms.

Contribution

It introduces a novel combined storage approach using XMLtapes and ARC files, with indexing and referencing strategies for improved digital object management.

Findings

Efficient storage of Digital Objects with concatenated XML representations.

Indexing enables protocol-based access via OAI-PMH and OpenURL.

Interconnection of XMLtapes and ARC files supports complex digital object retrieval.

Abstract

This paper introduces the write-once/read-many XMLtape/ARC storage approach for Digital Objects and their constituent datastreams. The approach combines two interconnected file-based storage mechanisms that are made accessible in a protocol-based manner. First, XML-based representations of multiple Digital Objects are concatenated into a single file named an XMLtape. An XMLtape is a valid XML file; its format definition is independent of the choice of the XML-based complex object format by which Digital Objects are represented. The creation of indexes for both the identifier and the creation datetime of the XML-based representation of the Digital Objects facilitates OAI-PMH-based access to Digital Objects stored in an XMLtape. Second, ARC files, as introduced by the Internet Archive, are used to contain the constituent datastreams of the Digital Objects in a concatenated manner. An index for the identifier of the datastream facilitates OpenURL-based access to an ARC file. The interconnection between XMLtapes and ARC files is provided by conveying the identifiers of ARC files associated with an XMLtape as administrative information in the XMLtape, and by including OpenURL references to constituent datastreams of a Digital Object in the XML-based representation of that Digital Object.