Metadata, GCM, and You

Migrating Data from MySQL

gcmwalker — Mon, 12 Apr 2010 08:37:42 +0000

This is a simple PHP script to write data from a CSV file into a new FOXML file for ingest into the repository. The process has been straightforward thus far. MySQL provides a quick way to export data from a table into a makeshift CSV file, and PHP’s fgetcsv function helps parse this file into a series of arrays which can be stepped through, value by value. This script has parsed the ~1000 rows of tabular data from the Hardware table into FOXML files. These files have been ingested into Fedora through the batch ingest tool available on the Java administration client.

Having this amount of records present in the repository has helped clarify how much functionality the base instance of Fedora possesses. The default search tool is fairly capable, and runs through several prominent Dublin Core fields specified in the FOXML file. Except for some notable absences, such as model number (for which there is no analogous DC term), the hardware records are as searchable and discoverable as they are on the MySQL/PHP interface. The major deficiency thus far is the lack of services associated with the RELS-EXT datastreams defined in the FOXML files. Associated these records with images and other parts is key, so this will continue to need to be developed.

Still, I have to say it feels good to migrate some plain CSV data into something a little more marked-up and meaningful!

" . "\n");
        fwrite($fh, "" . "\n");

		// Write Fedora object properties
		fwrite($fh, "" . "\n");
    	fwrite($fh, "" . "\n");
    	fwrite($fh, "" . "\n");
    	fwrite($fh, "" . "\n");
        fwrite($fh, "" . "\n");

        // Write the DC datastream
        fwrite($fh, "" . "\n");
    	fwrite($fh, "" . "\n");
        fwrite($fh, "" . "\n");

        // TODO: confirm this line is alright
        fwrite($fh, "" . "\n");

        echo " Wrote record number $row with $num fields 
\n";
        $row++;

            // Write certain array values into the new DC datastream of the FOXML file
       	    fwrite($fh, "PhysicalObject" . "\n");

       		// If it was set to 'system' before, it is marked as a system now
       		if (strcmp($data[2], "system") == 0 || strcmp($data[2], "System") == 0) {
       			fwrite($fh, "Y" . "\n");
       		} else {
       			fwrite($fh, "N" . "\n");
       		}

       		// Do not need this since Fedora applies its own upon ingest
       		// fwrite($fh, "h"   . $recordNumber . "" . "\n");

       		// 'Title' is taken from the former 'make' field
       		fwrite($fh, ""         . $data[4]     . ""      . "\n");
       	    // From the former 'manufacturer' field. Ideally, this value will be reaplced by a RELS-EXT relationship to some Agent object (a company, e.g.)
       	    fwrite($fh, ""       . $data[3]     . ""    . "\n");
       	    fwrite($fh, "". "\n");
       		// TODO: lots of scrubbing to normalize all the entries we did on the fly to the values of the 'easydeposit_metadata_hardwarecat' array in config/easydeposit.php
       		fwrite($fh, ""          . $data[2]     . ""       . "\n");
       		fwrite($fh, ""    . $data[6]     . "" . "\n");
       		fwrite($fh, ""             . $data[5]     . ""          . "\n");
       		fwrite($fh, ""       . $data[7]     . ""    . "\n");
		    fwrite($fh, " name=\"Manufacture span\"; start=0000-00-00; end=0000-00-00;" . "\n");
			fwrite($fh, ""            . $data[8]     . ""         . "\n");
		    fwrite($fh, ""                                . "\n");
 			fwrite($fh, ""    . $data[12]    . "" . "\n");

 			// all the Hardware record are "Gift" if it is filled in at all
 			if (strcmp($data[14], "Gift") == 0) {
 				fwrite($fh, "Donation" . "\n");
 			} else {
 				fwrite($fh, "" . "\n");
 			}

 			// fwrite($fh, "" . someValue . "" . "\n");
 			fwrite($fh, "" . $data[24]    . "" . "\n");
 			fwrite($fh, ""  . $data[27]    . ""  . "\n");
			fwrite($fh, ""       . $data[15]    . ""       . "\n");
		    fwrite($fh, ""                         . "\n");
		    fwrite($fh, ""                                   . "\n");
  			fwrite($fh, ""       . $data[25]    . ""       . "\n");
  			fwrite($fh, ""       . $data[19]    . ""       . "\n");
  			fwrite($fh, ""   . $data[20]    . ""   . "\n");
  			fwrite($fh, ""           . $data[23]    . " at "
  											      . $data[22]    . ""           . "\n");
			fwrite($fh, "                                   . "\n");
			fwrite($fh, "                                 . "\n");

  			fwrite($fh, "Functioning: " . $data[21] . "" . "\n");
  			fwrite($fh, "Technical: "   . $data[10] . "" . "\n");
  		    fwrite($fh, "Appearance: "                   . "\n");
  			fwrite($fh, "Condition: "   . $data[17] . "" . "\n");

 		fwrite($fh, "" . "\n");
      	fwrite($fh, "" . "\n");
      	fwrite($fh, "" . "\n");
      	fwrite($fh, "" . "\n");

        // Write the RELS-EXT datastream to point to collection:hardware
        fwrite($fh, "" . "\n");
        fwrite($fh, "" . "\n");
        fwrite($fh, "" . "\n");
        	fwrite($fh, "" . "\n");
        	fwrite($fh, "" . "\n");
        	fwrite($fh, "" . "\n");
        	fwrite($fh, "" . "\n");
        	fwrite($fh, "" . "\n");
        fwrite($fh, "" . "\n");
      	fwrite($fh, "" . "\n");
      	fwrite($fh, "" . "\n");

        // Write to close the root element
        fwrite($fh, "");

        fclose($fh); 

    } // while

    fclose($handle);

} // if
?>

Note: This script could use some cleaning up I know. I assume no one will really ever need it but once.

EasyDeposit Work

gcmwalker — Fri, 26 Mar 2010 19:24:01 +0000

EasyDeposit looks more and more promising as a straightforward way to manage the deposit interface for Fedora. I have written content models for the main categories of materials type (hardware, software, etc.). These content model objects also contain the

true

attribute elaborated on here, so they can function as “collections” in Fedora. Submitting test objects into these collections correctly gives them a RELS-EXT datastream stating they are a member of this collection, i.e.:

This is a good start I hope.

From this point two key issues stand out. The first is that EasyDeposit defaults to zipping submissions. This is appropriate when the submission is the actual data meant to be preserved, as is the case with a software submission, but is not appropriate when the submission is simply metadata. In this case a simple FOXML file is all that needs to be submitted. I am sure this can be changed somewhere.

The second is that a key feature of the repository is mapping relationships between pieces of hardware and software. Thus, a submitter needs to be able to specify such a relationship, ideally in the submission process. There needs to be a step where the user can pick from a list of relationships and previously submitted target resources. For example, some record isPartOf of x and y or hasPart a and b. Generating a list of possible relationships should be straightforward, but it is likely the user will simply have to specify manually what the target resource is.

Using SWORD for Deposit

gcmwalker — Thu, 04 Mar 2010 04:32:08 +0000

I’ve been looking for suitable front-ends for Fedora. The default installation comes with both a Java administrator client and a newer (though less full-featured) web administration tool. Developers are currently trying to transition over to the web client, adding functionality with each point release. Neither of these clients are suitable as comprehensive front-ends for Fedora however.

I identified a deposit interface as the foremost component for the front-end, and quickly went looking to the SWORD APP for a solution. SWORD (Simple web-service offering repository deposit) is a profile of the Atom Publishing Protocol (APP), a successful syndication specification for items on the web. SWORD retools the protocol a tiny bit and emphasizes the request and deposit functions over functions like delete or update. APP is a HTTP-based protocol, so of course SWORD functions over the web. The idea is that a simple protocol like this will find widespread use and acceptance, bolstering the awareness and bulk of repositories by simplifying the deposit process, all while keeping it as a remote function. For instance, there’s SWORD Facebook app and a (sample) SWORD plug-in for Microsoft Office. Ideally from either of these platforms you can send your work right away to any number of receiving repositories.

I have setup SWORD with Fedora. It runs as a web application inside Tomcat. After a few kinks everything seems to be ironed out, and it’s a little more clear how SWORD could fit into the repository as a whole. SWORD is best at depositing content, naturally, and it’s best if that content is pre-processed. The out-of-the-box demonstration client isn’t going to provide a way add metadata to your deposit.

EasyDeposit, written by Stuart Lewis, looks very promising in this area. EasyDeposit is a PHP front-end to SWORD, and allows the user to go through several steps prior to delivering the deposit. It allows the administrator to adjust, add, delete, and create steps, tailoring the process to the organization. For our purposes, we would add steps to receive metadata information about hardware, bibliographic materials, software and so on. I’ve implemented EasyDeposit on our test instance and it does safely deposit content along with some default metadata fields. Concerning the steps already present, configuration is straightforward, requiring modification of .php files. It should be easy to add in collections, or content models that define metadata and behavior requirements, and present them in these steps. Unfortunately (for me), EasyDeposit is presently at 0.1 (although it just got support for the CrossRef API), and documentation is not all there. This doesn’t put EasyDeposit completely out of the running per se, as the interface looks great, and the idea of customizing a template is pretty attractive.

More broadly, SWORD will not serve as an entire front-end solution, and the question becomes whether one wants to join a SWORD interface like EasyDeposit with other Fedora-compliant components like searching and disseminating. Alternatives to this approach are projects like Muradora and Islandora that attempt to provide more full-featured front-end. I plan to explore these options and get a better idea of the possibilities for full implementation.

Ingesting a Content Model

gcmwalker — Fri, 12 Feb 2010 04:55:06 +0000

I wanted to briefly post a Content Model for the repository. This is the “Common Metadata” Content Model, and it has been adapted from the Hydra Project. A digital object can be associated with a content model before, during or after ingestion. The content model in turn can point to a Service Definition object, which defines certain services for the digital object. Those service are in turn specifically defined in a Service Deployment object. The SDep object defines these services with the Web Services Description Language (WSDL). WSDL is an XML format for specifically detailing a web service. Documentation for Fedora 3 states that “Notably, Fedora currently only supports performing disseminations via HTTP GET.” This should be fine for our purposes, and it should make our .wsdl file pretty straightforward.

This is certainly a lot of layers involved in making Fedora really do something, but the modularity is key when you need to make changes on your server. If the port designations for your server have changed, you only need to change a few .wsdl files. If you need to assign new services to a new type of digital object, just add those service definitions to your content model. At least, I hope it’s that simple.

This content model points to one service call (info:fedora/gcm-sDef:commonMetadata). This service definition object will (through a SDep) deliver metadata common to all the objects in the repository: title, type (from the DCMI vocabulary), type (from our own categories), creation date and general description.

Finally, this content model is missing two compulsory datastreams, a basic Dublin Core datastream and an auditing one. These are added by Fedora automatically when the object is ingested.

Instance of Fedora 3.3 Up

gcmwalker — Thu, 28 Jan 2010 23:45:37 +0000

Whew. That was intense.

A bare bones installation of Fedora 3.3 is up and running on the GCM server. Address is http://10.10.24.35:8080/fedora if you’re in Goodwill’s network, authentication required. Installation was actually relatively painless, and took about 1/30 the time I spent setting up a default install of DSpace last year. I’m not sure why this is, DSpace is suppose to be the more plug and playable repository software. A key difference here: there is absolutely nothing you can do with Fedora right now; with DSpace you could start building communities and collections from the start.

Fedora is using our regular old MySQL instance to store its information. This will work fine. The next step will be to set up some basic services and become familiar with FOXML, the XML format Fedora uses to describe its digital objects. From there we can begin creating our schema and doing some simple test runs. When this is going smoothly enough, we can think more about putting in hooks into the Semantic Web (such as it stands) by pointing to elements in Dublin Core, Friend of a Friend, or other ontologies.

One final prerequisite that does need to get up as soon as possible is regular backup of the server. Our present MySQL database is mailed weekly to a couple different computers, so that data is fairly secure. But the drive running our server has no redundancy, so we’re not as secure as we should be.

Notes on the Open Archival Information System (OAIS)

gcmwalker — Mon, 28 Sep 2009 16:49:12 +0000

Back in 2002 Consultative Committee for Space Data Systems made a recommendation to the ISO for an Open Archival Information System. The recommendation has found broad acceptance and varying levels of compliance are usually elaborated upon in the digital repository software packages like DSpace. Since we want our archive to have a future as a federated or cooperating (OAIS terms) archive, and since the terminologies and concepts created in this document are widespread, I decided to take some notes on the recommendation as they relate to potential metadata elements we’ll employ.

The recommendation mostly concerns itself with the long term preservation of digital objects, although the framework incorporates metadata for physical objects as well. Broadly, OAIS defines an Information Object as a Data Object coupled with its Representation Information. The Representation Information allows a person to understand how the bits in the Data Object are to be interpreted. An example would be a TIFF file (Data Object) coupled with an ASCII document (Representation Information) detailing the headers, its compression method, etc., like here: TIFF description at Digital Preservation (The Library of Congress). Of course, one might also want Representation Information for the ASCII file, to explain how characters are interpreted in that format. OAIS terms this phenomenon recursive Representation Information and one might eventually accrue a Representation Network of such digital objects. One stops when the Knowledge Base of your Designated Community has the requisite knowledge to understand your top-most piece of Representation Information.

OAIS defines two types of Representation Information: Structure and Semantic. Structure Information describes the data format applied to the bit sequence to derive more meaningful values like characters, pixels, numbers, etc. Semantic Information describes the social meaning behind these higher values (for example that the text characters are English).

OAIS discourages using software that can access and use Data Objects as a replacement for comprehensive Representation Information. Although that would serve the end user well enough for a time, the software itself naturally poses its own obsolescence problem. Of course, the digital media we would like to preserve is mostly software itself. We may have datasets, images, scans, etc., but the majority of digital assets we hold are complete software packages. This includes operating systems, office suites, computer games, console games (on cartridges) and so on. Retrieving Representation Information for all these types of software will be a considerable and ongoing task, as most software will consist multiple file types.

On a side note, some research (or risk assessment) will need to be done as to what conditions will preservation copies and fair use copies. The Digital Millennium Copyright Act can be exceptionally restrictive with this. This issue falls into a broader topic concerning what control we have (and if it is sufficient control) for Content Information we receive for which we do not own the intellectual property rights. A Submission Agreement can iron out these details in the case of a donor, but materials coming from recycling is another issue to investigate.

Along with the Data Object and Representation Information, which together form the Content Information, OAIS specifies Preservation Description Information to ensure the comprehensibility of the former. OAIS lists four type of PDI, each one give us something to consider in regards to all our holding types (software, hardware and documents):

Provenance: custody information and processing history. Custody information is something we might want to really strive to extract from donors since it constitutes some of the history of personal and professional computing that we are trying to preserve. Other fields: revision history, license holder, registration, copyright.
Context: how the Content Information relates to information outside itself. This would be critical and quite formal if we were producing datasets, but as it stands this field seems open to a great deal of interpretation. Possible information could be use and purpose of the software, chip, computer, etc., conditions of use (where it was used, by who, etc.), and relations it may have with other materials we hold. Also manufacturing information, help files, user guides, the language of the package.
Reference: one or more unique identifiers. This should be contingent on our repository software and metadata set, and could include values like name, author/originator, version number or serial number.
Fixity: protection of the Content Information from undocumented alteration. Checksums, CRC, etc. It’s preferable to perform these operations for each file of a piece of software when we are not simply creating disk images of the media.

Wrapping around all this is the Packaging Information, which explains, identifies and relates the Content Information and PDI. I believe this is where we would store critical information on media-dependent attributes for our software such as tape block sizes, CD-ROM volume information, floppy block sizes and various filesystem information. This is vital information that needs to be preserved at every step, since it constitutes the history of information storage, the conditions of the initial write and so on. Along with a disk image there should be concrete information on these aspects of the original media. In addition if we have provided another compression/packaging layer on top of this, that information would be located here. For example if we had compressed our Content Information and PDI into a tarball, the Packaging Information would explain the .tar format.

The Content Information and PDI constitute the Archival Information Package (AIP), which should contain all needed information to allow Long Term Preservation. OAIS also specifies Archival Information Collections, which we may use in the case of donor collections.

OAIS also details four migration types we will want to be aware of:

Refreshment: simply copying a media instance from its original media to a new, identical piece of media. This would be when a floppy is getting old, looking old, etc., or when we are transferring data to a new hard drive (as routine backup or not). None of the infrastructure that points to AIPs needs to change, and all bits are preserved.
Replication: a transfer to a new media type where the Packaging Information, Content Information and PDI are exactly preserved. The infrastructure may need to be adjusted to point to a new location.
Repackaging: There are bits changed in the Packaging Information as a result of a transfer because new files and directories were created (though they may share the same structure and names of their originals).
Transformation: Bits are changed in the Content Information or PDI; the information content is preserved (supposedly).

Providing that we use disk images, and do not attempt to move a software’s individual files and directory structure to new media “by hand”, we should be able to stay in the safe realms of Refreshment and Replication.

The recommendation covers a great deal more in submission protocols, dissemination, management, policy and materials control, but I what’s covered here I feel is most relevant to keep in mind when designing the metadata schema for our holdings.

Interview Notes

gcmwalker — Wed, 09 Sep 2009 20:36:23 +0000

Future use of GCM’s digital archive can be divided into internal and external realms. Internally the archive will serve as a database of holdings. Inventory reports, audits, and storage organization are likely to be the primary purposes for internal reference. Since the museum is seeking accreditation by the American Association of Museums, the archive’s metadata should accomadate this goal. An area of the accreditation that falls into the scope of the database aspect of the archive is specified in AAM’s Commission’s Expectations Regarding Collections Stewardship [.pdf] document, which requires:

a system of documentation, records management, and inventory is in effect to describe each object and its acquisition (permanent or temporary), current condition and location, and movement into, out of, and within the museum

Note that an “object” is here defined as “materials used to communicate and motivate learning and instruments for carrying out the museum’s stated purpose,” so the database should track accessioned and non-accessioned materials. The latter is termed “ephemera” in our present database.

In the external realm, one foresees museum members and researchers availing themselves of an online GCM digital archive. This might be for general interest or for a specific project or research topic. Possible documents of interest would be detailed photos of machines and software, manufacturing information such as the number of machines produced in a certain model line and original price points, design documents (both institutional and individual) for machines and software, and manual texts. Where possible one will want to link to official specifications of software and hardware (filesystem specifications, manufacturer’s notes, etc.).

It’s not foreseen that internal components of hardware will receive extensive metadata as a priority. In this category are motherboards, sounds cards, network cards, hard drive, disk drives, etc. In the case of complete computer systems information on the internal components is readily available elsewhere. This leaves little reason to tear down the system and verify internal components unless doing so yielded particularly valuable information. In the case of incomplete or non-standard computer systems (systems custom-built, upgraded or downgraded) documentation of internal components might occur where those components deviate from the manufacturer’s norm.

Russ discussed some of the difficulty in labeling a manufacturer’s different model lines of hardware. For instance the silicon rails on chips can differ by millimeters or nanometers. When a manufacturer is able to tighten the distance between rails, a new model or version of a chip may go into production. Such a change is recorded by the manufacturer, but it may be quite difficult for the museum to measure this and thus record that piece of manufacturing data. The issue of what manufacturing specifications should be recorded into the the museum’s database is a key issue that will need to be resolved.

PAWN and Producers

gcmwalker — Sat, 05 Sep 2009 21:31:40 +0000

The Producer-Archive Workflow Network (PAWN) is a platform for handling the ingestion of artifacts into a long term digital repository like Fedora of DSpace. As such it focuses on the Producer-Archive interaction portion described in the Open Archival Information Systems [pdf]. It strives for flexibility in accommodating different producer-archive relationships, most likely found in a distributed system. For example an archivist or repository manager may use PAWN to handle disparate types of data or package producers (manufacturers, individual scholars, students, etc.) who are all going to have different metadata to fill out before submitting the package for ingestion, and for which the processing may be different in regards to individual metadata elements. PAWN is part of a larger tool set being developed by ADAPT (An Approach to Digital Archiving and Preservation Technology).

PAWN seems most applicable in a distributed repository that sees submissions from a variety of different producers. It’s unlikely the Goodwill Computer Museum would need such flexibility for its own repository. That repository will be fairly centralized, only maintaining multiple clients in the building (and perhaps a few remotely in time). Our producers will always be staff or trained volunteers. But the project does highlight that the museum will be seeing submissions from at least two different ‘producers’: the recycling department, individual donations, and perhaps institutional donations.

The recycling department ‘producer’ effectively makes the real producer anonymous. The only exception would be provenance information found on or in the artifact itself (stickers, names in books, disk storage, etc.). Despite the presence of such information, I can’t imagine the museum would be able to use it, as very likely it is outside of the recycling departments’ right to disburse such information.

Mechanisms: An Annotation

gcmwalker — Sat, 05 Sep 2009 21:16:00 +0000

Kirschenbaum, M. (2008). Mechanisms: New media and the forensic imagination. Cambridge: MIT Press.

Matthew Kirschenbaum, Associate Professor of English and Associate Director at the Maryland Institute for Technology in the Humanities (MITH), here examines digital media in the context of traditional textual studies and bibliography. Kirschenbaum presents to the reader forensic techniques for data recovery and investigation that reveal how digital media, typically assigned attributes like ephemeralness, repeatability and variability (what he terms a traditional “screen essentialism” attitude about digital media), actually fulfills traditional bibliographic requirements of individualism, provenance and inscription.

Central to understanding these qualities of new digital media is an understanding of the affordances and technical mechanics of the dominant storage device for the last twenty or so years: the magnetic hard disk drive. Kirschenbaum reveals how data inscription on these devices (the magnetic fluxes inscribed on the drive’s multiple platters) can identify past events and previous inscriptions in a discrete spatial territory, much like the clues traditionally found textual scholars. The author makes a distinction between this forensic materiality and the more familiar formal materiality of digital media: its carefully controlled and highly engineered behavior we see on the screen. The author elaborates on how software engineering and extensive error checking at every level of the computer works to migrate magnetic fluxes to actual human-readable documents on the screen. Even at the formal materiality level many bibliographic and textual details are overlooked for lack of close inspection: multiple versions, multiple operating environments, actual textual differences between works, etc.

Three case studies illuminate these topics: a forensic and textual analysis of a Mystery House disk image, a bibliographic and historic look at the multiple versions of Afternoon: A Story by Michael Joyce, and a look at the social and textual transmissions of William Gibson’s “Agrippa.”

Kirschenbaum’s central argument is that traditional characterizations of electronic texts and media (fluid, repeatable, identical, ephemeral) is insufficient for bibliographic, preservationist, and textual purposes, and that the media itself, upon closer examination, supports none of these characterizations.