OAI Harvesting System

Overview

DLXS hosts the OAIster project, as well as other OAI test portals (MODS, Aquifer, DLF). This document details how to run the harvester used for gathering records we use in these portals, and how to transform the harvested records into Bibliographic Class.

For more information on OAI and the Protocol for Metadata Harvesting (OAI-PMH), see the official site. For best practices related to OAI, see the Best Practices wiki.

To download just the harvester and transform engine, see the OAI Tools Package.

Harvester (UMHarvester)

N.B.: To use the harvester in your system, you may have to make changes to the Global Parameters located at the beginning of the UMHarvester script.

To start the harvester use ./UMHarvester from within /$DLXSROOT/bin/o/oaister/scripts/

These flags let you perform harvesting:

-i
The id of the repository. This is pulled from /$DLXSROOT/bin/o/oaister/scripts/id_URL_table.txt, a text file that lists the id and the baseURL of repositories, e.g., cogprints=http://cogprints.ecs.soton.ac.uk/perl/oai2. (Use id_URL_table.sample.txt to get started.) You can run multiple harvests at the same time by separating ids with commas.
-v
The verbs. You can specify:
- lr = ListRecords
- id = Identify
- ls = ListSets
- lf = ListMetadataFormats
When running ListRecords, if the folder that will contain the repository's record already exists, it will place the original folder in a backup place. Currently, this is set up as such:
- Records are placed in /$DLXSROOT/prep/h/harvester/[repository_id]
- Backup is in /$DLXSROOT/prep/h/harvester_other/backup/[repository_id]
Logs of ListRecords output are placed in /$DLXSROOT/bin/o/oaister/scripts/log/. The active.log is overwritten for each (set of) harvests run. The [repository_id].log is appended with successful harvests or harvesting errors.
-s
To harvest sets for a particular repository. Use the setSpec name for a particular set.
-f
To harvest records in the following metadata formats:
- mods
- oai_marc
- marc21
- marc21a
- marc21b
- marcxml
The harvester will harvest in oai_dc format, unless a metadata format is specified.
-n
To harvest records from the last harvest date for a repository (i.e., incremental harvesting). This flag checks the datestamp granularity from the Identify response and starts harvesting either the next second after the last harvest finished (e.g., last harvest finished 2005-03-15T10:45:46Z, incremental harvest starts at 2005-03-15T10:45:47Z) or the next day after the last harvest finished (e.g., last harvest finished 2005-03-15, incremental harvest starts at 205-03-16). The flag creates a tar.gz copy of the repository's records and puts that in the backup directory. As the incremental harvest runs, it checks the contents of the repository's directory and replaces those records that have been modified. If it finds no replacement for a harvested record, it places this in a directory specific to the incremental harvest date (e.g., cogprints1-1000_2007-03-28).
-a
To retry a harvest following a timeout. This flag will wait 5 minutes to retry a timeout on a particular resumptionToken. The limit is 10 retries for a particular harvest.
Examples:
- ./UMHarvester -i auburn,epsilondiss,rdn -v ls
- ./UMHarvester -i cogprints -v id
- ./UMHarvester -i cogprints -v id
- ./UMHarvester -i uiucimages -v lr -s ALA
- ./UMHarvester -i lcoa1 -v lr -f mods
- ./UMHarvester -i forex -v lr -n
- ./UMHarvester -i CCSDthesis -v lr -a

The Batch_UMHarvest file is used to run automated incremental harvests on repositories. See the /$DLXSROOT/bin/o/oaister/scripts/Batch_UMHarvest_sample file for an example.

my @Monday = 
(['uiucimages', 'ALA', 'oai_dc', 'dr'],
);

Add your own repository id, set, metadata format, and run specification (r to run, dr to not run OAITransform) for each repository you wish to batch harvest. Batch_UMHarvest will perform an incremental harvest from the last time you harvested, based on the .log file for that repository id.

Rename Batch_UMHarvest_sample to Batch_UMHarvest to use. To start the Batch_UMHarvest run
./Batch_UMHarvest -d M &
from within /$DLXSROOT/bin/o/oaister/scripts/. This will run all the repository ids within the "M" (or Monday) batch harvest group.

Transform engine (OAITransform)

OAITransform creates concatenated BibClass file of all oai_dc records, per repository. To start the transform tool use ./oaitransform/OAITransform [repository_id] from within /$DLXSROOT/bin/o/oaister/oaitransform/

Add the repository id you want to transform. This id is taken from repository_table.txt, which you will build using repository_table.sample.txt as your starting point.
e.g.,./oaitransform/OAITransform celebration

The transform program will process your oai_dc harvested files, first by concatenating them into raw files and then by transforming them into BibClass files. The /$DLXSROOT/bin/o/oaister/oaitransform/oai-bibclass3.xsl file is used to perform the mapping from oai_dc to BibClass.

The repository report at the end of the transform will provide a number of statistics.

Repository Report: bristol
        records with URLs       = 818
        records without URLs    = 5
        repository records      = 823
        success rate            = 99.39%
        ------------------------
        data conditioning msgs? = YES!
        deleted records (.del)  = 0
        normalization errors    = 2
        raw parse failures      = 0

records with URLs: OAIster is only interested in oai_dc records with a dc:identifier beginning with http or ftp, so the transform engines only transforms those records with those dc:identifiers.
records without URLs: The remainder of the records.
repository records: All the oai_dc records harvested.
data condtioning msgs: If there are character errors during transformation, these are fixed by OAITransform. See an explanation of the data conditioning that can be performed. To see character errors that have been fixed after a transformation, see /$DLXSROOT/bin/o/oaister/errors/utf8_status_log.txt
deleted records (.del): Not used unless you want to re-write the harvester to mark deleted records with a .del extension.
normalization errors: The transform tool uses the /$DLXSROOT/bin/o/oaister/oaitransform/normal_types.txt file to normalize the dc:type field values into five distinct BibClass TYPE values, i.e., text, image, audio, video, dataset. If the values in the dc:type fields can't be normalized because there are not mappings for them, these will be logged to an error file located at /$DLXSROOT/bin/o/oaister/errors/normalization_errors.txt
raw parse failures: If there are encoding errors which the transform tool cannot fix, these will be indicated during the transform.

For questions on how to transform MODS records, please contact Kat Hagedorn at khage at umich dot edu.

Contents

Overview

Harvester (UMHarvester)

Transform engine (OAITransform)