3.1 Overview

Index building is an intrinsic part of setting up an XPAT database. Indices provide a means by which text can be searched for rapidly. Index building is actually composed of a sequence of individual index-building steps, each performed by specialized utilities. Each of these utilities has its own command line requirements. Each utility has default characteristics that can be customized or optimized by the use of command-line parameters. These parameters can be adjusted in order to exert greater control over, and to improve, the index-building process.

Before discussing the actual details of the index-building process (see Chapter 3, Detailed Index Building), we first review the functionality of the individual utilities that comprise a complete set of indices. The actual sequence of utilities used depends greatly on the type of database in use. Later, Chapter 3, Detailed Index Building, provides full details for each of the individual index-building utilities.

3.2 Components of the Index-building Process

The index building process consists of two primary and required parts. The first of these is a string index, generated by xpatbld. When using MFS databases, other XPAT utilities help to prepare the data so that xpatbld can build a string index; these additional steps are not required in consolidated databases. The xpatbld string index is used to find occurrces of words, phrases, and ranges of values. The second component part of an index building process is the region index. The concept of region indexes is covered in Chapter 2, section 2, XPAT Regions. Briefly, region indexes make it possible to find information in within specified (and demarcated) segments of the text, and even segments nested within each other. These two indices--the string index and the region index--work together to provide users and system builders with the ability to find information in document collections using XPAT. The following table provides a brief inventory of the utilities used, their relevance to a consolidated index build, an MFS index build, or both, and the files read, modified, and created.

3.3 Memory Allocation for Index Building

Memory allocation is an important part of the index-building process. It is not required; however, its proper use will increase the speed with which the indices are built. There are two simple rules for deciding how much memory to allocate:

1. Allocate as much physical RAM as possible
2. Do not allocate more RAM than you have available.

The second rule sounds rather ridiculous; however attempting to allocate more RAM than available will cause the system to "thrash." Your machine will be so busy trying to allocate physical memory via virtual memory that it will not be able to begin building the indices. For example, you may have 200Mb of data and 256Mb of RAM. The operating system takes up, say, 6Mb of RAM, leaving 250Mb for index building. If you allocate the default, which is 512Kb, the index-building process will take far longer than if you allocate, say 50Mb, or even 100Mb. In this case, you can allocate up to 250Mb. However, if you allocate 300Mb, you have exceeded the available RAM. This will cause your machine to swap physical and virtual memory back and forth, and thus it will not get around to building the indices. Consult your system administrator for information on the amount of physical RAM your machine has available.

3.4 Summary

The type of database in use determines the sequence of individual index builds. Different index-building utilities with different user-selected parameters control the indexing process. Further details of each index-building utility are available in Chapter 3, Detailed Index Building.

3.5 Miscellaneous

The table below shows a list of the files that may be either read, created, or modified by an individual index-builder utility. In addition to the filename suffix, there is also a short form notation column. These short form notations are used in various chapters in the XPAT documentation.