Metadata++: A scalable hierarchical framework for digital libraries

Metadata++ is a digital library system that we are developing to serve the needs of the United States Department of Agriculture Forest Service, the United States Department of the Interior Bureau of Land Management, and the United States Department of the Interior Fish and Wildlife Service to support natural resource managers, scientists and publics as they analyze issues and make decisions. The system provides access to institutional knowledge consisting of formal and informal agency reports and documents – including Environmental Assessments, Decision Notices, Appeal Decisions, specialist reports, and so forth. Metadata++ uses a set of hierarchically structured controlled vocabularies – with synonyms and associations – as the primary organizational framework. Users browse the hierarchy to select search terms and see the search results directly in the context of the hierarchy. In order to be useful as a digital library infrastructure, this hierarchy must be implemented in an efficient and scalable manner. This paper introduces the Metadata++ system and evaluates the performance of four different approaches to managing the hierarchy. We present a novel approach that uses a common file system with an associated indexing engine to store terms as directories (with narrower terms as subdirectories) and show how we achieve both scalability and efficiency.