Metapack: Data Packaging System

Metapack is a data package format and packging system that uses Metatab formatted files for both metadata and for the instructions for building data packages.

Metatab is a metadata format that allows structured metadata – the sort of information about a dataset like title, origin, or date of publication that you’d normally store in JSON, YAML or XML – to be stored and edited in tabular forms like CSV or Excel. Metatab files look exactly like you’d expect, so they are very easy for non-technical users to read and edit, using tools they already have. Metatab is an excellent format for creating, storing and transmitting metadata. For more information about metatab, visit http://metatab.org, or get Metapack Overview to understand the format.

Using metapack, you can create a Metatab formatted file that describes the data you’d like to package and create an Excel or Zip file data package that holds that data. Metapack also includes programs to load data sets to AWS S3, Data.World and CKAN, and to use these packages in Jupyter notebooks.

This python module provides CLI tools and APIs for inspecting and using data packages, but does not provide support for building data packages. For building data packages, see the metapack-build. module.

Install

If you want to use metapack packages, just install metapack. But if you want to build packages, you should install:

• metapack-build for building packages.

• metapack-jupyter. for Jupyter notebook support.

• metapack-github. for managing Github repositories.

• metapack-wp. for publising packages to the web.

The most consistently reliable way to install Metapack, especially in MacOs, is within Conda, which will avoid having to compile the geographic libraries for geos and gdal. So, on MacOs, you may want to start with:

conda create --name metapack -c conda-forge python=3.10.2 pyproj proj-data geopandas
conda activate metapack

Then, on MacOS and Linux, install metapack with:

pip install metapack-build metpack-github metapack-jupyter
# Or
pip install metapack

For development, you’ll probably want the development package, with sub-modules for related repos:

$ git clone --recursive https://github.com/Metatab/metapack-dev.git
$ cd metapack-dev
$ bin/init-develop.sh

Metatab and Metapack Overview

• Metapack Overview
  • Just Enough Metatab
  • Root, Documentation and Contact Metadata
  • Resources and References
    • Resource Urls
  • Schemas: The Data Dictionary

Using Metapack Packages

• Using Metapack Packages
  • Command Line Operations
    • Opening Packages
  • Using Packages In Python

• Metpack CLI Commands
  • mp: Main program
  • config: Show Configuration Information
  • doc: Display package documentation
  • index: Add a package to the search index
  • info: Show Package information
  • open: Open a resource as a CSV file
  • run: Run a resource to produce data
  • search: Search for a package

Building Metapack Packages

Note

Creating Metapack packages requires the metapack-build. module.

• Getting Started Tutorial

• Generating Rows

• Altering rows with transforms

Indices and tables

• Index

• Module Index

• Search Page