- Ingestion definitions
- ACE products
- AEOLUS products
- Aerosol CCI products
- CALIPSO LIDAR products
- CLOUDNET products
- Cloud CCI products
- EARLINET products
- ECMWF GRIB products
- GEOMS products
- GOME products
- GOME-2 products
- GOMOS products
- GOSAT FTS products
- Green House Gases CCI products
- HIRDLS products
- IASI products
- MIPAS products
- MLS products
- NPP products
- OMI products
- OSIRIS products
- Ozone CCI products
- QA4ECV products
- SCIAMACHY products
- SMR products
- Sentinel-5P products
- TEMIS products
- TES products
- C library
- IDL interface
- MATLAB interface
- Python interface
- Command line tools
What is HARP?
HARP is a toolkit for reading, processing and inter-comparing satellite remote sensing data, model data, in-situ data, and ground based remote sensing data. The toolkit is composed of:
- A set of command line tools
- A library of analysis functions
The main goal of HARP is to assist in the inter-comparison of data sets. By appropriately chaining calls to HARP command line tools one can pre-process data sets such that two datasets that need to be compared end up having the same temporal/spatial grid, same data format/structure, and same physical unit. At the end of the toolchain you will have a set of data files that can be directly compared in e.g. Python, IDL or MATLAB.
In order for the HARP command line tools to handle each others output the toolkit uses its own data format conventions for intermediate files. The use of this harmonized formatting means that satellite products need to be converted first to HARP compliant files. Once in this format, further processing can be performed.
In order for HARP command line tools to handle each others output the toolkit makes uses of its own data format conventions.
There is one standard for representing and storing measurement data, which only prescribes a specific set of constraints to the data format and is still flexible enough to allow data storage using either netCDF, HDF4, or HDF5 and inclusion of metadata from other standards such as netCDF-CF.
The other data format convention is a standard for storing information on collocations, the Collocation Result File, which is based on CSV.
Both data formats are described in more detail in the Data formats section of the documentation.
HARP can perform a wide variety of operations on the data. A description of the algorithms as used in HARP can be found in the Algorithms section of the documentation.
Several HARP functions and tools can take a sequence of operations, such as filters or regridding operations, which are then performed on a product. A description of the operations expressions is provided in the Operations section of the documentation.
Common parts that are used by the various HARP command line tools are gathered in a single HARP C library. This library can be used to build custom applications or libraries that work with HARP compliant data products.
The HARP IDL interface provides a set of functions to import and export HARP products, and to ingest non-HARP products of a type supported by HARP from IDL.
The HARP MATLAB interface provides a set of functions to import and export HARP products, and to ingest non-HARP products of a type supported by HARP from MATLAB.
The HARP Python interface provides a set of functions to import
and export HARP products, and to ingest non-HARP products of a type supported by
HARP from Python. The Python interface depends on the
which is part of the C foreign function interface (cffi) package. This package
must be installed in order to be able to use the Python interface. See the cffi
documentation for details on how to install the cffi package.
Command line tools
The HARP command line tools perform various operations on HARP compliant data
products. All command line tools except
harpconvert expect one or more HARP
data products as input. The
harpconvert tool can be used to convert non-HARP
products into HARP products.
The available command line tools are described in more detail in the Command line tools section of the documentation.
A typical inter-comparison
Typical sequence when comparing two datasets of measurements:
- harpconvert …
- Convert all products that do not already follow the HARP format conventions to a common data format. Filter/convert the data in each file: remove unneeded parameters/measurements, add derived physical parameters, perform unit conversion, regrid dimensions, etc.
- harpcollocate matchup …
- Find the measurement pairs that match within the specified collocation criteria and save the result into a collocation result file.
- harpcollocate resample …
- Apply resampling to the collocation result file.
- harpconvert -a ‘collocate-left(<csvfile>)’ …
- Apply the collocation filter for each file in the primary dataset
- harpconvert -a ‘collocate-right(<csvfile>)’ …
- Apply the collocation filter for each file in the secondary dataset
Input files can be verified with
and after each stage, the contents of each file can be inspected from the command line using: