This document outlines the options available in terms of how file-based deliveries are made.
- File Format: CSV
- File Compression: GZIP
- Folder structure: See here
- Files are automatically partitioned
- Success File is by default provided
.csv with a header column and
, as the delimiter.
We deliver New-Line Delimited JSON with the
We offer to compress each file delivered with
This describes the folder structure used to separate consecutive deliveries of data. Files are partitioned by date and time of the shipment.
PREFIXis optional and customisable
dataset_version_stringis a unique name for the version of the dataset received
is_backfillif the delivery is a backfill
_backfillwill be appeneded to the
DELIVERY_DATE_AND_HOURthe date and hour of the delivery (UTC) e.g.
2023/01/01/13/would be data with delivery started on Jan 1st, 2023 at hour 13 UTC.
PERIOD_START_DATEtime partitions of data, indicates start date of each observation period, e.g.
2022/12/01would be data describing December 2022.
Files are automatically partitioned into several chunks/files. These files are numbered.
There is no guaranteed sort between the chunks/files in a single delivery.
If enabled, a success file can be provided.
The success file is simply a file that is written after all files within one delivery has been successfully written to the target.
For a backfill of data when multiple time periods of data is delivered at once, one success-file will be provided per period, in its respective folder.
In the backfill example below, there is one success-file for each period of data. Data is delivered for November and December 2022.
The name of the success file is
_SUCCESS and contains no particular information.