Records, import and export

Find, filter, change, import and export records, and make retried writes safe.

Beta

Data Hub is in Beta. It is off by default; an administrator turns it on per tenant.

Open an entity and select Records to see its records. A Data Hub Reader can read and export them. A Data Hub Editor or Designer can also create, edit, import and delete them.

Records#

A record holds the field values (data) and the system fields id, revision, createdAt, createdBy, updatedAt and updatedBy.

  • An update changes only the fields it sends. Sending null clears a field.
  • An update can send the revision you last read. If someone changed the record since, the update is refused with 409 dh_revision_conflict. Read the record again and reapply your change.
  • A record is at most 64 KiB of JSON.

Filter and sort#

Build a filter on Records, then select Copy filter as JSON to use the same filter in an automation or the API.

A filter is a JSON tree: a leaf {"field", "op", "value"}, or {"and": [...]} / {"or": [...]}. It can be at most 3 levels deep with 20 leaves.

Operator Text Number, decimal, date, date and time Boolean Choice
eq, ne Yes Yes Yes Yes
in, nin (up to 100 values) Yes Yes No Yes
gt, gte, lt, lte Yes Yes No No
contains, startsWith Yes No No No
isNull, isNotNull (no value) Yes Yes Yes Yes
  • Text compares without regard to case.
  • A missing value is null. ne and nin match records without a value; add isNotNull to exclude them. eq with null is refused: use isNull.
  • contains always scans the entity. See Indexed fields.
  • System fields can be filtered and sorted too.
JSONCopy
{"and":[{"field":"status","op":"eq","value":"open"},{"field":"amount","op":"gte","value":"1000"}]}

A sort is one string on every surface, with at most 3 keys: "amount desc, invoiceNumber". Without a sort, records come in id order.

Paging#

A query returns one page: 100 records by default, at most 1,000 records and 8 MiB. While more remain, it returns a nextCursor. Send the same query with the cursor to get the next page. Every record appears exactly once across pages.

  • A cursor lives for 24 hours. It stops working (400 dh_cursor_invalid) if the query changes or a field it uses is removed or retyped. Start again from the first page.
  • Ask for includeTotal to get a total. It is capped at 10,000.
  • Fetching all pages at once (Fetch All in Studio, iter_records in Python) stops at 10,000 records or 64 MiB with dh_result_too_large, unless you raise the limit on purpose.

Make retried writes safe#

Use When What happens
Create or update by a unique field (upsert) The record has a natural key, such as an invoice number Creates the record, or updates the one with that key. Never two.
An idempotency key on create A job that may be retried or resumed must create one record The same key and data within 24 hours return the first record. The same key with other data is 409 dh_idempotency_key_reused.
The Idempotency-Key HTTP header Any HTTP client retrying the same request The same key and body within 24 hours replay the first answer.

Import records#

Import a .csv or .json file into an entity. A file can be up to 20 MB and 100,000 rows.

  1. Choose a mode:
    • Add all as new creates a record for each row.
    • Update matching records matches rows on a unique field you pick, updates the record that has the key and creates the others. It also needs the right to edit records.
  2. Map each field to a column of the file (a CSV header or a JSON member). Fields you do not map are not imported.
  3. Start the import. It runs in the background.

When rows are refused, download the error file. It lists row, field, code, message for each refused row. If an import fails part way, the rows already committed stay, and the import says at which row it stopped.

Each entity runs one import or export at a time, and a tenant runs at most five at once (409 dh_import_running).

Export records#

Export the records that match the current filter, sort and selected fields as CSV or JSON. Every match is exported, up to 100,000 rows, and paging is ignored. A finished export file is kept for one day.

Deleted records#

Deleted records, deleted entities and removed fields are purged after 30 days by default.

Records

Filter and sort

Paging

Make retried writes safe

Import records

Export records

Deleted records