Skip to main content

Pipeline Incremental State

Many source connectors support incremental replication state which means a bookmark is saved between syncs so that the following sync picks up where the previous one left off. For example this is usually implemented in the connector using an API filter with something like a last modified timestamp. See the meltano docs for more techincal details on how this works.

In Arch you can view the current pipeline state and make edits if needed. This is usually only done when you need to manually backfill data and is sometimes accompanied by manual midifications to the destination table.

Modifying state incorrectly could lead to unexpected results so be careful and reach out to the Arch support team if needed.

Edit State​

To edit the pipeline state you navigate to the Settings tab for the pipeline, scroll down to where it says Edit pipeline state, and hit the carot to activate the editor. Modify the stream bookmarks as needed and save the changes by clicking Update.

In the example below you can see the HubSpot stream bookmarks. The replication_key represents the field thats being used to filter and the replication_key_value is the current filter value

Pipeline State Editor

The state can not be edited while a pipeline is actively running.

Reset State​

To remove all state and run a complete backfill you can set the state to the following:

{
  "completed": {
    "singer_state": {
      "bookmarks": {}
    }
  },
  "partial": {}
}