Skip to main content

Connection Incremental State

Many source connectors support incremental replication state which means a bookmark is saved between syncs so that the following sync picks up where the previous one left off. For example this is usually implemented in the connector using an API filter with something like a last modified timestamp. See the meltano docs for more technical details on how this works.

In Arch you can view the current connection state and make edits if needed. This is usually only done when you need to manually backfill data and is sometimes accompanied by manual midifications to the destination table.

Modifying state incorrectly could lead to unexpected results so be careful and reach out to the Arch support team if needed.

Edit State​

To edit the connection state you navigate to the Settings tab for the connection, scroll down to where it says Edit connection state, and hit the carot to activate the editor. Modify the stream bookmarks as needed and save the changes by clicking Update.

In the example below you can see the HubSpot stream bookmarks. The replication_key represents the field thats being used to filter and the replication_key_value is the current filter value

Pipeline State Editor

The state can not be edited while a connection is actively running.

Reset State​

To remove all state and run a complete backfill you can set the state to the following:

{
  "completed": {
    "singer_state": {
      "bookmarks": {}
    }
  },
  "partial": {}
}