Skip to main content

Configuring Sources

Once you've chosen a source for your pipeline you can update any configurations needed, depending on the source.

Configure Source

Under the Advanced settings section you will find more optional settings to configure. If you need help understanding what an advanced setting does, you can check MeltanoHub for the open source connector documentation or reach out to the Arch team for help.

OAuth Configuration​

Arch owns and maintains many OAuth apps (e.g. Facebook, HubSpot, etc.) in order to provide customers and their clients with a smooth authorization workflow. As part of configuring an OAuth supported source you will click the Authorize with... button that will redirect you to authorize Arch to access your data. Once authorized, Arch will be able to connect and extract data for you.

Configure OAuth Source

Stream Selection​

Sources contain streams of data that get written to tables in your destination database and those streams contain attributes that map to columns in your tables.

Selection Modes​

Pipelines have two selection modes:

  • Default - the pipeline will sync all streams that are default selected in the connector itself, usually this is every stream.
  • Selected - the pipeline will only sync the streams and attributes you've selected.

Every pipeline starts in Default selection mode.

Switching to Selected Mode​

When switching from Default to Selected mode Arch will generate a catalog of streams and attributes for your pipeline. To do this navigate to the Streams tab and hit the "Generate source schema" button. Sometimes this requires connecting directly to the source itself to discover what options are available, so all credentials need to be set to avoid connection errors.

Depending on the source, this stream discovery process could take several minutes.

Once that process is completed you can select the checkboxes next to the stream you want to select.

Select Streams

Then by clicking on the stream name you can chose your attribute selections.

Select Stream Attributes

Refreshing Source Schema​

Over time the streams available in your source may evolve, in order to pull them into your available streams to select from you'll need to hit the "Refresh source schema" button to kick off the discovery process again. Once completed you'll see your new streams and attributes available for selection.

Troubleshooting​

If the source schema discovery process fails you'll see a link in the Streams tab to the logs where you'll find more information about what caused the failure. The most common issue is incorrect or incomplete credentials.

Reach out the the Arch support team if you continue to experience issues.