# Fivetran

Source: https://developers.swell.is/app-guides/fivetran

The Fivetran app sends your Swell store data to Fivetran, which loads it into your data warehouse, such as Snowflake, BigQuery, Redshift or Databricks. It sends each change to the collections you choose to a Fivetran Webhooks connector as it happens, and backfills the records you already have.

## How it works

| When | What the app does |
| --- | --- |
| A record in a collection you chose is created, updated or deleted | Sends one row with the full current record to your Webhooks connector. Deletions are sent while Send deletions is on. |
| A payment succeeds, fails or is voided | Sends the payment as an update, since payments have no create or update events of their own. |
| You start a backfill | Sends the collection's existing records, one page at a time, and shows the progress on the backfill. |
| You run a Connection test backfill | Sends one test row to check the URL and the signature. |
| Your Fivetran connector syncs | Fivetran loads the rows into your destination, on the connector's sync schedule. |

## Before you begin

- A Fivetran account with a destination, such as Snowflake, BigQuery, Redshift or Databricks, and access to add connectors in it.

## Set up

Install Fivetran from the Swell App Store, and set it up in your test environment before the live one.

**1. Add a Webhooks connector**

In Fivetran, add a connector of type **Webhooks** and choose the destination schema and table name. Copy the **webhook URL** from the setup form.

**2. Set up signing**

This step is optional. In the connector's signature validation settings, choose the algorithm SHA-256, an encoding (hex or base64), a header key such as `X-Fivetran-Signature-256`, and a secret.

**3. Add the URL to the app**

In Swell, go to **Apps > Fivetran > Settings** and paste the **Webhook URL**. If you set up signing, enter the same **Signing secret**, **Signature header** and **Signature encoding**. Under **Collections to send events for**, check the collections to send, since none are checked at first, and click **Save settings**.

**4. Test the connection**

Open **Fivetran** in the dashboard sidebar and click **New fivetran backfill**. Choose the collection **Connection test** and save. Within a few seconds the backfill shows **Completed**, or **Failed** with the error from Fivetran.

**5. Backfill your existing records**

Click **Backfill all collections**, or start a backfill for one collection with **New fivetran backfill**. After the connector's next sync, the rows are in your destination.

**6. Place a test order**

If you checked Orders, place a test order. After the connector's next sync, your destination has a row for the order with the event `created`, and a row with the event `updated` for each change to it.

**7. Go live**

Settings are saved separately for each environment. In the live environment, repeat steps 3 to 5. Use a separate Webhooks connector for the test environment, so test data stays out of your live tables.

## Using Fivetran

### Live sync

The app can send orders, customers, products, subscriptions, payments, refunds, invoices, carts, coupons, promotions and categories. Changes are sent for the collections checked under **Collections to send events for**:

- Creating a record sends it with the event `created`.
- Updating a record sends the whole current record with the event `updated`, not just the fields that changed.
- With **Send deletions** on, deleting a record sends it, as it was before it was deleted, with the event `deleted`.
- Payments are sent with the event `updated` when they succeed, fail or are voided.

If Fivetran is down, times out or rate-limits the app, Swell sends the change again later. Other errors, such as a wrong URL or signature, are logged and the change isn't sent again.

### Backfills

A backfill sends the records a collection already has. Backfills are listed under **Fivetran** in the dashboard sidebar, with **Active** and **Failed** tabs.

- To backfill one collection, click **New fivetran backfill**, choose the collection and save. To send only newer records, set **Created on or after**.
- To backfill every collection, click **Backfill all collections**. It starts one backfill for each collection that doesn't have an active one, whether or not the collection is checked for live sync.
- Each backfill shows its status, the records sent, the total, the pages sent, when it started and finished, the last record sent and the last error.
- To stop a backfill, open it and choose **Cancel backfill**. It stops after the page being sent. To continue a failed or canceled backfill after its last record, choose **Resume backfill**.

### Rows in your warehouse

Each change is one row, so the table is an event log, not a copy of your store. Every row has the same fields:

| Field | Value |
| --- | --- |
| event_id | The Swell event ID for live rows, backfill:<backfill ID>:<record ID> for backfill rows, or test:<time> for test rows. |
| store_id | Your Swell store ID. |
| collection | The collection, such as orders, or test. |
| event | created, updated, deleted, backfill or test. |
| record_id | The ID of the Swell record. |
| sent_at | When the app sent the row. |
| data | The full Swell record. For deleted, the record as it was before it was deleted. |

Fivetran adds its own columns, and depending on the connector's settings, unpacks `data` into columns or keeps it as JSON. To get the current state of a record, take its latest row by `sent_at` for each `collection` and `record_id`, and drop records whose latest row is `deleted`.

### Signed requests

With a **Signing secret**, every request carries an HMAC SHA-256 signature of its body, in hex or base64, in the header set in **Signature header**. The connector checks it against the secret in its signature validation settings.

### In the dashboard

| Where | What it's for |
| --- | --- |
| Apps > Fivetran > Settings | The webhook URL, signing, the collections to send live, deletions, the backfill page size and the Enabled switch. |
| Fivetran in the sidebar | The list of backfills, with New fivetran backfill and Backfill all collections. |
| A backfill | Its status and progress, the last error, and Cancel backfill or Resume backfill. |

## Settings

Settings are in **Apps > Fivetran > Settings** and are saved separately for each environment.

| Setting | Description | Default |
| --- | --- | --- |
| Webhook URL | The URL of your Fivetran Webhooks connector, from its setup form. Must start with https. Required. | None |
| Signing secret | Signs each request with HMAC SHA-256. Use the same secret in the connector's signature validation. Masked on the settings page. | None |
| Signature header | The header the signature is sent in. Must match the connector's header key. | X-Fivetran-Signature-256 |
| Signature encoding | Hexadecimal or Base64. Must match the connector. | Hexadecimal |
| Collections to send events for | The collections whose changes are sent live. Backfill all collections always covers every collection. | None |
| Send deletions | Sends a row with the event deleted when a record is deleted. | On |
| Backfill page size | Records per request during a backfill, from 1 to 500. Lower it if Fivetran rejects requests as too large. | 100 |
| Enabled | Turns off all sending without uninstalling the app. Running backfills fail with a message. | On |

## Limitations

- Data reaches your warehouse on the connector's sync schedule, not instantly.
- Payments are only sent live when they succeed, fail or are voided. Use a backfill to send the rest.
- Rows can arrive out of order, and retries or overlapping backfills can send a record more than once. Deduplicate by `record_id` and `sent_at`.
- Records changed while a backfill runs are sent by both the backfill and live sync.
- A backfill's total is counted when it starts, so the records sent can end up higher than the total.
- Records are sent as the Swell API returns them, including customer contact details and payment card details such as the brand, the last 4 digits and the gateway token. Secure your destination to match.
- High-volume collections such as carts send a row on every change when they're checked.
- If Fivetran keeps failing for about 4 days, Swell turns off the app's event function.
- The app doesn't use Fivetran's REST API, so it doesn't start connector syncs or show the connector's status.

## Troubleshooting

A failed backfill shows the error on the backfill. The app also logs each change it can't send. In the Swell dashboard, the logs are under **Developer > Console > Logs**.

### The Connection test backfill fails

The error shows Fivetran's response. Check that the **Webhook URL** is the one from the connector's setup form. If you set up signing, check that the secret, header and encoding match the connector's signature validation settings.

### "Fivetran webhook URL is not set"

Paste the connector's webhook URL, which starts with https, in the app settings and save. Then open the backfill and choose **Resume backfill**.

### "The Fivetran app is turned off in settings"

Turn on **Enabled** in the app settings and save. Then open the backfill and choose **Resume backfill**.

### Nothing arrives in the warehouse

Rows arrive after the connector's next sync. Check that the collection is checked under **Collections to send events for** and that **Enabled** is on.

### A backfill fails with a size error

If the error is HTTP 413 or another size error, lower **Backfill page size**, save, and choose **Resume backfill**.

### Records show up more than once

The table is an event log, and retries or overlapping backfills can send a record again. Deduplicate by `record_id` and `sent_at`.

### Live changes stopped arriving

If Fivetran kept failing for about 4 days, Swell turned off the app's event function. Fix the cause, then reinstall the app to turn it back on.

## Uninstalling

Uninstalling the app stops sending changes to Fivetran and removes **Fivetran** from your dashboard sidebar. Uninstalling applies to the environment you are in, so remove the app from the test and live environments separately.

Rows already in your destination stay there. If you no longer need the Webhooks connector, remove it in Fivetran.

## For developers

| Function | Runs on | What it does |
| --- | --- | --- |
| record-events | Created, updated and deleted events of each supported collection, and payment.succeeded, payment.failed and payment.voided | Sends one row per change, reading the full record for updates. |
| backfill-run | backfill.created and backfill.updated, while the backfill is pending or running | Sends one page of the collection, sorted by ID after the last record sent, and saves the progress. Saving it starts the next page. |
| backfill-all | The Backfill all collections action | Starts a backfill for each collection that doesn't have an active one. |
| backfill-control | The Cancel backfill and Resume backfill actions | Sets the backfill to canceled, or back to pending. |
| test-connection | An action that isn't in the dashboard yet | Sends one test row and returns Fivetran's answer. |

Backfills are stored in the app collection `apps/fivetran/backfills`, with `collection`, `status`, `date_from`, `last_id`, `pages`, `sent`, `total`, `last_error`, `date_started` and `date_completed`. Live rows are sent one JSON object per request, and backfill rows as a JSON array, which Fivetran stores as one row per element.

To send a test row without a backfill:

```sh
swell api post /functions/fivetran/test-connection
```

## Support

Swell builds and maintains the Fivetran app. For help with the app, email support@swell.is. For questions about your Fivetran account or connector, see the Fivetran documentation.