App guides
The Fivetran app sends your Swell store data to Fivetran, which loads it into your data warehouse, such as Snowflake, BigQuery, Redshift or Databricks. It sends each change to the collections you choose to a Fivetran Webhooks connector as it happens, and backfills the records you already have.
| When | What the app does |
| A record in a collection you chose is created, updated or deleted | Sends one row with the full current record to your Webhooks connector. Deletions are sent while Send deletions is on. |
| A payment succeeds, fails or is voided | Sends the payment as an update, since payments have no create or update events of their own. |
| You start a backfill | Sends the collection's existing records, one page at a time, and shows the progress on the backfill. |
| You run a Connection test backfill | Sends one test row to check the URL and the signature. |
| Your Fivetran connector syncs | Fivetran loads the rows into your destination, on the connector's sync schedule. |
- A Fivetran account with a destination, such as Snowflake, BigQuery, Redshift or Databricks, and access to add connectors in it.
Install Fivetran from the Swell App Store, and set it up in your test environment before the live one.
Add a Webhooks connector
In Fivetran, add a connector of type Webhooks and choose the destination schema and table name. Copy the webhook URL from the setup form.
Set up signing
This step is optional. In the connector's signature validation settings, choose the algorithm SHA-256, an encoding (hex or base64), a header key such as X-Fivetran-Signature-256, and a secret.
Add the URL to the app
In Swell, go to Apps > Fivetran > Settings and paste the Webhook URL. If you set up signing, enter the same Signing secret, Signature header and Signature encoding. Under Collections to send events for, check the collections to send, since none are checked at first, and click Save settings.
Test the connection
Open Fivetran in the dashboard sidebar and click New fivetran backfill. Choose the collection Connection test and save. Within a few seconds the backfill shows Completed, or Failed with the error from Fivetran.
Backfill your existing records
Click Backfill all collections, or start a backfill for one collection with New fivetran backfill. After the connector's next sync, the rows are in your destination.
Place a test order
If you checked Orders, place a test order. After the connector's next sync, your destination has a row for the order with the event created, and a row with the event updated for each change to it.
Go live
Settings are saved separately for each environment. In the live environment, repeat steps 3 to 5. Use a separate Webhooks connector for the test environment, so test data stays out of your live tables.
The app can send orders, customers, products, subscriptions, payments, refunds, invoices, carts, coupons, promotions and categories. Changes are sent for the collections checked under Collections to send events for:
- Creating a record sends it with the event created.
- Updating a record sends the whole current record with the event updated, not just the fields that changed.
- With Send deletions on, deleting a record sends it, as it was before it was deleted, with the event deleted.
- Payments are sent with the event updated when they succeed, fail or are voided.
If Fivetran is down, times out or rate-limits the app, Swell sends the change again later. Other errors, such as a wrong URL or signature, are logged and the change isn't sent again.
A backfill sends the records a collection already has. Backfills are listed under Fivetran in the dashboard sidebar, with Active and Failed tabs.
- To backfill one collection, click New fivetran backfill, choose the collection and save. To send only newer records, set Created on or after.
- To backfill every collection, click Backfill all collections. It starts one backfill for each collection that doesn't have an active one, whether or not the collection is checked for live sync.
- Each backfill shows its status, the records sent, the total, the pages sent, when it started and finished, the last record sent and the last error.
- To stop a backfill, open it and choose Cancel backfill. It stops after the page being sent. To continue a failed or canceled backfill after its last record, choose Resume backfill.
Each change is one row, so the table is an event log, not a copy of your store. Every row has the same fields:
| Field | Value |
| event_id | The Swell event ID for live rows, backfill: |
| store_id | Your Swell store ID. |
| collection | The collection, such as orders, or test. |
| event | created, updated, deleted, backfill or test. |
| record_id | The ID of the Swell record. |
| sent_at | When the app sent the row. |
| data | The full Swell record. For deleted, the record as it was before it was deleted. |
Fivetran adds its own columns, and depending on the connector's settings, unpacks data into columns or keeps it as JSON. To get the current state of a record, take its latest row by sent_at for each collection and record_id, and drop records whose latest row is deleted.
With a Signing secret, every request carries an HMAC SHA-256 signature of its body, in hex or base64, in the header set in Signature header. The connector checks it against the secret in its signature validation settings.
| Where | What it's for |
| Apps > Fivetran > Settings | The webhook URL, signing, the collections to send live, deletions, the backfill page size and the Enabled switch. |
| Fivetran in the sidebar | The list of backfills, with New fivetran backfill and Backfill all collections. |
| A backfill | Its status and progress, the last error, and Cancel backfill or Resume backfill. |
Settings are in Apps > Fivetran > Settings and are saved separately for each environment.
| Setting | Description | Default |
| Webhook URL | The URL of your Fivetran Webhooks connector, from its setup form. Must start with https. Required. | None |
| Signing secret | Signs each request with HMAC SHA-256. Use the same secret in the connector's signature validation. Masked on the settings page. | None |
| Signature header | The header the signature is sent in. Must match the connector's header key. | X-Fivetran-Signature-256 |
| Signature encoding | Hexadecimal or Base64. Must match the connector. | Hexadecimal |
| Collections to send events for | The collections whose changes are sent live. Backfill all collections always covers every collection. | None |
| Send deletions | Sends a row with the event deleted when a record is deleted. | On |
| Backfill page size | Records per request during a backfill, from 1 to 500. Lower it if Fivetran rejects requests as too large. | 100 |
| Enabled | Turns off all sending without uninstalling the app. Running backfills fail with a message. | On |
- Data reaches your warehouse on the connector's sync schedule, not instantly.
- Payments are only sent live when they succeed, fail or are voided. Use a backfill to send the rest.
- Rows can arrive out of order, and retries or overlapping backfills can send a record more than once. Deduplicate by record_id and sent_at.
- Records changed while a backfill runs are sent by both the backfill and live sync.
- A backfill's total is counted when it starts, so the records sent can end up higher than the total.
- Records are sent as the Swell API returns them, including customer contact details and payment card details such as the brand, the last 4 digits and the gateway token. Secure your destination to match.
- High-volume collections such as carts send a row on every change when they're checked.
- If Fivetran keeps failing for about 4 days, Swell turns off the app's event function.
- The app doesn't use Fivetran's REST API, so it doesn't start connector syncs or show the connector's status.
A failed backfill shows the error on the backfill. The app also logs each change it can't send. In the Swell dashboard, the logs are under Developer > Console > Logs.
The error shows Fivetran's response. Check that the Webhook URL is the one from the connector's setup form. If you set up signing, check that the secret, header and encoding match the connector's signature validation settings.
Paste the connector's webhook URL, which starts with https, in the app settings and save. Then open the backfill and choose Resume backfill.
Turn on Enabled in the app settings and save. Then open the backfill and choose Resume backfill.
Rows arrive after the connector's next sync. Check that the collection is checked under Collections to send events for and that Enabled is on.
If the error is HTTP 413 or another size error, lower Backfill page size, save, and choose Resume backfill.
The table is an event log, and retries or overlapping backfills can send a record again. Deduplicate by record_id and sent_at.
If Fivetran kept failing for about 4 days, Swell turned off the app's event function. Fix the cause, then reinstall the app to turn it back on.
Uninstalling the app stops sending changes to Fivetran and removes Fivetran from your dashboard sidebar. Uninstalling applies to the environment you are in, so remove the app from the test and live environments separately.
Rows already in your destination stay there. If you no longer need the Webhooks connector, remove it in Fivetran.
| Function | Runs on | What it does |
| record-events | Created, updated and deleted events of each supported collection, and payment.succeeded, payment.failed and payment.voided | Sends one row per change, reading the full record for updates. |
| backfill-run | backfill.created and backfill.updated, while the backfill is pending or running | Sends one page of the collection, sorted by ID after the last record sent, and saves the progress. Saving it starts the next page. |
| backfill-all | The Backfill all collections action | Starts a backfill for each collection that doesn't have an active one. |
| backfill-control | The Cancel backfill and Resume backfill actions | Sets the backfill to canceled, or back to pending. |
| test-connection | An action that isn't in the dashboard yet | Sends one test row and returns Fivetran's answer. |
Backfills are stored in the app collection apps/fivetran/backfills, with collection, status, date_from, last_id, pages, sent, total, last_error, date_started and date_completed. Live rows are sent one JSON object per request, and backfill rows as a JSON array, which Fivetran stores as one row per element.
To send a test row without a backfill:
swell api post /functions/fivetran/test-connectionSwell builds and maintains the Fivetran app. For help with the app, email support@swell.is. For questions about your Fivetran account or connector, see the Fivetran documentation.