The customer experience
A customer connects Dropbox, selects a source folder in Dropbox, and chooses a destination folder in their Vimeo library. From then on, a video added to the selected Dropbox folder is imported automatically. Vimeo documents this same customer flow in its Help Center.
The feature sounds like a file copy, but the first message contains no file. Dropbox sends a webhook: a short HTTP request that says which connected account changed. The integration must then ask Dropbox for the actual changes. A A cursor is a saved position in Dropbox's change history. It marks where the previous check ended.
The initial connection uses OAuth. The customer signs in with Dropbox and grants specific access without sharing their Dropbox password with Vimeo.
Four ideas used in the rest of the story
Webhook
A wake-up saying that an account changed. Several file changes may be grouped into one webhook, and the same account may be announced again.
HMAC signature
A hash-based message authentication code: a cryptographic check over the original request bytes that verifies Dropbox produced the notification.
Kafka, consumer, and offset
Kafka stores the wake-up durably. A consumer is the worker that later reads it. Its offset records how far that worker has completed.
Idempotency
Repeating work for the same Dropbox file revision converges on the same Vimeo import instead of creating another video.
My role
I led the technical work on the integrations team, from the product discussion and design through implementation, release, onboarding, and production issues. For this integration, the important design decisions were the boundary between the webhook and file discovery, the identity of an import, and the recovery state customers would see when a file could not move.
The work I led
Define folder behavior
Specified connection, folder selection, destination choice, disconnect behavior, and the customer-visible failure states.
Design change discovery
Separated the signed account-change signal from cursor-based listing and file-level work.
Build the transfer path
Implemented with the team and made each eligible file an independently retryable import.
Operate the integration
Owned monitoring, production debugging, onboarding support, and the reliability changes that followed.
A webhook is a hint
How to read the diagrams
- Dropbox and its account-change signal
- Webhook verification and Dropbox change discovery
- Kafka and independent file-level work
- The Vimeo upload path
- A file or connection that needs attention
From folder change to Vimeo import
- folder changeDropbox → Webhook boundary
Notify that an account changed
account identity, not file contents
- intakeWebhook boundary
Verify the signature over the original bytes
reject a request that cannot be authenticated
- before HTTP successWebhook boundary → Kafka
Publish account-change work
acknowledge only after Kafka accepts it
- deliveryKafka → Import worker
Deliver to an import consumer
the same account signal may arrive again
- discoverImport worker → Dropbox
List changes from the saved cursor
zero, one, or many file changes
- for each videoImport worker → Vimeo
Transfer into the chosen destination
stable file identity and revision
- durable resultImport worker → Kafka
Commit the consumed offset
after cursor and import outcomes are stored
This separation prevents the webhook response from waiting on file listing or video transfer. It also handles grouping: several folder changes can be represented by one account notification, while the saved cursor still lets the consumer enumerate every change. Kafka can deliver the same account signal again, so replay starts from durable cursor and import state rather than from an assumption that every message is new.
Authenticate before parsing
Dropbox signs the exact request bytes before they are parsed as JSON. The receiver computes the expected HMAC with the app's private signing value and uses a comparison designed not to reveal partial matches through timing before trusting any account identifier in the request.
- check
- Signature over the original bytes
- accepted behavior
- Continue to account discovery
- rejected behavior
- Return an authentication error
- check
- Known connected account
- accepted behavior
- Publish change discovery to Kafka
- rejected behavior
- Ignore an account with no active connection
- check
- Durable hand-off
- accepted behavior
- Return success after Kafka accepts the work
- rejected behavior
- Let Dropbox retry if the publish fails
The signature proves that Dropbox produced the request bytes. It does not make the request body a file list, and it does not remove the need for idempotency after the request is accepted.
What at-least-once delivery means here
Kafka makes the accepted wake-up durable and replayable. It does not promise that a consumer sees it only once, and it cannot include a Vimeo upload in the same all-or-nothing operation. A crash after an upload but before the Kafka offset is committed can therefore produce another delivery.
- failure point
- The hand-off to Kafka fails
- what can happen
- The account-change work is not durable
- safe response
- Do not acknowledge the webhook; let Dropbox retry
- failure point
- Dropbox sends another account signal
- what can happen
- The consumer may discover the same range again
- safe response
- Resume from the stored cursor and reuse existing file imports
- failure point
- Consumer crashes before offset commit
- what can happen
- Kafka delivers the record again
- safe response
- Read durable cursor and file state before continuing
- failure point
- Vimeo accepts an upload but the response is lost
- what can happen
- The consumer cannot safely assume failure
- safe response
- Check the saved revision and any recorded Vimeo video before another upload
- failure point
- Dropbox access is revoked
- what can happen
- Automatic retries cannot restore permission
- safe response
- Stop retrying and ask the customer to reconnect
Idempotency means that every retry for the same Dropbox file revision converges on the same Vimeo import. The key is Dropbox's stable file identity plus its revision. The consumer stores that identity with the import outcome and checks it before repeating the outside upload. Kafka provides the replay; the saved import state prevents replay from creating another video.
Reconciliation means checking the saved file revision and any Vimeo video already associated with it before deciding whether another upload is safe.
Let each file recover independently
Once change discovery finds eligible videos, each file proceeds separately. One large transfer, a revoked connection, or a Dropbox limit should not erase the progress of unrelated files.
- state
- discovered
- meaning
- The file is new or changed and matches the selected source folder
- next action
- Create or find its import attempt
- state
- transferring
- meaning
- The file is moving through Vimeo's upload path
- next action
- Track progress and retain the same file identity
- state
- retry later
- meaning
- A timeout, rate limit, or temporary Dropbox error interrupted the attempt
- next action
- Wait longer between attempts and continue the same import
- state
- needs reconnection
- meaning
- Dropbox authorization can no longer be refreshed
- next action
- Ask the customer to connect again
- state
- complete
- meaning
- Vimeo has accepted the video and the destination is recorded
- next action
- Do not create a second video for the same file revision
A useful status has to answer two questions: which file failed, and what can be done next? A generic connection error is insufficient when most files imported successfully and one needs attention.
Why the boundaries mattered
The webhook is an authenticated wake-up signal. The cursor records which Dropbox changes have been examined. The file revision identifies one import. With those responsibilities separated, a duplicate notification replays known work, a grouped notification still reveals every change, and one failed transfer does not erase progress for the other files.