Troubleshooting: Webhook deliveries failing
This article describes how to resolve the issue of webhook event deliveries not being received or failing repeatedly.
This article describes how to resolve webhook event deliveries that are not being received or are failing repeatedly.
Problem description
Behavior: Cerby events are not arriving at your endpoint, or the webhook status shows Disabled unexpectedly.
Context: This can occur at any time after a webhook is created or after changes are made to your endpoint configuration.
Cause
Webhook deliveries fail when Cerby cannot complete an HTTPS POST request to your endpoint, or when your endpoint returns a response that Cerby treats as a failure.
The following are common triggers:
The endpoint URL is incorrect, unreachable, or not publicly accessible from the internet.
The endpoint does not use HTTPS, or its TLS certificate is invalid or expired.
The endpoint returns a 4xx response other than 408 or 429, which causes Cerby to stop retrying immediately.
All six delivery attempts were exhausted within the 24-hour delivery window.
The webhook is turned off.
Signature verification is rejecting requests that are actually valid.
Who this affects
Workspace Owners, Super Admins, and Admins who manage webhook endpoints.
Developers who build systems that consume Cerby webhook events.
Solution
Solution 1: Check the webhook status
Before investigating your endpoint, confirm that the webhook is turned on.
To check the webhook status, complete the following steps:
Log in to your Cerby workspace.
Select Settings from the left navigation. The Workspace Configuration page is displayed.
Select the Webhooks left tab.
Locate the webhook in the list and check the status chip on its card:
If the status shows Disabled, the webhook is not delivering events. Select the actions menu and select Enable webhook to resume deliveries.
If the status shows Enabled, continue to one of the solutions below based on the type of failure.
Expected result: When the webhook is enabled, Cerby resumes delivery for new events. Past events that occurred while the webhook was disabled are not replayed.
Solution 2: Resolve a 4xx fail-fast failure
If your endpoint is returning a 4xx response code other than 408 or 429, Cerby has permanently stopped retrying that delivery. A 4xx response indicates a configuration problem with your endpoint that will not resolve on its own.
The following are common causes:
401 Unauthorized: Your endpoint requires authentication that Cerby is not providing. Remove the authentication requirement or update your endpoint logic to accept Cerby's requests.
403 Forbidden: Your endpoint is rejecting Cerby's IP address or user agent. Update your allowlist or firewall rules.
404 Not Found: The webhook URL no longer resolves to a valid endpoint. Update the webhook URL.
410 Gone: The endpoint was intentionally removed. Update or delete the webhook.
To update the webhook URL, complete the following steps:
Log in to your Cerby workspace.
Select Settings from the left navigation. The Workspace Configuration page is displayed.
Select the Webhooks left tab.
Locate the webhook in the list and select the actions menu.
Select Edit webhook. The webhook detail page is displayed in edit mode.
Update the Endpoint URL field with the correct HTTPS endpoint URL.
Click the Save changes button. A success message is displayed.
Expected result: Future events are delivered to the corrected URL. Past failed deliveries are not replayed. To identify missed events, use the Cerby API's account or automation endpoints to query the current state of the affected resources.
Solution 3: Resolve delivery exhaustion (all retries failed)
If your endpoint was unavailable for more than 24 hours, Cerby will have exhausted all six delivery attempts and the events are permanently undeliverable. Cerby does not replay events after the 24-hour delivery window closes.
To prevent this from recurring, ensure your endpoint has sufficient availability and responds within 30 seconds. For planned maintenance windows, turn off the webhook in advance and turn it on again when your endpoint is available.
Events that fall outside the 24-hour delivery window cannot be retrieved via webhook. To audit missed events, use the Cerby API's account or automation endpoints to query the current state of the affected resources.
Expected result: After your endpoint is restored and the webhook is turned on, future events are delivered normally.
Solution 4: Fix signature verification rejections
If your endpoint is receiving requests but rejecting them during signature verification, the following issues are most common.
Issue: Timestamp skew rejection
Your verification code is rejecting requests because the X-Cerby-Signature-Timestamp value is more than five minutes old. To resolve this issue, do the following:
Ensure your server clock is synchronized with an NTP server. Clock drift causes valid requests to appear stale.
Do not buffer or queue incoming requests before verifying them. Verify the signature immediately upon receipt.
Issue: Incorrect signed bytes
Your verification code is computing the wrong bytes to verify against the signature. To resolve this issue, do the following:
The signed message is
<timestamp_ms>.<raw_body_bytes>, where<timestamp_ms>is the millisecond-precision Unix timestamp fromX-Cerby-Signature-Timestamp, followed by a literal period, followed by the raw request body bytes.Use the raw request body bytes as received. Do not parse, re-serialize, or modify the JSON before verifying.
Issue: Base64url padding
The signature value in X-Cerby-Signature is base64url-encoded without padding. If your base64 decoder requires padded input, re-pad the value before decoding. Append = characters until the string length is a multiple of four. Do not hard-code == across all algorithms — an Ed25519 signature needs == and an HMAC-SHA256 signature needs =.
Issue: Dual-sign during key rotation
When Cerby is rotating your webhook's signing key, each request includes two X-Cerby-Signature headers. Accept the request if either signature validates. Some HTTP frameworks collapse repeated headers into a comma-separated string. Split the value and try each row independently before rejecting.
For reference verification implementations in Python and Node.js, refer to Implement a webhook receiver.
Expected result: After correcting your verification code, requests from Cerby are accepted and processed normally.
Solution 5: Rotate a compromised signing key
If you suspect your signing key has been exposed, rotate it immediately. Rotating invalidates the old key after a 24-hour dual-sign grace period. During this window, Cerby signs requests with both the old and new keys.
Update your stored key within 24 hours of rotation. After the grace period, the old key expires and only the new key is accepted.
To rotate a signing key, complete the following steps:
Log in to your Cerby workspace.
Select Settings from the left navigation. The Workspace Configuration page is displayed.
Select the Webhooks left tab.
Click the name of the webhook whose key you want to rotate. The webhook detail page is displayed.
Locate the Signing and verification card and click the rotate button:
Ed25519: Click Rotate public key.
HMAC-SHA256: Click Rotate secret.
A confirmation dialog box is displayed.
Review the confirmation and click the confirm button:
Ed25519: Click Rotate public key.
HMAC-SHA256: Click Rotate secret.
For Ed25519, the rotation begins immediately. The new public key appears in the Signing and verification card.
For HMAC-SHA256, a dialog box is displayed showing the new signing secret. Copy the Signing secret value before closing — Cerby cannot recover it.
Update your stored key in your verification system.
Click the Done button. Cerby begins dual-signing requests with both the old and new keys. After 24 hours, the old key expires automatically.
Expected result: After you update your stored key, your verification code accepts requests signed with the new key.
Get help
If the issue persists, contact:
Your workspace Admin for workspace access or configuration issues.
Cerby Support at support@cerby.com.
Related articles
Feature guides:
Developer reference:
Last updated
Was this helpful?