Skip to main content
Version: latest

Create an Data Pipeline from Webhook to Slack

This guide shows an end-to-end event pipeline that reads an event from a webhook, generates a formatted string, and publishes the result to Slack. While this is a simple example, it has many event notification use cases, such as:

  • submission from website forms (via Cloudflare workers or your custom backend).
  • activity from e-commerce platforms on purchases and shopping carts.
  • notifications from github on your projects' activities.
  • alerts from financial products on your transactions.
  • notifications from any product that can invoke a webhook.

This pipeline uses the following features:

  • webhook: that creates a public API to receive external events, transform them, and publish them to a topic.
  • http-sink: that listens to the same topic and publishes them on Slack.

Objective​

Show an example of how to build an event streaming pipeline that receives webhook events, transforms the input into a readable form, and generates an alert. We assume the events are generated by a user submitting a form, and we'll format it accordingly.

Prerequisites​

Step-by-Step​

  1. Create webhook configuration file
  2. Create http-sink configuration file
  3. Download SmartModules
  4. Start Webhook and Connector
  5. Test Data Pipeline

Create webhook configuration file​

Create a webhook configuration file called form-webhook.yaml:

info

All versions are marked with x.y.z. To find the latest version, run:

  • fluvio hub connector list
  • fluvio hub smartmodule list
meta:
  name: form-webhook
  topic: form-events
webhook:
  outputParts: body
  outputType: json
transforms:
  - uses: infinyon-labs/json-formatter@x.y.z
    with:
      spec:
        match:
          - key: "/type"
            value: "subscribe"
            format:
              with: "📢 {} ({}) subscribed on {}"
              using:
                - "/name"
                - "/email"
                - "/source"
              output: "/formatted"
          - key: "/type"
            value: "use-case"
            format:
              with: ":confetti_ball: {} ({}) wants to solve the following '{}' use-case:\n>{}"
              using:
                - "/name"
                - "/email"
                - "/source"
                - "/description"
              output: "/formatted"
        default:
          format:
            with: "{} ({}) submitted a request"
            using:
              - "/name"
              - "/email"
            output: "/formatted"

The webhook reads the JSON body, applies the json-formatter smartmodule to generate readable text, and writes the new record to a topic called form-events. Checkout labs-json-formatter-sm in github for additional information.

Create http-sink configuration file​

Create an HTTP source connector configuration file called slack-form-alerts.yaml :

apiVersion: 0.1.0
meta:
  version: x.y.z
  name: slack-form-alerts
  type: http-sink
  topic: form-events
  secrets:
    - name: SLACK_USER_ALERTS
http:
  endpoint: "https://hooks.slack.com/services/${{ secrets.SLACK_USER_ALERTS }}"
  headers:
    - "Content-Type: application/json"
transforms:
  - uses: infinyon/jolt@x.y.z
    with:
      spec:
        - operation: shift
          spec:
            "formatted": "text"

The sink connector reads from the form-events topic and uses the jolt smartmodule to shift the formatted string into a field called text per the Slack instructions. Checkout fluvio-jolt in github for additional information.

Add Slack webhook token to InfinyOn Secrets​

The Slack webhook link is sensitive information, let's add the Access Token part to secret in InfinyOn Cloud :

$ fluvio cloud secret set SLACK_USER_ALERTS <webhook-token>

Check out Slack Webhooks on how to create the webhook token.

Download SmartModules​

Download the smartmodules used by the webhook ad the connector:

$ fluvio hub download infinyon/jolt@x.y.z
$ fluvio hub download infinyon-labs/json-formatter@x.y.z

Check fluvio smartmodule list to ensure they've been downloaded.

Start Webhook and Connector​

Start webhook listener:

$ fluvio cloud webhook create --config form-webhook.yaml

Check fluvio cloud webhook list to ensure it has been successfully provisioned. In checkout the webhook link that we'll use to test the pipeline: https://infinyon.cloud/webhooks/v1/[token]

Start sink connector:

$ fluvio cloud connector create -c slack-form-alerts.yaml

Check fluvio cloud connector list to ensure it has been successfully provisioned.

Test Data Pipeline​

Use curl to send a POST request with a fictitious user request to our webhook link. In production environments, this iw what a website would send:

$ curl -X POST https://infinyon.cloud/webhooks/v1/<token> \
  -H "Content-Type: application/json" \
  -d '{ "email": "alice@acme.com", "name": "Alice Liddell", "type": "subscribe", "source": "front-page" }'

The following alert is displayed in Slack:

`📢 Alice Liddell ("alice@acme.com) subscribed on front-page` will show-up in your slack channel.

That's all folks!

Reference​