Visualize fraudulent transactions via CEP with Kafka, Flink, SQL, D3.js and Mapbox.

Apr 8, 2020

Visualizing fraudulent transactions via heatmaps using Mapbox GL JS

I am not too proud to admit it — I like writing code in javascript. I’m a novice, but along with Python it’s my go-to language given a task at hand needing completion.

ObservableHQ.com describes itself as “Observable is a magic notebook for exploring data and thinking with code.” — and I tend to agree. Put another way, it’s javascript notebooks with D3. If you are familiar with D3.js and javascript, but want to have the usability and exploration characteristics of a notebook (think Jupyter), then Observable is for you. In this example, I will be using synthetic Kafka data generated via the Eventador Examples repo, and will be pulling that data into Observable and using it to create a heatmap of fraudulent transactions using Complex Event Processing in SQL.

Let’s get into it. Detailed instructions are in the repo, I will cover the highlights here.

First let’s build a streaming data pipeline

In this example, I assume you are using the Eventador Platform and Continuous SQL to identify fraudulent transactions via CEP, then we visualize them in the notebook. First I generate fake transactions (including fraudulent ones) via a python script running in a docker container: