Fetching OpenSky airspace to Kafka with Apache NIFI

The OpenSky Network makes air traffic data public. It is a community-based receiver network that publishes data realtime. In this post we show, how to fetch the data with Apache NIFI and push it into a Kafka cluster.

OpenSky Data and its API

We will use OpenSky’s REST API to fetch the data. Requesting an URL:

https://opensky-network.org/api/states/all?lamin=..

fetches JSON of a form:

{
  "time": 1541257080,
  "states": [
    [
      "4b1801",  # Unique ICAO 24-bit address of the transponder 
      "SWR1663 ", # Callsign of the vehicle (8 chars). 
      "Switzerland", # Country name from the ICAO 24-bit address.
      1541257078, # Unix timestamp (seconds) for the last position 
      1541257078, # Unix timestamp (seconds) for the last update in general
      8.5559, # Longitue
      47.456, # Latitude
      ...
    ], 
    ...

Additionally, API request parameters: lamin, lomin, lamax, lomax allow to filter an area for the records fetched.

Nifi Workflow

We create Nifi Workflow that fetches data from REST API and pushes records into Apache Kafka. The workflow will be created based on three Apache Nifi Processors:

  • GetHTTP – to fetch data over HTTP,
  • SplitJson – to extract separate records from a response,
  • PutKafka – to publish messages to Kafka.opensky1

The workflow can be scheduled each 30 seconds.

GetHTTP

HTTPS connection is required. To make it work, we need to create StandardRestrictedSSLContextService, generate client keystore:

keytool -keystore clientkeystore -genkey -alias client

and define path to it within service properties:

opensky2.png

SplitJson

This one requires proper JSONPath expression to explode single JSON record into multiple entries:

opensky3.png

PutKafka

The processor requires to specify known Kafka brokers and topic name where records will be published:

opensky4.png

Summary

At the end, we can check messages on a Kafka cluster with:

kafka-console-consumer --topic opensky --bootstrap-server localhost:9092

["400b82","BAW71P  ","United Kingdom",1541243529,1541243530,9.3172,45.8576,11567.16,false,237.81,327.99,0,null,11757.66,"0205",false,0]
["49d307","TVS2146 ","Czech Republic",1541243529,1541243529,7.4909,46.8319,6103.62,false,201.51,51.84,0,null,6301.74,"1000",false,0]
["44083c","EZY24NB ","Austria",1541243530,1541243530,10.0189,45.847,8130.54,false,212.59,79.26,6.5,null,8366.76,"0464",false,0]
...

which means our workflow works. It took us a couple of minutes to create it without a single line of code written.  This shows the power of Apache NIFI.

If you find it interesting, have some Apache Nifi related questions or need a custom processor, feel free to contact.

 

 

Leave a Reply