Fetching Bitcoin transactions with Apache NIFI

Bitcoin and its followers thrill finntech and many others. The cryptocurrency concept allows everyone to access all the history of transactions which has never happened to any currency before.

Transaction history can be fetched in several ways and NIFI is among the best tools that allow to create data pipeline smoothly without spending too much time on it.

blockchain.com allows API requests to fetch the data. The easier way to do so is to make use of Google public dataset bitcoin_blockchain which contains two tables: blocks and transactions. Tables are updated every 10 minutes.

We will grab the content with our custom ExecuteBigQueryProcessor. Its configuration requires only two parameters: Google service account credentials and query to be executed. The query parameter supports NIFI expression language  which may useful for specifying time interval of transactions we want to fetch. For example, we can run a daily cron processor schedule that executes a query:

select * from `bigquery-public-data.bitcoin_blockchain.transactions`
where timestamp between
${now():toNumber():minus(86400000):format("yyyy/MM/dd", "UTC"):toDate("yyyy/MM/dd", "UTC"):toNumber()}
and
${now():toNumber():format("yyyy/MM/dd", "UTC"):toDate("yyyy/MM/dd", "UTC"):toNumber()}

The expression

${now():toNumber():format("yyyy/MM/dd", "UTC")}

gets current date string based on current timestamp which is then parsed into Date. This allows us filter one full day of transactions.

The whole pipeline that stores data on disk may be as simple as:

nifi-bitcoint-putfile.png

At the end we end up with local files filled with JSON elements like the one below:

{
"timestamp":1531939208000,
"transaction_id":"fde93d7d51a977693efaac41ea5e5fdc0327d1dea9778abf6502ff812821270a",
"inputs":[{"input_script_string":"PUSHDATA(72)[30450221008dfa343e38672cb00559bea0308bbeac57ce5e3a4aae9667a23269d3a607326c022075116f5b77cea6e8445daba29f98bfe2db458a8437f375f0a580d10aec16e61101] PUSHDATA(33)[02a74b4340706978236e40ae70ee463728de914c325ae1808fb1b057c488a76249]","input_sequence_number":4294967295,"input_pubkey_base58":"15wsAvuYUWFmB8NjtXPhdUjWoLigmnr3fZ"}],
"outputs":[{"output_satoshis":8269,"output_script_string":"DUP HASH160 PUSHDATA(20)[d8286a48bfbfa97df6af0d9743e2af526020dc39] EQUALVERIFY CHECKSIG", <span id="mce_SELREST_start" style="overflow:hidden;line-height:0;">&#65279;</span>"output_pubkey_base58":"1LhwNXnaPqi6UhtCUFyTa46ZZb7eiioFDb"},{"output_satoshis":2740000,"output_script_string":"HASH160 PUSHDATA(20)[1cc21d105ab6cc1d230fe9600b0433d3ffb28c16] EQUAL",
"output_pubkey_base58":"34K5PAD4CRqMzKqM2HUFZdYN5th6aPZeyM"}],
"block_id":"0000000000000000002acb3228dddc94e67fa8e63fda905a05668ae49ffa8660",
"previous_block":"0000000000000000001f2c99f110fa1a09aca4bce27a8a59c720d2a8d9262433",
"merkle_root":"3986cf5ad9711e735ee5bd42ea6236c70acd64a16d9c777f272b54c12ce1b4e6",
"nonce":3272609194,
"version":536870912,
"work_terahash":222425622729
}

Leave a Reply