The era of high-variety data sources

According to Gartner definition, Big Data stands also for high-variety of data sources and requires innovative forms for data pipelines. In this blog we focus on pipelines that ingest and refine data from various storage technologies into a single Big Data system where we can merge data, extract insights and create machine learning algorithms. 

Big Data stands also for high-variety of data sources
and requires innovative forms for data pipelines.

Whatever our business is, there is plenty of data outside an organisation that may be extremely valuable. These may social media data from like Facebook posts on our official profile or profiles of the competitors. These may be also weather and climate data because this surely affects business performance. It may be any of the awesomedata – the awesome list of publicly available datasets.

Public datasets ingested and refined into your ecosystem can boost Big Data revolution at your company. The only missing gap is the ability to easily pick the data up from any source it is stored. Apache Nifi is the technology that fills that gap. It allows to create rich pipelines within Web Interface by dragging and dropping ready-to-go components. Those components are called Nifi processors. Creating pipeline with Apache Nifi does not require to write any single line of code.

Creating pipeline with Apache Nifi
does not require to write any single line of code.

Building a pipeline, as if it was LEGO construction, is great but it also requires Nifi components that perfectly match our needs. Apache Nifi comes with hundreds of components that fit several usecases well.

In case You find out they do lack something, we can help you with that.

Leave a Reply