Driven by specialized analytics systems and  software, big data analytics can point the way to various business benefits, including new revenue opportunities, more effective marketing, better customer service, improved operational efficiency and competitive advantages over competitors.

Big data analytics applications enable data scientists, predictive modelers, statisticians and other analytics professionals to analyze growing volumes of structured transaction data, plus other forms of data that are often left untapped by conventional business intelligence (BI) and analytics programs. That encompasses a mix of semi-structured and unstructured data — for example, internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile-phone call-detail records and machine data captured by sensors connected to the internet of things.

On a broad scale, data analytics technologies and techniques provide a means of analyzing data sets and drawing conclusions about them to help organizations make informed business decisions. BI queries answer basic questions about business operations and performance. Big data analytics is a form of advanced analytics, which involves complex applications with elements such as predictive models, statistical algorithms and what-if analyses powered by high-performance analytics systems.

The term big data was first used to refer to increasing data volumes in the mid-1990s. In 2001, Doug Laney, then an analyst at consultancy Meta Group Inc., expanded the notion of big data to also include increases in the variety of data being generated by organizations and the velocity at which that data was being created and updated. Those three factors — volume, velocity and variety — became known as the 3Vs of big data, a concept Gartner popularized after acquiring Meta Group and hiring Laney in 2005.

Separately, the Hadoop distributed processing framework was launched as an Apacheopen source project in 2006, planting the seeds for a clustered platform built on top of commodity hardware and geared to run big data applications. By 2011, big data analytics began to take a firm hold in organizations and the public eye, along with Hadoop and various related big data technologies that had sprung up around it.Initially, as the Hadoop ecosystem took shape and started to mature, big data applications were primarily the province of large internet and e-commerce companies, such as Yahoo, Google and Facebook, as well as analytics and marketing services providers. In ensuing years, though, big data analytics has increasingly been embraced by retailers, financial services firms, insurers, healthcare organizations, manufacturers, energy companies and other mainstream enterprises.

Big data analytics technologies and tools

Unstructured and semi-structured data types typically don’t fit well in traditional data warehouses that are based on relational databases oriented to structured data sets. Furthermore, data warehouses may not be able to handle the processing demands posed by sets of big data that need to be updated frequently — or even continually, as in the case of real-time data on stock trading, the online activities of website visitors or the performance of mobile applications.

As a result, many organizations that collect, process and analyze big data turn to NoSQL databases as well as Hadoop and its companion tools, including:

  • YARN: a cluster management technology and one of the key features in second-generation Hadoop.
  • MapReduce: a software framework that allows developers to write programs that process massive amounts of unstructured data in parallel across a distributed cluster of processors or stand-alone computers.
  • Spark: an open-source parallel processing framework that enables users to run large-scale data analytics applications across clustered systems.
  • HBase: a column-oriented key/value data store built to run on top of the Hadoop Distributed File System (HDFS).
  • Hive: an open-source data warehouse system for querying and analyzing large datasets stored in Hadoop files.
  • Kafka: a distributed publish-subscribe messaging system designed to replace traditional message brokers.
  • Pig: an open-source technology that offers a high-level mechanism for the parallel programming of MapReduce jobs to be executed on Hadoop clusters.

In some cases, Hadoop clusters and NoSQL systems are being used primarily as landing pads and staging areas for data before it gets loaded into a data warehouse or analytical database for analysis, usually in a summarized form that is more conducive to relational structures.

More frequently, however, big data analytics users are adopting the concept of a Hadoop data lake that serves as the primary repository for incoming streams of raw data. In such architectures, data can be analyzed directly in a Hadoop cluster or run through a processing engine like Spark. As in data warehousing, sound data management is a crucial first step in the big data analytics process. Data being stored in the Hadoop Distributed File System must be organized, configured and partitioned properly to get good performance on both extract, transform and load (ETL) integration jobs and analytical queries.

Once the data is ready, it can be analyzed with the software commonly used in advanced analytics processes. That includes tools for data mining, which sift through data sets in search of patterns and relationships; predictive analytics, which build models for forecasting customer behavior and other future developments; machine learning, which tap algorithms to analyze large data sets; and deep learning, a more advanced offshoot of machine learning.

Text mining and statistical analysis software can also play a role in the big data analytics process, as can mainstream BI software and data visualization tools. For both ETL and analytics applications, queries can be written in batch-mode MapReduce; programming languages, such as R, Python and Scala; and SQL, the standard language for relational databases that’s supported via SQL-on-Hadoop technologies.

Big data analytics uses and challenges

Big data analytics applications often include data from both internal systems and external sources, such as weather data or demographic data on consumers compiled by third-party information services providers. In addition, streaming analytics applications are becoming common in big data environments, as users look to do real-time analyticson data fed into Hadoop systems through Spark’s Spark Streaming module or other open source stream processing engines, such as Flink and Storm.

Home

Downloads

You are looking for answers. In our downloads area you’ll find plenty!

Big Data

What is Big Data? How can Big Data help you make better decisions?

Career

We are constantly looking for talent. Are you up for the challenge?

DNAVIOS

What, where, who we are

Our Team

Stefan H. RummelManaging Partner & Founder+
Living  Consulting 4.0   -   Stefan founded DNAVIOS in 2017 with the dedication to integrate Big Data analytics into the classic consultingRead more
Alexander LudorfPartner+
With his vast experience in machinery and machine tools industries Alexander is a powerhouse of knowledge. Located in Luxembourg he is wellRead more
Hadoop SpecialistBig Data Analytics+
We hire in Barcelona    -    We are constantly looking for talent. Specifically we are looking for a person which is experienced in  Hadoop,Read more
Business Analyst/ConsultantM&A and Strategy Consulting+
We hire in Barcelona     -    We are constantly looking for talent. In Barcelona we are looking for Business Analysts and Consultants for ouRead more

Our Services

Big Data Analytics

We offer you the full range of Big Data analytics
Learn more

M&A

Commerical Due Diligences, Vendor Due Diligences or Company Valuation. If you are looking for an experienced partner for your M&A processes we should talk!
Learn more

Strategy

Being ahead of your competition, outgrowing the market, develop market defining innovations. We are helping our customers to do exactly that! Functional competency paired with technological know-how and creativity is what we stand for
Learn more

Our Portfolio

Contact Us