Why Spark Data Strategy Architecture Advisory Data Engineering Data Science Technologies

Let's start a conversation
(or send a mail to hello@knoldus.com with your message.)

Apache Spark is a powerful open source processing engine built around speed, ease of use, and sophisticated analytics. It was originally developed at UC Berkeley in 2009.

The Spark engine runs in a variety of environments, from cloud services to Hadoop or Mesos clusters. It is used to perform ETL, interactive queries (SQL), advanced analytics (e.g. machine learning) and streaming over large datasets in a wide range of data stores (e.g. HDFS, Cassandra, HBase, S3). Spark supports a variety of popular development languages including Java, Python and Scala.

Since its release, Spark has seen rapid adoption by enterprises across a wide range of industries. It has quickly become the largest open source community in big data, with over 400 contributors from 100+ organizations.

Simple

Spark provides easy-to-use APIs for operating on large datasets. This includes a collection over 80 operators for transforming data and familiar data frame APIs for manipulating semi-structured data.

Fast

Spark provides easy-to-use APIs for operating on large datasets. This includes a collection over 80 operators for transforming data and familiar data frame APIs for manipulating semi-structured data.

A Unified Engine

Spark is packaged with higher level libraries, including support for SQL queries, streaming data, machine learning and graph processing. These standard libraries increase developer productivity and can be seamlessly combined to create complex workflows.

General Execution: Spark Core

Spark Core is the underlying general execution engine for the Spark platform that all other functionality is built on top of. It provides in-memory computing capabilities to deliver speed, a generalized execution model to support a wide variety of applications, and Java, Scala, and Python APIs for ease of development.

Structured Data: Spark SQL

Many data scientists, analysts, and general business intelligence users rely on interactive SQL queries for exploring data. Spark SQL is an engine for Hive data that enables unmodified Hadoop Hive queries to run up to 100x faster on existing deployments and data. It also provides powerful integration with the rest of the Spark ecosystem (e.g., integrating SQL query processing with machine learning).

Streaming Analytics: Spark Streaming

Many applications need the ability to process and analyze not only batch data, but also streams of new data in real-time. Running on top of Spark, Spark Streaming enables powerful interactive and analytical applications across both streaming and historical data, while inheriting Spark's ease of use and fault tolerance characteristics. It readily integrates with a wide variety of popular data sources, including HDFS, Flume, Kafka, and Twitter.

Machine Learning: MLlib

Machine learning has quickly emerged as a critical piece in mining Big Data for actionable insights. Built on top of Spark, MLlib is a scalable machine learning library that delivers both high-quality algorithms (e.g. multiple iterations to increase accuracy) and blazing speed (up to 100x faster than MapReduce). The library is usable in Java, Scala, and Python as part of Spark applications, so that you can include it in complete workflows.

Graph Computation: GraphX

GraphX is a graph computation engine built on top of Spark that enables users to interactively build, transform and reason about graph structured data at scale. It comes complete with a library of common algorithms.

Other Projects

One of the benefits of Spark's vibrant open-source community is continued innovation that helps extend Spark's capabilities, many of which originated in UC Berkeley's AMPLab. Here is a sampling of some on-going projects in the community (that are still in alpha):

BlinkDB

An approximate query engine for interactive SQL queries in Shark that allows users to trade-off query accuracy for response time. This enables interactive queries over massive data by using data samples and presenting results annotated with meaningful error bars.

Spark R

A package for the R statistical language that enables R-users to leverage Spark functionality interactively from within the R shell. It provides a light-weight frontend to use Apache Spark from R. It exposes the Spark API through the RDD class and allows users to interactively run jobs from the R shell on a cluster.