Enable Huawei to implement different functionalities and integration support with presto and hive in CarbonData

Knoldus Carbon data

Partner : Huawei Technologies.

Website : https://carbondata.apache.org/

Technologies Used : Scala, Java, Apache-Spark, Spark-Streaming, Presto, Hive, Hadoop, AWS S3

Domain : Data Storage and processing on Big Data

About CarbonData

Apache CarbonData is an indexed columnar data format for fast analytics on big data platform, e.g. Apache Hadoop, Apache Spark, etc. Knoldus enable Huawei to work in collaboration with them to implement different functionalities or integration support with different technologies including presto and hive in CarbonData. The below diagram illustrates CarbonData file structure:

Knoldus Carbon diagram

The Challenge

Huawei wants to explore a domain where backend, frontend, and continuos integration ensure backward compatibility of the older versions when the newer versions will be rolled out on a frequent basis. Knoldus worked along with Huawei Team to help CarbonData in becoming an Apache-licensed project from an incubating project.

Our Solution

Knoldus worked closely with the Huawei team and helped in building the crucial functionalities, some of which are listed below:

  • Development of Dictionary Generation Tool for CarbonData.
  • Pre-aggregate functionality to improve performance of aggregation queries.
  • CarbonData integration with Presto, Hive, Flink, and S3 technologies.
  • Setting up of continuous Integration via Jenkins.
  • Creation of Performance Testing tool to do benchmarking.
  • Achieving zero bugs with Automation Testing.
  • Development of Apache CarbonData website and its maintenance.
  • Automation of documentation from Git to website.
  • Development and enhancement in core packages of CarbonData.
  • Benchmarking CarbonData against available file formats like Parquet and ORC, against frameworks like Spark, Presto, and Impala and against different storage systems like Hadoop, S3 and Kudu.

Knoldus worked with Huawei to develop a file-format which is faster and efficient in processing and querying on big data. Now, Huawei clients able to speed up their system by utilizing the features of CarbonData.

Our team also developed a proprietary performance benchmarking tool for CarbonData. This benchmark tool tests the performance of the CarbonData in comparison with its competitors like Parquet and ORC Format. The key functionality supported by the Benchmark tool are as follows:

  • Generating the TPCH benchmarking data depending on the cluster size driven by configuration.
  • Defining workloads as a configuration for particular datasets.
  • Loading the data into all the formats into the Hive Store like CarbonData, Parquet, and ORC.
  • Configuration based Tuning for Spark that included parallelism settings as well as spark configuration based on different workloads.
  • Executing the workloads and capturing the response time and results with respect to load for all the formats.
  • Comparison of the results in all the formats.
  • Generating an Excel report showing the comparison of the results as well as success and failures of test execution.

Benefit

One of the CarbonData client - Hulu (one of the first users of CarbonData. It is a North American video industry internet company) used the CarbonData platform and able to filter out 2% to 5% of data for aggregation. The filters are majority based on 5 to 10 columns and the result set has 100+ columns. The finger grained index, columnar groups provided in CarbonData render speedier results than any other competitor platforms like Parquet/ ORC Format.

Get In Touch

If you are looking to build a similar platform or product with Big Data solution leveraging Spark, Knoldus is here to help. We are proven, experienced certified Databricks, Datastax and Lightbend Partner, available for partnering to make your product a reality. Get in touch with us or just send us an email on hello@knoldus.com

Stay Connected and Updated:

linkden newsletter
Trusted by innovative organizations, big and small.
Awards and Recognitions
2 Upcoming webinars:

Akka Clustering & Sharding made easy and Introduction to Akka Streams Know More!