Apache Spark is a well established tool in Big Data pipelines. There are open source as well as several commercial offerings for Spark. With such widespread usage, Spark is also often misused. Valuable customer dollars and resources are wasted due to poorly configured Spark clusters. Current profiling solutions for Spark lack :

• Full insights into resource wastage and scope for optimization

• Wide availability

• Completeness

ZettaProf was built to address users’ requirements for building optimized Spark solutions. Over a period of time, it has evolved into a mature product which can be used by novice and expert Spark developers alike. Based on the insights offered by ZettaProf, we have been able to provide 10-100x performance improvements to our partner customers. Key capabilities which differentiate ZettaProf over other solutions:

• Works with any Spark distribution : Open Source, Databricks, Azure Synapse, AWS EMR, nVidia RAPIDs (on cloud or on-prem)

• Most comprehensive profiler with rich set of features and recommendations

• Zero touch deployment with no change to existing Spark code or queriesZettaProf Dashboard provides a bird’s eye view of Spark application by giving a summary of performance, report on resource utilization and recommendations for improvement. Users can deep-dive into fine-grained details to further look at various aspects (CPU, RAM, disk writes/reads, stage/operator runtimes etc.) of the full application as well as individual queries.

  ZettaProf sits alongside the Spark UI and a click on “ZettaProf” tab will take the user to a dashboard as in the picture below: