Summary of PhD

Scale-out big data processing frameworks like Apache Spark have been designed to use the off the shelf machines where each machine has the modest amount of compute , memory and storage capacity. Recent advancement in the hardware technology motivate to revisit the definition of commodity machines. In this context, the thesis quantifies the impact of bottlenecks limiting the multi-core performance of Apache Spark on the modern scale-up server using empirical evaluation. It also quantifies the efficacy of hardware architectural features such as NUMA-awareness, Simultaneous Multi-threading, Hardware Prefetching and DRAM frequency in improving the performance bottlenecks. Based on the characteristics of workloads, it envisions near-memory and near storage hardware acceleration to improve the single-node performance of scale-out frameworks like Apache Spark and shows significant performance gain by cache-coherent DRAM accelerators.