SkewSense: ML-Driven Partition Optimization for Distributed Systems

Abstract

Data skew in distributed big data systems causes significant performance degradation by concentrating data around specific key values, leading to overloaded nodes and underutilized resources. This problem is particularly severe in operations requiring data redistribution, such as joins, aggregations, and window functions. In this study, we propose a machine learning (ML)-driven decision system that automatically selects the appropriate partitioning strategy by analyzing data skew. The developed system uses summary statistical metrics of the dataset to choose among hash, range, and custom salting strategies and generates executable Apache Spark Scala code for the recommended structure. The performance of the proposed approach is evaluated in comparison with the PostgreSQL-based distributed database system Citus and the Apache Spark baseline configuration. Experiments are conducted on a synthetic dataset with controlled data skew using four different query scenarios. Results show that the ML-driven custom salting strategy achieves significant performance improvements over Spark baseline, especially in aggregation queries involving hot keys. However, performance gains are limited in window function queries due to additional shuffle overhead. The findings demonstrate that data-aware partitioning strategies can enhance performance in distributed data processing systems and that ML-driven decision mechanisms offer an effective solution in this process. © 2026 IEEE.