Performance comparison of machine learning models on a novel in-vehicle controller area network dataset

Abstract

The advancement of technology has significantly enhanced comfort and welfare across all aspects of life, particularly in the field of transportation. One notable development is the growing adoption of autonomous vehicles, driven by the integration of smart systems into automobiles. However, the sophisticated systems and networks within autonomous vehicles have also opened new avenues for cyberattacks. These attacks typically aim to achieve one of three objectives: gaining unauthorized control of system components, overloading the system network to slow its operation, or causing a system crash. The potentially severe consequences of such cyberattacks have underscored the urgent need for robust security measures to protect autonomous vehicles. This study focuses on detecting cyberattacks targeting in-vehicle networks of smart vehicles using machine learning models. A simulation environment was developed to generate cyberattack scenarios, resulting in the creation of a dataset. This dataset was then analyzed using classification algorithms, including XGBoost, LightGBM, Random Forest, and Decision Trees. Performance comparisons revealed that XGBoost achieved the highest accuracy at 86.22% and F1 Score at 79.7%, while the Decision Tree algorithm had the lowest accuracy at 80.7% and F1 Score at 72.5%. In addition, the LightGBM algorithm had an accuracy rate of 85.83% and the Random Forest algorithm had an accuracy rate of 85.84%. The findings of this study are expected to contribute to the efforts of smart vehicle security experts in mitigating cyber threats and raising awareness about the importance of cybersecurity in autonomous vehicles.