Techniques for Malware Detection and Classification Using Machine Learning Classifiers

Abstract

The increasing sophistication of malware poses significant challenges to cybersecurity systems worldwide. This study presents a machine learning-based approach for the detection and classification of malware using behavioral features extracted from sandbox environments. Specifically, API call patterns were analyzed to represent the behavior of benign and malicious software. Three machine learning algorithms, Support Vector Machines (SVM), Naive Bayes, and Random Forest, were implemented and evaluated on a dataset comprising ten malware families and benign files. The evaluation was conducted under both multi-class and binary classification settings. Results revealed that the Random Forest algorithm outperformed others, achieving 95.69% accuracy for multi-class classification and 96.8% accuracy for binary classification, with minimal false positives and false negatives. SVM also delivered a robust performance, especially in binary classification, where it recorded zero false negatives. In contrast, Naive Bayes exhibited poor classification performance, mainly due to its assumption of feature independence, which proved unsuitable for the correlated nature of API features.

The findings underscore the effectiveness of ensemble-based models, particularly Random Forest, for behavioral malware detection. This work highlights the critical role of feature representation and algorithm selection in building reliable and scalable malware detection systems.