Open research questions in Advanced Malware Detection Techniques

What the literature leaves open

• A federated multimodal framework for malware fam- ily classification under non-IID client distributions was presented. By combining byte-level images with disas- sembly-derived token sequences, and by introducing a bidirectional cross-attention module with adaptive resid- ual fusion, each client learns a stable cross-modal repre- sentation while keeping raw data local. Adapting FedBN to this multimodal architecture further reduces the nor- malization mismatch caused by skewed label distribu- tions. Experiments on the Microsoft BIG 2015 dataset show that the proposed framework achieves competitive performance against representative unimodal and mul- timodal baselines, with stable convergence under highly heterogeneous client partitions. The security-oriented evaluation further shows that the framework retains useful robustness under moder- ate targeted and untargeted label-flipping attacks, while high poisoning ratios still degrade performance. Coordi- nate-wise median aggregation provides a supplementary defense under untargeted label flipping, but FedBN is designed for statistical heterogeneity rather than mali- cious behavior, and the current design should not be regarded as a complete defense against data poisoning, model-weight poisoning, backdoor attacks, or Byzantine clients.

  Federated multimodal malware classification under non-IID data · 2026 · DOI

• Using machine learning for trap selection to detect ransomware is an emerging area of research. Most related works focus on heuristic-based methods for selecting trap files, which often fail when ransomware employs multi-threading and 16 P. Mohan Anand et.al random encryption orders. In this study, we explore non-parametric clustering methods, including Affinity Propagation, GMM, Mean Shift, and Optics, to select traps at endpoints. We evaluate these selected traps to identify the most effective method that minimizes file loss and detection delay without causing significant performance overhead. Our analysis reveals that Affinity Propagation is the most suitable machine learning method for trap selection and can be easily deployed at endpoints compared to other methods. It resulted in a minimal file loss of 0.62% on 20,558 files, with a performance overhead of 22.3MB. However, ransomware variants like Babuk and AvosLocker, which follow alphabetical and reverse alphabetical file encryption orders, can significantly delay machine learning-based trap selection methods, leading to substantial file loss. To address this issue, we propose APFO—a combination of Affinity Propagation with file name order. APFO leverages both the machine learning-based trap selection method of Affinity Propagation and the heuristics of alphabetical and reverse alphabetical file ordering to select files across directories. APFO performed significantly better than Affinity Propagation alone, resulting in a very minimal file loss of 0.32%, a detection delay of 1.03 seconds, and a performance overhead of 24.5MB on a given endpoint. This makes APFO highly effective for deployment at endpoints to counteract even faster ransomware encryption methods. Future work will focus on pre-encryption scenarios of ransomware execution, aiming to identify and learn behavioral patterns to further improve early detection.

  A Comprehensive Analysis of Machine Learning Based File Trap Selection Methods to Detect Crypto Ransomware · 2026 · DOI

• This paper introduced a lightweight and useful disk-level anomaly detection framework to virtual machine backup settings called the Semantic-Aware File Metadata Generation Framework (SA-FMGF). SA-FMGF uses only file-system and disk-level metadata to operate and therefore does not require inspection of raw file content, and still achieves high detection and low computational overhead. The framework combines semantic abstraction, structural entropy analysis, temporal modification modeling, and block-level consistency indicators aware of meaningful disk behavior patterns to detect anomalies at an early stage. Experimental analysis of the DARPA Transparent Computing dataset proved that SA-FMGF has 98.7 detection accuracy, a false positive rate of 1.9, and metadata storing overhead is about 42 times less than traditional metadata-based methods. These findings affirm the strength, scalability, and feasibility of the given model for real-world backup monitoring. The semantic-inspired design of SA-FMGF not only increases the detection performance but also increases interpretability and anomaly localization, thus improving faster forensic evaluation and recovery planning. The decentralized structure makes it resistant to the never-before-seen variants of ransomware and new forms of attacks. SA-FMGF can be upgraded to incorporate the concept of adaptive learning in order to fine-tune the semantic representation dynamically in response to long-term behavior patterns as future work. R e f e r e n c e s 1. A l r a y e s, F. S., M. Z a k a r i a h, S. U. A m i n, Z. I. K h a n, M. H e l a l. Intrusion Detection in IoT Systems Using a Denoising Autoencoder. – IEEE Access, Vol. 12, 2024, pp. 122401-122425. 2. B i s h t, P. S., P. M i s h r a, P. C h a u h a n, R. C. J o s h i. HyperGuard: оn Designing an Out-Of- VM Malware Analysis Approach to Detect Intrusions from the Hypervisor in a Cloud Environment. – International Journal of Grid and Utility Computing, Vol. 14, 2023, No 4, pp. 356-367. 3 . G o r o k h o v, O., M. P e t r o v s k i y, I. M a s h e c h k i n, M. K a z a c h u k. Fuzzy CNN Autoencoder for Unsupervised Anomaly Detection in Log Data. – Mathematics, Vol. 11, 2023, No 18, 3995. 4 . Z h a n g, H., W. Z h o u. A Two-Stage Virtual Machine Abnormal Behavior-Based Anomaly Detection Mechanism. – Cluster Computing, Vol. 25, 2022, No 1, pp. 203-214. 5 . H u s e y n o v, H. Anomaly-Based Intrusion Detection System Through Remote Virtual Machine Introspection. PhD dissertation. The City College of New York, 2023. 6 . J i a n g, G. Artificial Intelligence-Based Adaptive Anomaly Detection Technology for IaaS Cloud Virtual Machines. – Journal of Engineering and Applied Science, Vol. 71, 2024, No 1, 102. 7 . L u, S ., N.

  A Semantic-Aware File Metadata Generation Framework for Disk-Level Anomaly Detection in Virtual Machine Backups · 2026 · DOI

• Future work will explore: (1) integration of dynamic behavioral features (API call sequences, network traffic patterns) alongside static permission analysis; (2) adversarial robustness training to counter obfuscated malware; (3) expansion of the Twitter-based update pipeline to include other threat intelligence sources such as VirusTotal and MISP feeds; and (4) deployment on cloud platforms for scalable, real-time protection at the enterprise level.

  Android Malware Detection Using Machine Learning: A Hybrid Approach with Twitter-Based Hash Updates and Deep Learning Classification · 2026 · DOI

• While our approach demonstrates promising results, several limitations should be acknowledged. First, the model shows strong overall AUC and AUCPR, yet strug- gles to maintain low false positive rates at very high recall thresholds (e.g., FPR above 0.4 when TPR ≥ 95%), limit- ing its immediate applicability in high sensitivity deploy- ment settings. Second, the malware family labels in our dataset are unevenly distributed and in many cases not well characterized, leaving gaps in understanding the similarities among malware classes and their proximity to benign software. This lack of granularity makes it dif- ficult to assess whether high anomaly scores reflect true malicious behavior or benign outliers that share surface level patterns with malware. Third, by design our pipe- line emphasizes simplicity and reproducibility (minimal preprocessing, straightforward image conversion, and a single GAN-based model). While this design achieves comparability with more complex baselines, it inevitably sacrifices some performance and robustness that could be gained from multi-stage or hybrid systems. Fourthly, our evaluation does not fully explore model explainability: although reconstruction errors and feature discrepancies provide anomaly scores, they do not offer interpretable insights into which regions or features of a binary most strongly contribute to detection decisions. In addition, per family variability on the Microsoft dataset reveals texture bias and packing sensitivity, with strong results on families such as Kelihos V1 but weaker detection on Simda or Kelihos V3, suggesting domain shift between self collected (2018–2023) and older curated samples. Label noise and ground truth uncertainty from public feeds (e.g., PUPs, multi label overlaps) may also confound transfer evaluations. Temporal drift further limits gen- eralization, as models trained on recent threats may fail on legacy malware, while our imaging approach, though Wijayasiri et al. Cybersecurity (2026) 9:165 Page 15 of 16 preserving locality through Hilbert mapping, imposes hand crafted assumptions that underrepresent sparse yet semantically important code regions. Performance also shows hyperparameter and seed sensitivity, and calibra- tion remains imperfect at high recall thresholds. Moreo- ver, we were not able to conduct a robustness evaluation against packing or evasion based perturbations. We emphasize that robustness to adaptive adversarial manip- ulation remains an open problem for all static malware detectors, including both deep learning and signature- based systems.

  Enhanced Consistency Bi-directional GAN (CBiGAN) for malware anomaly detection · 2026 · DOI

• Future research should explore integrating richer con- textual knowledge of malware families, including inter- family similarity measures and benign malicious overlap, to better calibrate anomaly detection thresholds and reduce false positives. Explainability is another key direc- tion. Visual saliency maps, feature attribution techniques, or latent space disentanglement could help us under- stand why a sample is flagged as anomalous. Moreover, calibration strategies (e.g., temperature scaling, isotonic regression) and threshold optimization tuned to resource budgets could improve deployment reliability in low false positive regimes. Also, while our current pipeline emphasizes simplicity, future iterations could combine our imaging approach with dynamic features, ensem- ble methods, or hybrid static dynamic frameworks. This could help balance interpretability, computational effi- ciency, and detection performance. Beyond these, time aware evaluation and continual learning could mitigate temporal drift, while class or cluster aware thresholds may address per family variance in the Microsoft dataset.

  Enhanced Consistency Bi-directional GAN (CBiGAN) for malware anomaly detection · 2026 · DOI

• Although family-wise validation was conducted to mitigate memorization effects, evaluation across multiple independent datasets was beyond the scope of this work. This study is limited to the CIC-MalMem- 2022 dataset, which focuses on memory-resident malware behaviors.

  Interpretable Memory-behavior Modeling for Malware Detection Using Additive Explainable Learning · 2026 · DOI

• Android's adaptability and ease of use have caused it to swiftly rise to the top of the mobile operating system market. The vast majority of persistent malevolent attacks also target it. This necessitates the quick installation of a strong malware detection system. With an accuracy of 99.16%, precision of 99.98%, recall of 99.87%, and F1score of 99.76%, this study's CNN+RNN hybrid model for Android malware classification achieves impressive results on the CIC-InvesAndMal2019 dataset. The suggested model proves in cybersecurity by surpassing more traditional ML algorithms like DT, RF, and SVM. However, challenges such as overfitting, indicated by fluctuations in validation loss, and high computational demands during model training, remain. To address these limitations, future work should focus on enhancing model regularization techniques, optimizing training procedures for faster convergence, and utilizing transfer learning to improve scalability.

  Deep Learning Ensemble Model for Precise and Robust Android Malware Detection · 2026 · DOI

• In this paper we identified two challenges for identifying IoT firmware versions: the on-wire behaviour resulting from different firmware versions running on the same device are more subtle than those for different device models, types or manufacturers; and there is limited data on different device versions due to lack of publicly available datasets. These challenges are novel in this area and have therefore not previously been addressed by the literature. We developed a technique using flow statistics to transform on-wire device behaviours into greyscale images. These images were fed into a Twin Neural Network model to output similarity scores. Our best performing model was 95.83% accurate at identifying stable versions and 84.38% accurate identifying version changes. By calculating the Hedges’ g effect size of similarity scores, we were able to detect the subtle changes resulting from a device running different firmware versions. We showed how Hedge’s g as a metric is approximately 20% more accurate than the standard TNN measure. By using a TNN model trained on device differences — but tested on version changes — we were able to overcome the limited data problem via transfer learning. This allowed the TNN to be able to correctly calculate the similarity of images that were truly unknown - the images of different device versions had not been used to train the TNN. To improve our technique, further work could be carried out in this area, including: (1) Using active approaches. Although stated to not be ideal for the reasons mentioned in the introduction, active approaches may have a place in augmenting techniques when identifying device versions to overcome the problem when a device has no change in passive on-wire signature. This could either be from banner grabbing on known ports (as the version change may update versions or banners of extra components such as libraries), or logging into device management pages and identifying version strings. (2) Using this technique to identify newly added devices, or expanding to a huge database of known devices as an identification technique for unknown devices. (3) We have shown our technique can identify anomalies in terms of version changes, there would be value in applying the same technique for other types of anomaly detection from compromised devices. REFERENCES [1] Georgios Agrafiotis, Eftychia Makri, Ioannis Flionis, Antonios Lalas, Konstantinos Votis, and Dimitrios Tzovaras. 2022. Image-based Neural Network Models for Malware Traffic Classification using PCAP to Picture Conversion. 17th International Conference on Availability, Reliability and Security. [2] Amazon Web Services. 2024. AWS IoT Device Defender FAQs. https://aws.amazon.com/iot-device-defender/faq/ [Accessed: 07.04.2024]. [3] Nesrine Ammar, Ludovic Noirie, and Sebastien Tixeuil. 2020. Autonomous Identification of IoT Device Types based on a Supervised Classification. International Conference on Communications (ICC). [4] Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, and Thomas Cattermole. 2023. Granular IoT Device Identification Using TF-IDF and Cosine Similarity. Proceedings of the 5th Workshop on CPS&IoT Security and Privacy. , 9 pages. [5] Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, and Thomas Cattermole. 2023. Keyword Extraction for Fine-Grained IoT Device Identification. 2022 Seventh International Conference on Fog and Mobile Edge Computing (FMEC). [6] Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, and Thomas Cattermole. 2023. Reliable Identification of IoT Devices from Passive Network Traffic Analysis: Requirements and Recommendations. World Forum on Internet of Things. [7] BBC. 2017. How hackers could use doll to open your front door. https://www.bbc.co.uk/news/av/technology-38966285. [Accessed: 07.04.2024]. [8] BBC. 2020. Smart doorbells ’easy target for hackers’ study finds. https://www.bbc.co.uk/news/technology-55044568. [Accessed: 07.04.2024]. [9] BBC. 2023. The tech flaw that lets hackers control surveillance cameras. https://www.bbc.co.uk/news/technology-65975446 [Accessed: 07.04.2024]. [10] Punam Bedi, Neha Gupta, and Vinita Jindal. 2020. Siam-IDS: Handling class imbalance problem in Intrusion Detection Systems using Siamese Neural Network. Procedia Computer Science. , 780-789 pages. [11] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature Verification using a "Siamese" Time Delay Neural Network. Advances in Neural Information Processing Systems. [12] Luigi Catuogno and Clemente Galdi. 2023. Secure Firmware Update: Challenges and Solutions. Cryptography 7, 2 (2023), 1–17. [13] Batyr Charyyev and Mehmet Hadi Gunes. 2021. Locality-Sensitive IoT Network Traffic Fingerprinting for Device Identification. IEEE Internet of Things Journal 8 (2021), 1272–1281. Issue 3.

  IoT Firmware Version Identification Using Transfer Learning with Twin Neural Networks · 2026 · DOI

• This research highlights that while Gated Recurrent Unit and Decision Tree models achieve state-of-the-art performance in malware detection on the EMBER dataset, with accuracies of 94% and 93% respectively, their transition from experimental benchmarks to practical deployment reveals critical limitations. Deep learning architectures, such as GRUs, incur substantial computational overhead, demanding 20–30 times more training time and exhibiting 15–20 times higher inference latency compared to less complex models like Decision Trees. This renders GRUs impractical for resource constrained environments, including endpoint devices, IoT systems, or real-time network gateways. Conversely, Decision Tree models offer rapid inference and minimal memory footprint but exhibit significant performance degradation under dataset shift, underscoring their dependence on the specific feature engineering of the EMBER dataset. This vulnerability compromises their robustness within diverse and evolving threat landscapes. Critically, both model types demonstrate high susceptibility to adversarial attacks, a factor not within the scope of this investigation but extensively documented in extant literature. Decision trees are vulnerable to attacks that manipulate influential features, often identified via methods such as L1 regularization or genetic algorithms. Recurrent architectures, including GRUs, can be compromised through gradient-based or malware-specific attacks that subtly alter API call sequences or byte entropy while preserving malicious functionality. Without dedicated adversarial hardening e.g., adversarial training, ensemble defenses, or input randomization deploying these models is inherently precarious, as their high accuracy on clean data can precipitously decline below 60%. Furthermore, the inherent opacity of GRU and XGBoost models impedes operational trust and incident response, as security analysts require interpretable, human-understandable rationales for detection decisions. Emerging eXplainable AI techniques, such as SHAP and attention mechanisms, offer promising avenues to mitigate this deficiency. To advance these high-performing models into robust, production-ready systems, future research should focus on four pivotal areas: Adversarial Hardening: Implementing adversarial training and metric learning ensembles to bolster robustness without significantly compromising clean-data accuracy. Efficiency Optimization: Employing model compression, quantization, and knowledge distillation to achieve GRU-level accuracy on edge devices. Explainability Integration: Incorporating modules like SHAP, LIME, or attention mechanisms to provide transparent and actionable insights for Security Operations Center analysts. Hybrid Architecture Design: Developing tiered detection systems that integrate lightweight machine learning models for initial triage with deep learning models for in-depth forensic analysis, thereby balancing speed, accuracy, and robustness. By transparently addressing these limitations and articulating a clear research roadmap, this study not only advances the state-of-the-art in malware detection but also provides a pragmatic, operationally grounded framework for developing the next generation of reliable, adaptive, and deployable cybersecurity systems.

  Advances in Malware Detection using Machine Learning and Deep Learning: A Comprehensive Comparative Analysis · 2026 · DOI

• This research leverages the Elastic Malware Benchmark for Empowering Researchers (EMBER) dataset to develop a lightweight, portable tactical decision aid that enables incident responders to rapidly determine whether a binary warrants further investigation.

  Static Malware Analysis for Incident Response: Developing a Tactical Aid with EMBER · 2025 · DOI

• The synthetic dataset was created from ESXi logs using an algorithm to mimic the pattern of real logs for machine learning testing, which may not capture the complexity and diversity of actual ransomware attacks.

  Behavioral analysis of ransomware threats to ESXi Hypervisors: a machine learning-based predictive model · 2026 · DOI

• Extreme gradient boosting has limitations such as high computational cost and hyper plane interpretation which make it limited in various cases, especially in detecting malware.

  Malware Detection Using Machine Learning Techniques · 2026 · DOI

• Random forest classification model has drawbacks such as less interpretation, the highest computational cost, and difficulty in potential biasing which makes it limited in several cases.

  Malware Detection Using Machine Learning Techniques · 2026 · DOI

• Scalability and performance characteristics of the weighted quorum scoring and adaptive challenge escalation mechanisms under high-volume attestation requests are not evaluated.

  Quorum Seal: Cross-Sensor Challenge and Response Attestation for Compromise Detection with Adaptive Multi-Surface Verification · 2026 · DOI

• The system does not promise perfect compromise diagnosis under every attacker model, especially against highly privileged adversaries who control the operating system or can instrument the app deeply.

  Quorum Seal: Cross-Sensor Challenge and Response Attestation for Compromise Detection with Adaptive Multi-Surface Verification · 2026 · DOI

• Development of malware detection systems that automatically and opportunistically rethink their detection measures on the basis of newly available data is a prerequisite for effective defense against new malware technologies.

  Efficient machine learning approach for classification of malware with portable executable (PE) headers · 2026 · DOI

• While much effort has been made to detect and measure the privacy leakage caused by the advertising (ad) libraries integrated in mobile applications, analytics libraries, which are also widely used in mobile apps have not been systematically studied for their privacy risks.

  Privacy Risk Analysis and Mitigation of Analytics Libraries in the Android Ecosystem · 2019 · DOI

Most-cited papers in Advanced Malware Detection Techniques

• Image-Based malware classification using ensemble of CNN architectures (IMCEC) · Computers & Security · 2020 · 347 citations
• Ransomware: Recent advances, analysis, challenges and future research directions · Computers & Security · 2021 · 217 citations
• Malware detection in industrial internet of things based on hybrid image visualization and deep learning model · Ad Hoc Networks · 2020 · 188 citations
• A dynamic Windows malware detection and prediction method based on contextual understanding of API call sequence · Computers & Security · 2020 · 183 citations
• Byte-level malware classification based on markov images and deep learning · Computers & Security · 2020 · 179 citations
• GDroid: Android malware detection and classification with graph convolutional network · Computers & Security · 2021 · 176 citations
• Enhancing malware detection through self-union feature selection using gray wolf optimizer · Indonesian Journal of Electrical Engineering and Computer Science · 2024 · 154 citations
• GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis · 2024 · 148 citations
• A Survey of IoT Privacy Security: Architecture, Technology, Challenges, and Trends · IEEE Internet of Things Journal · 2024 · 141 citations
• EfficientNet convolutional neural networks-based Android malware detection · Computers & Security · 2022 · 140 citations

Most recent work

• AI-Powered and Conventional Malware Detection Approaches: Challenges and Future Trends · Journal of Engineering Research and Education (JERE) · 2026
• Efficient machine learning approach for classification of malware with portable executable (PE) headers · Computer Science and Engineering Research · 2026
• Quorum Seal: Cross-Sensor Challenge and Response Attestation for Compromise Detection with Adaptive Multi-Surface Verification · Indian Journal of Computer Science and Technology · 2026
• Malware Detection Using Machine Learning Techniques · International Research Journal of Modernization in Engineering Technology & Science · 2026
• INTERNET BEHAVIOR OF MALWARE DETECTION · International Journal of Progressive Research in Engineering Management and Science · 2026
• Behavioral analysis of ransomware threats to ESXi Hypervisors: a machine learning-based predictive model · Journal of Computer Sciences Institute · 2026
• Machine Learning Based Malware Trend Analysis using Cyber Threat Intelligence Data · International Scientific Journal of Engineering and Management · 2026
• Advances in Malware Detection using Machine Learning and Deep Learning: A Comprehensive Comparative Analysis · International Journal of Computer Network and Information Security · 2026
• A Comparative Study of Federated Learning and Amino Acid Encoding with IoT Malware Detection as a Case Study · Big Data and Cognitive Computing · 2026
• App Risk Analyzer: Android-Based Application for Risk Scoring and Safe Uninstallation · International Journal for Research in Applied Science and Engineering Technology · 2026

Related topics in Computer Science