Open research questions in Advanced Malware Detection Techniques
123 unresolved questions extracted from the limitations and future-work sections of 498 Advanced Malware Detection Techniques papers in our library. Each links back to the study that raised it.
What the literature leaves open
The evaluation was limited to 15 Linux-based MIPS firmware targets, - The study did not consider other types of firmware or architectures, - The study relied on a specific set of network interactions and protocol logic
Exploring the application of STAFF to other types of firmware or architectures, - Investigating the use of STAFF in conjunction with other fuzzing techniques, - Developing new methods for improving the efficiency and effectiveness of STAFF
The model was using the density as a discriminating factor, - The current work does not address generalization to Linux and Android malware, - The current work does not address binaries that exceed the 64^3 volume
3D volumetric malware detection using morton curves and multi-channel semantic features · 2026 · DOIThe need for a more effective and proactive static malware detection model. The limitations of current 2D visualization techniques for malware detection. The lack of a systematic evaluation of the generalization of malware detection models to unseen malware families.
3D volumetric malware detection using morton curves and multi-channel semantic features · 2026 · DOICentrality-aware protection and weighted-contact models are needed for precise estimation - Intervention levers and governance differ materially at the global internet level - The work focuses on organizational environments, while the same outbreak logic can describe internet-scale worm spread
The growing complexity of malware demands advanced detection techniques. Conventional mechanisms face great challenges compared to the growing malware threats. There is a need for improved detection techniques to protect internet users' privacy.
Signature-based detection algorithms are inherently unable to identify new zero-day malware. There is a need for more adaptive methods. Ransomware poses a unique threat which requires a whole new detection protocol.
Efficient machine learning approach for classification of malware with portable executable (PE) headers · 2026 · DOIA single software-only check can be hidden, replayed, or forged by a capable attacker. There is a need for a trust-attestation framework that can evaluate whether a device is trustworthy enough for sensitive actions.
Quorum Seal: Cross-Sensor Challenge and Response Attestation for Compromise Detection with Adaptive Multi-Surface Verification · 2026 · DOITo improve the performance of the proposed model. To apply the proposed model to other cyber security problems. To compare the proposed model with other machine learning techniques.
There is a need for a more accurate malware detection model. Existing models have limitations such as high computational costs and difficulty in parameter tuning.
Detecting newly emerging or unknown threats. Analyzing application behavior in real-time. Providing a comprehensive dashboard for visualizing threat statistics and network activity.
Traditional signature-based detection systems are often unable to identify newly emerging or unknown threats. There is a need for a behavior-based malware detection system that can detect malicious activities that do not match known signatures.
The lack of dedicated security tools for ESXi hypervisors. The expanding attack surface targeting virtualization infrastructure. The use of data encryption by attackers. The exploitation of known vulnerabilities.
Behavioral analysis of ransomware threats to ESXi Hypervisors: a machine learning-based predictive model · 2026 · DOIThe lack of dedicated security tools and expanding attack surfaces make ESXi hypervisors vulnerable to ransomware threats. Existing security mechanisms are not sufficient to prevent ransomware attacks. There is a need for effective machine learning approaches to detect and prevent ransomware threats.
Behavioral analysis of ransomware threats to ESXi Hypervisors: a machine learning-based predictive model · 2026 · DOIThe lack of robust and adaptable solutions for detecting and classifying previously unseen malware threats. The need for a comprehensive evaluation of the effectiveness of machine learning and deep learning methods for malware detection.
Advances in Malware Detection using Machine Learning and Deep Learning: A Comprehensive Comparative Analysis · 2026 · DOIThis research highlights that while Gated Recurrent Unit and Decision Tree models achieve state-of-the-art performance in malware detection on the EMBER dataset, with accuracies of 94% and 93% respectively, their transition from experimental benchmarks to practical deployment reveals critical limitations. Deep learning architectures, such as GRUs, incur substantial computational overhead, demanding 20–30 times more training time and exhibiting 15–20 times higher inference latency compared to less complex models like Decision Trees. This renders GRUs impractical for resource constrained environments, including endpoint devices, IoT systems, or real-time network gateways. Conversely, Decision Tree models offer rapid inference and minimal memory footprint but exhibit significant performance degradation under dataset shift, underscoring their dependence on the specific feature engineering of the EMBER dataset. This vulnerability compromises their robustness within diverse and evolving threat landscapes. Critically, both model types demonstrate high susceptibility to adversarial attacks, a factor not within the scope of this investigation but extensively documented in extant literature. Decision trees are vulnerable to attacks that manipulate influential features, often identified via methods such as L1 regularization or genetic algorithms. Recurrent architectures, including GRUs, can be compromised through gradient-based or malware-specific attacks that subtly alter API call sequences or byte entropy while preserving malicious functionality. Without dedicated adversarial hardening e.g., adversarial training, ensemble defenses, or input randomization deploying these models is inherently precarious, as their high accuracy on clean data can precipitously decline below 60%. Furthermore, the inherent opacity of GRU and XGBoost models impedes operational trust and incident response, as security analysts require interpretable, human-understandable rationales for detection decisions. Emerging eXplainable AI techniques, such as SHAP and attention mechanisms, offer promising avenues to mitigate this deficiency. To advance these high-performing models into robust, production-ready systems, future research should focus on four pivotal areas: Adversarial Hardening: Implementing adversarial training and metric learning ensembles to bolster robustness without significantly compromising clean-data accuracy. Efficiency Optimization: Employing model compression, quantization, and knowledge distillation to achieve GRU-level accuracy on edge devices. Explainability Integration: Incorporating modules like SHAP, LIME, or attention mechanisms to provide transparent and actionable insights for Security Operations Center analysts. Hybrid Architecture Design: Developing tiered detection systems that integrate lightweight machine learning models for initial triage with deep learning models for in-depth forensic analysis, thereby balancing speed, accuracy, and robustness. By transparently addressing these limitations and articulating a clear research roadmap, this study not only advances the state-of-the-art in malware detection but also provides a pragmatic, operationally grounded framework for developing the next generation of reliable, adaptive, and deployable cybersecurity systems.
Advances in Malware Detection using Machine Learning and Deep Learning: A Comprehensive Comparative Analysis · 2026 · DOIRansomware attacks are becoming more sophisticated and difficult to detect - The system needs to be able to differentiate between good and bad behavior - The system needs to be able to respond quickly to ransomware attacks
The old security systems like antivirus software and firewalls are not good enough to find ransomware attacks - There is a need for a new method of detecting and preventing ransomware attacks
The diverse and sophisticated nature of modern attacks. The limitations of traditional malware detection techniques. The need to detect zero-day attacks, which are typically missed by traditional signature-based systems.
AUTOMATED ANDROID MALWARE DETECTION USING OPTIMAL ENSEMBLE LEARNING APPROACH FOR CYBER SECURITY · 2026 · DOITraditional malware detection techniques are no longer sufficient to combat modern sophisticated attacks, especially zero-day threats. Existing Android malware detection systems primarily rely on traditional techniques such as signature-based detection and single machine learning classifiers.
AUTOMATED ANDROID MALWARE DETECTION USING OPTIMAL ENSEMBLE LEARNING APPROACH FOR CYBER SECURITY · 2026 · DOIFurther evaluation of the technique in various IoT applications. Exploration of other machine learning techniques for IoT device version identification. Investigation of the use of other features and protocols for IoT device version identification.
In this paper we identified two challenges for identifying IoT firmware versions: the on-wire behaviour resulting from different firmware versions running on the same device are more subtle than those for different device models, types or manufacturers; and there is limited data on different device versions due to lack of publicly available datasets. These challenges are novel in this area and have therefore not previously been addressed by the literature. We developed a technique using flow statistics to transform on-wire device behaviours into greyscale images. These images were fed into a Twin Neural Network model to output similarity scores. Our best performing model was 95.83% accurate at identifying stable versions and 84.38% accurate identifying version changes. By calculating the Hedges’ g effect size of similarity scores, we were able to detect the subtle changes resulting from a device running different firmware versions. We showed how Hedge’s g as a metric is approximately 20% more accurate than the standard TNN measure. By using a TNN model trained on device differences — but tested on version changes — we were able to overcome the limited data problem via transfer learning. This allowed the TNN to be able to correctly calculate the similarity of images that were truly unknown - the images of different device versions had not been used to train the TNN. To improve our technique, further work could be carried out in this area, including: (1) Using active approaches. Although stated to not be ideal for the reasons mentioned in the introduction, active approaches may have a place in augmenting techniques when identifying device versions to overcome the problem when a device has no change in passive on-wire signature. This could either be from banner grabbing on known ports (as the version change may update versions or banners of extra components such as libraries), or logging into device management pages and identifying version strings. (2) Using this technique to identify newly added devices, or expanding to a huge database of known devices as an identification technique for unknown devices. (3) We have shown our technique can identify anomalies in terms of version changes, there would be value in applying the same technique for other types of anomaly detection from compromised devices. REFERENCES [1] Georgios Agrafiotis, Eftychia Makri, Ioannis Flionis, Antonios Lalas, Konstantinos Votis, and Dimitrios Tzovaras. 2022. Image-based Neural Network Models for Malware Traffic Classification using PCAP to Picture Conversion. 17th International Conference on Availability, Reliability and Security. [2] Amazon Web Services. 2024. AWS IoT Device Defender FAQs. https://aws.amazon.com/iot-device-defender/faq/ [Accessed: 07.04.2024]. [3] Nesrine Ammar, Ludovic Noirie, and Sebastien Tixeuil. 2020. Autonomous Identification of IoT Device Types based on a Supervised Classification. International Conference on Communications (ICC). [4] Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, and Thomas Cattermole. 2023. Granular IoT Device Identification Using TF-IDF and Cosine Similarity. Proceedings of the 5th Workshop on CPS&IoT Security and Privacy. , 9 pages. [5] Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, and Thomas Cattermole. 2023. Keyword Extraction for Fine-Grained IoT Device Identification. 2022 Seventh International Conference on Fog and Mobile Edge Computing (FMEC). [6] Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, and Thomas Cattermole. 2023. Reliable Identification of IoT Devices from Passive Network Traffic Analysis: Requirements and Recommendations. World Forum on Internet of Things. [7] BBC. 2017. How hackers could use doll to open your front door. https://www.bbc.co.uk/news/av/technology-38966285. [Accessed: 07.04.2024]. [8] BBC. 2020. Smart doorbells ’easy target for hackers’ study finds. https://www.bbc.co.uk/news/technology-55044568. [Accessed: 07.04.2024]. [9] BBC. 2023. The tech flaw that lets hackers control surveillance cameras. https://www.bbc.co.uk/news/technology-65975446 [Accessed: 07.04.2024]. [10] Punam Bedi, Neha Gupta, and Vinita Jindal. 2020. Siam-IDS: Handling class imbalance problem in Intrusion Detection Systems using Siamese Neural Network. Procedia Computer Science. , 780-789 pages. [11] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature Verification using a "Siamese" Time Delay Neural Network. Advances in Neural Information Processing Systems. [12] Luigi Catuogno and Clemente Galdi. 2023. Secure Firmware Update: Challenges and Solutions. Cryptography 7, 2 (2023), 1–17. [13] Batyr Charyyev and Mehmet Hadi Gunes. 2021. Locality-Sensitive IoT Network Traffic Fingerprinting for Device Identification. IEEE Internet of Things Journal 8 (2021), 1272–1281. Issue 3.
Conventional detection technologies are not effective against changing threats. There is a need for a more accurate and robust malware detection system.
Android's adaptability and ease of use have caused it to swiftly rise to the top of the mobile operating system market. The vast majority of persistent malevolent attacks also target it. This necessitates the quick installation of a strong malware detection system. With an accuracy of 99.16%, precision of 99.98%, recall of 99.87%, and F1score of 99.76%, this study's CNN+RNN hybrid model for Android malware classification achieves impressive results on the CIC-InvesAndMal2019 dataset. The suggested model proves in cybersecurity by surpassing more traditional ML algorithms like DT, RF, and SVM. However, challenges such as overfitting, indicated by fluctuations in validation loss, and high computational demands during model training, remain. To address these limitations, future work should focus on enhancing model regularization techniques, optimizing training procedures for faster convergence, and utilizing transfer learning to improve scalability.
Malware is evolving at an alarming pace. Current malware families are using dynamic execution, fileless, and memory persistence to evade old-fashioned defenses. Existing malware detection techniques lack transparency.
Interpretable Memory-behavior Modeling for Malware Detection Using Additive Explainable Learning · 2026 · DOI
Most-cited papers in Advanced Malware Detection Techniques
- Survey of machine learning techniques for malware analysis · Computers & Security · 2018 · 479 citations
- DL-Droid: Deep learning based android malware detection using real devices · Computers & Security · 2019 · 375 citations
- Image-Based malware classification using ensemble of CNN architectures (IMCEC) · Computers & Security · 2020 · 347 citations
- Ransomware threat success factors, taxonomy, and countermeasures: A survey and research directions · Computers & Security · 2018 · 305 citations
- Malware identification using visualization images and deep learning · Computers & Security · 2018 · 271 citations
- Early-stage malware prediction using recurrent neural networks · Computers & Security · 2018 · 247 citations
- A Survey on Ransomware: Evolution, Taxonomy, and Defense Solutions · ACM Computing Surveys · 2022 · 241 citations
- The Emerging Threat of Ai-driven Cyber Attacks: A Review · Applied Artificial Intelligence · 2022 · 233 citations
- Ransomware: Recent advances, analysis, challenges and future research directions · Computers & Security · 2021 · 217 citations
- AndroDialysis: Analysis of Android Intent Effectiveness in Malware Detection · Computers & Security · 2016 · 204 citations
Most recent work
- vul-rag : Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG · ACM Transactions on Software Engineering and Methodology · 2026
- Systematic Evaluation of Machine Learning and Deep Learning Models for IoT Malware Detection Across Ransomware, Rootkit, Spyware, Trojan, Botnet, Worm, Virus, and Keylogger · Sensors · 2026
- Behavioural Detection of Ransomware and a Comprehensive Approach to Ransomware Prevention · IEEE Communications Standards Magazine · 2026
- Engaging with cybercriminals: phases and influence strategies in ransomware negotiations · Computers in Human Behavior · 2026
- The Commercialization of Ransomware: From Spray-and-Prey to Big Game Hunting to the Ransom Bazaar · Crime & Delinquency · 2026
- Hybrid cryptographic and AI frameworks for cheat detection in online games · Discover Artificial Intelligence · 2026
- AI used to design brand new viruses raises safety and security concerns · BMJ · 2026
- A novel lightweight binary-level malware hybrid obfuscation against black-box anti-malware engines · High-Confidence Computing · 2026
- DeRed: Enhancing third-party library detection in binaries via deceptive reuse mitigation · Computers & Security · 2026
- Unifying mixed boolean-arithmetic obfuscation by architectural and anti-generalization hardening · Computers & Security · 2026
Find a gap in your own Advanced Malware Detection Techniques sub-topic
This page shows what the Advanced Malware Detection Techniques literature already flags as unresolved. To narrow it to your specific question, run the guided finder — it searches the gap library on demand and checks candidates against 250M+ OpenAlex works.
Open the Research Gap Finder →