DETECTING MALICIOUS API CALL SEQUENCES IN BINARY PROGRAMS USING DYNAMIC SYMBOLIC EXECUTION

Date

2022-10-01

Author

Tatar, Fatih Tamer
Betin Can, Aysu

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

214
views

0
downloads

As malicious software gets more stealthy and smarter, software analysis has become an essential part of malware detection. Modern malware does not immediately display its malicious behavior, especially if they are aware that it is being analyzed. For instance, malware can detect the runtime environment and use certain triggers, such as time, to avoid detection. Static analysis fails on obfuscated code whereas dynamic analysis struggles to find the right actions and conditions to trigger malicious activity of software that can sense being monitored. In this paper, we propose a behavior-based malware detection methodology using API call sequence analysis based on dynamic symbolic execution. We propose API function models with the symbolic execution engine to extract possible call sequences of a given binary program; identify if there is a malicious sequence even if it is hidden, and provide evidence by showing what data values will reveal this malicious API sequence. Our experiments showed that our methodology detects suspicious behavior hiding behind evasion techniques and its applicability to a real malware.

URI

https://arxiv.org/submit/4560713/view
https://hdl.handle.net/11511/100506

Collections

Graduate School of Informatics, Article

Suggestions

OpenMETU
Core

Detecting malicious behavior in binary programs using dynamic symbolic execution and API call sequences Tatar, Fatih Tamer; Betin Can, Aysu; Department of Bioinformatics (2021-6) Program analysis becomes an important part of malware detection as malware become stealthier and more complex. For example, modern malware may detect whether they are under analysis and they may use certain triggers such as time to avoid detection. However, current detection techniques turn out to be insufficient as they have limitations to detect new, obfuscated, and intelligent malware. In this thesis, we propose a behavior based malware detection methodology using API call sequence analysis. In our metho...
Malicious user input detection on web-based attacks with the negative selection algorithm Karataş, Mustafa Mer; Acar, Aybar Can; Department of Cyber Security (2019) In the cyber security domain, detection and prevention of intrusions is a crucial task. Intrusion attempts exploiting vulnerabilities in an organization’s servers or applications may lead to devastating consequences. The malicious actor may obtain sensitive information from the application, seize database records or take over the servers completely. While protecting web applications/services, discrimination of legitimate user inputs from malicious payloads must be done. Taking inspiration from the Human Imm...
Application of subspace clustering to scalable malware clustering Işıktaş, Fatih; Betin Can, Aysu; Department of Information Systems (2019) In recent years, massive proliferation of malware variants has made it necessary to employ sophisticated clustering techniques in malware analysis. Choosing an appropriate clustering approach is very important especially for rapidly and accurately mining clustering information from a large malware set with high number of attributes. In this study, we propose a clustering method that is based on subspace clustering and graph matching techniques and presents an enhanced clustering ability and scalable runtime...
Modelling the effects of malware propagation on military operations by using bayesian network framework Şengül, Zafer; Acartürk, Cengiz; Department of Cyber Security (2019) Malware are malicious programs that cause unwanted system behavior and usually result in damage to IT systems or its users. These effects can also be seen during military operations because high-tech military weapons, command, control and communication systems are also interconnected IT systems. This thesis employs conventional models that have been used for modeling the propagation of biological diseases to investigate the spread of malware in connected systems. In particular, it proposes a probabilistic l...
Using Assurance Cases to Develop Iteratively Security Features Using Scrum BEN OTHMANE, Lotfi; Angın, Pelin; BHARGAVA, Bharat (2014-09-12) A security feature is a customer-valued capability of software for mitigating a set of security threats. Incremental development of security features, using the Scrum method, often leads to developing ineffective features in addressing the threats they target due to factors such as incomplete security tests. This paper proposes the use of security assurance cases to maintain a global view of the security claims as the feature is being developed iteratively and a process that enables the incremental developm...

Citation Formats

F. T. Tatar and A. Betin Can, “DETECTING MALICIOUS API CALL SEQUENCES IN BINARY PROGRAMS USING DYNAMIC SYMBOLIC EXECUTION,” 2022, Accessed: 00, 2022. [Online]. Available: https://arxiv.org/submit/4560713/view.