Detecting malicious behavior in binary programs using dynamic symbolic execution and API call sequences

Download

fttatar_thesis_20210702.pdf

Date

2021-6

Author

Tatar, Fatih Tamer

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

389
views

630
downloads

Program analysis becomes an important part of malware detection as malware become stealthier and more complex. For example, modern malware may detect whether they are under analysis and they may use certain triggers such as time to avoid detection. However, current detection techniques turn out to be insufficient as they have limitations to detect new, obfuscated, and intelligent malware. In this thesis, we propose a behavior based malware detection methodology using API call sequence analysis. In our methodology, we combine dynamic symbolic execution and API function models to extract call sequences of a given binary program and decide whether it has a malicious sequence. In our experiments, we showed that our methodology is capable of detecting malware hiding behind evasion techniques and our methodology is applicable to a real-world problem.

Subject Keywords

Call sequence analysis, Dynamic symbolic execution, Function modeling, Malware analysis

URI

https://hdl.handle.net/11511/91234

Collections

Graduate School of Informatics, Thesis

Suggestions

OpenMETU
Core

DETECTING MALICIOUS API CALL SEQUENCES IN BINARY PROGRAMS USING DYNAMIC SYMBOLIC EXECUTION Tatar, Fatih Tamer; Betin Can, Aysu (2022-10-01) As malicious software gets more stealthy and smarter, software analysis has become an essential part of malware detection. Modern malware does not immediately display its malicious behavior, especially if they are aware that it is being analyzed. For instance, malware can detect the runtime environment and use certain triggers, such as time, to avoid detection. Static analysis fails on obfuscated code whereas dynamic analysis struggles to find the right actions and conditions to trigger malicious act...
Malicious code detection: run trace analysis by LSTM Şırlancı, Melih; Acartürk, Cengiz; Gürkan Balıkçıoğlu, Pınar; Department of Cybersecurity (2021-6) Malicious software threats and their detection have been gaining importance as a subdomain of information security due to the expansion of ICT applications in daily settings. A major challenge in designing and developing anti-malware systems is the coverage of the detection, particularly the development of dynamic analysis methods that can detect polymorphic and metamorphic malware efficiently. In the present study, we propose a methodological framework for detecting malicious code by analyzing run trace ou...
Application of subspace clustering to scalable malware clustering Işıktaş, Fatih; Betin Can, Aysu; Department of Information Systems (2019) In recent years, massive proliferation of malware variants has made it necessary to employ sophisticated clustering techniques in malware analysis. Choosing an appropriate clustering approach is very important especially for rapidly and accurately mining clustering information from a large malware set with high number of attributes. In this study, we propose a clustering method that is based on subspace clustering and graph matching techniques and presents an enhanced clustering ability and scalable runtime...
Malicious user input detection on web-based attacks with the negative selection algorithm Karataş, Mustafa Mer; Acar, Aybar Can; Department of Cyber Security (2019) In the cyber security domain, detection and prevention of intrusions is a crucial task. Intrusion attempts exploiting vulnerabilities in an organization’s servers or applications may lead to devastating consequences. The malicious actor may obtain sensitive information from the application, seize database records or take over the servers completely. While protecting web applications/services, discrimination of legitimate user inputs from malicious payloads must be done. Taking inspiration from the Human Imm...
Modelling the effects of malware propagation on military operations by using bayesian network framework Şengül, Zafer; Acartürk, Cengiz; Department of Cyber Security (2019) Malware are malicious programs that cause unwanted system behavior and usually result in damage to IT systems or its users. These effects can also be seen during military operations because high-tech military weapons, command, control and communication systems are also interconnected IT systems. This thesis employs conventional models that have been used for modeling the propagation of biological diseases to investigate the spread of malware in connected systems. In particular, it proposes a probabilistic l...

Citation Formats

F. T. Tatar, “Detecting malicious behavior in binary programs using dynamic symbolic execution and API call sequences,” M.S. - Master of Science, Middle East Technical University, 2021.