Detecting malicious behavior in binary programs using dynamic symbolic execution and API call sequences

2021-6
Tatar, Fatih Tamer
Program analysis becomes an important part of malware detection as malware become stealthier and more complex. For example, modern malware may detect whether they are under analysis and they may use certain triggers such as time to avoid detection. However, current detection techniques turn out to be insufficient as they have limitations to detect new, obfuscated, and intelligent malware. In this thesis, we propose a behavior based malware detection methodology using API call sequence analysis. In our methodology, we combine dynamic symbolic execution and API function models to extract call sequences of a given binary program and decide whether it has a malicious sequence. In our experiments, we showed that our methodology is capable of detecting malware hiding behind evasion techniques and our methodology is applicable to a real-world problem.
Citation Formats
F. T. Tatar, “Detecting malicious behavior in binary programs using dynamic symbolic execution and API call sequences,” M.S. - Master of Science, Middle East Technical University, 2021.