Antivirus software initially relied heavily on signatures to identify malware and other object based threats. Indeed, even today’s current AV products still primarily use a signature engine for detection. Signatures were and are determined when a malware (or malicious file object) arrives in the hands of an antivirus firm, and is analyzed by malware researchers or by dynamic analysis systems. Once a file is determined to be a malicious, a signature (typically an MD5 or SHA 256 hash) of the file is computed and added to the antivirus software’s database of known bad files.
This method of detection works well when the malicious file is known ahead of time and appears in the same [known] form on the infected machine, but as antivirus software became commonplace, malware authors began to write “polymorphic” or “metamorphic” viruses, which encrypt parts of themselves or otherwise modify themselves as a method of disguise, so as to not match the virus signatures in the antivirus software’s database.
Because metamorphic files cannot be reliably detected with a simple signature based approach, it became necessary to devise a new method of detection: the sandbox.
Instead of relying on pre-defined signatures, sandbox based detection executes a program in a virtual environment, logging what actions the program performs. Depending on the actions logged, the sandbox can determine if the program is malicious or not. By 2006, this technique proved to be more effective than signature based detection and spawned multiple sandbox based products from a variety of companies.
While sandbox based products did provide value for a short period of time, today’s threats easily evade the technology. What follows is a description of the difficulties sandbox makers face when trying to design a detection system and clues about what to watch out for if this is your preferred method of detection.