== Limitations <sec:rasta-limit>

Some limitations of our approach should be kept in mind.

Our application dataset is biased in favour of Androguard, because Androzoo have already used Androguard internally when collecting applications and discarded any application that cannot be processed with this tool.

Despite our best efforts, it is possible that we made mistakes when building or using the tools. 
It is also possible that we wrongly classified a result as a failure. 
To mitigate this possible problem, we contacted the authors of the tools to confirm that we used the right parameters and chose a valid failure criterion. 
Before running the final experiment, we also ran the tools on a subset of our dataset and manually investigated the most common errors to ensure that they are not trivial errors that can be solved.

The timeout value and memory limits are arbitrarily fixed. 
To mitigate this issue, a small extract of our dataset has been analysed with more memory/time, and we checked that there was no significant difference in the results.

Finally, the use of VirusTotal for determining if an application is malware or not may be wrong. 
To limit the impact of errors, we used a threshold of at most 5 antiviruses (resp. no more than 0) reporting an application as being malware (resp. goodware) for taking a decision about maliciousness (resp. benignness).
