thesis/3_rasta/6_recommendations.typ

#import "../lib.typ": eg, jfl-note, MWE

== Recommendations <sec:rasta-reco>

In light of our findings in @sec:rasta-failure-analysis and the issues we encountered while packaging the tools, we summarise some takeaways that we believe developers should follow to improve the success of reusing their software.

//*developer*: dire que a la lumiere de ces resultats, on peut pense que certain pbs peuvent être évité ou bien corrigé par l'utilisateur]
We understand software developed for research purposes is not and should not be held to the same standards as production software.
However, research is incremental, and it is not sustainable to start each tool from scratch.
It is critical to be able to build upon tools already published, and efforts should be made to allow that when releasing a tool.

During the packaging and testing of the tools we examined in our experiment, the most notable issues we encountered could have been avoided by following classical development best practices.
To make a tool easy to reuse, it should have documentation with at least:
- Instructions about how to install the dependencies.
- Instructions about how to build the tool (if the tool needs to be built).
- Instructions about how to use the tool (#eg command line arguments)
- Instructions about how to interpret the results of the tools (we only checked for the existence of the results in our experiment, but we found that some results can be quite obscure)
In addition to the documentation, a minimum working example with the expected result of the tools allows a potential user to check if everything is working as intended.
This #MWE have the additional benefit that it can serve as an example in the documentation.

Another best practice to follow is to pin the version of dependencies of the tools.
Many modern dependency management tools can handle that: for instance, for Python, Poetry or uv generate a lock file with the exact version of the libraries to use, Cargo does the same for Rust, in Java, this can be an option in Gradle, and dependencies in Maven `pom.xml` files are usually the exact version.
For other dependencies that are not managed by a dependency manager -- for instance, the Java virtual machine to use, the Python interpreter, resource files -- the version to use should be clearly documented.
Alternatively, tools like nixpkg can be used to pin every dependency.
The worst case we encountered during our experiment was a tool whose documentation instructed us to install the z3 dependencies with a simple `git clone`, without specifying the commit to use.
The z3 project is still actively maintained, so the dependency installed was not compatible, and finding a compatible version required checking releases one by one.
Dependencies fetched with a version control system should always indicate the exact version to use (in the case of git, a commit, tag, or release should be used).

We also found that interactions with the running environment can become very problematic when the environment changes.
To minimise the issues, packaging the tool inside a Docker container or even a virtual machine can ensure that future users have at least access to a working version of the tool.

Finally, when possible, continuous integration, tests and code reviews should be implemented to improve the reliability of the developed tool.

Concerning the actual code of the tool, more attention should be paid to error reporting.
When a tool failed to perform its analysis, it should be clear to the user, and the reason should be clearly reported.
In some cases, this may imply _not_ trying to recover from unrecoverable errors: this often leads to errors seemingly unrelated to the initial issue.
This is often a problem in Java code, where the developers are strongly encouraged to catch all exceptions, and in bash scripts that run several programs in a row without checking the exit statuses.

Good error reporting can allow future users to solve issues encountered using the tools: for instance, the log generated by Androguard's decompiler clearly shows that the issue is file names exceeding the size limit.
This issue could easily be fixed by changing the filenames used to store the results.
In contrast, the errors generated by Flowdroid are so opaque that we have no idea how we could solve them.

And at last, an important remark concerns the libraries used by a tool. 
We have seen two types of libraries:
- internal libraries manipulating internal data of the tool.
- external libraries that are used to manipulate the input data (APKs, bytecode, resources).
We observed during our manual investigations that external libraries are the ones leading to crashes because of variations in recent APKs (file format, unknown bytecode instructions, multi-DEX files). 
We believe that the developer should provide enough documentation to make a later upgrade of these external libraries possible.
For example, old versions of Apktool are the top-most libraries raising errors, but breaking changes introduced by upgrading from v1.X versions to v2.X versions prevent us from upgrading Apktool.