thesis/5_theseus/3_static_transformation.typ

#import "../lib.typ": todo, APK, APKs, DEX, JAR, OAT, SDK, eg, ART, jm-note, jfl-note

== Code Transformation <sec:th-trans>

In this section, we will see how we can transform the application code to make dynamic code loading and reflective calls more analysable by static analysis tools.

=== Transforming Reflection <sec:th-trans-ref>

In Android, reflection allows applications to instantiate a class or call a method without having this class or method appear in the bytecode.
Instead, the bytecode uses the generic classes `Class`, `Method` and `Constructor`, which represent any existing class, method or constructor.
Reflection often starts by retrieving the `Class` object representing the class to use.
This class is usually retrieved using a `ClassLoader` object (though there are other ways to get it).
Once the class is retrieved, it can be instanciated using the deprecated method `Class.newInstance()`, as shown in @lst:-th-expl-cl-new-instance, or a specific method can be retrieved.
The current approach to instantiate a class is to retrieve the specific `Constructor` object, then call `Constructor.newInstance(..)` like in @lst:-th-expl-cl-cnstr.
Similarly, to call a method, the `Method` object must be retrieved, then called using `Method.invoke(..)`, as shown in @lst:-th-expl-cl-call.

Although the process seems to differ between class instantiation and method call from the Java standpoint, the runtime operations are very similar.
When instantiating an object with `Object obj = cst.newInstance("Hello Void")`, the constructor method `<init>(Ljava/lang/String;)V`, represented by the `Constructor` `cst`, is called on the object `obj`.
Thus, even for instantiation, a method is called at some point.

#figure(
  ```java
  ClassLoader cl = MainActivity.class.getClassLoader();
  Class clz = cl.loadClass("com.example.Reflectee");
  Object obj = clz.newInstance();
  ```,
  caption: [Instantiating a class using `Class.newInstance()`]
) <lst:-th-expl-cl-new-instance>

#figure(
  ```java
  Constructor cst = clz.getDeclaredConstructor(String.class);
  Object obj = cst.newInstance("Hello Void");
  ```,
  caption: [Instantiating a class using `Constructor.newInstance(..)`]
) <lst:-th-expl-cl-cnstr>

#figure(
  ```java
  Method mth = clz.getMethod("myMethod", String.class);
  Object[] args = {(Object)"an argument"};
  String retData = (String) mth.invoke(obj, args);
  ```,
  caption: [Calling a method using reflection]
) <lst:-th-expl-cl-call>

One of the main reasons to use reflection is to access classes that are neither platform classes nor in the application bytecode, as is often the case when dealing with classes from dynamically loaded bytecode.
Indeed, if the #ART were to encounter an instruction referencing a class that cannot be loaded by the current class loaded, it would crash the application.

To allow static analysis tools to analyse an application that uses reflection, we want to replace the reflection call with a bytecode chunk that actually calls the method and can be analysed by any static analysis tool.
In @sec:th-trans-cl, we deal with the issue of dynamic code loading so that the classes used are, in fact, present in the application.

A notable issue is that a specific reflection call can call different methods.
@lst:th-worst-case-ref illustrates a worst-case scenario where any method can be called at the same reflection call.
In those situations, we cannot guarantee that we know all the methods that can be called (#eg the name of the method called could be retrieved from a remote server).
In addition, the method we propose in @sec:th-dyn is a best effort approach to collect reflection data: like any dynamic analysis, it is limited by its code coverage.

#figure(
  ```java
  Object myInvoke(Object obj, Method mth, Object[] args) throws .. {
    return mth.invoke(obj, args);
  }
  ```,
  caption: [A reflection call that can call any method]
) <lst:th-worst-case-ref>

To handle those situations, instead of entirely removing the reflection call, we can modify the application code to test if the `Method` (or `Constructor`) object matches any of the methods observed dynamically, and if so, directly call the method.
If the object does not match any expected method, the code can fall back to the original reflection call.
DroidRA~@li_droidra_2016 has a similar solution, except that reflective calls are always evaluated, and the static equivalent follows just after, guarded behind an opaque predicate that is always false at runtime.
@lst:-th-expl-cl-call-trans demonstrates this transformation for the code originally in @lst:-th-expl-cl-call.
Let's suppose that we observed dynamically a call to a method `Reflectee.myMethod(String)` at line 3 when monitoring the execution of the code of @lst:-th-expl-cl-call.
In @lst:-th-expl-cl-call, at line 25, the `Method` object `mth` is checked using a method we generated and injected in the application (defined at line 2 in the listing).
This method checks if the method name (line 5), its parameters (lines 6-9), its return type (lines 10-11) and its declaring class (lines 13-14) match the expected method.
If it is the case, the method is used directly (line 26) after casting the arguments and associated object into the types/classes we just checked.
If the check line 25 does not pass, the original reflective call is made (line 28).
If we were to expect other possible methods to be called in addition to `myMethod`, we would add `else if` blocks between lines 26 and 27, with other check methods reflecting each potential method call.
/*
#jfl-note[It should be noted that we do the transformation at the bytecode level, the code in the listing correspond to the output of JADX][
  J'aurais bien fait une section a part sur "comment on fait ces transformation concretement; 
  plus pedagique de décrire les transformation sans bytecode, ensuite, sous section qui discute 
  les facon de modifier le bytecode, soot, apktool, ect et qui explique les limites, puis dire comment tu fait mes modifications
] #todo[Ref to list of common tools?] reformated for readability.
*/

The check of the `Method` value is done in a separate method injected inside the application to avoid cluttering the application too much.
Because Java (and thus Android) uses polymorphic methods, we cannot just check the method name and its class, but also the whole method signature.
We chose to limit the transformation to the specific instruction that calls `Method.invoke(..)`.
This drastically reduces the risks of breaking the application, but leads to a lot of type casting.
Indeed, the reflection call uses the generic `Object` class, but actual methods usually use specific classes (#eg `String`, `Context`, `Reflectee`) or scalar types (#eg `int`, `long`, `boolean`).
This means that the method parameters and object on which the method is called must be downcasted to their actual type before calling the method, then the returned value must be upcasted back to an `Object`.
Scalar types especially require special attention. 
Java (and Android) distinguish between scalar types and classes, and they cannot be mixed: a scalar cannot be cast into an `Object`.
However, each scalar type has an associated class that can be used when doing reflection.
For example, the scalar type `int` is associated with the class `Integer`, the method `Integer.valueOf()` can convert an `int` scalar to an `Integer` object, and the method `Integer.intValue()` converts back an `Integer` object to an `int` scalar.
Each time the method called by reflection uses scalars, the scalar-object conversion must be made before calling it.
And finally, because the instruction following the reflection call expects an `Object`, the return value of the method must be cast into an `Object`.

This back and forth between types might confuse some analysis tools. 
This could be improved in future works by analysing the code around the reflection call. 
For example, if the result of the reflection call is immediately cast into the expected type (#eg in @lst:-th-expl-cl-call, the result is cast to a `String`), there should be no need to cast it to Object in between.
Similarly, it is common to have the method parameter arrays generated just before the reflection call and never be used again (This is due to `Method.invoke(..)` being a varargs method: the array can be generated by the compiler at compile time).
In those cases, the parameters could be used directly without the detour inside an array.

#figure(
  ```java
  class T {
    static boolean check_is_reflectee_mymethod_e398(Method mth) {
      Class<?>[] paramTys = mth.getParameterTypes();
      return (
        meth.getName().equals("myMethod") &&
        paramTys.length == 1 &&
        paramTys[0].descriptorString().equals(
          String.class.descriptorString()
        ) &&
        mth.getReturnType().descriptorString().equals(
          String.class.descriptorString()
        ) &&
        mth.getDeclaringClass().descriptorString().equals(
          Reflectee.class.descriptorString()
        )
      )
    }
  }

  ...

  Method mth = clz.getMethod("myMethod", String.class);
  Object[] args = {(Object)"an argument"}
  Object objRet;
  if (T.check_is_reflectee_mymethod_e398abf7d3ce6ede(mth)) {
    objRet = (Object) ((Reflectee) obj).myMethod((String)args[0]);
  } else {
    objRet = mth.invoke(obj, args);
  }
  String retData = (String) objRet;
  ```,
  caption: [@lst:-th-expl-cl-call after the de-reflection transformation]
) <lst:-th-expl-cl-call-trans>

=== Transforming Code Loading (or Not) <sec:th-trans-cl>

#jfl-note[Ici je pensais lire comment on tranforme le code qui load du code, mais on me parle de multi dex]

An application can dynamically import code from several formats like #DEX, #APK, #JAR or #OAT, either stored in memory or in a file.
Because it is an internal, platform-dependent format, we elected to ignore the #OAT format.
Practically, #JAR and #APK files are zip files containing #DEX files.
This means that we only need to find a way to integrate #DEX files into the application.

We saw in @sec:cl the class loading model of Android.
When doing dynamic code loading, an application defines a new `ClassLoader` that handles the new bytecode, and starts accessing its classes using reflection.
We also saw in @sec:cl that Android now use the multi-dex format, allowing it to handle any number of #DEX files in one class loader.
Therefore, the simpler way to give access to the dynamically loaded code to static analysis tools is to add the dex files in the application as additional multi-dex bytecode files.
This should not impact the class loading model as long as there is no class collision (we will explore this in @sec:th-class-collision) and as long as the original application did not try to access inaccessible classes (we will develop this issue in @sec:th-limits). 

#figure(
  image(
    "figs/dex_insertion.svg",
    width: 80%,
    alt: "A diagram showing a box labelled 'app.apk', a box labelled 'lib.jar', and a single file outside the boxes labelled 'lib.dex'. The lib.jar box contains the files classes.dex and classes2.dex. Inside the app.apk box, the files AndroidManifest.xml, resources.arsc, classes.dex, classes2.dex, classes3.dex and the folders lib, res and assets are circled by dashes and labelled 'original files', and, still inside app.apk, the files classes4.dex, classes5.dex and classes5.dex are circled by dashes and labelled 'Added Files'. Arrows go from lib.dex to classes4.dex, from the classes.dex inside lib.jar to classes5.dex inside app.apk and from classes2.dex inside lib.jar to classes6.dex inside app.apk"
  ),
  caption: [Inserting #DEX files inside an #APK]
) <fig:th-inserting-dex>

In the end, we decided *not* to modify the original code that loads the bytecode.
Most tools already ignore dynamic code loading, and, with the dynamically loaded bytecode added using the multi-dex format, they already have access to it.
At runtime, although the bytecode is already present in the application, the application will still dynamically load the code.
This ensures that the application keeps working as intended, even if the transformation we applied is incomplete.
Specifically, to call dynamically loaded code, an application needs to use reflection, and we saw in @sec:th-trans-ref that we need to keep reflection calls, and in order to keep reflection calls, we need the class loader created when loading bytecode.

To summarise, we do not modify the existing bytecode.
Instead, we add the intercepted bytecode to the application as additional #DEX files using the multi-dex format, as represented in @fig:th-inserting-dex.

=== Class Collisions <sec:th-class-collision>

We saw in @sec:cl/*-obfuscation*/ that having several classes with the same name in the same application can be problematic.
In @sec:th-trans-cl, we are adding new code.
By doing so, we increase the probability of having class collisions:
The developer may have reused a helper class in both the dynamically loaded bytecode and the application, or an obfuscation process may have renamed classes without checking for intersection between the two sources of bytecode.
When loaded dynamically, the classes are in a different class loader, and the class resolution is resolved at runtime, like we saw in @sec:cl-loading.
We decided to restrain our scope to the use of class loaders from the Android #SDK.
In the absence of class collision, those class loaders behave seamlessly and adding the classes to the application maintains the behaviour.
#jfl-note[Un example aiderait a comprendre \ jm: j'en ai pas qui prennent pas 3 pages de listing]

When we detect a collision, we rename one of the colliding classes in order to be able to differentiate between classes.
To avoid breaking the application, we then need to rename all references to this specific class and be careful not to modify references to the other class.
To do so, we regroup each class by the class loaders that define them.
Then, for each colliding class name and each class loader, we check the actual class used by the class loader.
If the class has been renamed, we rename all references to this class in the classes defined by this class loader.
To find the class used by a class loader, we reproduce the behaviour of the different class loaders of the Android #SDK.
This is an important step: remember that the delegation process can lead to situations where the class defined by a class loader is not the class that will be loaded when querying the class loader.
The pseudo-code in @lst:renaming-algo shows the three steps of this algorithm: 
- First, we detect collisions and rename class definitions to remove the collisions.
- Then we rename the reference to the colliding classes to make sure the right classes are called.
- Ultimately, we merge the modified #DEX files of each class loader into one Android application.

#figure(
  ```python
  defined_classes = set()
  redifined_classes = set()

  # Rename the definition of redifined classes
  for cl in class_loaders:
    for clz in defined_classes.intersection(cl.defined_classes):
      cl.rename_definition(clz)
      redifined_classes.add(clz)
    defined_classes.update(cl.defined_classes)

  # Rename reference of redifined classes
  for cl in class_loaders:
    for clz in redifined_classes:
      defining_cl = cl.resolve_class(clz).class_loader
      cl.rename_reference(clz, defining_cl.new_name(clz))

  # Merge the class loader into a flat APK
  new_apk = Apk()
  for cl in class_loaders:
    for dex in cl.get_dex():
      new_apk.add_dex(dex)
  ```,
  caption: [Pseudo-code of the renaming algorithm]
) <lst:renaming-algo>

/*
* Although we limited ourselves to replacing one specific bytecode instruction, we encontered many technical challenges
* #todo[interupting try blocks: catch block might expect temporary registers to still stored the saved value] ?
*/

=== Implementation Details

Most of the contributions we saw performing instrumentation rely on Soot.
Soot works on an intermediate representation, Jimple, that is easier to manipulate.
However, Soot can be cumbersome to set up and use, and we initially wanted better control over the modified bytecode.
Our initial idea was to use Apktool, but in @sec:rasta, we found that many errors raised by tools were due to trying to parse Smali incorrectly.
So, rather than parsing, modifying and regenerating the Smali text files, we decided to make our own instrumentation library from scratch.
It was not as difficult as one would expect, thanks to the clear documentation of the Dalvik format from Google#footnote[https://source.android.com/docs/core/runtime/dex-format].
In addition, when we had doubts about the specification, we had the option to check the implementation used by Apktool#footnote[https://github.com/JesusFreke/smali], or the code used by Android to check the integrity of the #DEX files#footnote[https://cs.android.com/android/platform/superproject/main/+/main:art/libdexfile/dex/dex_file_verifier.cc;drc=11bd0da6cfa3fa40bc61deae0ad1e6ba230b0954].

One thing we noticed when manually instrumenting applications with Apktool is that sometimes the repackaged applications cannot be installed or run due to some files being stored incorrectly in the new application (#eg native library files must not be compressed).
We also found that some applications deliberately store files with names that will crash the zip library used by Apktool.
For this reason, we also used our own library to modify the #APK files.
We took special care to process the least possible files in the #APKs, and only strip the #DEX files and signatures, before adding the new modified #DEX files at the end.

Unfortunately, we did not have time to compare the robustness of our solution to existing tools like Apktool and Soot. 
In hindsight, we probably should have taken the time to find a way to use smali/backsmali (the backend of Apktool) as a library or use SootUp to do the instrumentation, but neither option has documentation to instrument applications this way.
At the time of writing, the feature is still being developed, but in the future, Androguard might also become an option to modify #DEX files.
Nevertheless, we published our instrumentation library, Androscalpel, for anyone who wants to use it. #todo[ref to code]

#v(2em)

Now that we saw the transformations we want to make, we know the runtime information we need to do it.
In the next section, we will propose a solution to collect that information.