thesis/5_theseus/2_static_transformation.typ

#import "../lib.typ": todo, APK, DEX, JAR, OAT, SDK, eg, ART, jm-note, jfl-note

== Code Transformation <sec:th-trans>

#todo[Define code loading and reflection somewhere]
#todo[This is a draft, clean this up]
#todo[Reflectif call? Reflection call?]

In this section, we will see how we can transform the application code to make dynamic codeloading and reflexive calls more analysable by static analysis tools.

=== Transforming Reflection <sec:th-trans-ref>


In Android, reflection allows to instanciate a class, or call a method, without having this class or method appear in the bytecode.
Instead, the bytecode uses the generic classes `Class`, `Method` and `Constructor`, that represent any existing class, method or constructor.
Reflection often starts by retrieving the `Class` object representing the class to use.
This class is usually retrieved using a `ClassLoader` object (though they are other ways to get it).
Once the class is retrieved, it can be instanciated using the deprecated method `Class.newInstance()`, as shown in @lst:-th-expl-cl-new-instance, or a specific method can be retrieved.
The current approach to instanciate a class is to retrieve the specific `Constructor` object, then calling `Constructor.newInstance(..)` like in @lst:-th-expl-cl-cnstr.
Similarly, to call a method, the `Method` object must be retrieved, then called using `Method.invoke(..)`, as shown in @lst:-th-expl-cl-call.

Although the process seems to differ between class instanciation and method call from the Java stand point, the runtime opperations are very similar.
When instanciating an object with `Object obj = cst.newInstance("Hello Void")`, the constructor method `<init>(Ljava/lang/String;)V`, represented by the `Constructor` `cst`, is called on the object `obj`.

#figure(
  ```java
  ClassLoader cl = MainActivity.class.getClassLoader();
  Class clz = cl.loadClass("com.example.Reflectee");
  Object obj = clz.newInstance();
  ```,
  caption: [Instanciating a class using `Class.newInstance()`]
) <lst:-th-expl-cl-new-instance>

#figure(
  ```java
  Constructor cst = clz.getDeclaredConstructor(String.class);
  Object obj = cst.newInstance("Hello Void");
  ```,
  caption: [Instanciating a class using `Constructor.newInstance(..)`]
) <lst:-th-expl-cl-cnstr>

#figure(
  ```java
  Method mth = clz.getMethod("myMethod", String.class);
  Object[] args = {(Object)"an argument"};
  String retData = (String) mth.invoke(obj, args);
  ```,
  caption: [Calling a method using reflection]
) <lst:-th-expl-cl-call>

One of the main reasons to use reflection is to access classes that are not present in the application bytecode, nor are platform classes.
Indeed, the application will crash if the #ART encounter references to a class that is cannot be found by the current classloader.
This is often the case when dealing with classes from bytecode loaded dynamically.

To allow static analysis tools to analyse an application that use reflection, we want to replace the reflection call by the bytecode that actually call the method.
In @sec:th-trans-cl, we deal with the issue of dynamic code loading so that the classes used are in fact present in the application.

A notable issue is that a specific reflection call can call different methods.
@lst:th-worst-case-ref illustrates a worst case scenario where any method can be called at the same reflection call.
In those situation, we cannot garanty that we know all the methods that can be called (#eg the name of the method called could be retrieved from a remote server).
In addition, the method we propose in @sec:th-dyn is a best effort approach to collect reflection data: like any dynamic analysis, it is limited by its code coverage.

#figure(
  ```java
  Object myInvoke(Object obj, Method mth, Object[] args) throws .. {
    return mth.invoke(obj, args);
  }
  ```,
  caption: [A reflection call that can call any method]
) <lst:th-worst-case-ref>

To handle those situation, instead of entirely removing the reflection call, we can modify the application code to test if the `Method` (or `Constructor`) object match any expected method, and if yes, directly call the method.
If the object does not match any expected method, the code can fallback to the original reflection call.
DroidRA~@li_droidra_2016 has a similar solution, except that reflective calls are always evaluated, and the static equivalent follow just after, guarded behind an opaque predicate that is always false at runtime.
@lst:-th-expl-cl-call-trans demonstrate this transformation on @lst:-th-expl-cl-call:
at line 25, the `Method` objet `mth` is checked using a method we generated and injected in the application (defined at line 2 in the listing).
This method check if the method name, (line 5), its parameters (lines 6-9), its return type (lines 10-11) and its declaring class (lines 13-14) match the expected method.
If it is the case, the method is used directly (line 26) after casting the arguments and associated object into the types/classes we just checked.
If the check line 25 does not pass, the original reflectif call is made (line 28).
If we were to expect other possible methods to be called in addition to `myMethod`, we would add `else if` blocks between lines 26 and 27, with other check methods reflecting each potential method call.
/*
#jfl-note[It should be noted that we do the transformation at the bytecode level, the code in the listing correspond to the output of JADX][
  J'aurais bien fait une section a part sur "comment on fait ces transformation concretement; 
  plus pedagique de décrire les transformation sans bytecode, ensuite, sous section qui discute 
  les facon de modifier le bytecode, soot, apktool, ect et qui explique les limites, puis dire comment tu fait mes modifications
] #todo[Ref to list of common tools?] reformated for readability.
*/

The method check is done in a separate method injected inside the application to avoid clutering the application too much.
Because Java (and thus Android) uses polymorphic methods, we cannot just check the method name and its class, but also the whole method signature.
We chose to limit the transformation to the specific instruction that call `Method.invoke(..)`.
This drastically reduce the risks of breaking the application, but leads to a lot of type casting.
Indeed, the reflection call uses the generic `Object` class, but actual methods usually use specific classes (#eg `String`, `Context`, `Reflectee`) or scalar types (#eg `int`, `long`, `boolean`).
This means that the method parameters and object on which the method is called must be downcast to their actual type before calling the method, then the returned value must be upcasted back to an `Object`.
Scalar types especially require special attention. 
Java (and Android) distinguish between scalar type and classes, and they cannot be mixed: a scalar cannot be cast into an `Object`.
However, each scalar type has an associated class that can be use when doing reflection.
For example, the scalar type `int` is associated with the class `Integer`, the method `Integer.valueOf()` can convert an `int` scalar to an `Integer` object, and the method `Integer.intValue()` convert back an `Integer` object to an `int` scalar.
Each time the method called by reflection used scalars, the scalar-object convertion must be made before calling it.
And finally, because the instruction following the reflection call expect an `Object`, the return value of the method must be cast into an `Object`.

This back and forth between types might confuse some analysis tools. 
This could be improved in futur works by analysing the code around the reflection call. 
For example, if the result of the reflection call is immediatly cast into the expected type (#eg in @lst:-th-expl-cl-call, the result is cast to a `String`), they should not be any need to cast it to Object in between.
Similarly, it is common to have the method parameter arrays generated just before the reflection call never be used again (This is due to `Method.invoke(..)` beeing a varargs method: the array can be generated by the compiler at compile time).
In those cases, the parameters could be used directly whithout the detour inside an array.

#figure(
  ```java
  class T {
    static boolean check_is_reflectee_mymethod_e398(Method mth) {
      Class<?>[] paramTys = mth.getParameterTypes();
      return (
        meth.getName().equals("myMethod") &&
        paramTys.length == 1 &&
        paramTys[0].descriptorString().equals(
          String.class.descriptorString()
        ) &&
        mth.getReturnType().descriptorString().equals(
          String.class.descriptorString()
        ) &&
        mth.getDeclaringClass().descriptorString().equals(
          Reflectee.class.descriptorString()
        )
      )
    }
  }

  ...

  Method mth = clz.getMethod("myMethod", String.class);
  Object[] args = {(Object)"an argument"}
  Object objRet;
  if (T.check_is_reflectee_mymethod_e398abf7d3ce6ede(mth)) {
    objRet = (Object) ((Reflectee) obj).myMethod((String)args[0]);
  } else {
    objRet = mth.invoke(obj, args);
  }
  String retData = (String) objRet;
  ```,
  caption: [@lst:-th-expl-cl-call after the de-reflection transformation]
) <lst:-th-expl-cl-call-trans>


=== Transforming Code Loading (or Not) <sec:th-trans-cl>

#jfl-note[Ici je pensais lire comment on tranforme le code qui load du code, mais on me parle de multi dex]

An application can dynamically import code from several format like #DEX, #APK, #JAR or #OAT, either stored in memory or in a file.
Because it is an internal, platform dependant format, we elected to ignore the #OAT format.
Practically, #JAR and #APK files are zip files containing #DEX files.
This means that we only need to find a way to integrate #DEX files to the application.


We saw in @sec:cl the class loading model of Android.
When doing dynamic code loading, an application define a new `ClassLoader` that handle the new bytecode, and start accessing its classes using reflection.
We also saw in @sec:cl that Android now use the multi-dex format, allowing it to handle any number of #DEX files in one classloader.
Therefore, the simpler way to give access to the dynamically loaded code to static analysis tool is add the dex files to the application.
This should not impact the classloading model as long as there is no class collision (we will explore this in @sec:th-class-collision) and as long as the original application appliaction did not try to access unaccessible classes (we will develop this issue in @sec:th-limits). 

#figure(
  image(
    "figs/dex_insertion.svg",
    width: 80%,
    alt: "A diagram showing a box labelled 'app.apk', a box labelled 'lib.jar', and single file ouside the boxes labelled 'lib.dex'. The lib.jar boxe contains the files classes.dex and classes2.dex. Inside the app.apk box, the files AndroidManifest.xml, resources.arsc, classes.dex, classes2.dex, classes3.dex and the folders lib, res and assets are circled by dashes and labelled 'original files', and, still inside app.apk, the files classes4.dex, classes5.dex and classes5.dex are circled by dashes and labelled 'Added Files'. Arrows go from lib.dex to classes4.dex, from the classes.dex inside lib.jar to classes5.dex inside app.apk and from classe2.dex inside lib.jar to classes6.dex inside app.apk"
  ),
  caption: [Inserting #DEX files inside an #APK]
) <fig:th-inserting-dex>

In the end, we decided to *not* modify the original code that load the bytecode.
Statically, we already added the bytecode loaded dynamically, and most tools already ignore dynamic code loading.
At runtime, although the bytecode is already present in the application, the application will still dynamically load the code.
This ensure that the application keep working as intended even if the transformation we applied are incomplete.
Specifically, to call dynamically loaded code, an application needs to use reflection, and we saw in @sec:th-trans-ref that we need to keep reflection calls, and in order to keep reflection calls, we need the classloader created when loading bytecode.

=== Class Collisions <sec:th-class-collision>

We saw in @sec:cl/*-obfuscation*/ that having several classes with the same name in the same application can be problematic.
In @sec:th-trans-cl, we are adding new code.
By doing so, we increase the probability of having class collisions:
The developper may have reuse a helper class in both the dynamically loaded bytecode and the application, or an obfuscation process may have rename classes without checking for intersection between the two sources of bytecode.
When loaded dynamically, the classes are in a different classloader, and the class resolution is resolved at runtime like we saw in @sec:cl-loading.
We decided to restrain our scope to the use of class loader from the Android #SDK.
In the abscence of class collision, those class loader behave seamlessly and adding the classes to application maintains the behavior.

When we detect a collision, we rename one of the colliding classes in order to be able to differenciate both classes.
To avoid breaking the application, we then need to rename all references to this specific class, an be carefull not to modify references to the other class.
To do so, we regroup each classes by the classloaders defining them.
Then, for each colliding class name and each classloader, we check the actual class used by the classloader.
If the class has been renamed, we rename all reference to this class in the classes defined by this classloader.
To find the class used by a classloader, we reproduce the behavior of the different classloaders of the Android #SDK.
This is an important step: remember that the delegation process can lead to situation where the class defined by a classloader is not the class that will be loaded when querying the classloader.
The pseudo-code in @lst:renaming-algo show the three steps of this algorithm: 
- First we detect collision and rename classes definitions to remove the collisions.
- Then we rename the reference to the colliding classes to make sure the right classes are called.
- Ultimately, we merge the modified dexfiles of each class loaders into one android application.

#figure(
  ```python
  defined_classes = set()
  redifined_classes = set()

  # Rename the definition of redifined classes
  for cl in class_loaders:
    for clz in defined_classes.intersection(cl.defined_classes):
      cl.rename_definition(clz)
      redifined_classes.add(clz)
    defined_classes.update(cl.defined_classes)

  # Rename reference of redifined classes
  for cl in class_loaders:
    for clz in redifined_classes:
      defining_cl = cl.resolve_class(clz).class_loader
      cl.rename_reference(clz, defining_cl.new_name(clz))

  # Merge the classloader into a flat APK
  new_apk = Apk()
  for cl in class_loaders:
    for dex in cl.get_dex():
      new_apk.add_dex(dex)
  ```,
  caption: [Pseudo-code of the renaming algorithm]
) <lst:renaming-algo>

/*
* Although we limited ourselves to replacing one specific bytecode instruction, we encontered many technical challenges
* #todo[interupting try blocks: catch block might expect temporary registers to still stored the saved value] ?
*/

#h(2em)

Now that we saw the transformations we want to make, we know the runtime information we need to do it.
In the next section, we will propose a solution to collect those informations.