This commit is contained in:
parent
c9752714db
commit
f23390279c
7 changed files with 177 additions and 182 deletions
|
@ -1,23 +1,18 @@
|
|||
#import "../lib.typ": etal, ie, ART, DEX, APK, SDK
|
||||
#import "../lib.typ": etal, ie, ART, DEX, APK, SDK, eg
|
||||
#import "X_var.typ": *
|
||||
|
||||
== Introduction
|
||||
|
||||
In this chapter, we study how Android handles the loading of classes in the case of multiple versions of the same class.
|
||||
Such collision can exist inside the #APK file or between the #APK file and #Asdkc.
|
||||
We intend to understand if a reverser would be impacted during a static analysis when dealing with such an obfuscated code.
|
||||
Because this problem is already complex enough with the current operations performed by Android, we exclude the case where a developer recodes a specific class loader or replace a class loader by another one, as it is often the case for example in packed applications~@Duan2018.
|
||||
We present a new technique that "shadows" a class #ie embeds a class in the #APK file and "presents" it to the reverser instead of the legitimate version.
|
||||
The goal of such an attack is to confuse them during the reversing process: at runtime the real class will be loaded from another location of the #APK file or from the #Asdk, instead of the shadow version.
|
||||
This attack can be applied to regular classes of the #Asdk or to hidden classes of Android~@he_systematic_2023 @li_accessing_2016.
|
||||
We show how these attacks can confuse the tools of the reverser when he performs a static analysis.
|
||||
In order to evaluate if such attacks are already used in the wild, we analysed #nbapk applications from 2023 that we extracted randomly from AndroZoo~@allixAndroZooCollectingMillions2016.
|
||||
Our main result is that #shadowsdk of these applications contain shadow collisions against the #SDK and #shadowhidden against hidden classes.
|
||||
Our investigations conclude that most of these collisions are not voluntary attacks, but we highlight one specific malware sample performing strong obfuscation revealed by our detection of one shadow attack.
|
||||
Such collisions can exist inside the #APK file or between the #APK file and #Asdkc.
|
||||
We intend to understand if a reverser would be impacted during a static analysis when dealing with such obfuscated code.
|
||||
Because this problem is already complex enough with the current operations performed by Android, we exclude the case where the application uses class loaders dynamically (#eg dynamic code loading).
|
||||
We present a new technique that "shadows" a class, #ie embeds a class in the #APK file and "presents" it to the reverser instead of the legitimate version.
|
||||
We show how these attacks can confuse the tools of the reverser when he performs a static analysis, and, in order to evaluate if such attacks are already used in the wild, we analysed #nbapk applications from 2023 that we extracted randomly from AndroZoo~@allixAndroZooCollectingMillions2016.
|
||||
|
||||
The chapter is structured as follows.
|
||||
@sec:cl-loading investigates the internal mechanisms about class loading and presents how a reverser can be confused by these mechanisms.
|
||||
Then in @sec:cl-obfuscation, we design obfuscation techniques and we show their effect on static analysis tools.
|
||||
Next, @sec:cl-wild evaluates if these obfuscation techniques are used in the wild, by searching inside #nbapk APKs if they exploit these techniques.
|
||||
@sec:cl-conclusion extends on the possible countermesures against those shadow attacks, how they interact with other obfuscation techniques, as well as the limitations of this work and avenues left to explore.
|
||||
@sec:cl-loading investigates the internal mechanisms of class loading and presents how a reverser can be confused by these mechanisms.
|
||||
Then, in @sec:cl-obfuscation, we design obfuscation techniques and show their effect on static analysis tools.
|
||||
Next, @sec:cl-wild evaluates if these obfuscation techniques are used in the wild by scanning #nbapk APKs.
|
||||
@sec:cl-conclusion extends on the possible countermeasures against those shadow attacks, how they interact with other obfuscation techniques, as well as the limitations of this work and avenues left to explore.
|
||||
Finally, @sec:cl-conclusion concludes the chapter.
|
||||
|
|
|
@ -1,17 +1,17 @@
|
|||
#import "../lib.typ": todo, ie, etal, num, DEX, ART, SDK, API, APK, APIs, AOSP
|
||||
#import "X_var.typ": *
|
||||
|
||||
== Analyzing the Class Loading Process <sec:cl-loading>
|
||||
== Analysing the Class Loading Process <sec:cl-loading>
|
||||
|
||||
For building obfuscation techniques based on the confusion of tools with class loaders, we manually studied the code of Android that handles class loading.
|
||||
In this section, we report the inner workings of #ART and we focus on the specificities of class loading that can bring confusion.
|
||||
Because the class loading implementation has evolved over time during the multiple iterations of the Android operating system, we mainly describe the behavior of #ART from Android version 14 (#SDK 34).
|
||||
In this section, we report the inner workings of #ART, and we focus on the specificities of class loading that can bring confusion.
|
||||
Because the class loading implementation has evolved over time during the multiple iterations of the Android operating system, we mainly describe the behaviour of #ART from Android version 14 (#SDK 34).
|
||||
|
||||
=== Class Loaders
|
||||
|
||||
When #ART needs to access a class, it queries a `ClassLoader` to retrieve its implementation.
|
||||
When #ART needs to access a class, it queries an object implementing the `ClassLoader` class to retrieve its implementation.
|
||||
Each class has a reference to the `ClassLoader` that loaded it, and this class loader is the one that will be used to load supplementary classes used by the original class.
|
||||
For example in @lst:cl-expl-cl-loading, when calling `A.f()`, the #ART will load `B` with the class loader that was used to load `A`.
|
||||
For example, in @lst:cl-expl-cl-loading, when calling `A.f()`, the #ART will load `B` with the class loader that was used to load `A`.
|
||||
|
||||
#figure(
|
||||
```java
|
||||
|
@ -24,16 +24,16 @@ For example in @lst:cl-expl-cl-loading, when calling `A.f()`, the #ART will load
|
|||
caption: [Class instantiation],
|
||||
) <lst:cl-expl-cl-loading>
|
||||
|
||||
This behavior has been inherited from Java and most of the core classes regarding class loaders have been kept in Android.
|
||||
Nevertheless, the Android implementation has slight differences and new class loaders have been added.
|
||||
For example, the java class loader `URLClassLoader` is still present in Android, but contrary to the official documentation, most of its methods have been removed or replaced by a stub that just raises an exception.
|
||||
This behaviour has been inherited from Java, and most of the core classes regarding class loaders have been kept in Android.
|
||||
Nevertheless, the Android implementation has slight differences, and new class loaders have been added.
|
||||
For example, the Java class loader `URLClassLoader` is still present in Android, but contrary to the official documentation, most of its methods have been removed or replaced by a stub that just raises an exception.
|
||||
Moreover, rather than using the Java class loaders `SecureClassLoader` or `URLClassLoader`, Android has several new class loaders that inherit from `ClassLoader` and override the appropriate methods.
|
||||
|
||||
The left part of @fig:cl-class_loading_classes shows the different class loaders specific to Android in white and the stubs of the original Java class loaders in grey.
|
||||
The main difference between the original Java class loaders and the ones used by Android is that they do not support the Java bytecode format.
|
||||
Instead, the Android-specific class loaders load their classes from (many) different file formats specific to Android.
|
||||
Usually, when used by a programmer, the classes are loaded from memory or from a file using the #DEX format (`.dex`).
|
||||
When used directly by #ART, the classes are usually stored in an application file (`.apk`) or in an optimized format (`OAR/ODEX`).
|
||||
When used directly by #ART, the classes are usually stored in an application file (`.apk`) or in an optimised format (`OAR/ODEX`).
|
||||
|
||||
#figure([
|
||||
#image(
|
||||
|
@ -50,7 +50,7 @@ When used directly by #ART, the classes are usually stored in an application fil
|
|||
On the runtime side, there are 5 boxes: bootClassLoader, appClassLoader (multi dex), systemClassLoader,
|
||||
Specific delegator with two delegates, X.
|
||||
Arrows labeled delegate go from appClassLoader, systemClassLoader, and Specific delegator to bootClassLoader, and from Specific delegator to X.
|
||||
bootClassLoader, appClassLoader, and systemClassLoader are grouped in a dotted box labeled Android default behavior.
|
||||
bootClassLoader, appClassLoader, and systemClassLoader are grouped in a dotted box labeled Android default behaviour.
|
||||
Dotted lines labeled instance go across the central demarcation from appClassLoader to PathClassLoader, from systemClassLoader to PathClassLoader, and from Specific delegator to DelegateLastClassLoader.
|
||||
Another dotted line labeled instance singleton goes from bootClassLoader to BootClassLoader.
|
||||
"
|
||||
|
@ -63,15 +63,15 @@ When used directly by #ART, the classes are usually stored in an application fil
|
|||
=== Delegation <sec:cl-delegation>
|
||||
|
||||
The order in which classes are loaded at runtime requires special attention.
|
||||
All the specific Android class loaders (`DexClassLoader`, `InMemoryClassLoader`, etc.) have the same behavior (except `DelegateLastClassLoader`) but they handle specificities for the input format.
|
||||
All the specific Android class loaders (`DexClassLoader`, `InMemoryClassLoader`, etc.) have the same behaviour (except `DelegateLastClassLoader`), but they handle specificities for the input format.
|
||||
Each class loader has a delegate class loader, represented in the right part of @fig:cl-class_loading_classes by black plain arrows for an instance of `PathClassLoader` and an instance of `DelegateLastClassLoader` (the other class loaders also have this delegate).
|
||||
This delegate is a concept specific to class loaders and has nothing to do with class inheritance.
|
||||
By default, class loaders will delegate to the singleton class `BootClassLoader`, except if a specific class loader is provided when instantiating the new class loader.
|
||||
When a class loader needs to load a class, except for `DelegateLastClassLoader`, it will first ask the delegate, i.e. `BootClassLoader`, and if the delegate does not find the class, the class loader will try to load the class on its own.
|
||||
This behavior implements a priority and avoids redefining by error a core class of the system, for example redefining `java.lang.String` that would be loaded by a child class loader instead of its delegates.
|
||||
`DelegateLastClassLoader` behaves slightly differently: it will first delegate to `BootClassLoader` then, it will check its files and finally, it will delegate to its actual delegate (given when instantiating the `DelegateLastClassLoader`).
|
||||
This behavior is useful for overriding specific classes of a class loader while keeping the other classes.
|
||||
A normal class loader would prioritize the classes of its delegate over its own.
|
||||
This behaviour implements a priority and avoids redefining by error a core class of the system, for example, redefining `java.lang.String` that would be loaded by a child class loader instead of its delegates.
|
||||
`DelegateLastClassLoader` behaves slightly differently: it will first delegate to `BootClassLoader`, then it will check its files, and finally, it will delegate to its actual delegate (given when instantiating the `DelegateLastClassLoader`).
|
||||
This behaviour is useful for overriding specific classes of a class loader while keeping the other classes.
|
||||
A normal class loader would prioritise the classes of its delegate over its own.
|
||||
|
||||
#figure(
|
||||
```python
|
||||
|
@ -105,19 +105,19 @@ It is the direct delegate of the two other class loaders instantiated by Android
|
|||
`systemClassLoader` is a `PathClassLoader` pointing to `'.'`, the working directory of the application, which is `'/'` by default.
|
||||
The documentation of `ClassLoader.getSystemClassLoader` reports that this class loader is the default context class loader for the main application thread.
|
||||
In reality, the #platc are loaded by `bootClassLoader` and the classes from the application are loaded from `appClassLoader`.
|
||||
`systemClassLoader` is never used.
|
||||
`systemClassLoader` is never used in production according to our careful reading of the #AOSP sources.
|
||||
|
||||
In addition to the class loaders instantiated by ART when starting an application, the developer of an application can use class loaders explicitly by calling to ones from the #Asdk, or by recoding custom class loaders that inherit from the `ClassLoader` class.
|
||||
At this point, modeling accurately the complete class loading algorithm becomes impossible: the developer can program any algorithm of their choice.
|
||||
For this reason, this case is excluded from this chapter and we focus on the default behavior where the context class loader is the one pointing to the `.apk` file and where its delegate is `BootClassLoader`.
|
||||
With such a hypothesis, the delegation process can be modeled by the pseudo-code of method `load_class` given in <lst:cl-listing3>.
|
||||
In addition to the class loaders instantiated by ART when starting an application, the developer of an application can use class loaders explicitly by calling ones from the #Asdk, or by recoding custom class loaders that inherit from the `ClassLoader` class.
|
||||
At this point, accurately modelling the complete class loading algorithm becomes impossible: the developer can program any algorithm of their choice.
|
||||
For this reason, this case is excluded from this chapter, and we focus on the default behaviour where the context class loader is the one pointing to the `.apk` file and where its delegate is `BootClassLoader`.
|
||||
With such a hypothesis, the delegation process can be modelled by the pseudo-code of method `load_class` given in @lst:cl-loading-alg.
|
||||
|
||||
In addition, it is important to distinguish the two types of #platc handled by `BootClassLoader` and that both have priority over classes from the application at runtime:
|
||||
|
||||
- the ones available in the *#Asdk* (normally visible in the documentation);
|
||||
- the ones that are internal and that should not be used by the developer. We call them *#hidec*~@he_systematic_2023 @li_accessing_2016 (not documented).
|
||||
|
||||
As a preliminary conclusion, we observe that a priority exists in the class loading mechanism and that an attacker could use it to prioritize an implementation over another one.
|
||||
As a preliminary conclusion, we observe that a priority exists in the class loading mechanism and that an attacker could use it to prioritise an implementation over another one.
|
||||
This could mislead the reverser if they use the one that has the lowest priority.
|
||||
To determine if a class is impacted by the priority given to `BootClassLoader`, we need to obtain the list of classes that are part of Android #ie the #platc.
|
||||
We discuss in the next section how to obtain these classes from the emulator.
|
||||
|
@ -147,22 +147,22 @@ On the top right, a diagram of a web browser open at https//develoer.android.com
|
|||
|
||||
@fig:cl-archisdk shows how classes of Android are used in the development environment and at runtime.
|
||||
In the development environment, Android Studio uses `android.jar` and the specific classes written by the developer.
|
||||
After compilation, only the classes of the developer, and sometimes extra classes computed by Android Studio are zipped in the #APK file, using the multi-dex format.
|
||||
After compilation, only the classes of the developer, and sometimes extra classes computed by Android Studio, are zipped in the #APK file, using the multi-dex format.
|
||||
At runtime, the application uses `BootClassLoader` to load the #platc from Android.
|
||||
Until our work, previous works~@he_systematic_2023 @li_accessing_2016 considered both #Asdk and #hidec to be in the file `/system/framework/framework.jar` found in the phone itself, but we found that the classes loaded by `bootClassLoader` are not all present in `framework.jar`.
|
||||
For example, He #etal~@he_systematic_2023 counted 495 thousand #APIs (fields and methods) in Android 12, based on Google documentation on restriction for non #SDK interfaces#footnote[https://developer.android.com/guide/app-compatibility/restrictions-non-sdk-interfaces].
|
||||
Until our work, previous works~@he_systematic_2023 @li_accessing_2016 considered both #Asdk and #hidec to be in the file `/system/framework/framework.jar` found in the phone itself, but we found that the classes loaded by `bootClassLoader` are not all present in `framework.jar`.
|
||||
For example, He #etal~@he_systematic_2023 counted 495 thousand #APIs (fields and methods) in Android 12, based on Google documentation on restrictions for non #SDK interfaces#footnote[https://developer.android.com/guide/app-compatibility/restrictions-non-sdk-interfaces].
|
||||
However, when looking at the content of `framework.jar`, we only found #num(333) thousand #APIs.
|
||||
Indeed, classes such as `com.android.okhttp.OkHttpClient` are loaded by `bootClassLoader`, listed by Google, but not in `framework.jar`.
|
||||
|
||||
For optimization purposes, classes are now loaded from `boot.art`.
|
||||
For optimisation purposes, classes are now loaded from `boot.art`.
|
||||
This file is used to speed up the start-up time of applications: it stores a dump of the C++ objects representing the *#platc* (#Asdk and #hidec) so that they do not need to be generated each time an application starts.
|
||||
Unfortunately, this format is not documented and not retro-compatible between Android versions and is thus difficult to parse.
|
||||
An easier solution to investigate the #platc is to look at the `BOOTCLASSPATH` environment variable in an emulator.
|
||||
This variable is used to load the classes without the `boot.art` optimization.
|
||||
This variable is used to load the classes without the `boot.art` optimisation.
|
||||
We found 25 `.jar` files, including `framework.jar`, in the `BOOTCLASSPATH` of the standard emulator for Android 12 (#SDK 32), 31 for Android 13 (#SDK 33), and 35 for Android 14 (#SDK 35), containing respectively a total of #num(499837), #num(539236) and #num(605098) API methods and fields.
|
||||
@tab:cl-platform_apis) summarizes the discrepancies we found between Google's list and the #platc we found in Android emulators.
|
||||
@tab:cl-platform_apis summarises the discrepancies we found between Google's list and the #platc we found in Android emulators.
|
||||
Note also that some methods may also be found _only_ in the documentation.
|
||||
Our manual investigations suggest that the documentation is not well synchronized with the evolution of the #platc and that Google has almost solved this issue in #API 34.
|
||||
Our manual investigations suggest that the documentation is not well synchronised with the evolution of the #platc and that Google has almost solved this issue in #API 34.
|
||||
|
||||
|
||||
#figure({
|
||||
|
@ -175,10 +175,10 @@ Our manual investigations suggest that the documentation is not well synchronize
|
|||
table.hline(),
|
||||
table.header(
|
||||
table.cell(colspan: 5, inset: 3pt)[],
|
||||
table.cell(rowspan: 2)[*SDK version*],
|
||||
table.cell(rowspan: 2)[*#SDK version*],
|
||||
table.vline(end: 3),
|
||||
table.vline(start: 4),
|
||||
table.cell(colspan: 4)[*Number of API methods*],
|
||||
table.cell(colspan: 4)[*Number of #API methods*],
|
||||
[Documented], [In emulator], [Only documented], [Only in emulator],
|
||||
),
|
||||
table.cell(colspan: 5, inset: 3pt)[],
|
||||
|
@ -193,7 +193,7 @@ Our manual investigations suggest that the documentation is not well synchronize
|
|||
table.hline(),
|
||||
)},
|
||||
|
||||
caption: [Comparison for #API methods between documentation and emulators],
|
||||
caption: [Comparison of #API methods between documentation and emulators],
|
||||
)<tab:cl-platform_apis>
|
||||
|
||||
We conclude that it can be dangerous to trust the documentation and that gathering information from the emulator or phone is the only reliable source.
|
||||
|
@ -202,20 +202,20 @@ Gathering the precise list of classes and the associated bytecode is not a trivi
|
|||
=== Multiple #DEX Files <sec:cl-collision>
|
||||
|
||||
For the application class files, Android uses its specific format called #DEX: all the classes of an application are loaded from the file `classes.dex`.
|
||||
With the increasing complexity of Android applications, the need arrised to load more methods than the #DEX format could support in one #dexfile.
|
||||
With the increasing complexity of Android applications, the need arose to load more methods than the #DEX format could support in one #dexfile.
|
||||
To solve this problem, Android started storing classes in multiple files named `classesX.dex` as illustrated by the @lst:cl-dexname that generates the filenames read by class loaders.
|
||||
Android starts loading the file `GetMultiDexClassesDexName(0)` (`classes.dex`), then `GetMultiDexClassesDexName(1)` (`classes2.dex`), and continues until finding a value `n` for which `GetMultiDexClassesDexName(n)` does not exist.
|
||||
Even if Android emits a warning message when it finds more than 100 #dexfiles, it will still load any number of #dexfiles that way.
|
||||
This change had the unintended consequence of permitting two classes with the same name but different implementations to be stored in the same `.apk` file using two #dexfiles.
|
||||
This change had the unintended consequence of permitting two classes with the same name but different implementations to be stored in the same `.apk` file using two #dexfiles (#eg the class `Foo` can be defined both in `classes.dex` and `classes2.dex`).
|
||||
|
||||
Android explicitly performs checks that prevent several classes from using the same name inside a #dexfile.
|
||||
However, this check does not apply to multiple #dexfiles in the same `.apk` file, and a `.dex` can contain a class with a name already used by another class in another #dexfile of the application.
|
||||
Of course, such a situation should not happen when multiple #dexfiles have been generated by properly Android Studio.
|
||||
Of course, such a situation should not happen when multiple #dexfiles have been generated properly by Android Studio.
|
||||
Nevertheless, for an attacker controlling the process, this issue raises the question of which class is selected when several classes sharing the same name are present in `.apk` files.
|
||||
|
||||
We found that Android loads the class whose implementation is found first when looking in the order of multiple `dexfiles`, as generated by the method `GetMultiDexClassesDexName`.
|
||||
We will show later in @sec:cl-evaltools that this choice is not the most intuitive and can lead to fool analysis tools when reversing an application.
|
||||
As a conclusion, we model both the multi-dex and delegation behaviors in the pseudo-code of @lst:cl-loading-alg.
|
||||
We will show later in @sec:cl-evaltools that this choice is not the most intuitive and can lead to fooling analysis tools when reversing an application.
|
||||
As a conclusion, we model both the multi-dex and delegation behaviours in the pseudo-code of @lst:cl-loading-alg.
|
||||
|
||||
#figure(
|
||||
```C
|
||||
|
|
|
@ -6,9 +6,9 @@
|
|||
|
||||
In this section, we present new obfuscation techniques that take advantage of the complexity of the class loading process.
|
||||
Then, in order to evaluate their efficiency, we reviewed some common Android reverse analysis tools to see how they behave when collisions occur between classes of the #APK or between a class of the #APK and classes of Android (#Asdk or #hidec).
|
||||
We call this collision "*class shadowing*", because the attacker version of the class shadows the one that will be used at runtime.
|
||||
To evaluate if such shadow attacks are working, we handcrafted three applications implementing shadowing techniques to test their impact on static analysis tools.
|
||||
Then, we manually inspected the output of the tools in order to check its consistency with what Android is really doing at runtime.
|
||||
We call this collision "*class shadowing*", because the attacker's version of the class shadows the one that will be used at runtime.
|
||||
To evaluate if such shadow attacks are working, we handcrafted three applications implementing shadowing techniques to test their impact on static analysis tools.
|
||||
Then, we manually inspected the output of the tools in order to check their consistency with what Android is really doing at runtime.
|
||||
For example, for Apktool, we look at the output disassembled code, and for Flowdroid~@Arzt2014a, we check that a flow between `Taint.source()` and `Taint.sink()` is correctly computed.
|
||||
|
||||
|
||||
|
@ -24,7 +24,7 @@ on peut shadow une classe hidden
|
|||
|
||||
=== Obfuscation Techniques
|
||||
|
||||
From the results presented in @sec:cl-loading, three approaches can be designed to hide the behavior of an application.
|
||||
From the results presented in @sec:cl-loading, three approaches can be designed to hide the behaviour of an application.
|
||||
|
||||
/*
|
||||
#paragraph([Hidden classes])[
|
||||
|
@ -41,24 +41,24 @@ On the other hand, using #hidec leave classes without implementation in the appl
|
|||
*/
|
||||
|
||||
#paragraph([*Self shadow*: shadowing a class with another from #APK])[
|
||||
This method consists in hiding the implementation of a class with another one by exploiting the possible collision of class names, as described in @sec:cl-collision with multiple #dexfiles.
|
||||
This method consists of hiding the implementation of a class with another one by exploiting the possible collision of class names, as described in @sec:cl-collision with multiple #dexfiles.
|
||||
If reversers or tools ignore the priority order of a multi-dex file, they can take into account the wrong version of a class.
|
||||
]
|
||||
|
||||
//priorité aux classes SDK meme si une shadow classe est définie dans l'APK (tout ca a cause de Boot)
|
||||
#paragraph([*SDK shadow*: shadowing a #SDK class])[
|
||||
This method consists in presenting to the reverser a fake implementation of a class of the #SDK.
|
||||
This class is embedded in the #APK file and has the same name as the one of the #SDK.
|
||||
This method consists of presenting to the reverser a fake implementation of a class of the #SDK.
|
||||
This class is embedded in the #APK file and has the same name as one of the #SDK.
|
||||
Because `BootClassLoader` will give priority to the #Asdk at runtime, the reverser or tool should ignore any version of a class that is contained in the #APK.
|
||||
The only constraint when shadowing an #SDK class is that the shadowing implementation must respect the signature of real classes.
|
||||
Note that, by introducing a custom class loader, the attacker could inverse the priority, but this case is out of the scope of this chapter.
|
||||
Note that, by introducing a custom class loader, the attacker could invert the priority, but this case is out of the scope of this chapter.
|
||||
]
|
||||
|
||||
// priorité aux classes hidden (car du SDK) meme si une shadow classe est définie dans l'APK
|
||||
#paragraph([*Hidden shadow*: shadowing an hidden class])[
|
||||
#paragraph([*Hidden shadow*: shadowing a hidden class])[
|
||||
This method is similar to the previous one, except the class that is shadowed is a #hidecsingular.
|
||||
Because #ART will give priority to the internal version of the class, the version provided in the #APK file will be ignored.
|
||||
Such shadow attacks are more difficult to detect by a reverser, that may not know the existence of this specific hidden class in Android.
|
||||
Such shadow attacks are more difficult to detect by a reverse engineer, who may not know the existence of this specific hidden class in Android.
|
||||
]
|
||||
|
||||
=== Impact on Static Analysis Tools <sec:cl-evaltools>
|
||||
|
@ -73,7 +73,7 @@ Such shadow attacks are more difficult to detect by a reverser, that may not kno
|
|||
}
|
||||
}
|
||||
|
||||
// customized for each obfuscation technique
|
||||
// customised for each obfuscation technique
|
||||
public class Obfuscation {
|
||||
public static String hide_flow(String personal_data) { ... }
|
||||
}
|
||||
|
@ -81,28 +81,28 @@ Such shadow attacks are more difficult to detect by a reverser, that may not kno
|
|||
caption: [Main body of test apps]
|
||||
)<lst:cl-testapp>
|
||||
|
||||
|
||||
|
||||
We selected tools that are commonly used to unpack and reverse Android applications.
|
||||
The only two tools that we found to still be alive in @sec:rasta-src-select: Androguard#footnote[https://github.com/androguard/androguard] and Flowdroid~@Arzt2014a.
|
||||
We also selected Jadx#footnote[https://github.com/skylot/jadx], a state-of-the-art decompiler for Android applications, as well as Apktool#footnote[https://apktool.org/], a disassembler/repackager used by 9 of the tools tested in @sec:rasta and often used by reverser when Jadx fails.
|
||||
In @sec:rasta (@sec:rasta-src-select), we found only two tools to be still actively maintained: Androguard#footnote[https://github.com/androguard/androguard] and Flowdroid#footnote[https://github.com/secure-software-engineering/FlowDroid].
|
||||
We also noticed that Apktool#footnote[https://apktool.org/] was a common dependency for a lot of the tools we tested in @sec:rasta (see @tab:rasta-rec-deps), and is still used today.
|
||||
Consequently, we will test the impact of shadow attacks on those three tools.
|
||||
Lastly, because it is a state-of-the-art decompiler for Android applications, we added Jadx#footnote[https://github.com/skylot/jadx] to the list of tools we tested.
|
||||
|
||||
To evaluate the tools, we designed a single application that we can customize for different tests.
|
||||
To evaluate the tools, we designed a single application that we can customise for different tests.
|
||||
@lst:cl-testapp shows the main body implementing:
|
||||
- a possible flow to evaluate FlowDroid: a flow from a method `Taint.source()` to a method `Taint.sink(Activity, String)` through a method `Obfuscation.hide_flow(String)`.
|
||||
- a possible use of a #SDK or hidden class inside the class `Obfuscation` to evaluate #platc shadowing for other tools.
|
||||
|
||||
We used 4 versions of this application:
|
||||
|
||||
+ A control application that does not do anything special: `Obfuscation.hide_flow(String personal_data)` simply return `personal_data`.
|
||||
It will be used for checking the expecting result of tools.
|
||||
+ A version that implements self shadowing: the class `Obfuscation` is duplicated: one is the same as the in the control app (`Obfuscation.hide_flow(String)` returns its arguments), and the other version returns a constant string.
|
||||
+ A control application that does not do anything special: `Obfuscation.hide_flow(String personal_data)` returns `personal_data`.
|
||||
It will be used for checking the expected result of tools.
|
||||
+ A version that implements self-shadowing: the class `Obfuscation` is duplicated: one is the same as the one in the control app (`Obfuscation.hide_flow(String)` returns its arguments), and the other version returns a constant string.
|
||||
These two versions are embedded in several #DEX of a multi-dex application.
|
||||
+ The third version implement #SDK shadowing and needs an existing class of the #SDK.
|
||||
We used the #SDK class `Pair` that we try to shadow.
|
||||
+ The third version implements #SDK shadowing and needs an existing class of the #SDK.
|
||||
We used the #SDK class `Pair` as the class to shadow.
|
||||
We put data in a new `Pair` instance and reread the data from the `Pair`.
|
||||
The colliding `Pair` class we created discards the data at the initialisation and stores `null` instead of the argument values.
|
||||
This decoy class break the flow of information: Flowdroid will detect the information flow if it uses the actuall #SDK implementation of `Pair` to compute the #DFG, but not if it uses the decoy.
|
||||
This decoy class break the flow of information: Flowdroid will detect the information flow if it uses the actual #SDK implementation of `Pair` to compute the #DFG, but not if it uses the decoy.
|
||||
+ The last version tests for Hidden #API shadowing.
|
||||
Like for the third one, we similarly store data in `com.android.okhttp.Request` and then retrieve it.
|
||||
Again, the shadowing implementation discards the data.
|
||||
|
@ -113,7 +113,7 @@ In @tab:cl-results, we report on the types of shadowing that can trick each tool
|
|||
A plain circle is a shadow attack that leads to a wrong result.
|
||||
A white circle indicates a tool emitting warnings or that displays the two versions of the class.
|
||||
A cross is a tool not impacted by a shadow attack.
|
||||
We explain in more detail in the following the results for each considered tool.
|
||||
//We explain in more detail in the following the results for each considered tool.
|
||||
|
||||
#figure({
|
||||
table(
|
||||
|
@ -128,7 +128,7 @@ We explain in more detail in the following the results for each considered tool.
|
|||
table.vline(end: 3),
|
||||
table.vline(start: 4),
|
||||
table.cell(colspan: 3)[Shadow Attack],
|
||||
[Self], [SDK], [Hidden],
|
||||
[Self], [#SDK], [Hidden],
|
||||
),
|
||||
table.cell(colspan: 5, inset: 3pt)[],
|
||||
table.hline(),
|
||||
|
@ -149,22 +149,22 @@ We explain in more detail in the following the results for each considered tool.
|
|||
|
||||
==== Jadx
|
||||
|
||||
Jadx is a reverse engineering tool that regenerates the Java source code of an application.
|
||||
It processes all the classes present in the application, but only save/display one class by name, even if two versions are present in multiple #dexfiles.
|
||||
//Jadx is a reverse engineering tool that regenerates the Java source code of an application.
|
||||
Jadx processes all the classes present in the application, but only saves/displays one class by name, even if two versions are present in multiple #dexfiles.
|
||||
Nevertheless, when multiple classes with the same name are found, Jadx reports it in a comment added to the generated Java source code.
|
||||
This warning stipulates that a possible collision exists and lists the files that contain the different versions of the class.
|
||||
Unfortunately, after reviewing the code of Jadx, we believe that the selection of the displayed class is an undefined behavior.
|
||||
At least for the version 1.5.0 that we tested, we found that Jadx selects the wrong implementation when a class with the same name is present.
|
||||
For example in `classes2.dex` and `classes3.dex`.
|
||||
Unfortunately, after reviewing the code of Jadx, we believe that the selection of the displayed class is an undefined behaviour.
|
||||
At least for version 1.5.0 that we tested, we found that Jadx selects the wrong implementation when a class with the same name is present.
|
||||
For example, in `classes2.dex` and `classes3.dex`.
|
||||
We report it with a "#warn" because warnings are issued.
|
||||
|
||||
//Using #hidec does not affect Jadx beyond the fact that #hidec are not decompiled, which is to be expected by the user anyway.
|
||||
|
||||
Shadowing #Asdk and #hidec is possible in Jadx: there is only one implementation of the class in the application and Jadx does not have a list of the internal classes of Android: no warning is issued to the reverser that the displayed class is not the one used by Android.
|
||||
Shadowing #Asdk and #hidec is possible in Jadx: there is only one implementation of the class in the application, and Jadx does not have a list of the internal classes of Android: no warning is issued to the reverser that the displayed class is not the one used by Android.
|
||||
|
||||
==== Apktool
|
||||
|
||||
Apktool generates Smali files, an assembler language for #DEX bytecode.
|
||||
//Apktool generates Smali files, an assembler language for #DEX bytecode.
|
||||
Apktool will store the disassembled classes in a folder that matches the #dexfile that stores the bytecode.
|
||||
This means that when shadowing a class with two versions in two #dexfiles, the shadow implementations will be disassembled into two directories.
|
||||
No indication is displayed that a collision is possible.
|
||||
|
@ -178,8 +178,8 @@ Androguard has different usages, with different levels of analysis.
|
|||
The documentation highlights the analysis commands that compute three types of objects: an #APK object, a list of #DEX objects, and an Analysis object.
|
||||
The #APK and the list of #dexfiles are a one-to-one representation of the content of an application, and have the same issues that we discussed with Apktool: they provide the different versions of a shadow class contained in multiple #dexfiles.
|
||||
|
||||
The Analysis object is used to compute a method call graph and we found that this algorithm may choose the wrong version of a shadowed class when using the cross references that are computed.
|
||||
This leads to an invalid call graph as shown in @fig:cl-andro_obf_cg: the two methods `doSomething()` are represented in the graph, but the one linked to `main()` on the graph is the one calling the method `good()` when in fact the method `bad()` is called when running the application.
|
||||
The Analysis object is used to compute a method call graph, and we found that this algorithm may choose the wrong version of a shadowed class when using the cross-references that are computed.
|
||||
This leads to an invalid call graph, as shown in @fig:cl-andro_obf_cg: the two methods `doSomething()` are represented in the graph, but the one linked to `main()` on the graph is the one calling the method `good()` when in fact the method `bad()` is called when running the application.
|
||||
|
||||
Androguard has a method `.is_external()` to detect if the implementation of a class is not provided inside the application and a method `.is_android_api()` to detect if the class is part of the Android #API.
|
||||
Regrettably, the documentation of `.is_android_api()` explains that the method is still experimental and just checks a few package names.
|
||||
|
@ -224,11 +224,12 @@ Because of that, like for Apktool and Jadx, Androguard has no way to warn the re
|
|||
) <fig:cl-andro_obf_cg>
|
||||
])
|
||||
h(1em)},
|
||||
caption: [Call Graphs of an application calling `Main.bad()` from a shadowed `Obfuscation` class.],
|
||||
caption: [Call Graphs of an application calling `Main.bad()` from a shadowed `Obfuscation` class],
|
||||
)<fig:cl-androguard_call_graph>
|
||||
|
||||
==== Flowdroid
|
||||
|
||||
/*
|
||||
#jfl-note[Flowdroid~@Arzt2014a is used to detect if an application can leak sensitive information.
|
||||
To do so, the analyst provides a list of source and sink methods.
|
||||
The return value of a method marked as source is considered sensitive and the argument of a method marked as sink is considered to be leaked.
|
||||
|
@ -237,22 +238,24 @@ Flowdroid is built on top of the Soot~@Arzt2013 framework that handles, among ot
|
|||
deja dit dans chap2?
|
||||
|
||||
Non mais on aurait du, ca viendra et il faudra modifier a ce moment là
|
||||
]
|
||||
]*/
|
||||
|
||||
We found that when selecting the classes implementation in a multi-dex #APK, Soot uses an algorithm close to what #ART is performing:
|
||||
Soot sorts the `.dex` bytecode file with a specified `prioritizer` (a comparison function that defines an order for #dexfiles) and selects the first implementation found when iterating over the sorted files.
|
||||
Unfortunately, the `prioritizer` used by Soot is not exactly the same as the one used by the ART.
|
||||
The Soot `prioritizer` will give priority to `classes.dex` and then give priority to files whose name starts with `classes` over other files and finally will use the alphabetical order.
|
||||
This order is good enough for application with a small number of #dexfiles generated by Android Studio, but because it uses the alphabetical order and does not check the exact format used by Android, a malicious developer could hide the implementation of a class in `classes2.dex` by putting a false implementation in `classes0.dex`, `classes1.dex` or `classes12.dex`.
|
||||
The Soot `prioritizer` will give priority to `classes.dex` and then give priority to files whose name starts with `classes` over other files, and finally will use alphabetical order.
|
||||
This order is good enough for application with a small number of #dexfiles generated by Android Studio, but because it uses the alphabetical order and does not check the exact format used by Android, a malicious developer could hide the implementation of a class in `classes2.dex` by putting a false implementation in `classes0.dex`, `classes1.dex` or `classes12.dex`.
|
||||
Because Flowdroid is based on Soot, it inherits this issue from it.
|
||||
|
||||
// TODO This could use more investigation
|
||||
In addition to self shadowing, Flowdroid is sensitive to the use of #platc, as it needs the bytecode of those classes to be able to track data flows.
|
||||
This is solved for #SDK classes by providing `android.jar` to Flowdroid.
|
||||
Flowdroid gives priority to the classes from the #SDK over the classes implemented in the application, thus defeating #SDK shadow attacks.
|
||||
Unfortunately, `android.jar` only contains classes from the #Asdk, meaning that using #hidec breaks the flow tracking.
|
||||
Solving this issue would require finding the bytecode of all the platform classes of the Android version targeted and as we said previously it requires extracting this information from the emulator.
|
||||
In addition to self-shadowing, Flowdroid is sensitive to the use of #platc, as it needs the bytecode of those classes to be able to track data flows.
|
||||
//This is solved for #SDK classes by providing `android.jar` to Flowdroid.
|
||||
Flowdroid does have a record of #SDK classes, and gives priority to the actual #SDK classes over the classes implemented in the application, thus defeating #SDK shadow attacks.
|
||||
//Unfortunately, `android.jar` only contains classes from the #Asdk, meaning that using #hidec breaks the flow tracking.
|
||||
Unfortunately, Flowdroid does not have a record of all platform classes, meaning that using #hidec breaks the flow tracking.
|
||||
Solving this issue would require finding the bytecode of all the platform classes of the Android version targeted, and, as we said previously, it requires extracting this information from the emulator or phone.
|
||||
|
||||
#v(2em)
|
||||
|
||||
We have seen that tools can be impacted by shadow attacks. In the next section, we will investigate if these attacks are used in the wild.
|
||||
We have seen that tools can be impacted by shadow attacks. In the next section, we will investigate whether these attacks are used in the wild.
|
||||
|
||||
|
|
|
@ -3,16 +3,16 @@
|
|||
|
||||
== Shadow Attacks in the Wild <sec:cl-wild>
|
||||
|
||||
In this section, we evaluate in the wild if applications that can be found in the Play store or other markets use one of the shadow techniques.
|
||||
In this section, we evaluate in the wild if applications that can be found in the Play Store or other markets use one of the shadow techniques.
|
||||
Our goal is to explore the usage of shadow techniques in real applications.
|
||||
Because we modeled the behavior of a rescent version of Android (#SDK 34), we decided to not used our dataset from @sec:rasta.
|
||||
The applications in the RASTA dataset span over more than 10 years and we cannot garanties that sandow attacks behaved the same during those 10 years.
|
||||
At the verry least, self-shadowing would not be possible before the introduction of multi-dex in 2014 -- about a fourth of the applications in the RASTA dataset.
|
||||
Instead, sampled another dataset of recent applications.
|
||||
We want to include malicious applications (in case such techniques would be used to hide malicious code) so we selected #num(50000) applications randomly from AndroZoo~@allixAndroZooCollectingMillions2016 that appeared in 2023.
|
||||
Malicious applications are spot in our dataset by using a threshold of 3 over the number of VirusTotal engines reporting an application as a malware.
|
||||
This number is provided by Androzoo, for scans performed between january 2023 and january 2024 depending on the application.
|
||||
A few applications over the total could not be retrieved or parsed leading to a final dataset of #nbapk applications.
|
||||
Because we modelled the behaviour of a recent version of Android (#SDK 34), we decided not to use our dataset from @sec:rasta.
|
||||
The applications in the RASTA dataset span over more than 10 years, and we cannot guarantee that sandow attacks behaved the same during those 10 years.
|
||||
At the very least, self-shadowing would not be possible before the introduction of multi-dex in 2014 -- about a fourth of the applications in the RASTA dataset.
|
||||
Instead, we sampled another dataset of recent applications.
|
||||
This way, we can also include malicious applications (in case such techniques would be used to hide malicious code), so we selected #num(50000) applications randomly from AndroZoo~@allixAndroZooCollectingMillions2016 that appeared in 2023.
|
||||
Malicious applications are spotted in our dataset by using a threshold of 3 over the number of VirusTotal engines reporting an application as malware.
|
||||
This number is provided by Androzoo for scans performed between January 2023 and January 2024, depending on the application.
|
||||
A few applications over the total could not be retrieved or parsed, leading to a final dataset of #nbapk applications.
|
||||
We automatically disassembled the applications to obtain the list of included classes.
|
||||
Then, we check if any shadow attack occurs in the #APK itself or with #platc of #SDK 34.
|
||||
|
||||
|
@ -24,7 +24,6 @@ on prend les classes des platform classes et
|
|||
comparé à SDK 32 33 34: si la shadow class match, alors match
|
||||
*/
|
||||
|
||||
|
||||
#todo[cl-shadow]
|
||||
#figure({
|
||||
show table: set text(size: 0.80em)
|
||||
|
@ -49,8 +48,7 @@ comparé à SDK 32 33 34: si la shadow class match, alors match
|
|||
|
||||
[],
|
||||
[], [*%*], [*% malware*],
|
||||
[*Shadow classes*], [*Median*], [*Target SDK*], [*Min SDK*],
|
||||
|
||||
[*Shadow classes*], [*Median*], [*Target #SDK*], [*Min #SDK*],
|
||||
),
|
||||
table.cell(colspan: 9, inset: 3pt)[],
|
||||
table.hline(),
|
||||
|
@ -87,23 +85,23 @@ comparé à SDK 32 33 34: si la shadow class match, alors match
|
|||
//The metadata provided by AndroZoo helps to have the flags reported by antiviruses used by VirusTotal#footnote[https://www.virustotal.com].
|
||||
|
||||
|
||||
We report in the upper part of @tab:cl-shadow the statistics about the whole dataset and the three shadow attacks: "self" when a class shadows another one in the #APK, "#SDK" when a class of the #SDK shadows one of the #APK, and "Hidden" when a hidden class of Android shadows one of the #APK.
|
||||
We report in the upper part of @tab:cl-shadow the statistics about the whole dataset and the three shadow attacks: "self" when a class shadows another one in the #APK, "#SDK" when a class of the #SDK shadows one of the #APK, and "Hidden" when a hidden class of Android shadows one of the #APK.
|
||||
We observe that, on average, a few classes are shadowed by another class.
|
||||
Note that the median value is 0 meaning that few apps shadow a lot of classes, but the majority of apps do not shadow anything.
|
||||
Note that the median value is 0, meaning that few apps shadow a lot of classes, but the majority of apps do not shadow anything.
|
||||
The number of applications shadowing a hidden #API is low, which is an expected result as these classes should not be known by the developer.
|
||||
We observe a consequent number of applications, 23.52%, of applications that perform #SDK shadowing.
|
||||
It can be explained by the fact that some classes that newly appear are embedded in the #APK for end users that have old versions of Android: it is suggested by the average value of Min #SDK which is 21.7 for the whole dataset: on average, an application can be run inside a smartphone with #API 21, which would require to embed all new classes from 22 to 34.
|
||||
This hypothesis about missing classes is further investigated later in this section.
|
||||
|
||||
In the bottom part of @tab:cl-shadow, we give the same statistics but we excluded applications that do not perform any shadowing.
|
||||
In the bottom part of @tab:cl-shadow, we give the same statistics, but we excluded applications that do not perform any shadowing.
|
||||
For those pairs of shadow classes, we disassembled them using Apktool to perform a comparison using instructions represented in the Smali language.
|
||||
For self-shadow, we compare the pair.
|
||||
For the shadowing of the #SDK or Hidden class, we compare the code found in the #APK with implementations found in the emulator and `android.jar` of #SDK 32, 33, and 34.
|
||||
|
||||
#paragraph([Self-shadowing])[
|
||||
We observe a low number of applications doing self-shadow attacks.
|
||||
For each class that is shadowed, we compared its bytecode with the shadowed one.
|
||||
We observe that 74.8% are identical which suggests that the compilation process embeds the same class multiple times but makes variations in headers or metadata values.
|
||||
For each class that is shadowed, we compared its bytecode with the shadowed one (we compared the Smali instructions generated by Apktool for each method).
|
||||
We observe that 74.8% are identical, which suggests that the compilation process embeds the same class multiple times but makes variations in headers or metadata values.
|
||||
We investigate later in @sec:cl-malware the case of malicious applications.
|
||||
]
|
||||
|
||||
|
@ -122,13 +120,13 @@ We investigate later in @sec:cl-malware the case of malicious applications.
|
|||
The remaining bars are between 0 and 5,000.
|
||||
"
|
||||
),
|
||||
caption: [Redefined #SDK classes, sorted by the first #SDK they appeared in.]
|
||||
caption: [Redefined #SDK classes, sorted by the first #SDK they appeared in]
|
||||
)<fig:cl-classes_by_first_sdk>
|
||||
|
||||
#paragraph([#SDK shadowing])[
|
||||
For the shadowing of #SDK classes, we observe a low ratio of identical classes.
|
||||
This result could lead to the wrong conclusion that developers embed malicious versions of the #SDK classes, but our manual investigation shows that the difference is slight and probably due to compiler optimization.
|
||||
To go further in the investigation, in @fig:cl-classes_by_first_sdk we represent these redefined classes with the following rules:
|
||||
This result could lead to the wrong conclusion that developers embed malicious versions of the #SDK classes, but our manual investigation shows that the difference is slight and probably due to compiler optimisation.
|
||||
To go further in the investigation, in @fig:cl-classes_by_first_sdk, we represent these redefined classes with the following rules:
|
||||
|
||||
- The class is classified on the X abscissa in the figure according to the #SDK it first appeared in.
|
||||
- The class is counted as "green" (solid) if it first appeared in the #SDK *after* the #APK min #SDK (retro compatibility purpose).
|
||||
|
@ -138,19 +136,19 @@ We observe that the majority of classes are legitimate retro-compatibility addit
|
|||
Abnormal cases are observed for classes that appeared in #API versions 7 and before, 8, and 16.
|
||||
@tab:cl-topsdk reports the top ten classes that shadow the #SDK for the three mentioned versions.
|
||||
For #SDK before 7, it mainly concerns HTTP classes: for example, the class `HttpParams` is an interface, containing limited bytecode that mostly matches the class already present on the emulator (98.03% of shadowed classes are identical).
|
||||
`HttpConnectionParams` on the other hand differs from the platform class and we observe only 4.99% of identical classes.
|
||||
`HttpConnectionParams`, on the other hand, differs from the platform class, and we observe only 4.99% of identical classes.
|
||||
Manual inspection of some applications revealed that the two main reasons are:
|
||||
|
||||
|
||||
- instead of checking if the methods attributes are null inline like Android does, applications use the method `org.apache.http.util.Args.notNull()`. According to comments in the source code of Android#footnote[https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/org/apache/http/params/HttpConnectionParams.java;drc=3bdd327f8532a79b83f575cc62e8eb09a1f93f3d?], the class was forked in 2007 from Apache 'httpcomponents' project. Looking at the history of the project, the use of `Args.notNull()` was introduced in 2012#footnote[https://github.com/apache/httpcomponents-core/commit/9104a92ea79e338d876b1b60f5cd2b243ba7069f?]. This shows that applications are embedding code from more recent version of this library without realizing their version will not be the used one.
|
||||
- very small changes that we found can be attributed to the compilation process (e.g. swapping registers: `v0` is used instead of `v1` and `v1` instead of `v0`), but even if we consider them different, they are very similar.
|
||||
- Instead of checking if the method's attributes are null inline, like Android does, applications use the method `org.apache.http.util.Args.notNull()`. According to comments in the source code of Android#footnote[https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/org/apache/http/params/HttpConnectionParams.java;drc=3bdd327f8532a79b83f575cc62e8eb09a1f93f3d?], the class was forked in 2007 from the Apache 'httpcomponents' project. Looking at the history of the project, the use of `Args.notNull()` was introduced in 2012#footnote[https://github.com/apache/httpcomponents-core/commit/9104a92ea79e338d876b1b60f5cd2b243ba7069f?]. This shows that applications are embedding code from more recent versions of this library without realising their version will not be the one used.
|
||||
- Very small changes that we found can be attributed to the compilation process (e.g. swapping registers: `v0` is used instead of `v1` and `v1` instead of `v0`), but even if we consider them different, they are very similar.
|
||||
|
||||
The remaining 4.99% of classes that are identical to the Android version are classes where the body of the methods is replaced by stubs that throw `RuntimeException("Stub!")`.
|
||||
This code corresponds to what we found in `android.jar` but not the code we found in the emulator, which is surprising.
|
||||
Nevertheless, we decided to count them as identical, because `android.jar` is the official jar file for developer, and stubs are replaced in the emulator: it is intended by Google developers.
|
||||
This code corresponds to what we found in `android.jar`, but not the code we found in the emulator, which is surprising.
|
||||
Nevertheless, we decided to count them as identical, because `android.jar` is the official jar file for developers, and stubs are replaced in the emulator: it is intended by Google developers.
|
||||
|
||||
Other results of @tab:cl-topsdk can be similarly discussed: either they are identical with a high ratio, or they are different because of small variations.
|
||||
When substantial differences appear it is mainly because different versions of the same library have been used or an #SDK class is embedded for retro-compatibility.
|
||||
When substantial differences appear, it is mainly because different versions of the same library have been used or an #SDK class is embedded for retro-compatibility.
|
||||
]
|
||||
|
||||
#figure({
|
||||
|
@ -168,7 +166,7 @@ When substantial differences appear it is mainly because different versions of t
|
|||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
table.hline(),
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
[redefined for SDK $<=$ 7], [], [],
|
||||
[redefined for #SDK $<=$ 7], [], [],
|
||||
table.hline(),
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
..redef_sdk_7minus.map(e => (
|
||||
|
@ -178,7 +176,7 @@ When substantial differences appear it is mainly because different versions of t
|
|||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
table.hline(),
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
[redefined for SDK $=$ 8], [], [],
|
||||
[redefined for #SDK $=$ 8], [], [],
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
table.hline(),
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
|
@ -189,7 +187,7 @@ When substantial differences appear it is mainly because different versions of t
|
|||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
table.hline(),
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
[redefined for SDK $=$ 16], [], [],
|
||||
[redefined for #SDK $=$ 16], [], [],
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
table.hline(),
|
||||
table.cell(colspan: 3, inset: 2pt)[],
|
||||
|
@ -209,7 +207,7 @@ For applications redefining hidden classes, on average, 16.1 classes are redefin
|
|||
The top 3 packages whose code actually differs from the ones found in Android are `java.util.stream`, `org.ccil.cowan.tagsoup` and `org.json`:
|
||||
|
||||
- stream: when looking in more detail, we found that `java.util.stream` was only redefined by 6 applications, but the large number of classes redefined artificially puts the package at the top of the list.
|
||||
It is explained by the fact that developers have included this library containing a lot of classes colliding with Android.
|
||||
It is explained by the fact that developers have included this library, containing a lot of classes that collide with Android.
|
||||
- tagsoup: `TagSoup` is a library for parsing HTML.
|
||||
Developers do not know that it is part of Android as hidden classes.
|
||||
- json: there is only one hidden class in `org.json`, redefined by #num(821) applications: `JSONObject$1`.
|
||||
|
@ -265,32 +263,32 @@ All these hidden shadow classes are libraries included by the developers who pro
|
|||
// ...
|
||||
}
|
||||
```,
|
||||
caption: [Implementation of Reflection executed by #ART (shadowed by @lst:cl-refl2],
|
||||
caption: [Implementation of Reflection executed by #ART (shadowed by @lst:cl-refl2)],
|
||||
) <lst:cl-refl1>
|
||||
|
||||
The last column of @tab:cl-shadow shows the proportion of applications considered as malware because we arbitrarily fixed a threshold of 3 positive detections from VirusTotal reports.
|
||||
For the whole dataset, we have 0.53% of applications considered as malware.
|
||||
We can see that an application that uses self-shadowing is 10 times more likely to be a malware, when the proportion of malware among application shadowing #platc is the same as in the rest of the dataset.
|
||||
Thus, we manually reversed self-shadowing malware, and found that the self-shadowing does not look to be voluntary.
|
||||
We can see that an application that uses self-shadowing is 10 times more likely to be malware, when the proportion of malware among application shadowing #platc is the same as in the rest of the dataset.
|
||||
Thus, we manually reversed self-shadowing malware and found that the self-shadowing does not look to be voluntary.
|
||||
The colliding classes are often the same implementation, occasionally with minor differences, like different versions of a library.
|
||||
Additionally, we noticed multiple times internal classes from `com.google.android.gms.ads` colliding with each other, but we believe that it is due to bad processing during the compilation of the application.
|
||||
|
||||
// Nom de l'app: ShareCRM, mais ca a l'air d'exister sur le store donc on va eviter un process et pas la nommer
|
||||
// https://play.google.com/store/apps/details?id=com.facishare.fsplay&hl=en
|
||||
|
||||
The most notable case we found was an application that still exists on the Google Play Store with the same package name#footnote[SHA256: `C46A65EA1A797119CCC03C579B61C94FE8161308A3B6A8F55718D6ADAD112546`]. This application contains a self-shadow class `me.weishu.reflection.Reflection` that can be found in github, in the repository `tiann/FreeReflection`#footnote[https://github.com/tiann/FreeReflection]. This class is used to disable Android restrictions on hidden #API.
|
||||
The most notable case we found was an application that still exists on the Google Play Store with the same package name#footnote[SHA256: `C46A65EA1A797119CCC03C579B61C94FE8161308A3B6A8F55718D6ADAD112546`]. This application contains a self-shadow class `me.weishu.reflection.Reflection` that can be found in Github, in the repository `tiann/FreeReflection`#footnote[https://github.com/tiann/FreeReflection]. This class is used to disable Android restrictions on hidden #API.
|
||||
At first glance, we believed the shadowing to be done voluntarily for obfuscation purposes.
|
||||
The shadow class that would be seen by a reverser is given in @lst:cl-refl2: it contains some Java bytecode performing reflection and loading a native library named "free-reflection" (the associated `.so` is missing).
|
||||
The shadowed class that is really executed is summarized in @lst:cl-refl1.
|
||||
It contains a more obfuscated code: a `DEX` field storing base64 encoded #DEX bytecode that is later used to load some new code.
|
||||
When looking at this new code stored in the field, we found that it does almost the same thing as the code in the shadow class.
|
||||
Thus, we believe that the developer has upgraded their obfuscation techniques, replacing a native library by inline base64 encoded bytecode.
|
||||
The shadow attack could be unintentional, but it strengthens the masking of the new implementation.
|
||||
The shadow class that would be seen by a reverser is given in @lst:cl-refl2: it contains some Java bytecode performing reflection and loading a native library named "free-reflection" (the associated `.so` is missing).
|
||||
The shadowed class that is really executed is summarised in @lst:cl-refl1.
|
||||
It contains a more obfuscated code: a `DEX` field storing base64 encoded #DEX bytecode that is later used to load some new code.
|
||||
When looking at this new code stored in the field, we found that it does almost the same thing as the code in the shadow class.
|
||||
Thus, we believe that the developer has upgraded their obfuscation techniques, replacing a native library with inline base64 encoded bytecode.
|
||||
The shadow attack could be unintentional, but it strengthens the masking of the new implementation.
|
||||
|
||||
#v(2em)
|
||||
|
||||
As a conclusion, we observed that:
|
||||
- #SDK shadowing is performed by #shadowsdk of applications but are unintentional: these classes are embedded for retro-compatibility purpose or because the developer added a library already present in Android;
|
||||
- Hidden shadowing rarely occurs and is mainly due to the usage of libraries that Android already contains;
|
||||
- Malware perform more self-shadowing than goodware applications, and we found a sample where self-shadowing would clearly mislead the reverser.
|
||||
- #SDK shadowing is performed by #shadowsdk of applications, but is unintentional: these classes are embedded for retro-compatibility purposes or because the developer added a library already present in Android.
|
||||
- Hidden shadowing rarely occurs and is mainly due to the usage of libraries that Android already contains.
|
||||
- Malware performs more self-shadowing than goodware applications, and we found a sample where self-shadowing would clearly mislead the reverser.
|
||||
|
||||
|
|
|
@ -8,89 +8,88 @@
|
|||
|
||||
Countermeasures against shadow attacks depend on each tool and its objectives.
|
||||
The first important recommendation is to implement the class selection algorithm according to the algorithm described in Listing @lst:cl-loading-alg.
|
||||
It should solve any case of self-shadowing, except for tools like Apktool, which do not have to select a class for computing the result but show the whole application's content.
|
||||
It should solve any case of self-shadowing, except for tools like Apktool, which do not have to select a class for computing the result, but show the whole application's content.
|
||||
For those tools, a clear warning should be added, pointing out that multiple implementations have been found and displaying the one that will be used at runtime.
|
||||
|
||||
Countermeasures against #SDK shadow and Hidden shadow attacks are more complex to handle: it requires the list of platform classes on the target smartphone.
|
||||
The list of #SDK classes can be extracted easily from android.jar, but hidden classes need to be obtained by another means.
|
||||
They could be listed directly from the #AOSP tree of the Android source code, or obtained from Android documentation, or extracted from the phone itself.
|
||||
The first approach requires statically analyzing the source code, which can be difficult to achieve as several programming languages are used, and the code base is large andd fragmented.
|
||||
As discussed earlier in the chapter, the documentation can lack some classes.
|
||||
Consequently, the most reliable source is the smartphone itself.
|
||||
It should be noted that none of these methods can be generalized for all possible versions of Android, as the exact list will depend on the exact targeted device, possibly modified by the manufacturer.
|
||||
Thus, to conter Shadow attaks, the static analysis tools that we evaluated need to embed multiple lists of platform classes, one for each Android version.
|
||||
Countermeasures against #SDK shadow and Hidden shadow attacks are more complex to handle: they require the list of platform classes on the target smartphone and, in some cases, their implementation.
|
||||
The list of #SDK classes can be extracted easily from android.jar, but hidden classes need to be obtained by other means.
|
||||
They could be listed directly from the #AOSP tree of the Android source code, obtained from Android documentation, or extracted from the phone itself.
|
||||
The first approach requires statically analysing the source code, which can be difficult to achieve as several programming languages are used, and the code base is large and fragmented.
|
||||
Ideally, the documentation would be the best solution, but as discussed earlier in the chapter, it can lack some classes.
|
||||
For this solution to be viable, Google would need to keep the documentation closer to the released version of Android than it currently is.
|
||||
Also, smartphone manufacturers might add additional classes that would not appear in Google documentation.
|
||||
In fact, neither the documentation nor the source code approach can be generalised for all possible versions of Android, as the exact list will depend on the exact targeted device, possibly modified by the manufacturer.
|
||||
Thus, to counter Shadow attacks, the static analysis tools that we evaluated need to embed multiple lists of platform classes, one for each Android version.
|
||||
Then, the best heuristic would be to use the list of platform classes that is closest to the target #SDK of the analysed application.
|
||||
|
||||
Some tools like Flowdroid would require additional countermeasures: to compute the exact flow of data, Flowdroid also needs to analyse the code of platform classes.
|
||||
Some tools, like Flowdroid, would require additional countermeasures: to compute the exact flow of data, Flowdroid also needs to analyse the code of platform classes.
|
||||
For the #SDK classes, Flowdroid has already analysed them, but the hidden classes have not.
|
||||
In addition to the data flow in hidden classes, Flowdroid needs a list of data sources and sinks from those classes.
|
||||
Other analysis tools may require additional data from platform classes, which may be too difficult to obtain.
|
||||
|
||||
We believe that analysis tools can handle shadow attacks to some degree.
|
||||
The implementation of the solution will differ depending on the nature tool and may not always require the same implementation effort.
|
||||
The implementation of the solution will differ depending on the nature of the tool and may not always require the same implementation effort.
|
||||
|
||||
=== Relation with Obfuscation Techniques <sec:cl-cross-obf>
|
||||
|
||||
As described in the state of the art, reverse engineers face other techniques of obfuscation such as packers or native code.
|
||||
As described in the state of the art, reverse engineers face other techniques of obfuscation, such as packers or native code.
|
||||
These techniques rely on custom class loaders that load new parts of the application from ciphered assets or from the network.
|
||||
The reverse engineers have to study the application dynamically, to recover new classes, and eventually go back to a static phase to understand the behavior of the application.
|
||||
In this section, we compare shadow attacks with these techniques and we discuss how they interact with them.
|
||||
The reverse engineers have to study the application dynamically to recover new classes and eventually go back to a static phase to understand the behaviour of the application.
|
||||
In this section, we compare shadow attacks with these techniques and discuss how they interact with them.
|
||||
|
||||
Advanced obfuscation techniques relying on packers have a higher impact on the difficulty of performing a static analysis compared to shadow attacks.
|
||||
Most of the time, the reverse engineer cannot deobfuscate the application without performing a dynamic analysis.
|
||||
For this reasons, approaches have been designed to assist the capture of the bytecode that is loaded dynamically, after the precise time where the deobfuscation methods have been executed~@zhang2015dexhunter @xue2017adaptive @wong2018tackling.
|
||||
For these reasons, approaches have been designed to assist the capture of the bytecode that is loaded dynamically, after the precise time where the deobfuscation methods have been executed~@zhang2015dexhunter @xue2017adaptive @wong2018tackling.
|
||||
On the contrary, a shadow attack can be easily defeated by implementing our algorithm in the static analysis tool, as discussed earlier in @sec:cl-countermeasures.
|
||||
Nevertheless, shadow attacks are stealthier than packers or native code.
|
||||
Packers can be easily spotted by artifacts left behind in the application or by detecting classes implementing a custom class loading mechanism.
|
||||
On the contrary, an extra class implementing a shadow attack, that would not be executed, could contain voluntarily few code, compared to the executed class of Android.
|
||||
Such attack would be more discrete than a packer that adds in the application a lot of possibly native code
|
||||
Packers can be easily spotted by artefacts left behind in the application or by detecting classes implementing a custom class loading mechanism.
|
||||
On the contrary, an extra class implementing a shadow attack, that would not be executed, could contain voluntarily little code, compared to the executed class of Android.
|
||||
Such an attack would be more discreet than a packer that adds in the application a lot of possibly native code.
|
||||
|
||||
Combining regular obfuscation techniques with shadow attacks can be achieved in two ways.
|
||||
|
||||
First, the attacker could hide the code of a packer or a native call by using a shadow attack.
|
||||
For example, by colliding a class of the #SDK, a control flow analysis could be wrongly computed, leading to consider that part of the code is dead, which would mislead the reverse engineer about the use of this part that contains a packer.
|
||||
For example, by colliding a class of the #SDK, a control flow analysis could be wrongly computed, leading to considering that part of the code to be dead, which would mislead the reverse engineer about the use of this part that contains a packer.
|
||||
At runtime, this code would be triggered, unpacking new code.
|
||||
|
||||
Second, the attacker could use a packer to unpack code at runtime in a first phase.
|
||||
The reverse engineer would have to perform a dynamic analysis, for example uising a tool such as Dexhunter~@zhang2015dexhunter, to recover new #DEX files that are loaded by a custom class loader.
|
||||
The reverse engineer would have to perform a dynamic analysis, for example, using a tool such as Dexhunter~@zhang2015dexhunter, to recover new #DEX files that are loaded by a custom class loader.
|
||||
Then, the reverse engineer would go back to a new static analysis and could have the problem of solving shadow attacks, for example, if a class is defined multiple times in the loaded #DEX files.
|
||||
|
||||
Because the interaction between shadow attacks and other obfuscations techniques often rely on a loading mechanism implemented by the developer, investigating these cases require to analyse the Java bytecode that is handling the loading.
|
||||
Because the interactions between shadow attacks and other obfuscation techniques often rely on a loading mechanism implemented by the developer, investigating these cases requires analysing the Java bytecode that is handling the loading.
|
||||
This problem is left as future work.
|
||||
|
||||
=== Limitations <sec:cl-ttv>
|
||||
|
||||
During the analysis of the #ART internals, we made the hypothesis that its different operating modes are equivalent: we analysed the loading process for classes stored as non-optimized `.dex` format, and not for the pre-compiled `.oat`.
|
||||
During the analysis of the #ART internals, we made the hypothesis that its different operating modes are equivalent: we analysed the loading process for classes stored as non-optimised `.dex` format, and not for the pre-compiled `.oat`.
|
||||
It is a reasonable hypothesis to suppose that the two implementations have been produced from the same algorithm using two compilation workflows.
|
||||
Similarly, we assumed that the platform classes stored in `boot.art` are the same as the ones in `BOOTCLASSPATH`.
|
||||
We confirm empirically our hypothesis on an Android Emulator, but we may have missed some edge cases.
|
||||
|
||||
The comparison of Smali code can lead to underestimated values, for example, if the compilation process performs minor modifications such as instruction reordering.
|
||||
The ratios reported in this study for the comparison of code are thus a lower bound and would be higher with a more precise comparison.
|
||||
In addition, platform classes are stored differently in older versions of Android and could not be easily retrieved.
|
||||
For this reason, we did not compared the classes found in applications to their versions older than #SDK 32 to avoid producing unreliable statistics for those versions.
|
||||
In addition, platform classes are stored differently in older versions of Android and cannot be easily retrieved.
|
||||
For this reason, we did not compare the classes found in applications to their versions older than #SDK 32 to avoid producing unreliable statistics for those versions.
|
||||
|
||||
|
||||
=== Futur Works <sec:cl-futur>
|
||||
|
||||
#todo[Develop @sec:cl-futur]
|
||||
As we just said, our Smali-based comparison of class implementation is quite naive and could use more work.
|
||||
It could be insightful to be able to detect exactly when two classes are from the same source file, or which version of a library a class belong to.
|
||||
More importantly, a better comparison technique would allow us to detect cases where the shadowed library has actual malicious bytecode added that we could have missed manually.
|
||||
|
||||
As we said, our comparison technique is quite naive and could use more work.
|
||||
It could be insightful to be able to detect excatlly when two classes are from the same fource file, or which version of a library a class belong.
|
||||
More importantly, a better comparision technique would allow to detect cases where the shadowed library has actual malicious bytecode added that we could have missed manually.
|
||||
Additionally, the question of dynamic class loaders, used manually by the application developer, is interesting.
|
||||
This is reaching the limits of static analysis; those cases involve dynamically loading bytecode, and in many cases, the classes loaded by those class loaders are not even available for analysis.
|
||||
However, even with dynamic analysis, the behaviour of class loaders can still be an issue, especially when the analysis is performed by alternating static and dynamic analysis, as is often the case when manually reversing an application.
|
||||
To handle those cases, it could be interesting to develop a method to model any arbitrary class loader, either by analysing its bytecode or by interacting with an instance of the class loader dynamically.
|
||||
|
||||
Additionally, the question of dynamic class loaders, used manually by the application developer is interesting.
|
||||
This is reaching the limits of static analysis, thoses cases involve dynamically loading bytecode, and in many cases the classes loaded by those classe loaders are not even available for analysis.
|
||||
However, even with dynamic analysis, the behavior of class loaders can still be an issue, especially when the analysis is performed by alternating static and dynamic analysis, as it is often the case in when manually reversing an application.
|
||||
To handle those cases, it could be interesting to develop a method to model any arbitrary class loaders, either by analysing its bytecode or by interacting with an instance of the class loader dynamically.
|
||||
|
||||
In september 2024 (just after we finished this work), Android 15 introduce support for the new version 41 of the #DEX format.
|
||||
In September 2024 (just after we finished this work), Android 15 introduced support for the new version 41 of the #DEX format.
|
||||
We can expect this version of #DEX to become the norm in a few years.
|
||||
The most notable change in version 41 is the new container format: instead of storing the bytecode in separated #DEX files, the different files can now be concatenated into one unique file.
|
||||
There is also some permeability between the concatenated files: some structures stored in one file can be used by the nexts concatenated files.
|
||||
The most notable change in version 41 is the new container format: instead of storing the bytecode in separate #DEX files, the different files can now be concatenated into one unique file.
|
||||
There is also some permeability between the concatenated files: some structures stored in one file can be used by the next concatenated files.
|
||||
This significant change in the bytecode storage is similar to the introduction of the multi-dex format.
|
||||
Considering that self shadowing is only possible because of the multi-dex format, be expect this change to have the potential to introduce new similar issues.
|
||||
Thus, we believe that the implementation details of this new version need to be studied and model properly to avoid introducing new issues when updating analysis tools to support it.
|
||||
Just by reading the specification#footnote[https://source.android.com/docs/core/runtime/dex-format#container], we believe that self shadowing between concatenated #DEX files is possible, unless additionnal checks are enforced by the #ART when loading the file.
|
||||
Considering that self-shadowing is only possible because of the multi-dex format, we can expect this change to have the potential to introduce new, similar issues.
|
||||
Thus, we believe that the implementation details of this new version should be studied and modelled properly to avoid introducing new issues when updating analysis tools to support it.
|
||||
Just by reading the specification#footnote[https://source.android.com/docs/core/runtime/dex-format#container], we believe that self-shadowing between concatenated #DEX files is possible, unless additional checks are enforced by the #ART when loading the file.
|
||||
|
||||
#jm-note[Maybe talk about v41 in RASTA? this will break a lot of things]
|
||||
|
|
|
@ -6,13 +6,13 @@
|
|||
#todo[Ca serait bien de faire un PR ou deux a Jadx/Androguard/Soot quand même]
|
||||
|
||||
This chapter has presented three shadow attacks that allow malware developers to fool static analysis tools when reversing an Android application.
|
||||
By including multiple classes with the same name or by using the same name as a class of the #Asdk, the developer can mislead a reverse engineer or impact the result of a flow analysis, such as the ones of Androguard or Flowdroid.
|
||||
By including multiple classes with the same name or by using the same name as a class of the #Asdk, the developer can mislead a reverse engineer or impact the result of a flow analysis, such as those of Androguard or Flowdroid.
|
||||
|
||||
We explored if such shadow attacks are present in as dataset of #nbapk applications .
|
||||
We explored whether such shadow attacks are present in a dataset of #nbapk applications.
|
||||
We found that on average, #shadowsdk of applications are shadowing the #SDK, mainly for retro-compatibility purposes and library embedding.
|
||||
More suspiciously, #shadowhidden of applications are shadowing a hidden class, which could lead to unexpected execution as these classes can appear/disappear with the evolution of Android internals.
|
||||
Investigations for applications that defined classes multiple times suggest that the compilation process or the inclusion of different versions of the same library is the main explanation.
|
||||
Finally, when investigating malware samples, we found a specific sample containing a shadow attack that would hide a part of the critical code from a reverser studying the application.
|
||||
Finally, when investigating malware samples, we found a specific sample containing a shadow attack that would hide a part of the critical code from a reverse engineer studying the application.
|
||||
|
||||
#v(2em)
|
||||
|
||||
|
@ -21,6 +21,6 @@ Finally, when investigating malware samples, we found a specific sample containi
|
|||
#v(0.75em)
|
||||
@lst:cl-loading-alg model the class loading algorithm: platform classes have priority over classes stored in `classes.dex` which have priority over `classes<n>.dex` (where $n in [| 2, +infinity [| $ and $forall i in [| 2, n [|, exists $ `classes<i>.dex`) which has priority over `classes<n+1>.dex`.
|
||||
|
||||
Failing to implement this model (#ie by ignoring some platform classes or by sorting the `classes<n>.dex` alphabetically instead of numerically) can cause static analysis tools to compute an incorrect representation of the analyzed application.
|
||||
Failing to implement this model (#ie by ignoring some platform classes or by sorting the `classes<n>.dex` alphabetically instead of numerically) can cause static analysis tools to compute an incorrect representation of the analysed application.
|
||||
])))
|
||||
|
||||
|
|
|
@ -10,8 +10,8 @@
|
|||
In particular, if the developer adds a class whose name collides with the name of a class of the Android operating system or another class in the application, they may confuse a reverse engineer in charge of studying such an application.
|
||||
In this chapter, we explore the consequences of those collisions.
|
||||
We highlight three attacks that we call shadow attacks because the class implementation that a reverser would find shadows a second implementation with a higher priority.
|
||||
In particular, we show that a static analysis tools used by a reverser choose the shadow implementation for most of the evaluated tools, and outputs a wrong result.
|
||||
In a dataset of #nbapk applications, we also investigate whether shadow attacks are used in the wild and show that, most of the time, there is no malicious behavior behind them.
|
||||
In particular, we show that static analysis tools used by a reverser choose the shadow implementation for most of the evaluated tools, and output a wrong result.
|
||||
In a dataset of #nbapk applications, we also investigate whether shadow attacks are used in the wild and show that, most of the time, there is no malicious behaviour behind them.
|
||||
])))
|
||||
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue