Implementation of an Android Framework for USB storage access without root rights

Contents

Acknowledgements

Abstract

Outline of the Thesis

Acronyms

I. Introduction and Theory
1. Introduction
2. Basics about USB
2.1. Client device hierarchy
2.2. Endpoints
2.3. USB On the go
3. USB on Android
3.1. USB Host API
3.2. Enumerating devices
3.3. Requesting permission for communication
3.4. Communication
3.5. Tearing down the communication
3.6. Listening to attach and detach events
3.6.1. Via BroadcastReceiver
3.6.2. Via AndroidManifest.xml
4. USB Mass storage class
4.1. Bulk-only Transport
4.2. SCSI transparent command set
4.3. SCSI commands
4.3.1. INQUIRY
4.3.2. TEST UNIT READY
4.3.3. READ CAPACITY
4.3.4. READ(10) and WRITE(10)
4.3.5. REQUEST SENSE
5. File systems
5.1. General
5.1.1. Examples
5.1.2. Partition table
5.2. The FAT32 file system
5.2.1. General layout
5.2.2. Boot Sector and FS Information Structure
5.2.3. File Allocation Table
5.2.4. Directories and files
5.2.5. Subdirectories
5.2.6. Long File Name entries

II. Implementation
6. Purpose and Overview
6.1. Using the Framework
7. Inside the packages
7.1. The driver package
7.2. The partition package
7.3. The file system package

III. Quality Management
8. Testing
8.1. Overview
8.1.1. Testing criteria
8.2. Results
8.2.1. Native support
8.2.2. Performance test . .

IV. Results
9. Summary
9.1. Current status
9.2. Conclusion
10. Outlook
Appendix
A. Debugging applications via Wifi
B. Isochronous USB transfers
C. API difference
Bibliography

Acknowledgments

Many acknowledgment goes to the operating system chair of the TUM, especially to Prof. Dr. Uwe Baumgarten and Nils Kannengießer for the ability to write this thesis and especially to Nils for guiding me. The operating system chair also borrowed me a lot of Android devices to test my work!

I also want to thank Jan Axelson for his book USB Mass Storage[2]. Without this book the thesis would have been a lot more cumbersome. It was an important resource throughout the whole work.

All people who read, corrected and gave me hints for improvement on this thesis in advance also deserve credit. I want to thank all of them at this place!

Abstract

This bachelor thesis describes the implementation of an Android framework to access mass storage devices over the USB interface of a smartphone. First the basics of USB (i.e. interfaces, endpoints and USB On the go) and accessing USB devices via the official Android API are discussed. Next the USB mass storage class is explained, which was designed by the USB-IF to access mobile mass storage like USB pen drives or external HDDs. For communication with mass storage devices, most important are the bulk-only transfer and the SCSI transparent command set. Furthermore file systems, for accessing directories and files, are described. This thesis focuses on the FAT32 file system from Microsoft, because it is the most commonly used file system on such devices.

After the theory part it is time to look at the implementation of the framework. In this section, the first concern is the purpose in general. Then the architecture of the framework and the actual implementation are presented. Important parts are discussed in detail.

The thesis finishes with an overview of the test results on various Android devices, a short conclusion and an outlook to future developments. Moreover the current status of the developed framework is visualized.

Outline of the Thesis

Part I: Introduction and Theory

CHAPTER 1: INTRODUCTION

Overview of the thesis and its purpose.

CHAPTER 2: BASICS ABOUT USB

Basics of USB, how it is structured and how it generally works.

CHAPTER 3: USB ON ANDROID

Introduction to the USB API on Android. How can USB devices be enumerated and accessed, and how can the communication be initiated?

CHAPTER 4: USB MASS STORAGE CLASS

Overview of the USB mass storage class. The main goal is to understand SCSI commands to communicate with the device and read and write from and to its storage.

CHAPTER 5: FILE SYSTEMS

File systems in general and a detailed description of how FAT32 from Microsoft works.

Part II: Implementation

CHAPTER 6: PURPOSE AND OVERVIEW

Purpose and requirements of the implementation and a short overview of the framework’s architecture.

CHAPTER 7: INSIDE THE PACKAGES

Deeper insight into the developed framework, it’s packages, classes and interaction between the important parts.

Part III: Quality Management

CHAPTER 8: TESTING

Results of different tests regarding the developed framework on various devices.

Part IV: Results

CHAPTER 9: SUMMARY

Short summary of the current status of the framework and a conclusion of the thesis.

CHAPTER 10: OUTLOOK

Some hints about further development and advices what future may hold.

Acronyms

illustration not visible in this excerpt

Part I. Introduction and Theory

1. Introduction

Since Android 3.1, which was originally designed for tablet computers, a lot of Android devices come with USB host support (USB On the go). That means a normal Android tablet or phone can not only act as an USB client when connected to a computer. It can also act as an USB host for peripherals by powering the bus with the needed five Volt and changing into USB host mode which enables enumeration of connected USB devices[8]. Android currently supports interrupt, bulk and control transfers¹. That means almost every USB device can, theoretically, be used with an Android device². The Android host API allows to communicate with connected USB devices, i.e. a high level USB driver can be written in Java.

Whereby the idea of connecting a USB mass storage device like USB flash drives or external HDDs is not far-fetched. Especially when looking at recent occurrences where a lot of devices lack a slot for SD-Cards and only offer a solid, mostly small, internal storage. Unfortunately the stock Android comes without support for USB storage devices. That means when connecting mass storage devices to an Android phone, nothing happens. The data cannot be accessed via a file manager or something similar. On rooted devices this is possible because the alternative Android ROMs often provide support for it. But with the Android USB Host API it should also be possible to access such devices without rooting the device and flashing an alternative ROM. The only thing needed is to implement the low level USB communication via e.g. bulk transfers and the abstraction of directories and files via a file system.

Currently there are two applications in the Google Play Store which allow accessing mass storage devices without root rights! First there is a plugin for the Total Commander called USB-Stick Plugin-TC. The plugin extends the Total Commander application by USB mass storage access. It currently supports FAT12, FAT16, FAT32, exFAT and NTFS (read only). There is a free trial version available. The second application is called Nexus Media Importer. It supports FAT16, FAT32 and NTFS (also read only). There is no free trial version available. In general both apps support USB sticks, external HDDs and card readers.

The disadvantage of both applications is that there is no solution to access the mass storage from other apps. That means all accessed data has to be cached and copied to the internal storage before any other app can use it. Unfortunately it seems that these limitations cannot be removed.

Both applications are proprietary and the source code is not available for reference or modification. This is why an open source Android framework for accessing mass storage devices is developed in this bachelor thesis. The desired license is the very liberal Apache License, Version 2.0, the same license Android is licensed under.

Due to the same licensing it would be possible for Google to integrate this solution into the official Android. But there are some factors which make the integration unlikely. First of all, necessary things, like file systems (e.g. FAT32) or the SCSI transparent command set, for mounting USB mass storage are already implemented in the underlying Linux ker- nel. Google just deactivated the support for it. Second, with the solution described in this thesis, only apps which use the framework can access USB storage devices. It would be more straightforward if the connected devices would be mounted in the normal unix file system like SD-cards. For example under /mnt/usbstick0. This would allow other apps to easily access data from USB mass storage without extra changes to the application. There- fore it is very unlikely that Google will integrate this framework into the official Android. If Google decides to support mounting USB mass storage devices, they will most likely enable support for it in the kernel and mount the devices in the normal file system, like some manufacturers (e.g. Samsung) already do.

Another reason for Google not to implement the support for mass storage devices over USB, is to move more people to use their own cloud service Google Drive. But maybe, if Google notices the growing popularity of applications allowing access of USB mass storage, Google may enable support for it in their mobile operating system.

Numbers in this thesis

There are a lot of numbers in this thesis. If the number has a trailing ’h’, for example 08h, this number shall be interpreted as a hex number. Numbers without this ’h’ shall be interpreted as decimal numbers.

2. Basics about USB

USB stands for Universal Serial Bus and is a standard for a serial bus system, for connecting multiple peripheral devices to a personal computer. The first version was introduced by Intel in 1996. Today the specification is done by the USB Implementers Forum (USB-IF). The USB-IF is a corporation founded by various companies which work non-profit on the USB specification. In USB communication there are two kinds of devices, one USB host controller (e.g. computer) and one or more clients (slaves). The host is responsible for the communication process. The client only sends data when the host asks for it. The USB host is responsible for powering the connected client, thus an external power source is only necessary in some special cases[20].

2.1. Client device hierarchy

A USB client is structured into four different USB descriptors:

- Device descriptor
- Configuration descriptor
- Interface descriptor
- Endpoint descriptor

The device descriptor represents the USB device as a whole device which is connected to the USB bus. This can for example be a loud speaker with volume control buttons. The configuration descriptor represents the current state of the USB device. This can for example be standby or active.

A USB interface descriptor describes every logical device which belongs to the USB device. Often USB devices consist of multiple logical device units. For example a loud speaker could consist of the speakers as an audio device and buttons to control the volume as a human interface device.

Lastly there are endpoint descriptors which represent unidirectional communication pipes. This is where the actual communication happens. Endpoints can either be of type IN (device to host) or OUT (host to device). Additionally there are four different types of endpoints, to fit the different requirements of communication[11].

2.2. Endpoints

Every USB device has different requirements on the underlying communication pipe. To satisfy the different needs the USB protocol offers four different types of communication (endpoints).

Control endpoints are used to configure the device and retrieve status information. Con- trol transfers are typically very small. Every device has a control endpoint called endpoint 0 which plays an important role at insertion time[11].

Interrupt transfers carry a small amount of data to the host every time the host asks for it. This happens at a fixed rate resulting in a fixed and guaranteed bandwidth. These transfers are used by human interface devices (e.g. mouse, keyboard, gamepads) which need a low latency and a low packet size.

Next there are bulk endpoints. These are useful when the amount of data transferred varies often and happens infrequently. The remaining available bandwidth the bus offers is used. Hence, there is no guarantee on bandwidth or latency. However bulk transfers offer consistent data transfers, meaning that no data is lost. They are typically used for printers, network or mass storage devices. Everywhere data loss is unacceptable and no guaranteed bandwidth is needed.

Finally there are the isochronous transfers. They offer a guaranteed bandwidth while resigning consistency. The guaranteed bandwidth is mostly as fast as possible and valuable for real time transfers (e.g. audio or video). Mostly these transfers are used for webcams or audio devices (e.g. external audio cards/audio interfaces)[11].

2.3. USB On the go

As already mentioned, in USB communication there is always a host (master) and a client (device). The host initiates the communication and acts as a master. This means that the client can only send data after being explicitly asked to do so by the host. The client is only able to signal that it requires attention. Then the host must react and ask for receiving data. When connecting a smartphone or tablet to the computer, the computer acts as the host and the smartphone acts as the client device. That means the smartphone normally acts as a client device and not as the USB host. In the desired constellation described in this thesis, however, it has obviously to act as a host.

For that reason the USB-IF developed the USB On the go (USB OTG) feature in 2001 as part of the USB 2.0 specification[20]. This feature allows a USB device to act as a client or either as a host depending on the present situation. To use the USB OTG mode, on a smartphone, a special USB OTG adapter is needed. This is necessary for two reasons. First for signaling that the smartphone should act as a host and not as usual as a client and second because most smartphones and tablets do not provide a normal USB port of type A. Instead they offer a mini (older devices) or micro port of type Mini-A or Micro-B[21].

3. USB on Android

As already mentioned, Google added USB features to the Android OS with Android 3.1 Honeycomb. There are two different modes Android keeps under control. The already mentioned host support and a special USB accessory mode. The accessory mode is only available on Android. It is supposed for developing USB host hardware specifically designed for Android devices, where the accessory is the USB host and powers the Android device[7]. The Android device is the client and can for example charge through and interact with the hardware (e.g. playing music through external speakers).

The USB Accessory mode is also backported to Android 2.3.4[7]. The developed framework solely relies on the USB host functionality since a memory stick is a USB client.

3.1. USB Host API

Android offers classes to enumerate, connect to and communicate with connected USB devices. Table 3.1 gives an overview of the classes which can be found in the package android.hardware.usb.

Table 3.1.: USB host APIs, compare to [8]

illustration not visible in this excerpt

The UsbRequest class is only needed when communicating asynchronously¹. The rough procedure of getting in touch with a USB device includes following steps:

1. Retrieve the desired UsbDevice via the UsbManager
2. Get the appropriate UsbInterface and UsbEndpoint
3. Begin the communication by opening a UsbDeviceConnection via the UsbEndpoint

To understand the following sections, fundamental knowledge of Android program- ming is recommended. Basics² are not described in detail here. An introduction to An- droid programming can be found in the official Android developer documentation³.

3.2. Enumerating devices

To enumerate through all connected USB devices the singleton UsbManager is used. It allows looping through the device list. The device list is returned by the method getDeviceList() of the UsbManager.

Listing 3.1: Enumerating connected USB devices

illustration not visible in this excerpt

Accessing the UsbInterface and the UsbEndpoint is also straightforward. The UsbDevice has a method to get the desired interfaces and the UsbInterface has a method to get the desired endpoints on the other hand. Listing 3.2 illustrates the process.

Listing 3.2: Accessing UsbInterface and UsbEndpoint

illustration not visible in this excerpt

While looping through the devices, it can easily be checked if any device fits the desired needs. UsbDevice and UsbInterface offer methods to get the class, subclass and the protocol of the device, resp. the interface. The UsbEndpoint class has methods to get the type, the direction and other attributes of the corresponding endpoint. The UsbDevice also offers methods to get the vendor and product ID of the connected device.

3.3. Requesting permission for communication

After discovering a suitable USB device Android requires the user to accept the communication between an application and the USB device first. To do so the permission to communicate with the USB device has to explicitly be requested. A dialog is shown to the user asking for permission, where the user can click okay or cancel. Therefore the UsbManager offers a method called requestPermission which takes a UsbDevice and a PendingIntent as parameter. The PendingIntent in this case is a Broadcast which can be received via registering a BroadcastReceiver with a specific IntentFilter.

Listing 3.3 shows how a BroadcastReceiver, for receiving notifications about the per- mission, can look like. First the intent action is validated, this step is only needed if the BroadcastReceiver receives multiple different actions. In this example this is not the case. After that the UsbDevice can be accessed via an extra of the intent. Another extra of the intent is the permission state. If it is granted the extra is true and it is permitted to com- municate with the device.

Listing 3.3: Permission BroadcastReceiver [8]

illustration not visible in this excerpt

The next step is to register this BroadcastReceiver that it can actually receive broadcasts from the system. This normally happens in the onCreate method of an Activity via the method registerReceiver which takes the BroadcastReceiver (mUsbReceiver) and the IntentFilter as parameter. The IntentFilter uses the ACTION USB PERMISSION string, declared in Listing 3.3, to filter undesired actions.

Listing 3.4: Registering the BroadcastReceiver [8]

illustration not visible in this excerpt

The last step consists of requesting the permission using the UsbManager:

Listing 3.5: Requesting permission [8]

illustration not visible in this excerpt

3.4. Communication

After succeeding all necessary steps to set up the device, communication is possible. To do so the desired device has to be opened and an UsbDeviceConnection has to be retrieved. The class UsbDeviceConnection offers a method to claim a certain UsbInterface. After that communication is possible via the UsbDeviceConnection. It offers methods like bulkTrans- fer or controlTransfer.

Listing 3.6: Communicating with a connected device [8]

illustration not visible in this excerpt

Listing 3.6 uses, for simplicity reasons, the first interface and endpoint. Normally the endpoint to communicate with, should be chosen wisely by examining for example the interface class, or the vendor ID of the device. A control transfer would look similar.

3.5. Tearing down the communication

When the communication between the Android application and the USB device is done, it has to be shut down. This is done by releasing the interface and closing the connection. Listing 3.7 gives an example about how to do that.

Listing 3.7: Closing communication

illustration not visible in this excerpt

3.6. Listening to attach and detach events

Android does not only allow enumerating connected devices, an application can also register for attach and detach events of USB devices. The application then gets notified whenever a USB device is connected to, or disconnected from the Android device. There are two different ways to do that. The first one is via a BroadcastReceiver, the second one via the AndroidManifest.xml file. The second method has the advantage that the application is notified even if it has not been started before.

3.6.1. Via BroadcastReceiver

Listing 3.8: Attach and detach notification of USB devices via a BroadcastReceiver

illustration not visible in this excerpt

To use this BroadcastReceiver it has to be registered in an Activity or Service with the corresponding IntentFilter like this:

Listing 3.9: Registering the BroadcastReceiver with the desired actions

illustration not visible in this excerpt

3.6.2. Via AndroidManifest.xml

If an application wants to be notified about the attachment of an USB device this can also be specified in the AndroidManifest.xml. This has the advantage that the application does not have to be started before. In fact it is started when a desired USB device is connected. The user is then asked if he wants to start the application which can handle the attached device. The next benefit is that the step of requesting permission, described in 3.3, is not required because the user already gave his consent by allowing the application to start.

Additionally a device filter can be specified, which allows the application to be notified only if an appropriate device is attached. Following attributes can be specified[8]:

- Vendor ID
- Product ID
- Class
- Subclass
- Protocol (device or interface)

Below is an example how a device filter could look like:

Listing 3.10: Example device filter [8]

illustration not visible in this excerpt

This resource file should be located at res/xml/device filter.xml in the project directory[8]. The device filter can then be used in the AndroidManifest.xml, like in Listing 3.11.

Listing 3.11: AndroidManifest.xml [8]

illustration not visible in this excerpt

The intent filter for the action uses, again, the USB DEVICE ATTACHED string, like when using a BroadcastReceiver. This time no broadcast is sent, but an activity is started. The manifest also contains a uses-feature entry, because not all Android devices guarantee to support the USB host feature[8]. The minimum sdk version is set to 12 here, because on lower API levels the USB host API is not available.

After that the UsbDevice can be accessed anywhere within the Activity like this:

Listing 3.12: Accessing the UsbDevice in the Activity, compare to: [8]

illustration not visible in this excerpt

[...]

---

¹ Isochronous transfers are currently unsupported[9].

² Webcams or audio devices mostly use isochronous transfers and can thus not be used at the moment.

¹ Asynchronous communication is passed in the implementation of the framework.

² Following things, for example, are seen as basic: Activity, Intent, PendingIntent, IntentFiler, Broadcast, BroadcastReceiver.

³ http://developer.android.com