Application Casting

Application Casting

Sammaiah Kyatham

ABSTRACT

This thesis describes the research and development of Application casting, a low bandwidth application sharing system.

A software system for all computer users right from students to a normal user which lets users show and give control of a part of their desktop area to some other person over a network using extremely low bandwidth would be of great help for computer users. In the current scenario where every platform is having its own standards, such a system should work on any platform with very less dependencies. Such a system would be of great help for online teaching, remotely accessing a machine, remote trouble shooting a machine, remotely demonstrating an application etc.,

Approach for sharing application:

Initial part of the research was to grab one user’s desktop and share it with another user. This can be done in several ways like grabbing the stream that was sent to user’s monitor. But this involves native coding, which makes an application dependent on the operating system it is using. Further analysis of this problem resulted in capturing a user’s desktop as an image and sending it over a network.

Design of Application Casting has been laid out by taking bandwidth into consideration. The goal of the design was to make it work over very low bandwidth. There are two components for Application Casting, one at the presenter end and the other at the viewer end. In this document the component at the presenter end is called broadcaster and the component at the viewer end is called client.

Protocol:

Application Casting follows RTP packet structure based on RFC 1889. The data packet senders report and receivers report are constructed according to the RFC. The RTP data packet contains a protocol hidden with in the data in the RTP data packet. Based in this content an RTP packet will be a control packet for application casting or a data packet for application casting. The RTP, A real time protocol which provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services.

The Main aim of this thesis is implementation of low bandwidth application casting system.

1. INTRODUCTION

A tool for all computer users’ right from students to a normal user which lets users show and give control of a part of their desktop area to some other person over a network using extremely low bandwidth would be of great help for computer users. In the current scenario where every platform is having its own standards, such a tool should work on any platform with very less dependencies. Such a tool would be of great help for online teaching, remotely accessing a machine, remote trouble shooting a machine, remotely demonstrating an application etc.,

A solution to this problem is a tool in Java. This tool enables users to choose the part on their screen for sharing. This tool does not give complete control over machine but gives control over a selective portion of the screen. Some features this tool accomplishes are:

i.  Platform independent.

ii.  Displays an interface to the GPD for approving or declining the student’s request.

iii.  Allow the students to drop the courses requested for advising if they change their mind even if they have been approved of their previous application.

Primary reason for developing this tool is to enable sharing of applications over a chat system. The system should utilize lowest possible bandwidth giving better clarity and should be user friendly.

Application Casting follows RTP packet structure based on RFC 1889. The data packet senders report and receivers report are constructed according to the RFC. The RTP data packet contains a protocol hidden with in the data in the RTP data packet. Based in this content an RTP packet will be a control packet for application casting or a data packet for application casting. The RTP, A real time protocol which provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services.

The whole application is based on java and it does not allow accessing resources of another program. For the viewer to see what the presenter is pointing to, it was necessary that the client application to know the mouse

Coordinates of presenter. For accomplishing this native c++ code was written using windows API to get the actual location of the mouse.

Broadcaster application uses jpg and png compression and we needed to send tile information in a best-compressed way. Tests showed results that some tiles are best compressed with jpg and some using png. There was requirement where we need to choose a format that is best suitable for a particular tile. Logic was developed for broadcaster to use a format that is best suitable for a tile.

PNG is an extensible file format for the loss less, portable, well-compressed storage of raster images. PNG provides a patent-free replacement for GIF and can also replace many common uses of TIFF. Indexed-colour, greyscale, and truecolor images are supported, plus an optional alpha channel for transparency. Sample depths range from 1 to 16 bits per component (up to 48bit images for RGB, or 64bit for RGBA).

3. PROPOSED WORK

3.1.  APPROACH FOR SHARING APPLICATIONS

Initial part of the research was to grab one user’s desktop and share it with another user. This can be done in several ways like grabbing the stream that was sent to user’s monitor. But this involves native coding, which makes an application dependent on the operating system it is using. Further analysis of this problem resulted in capturing a user’s desktop as an image and sending it over a network. This could be done completely in java with out any dependencies on operating system.

Java API provides specific classes to capture screen, one of the class java.awt.Robot helps to capture the screen. Robot class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed. The primary purpose of Robot is to facilitate automated testing of Java platform implementations.

Using the class to generate input events differs from posting events to the AWT event queue or AWT components in that the events are generated in the platform′s native input queue. For example, Robot.mouseMove will actually move the mouse cursor instead of just generating mouse move events.

Note that some platforms require special privileges or extensions to access low-level input control. If the current platform configuration does not allow input control, an AWTException will be thrown when trying to construct Robot objects. For example, X-Window systems will throw the exception if the XTEST 2.2 standard extension is not supported (or not enabled) by the X server.

Constructors:

Robot()
 Constructs a Robot object in the coordinate system of the primary screen.

Robot(GraphicsDevice screen)
 Creates a Robot for the given screen device.

Methods:

3.2.  APPROACH FOR TRANSFERRING ONLY CHANGES

Next concern was regarding the bandwidth usage. Transferring a user’s desktop image for every 500 milliseconds involves a lot of bandwidth. An analysis of some of the applications that a user generally shares like word documents, presentations, browsers etc., showed that only parts of it will be changed very frequently and other parts of the users screen will remain. This led to the conclusion that the image of the users sharing area has to be split into tiles as shown in the figure below and only the changed tiles have to be transmitted.

Fig. Changed Tiles of the Image.

If above two share areas are considered there is a difference only in some tiles. For transmitting the changed information with a normal approach we are transmitting many times the actual information that is changed.

For this reason whole image is broken into small tiles and then each tile is compared with old tile at corresponding location. This way bandwidth consumption is brought down considerably

3.3.  APPROACH FOR REDUCING COMPUTATIONS

Each new tile has to be compared with the existing tile for any changes. This can be done by taking the colour at each pixel and comparing it with an earlier copy. This involved checking each and every pixel, which results in lot of computations. In order to reduce the number of computations for calculating if a particular tile has changed a grid was laid out over the tile and only the points on the grid for checked for changes. This grid is shown in the figure below. The lines in the grid were only a few pixels a part say 3 or 5 pixels. The points of intersection of vertical and horizontal lines are checked for changes.

Grid layout on a tile

Fig. Grid layout on a tile

3.4.  APPROACH FOR IMAGE COMPRESSION

Once a changed tile is recognized it is not yet compressed. Compression results in smaller packet sizes, hence lesser bandwidth. Compression is necessary for such applications where images are transferred over a network very frequently. For this purpose PNG compression was used. PNG is an open source image format and can be used freely. A few experimentations with PNG images showed that PNG compression was resulting in a larger file size when compared to jpg for colourful images. Good compression for PNG can be achieved by using less number of colours. For this reason quantization is applied on an image before compressing. Quantization is a process reducing the number of colours in an image. Pixel colours are adjusted with the colours of the adjacent pixels to use less number of colours in the image.

The JPEG lossy image compression standard is currently in worldwide use, and is becoming a critical element in the storage of digital images captured with the optical microscope. This interactive tutorial explores compression of digital images with the JPEG algorithm, and how the lossy storage mechanism affects file size and the final image appearance.

Java API provides classes and methods to convert one type of image to another type of image, sample code to convert gif image to png image is,

The Image I/O libraries added to Java 1.4

File inputFile = new File(filename);

BufferedImage input = ImageIO.read(inputFile);

File outputFile = new File(outfilename);

ImageIO.write(input, "PNG", outputFile);

To get a list of available format one can read, use ImageIO.getReaderFormatNames(). (gif, jpg, png)

To get the list of available writable formats, use  

getWriterFormatNames() (jpg, png).

Javax. Imageio.ImageIO  class containing static convenience methods for locating ImageReaders and ImageWriters, and performing simple encoding and decoding.

3.5.  APPROACH FOR RTP PROTOCOL MPLEMENTATION

Application Casting follows RTP packet structure based on RFC 1889. The data packet senders report and receivers report are constructed according to the RFC.

The RTP data packet contains a protocol hidden with in the data in the RTP data packet. Based in this content an RTP packet will be a control packet for application casting or a data packet for application casting.

Here an octet is an 8-bit byte.

1 to 12 octets à Standard RTP header described in RFC 1889.

Packet contains a custom header designed for Application Casting from 13^th octet. This custom header is different for data packet and for control packet. The first value of the custom header describes if it is a control packet or a data packet. This is the 13^th octet in the whole RTP data packet. If 13^th octet contains a value less than 0 packet is a control packet for Application Casting else it is a Data Packet.

3.5.1.  CONTROL PACKET

13^th Octet à less than 0

14^th Octet à Command

Only if 13^th octet is less than 0 it is considered a control packet. 14^th octet contains the type of control message it is passing. 14^th octet decides the rest of the packet structure. Packet structure from here will depend on the type of the command. A standard structure cannot be defined as each command may have various requirements. For example a change in the share area of the broadcaster involves the client knowing the width and height of the new sharing area and mouse movement should include mouse coordinates and may also include the state of clients machine like busy or other that may require client to change the type of mouse pointer being displayed.

If 14^th packet is 1 then it is considered a control packet for resize event. This informs the client that broadcaster has changed his sharing area and gives the new sharing area to client.

15^th to 18^th Octet à Contain a 32 bit integer indicating the new height.

19^th to 22^nd Octet à Contain a 32 but integer indicating new width.

If 14^th packet is 0 then it is considered a control packet for mouse movement. This informs client about a change in the mouse position of the broadcaster.

15^th to 18^th Octet à Contain a 32 bit integer indicating new mouse coordinate on X axis.

19^th to 22^nd Octet à Contain a 32 bit integer indicating new mouse coordinate on Y axis.

3.5.2.  DATA PACKET

A description of the Data packets needs a description of tiles, fragments, fragment numbers and the method in which they are computed.

The sharing area is broken into smaller images of size 128 X 128 each. Each 128 X 128 part is called a tile. Tiles are numbered from 0 starting from left to right and then going towards the bottom of the sharing area. At the instant of sending this packet over RTP it may not be possible to send complete tile in one packet. For this reason image is converted into bytes and these are further divided into parts. Each of such part is called a fragment. These fragments are numbered accordingly.

13^th Octet à Contains tile number

14^th Octet à Contains fragment number in this packet

15^th to 18^th Octet à Contain the total length of fragment in this packet.

19^th Octet à Contains total number of fragments for this tile number.

Congestion Control

All transport protocols used on the Internet need to address congestion control in some way. RTP is not an exception, but because the data transported over RTP is often inelastic, the means to control congestion in RTP may be quite different from those for other transport protocols such as TCP. In one sense, inelasticity reduces the risk of congestion because the RTP stream will not expand to consume all available bandwidth as a TCP stream can. However, inelasticity also means that the RTP stream cannot arbitrarily reduce its load on the network to eliminate congestion when it occurs.

Since RTP may be used for a wide variety of applications in many different contexts, there is no single congestion control mechanism that will work for all. Therefore, congestion control should be defined in each RTP profile as appropriate. For some problems, it may be sufficient to include an applicability statement restricting the use of that profile to environments where congestion is avoided by engineering. For other profiles, specific methods such as data rate adaptation based on RTCP feedback may be required.