Semantic Annotation of View-based Object Representations Using Online Search Engines

Master's Thesis , 2015 , 105 Pages , Grade: 1.3

Computer Science - Internet, New Technologies

In this master’s thesis, an automatic image annotation algorithm for object recognition systems is being developed and evaluated. Addressing the semantic gap problem, it is investigated by which means images of objects can automatically be labelled with textual tags that represent corresponding semantic information. A central question in this context is to what extent online databases are exploitable for such an annotation algorithm. Representative works from the literature normally use large scale databases containing images along their features and their semantic information. Queued images can be annotated by searching the database for images with similar features and using the provided semantics.

This work focuses another approach, where the semantic information are extracted from additional sources, such as texts according to visually similar images on websites. Those images are found in online databases or on the web using search engines. By this means, a broad range of objects can be handled without the need to create a database. Techniques will be implemented and analyzed that find websites containing images similar to the image to be annotated and that mine semantic information from corresponding texts. Eventually the system will be tested with a range of images with objects of different difficulties.

Inhaltsverzeichnis (Table of Contents)

• 1 Introduction
• 2 Fundamentals
  • 2.1 Semantic Image Annotation
  • 2.2 The Semantic Gap Problem
  • 2.3 Object Representations in Semantic Object Recognition Systems
  • 2.4 Image Comparison Methods
  • 2.5 Google’s Reverse Image Search Engine
• 3 Related Works
• 4 The Semantic Annotation System
  • 4.1 Overview
  • 4.2 The Online-based Annotation System
    • 4.2.1 Processing of Websites and Tag Extraction
    • 4.2.2 The Score Value
    • 4.2.3 Alternate Tags
    • 4.2.4 Substrings in URL
    • 4.2.5 Plural to Singular Conversion
    • 4.2.6 Words from Similar Semantic Context
  • 4.3 The Offline-based Annotation System
    • 4.3.1 Transformations
    • 4.3.2 Invariant Features
    • 4.3.3 Invariant Feature Histogram
    • 4.3.4 Jensen-Shannon Divergence
    • 4.3.5 The Offline Database
• 5 Implementation
  • 5.1 Robot Operating System
  • 5.2 Libcurl
  • 5.3 WordNet
  • 5.4 Flexible Image Retrieval Engine
• 6 Testing and Evaluation
  • 6.1 Testing Environment and Notation
  • 6.2 The Online-based Annotation System
  • 6.3 The Offline-based Annotation System
  • 6.4 Combining the Offline and Online Annotation Engine
• 7 Summary and Outlook
  • 7.1 Summary
  • 7.2 Outlook

Zielsetzung und Themenschwerpunkte (Objectives and Key Themes)

This master's thesis develops and evaluates an automatic image annotation algorithm for object recognition systems, focusing on leveraging online databases to address the semantic gap problem. The goal is to automatically label images with textual tags representing semantic information without relying on large, manually-annotated local databases.

• Automatic semantic image annotation
• Addressing the semantic gap problem
• Utilizing online search engines for image annotation
• Comparison of online and offline-based annotation approaches
• Evaluation of the system's performance on diverse objects

Zusammenfassung der Kapitel (Chapter Summaries)

Chapter 2: Fundamentals introduces content-based image retrieval (CBIR) and image annotation, focusing on the semantic gap problem and various image representation methods. It details image comparison methods and introduces Google's Reverse Image Search Engine, a key component of the developed system.

Chapter 3: Related Works reviews existing automatic image annotation systems, highlighting their approaches and limitations, particularly concerning the use of large local databases.

Chapter 4: The Semantic Annotation System presents the developed system, which combines online and offline annotation engines. The online engine uses Google's Reverse Image Search to find visually similar images and extracts tags from associated web texts. The offline engine utilizes a small local database for improved accuracy on specific objects.

Chapter 5: Implementation details the system's implementation using ROS, Libcurl, WordNet, and FIRE, explaining their roles in the annotation process.

Chapter 6: Testing and Evaluation describes the testing environment and presents the evaluation results of both online and offline engines, separately and combined. Different parameters and modules are tested to optimize performance.

Schlüsselwörter (Keywords)

Semantic image annotation, semantic gap, content-based image retrieval (CBIR), online search engines, object recognition, Google Reverse Image Search, offline database, invariant feature histograms, Jensen-Shannon divergence, Robot Operating System (ROS).

• College: University of Paderborn (Institute of Electrical Engineering and Information Technology)
• Grade: 1.3
• Author: Anonym (Author)
• Publication Year: 2015
• Pages: 105
• Catalog Number: V1419626
• ISBN (PDF): 9783346979339
• ISBN (Book): 9783346979346
• Language: English
• Tags: semantic, annotation, view-based, object, representations, using, online, search, engines