# Combining Intra-Image and Inter-Class Semantics for Image Matching

### images query similarity ranking

**Definition:** Both local (intra-image) and global (inter-class) similarities play complementary roles in image matching and ranking, so a simple linear combination scheme has been experimented with significant performance improvement over single image matching schemes.

Given an image retrieval system, the information need of a user can be modeled as the posterior probability of the set of relevant images *R* given an expression of the information need in the form of query specification *q* and an image *x* in the current database, *P* ( *R* | *q,x* ). The objective of the system is to return images with high probabilities of relevance to the user.

In Query By Example, *P* ( *R* | *q,x* ) depends on the similarity between query *q* and image *x* . On the other hand, we note that the set of relevant images *R* does not exist until a query has been specified. However we can construct prior categories of images C k , *k* = 1, 2, *M* as some prototypical instances of *R* and compute the memberships of *q* and *x* to these prior categories for contextual similarity (see an article on **Semantic Class-Based Image Indexing** ).

Both local (intra-image) and global (inter-class) similarities play complementary roles in image matching and ranking. Using a Bayesian formulation, we have:

We observe that *P* ( *q,x* ) tends to be small if *q* and *x* are similar (i.e. less likely to find similar images than dissimilar pair in a large database). On the other hand, *P* (q,x | R) tends to be large if *q* and *x* are similar with respect to *R* (i.e. q and x are more likely to co-occur in R if they belong to *R* ). And *P* ( *R* ) is constant for a given query session. Hence *P* ( *R* | *q,x* ) is proportional to the similarity between *q* and *x* given *R* (denoted as µ( *q,x* )) arid the similarity between *q* and *x* in terms of their image contents (denoted as ?( *q* , *x* )) i.e.:

For the purpose of retrieval, Equation (2) provides a principled way to rank images *x* by their probabilities of relevance to the user’s information need as represented by the query example *q* . When the similarities µ( *q,x* ) and ?( *q,x* ) are expressed in the form of probabilistic distance (i.e. inverse of similarity) such as the Kullback-Leibler distance, ordering images from the smallest distance to the largest distance is the manifestation of the minimum cross-entropy principle This echoes the *Probability Ranking Principle* in text information retrieval ].

To realize local and global semantics, a good choice for ?( *q, x* ) and µ( *q,x* ) are Semantic Support Regions (see an article on **Semantic Image Representation and Indexing** ) and Semantic Support Classes (see an article on **Semantic Class-Based Image Indexing** ) respectively. Since only ranking matters for practical image retrieval, a simple linear combination scheme has been experimented with significant performance improvement over single image matching schemes. That is, with ? [0,1] and integrated similarity ?( *q,x* ) replacing *P* ( *R* | *q,x* ):

## User Comments