Semantic Visual Query and Retrieval

Definition: As an alternative to content based queries, semantic visual queries, such as Query by Spatial Icons, allow user to specify visual semantics explicitly based on a predefined visual vocabulary used to index the images.

Searching images by text queries is effective if the images are associated with comprehensive keywords. Query By Example (QBE) and Query Canvas (QBC) allow image retrieval based on image contents. However QBE and QBC are implicit as they expect the image retrieval systems to understand the visual semantics embedded in the image or drawn respectively. As an alternative, Query by Spatial Icons (QBSI) allows user to specify visual semantics explicitly based on a predefined visual vocabulary used to index the images (see example in Figure 1).

A QBSI query is composed as a spatial arrangement of visual semantics. A Visual Query Term (VQT) q specifies a region R where a Semantic Support Region (SSR) S i should appear and a query formulas chains these terms up via logical operators. The truth value ?(q,x) of a VQT q for any image x is simply defined as

where T i® is as defined in Equation (3) in the article on Semantic Image Representation and Indexing .

A QBSI query Q is a disjunctive normal form of VQTs (with or without negation):

The query processing of query Q for any image x is to compute the truth value ?(Q,x) using appropriate logical operators. As uncertainty values are involved in SSR detection and indexing, we adopt fuzzy operations as follows (for efficient approximation and other details, please refer to ),

A typical QBSI interface allows a user to specify a VQT by clicking on a SSR icon from a palette of icons associated with the SSRs and dragging it as a rectangle into a canvas. To apply negation, the user can click on the ‘NOT’ button and then the rectangle (a yellow cross will be shown on the selected rectangle). The user can continue to specify more VQT in a conjunct by repeating the above steps. To start a new conjunct in the disjunctive normal form (Equation (2)), the user can click on the ‘OR’ button to start a new canvas with SSR icons.