# Data Modeling, Multimedia

### time intervals relations relation

*Simone Santini
University of California, San Diego, USA and Universidad Autónoma de
Madrid, Spain*

**Definition:** Multimedia data modeling refers to creating the relationship between data in a multimedia application.

Data modeling ( *sans* multimedia), mush like requirement engineering, is in a sense a hybrid discipline that operates as a bridge between, on one side, some branch of the computing profession and, on the other, the problems from other disciplines to which the computing profession applies its solutions. In the general acceptance of the term, data modeling has the purpose of creating a formal data structure that captures as well as possible the (often informal) data of the application, and the relations between them. Standard data modeling is often done using semi-formal methods such as the *entity-relationship diagrams* . These methods, in general, put the emphasis on the relationship between the data, rather than on the data themselves: the data are often represented as simple data types (integers, strings, etc.) connected by the relations of the model.

A greater emphasis on the data themselves is placed by so-called *object-oriented* approaches, most of which make use of graphic formats such as UML . (Object oriented methods in general, and UML in particular, are designed as notations to specify the structure of programs and, as such, are not quite suitable for data modeling. To name but one problem, in UML design it is necessary to decide immediately which entities are objects and which are attributes, a decision that should properly be left to a later stage of the design process.) It is worth noticing that an object in the typical object oriented modeling formalism is not an object at all, since it is not a truly abstract data type, in that its internal structure is always explicit and part of the model: in this sense, an object in a data model is closer to Pascal’s records or C’ structures than to objects.

Multimedia data are often complex and have a rich structure and, at least in their static aspects, are often modeled using object oriented modeling formalisms, possibly with a higher level of abstraction since the extreme complexity of certain multimedia structure pushes the designer to place the emphasis of a design on the *behavior* of multimedia data rather than on their *structure* . But, apart from some adjustment to allow a higher degree of abstraction, multimedia data would probably have little to add to the general area of data modeling were it not for the issue of *time* .

Many multimedia data have a time component, and the existence of this component creates issues, such as data flow or synchronization, that are quite alien to traditional modeling techniques. The timing problems of multimedia data are, in fact, quite independent of those of areas such as historical data bases, temporal queries, or real time data bases, *vis à vis* which data modeling has developed its time modeling techniques. Multimedia data are often retrieved and processed in order to be presented, and the technical or psychophysical differences in the perception of different types of data imposes, in the multimedia arena, considerations unknown in the case of other data modeling problems.

To make but an example, in a multimedia presentation the synchronization between the presentation material (typically: transparencies) and the audio-video content is relatively coarse—a mismatch of the order of seconds can be tolerated—while the synchronization between the audio and the video of the speaker must be considerably finer: a mismatch of a single frame (1/30 th sec.) is already noticeable. (The means to achieve synchronization also depend on the characteristics of the medium and of its perception: video frames, for example, can be dropped more readily than audio packets, as the human sensory system discerns gaps in audio as more prominent and disruptive than comparative gaps in video; this kind of problems, however, belong to a level different than that of data modeling, and will not be considered here.)

Many of these problems are solved by suitable querying, access, or indexing techniques. The purpose of data modeling is, for multimedia as for other types of data, to provide a suitably formalized view of the relevant data structures, so that a suitable solution can be designed. In the specific area of multimedia, next to the traditional aspects of data modeling, this entails the specification of the temporal relations between the data, as a prolegomenon, e.g., to the temporal schema design of the data bases involved in a system.

An important representation often used in multimedia data modeling is that of *intervals* (in time). Given a set of instants *T* , a partial order on *T* , and two elements *a* , *b* *T* with *a* = *b* , the interval *[a,b]* is the set { *t* | *t* *T* , *a* = *t* = *b* }. Note that, contrary to the common intuition, time is represented here as a *partially* ordered set. This stipulation is useful, e.g. for modeling situations of time uncertainty when, given two events *e* 1 ? *e* 2 , at instants *t* 1 , *t* 2 , the uncertainty in the measurement of time doesn’t allow one to determine which of the two events came first.

Many multimedia modeling problems have to do with synchronization, that is, with the temporal relation between pieces of multimedia data, each one considered as an interval. The simplest case of such relation is that between *two* intervals, in which case there are 13 topologically distinct relations in which they can be, represented, using the graphical notation of Allen by the seven relations in Figure 1 and by the inverses of the first six (the inverse of the seventh is clearly equal to the relation itself).

Here, *a* , *a* (resp. *b* , *b* ) are the start time and duration of event *a* (resp. *b* ), , their relative position (t *d* = |p b – p a |) and ? T the duration of the composition of intervals. The relations between intervals can be expressed in terms of conditions on these parameters. For instance:

a before b = (p a +t a < p b )

and so on. Binary relations can be extended to lists of intervals by assuming that each pair of contiguous intervals stands in the same binary relation. In this case, for instance, the n-meets relation would be represented as

Several partial order relations can be defined for intervals. Intervals being sets, the most obvious one is probably the set inclusion relation which, in terms of the time parameters of the intervals, translates to

*a* = *b* (p *a* = p *b* ) (p *a* t *a* = p *b* t *b* )

but this order doesn’t capture the temporal semantics, so to speak, of the interval relation: one would like, at least, a partial order that, in the limit when the length of the intervals tends to zero, becomes isomorphic to the total order of time instants (without uncertainty).

A partial ordering such as

*a* = *b* (p *a* = p *b* ) (p *a* t *a* = p *b* t *b* )

would do the job. This partial ordering is related somewhat to the qualitative interval relations since, for instance,

a before b *a* = *b*

a meets b *a* = *b*

## User Comments