Other Free Encyclopedias » Online Encyclopedia » Encyclopedia - Featured Articles » Contributed Topics from K-O

Multimedia Semiotics - Syntagm and Paradigm, The formalization of semiotics

sign video interface iconic

Simone Santini
University of California, San Diego, USA

Definition : Multimedia signals are artifacts intended to convey messages and, as such, they are a legitimate object of semiotic analysis .

Some care must be taken in semiotic analysis because, if it is legitimate to consider multimedia as a form of text, the peculiar characteristics of this text, and the intertwined reading modalities that it presents, impose the use of specific analysis instruments.

The first issue that presents itself is about the very nature of the multimedia sign: to put it in Piercean terms: is the multimedia sign an icon, an index, or a symbol? (See semiotics for a definition of these and other technical terms.) The complexity of the multimedia sign is revealed by the fact that it participates in all these natures: it is at one time an icon, an index, and a symbol.

The iconic status of the multimedia sign that, at a superficial analysis would appear the most obvious, is in reality quite complex and problematic. It won’t do, in fact, to say that a picture of Charles V “stands for” Charles V by virtue of its resemblance with the king. It has long since been recognized in semiotics that similarity is too generic to create a sign relation. With suitably elastic similarity criteria, virtually everything can be found to be similar in some respect to anything else, dissolving signification in a sea of symbols worth of the hermetic tradition. Things are not such, of course: both a red apple and the picture of a Ferrari sports car are similar to a Ferrari sports car by virtue of their color, but only in the case of the picture does the color form an iconic sign. Sonesson distinguishes between an iconic ground and an iconic sign. The iconic ground is any relation of similarity that, under opportune circumstances, could generate an iconic sign. Color is an iconic ground but, in the example above, it generates an iconic sign only in the case of the picture.

Equally problematic is the indexicality of the multimedia sign, that is, the existence of constitutive causal connections between the signifier and the signified. In some cases such a connection does clearly exist: in photographs there is a causal connection between the subject and its representation, which includes the optical properties of the lens and the chemical properties of the photographic film (or, these days, the properties of the CCD device and of the printer). The problematicity of indexicality comes from the fact that these properties can’t, in general, be ascertained. This is especially true in an age of almost unlimited possibilities of digital manipulation, but it is also true at a more fundamental level. There is a fundamental difference between what a picture or a film represents by cultural convention and what they represent as the end point of a causal chain. The famous photograph l’embrasse de l’Hôtel de Ville by Doisenau, owes its signification to the historical and social position it has (France after World War II), and not to its causal connection with two lovers in Paris: its meaning—and its place in the history of photography—didn’t change after Doisenau revealed that the picture was staged so that the two lovers were not, quite likely, lovers at all. Every studio shot— Humphrey Bogart as Sam Spade, say—works at two different levels: an indexical one— the chain from Humphrey Bogart in a studio—and the one that indexical is not—Sam Spade in his office. Which one is the true signifier depends on what one is doing: whether one is writing a biography of Humphrey Bogart, or telling the story of The maltese falcon .

These considerations lead us to the analysis of the multimedia sign as symbolic and culturally mediated. The meaning of, say, an image is often a cultural construction that has little to do with the possible iconic or indexical aspects of the sign: the picture of Charles V mentioned earlier has that representational meaning because of its cultural placement, regardless of whether the man in the portrait resembles Charles V or not—a fact that nobody today is in a position to ascertain.

One particularly clear example of this fact is the iconography of Jesus. The contemporary accepted image of Jesus, with the beard and the long hair, is a creation of the middle ages: until the VII century, the Roman iconography prevailed: Jesus had short hair, no beard, appeared much younger, and was in general represented as a shepherd. Both iconographies co-exist face to face in the magnificent mosaics of the basilic of Sant’Apollinare in Classe , near Ravenna (Italy). The iconography of Jesus is clearly symbolic: it signifies because of thirteen hundred of cultural tradition, quite independently of its resemblance to the person the paintings signify, or to the actual existence of such a person.

The symbolic and cultural aspects of the multimedia sign are arguably the most relevant for the problem of signification, in spite of the superficial preponderance of the iconic aspects. The evolution of cinema in the early XX century is exemplar in this respect. At the turn of the century, cinema was little more than a way to register observables in an iconic way (but, even in this case, it is to be noted that the choice of what should be registered was a cultural act: cinema was never neutral in the way a security camera is). Less then thirty years later its complex symbolic language, theorized by the likes of Eisenstein and Vertov, was firmly established, to the point that many people regarded the introduction of audio as a useless intrusion that didn’t add anything substantial to the language of the medium. Of course, the different aspects of the multimedia sign often interact and are present together in the same semiotic act. A picture of a red Ferrari, for instance, denotes the red Ferrari by iconicity, but connotes things such as wealth or success through the cultural associations of the Ferrari: the sign, as a denotative sign, is an icon, but as a myth is a symbol. The situation is by no means unusual: myth (Althausser calls the mythological phenomenon ideology depends on social relations of power and, although the sign that the myth uses as a signifier may be iconic or indexical, the myth itself is in general symbolic.

The symbolism of the visual sign, especially in video, also takes place according to different modalities. In the case of the Ferrari, the cultural and symbolic aspect of the sign is in the connotations of the subject of the multimedia sign: the video stands for a Ferrari, which stands for wealth and success. The signification that involves multimedia (the first stands for ) is iconic (with the caveat of the cultural influence in the transition from an iconic ground to an iconic sign), quite independent of the symbolic connotations of its subject, and prior to it: the same iconic relation would have occurred had the video represented a Ford Pinto, with quite different connotations. In this case, the interpretation of the multimedia sign takes place in two distinct phases: the interpretation of the sign itself (“the video represents a Ferrari”) and that of its connotation (“therefore it stands for wealth and success”). It is worth noticing that a lot of attention has been given to the standardization of the first of these problems (in standards such as MPEG7, RDF, ) but not to the second, which, in light of these observations, is the most interesting and relevant of the two.

In a second modality, the symbolism is in the content itself. This modality is often very evident in cartoons: when a character starts running its legs move for a while before it actually starts, and when Wile E. Coyote falls at the bottom of a canyon the whole frame shakes, and we see a cloud of dust coming up from the bottom. These signs are not completely unmotivated, and do have a residue of iconicity in them. The characters beginning to run, for instance, are reminiscent of the way in which a powerful automobile skids its tires before starting at high speed, an obvious signifier of the fast run. Certain exaggerations, however (the shaking of the frame) are part of the language of cartoons that is symbolic and that, associated with certain genre markers, has made its way into films.

A third modality of symbolism has to do not with the content of the video, but with the way in which the content is presented. Film, for example, has a language of its own, a sign system whose elements are editing techniques, containing cultural norms such as the one, observed at least until the 60’s, prescribing that a “dissolve” marks the passage of a relatively long time, and always implies a discontinuity in the action, while a “cut” marks the passage of a typically shorter time or no time at all preserving, in this case, the continuity of the action; or the rules about the interest line and the positioning of the dominant character. This language is symbolic, although one sometimes finds surprising and tortuous indexical connections. The American film noir of the 1940’s was at first shot in black and white for economic reasons: these were low budget films, and color film was very expensive. The cult status of these films in the following decades, however, contributed to give a completely different meaning to black and white: to shoot films in black and white today hasn’t the same connotations it would have had the film noir never existed. This symbolic modality is very useful technically because it is prior and, often, independent of the contents of the video, therefore it can provide access to certain meanings of the video without requiring the interpretation of the fames and, sometimes, drive the interpretation process. To this day, the use of this type of language is in general limited to its temporal aspects, such as the identification of cuts and dissolves

Syntagm and Paradigm

One of the important semiotic shifts caused by the digital medium and by the computer interface is that from the preponderance of the syntagmatic plane to that of the paradigmatic . In a text, speech, or video, the semiotic units are linked on the syntagmatic plane: they form a chain connected by relations of contiguity and complementarity. By contrast, a computer interface typically presents a choice of different alternatives, only one of which will be chosen by the user. The choices are all equivalent in the sense that, at the point of the interaction when they appear, each one of them could be made. They are, in Saussurean terms, in a paradigmatic relation. Of course, during each interaction episode, a user will go through a specific sequence of choices, that will therefore constitute a syntagm; the syntagm, however, is a product of the interaction act: the interface in itself is paradigmatic and, in this sense, it resembles more a dictionary from which the textemes that constitute a text come from than the text itself. This semiotic discordance between the paradigm of the interface and the syntagm of the video is at the basis of the difficulties experienced by many researchers of creating a good interface for video that could take advantage of the possibilities of the computer: the vast majority of computer interfaces for video are but glorified VCR interfaces. To the same semiotic discordance one can trace the interest of many researchers in the so-called video summarization techniques, which try to transform a video into a series of representative images that can be presented simultaneously on a computer screen, transforming the syntagm into a paradigm.

These problems are compounded by the dichotomy between opaque screens and transparent ones (the terminology and the concept are due to Lev Manovich. When looking at a video, the screen works as a window through which we observe the world of the video; that is, the screen becomes transparent and disappears from view. By contrast, in an interface, the different elements are placed on the screen: the user looks at the screen as an opaque support: there is nothing beyond it. Changing between these two modes of operation requires a cognitive shift that disturbs the integration of video and its interface. One interesting possibility in this respect would be—at least for videos produced specifically for the digital medium—to place the manipulation interface in the video itself, so that the transparency of the screen can be preserved. The few attempts in this direction (dubbed hyper-linking) have been quite limited in scope, often directed to linking products placed in films to a web page where one can buy the product. One of the limitations of these attempts is that, while the links were placed in the video, they were selected using the mouse, and the mouse’s cursor, of course, leads one back to an opaque screen. An even more reductive circumstance is that, so far, digital video has been thought more or less as the digitalization of broadcast video, and the expressive possibilities of the multimedia sign haven’t been explored at all. Experimental videos of the 1960’s and 70’s, with their exploration of non-linear narratives and pastiche foreshadowed the future use of video in interactive media but mainstream internet video appears to be tied to a more traditional narrative models, that deal problematically with interaction and interfaces.

A computer interface and a video are both sign systems but, possibly for the first time since computer scientists started to think about interfaces, the interface reveals profound semiotic discrepancies with the sign system that constitutes the data, the interface working on the paradigmatic plane on an opaque screen, the video working on the syntagmatic plane “behind” a transparent screen. The difficulty of reconciliation of these very different sign systems is responsible in no small measure of the difficulty of creating an interface for the effective manipulation of video.

Quite interestingly, things work better for images. The relation between the different parts of an image is syntagmatic: they are associated by juxtaposition to form one image-text. Their stability, however, makes it easy to intervene paradigmatically on an image, that is, to replace some of its parts with other elements that stand in a paradigmatic relation with them. In a picture, say Rembrandt’s Aristotle contemplating the bust of Homer , the various elements (Aristotle and the and the bust of Homer) stay in a syntagmatic relation about which—as in the case of video—little can be done apart from showing it on the screen. But the stability of the image creates the possibility of an interface that works on the paradigmatic plane, with elements that could be substituted to those in the picture and that would stay in the same syntagmatic relation (Thelonious Monk instead of Aristotle, a picture of the Queen of England instead of the bust of Homer): the interfaces of all image manipulation systems work on the elements that are not in the images, but that could be placed in the image. In the case of the images, moreover, the screen becomes opaque: we usually look at a picture on the screen rather than at a scene through a screen.

The formalization of semiotics

One of the difficulties that one has when dealing with the semiotics of multimedia from the point of view of computing science (or, indeed, in any form of study of the communication between computers and people) is the dichotomy between the qualitative nature of semiotic statements and the quantitative and fully formalized form that any computing activity must take. An attempt to a formal theory of the communication of signification in a rather general setting is Joseph Goguen’s algebraic semiotics . A sign system, in algebraic semiotic, is a theory , defined as follows:

A sign system S is a 7-tuple S =( T,V,= T ,= V ,F,A ), where:

  1. T is a set of data types;
  2. V is a set of parameter types;
  3. = T is a partial order on T , called the subtype relation;
  4. = V is a partial order on T V , called the level relation such that for each v e V , and t e T is it v = V t , and there is a unique type t ? (called the top) such that for all t, t = V t? ;
  5. C is a set of partially ordered sets of constructors, one set of constructors for each level;
  6. F is the set of functions and relations on signs;
  7. A is a set of logic sentences called axioms.

The data types describe the “informative” elements of an interface, and their organization allows one to mirror the structure of the data that the interface must deal with; the parameter types are, typically, used to describe interface elements, and contain things such as colors, sizes, and so on. With these systems, one can define semiotic morphisms , that is, morphisms that transform a sign system into another one while maintaining as much structure as possible, allowing one to create a hierarchy of expressivity of sign system, and to decide whether an interface, model as a sign system, has sufficient expressive power to deal with a given set of data.

It is not clear, as yet, the extent to which this or other formalizations will help elucidate the nature of the multimedia sign and help users interact with them through suitable interfaces. Algebraic semiotics has been used to study the transformation of the “message” of image interfaces for image data bases, but so far, there doesn’t seem to be any study of their applicability to other multimedia data.

Multimedia Sensor Networks - Multimedia Sensor Network Systems [next] [back] Multimedia Semantics - Conceptualization of the multimedia content, Extraction and representation of

User Comments

Your email address will be altered so spam harvesting bots can't read it easily.
Hide my email completely instead?

Cancel or