Ole C Brudvik's Bookmarks tagged multimodality →  View Popular

Interview

Q: Unlike audience research or parts cultural studies which address “appropriations” of texts/cultural artefacts, your method called ‘text-centered’ insofar as it analyzes preferred meanings of texts rather than actual meanings derived by recipients. How do your respond to criticism that maintains that, in fixing a “reading” for a text – without doing audience research – you are ignoring/reducing the multiplicity of possible meanings for any given text?

A: The emphasis in my work is not on 'the' meanings of texts, preferred or otherwise, but on the resources people use for making meanings, both in 'production' and 'reception'. It attempts to describe these resources and their meaning potential. This meaning potential derives from the past uses of such resources, and is always widening and increasing. As for the 'multiplicity' of meanings, I would like to say that, from a social semiotic point of view, it is not possible to make categorical statements about whether meanings are multiple or not. How multiple they are depends on the way their use is regulated in a given domain. In some domains it is quite strongly regulated. A traffic sign, on the street, does not have multiple meanings. It cannot have, otherwise people would crash into each other. But if it were placed in an art gallery, it would. On the other hand, if the interpretation of this art work were taught in an environment that is heavily committed to a particular theoretical point of view, the multiplicity of meaning would diminish again.

Q: Considering the complexity and relative openness of these contextual conditions for how meaning is actually “made”, how do you reconcile this with the practical necessities of publishing specific analyses or presenting material?

A: I do of course analyse texts in my work, as examples, and I am quite willing to say that these analyses represent my uses of semiotic resources and that these uses are given in by my interests. I hope people will find them interesting and enlightening. And I would like to add that social semioticians, who make it their life's work to study semiotic resources and their uses past and present, have a greater knowledge of the meaning potential of these resources than non-semioticians. This, I would hope, could be one of the contributions semioticians can make. They can open people's eyes for this. It is my greatest reward as a teacher when students say: I look at images, or listen to music, in a new way now. At the same time, in recent articles and in my new book, I try to combine different methods. I have for instance written an article about baby toys, together with Carmen Caldas-Coulthard. In that article we first analyse the meaning potential of the toys themselves, then the regulative discourses that exist around them, for instance in parenting books and magazines, and then the actual way mothers use the toys in interacting with their babies. You need to look at all these things to get the whole picture, and relate them together. There is no point in creating some kind of theoretical opposition between 'meanings made by the text' and  'meanings made by the user of the text'. 

Q: Since your approach is based on Hallidayan functional grammar, isn’t this arguably problematic position something you “inherit” from this tradition? Are there any points where you would distance yourself (and your approach) from SFL, for instance with respect to its understanding of the text/reader relationship or its dyadic notion of the “signifier” vis-à-vis the “signified”?

A: Systemic-functional grammar has given me a number of crucial ideas for moving my work forwards. Because it is based, not on form categories, but on semantic-functional categories, it makes it possible to see that meanings belong to culture rather than to one semiotic mode or another. Such semantic-functional categories can in principle be used across different semiotic modes in a way that formal linguistic categories cannot. At the same time it became increasingly clear to me as I went along that language cannot realize all the meanings of a given culture, so the semantic-functional categories that have been uncovered in systemic-functional linguistics have to be augmented by categories that do not come to light if you study language only. Other crucial ideas are the idea of 'semiotic resources', on which I have already commented, and the notion of 'register'. It is never the whole of language, or of some other semiotic mode, that is relevant to a specific social context of communication. Each context, whether small or large, makes use only of part of the total resources of language, and that part is its 'register', the total of meanings made in a given context, and the total of the resources with which they are made.

As for your question of the text/reader relationship, I see systemic-functional linguistics not as a theory of texts and readers, but as a theory of linguistic resources and their meaning potentials. You can then use the theory to study how, in specific contexts, these resources are taken up, whether by text producers and/or by text 'readers'.

Q: You frequently refer to your theory and methodology as a “social semiotics” and thereby situate yourself within a particular field of research. (In fact, a forthcoming book of yours is an introductory textbook called “Social Semiotics”.) What exactly is “the social” in your social semiotics? How is it different, say, from Thibault’s definition of it as “the study of human social meaning-making practices of all types”? [1]

A: I don't think my definition is particularly different from Thibault's, although he puts perhaps more stress on analytical constructs and theoretical frameworks, and less on the social and cultural history of semiotic resources. What is 'social' about social semiotics? First or all: the fact that semiotic resources are created by humans in social contexts, as a response, at a given historical moment, to certain social and cultural (and economic) needs, and that they have to be studied as such. That is why I increasingly include in my books and articles the history of the semiotic resources which I discuss. Secondly: because semiotic resources do not 'have' immanent rules. They are regulated in different ways and to different degrees in the context of various kinds of 'semiotic regimes'. I spell these out in my new book. They include personal authority, impersonal authority, tradition, conformism, role modelling and expertise. Thirdly, because every instance of using semiotic resources takes place within a particular social setting, and needs to be explained on the basis of the interests prevailing in that setting, whether they are highly institutionalised or arising from more contingent circumstances.

Q: On a more general level, where do you see the advantages and disadvantages of a text-centred approach to analysis? What, in an ideal case, can be the contributions of such approaches to our understanding of communication/social meaning-making?

A: Again I would stress that my approach is not text-centered, but centered on the study of resources for making texts - and that includes also semiotic artefacts, communicative events, and so on. What this can contribute is, above all, the possibility of asking new questions. For instance, my work on colour has led me to think that, today, colour schemes are becoming a more important category than individual colours. The question then arises why, and that question can only be answered on the basis of a broader study of the social and cultural contexts in which this occurs. Semiotics can sharpen your perception, but not necessarily explain what you observe.

Q: What do you see as the role or place and functionality of your own approach in a larger scientific context, especially with respect to cultural studies and critical discourse analysis? How can your work be integrated? Does it need to be adapted or modified to work well within such approaches?

A: Yes, I now see social semiotics as involving three areas, the study of semiotic resources (and their histories), the study of the uses of semiotic resources in specific contexts, and the development of new semiotic resources. These areas require different approaches, and different interdisciplinary combinations. For the study of semiotic resources, you need to combine semiotic analysis and cultural history; for the study of the use of semiotic resources you need to combine semiotic analysis and ethnography; in the case of developing new resources, you need to collaborate with designers, artists and so on. Take the combination of semiotic analysis and cultural history. Linguists can describe the conventions that govern how people talk and write in specific contexts, but they do not have the means to explain how and why these conventions came about. Art historians know how visual conventions came about in the work of specific innovators, in specific historical periods, for specific reasons, but they do not describe the result of all these cultural inventions and innovations, the visual 'language' that was created, and that is now used by every illustrator, photographer, designer etc, whether in clichéd or innovative ways. Social semiotics as I want to practice it, brings these two things together. The best of both worlds. As for CDA, I think social semiotics can contribute some tools for the critically analysis of the ideological dimensions of visual texts, musical texts and so on. In articles such as my 'Visual Racism' article, or in my work on music and ideology I try to show how that can be done. 

Q: Regarding the all too often problematic relation between theory and practice, has your theory-grounded approach proven applicable in your own experience and – as far as you know – the experience of others?

A: Yes, as I just said, semiotics is not only about what is, but also about what could be, and in that area you have to work with practitioners, as I have been fortunate to be able to do particularly when I worked at the London Institute (now London University of the Arts), where PhD students did practice-based PhDs that had to combine theory and practice. Usually this took the form of a posthoc rationalization or intellectualisation of creative work that was made on the basis of intuition. I advocated putting theory first, using it as a way of asking 'what if' questions, as a way of unlocking doors, and stressing that this was done by the crucial innovators of the early 20th century, the artists and designers of the Bauhaus, the constructivist filmmakers of the Soviet Union, and so on. And I found that young artists were very receptive, particularly if they wanted to work with new media, and that their practice teachers often felt intimidated by it.

A) READING IMAGES

Q: Your approach to visual design in Reading Images seems largely based on the assumption that the language-derived categories of Systemic Functional Linguistics are by and large suitable for a pansemiotic theory or at least a theory of visual design. How has your attitude towards this analogy changed after having spent so much time on other “modes” such as sound, and recently colour and typography?

A: Yes, in Reading Images we did draw parallels between language and image, but we also stressed, constantly, that it is possible to draw parallels in some areas, for instance in the case of  'narrative processes', but not in others. For instance, many visual-spatial conceptual processes have no direct analogue in language, just as some type of semantic linking have no parallels in any current form of visual communication. As a result the changes from predominantly language-based to predominantly visual forms of science education can lead to quite different epistemologies, quite different forms of science, as Gunther Kress has shown in his recent work.

What changes as you look in some depth at more and more semiotic modes, is that you start to look at language in a different way. You discover, for instance, that some semiotic resources are not systematically organized, at least not at the moment. That leads to a new discovery: some areas of linguistic communication are also not systematically organized, and these have either been ignored, precisely because they are not amenable to systematic analysis, or been systematized in ways that resulted in enormous impoverishment and simplification, as for instance in some linguistic approaches to intonation.

Q: Personally, the one point in Reading Images, that we have always felt uneasy about, is that relatively little space is given to the exploration of moving images. Since you have considerable experience in both film production and the analysis of films (reference), are you planning to take your theories into the domain of film? And what are the main difficulties with such a move?

A: Everything in Reading Images is applicable to film images as well, and there are in fact quite a few film examples in the book. But there are two differences. The ideas we unfold in Reading Images apply only to the spatial aspects of film (e.g. the composition of shots) and not to the temporal aspects of film (rhythm, editing and so on), and, secondly, because of the added time dimension, in film the systems of visual communication we have described become dynamic. The movement of the camera and/or the actors can change the composition, or the relation between the image and the viewer in front of your eyes. There are then some areas of overlap where film has the option to represent something either spatially or temporally, for instance in what we call the 'reaction' syntagm, the link between a 'reacter' (a person looking at something in some way) and a 'phenomenon' (what the person looks at), which can be done either in one shot, or in a syntagm of three shots (the 'point of view' syntagm). The new edition of Reading Images will have a section on the moving image, and I have in fact written a number of articles on film over the years, particular on rhythm and 'conjunction' (which links in with theories of montage). But I would like to stress that I would not want now to see the semiotics of film as a separate field. Movement exists in film, but also in many other semiotics, e.g. dance, and now kinetic typography. My approach would be to look at movement in all the fields where it does semiotic work, just as with sound, spatial composition, colour, etc, I try to look at it across all the fields in which it plays a semiotic role.

B) SPEECH MUSIC SOUND

Q: In your 1999 book Speech, Music, Sound, you noted that, unlike with your work on images, in ‘applying’ the systemic functional categories of ‘metafunctions’ to sound, you found it difficult to assign singular functions to actual data. In fact, with sound, many or most instances could be understood only in terms of “metafunctional ensembles”, that is specific combinations of metafunctions. What do you make of this difference between visuals and sound? Is visual communication more like language than sound is?

A: I have found it useful recently to compare different approaches to the functions of language (and of communication generally) to the Hallidayan metafunctions. For instance, while Halliday sees his three metafunctions fulfilling a more or less equal role (he often uses the metaphor of polyphony here), Jakobson stresses that different functions dominate in different contexts. In my book on sound I said that, in sound, the interpersonal function is more fully developed than the ideational function, and the ideational has to ride on the back of the interpersonal. In the field of the visual it is the other way around, the ideational is dominant, and the interpersonal has to ride on the back of the ideational. So the visual and the sound are both different from language here. But this is not some universal truth about vision and hearing. In both the practice and the theory of music, representation has, for two centuries now, been marginalized.  Many have argued that music 'cannot represent'. Yet this idea would not have been understood in the eighteenth century, when opera introduced what was actually called the 'representative style', nor does it apply to the way music is used and talked about, for instance, in advertising, or in film soundtracks. So in my book I at once recognize the marginalization of representation, and try to bring representation back and to open people's eyes for its ubiquity in everyday music.

C) MULTIMODAL DISCOURSE

Q: In the introduction to Multimodal Discourse, a book you also co-wrote with Gunther Kress, you mention that the book is something of a compromise and that it perhaps came out too early. Without pressing you for details, perhaps you could elaborate a little on the following: How does Multimodal Discourse relate to your previous work (e.g. with respect to several concepts that are central to your argument in Multimodal Discourse – i.e., discourse, design, production, and distribution – you had already introduced these at the end of Speech, Sound, Music)? Is it an attempt to create a theory of communication, a framework for the different strands of research in visual communication: production analysis, product analysis, and reception studies?

A: Multimodal Discourse tries to do three things, first, to create a social semiotic theory of what linguists call ‘stratification’, to merge linguistic theories of stratification with theories of social stratification in communication, e.g. in the work of Goffman. Secondly, we try to show that meaning is made at every one of these levels, or strata. In linguistics, sound has been seen as meaningless, as merely realizing a linguistic 'design' that could also be realized through writing. We argue against this. The material realization of the linguistic design, whether through speech or through various forms of writing, also produces meaning. Thirdly we argue that modern technology, which has by and large been kept out of semiotics, should become an integral part of semiotics. So the work we did there provides, we hope, a clear overview of the levels of analysis needed to get the whole picture. We can now see that the 'systems' we described in Reading Images are of the level of 'design', able to be realized in a variety of material forms, just like 'language', and that ReadingImages therefore needs to be complemented by studies of the 'production' and 'distribution' levels of visual communication, and the interactions between the three levels.

Q: In what direction would you have liked the trajectory of the book to go?

A: Overall I am happy with the book, and feel that it moved my thinking forwards. But I would have liked to take more examples through all of the strata, as we do with the example of Stephanie's room in the first chapter. And I would have liked to mull over it a bit longer. Unfortunately we were already way past the original publisher's deadline. 

Q: What were your reasons for focusing, in Multimodal Discourse, on other things than the specific methodological proposals – say systematic networks, specific analytical categories etc. – which are so characteristic of the two proceeding books Reading Images and Speech, Music, Sound?

A: Originally we thought the book was going to be much more similar to ReadingImages. The main difference would be that it would concentrate not on one mode, but on the way modes work together in multimodal texts. But we got into lengthy and complex discussions about what a 'mode' actually is, whether it is different from a 'medium' and so on, and we felt it was important to come to terms with these problems in a social-semiotic way, a way that relates the structuring of semiotic resources to the social structuring of their uses. In the end we had enough ideas for a book - a quite different book from the one we originally envisaged.

Q: One of the main innovations of the book is your distinction between ‘mode’ and ‘medium’. You concede, however, that it is not always easy to distinguish between the two – has your appraisal of the situation changed in any way or have you found a better means of differentiating the two?

A: The difficulty with the concepts of 'mode' and 'medium' is that mediums can become modes and vice versa, that a particular semiotic resource may be used by some users as a medium and by others as a mode, and that some resources are in between the two, for instance colour, which formally has long been systematized, but semiotically remains unsystematic and fragmented. But that does not make the two concepts any less useful for exploring these complexities.

Q: The term ‘multimodality’ has gained wide currency in the course of the past ten years. As you were one of the first semioticians to use that term – the earliest use in your publications seems to be 1993 – could you perhaps tell us when you began to think in terms of a) multimodality and b) a pansemiotic theory?

A: It is difficult to pinpoint when this idea fully crystallized. There was a gradual realization, first that some concepts (but by no means all) could be applied to both language and image, then that some of the resources we described as resources for images, were in fact also resources for other kinds of things, such as 3D objects, architecture, etc.

As for multimodality, from the start we wanted to address the fact that many important texts are multimodal, using both language and image, and we wanted to study the way they were brought in relation to each other. In one of our very first meetings we coined the term 'integration code', which we later dropped. But to talk about the integration of word and image in a way that could do justice to both, we first had to know more about images. So the moment of multimodality was postponed, even if the idea was there from the start. 

Q: With respect to the theoretical trajectories of ‘multimodality’ and ‘pansemiotics’, how would you describe the possible interconnections between the two?

A: Perhaps you could say that multimodality is about resources for integrating semiotic modes into multimodal texts and communicative events, while 'pansemiotics' is about developing concepts you can use in thinking about each and every semiotic mode. For instance, even though I have just said that sound and image are different kinds of 'metafunctional ensembles', the metafunctions still are good tools for exploring these differences, good 'pansemiotic tools'.

Concluding questions:

Q: Where, approximately ten years after you first formulated the idea of a linguistics-based ‘pansemiotic theory’ in Reading Images, do you see the project today? What has been achieved and what remains to be done?

A: Both Gunther and I see Reading Images as a beginning. What remains to be done? Deepening out the histories of key semiotic resources and their uses; engaging more with 'non-Western' semiotic resources and their uses; working out some of the 'systems' in more detail; incorporating the 'production' and 'distribution' strata, since we have now realized that what we described in Reading Images was at the level of 'design'.

Q: What are currently, or have recently been your most important projects and research interests?

A: There are a number of balls in the air. In the context of the ‘Language and Global Communication’ research project I can now finally realize the idea of combining semiotic analysis and ethnography, thanks to my close collaboration with David Machin, who is an anthropologist. This project also involves a global network of collaborators from many countries who contribute data, local information and interviews for our work, and who we get to meet together from time to time. Also in the context of this project, we are developing software to analyse images along the lines of Reading Images, and applying it to very large sets of images from the Getty Image bank. Getty is a company that boasts it is creating a visual language for the globe. We would like to know what you can 'say' and what you cannot 'say' in this 'language'. And then I continue working on colour and typography when I can.

Q: Do you plan to integrate, on a methodological level, the different strands of your work? To give analysts an easy-to-use tool kit for analyzing multimodal discourse?

A: Every good 'applied' project challenges the theory. No 'toolkit' can ever be used without adaptation and modification. My own first project was on the intonation of radio announcers and disc jockeys, and I quickly found that none of the methods of intonation transcription and analysis available could quite do what I needed to do, because they were designed to do something else. In my own teaching I increasingly encourage students to make their own tools - for which of course they will have to use existing tools as their model. Still, I continue to believe in the idea of making 'tools' and 'resources' that will be useful to people, and I continue to try to be as explicit as I can when I make them.

Q: A look at your long-time collaborator and co-author Gunther Kress’ publishing activity suggests that he has moved from primarily theoretical concerns toward applying his work in the classroom. Are there any joint projects between you at this time or planned for the future? And what do you see as the principal areas of application of pansemiotics/multimodal theory?

A: We have just completed a second edition of Reading Images, and have no immediate plans for another joint publication, but we continue to talk. So watch this space. By the way, I think that Gunther is still producing theory. One thing I have also learnt from Michael Halliday is that you cannot really separate theory and application, and that the best theories come about in the context of 'applied' projects. So watch that space too.

Markus Rheindorf and Judith Reitstaetter would like to thank Theo van Leeuwen for agreeing, first of all, to collaborating on this piece. His openness to questions, even those of the challenging, uncomfortable kind, is much appreciated.

The kind of literacy he looks at includes the reading all kinds
of semiotic, meaning loaded forms. Technological changes and their connection
with social and economic factors form the framework for his book. As he states
on the first page:

Just as he widens literacy to include the image, he also widens his concerns
from reading images and words to the design of meaning in these modes. All
people are capable of learning how to design images and words.

The next conceptual tool he introduces is that of the mode in which the sign
is communicated.  We have time-based modes, such as speech, dance, gesture,
action, music and space based modes, such as image, sculpture, lay-out in
architecture and streetscapes (p45). Each mode has its logic and this exerts an
influence on the meaning afforded by the signs carried in the respective mode.
Writing partakes of a spatial logic, in that words are presented graphically and
it also partakes of the temporal logic because words follow sequentially. The
most potent expression of the temporal modes is the genre of narrative and the
most potent expression of the spatial modes is the genre of display. This
dichotomy between the temporal and spatial makes it possible for Kress to state
the following:

Benjamin proposed dialectical images as a way of directing attention towards
on the one hand, the human faculty to make images, and on the other hand the
dialectical manner in which two or more opposing entities can be drawn together
to bring about a shock with creative consequences. Thus, in the Arcades Project
(1999) he suggests the following:

 The shocking experience is meant to stop time and hence disrupt
precisely, how the way ‘things "just keep on going" is the catastrophe.’
(9a,1)

Media and their
interrelation with modes

Making texts and
reading texts

As Kress (2003) explains, different modes have different organizing logics, and, as such, different affordances (Gibson, 1979) for meaning-making.  For instance, ideas encoded in imagery may be said to offer a different, more spatial and simultaneously apprehended kind of meaning than the same ideas⁵ encoded in oral language, which presents ideas in a sequentially and temporally organized way.  Moreover, Kress goes on to explain, the logics and affordances of different modes also necessarily entail certain “epistemological commitments” on the part of the user:

If I say "a plant cell has a nucleus", I have been forced by the mode to provide a name for the relation between the cell and the nucleus.  I have named it as a relation of possession, "have."  If I draw the cell, and have been asked to indicate the nucleus, my drawing requires me to place the element that indicates the nucleus somewhere; I cannot avoid that epistemological commitment. (2003, p. 57)

Reflecting upon Kress’s illustration, one might feel that the epistemological peculiarities of each mode, and the weltanschauung structured according to its meaning-making affordances, may seem academic, i.e. without practical ramification.  However, the consequences of mode for meaning can indeed be concrete.  Tufte (1997) offers one powerful illustration of such a case.  He shows how the Challenger space shuttle disaster might have been averted had the involved scientists better understood relations between their intended meanings and their chosen graphic presentation materials when discussing faulty o-rings and the possible danger of explosion (pp. 27-54).

So if, in fact, the quality of meaning of a sign is somehow ineluctably bound to the semiotic mode in which it is made manifest, how is it that coherent meaning is made in a text that is constituted by elements of different modalities that entail respectively different organizing logics and epistemological commitments?  This is a question that cuts to the core of the notion of synaesthesia.