Ole C Brudvik's Bookmarks tagged multimodality → View Popular
You are here: Diigo Home > Ole C Brudvik's Bookmarks
4-ICOM Technology, Multimodal Representation and Knowledge
Tags: conference, multimodality on 2008-05-15 and saved by2 people -All Annotations (0) -About
more frommultimodal-analysis-lab.org
From the Special Issue Editors, Fall 2007
Tags: journal, multimodality on 2008-02-04 and saved by2 people -All Annotations (0) -About
more fromwww.bgsu.edu
-
The Theory into Practice section of the special issue begins with Jody Shipka's "This Was (NOT!!) an Easy Assignment." This web text offers an excellent exploration of students' experiences of multimodal writing instruction. Considering how the modalities students can access and use online likewise play a significant role offline, Shipka examines how 29 first-year composition students negotiated a multimodal framework for composing that challenged their expectations for, and/or experiences in, academic writing courses. Enacting the very practices she discusses, Shipka employs words, drawings, sound, and video together to create a scholarly work that uses multiple modes online rather than just reports on them in print.
Reconstruction 5.2 (Spring 2005)
Tags: interview, multimodality, vanleeuwen on 2006-11-24 -All Annotations (0) -About
more fromreconstruction.eserver.org
-
Interview
Q: Unlike audience research or parts cultural studies which address “appropriations” of texts/cultural artefacts, your method called ‘text-centered’ insofar as it analyzes preferred meanings of texts rather than actual meanings derived by recipients. How do your respond to criticism that maintains that, in fixing a “reading” for a text – without doing audience research – you are ignoring/reducing the multiplicity of possible meanings for any given text?
A: The emphasis in my work is not on 'the' meanings of texts, preferred or otherwise, but on the resources people use for making meanings, both in 'production' and 'reception'. It attempts to describe these resources and their meaning potential. This meaning potential derives from the past uses of such resources, and is always widening and increasing. As for the 'multiplicity' of meanings, I would like to say that, from a social semiotic point of view, it is not possible to make categorical statements about whether meanings are multiple or not. How multiple they are depends on the way their use is regulated in a given domain. In some domains it is quite strongly regulated. A traffic sign, on the street, does not have multiple meanings. It cannot have, otherwise people would crash into each other. But if it were placed in an art gallery, it would. On the other hand, if the interpretation of this art work were taught in an environment that is heavily committed to a particular theoretical point of view, the multiplicity of meaning would diminish again.
-
Q: Considering the complexity and relative openness of these contextual conditions for how meaning is actually “made”, how do you reconcile this with the practical necessities of publishing specific analyses or presenting material?
A: I do of course analyse texts in my work, as examples, and I am quite willing to say that these analyses represent my uses of semiotic resources and that these uses are given in by my interests. I hope people will find them interesting and enlightening. And I would like to add that social semioticians, who make it their life's work to study semiotic resources and their uses past and present, have a greater knowledge of the meaning potential of these resources than non-semioticians. This, I would hope, could be one of the contributions semioticians can make. They can open people's eyes for this. It is my greatest reward as a teacher when students say: I look at images, or listen to music, in a new way now. At the same time, in recent articles and in my new book, I try to combine different methods. I have for instance written an article about baby toys, together with Carmen Caldas-Coulthard. In that article we first analyse the meaning potential of the toys themselves, then the regulative discourses that exist around them, for instance in parenting books and magazines, and then the actual way mothers use the toys in interacting with their babies. You need to look at all these things to get the whole picture, and relate them together. There is no point in creating some kind of theoretical opposition between 'meanings made by the text' and 'meanings made by the user of the text'.
-
Q: Since your approach is based on Hallidayan functional grammar, isn’t this arguably problematic position something you “inherit” from this tradition? Are there any points where you would distance yourself (and your approach) from SFL, for instance with respect to its understanding of the text/reader relationship or its dyadic notion of the “signifier” vis-à-vis the “signified”?
A: Systemic-functional grammar has given me a number of crucial ideas for moving my work forwards. Because it is based, not on form categories, but on semantic-functional categories, it makes it possible to see that meanings belong to culture rather than to one semiotic mode or another. Such semantic-functional categories can in principle be used across different semiotic modes in a way that formal linguistic categories cannot. At the same time it became increasingly clear to me as I went along that language cannot realize all the meanings of a given culture, so the semantic-functional categories that have been uncovered in systemic-functional linguistics have to be augmented by categories that do not come to light if you study language only. Other crucial ideas are the idea of 'semiotic resources', on which I have already commented, and the notion of 'register'. It is never the whole of language, or of some other semiotic mode, that is relevant to a specific social context of communication. Each context, whether small or large, makes use only of part of the total resources of language, and that part is its 'register', the total of meanings made in a given context, and the total of the resources with which they are made.
As for your question of the text/reader relationship, I see systemic-functional linguistics not as a theory of texts and readers, but as a theory of linguistic resources and their meaning potentials. You can then use the theory to study how, in specific contexts, these resources are taken up, whether by text producers and/or by text 'readers'.
-
Q: You frequently refer to your theory and methodology as a “social semiotics” and thereby situate yourself within a particular field of research. (In fact, a forthcoming book of yours is an introductory textbook called “Social Semiotics”.) What exactly is “the social” in your social semiotics? How is it different, say, from Thibault’s definition of it as “the study of human social meaning-making practices of all types”? [1]
A: I don't think my definition is particularly different from Thibault's, although he puts perhaps more stress on analytical constructs and theoretical frameworks, and less on the social and cultural history of semiotic resources. What is 'social' about social semiotics? First or all: the fact that semiotic resources are created by humans in social contexts, as a response, at a given historical moment, to certain social and cultural (and economic) needs, and that they have to be studied as such. That is why I increasingly include in my books and articles the history of the semiotic resources which I discuss. Secondly: because semiotic resources do not 'have' immanent rules. They are regulated in different ways and to different degrees in the context of various kinds of 'semiotic regimes'. I spell these out in my new book. They include personal authority, impersonal authority, tradition, conformism, role modelling and expertise. Thirdly, because every instance of using semiotic resources takes place within a particular social setting, and needs to be explained on the basis of the interests prevailing in that setting, whether they are highly institutionalised or arising from more contingent circumstances.
-
Q: On a more general level, where do you see the advantages and disadvantages of a text-centred approach to analysis? What, in an ideal case, can be the contributions of such approaches to our understanding of communication/social meaning-making?
A: Again I would stress that my approach is not text-centered, but centered on the study of resources for making texts - and that includes also semiotic artefacts, communicative events, and so on. What this can contribute is, above all, the possibility of asking new questions. For instance, my work on colour has led me to think that, today, colour schemes are becoming a more important category than individual colours. The question then arises why, and that question can only be answered on the basis of a broader study of the social and cultural contexts in which this occurs. Semiotics can sharpen your perception, but not necessarily explain what you observe.
-
Q: What do you see as the role or place and functionality of your own approach in a larger scientific context, especially with respect to cultural studies and critical discourse analysis? How can your work be integrated? Does it need to be adapted or modified to work well within such approaches?
A: Yes, I now see social semiotics as involving three areas, the study of semiotic resources (and their histories), the study of the uses of semiotic resources in specific contexts, and the development of new semiotic resources. These areas require different approaches, and different interdisciplinary combinations. For the study of semiotic resources, you need to combine semiotic analysis and cultural history; for the study of the use of semiotic resources you need to combine semiotic analysis and ethnography; in the case of developing new resources, you need to collaborate with designers, artists and so on. Take the combination of semiotic analysis and cultural history. Linguists can describe the conventions that govern how people talk and write in specific contexts, but they do not have the means to explain how and why these conventions came about. Art historians know how visual conventions came about in the work of specific innovators, in specific historical periods, for specific reasons, but they do not describe the result of all these cultural inventions and innovations, the visual 'language' that was created, and that is now used by every illustrator, photographer, designer etc, whether in clichéd or innovative ways. Social semiotics as I want to practice it, brings these two things together. The best of both worlds. As for CDA, I think social semiotics can contribute some tools for the critically analysis of the ideological dimensions of visual texts, musical texts and so on. In articles such as my 'Visual Racism' article, or in my work on music and ideology I try to show how that can be done.
-
Q: Regarding the all too often problematic relation between theory and practice, has your theory-grounded approach proven applicable in your own experience and – as far as you know – the experience of others?
A: Yes, as I just said, semiotics is not only about what is, but also about what could be, and in that area you have to work with practitioners, as I have been fortunate to be able to do particularly when I worked at the London Institute (now London University of the Arts), where PhD students did practice-based PhDs that had to combine theory and practice. Usually this took the form of a posthoc rationalization or intellectualisation of creative work that was made on the basis of intuition. I advocated putting theory first, using it as a way of asking 'what if' questions, as a way of unlocking doors, and stressing that this was done by the crucial innovators of the early 20th century, the artists and designers of the Bauhaus, the constructivist filmmakers of the Soviet Union, and so on. And I found that young artists were very receptive, particularly if they wanted to work with new media, and that their practice teachers often felt intimidated by it.
-
- Questions to specific works -
-
A) READING IMAGES
Q: Your approach to visual design in Reading Images seems largely based on the assumption that the language-derived categories of Systemic Functional Linguistics are by and large suitable for a pansemiotic theory or at least a theory of visual design. How has your attitude towards this analogy changed after having spent so much time on other “modes” such as sound, and recently colour and typography?
A: Yes, in Reading Images we did draw parallels between language and image, but we also stressed, constantly, that it is possible to draw parallels in some areas, for instance in the case of 'narrative processes', but not in others. For instance, many visual-spatial conceptual processes have no direct analogue in language, just as some type of semantic linking have no parallels in any current form of visual communication. As a result the changes from predominantly language-based to predominantly visual forms of science education can lead to quite different epistemologies, quite different forms of science, as Gunther Kress has shown in his recent work.
What changes as you look in some depth at more and more semiotic modes, is that you start to look at language in a different way. You discover, for instance, that some semiotic resources are not systematically organized, at least not at the moment. That leads to a new discovery: some areas of linguistic communication are also not systematically organized, and these have either been ignored, precisely because they are not amenable to systematic analysis, or been systematized in ways that resulted in enormous impoverishment and simplification, as for instance in some linguistic approaches to intonation.
-
Q: Personally, the one point in Reading Images, that we have always felt uneasy about, is that relatively little space is given to the exploration of moving images. Since you have considerable experience in both film production and the analysis of films (reference), are you planning to take your theories into the domain of film? And what are the main difficulties with such a move?
A: Everything in Reading Images is applicable to film images as well, and there are in fact quite a few film examples in the book. But there are two differences. The ideas we unfold in Reading Images apply only to the spatial aspects of film (e.g. the composition of shots) and not to the temporal aspects of film (rhythm, editing and so on), and, secondly, because of the added time dimension, in film the systems of visual communication we have described become dynamic. The movement of the camera and/or the actors can change the composition, or the relation between the image and the viewer in front of your eyes. There are then some areas of overlap where film has the option to represent something either spatially or temporally, for instance in what we call the 'reaction' syntagm, the link between a 'reacter' (a person looking at something in some way) and a 'phenomenon' (what the person looks at), which can be done either in one shot, or in a syntagm of three shots (the 'point of view' syntagm). The new edition of Reading Images will have a section on the moving image, and I have in fact written a number of articles on film over the years, particular on rhythm and 'conjunction' (which links in with theories of montage). But I would like to stress that I would not want now to see the semiotics of film as a separate field. Movement exists in film, but also in many other semiotics, e.g. dance, and now kinetic typography. My approach would be to look at movement in all the fields where it does semiotic work, just as with sound, spatial composition, colour, etc, I try to look at it across all the fields in which it plays a semiotic role.
-
B) SPEECH MUSIC SOUND
Q: In your 1999 book Speech, Music, Sound, you noted that, unlike with your work on images, in ‘applying’ the systemic functional categories of ‘metafunctions’ to sound, you found it difficult to assign singular functions to actual data. In fact, with sound, many or most instances could be understood only in terms of “metafunctional ensembles”, that is specific combinations of metafunctions. What do you make of this difference between visuals and sound? Is visual communication more like language than sound is?
A: I have found it useful recently to compare different approaches to the functions of language (and of communication generally) to the Hallidayan metafunctions. For instance, while Halliday sees his three metafunctions fulfilling a more or less equal role (he often uses the metaphor of polyphony here), Jakobson stresses that different functions dominate in different contexts. In my book on sound I said that, in sound, the interpersonal function is more fully developed than the ideational function, and the ideational has to ride on the back of the interpersonal. In the field of the visual it is the other way around, the ideational is dominant, and the interpersonal has to ride on the back of the ideational. So the visual and the sound are both different from language here. But this is not some universal truth about vision and hearing. In both the practice and the theory of music, representation has, for two centuries now, been marginalized. Many have argued that music 'cannot represent'. Yet this idea would not have been understood in the eighteenth century, when opera introduced what was actually called the 'representative style', nor does it apply to the way music is used and talked about, for instance, in advertising, or in film soundtracks. So in my book I at once recognize the marginalization of representation, and try to bring representation back and to open people's eyes for its ubiquity in everyday music.
-
C) MULTIMODAL DISCOURSE
Q: In the introduction to Multimodal Discourse, a book you also co-wrote with Gunther Kress, you mention that the book is something of a compromise and that it perhaps came out too early. Without pressing you for details, perhaps you could elaborate a little on the following: How does Multimodal Discourse relate to your previous work (e.g. with respect to several concepts that are central to your argument in Multimodal Discourse – i.e., discourse, design, production, and distribution – you had already introduced these at the end of Speech, Sound, Music)? Is it an attempt to create a theory of communication, a framework for the different strands of research in visual communication: production analysis, product analysis, and reception studies?
A: Multimodal Discourse tries to do three things, first, to create a social semiotic theory of what linguists call ‘stratification’, to merge linguistic theories of stratification with theories of social stratification in communication, e.g. in the work of Goffman. Secondly, we try to show that meaning is made at every one of these levels, or strata. In linguistics, sound has been seen as meaningless, as merely realizing a linguistic 'design' that could also be realized through writing. We argue against this. The material realization of the linguistic design, whether through speech or through various forms of writing, also produces meaning. Thirdly we argue that modern technology, which has by and large been kept out of semiotics, should become an integral part of semiotics. So the work we did there provides, we hope, a clear overview of the levels of analysis needed to get the whole picture. We can now see that the 'systems' we described in Reading Images are of the level of 'design', able to be realized in a variety of material forms, just like 'language', and that ReadingImages therefore needs to be complemented by studies of the 'production' and 'distribution' levels of visual communication, and the interactions between the three levels.
-
Q: In what direction would you have liked the trajectory of the book to go?
A: Overall I am happy with the book, and feel that it moved my thinking forwards. But I would have liked to take more examples through all of the strata, as we do with the example of Stephanie's room in the first chapter. And I would have liked to mull over it a bit longer. Unfortunately we were already way past the original publisher's deadline.
-
Q: What were your reasons for focusing, in Multimodal Discourse, on other things than the specific methodological proposals – say systematic networks, specific analytical categories etc. – which are so characteristic of the two proceeding books Reading Images and Speech, Music, Sound?
A: Originally we thought the book was going to be much more similar to ReadingImages. The main difference would be that it would concentrate not on one mode, but on the way modes work together in multimodal texts. But we got into lengthy and complex discussions about what a 'mode' actually is, whether it is different from a 'medium' and so on, and we felt it was important to come to terms with these problems in a social-semiotic way, a way that relates the structuring of semiotic resources to the social structuring of their uses. In the end we had enough ideas for a book - a quite different book from the one we originally envisaged.
-
Q: One of the main innovations of the book is your distinction between ‘mode’ and ‘medium’. You concede, however, that it is not always easy to distinguish between the two – has your appraisal of the situation changed in any way or have you found a better means of differentiating the two?
A: The difficulty with the concepts of 'mode' and 'medium' is that mediums can become modes and vice versa, that a particular semiotic resource may be used by some users as a medium and by others as a mode, and that some resources are in between the two, for instance colour, which formally has long been systematized, but semiotically remains unsystematic and fragmented. But that does not make the two concepts any less useful for exploring these complexities.
-
Q: The term ‘multimodality’ has gained wide currency in the course of the past ten years. As you were one of the first semioticians to use that term – the earliest use in your publications seems to be 1993 – could you perhaps tell us when you began to think in terms of a) multimodality and b) a pansemiotic theory?
A: It is difficult to pinpoint when this idea fully crystallized. There was a gradual realization, first that some concepts (but by no means all) could be applied to both language and image, then that some of the resources we described as resources for images, were in fact also resources for other kinds of things, such as 3D objects, architecture, etc.
As for multimodality, from the start we wanted to address the fact that many important texts are multimodal, using both language and image, and we wanted to study the way they were brought in relation to each other. In one of our very first meetings we coined the term 'integration code', which we later dropped. But to talk about the integration of word and image in a way that could do justice to both, we first had to know more about images. So the moment of multimodality was postponed, even if the idea was there from the start.
-
Q: With respect to the theoretical trajectories of ‘multimodality’ and ‘pansemiotics’, how would you describe the possible interconnections between the two?
A: Perhaps you could say that multimodality is about resources for integrating semiotic modes into multimodal texts and communicative events, while 'pansemiotics' is about developing concepts you can use in thinking about each and every semiotic mode. For instance, even though I have just said that sound and image are different kinds of 'metafunctional ensembles', the metafunctions still are good tools for exploring these differences, good 'pansemiotic tools'.
-
Concluding questions:
Q: Where, approximately ten years after you first formulated the idea of a linguistics-based ‘pansemiotic theory’ in Reading Images, do you see the project today? What has been achieved and what remains to be done?
A: Both Gunther and I see Reading Images as a beginning. What remains to be done? Deepening out the histories of key semiotic resources and their uses; engaging more with 'non-Western' semiotic resources and their uses; working out some of the 'systems' in more detail; incorporating the 'production' and 'distribution' strata, since we have now realized that what we described in Reading Images was at the level of 'design'.
Q: What are currently, or have recently been your most important projects and research interests?
A: There are a number of balls in the air. In the context of the ‘Language and Global Communication’ research project I can now finally realize the idea of combining semiotic analysis and ethnography, thanks to my close collaboration with David Machin, who is an anthropologist. This project also involves a global network of collaborators from many countries who contribute data, local information and interviews for our work, and who we get to meet together from time to time. Also in the context of this project, we are developing software to analyse images along the lines of Reading Images, and applying it to very large sets of images from the Getty Image bank. Getty is a company that boasts it is creating a visual language for the globe. We would like to know what you can 'say' and what you cannot 'say' in this 'language'. And then I continue working on colour and typography when I can.
Q: Do you plan to integrate, on a methodological level, the different strands of your work? To give analysts an easy-to-use tool kit for analyzing multimodal discourse?
A: Every good 'applied' project challenges the theory. No 'toolkit' can ever be used without adaptation and modification. My own first project was on the intonation of radio announcers and disc jockeys, and I quickly found that none of the methods of intonation transcription and analysis available could quite do what I needed to do, because they were designed to do something else. In my own teaching I increasingly encourage students to make their own tools - for which of course they will have to use existing tools as their model. Still, I continue to believe in the idea of making 'tools' and 'resources' that will be useful to people, and I continue to try to be as explicit as I can when I make them.
Q: A look at your long-time collaborator and co-author Gunther Kress’ publishing activity suggests that he has moved from primarily theoretical concerns toward applying his work in the classroom. Are there any joint projects between you at this time or planned for the future? And what do you see as the principal areas of application of pansemiotics/multimodal theory?
A: We have just completed a second edition of Reading Images, and have no immediate plans for another joint publication, but we continue to talk. So watch this space. By the way, I think that Gunther is still producing theory. One thing I have also learnt from Michael Halliday is that you cannot really separate theory and application, and that the best theories come about in the context of 'applied' projects. So watch that space too.
Markus Rheindorf and Judith Reitstaetter would like to thank Theo van Leeuwen for agreeing, first of all, to collaborating on this piece. His openness to questions, even those of the challenging, uncomfortable kind, is much appreciated.
Ilana Snyder
Tags: kress, multimodality, transduction, transformation on 2006-11-11 -All Annotations (0) -About
more fromwww.arts.uwaterloo.ca
seminar.net - Literacy in the New Media Age by Gunther Kress
Tags: kress, multimodality, transduction, transformation on 2006-11-11 -All Annotations (0) -About
more fromwww.seminar.net
-
The kind of literacy he looks at includes the reading all kinds
of semiotic, meaning loaded forms. Technological changes and their connection
with social and economic factors form the framework for his book. As he states
on the first page:
Two distinct yet related factors deserve to be particularly
highlighted. These are, on the one hand, the broad move from the now
centuries-long dominance of writing to the new dominance of the image and, on
the other hand, the move from the dominance of the medium of the book to the
dominance of the medium of the screen. These two together are producing a
revolution in the uses and effects of literacy and of associated means for
representing and communicating at every level and in every domain. Together they
raise two questions: what is the likely future of literacy, and what are the
likely larger-level social and cultural effects of that change?
(p1)Just as he widens literacy to include the image, he also widens his concerns
from reading images and words to the design of meaning in these modes. All
people are capable of learning how to design images and words. -
How then does he argue for enhancing the ability of children to in the direction
of literacy as a design oriented activity? He suggests a chain of connected
concepts, what he calls a ‘toolkit’. -
Meaning in speech, writing and encounters with images is based upon semiotic
work, whether as articulation of outwardly made signs, or the interpretation of
inwardly made signs, such that the signs are filled with meaning. The meaning of
signs is not therefore, something given by the sign on its own. S -
S
igns have the potential of becoming signifiers
with meaning and the sign makers original meaning can be changed through
interpretation by the listener, reader or viewer of the sign. The conception of
signs as signifiers of potential meaning is derived from the work of Ferdinand
de Saussure. But, Kress goes further in his toolkit by appropriating the work of
Peirce who was concerned with the uses of the sign by readers/users. Peirce
distinguished between iconic signs (e.g. the form of flame paralleled in the
drawing of the flame), indexical signs (e.g. smoke signaling combustion is a
relation of consequence between the sign and its signifying something) and
symbolic signs (e.g the red cross is a sign signifying a meaning established by
convention). -
Kress rejects the ‘idea of arbitrariness’ found in de Saussure’s conception of
the relation between the sign as signifier and what it signifies. On this point
he is closer to Peirce in the view that the relation between the signifier and
signified is always motivated (p42). That it is motivated means that the reader,
speaker or viewer can change the content of the meaning according to the
changing local context in which the sign is encountered. Social conventions can
never determine meaning in an absolute sense. -
The next conceptual tool he introduces is that of the mode in which the sign
is communicated. We have time-based modes, such as speech, dance, gesture,
action, music and space based modes, such as image, sculpture, lay-out in
architecture and streetscapes (p45). Each mode has its logic and this exerts an
influence on the meaning afforded by the signs carried in the respective mode.
Writing partakes of a spatial logic, in that words are presented graphically and
it also partakes of the temporal logic because words follow sequentially. The
most potent expression of the temporal modes is the genre of narrative and the
most potent expression of the spatial modes is the genre of display. This
dichotomy between the temporal and spatial makes it possible for Kress to state
the following:
‘The world narrated’ is a different world to ‘the world depicted and
displayed’. (p2) -
The mixing of modes brings the reader to Kress’s view on the multi-modal, for
that is precisely what is happening as writing and image are present on
web-pages, in hyper-text, on signs in streets, in sciences textbooks. Two logics
are brought together, that of the logic of the written and the logic of the
image. -
The main point however, is that the multi-modal is not a melting together of the
modes, it is rather an interaction, with one or other of the modes occupying a
dominant position. Their distinct logics remain intact. Within modes there can
be transformation as links are established within a mode and transduction as a
message is re-configured in a different mode, from image to writing for
example. -
Kress developed his concept of the multi-modal in texts. Texts are defined as
‘any instance of communication in any mode or in any combination of modes’
(p48). -
A key component of his tool-kit of concepts in the new media age is that of
genre. This deals not with, ‘what is represented in the sense of what issues,
but with who acts (and) in relation to whom, with the question of purposes’
(p84). Put simply, social actions shape the generation of texts in different
modes and we always encounter texts as genres reflecting these social
relations. Genre knowledge, in terms of understanding their constitution,
their valuations in hierarchies of power, and being able to produce them become
the sign of a truly literate person. He gives several examples of genres, such
as swimming club rules, some beach holiday lets, an aboriginal community law. In
each instance the social group who have made the rules have designed the signs
to carry certain meanings. -
his argument that understanding the social relations of genres and the modes
they use, is the case of some 12-13 year olds in a science lesson as they write
and illustrate plant cells studied under a microscope. The pupils produce images
and text, some in the genre of a recount, some in the genre of the procedure. In
the different genres they have chosen to communicate their meanings they
construct different social relations. In the recount for example, the social
relations are a ‘friendly telling of what happened so that you might do the
same’ (p113), while in the procedure it is a case of social relations telling
the reader what to do in a more instructive, commanding manner. -
In one of the later chapters, entitled Meaning and frames: punctuations and
semiosis, Kress looks at the role of punctuation in parceling the world. His
subject in this chapter seems to be the manner in writing can imitate speech to
varying degrees through the use of punctuation. Some forms of writing show
little punctuation and are hence less close to the transduction of meanings from
speech into the mode of writing. Obviously the punctuation he has in mind can
take place in different forms and in different modes – it is not merely
dependent on the comma and semi-colons. The point is that the messages
communicated in different modes are parceled for consumption by recipients. Thus
punctuation, a topic to be expected from a professor in English Education,
becomes another concept in his tool-kit to explain the functioning of texts and
their realisation in different modes. -
Are his arguments new?
-
Walter Benjamin in the 1930s thought deeply about the effects of new visual
media on human experience (Dobson, 2003). He pre-dated Kress’s concern with the
visual image, even though he undoubtedly lacked the reasoned argumentation and
theorization Kress provides. -
Benjamin proposed dialectical images as a way of directing attention towards
on the one hand, the human faculty to make images, and on the other hand the
dialectical manner in which two or more opposing entities can be drawn together
to bring about a shock with creative consequences. Thus, in the Arcades Project
(1999) he suggests the following:
The dialectical image is a lightning flash. The Then must be held fast
as it flashes its lightning image in the Now of recognizability. The rescue that
is thus - and only thus - effected, can only take place for that which, in the
next moment, is already lost. (9,7)The shocking experience is meant to stop time and hence disrupt
precisely, how the way ‘things "just keep on going" is the catastrophe.’
(9a,1) -
From Benjamin we find an awareness of the manner in which modes, to use Kress’s
phrase, interact in a dialectical fashion. Perhaps, Kress has something to learn
from Benjamin. Perhaps, Benjamin can jog his theory along to a new set of
questions. Kress speculates towards the end of his book (p174-175) if the image
based mode inhabited by youth lacks or precludes experience of reflection,
something gained in reading books in a sustained, concentrated fashion over an
extended period. Benjamin might argue that with dialectical images and the shock
of the differences experienced, pause for reflection is created even amongst
youth looking for the latest image based experience.
IIID Gunther Kress: Reading Images: Multimodality, Representation and New Media
Tags: 2004, conference, kress, multimodality, papers on 2006-10-30 -All Annotations (0) -About
more fromwww.knowledgepresentation.org
-
One of the present tasks of a social semiotic approach to multimodality is to
describe the potentials and limitations for meaning which inhere in different
modes. For that, it is essential to consider the materiality of modes. -
Media and their
interrelation with modesModes
and media exist in culturally and historically shaped ‘constellations’. The one
that has dominated the alphabetic cultures of the ‘West’ over the last 300 years
or so is that of mode of writing with medium of book and page. Writing as mode
and book as medium have shaped western imagination, forms of knowledge,
practices of reading; the technology of writing has shaped the book, and the
technology of the book has shaped how writing has developed. The traditional
book represented the work of the author, who had laboured to produce a text,
which in its ordering represented a ‘body of knowledge’ or the shape of the
world – whether fictional or actual. Chapters in the book were coherent and
complete in themselves; paragraphs had their logic; and sentences derived their
form and purpose from the organization of the paragraph and the larger
text. -
In that world the reader’s task was to attempt to follow the pre-given ordering
of the written text, embodying the authority of the author, working assiduously
to reproduce the meaning which the author had intended for the reader. In that
world, authors could confidently speak and act on behalf of the reader, -
Contemporary texts - whether information books of all kinds, web-pages, the
screens of CD ROMs, and so on - in their increasingly often image-like textual
organization, ask the reader to perform different semiotic work, namely to
design the order of the text for themselves. Consequently two phenomena are now
becoming noticeable, as in Figure 4, which had been present but never noticed
before: the entry point of the ‘page’ and its reading path. -
Making texts and
reading textsIn the
conception outlined here, the processes of making texts and reading texts are
both are processes of design; and both are in important sense inversions of the
social and semiotic arrangements of the era of the dominance of the
constellation of writing and book. It has now been overtaken by the new
constellation of image and screen. The (at least mythically) dominant media are
now those of the screen - whether of the Gameboy, the mobile telephone, the PC,
or still the TV and video. The book and its page had been the site of writing
and the logic of writing had shaped the order of the page and the book; the
screen is the site of the image and the logic of the image is shaping the order
and the arrangements of the screen.
LLT Vol10 Num2: MODE, MEANING, AND SYNAESTHESIA IN MULTIMEDIA L2 WRITING
Tags: literacy, meaning-making, multimodality on 2006-10-13 -All Annotations (0) -About
more fromllt.msu.edu
-
As a point of departure, I take Gunther Kress’s (2003) assertion that a theory of multimodal meaning-making must account for the complementary processes of transformation and transduction, which he explains as the purposive reshaping of semiotic resources within and across modes, respectively.
-
I aim (a) to demonstrate practical evidence of the synaesthetic functions of transformation and transduction at work in the multimodal text creation process, (b) to specifically show how the synaesthetic functions of transformation and transduction can actually serve to both facilitate and hinder authorial voice, understood as the purposive expression of personal meaning, in consequential ways, and (c) to point out some possible implications of synaesthesia and multimodal communication for L2 authors in particular.
-
The status of the author in the postmodern world is an uncertain one. Though it seems self-evident that each of us has our own meanings to express, there are those, most notably French semiotician Roland Barthes, who take the notion of authorship to be a mere chimera. Barthes (1977) famously argued that if there ever were such a thing as an author, s/he is presumed "dead." Barthes writes, "a text is not a line of words releasing a single theological meaning (the 'message' of the Author-God) but a multidimensional space in which a variety of writings, none of them original, blend and clash" (p. 146). The defining characteristic of Barthes’ (nonexistent) author is the ability to conceive and express original ideas. He does not say that we cannot write, simply that we cannot write anything that is unique to ourselves. This resonates, in large part, with Bakhtin’s (1981) theory of speech genres: stable discourse types, the words of others, which we cannot help but draw upon for our own communication purposes. In an important sense, Barthes may be right. There may well be no new stories, as the saying goes; we do speak and write in re-combinations of the words and ideas of others. However, I believe that we can authentically redesign these or, as Bakhtin optimistically explains, "populate" the utterances of others with our "own intentions" (p. 293).
-
As Kress (2003) explains, different modes have different organizing logics, and, as such, different affordances (Gibson, 1979) for meaning-making. For instance, ideas encoded in imagery may be said to offer a different, more spatial and simultaneously apprehended kind of meaning than the same ideas5 encoded in oral language, which presents ideas in a sequentially and temporally organized way. Moreover, Kress goes on to explain, the logics and affordances of different modes also necessarily entail certain “epistemological commitments” on the part of the user:
If I say "a plant cell has a nucleus", I have been forced by the mode to provide a name for the relation between the cell and the nucleus. I have named it as a relation of possession, "have." If I draw the cell, and have been asked to indicate the nucleus, my drawing requires me to place the element that indicates the nucleus somewhere; I cannot avoid that epistemological commitment. (2003, p. 57)
Reflecting upon Kress’s illustration, one might feel that the epistemological peculiarities of each mode, and the weltanschauung structured according to its meaning-making affordances, may seem academic, i.e. without practical ramification. However, the consequences of mode for meaning can indeed be concrete. Tufte (1997) offers one powerful illustration of such a case. He shows how the Challenger space shuttle disaster might have been averted had the involved scientists better understood relations between their intended meanings and their chosen graphic presentation materials when discussing faulty o-rings and the possible danger of explosion (pp. 27-54).
So if, in fact, the quality of meaning of a sign is somehow ineluctably bound to the semiotic mode in which it is made manifest, how is it that coherent meaning is made in a text that is constituted by elements of different modalities that entail respectively different organizing logics and epistemological commitments? This is a question that cuts to the core of the notion of synaesthesia.
-
According to its common, clinical definition, synaesthesia is a condition
whereby a person experiences one sensation, e.g. smelling a scent or seeing a
color, in regular correspondence with a seemingly unrelated sensation.
Moreover, these experiences are physical and real. In his well-known book
The Man Who Tasted Shapes, neurologist Richard Cytowic (1993) presents
the case of "Michael" who, upon tasting a chicken dish, complained that the
chicken did not have "enough points," indicating that the flavor would have been
better if it had been pricklier (pp. 3-6).
(Mavers et al, 2001a). - Google Search
Tags: digitaldivide, google:, multimodality on 2006-09-16 -All Annotations (0) -About
more fromwww.google.com
VJ_BOOK.pdf (application/pdf Object)
Tags: media, montage, multimodality, vj on 2006-08-31 -All Annotations (0) -About
more fromwww.vj-book.com
Towards a metalanguage for multiliteracies education: Describing the meaning- making resources of language-image interaction
Tags: literacy, media, multiliteracy, multimodality on 2006-08-29 -All Annotations (0) -About
more from72.14.235.104
-
The increasingly integrative use of images with language inmany different types of texts in electronic and paper media has created anurgent need to go beyond logocentric accounts of literacy and literacypedagogy. Correspondingly there is a need to augment the genre, grammarand discourse descriptions of verbal text as resources for literacy pedagogy toinclude descriptions of the meaning-making resources of images. Someaugmentation along these lines has involved the articulation of Hallidayansystemic functional descriptions of language, mainly focussed on verbalgrammar, with the social semiotic descriptions of the meaning-makingresources of images described in a grammar of visual design proposed byKress and van Leeuwen.However, current research indicates thatarticulating discrete visual and verbal grammars is not sufficient to accountfor meanings made at the intersection of language and image. This paperadopts a systemic functional semiotic perspective in outlining a range ofdifferent types of such meanings in different kinds of texts, suggesting thesignificance of such meanings in comprehending and composingcontemporary multimodal texts, and the importance of developing anappropriate metalanguage to enable explicit discussion of these meaning-making resources by teachers and students.KEYWORDS: Multiliteracies, new literacies, metalanguage, visual literacy,visual grammar, image-text relations, multimodality, systemic functionallinguistics, social semiotics
-
No such comparable accounts of a metalanguage describing the meaning-makingresources of images and image/text interaction accompany these governmentcurriculum documents and syllabi.
-
Faced with the requirement to address themultimodality of texts, the prescription of verbal grammar, and the absence in syllabiof comparably theorized resources for describing the meaning-making resources ofimages, some teacher educators and teachers have made use of the “grammar ofvisual design” developed by Kress and van Leeuwen (1996), extrapolating from SFLaccounts of language. The commonality of the systemic functional theoreticalapproach to language and image as social semiotic systems facilitates an articulationof visual and verbal grammar as descriptive and analytical resources in developingstudents’ comprehension and composition of multimodal texts. However, beyondaccounting for the independent, albeit sometimes strategically aligned, contributionsof language and image to the meaning of composite texts, is the challenge ofsystematically theorising and describing resources for the construction of meaning atthe intersection of language and image
-
The purpose of this paper is to outline recent work addressing this challenge, and inso doing to indicate the pedagogic utility of formulating such a metalanguage ofmultimodality for the development of the multiliteracies education needed by studentsto engage with contemporary multimodal texts and texts of electronic multimedia.
-
an introductory example of one type ofmeaning made at the intersection of language and image in Anthony Browne’s (1994)picture book Zoo.
-
In the next section of the paper I will outline the key tenets ofsystemic functional semiotic theory that facilitate its use in describing meaning-making resources within and across a variety of modes of meaning includinglanguage, images, music and gesture.
-
The subsequent section, and main body of thepaper, will outline recent research dealing with the development of descriptions ofmeaning-making resources of image-language interaction.
-
Finally, I will suggest – onthe basis of research reporting the pedagogic efficacy of the metalanguage of SFL,some work on the pedagogic use of the grammar of visual design, and the discussionin previous sections of the emerging research on descriptions of image-languageinteraction – that teachers, teacher-educators and researchers consider further thepedagogic potential of existing and emerging metalanguage drawing on systemicfunctional semiotic approaches to multimodal texts.
-
According to SFL, the structures of language have evolved (and continue to evolve)as a result of the meaning-making functions they serve within the social systems orcultures in which they are used. Language is considered as a meaning-making systemwhere the options available to individuals to achieve their communicative goals areinfluenced by the nature of the social context and how individuals are positioned inrelation to it. However, although Halliday focused on language, he was very clearthat this was only one semiotic system among many other modes of meaning in anyculture, which might include… both art forms such as painting, sculpture, music, the dance, and so forth, andother modes of cultural behaviour that are not classified under the heading of formsof art, such as modes of exchange, modes of dress, structures of the family, and soforth. These are all bearers of meaning in the culture. Indeed we can define a cultureas a set of semiotic systems, as a set of systems of meaning, all of which interrelate(Halliday & Hasan, 1985, p. 4).
-
Any communicative context can be described in terms of these three main variablesthat are important in influencing the semiotic choices that are made. The first of these,FIELD, is concerned with the social activity, its content or topic; the second, TENOR, isthe nature of the relationships among the people involved in the communication; andthe third, MODE, is the medium and channel of communication.
-
SFL and its link to the situational variables of socialcontexts that has provided a common theoretical basis for the development of similar“grammatical” descriptions of the meaning-making resources of other semioticmodes.
-
For example, Kress and van Leeuwen (1996) proposed that images, likelanguage, also always simultaneously realize three different kinds of
