Call Me What You Want's Bookmarks tagged Downloadable Essays on Visual Language →  View Popular

Visual
Syntactic Structures
• Spring 2003

Straightforwardly, communication made by a sequence
of images might be viewed in terms of linear "panel transitions."
However, while intuitive, a linear approach ultimately has many
problems with it.

Part
1 addresses
the problems found in a panel transition model (22 pages,
452K)

Part 2 proposes
an alternative method of analysis in the form of hierarchic
rules, and can be found in my book, Early
Writings on Visual Language

Linguist David Wilkins has documented that the Central Australian community of the Arrernte (pronounced "Ar-un-da") create narratives by drawing in the sand, in what could be interpreted as a visual language. These narratives are sequential expressions with highly developed features. Notably, in comparison to our own culture the Arrernte have no writing system, but call their VL "writing" when asked. Their VL seems to have developed without the influence of writing, which perhaps shows in its structure—it does not feature spatially juxtaposed images like in comics. Rather, it is drawn temporally in one location with structurally simple icons. More interestingly though, these narratives are made in real-time exchanges, drawn into the culturally accessible media of sand.

Part of what makes the Arrernte system so effective is that their landscape provides them with the resource for expression—the sand allows for easy real-time interactivity, without the need for extraneous tools (such as pens or paper). Without that accessible sand, their VL would be forced to take on different structural features to adapt to a different ecological environment or might not have even developed as language at all.

At the same time, the print culture could also have contributed to development of a more widespread visual language. The advantage of print is its potential to reach a large quantity of people not in the same time and space as the producer. Though it may have had constraining effects on the actual structure of the VL, mass distribution has offered many people the exposure necessary for the language as a whole to flourish. Learners may need to contribute more to the language's structure during their development than in a real-time exchange, but they are still provided with stimuli that they can use to develop their own fluency.

However, a print culture alone does not limit widespread regularity. Take for example the generalized style that permeates most Japanese comics, with facial features like big eyes, pointy noses and slender chins. Originally, that style stemmed from the "God of Comics" Osamu Tezuka (who himself emulated Walt Disney). First, his single popular "voice" influenced the styles of several others. In time though, it spread to so many people that it no longer could be identifiable as the way a small group of individuals drew, but fossilized as a "manga style" permeating a culture. At this point, new learners (such as the American children now reading manga) become more interested in learning the generalized system, regardless of the individual authors associated to it.

In contrast, American comics authors by and large have styles that slightly resemble those of other authors, but not to the degree of allowing for a complete generalized style. Widespread regularity would have difficulty emerging in a culture emphasizing originality of style. For instance, recall the many Jim Lee clone artists from the early and mid-1990s. These people started out like those who originally imitated Tezuka—they all shared common styles derived from an individual influence. However, unlike the Japanese example, most Jim Lee clones that have survived continued to develop their own individual styles, using his as a foundation for broader personal development. As a result, they might be systematic in their own work, but have only with tenuous relations to the rest of the language group. Thus, though the print culture might play a role in the exposure that individual "voices" have on the language users, it alone does not determine how the learners of the visual language might develop.

By comparison, this highlights the features of our own modern VL's structure, which mimics the spatial juxtaposition employed in written language to circumvent such real-time interactivity. By having a sequence of panels, VL in comics takes on the spatial characteristics of writing, making it understandable even when preserved temporally. Interestingly, with digital tools, the preservation of a temporally unfurling sequence has become possible, such as in Scott McCloud's The Right Number.

Our cultural views on the medium could also have contributed to the lack of the interactivity in our visual language as well. The current perception grounds VL users to the roles of a print culture—a static state emerging from an "artist" and interpreted by a "reader," most often in a separate time and place than that of the creation. This is a far cry from an interactive exchange using visual material merely as a tool for expression rather than the expression itself. Culturally, such a position comes from the perception of the form as some sort of "Art" as opposed to "Language," again resulting in inhibiting the social interactivity of VL.

Interactivity in visual creation has also been shown to aid learning as well.

Historical findings of graphic representation such as cave paintings, drawings by native cultures, and other such visual expressions could be instances of visual language use, though overlooked because our analysis of them sees only the finality of their static preservation (not to mention our own inability to decode their visual grammar). Our perception of them is lacking in both temporal sequence and understanding of them as potentially multimodal communicative acts. Forms such as these might very well be examples of visual languages, though unrecognizable because of our limited categorization of them. Granted, this does not prove that such fully developed structures existed, though pictures that we consider as static works of "art" may have been part of a multimodal interactive process, accompanied by speech and or gestures.

When manual expression uses systematized signs with sequential structures—a grammar—it becomes a "sign language," found most often in deaf communities, such as those that use American Sign Language. Likewise, communication with individual drawings occurs frequently, but when put into a grammatical sequence those drawings take on the qualities of a visual language—as found in the social objects of our culture's "comics." Despite the shift towards the interactive poverty of our print culture, it has not restricted multimodality. In print, comics feature both written and visual language, both of which can take on fully grammatical forms—evidenced in their structured sequences.

The sequential aspect to language could in fact play a role in the inability to recognize older artifacts of visual language as well. Because VL is by nature visual, any temporally made sequential production of its form will be preserved statically. Thus, if a person were to "visually converse" in real-time (as we do in spoken language) but drawing only in one physical area, those images would lose their sequential characteristics. The temporal aspect of that succession would be preserved merely as a static representation: only the aftermath of that sequence would be seen. Furthermore, the accompaniment of any spoken words or gesture might also be lost because they are not physically fossilized, negating any semblance of a multimodal act.

As humans' primary means of communication, language serves as an important connection between people. Given that most language is produced dynamically in real-time, while drawings usually are not, it is no wonder that the prospect of a visual language might seem perplexing in light of language's social aspects. However, such concerns can be answered by considering where our modern VL finds most of its exposure: comics, a recognizable print culture. In print, VL faces the same restrictions as written language, separating the "encoder" from the "decoder" across both space and time.

This was not always the case. Some early comic authors actually performed their works. In the early 1900's, many well-known cartoonists had vaudeville acts where they would speedily draw pictures on a sketch-pad, canvas, or blackboard, often while performing voices and commentary. These "chalk talks" or "lightning sketches" emerged from Victorian parlor acts in the late nineteenth century, and quickly produced several celebrity performers both in America and Europe.

In fact, it was the connections to these sorts of real-time performances that inspired Winsor McCay (1869-1934) to create his famous cartoon Gertie the Dinosaur, which opened the door for animated features in movie theatres. While performance is not entirely the same as the interactive use of language, it does provide a middle ground between static print and interactive exchange. Moreover, this mixing of verbal and visual expression in real-time reflects a common practice of storytelling that has pervaded human history across cultural boundaries.