Two for No: theory in practice

I'm happy to be able to announce that my friends Alexander Danner and Tym Godek have a new weekly comic series called "Two for No." What makes it even more exciting for me is that the comics are all based around ideas from my theories!

Several years back, I wrote a paper about "visual language poetry" for ComixTalk where I proposed a "formal" type of visual language poetry that did not imitate the structures of verbal poetry, like when people create "comic haikus." Rather, my poetic form—and my proposal for how to make more poetic forms in visual language—drew directly from the properties of the structure of the the visual language of comics.

That poetic form was called a "Reducto" and it's been played with a bit previously by Grant Thomas, who also explored some of his own manipulations of it.

Now, Alex and Tym offer a whole series that is based on these ideas. They also go further than just using my form and have started making another visual language poetic form, the "Dissociation."

When I lived in Boston, I'd often wax theoretical with Alex and visited classes he taught on comic literature and creation. He frequently queried my thoughts on how my ideas might translate back around to the "practice" side of things, and got very excited when I recalled these ideas about poetry. I'm really glad he and the ever-talented Tym have been exploring these further, and I was glad to be able to see a sneak peak as they started on them.

In fact, Alex's pushing me about this stuff lead me to seriously consider adding a chapter in my upcoming book about visual language poetry. Alas though, I started running out of space and it didn't make the cut. Who knows, maybe Alex and I will one day get around to writing the "visual language theory in practice" book we've discussed in the past, with a whole chapter on this stuff?

For now though, go check out "Two for No," bookmark it, and visit every Monday when they post new comics!

11 years and pretty cover

I always seem to miss these things when they happen, but as of May 31st, my website has now been online for 11 years!

The amusing thing is that as another year goes by with my website, I am currently working hard to completely redesign and relaunch the site with the new url, www.visuallanguagelab.com. As I've mentioned before, now would be a good time to update your links if you haven't already. Full launch of the new site should be in a month or so.

In the meantime, production is still mounting on my new book. I expect to receive the proofs any day now. Even though it won't be released until December in the UK and January in the US, it has a cover!  Check it out:

Updates and such

Things have been a bit busy lately, so I've had less time to devote to blogging (suggested topics always welcomed, FYI). I've been hard at work on several revisions of papers for journals, as well as prepping several more papers for submissions. I'm also gearing up to start running a new brainwave study here at UCSD, so that's exciting to have underway.

I've also been hard at work on a major redesign/relaunch of this website. The site is getting a new look, new organization, and even a new address. If you'd like to plan ahead for the switch, the new url will be: www.visuallanguagelab.com (currently a redirect to the present site, which will then be flipped on relaunch).

Finally, I'm happy to report that my upcoming book, The Visual Language of Comics, has now entered the production stage! It will be fun to see the proofs in a few weeks. For now though, it's exciting to see that my publisher has now created a webpage promoting the book, including a growing list of endorsements. Looks like there's even a page on amazon for it. Let the countdown until it's release in December begin!

CFP: Interdisciplinary approaches to visual narrative

For those that might be interested, one of my projects is trying to organize a book summarizing important research on visual narrative. This book will be a companion volume to my book out later this year, The Visual Language of Comics. If you may be interested in contributing, here's a Call For Papers for it...

CFP: Interdisciplinary approaches to visual narrative

While there have been a growing number of books on comics in recent years, very few have addressed aspects of structure, particularly from theoretical, cognitive, or experimental points of view and outside the realm of literary or sociocultural theory. I am working to organize a compilation of important papers on the understanding of sequential images. Most of the chapters will be either 1) summary papers that provide extensive bibliographies that can provide an overview to students and a resource to other researchers, or 2) reprints of significant research that remain under-recognized or hard-to-find.

This Call for Papers asks for proposals for papers of two types of chapters focused particularly on research outside of English, presented for an English speaking audience:

1. Chapters that summarize, in English, advances in comic theory from non-English speaking researchers. Such chapters should be large in scope with extensive reference sections.

2. Translations into English of significant non-English comic theory (structural, cognitive, experimental, etc.) from important papers or book chapters.

Topics or chapters outside this scope may be considered, though best to contact me directly with inquiries. (Of interest may be: review papers of other types, historical development of comic “symbology”, empirically grounded discussions of differences between comics cross-culturally, etc.). Importantly, papers should be relevant not only to scholars of comics, but also to linguists, cognitive scientists, and psychologists.

Contributor Guidelines

1. Abstracts of 400-500 words accepted. Papers of 5000-9000 words, including notes and bibliography, accepted. Please also include a short biographical statement.

2. All documents should be submitted as Word or Word-compatible files. PDFs are also acceptable.

3. Submission deadline: May 15, 2013. June 15, 2013

4. Materials should be sent to me via email (Neil Cohn)

Comics, games, and bad science

I'm often very excited when I find new research on comics, especially when it's experimental. There is so little done that it's a treat to find something I didn't know about. Unfortunately, sometimes my excitement at new data turns sour when I see what was actually done...

I recently found this study (pdf) by Kinzer and colleagues that compares the comprehension and eye movements of readers for narratives in comics and video games. Their main goal is to help provide support for the use of comics and video games in educational contexts.

In this study, they presented sixth graders with either a video game version of a story or a comic created from the images of the video game. Overall, they find that participants understood the story in the video game version better than the comic version. They also found people spent more time engaged with the game than the comic.

I would take all of these findings with a grain of salt...

...because the stimuli appear to be extremely confounded because the comic versions of the story appear to be so badly created. Judging by the example in the text, the comic pages clearly were created by someone who had no real fluency in the visual language of comics. This is clear at a glance just by the example that they provide in the paper:

First off, the images make it extremely hard to tell what's going on. Second, almost all of the balloons are placed outside of their originating panels to the extent that they completely overlap in panels forward and backward in the text. I don't even need to know what's in the text to know this will be confusing to a reader. This is so "illegal" in the rules of page construction that it is almost painful.

If this is their example (what is probably the best example they have), I shudder to think what other pages in the experiment look like. Seriously, if I wanted to design an experiment that had "incomprehensible comics pages" as one type of stimuli, I'd use pages like these.

It's no wonder they found that their participants had poorer comprehension for the comic version—their stimuli are the equivalent of trying to test English comprehension while using broken English. It tells you next to nothing of interest.

There are two main points I'd like to make about this:

First, good experiments are hard to design, and having something worth saying must follow from having successfully designed an experiment that can give you good information. It pays to be critical as a creator and reader of scientific research (no matter what the topic).

Second, doing experiments using the visual language of comics is not trivial. Stimuli cannot be created by anyone, regardless of their fluency in comic creation. Just because you can throw together some images and words into panels on a page does not mean you've successfully created an example of "native" visual language. Believing otherwise does a disservice to yourself and to others who might read and cite your research.

Kinzer, C. K., Turkay, S., Hoffman, D. L., Gunbas, N., Chantes, P., Chaiwinij, A., & Dvorkin, T. (2012). Examining the Effects of Text and Images on Story Comprehension: An Eye-Tracking Study of Reading in a Video Game and Comic Book. In P. J. Dunston, S. K. Fullerton, C. C. Bates, K. Headley, & P. M. Stecker (Eds.), Literacy Research Association Yearbook 61 (pp. 259-275). LRA: Chicago, IL.

New paper: Navigating Comics

I am happy to announce I have a new paper published in the journal Frontiers in Cognitive Science that outlines a theory of how people navigate page layouts. A previous version of this paper was posted here several years ago, though this is now the final version. You can download a pdf of the paper here: "Navigating Comics."

The paper describes an experiment I conducted that examined people's intuitions for moving through page layouts. I then incorporate the results into a larger theory of the structure of page layout. Here's the official abstract:

Like the sequence of words in written language, comic book page layouts direct images into a deliberate reading sequence. Conventional wisdom would expect that comic panels follow the order of text: left-to-right and down—a “Z-path”—though several layouts can violate this order, such as Gestalt groupings of panels that deny a Z-path of reading. To examine how layouts pressure readers to choose pathways deviating from the Z-path, we presented participants with comic pages empty of content, and asked them to number the panels in the order they would read them. Participants frequently used strategies departing from both the traditional Z-path and Gestalt groupings. These preferences reveal a system of constraints that organizes panels into hierarchic constituents, guiding readers through comic page layouts.

Cohn, Neil. 2013. Navigating Comics: An empirical and theoretical approach to strategies of reading comic page layouts. Frontiers in Cognitive Science. 4: 1-15.

Please help

Just one more pleas for helping my colleague... www.nalinineedsyou.com. Please watch, repost, and help us save the life of an amazing woman:

Help Nalini Now

This is not going to be your normal post on comics and visual language. This post is a plea for help from anyone who might read this blog and anyone compelled to help save someone's life. I summarize here much of what's said in a blog post from my colleague Sam Sommers.

From 2006 to 2012 I was a graduate student at Tufts University. One of the professors there was Nalini Ambady, who happened to be the advisor of many of my closest friends. You probably know Nalini's research in social psychology, though you may not have heard of her. Have you heard of Malcom Gladwell's book Blink, which talks about the "ability to extract an enormous amount of meaningful information from the very thinnest slice of experience"? The idea of "thin slices" comes from Nalini's work.

I am sad to say now that Nalini is battling with leukemia and is dire need of a bone marrow transplant. Because she is Indian, she has a very specific genetic marker that likely needs a match with another South Asian. Time is of the essence—she likely needs a match within 8 weeks. Her story is detailed much more at this website: http://www.helpnalininow.org

Helping save this woman's life is easy:

1) Register today as a potential bone marrow donor in the national registry. It’s easy: if you’re between the ages of 18-44 you can simply go here: http://marrow.org/Join/Join_Now/Join_Now.aspx?promo_code=MatchNalini 

Make sure to enter the promo code “nalini” and your request for a cheek swab will be rushed to you and its processing expedited.

2) Everyone has the potential to save a life by registering on the site. In Nalini’s case, though, it’s particularly Indian donors who are likely to be a match. Accordingly, please forward this blog post or the http://www.helpnalininow.org website to any websites, email lists, or organizations with large South Asian memberships.

Please help. It takes just a few clicks of a mouse and time is running out. Your effort may help save a woman's life. Thank you.

New article: Visual narrative structure

I am happy to announce I have a new paper published in the journal Cognitive Science that outlines my theory of sequential image understanding. I argue that approaches like "panel transitions" cannot account for the creation of meaning in sequential images, and I offer a new theory of "Visual Narrative Grammar" that better accounts for how we comprehend visual narratives.

You can download this study along with my other papers, or directly to the pdf here.

This new theory uses narrative categories similar to traditional notions of narrative (though operationalized) that are organized into hierarchic constituents. The basics of this theory are outlined, along with diagnostic methods for testing these categories and constituents. Finally, I outline how this theory can apply beyond the sequential images found in comics to the understanding of film and verbal discourse.

This paper presents the basics of this theory for the first time in a cohesive paper, though this paper actually only consists of part of the broader theory of Visual Narrative Grammar. I hope to discuss the theory in full in subsequent papers and books. I should note also that my experiments on the cognition of understanding comics (also available for download) use this theory as their basis.

Here's the full abstract:

Narratives are an integral part of human expression. In the graphic form, they range from cave paintings to Egyptian hieroglyphics, from the Bayeux Tapestry to modern day comic books (Kunzle, 1973; McCloud, 1993). Yet not much research has addressed the structure and comprehension of narrative images, for example, how do people create meaning out of sequential images? This piece helps fill the gap by presenting a theory of Narrative Grammar. We describe the basic narrative categories and their relationship to a canonical narrative arc, followed by a discussion of complex structures that extend beyond the canonical schema. This demands that the canonical arc be reconsidered as a generative schema whereby any narrative category can be expanded into a node in a tree structure. Narrative “pacing” is interpreted as a reflection of various patterns of this embedding: conjunction, left-branching trees, center-embedded constituencies, and others. Following this, diagnostic methods are proposed for testing narrative categories and constituency. Finally, we outline the applicability of this theory beyond sequential images, such as to film and verbal discourse, and compare this theory with previous approaches to narrative and discourse.

Cohn, Neil. (2013). Visual Narrative Structure. Cognitive Science, 37 (3), 413-452 DOI: 10.1111/cogs.12016

A Caveat: misunderstanding comics and the brain

Via this article I stumbled onto this dissertation which promotes using comics in educational contexts (a topic I am very interested in).

In one of the chapters of the thesis, it looks at the understanding of comics and includes a "neuroscience" section. Now, even when I disagree with them, I am always one to encourage research and writing on the structure and comprehension of comics (and I am wholly in support of the effort that this thesis is trying to make with regard to graphic textbooks). However, I have also railed on the invocation of neuroscience when used inappropriately.

Given my commitment to educating about comics and the brain, it behooves me to explain exactly what is wrong with a discussion like this, and especially what is misleading/wrong with an image like this:

This graphic—and the accompanying description—reflect the worst of misunderstandings and invocations about the brain. Here are among the problems:

1) It suffers from the "localization" fallacy. Contrary to what the image implies, there is not a "speech" location in the brain. Nor is there a "writing" or "vision" location in the brain. The brain does not localize full behaviors in the brain in modular locations. Rather, the brain is highly interconnected, with portions of the brain devoted to particular functions that interface to produce complex behaviors like speech, language, or vision.

For example, while the occipital cortex (back of the head) is the "primary" visual area, the visual system alone goes up through the top of the cortex (the "dorsal stream"), down into the bottom, temporal lobe (the "ventral stream"), and connections from the eyes go through the middle of the brain on its way to the occipital lobe in the first place. In other words: the visual system engages lots of different parts of the brain, not just back of the head. (...and where is the rest of the brain in this image???)

Reducing the brain into localized areas for each of these complex behaviors displays a lack of understanding about how the brain works (at least, to the extent we understand it so far) and for how these complex behaviors work ("language" alone can be subdivided into at least four different major substructures, which each have more substructures). Beyond this...

2) Discussing the brain is entirely unnecessary in a thesis like this. The overall point that this thesis is trying to make is that comics are very complex and involve numerous interacting parts that we understand almost effortlessly. This point does not rely on discussion of the brain to come across. In fact, I myself have done quite a lot of work describing the complexity of structure found in comics, some of which discusses the brain, and some does not.

This thesis does not need it, and including a poorly-understood discussion of the brain only hurts its overall point. Now, studies have shown that just including an image of the brain somehow convinces people that its point is more believable. However, unless the thesis is actually about the brain, such discussion is unnecessary and borderline dishonest (albeit unintentionally). Which leads us to...

3) If you're going to talk about comics and the brain, at least do the basic research to discuss what work has actually been done on this topic. Granted, there are a limited number of studies that have directly examined this issue. However, they are out there, and reviews of some of these papers can be found on this very blog: Here, here, here, and here.

I myself have now done three studies looking at comics and the brain. I'm currently writing up my latest two studies (which were my dissertation), but my first experiment is online here: Full pdf article, Short "comic" version.

I'll close by reiterating what I said in my previous post on this topic:

My point overall is this: as cool and interesting as it is, not all arguments need to be tied to the brain and cognition. And, in fact, some arguments are made weaker by doing so, since appealing to neuroscience is unnecessary at best and hand-waving at worst.

Figure out what your point is and talk about it. I'm guessing it actually has little to do with neuroscience directly.

Hearts and bulbs

Here's another great recent Savage Chickens comic by Doug Savage. I've commented on this great strip before, and here's another good "meta"-comic:

As we all know, hearts are symbolic of love (especially over the head), while lightbulbs over the head represent inspiration. I call "over the head" symbols like these "upfixes" since they are like affixes that are "up" (a term coined by my mentor, Ray Jackendoff).

Of course, as in the last panel, a heart and a lightbulb together show a love of lightbulbs, not a love of inspiration or ideas. This combination also does not give you inspiration about love either. What's interesting about Savage's observation here is that it nicely shows that you cannot combine the upfixes together. This is a first interesting trait: these upfixes have meaning on their own, but not in combination.

Related to this, the nature of the lightbulb upfix is to give you "value added" for its meaning. It no longer is just a lightbulb, but out of the relation above the head generates a new conventionalized meaning of inspiration. By adding the heart, it effectively removes this additive meaning, making the lightbulb simply a lightbulb once again.

The heart also changes meaning a bit as well. When the heart is an upfix, it describes the mental state of the person: The chicken is in love—with what, it doesn't matter. However, in the third panel, their combination makes the heart modify the lightbulb now—it's a love of lightbulbs—not merely reflecting a general mental state of the lover (the chicken).

Finally, this combination also changes the thought bubble. In the first two upfixes, the thought bubble mostly gives a depicted link between head and upfix. It doesn't mean thinking about love or about inspiration, but just reinforces these signs as being mental states. However, it is mostly unnecessary. The heart or the lightbulb would retain their meaning without the thought bubble. In the combination though, the bubble now returns to it's usual meaning as encapsulating thoughts. Having a heart and a lightbulb floating above the head wouldn't work as an upfix, nor would it work to convey thoughts.

Altogether, this simple, quirky comic tells us a lot about the structure of these types of signs!

Science and Eddie Campbell's rules of comics comprehension

In Eddie Campbell's recent article at The Comics Journal, he described several potential "rhetorical rules" that authors of comics can follow in order to make them more understandable to inexperienced readers. In a previous post, I tackled the broader issue of what factors go into limitations to understanding. In this post, I discuss what scientific research tells us about his actual stated rules.

Rule 1: All the information necessary to understand the drama of a sequence must be contained in every panel of the sequence.

This rule actually reflects a very "Western" dialect of visual language that I would claim is even stronger in European comics than American comics (though I don't have data to support that). In two separate studies (first, second) I coded comics from America and Japan for how many characters appeared in each panel (the second breaks down American books into Indy and Mainstream genres).

Overall, I found that American comics used far more panels showing multiple interacting characters than Japanese manga, which used overwhelmingly more panels of single characters or close ups. This would support that American books use more sequences following "Rule #1" than Japanese books.

This difference has an impact on comprehension. Being provided with only parts of a scene (single characters) forces you to infer the larger scene. This requires more machinery in the narrative grammar (what I call "Environmental-Conjunction"), i.e., the rules in people's heads that allows them to comprehend sequential images. Yet, this does not necessarily lead to poor comprehension. Rather, it simply reflects a different grammar along with the need for a different type of fluency. Neither is better or worse. Just different.

So, as described in my last post (and the comments), the problem isn't that sequences like this are "incomprehensible" in some "universal" sense, but rather that those that have difficulty with them either 1) lack fluency in this grammar, or 2) have a different set of patterns in their heads from being fluent in a different visual language (such as European VL vs. American VL vs. Japanese VL).

Nevertheless, for Campbell's purposes of whether this would help an inexperienced (i.e., non-fluent) reader: There is no data at this point suggesting that the "Western" way leads to easier comprehension. I wouldn't doubt that this might be the case though, because it forces less inference.

However, it is worth also considering that were an actual author of comics to change their dialect in this way, it may have an effect on experienced readers. I would bet that doing as Campbell suggests would actually have an adverse effect on the reading experience for a Japanese manga reader, and possibly for a reader of mainstream American comics. So, for an author considering "changing their dialect" to that of Campbell's, they may have to weigh these issues (and for which audience is intended).

Rule 2: Ordering of speech balloons and Rule #3: Speech balloons should follow a system that can be intuited and doesn’t need to be explained.

Campbell claims that "After reading the contents of one balloon, the eye is likely to go to the next nearest balloon, even if that balloon is in another panel and the eye has not yet taken in all the balloons in the current panel. "

Some eye-tracking studies give us insight on this...

First, one study found that balloon position did have an impact on how often people skipped over content. They found that panels were skipped if they followed a panel that had a balloon with a dense amount of text. Breaking apart that balloon into smaller balloons with less text lead to less skipping over panels. This implies that alterations to balloons can have an impact on reading behavior.

Another study compared the eye-movements of "novice" and "expert" readers as they navigated through comic pages. The inexperienced comic reader had erratic eye-movements across a page and focused much more on the text. In contrast, the experienced comic reader had a very deliberate order of reading, and focused on the images much more than the text. This implies that an experienced reader would not jump around to whatever balloons are closest, but an inexperienced reader might.

Thus, in this case Campbell's rules might, as is their aim, help an inexperienced reader.

Rule #4: Timing only exists in comics if the reader agrees to play the game.

Unless I am mistaken, this rule has to do with people who read ahead in a book so that crucial information is known before it's read in the narrative. This has little to do with the structure of the narrative, and has to do with people flipping through a book beforehand. Contrary to Campbell's claim, I don't see how it's any different with skipping ahead in a novel except that images show you content. 

I'd be curious for Campbell to expand here on just how authors should prevent readers from skipping ahead. The example he gives is fairly constrained and clever for preventing people from getting too much information by inadvertently reading ahead. How would this work for something like his example of "Magneto [showing] up surprisingly on the last page"?

It's worth mentioning here that how we interact with a comic as a physical object (flipping pages, accidentally looking ahead to the next page, etc) is different from our actual comprehension (the patterns in our heads that allow us to understand/produce sequential images). I don't think that Campbell confuses this issue, but it's a point worth remembering.

Rule #5: In a visual medium, a thing does not exist unless it is seen to exist

This rule applies to characters off-panel and indicated with the tail of a balloon, or to implied aspects of a depiction. Campbell notes in his example that we are to assume a character has his arms tied behind him because we don't see them (and it matches the context of the page), and that this is something that should be depicted. 

However, this rule applies also to the legs of the characters: No character on that page is depicted below the waist, yet Campbell doesn't have an issue about that except to say that having a panel where the whole figures shown (feet and all) would help provide good spacing (i.e., Rule #10: have a panel with feet on every page). He doesn't say necessarily that if not shown, we won't believe they have feet.

It is certainly the case that "undepicted" elements are part of the conventional grammar of panel framing. Studies of children show the ability to treat a panel as a "window" on a scene is correlated with experience reading comics. Comic industry lore also tells of interactions with indigenous people who did not read comics who wonder why figures in panels without legs "had no legs at all." Unless you fully lack some type of basic fluency in the visual language, then this shouldn't be an issue. 

Nevertheless, there is some validity in providing at least some notice of an element in a panel and not leaving too much to be inferred. People do track elements and concepts across panels. An element even subtly depicted once can then pervade inferentially across a sequence.

In contrast, another constraint on sequences aims at reducing how many elements need to be tracked across panels. Including too much information can lead to overload in working memory and can adversely impact comprehension. Thus, balancing these issues—what should or should not be shown—can be a delicate issue.

---------

So, Campbell's rules cover a wide range of issues. In most cases though, he is correct that inexperienced readers may have trouble with these issues. It is not clear though whether following such rules would help those people further comprehend sequences or if some of these alterations may have adverse ramifications on the reading experience of people who are actually fluent already. However, experiments could easily test these ideas...