In this interesting talk combining cognitive linguistics and cognitive semiotics, our MA student Daiya Kuryu will present ongoing research on "multimodal constructions" involving language and gesture, with focus on a particular English idiomatic expression: "back and forth". All are warmly welcome to H402, and if not possible, to the usual zoom link!

Considering the meaningful information conveyed by non-verbal elements in face-to-face interaction, usage-based linguistics has increasingly focused on the multimodal (and polysemiotic) instantiation of constructions and their conventionalization and entrenchment as holistic multimodal units (Zima & Bergs, 2017). However, much of the nature of such “multimodal constructions” remains unclear, and therefore more research should be conducted both from theoretical and empirical perspectives. 

To this end, I report on a study that investigates the English expression “back and forth” and its accompanying gestures, using the NewsScape 2016 gesture-annotated corpus of Red Hen Lab. Out of all the 295 instances of “back and forth” where the speaker’s hands are visible on screen, 215 (72%) were the cases in which the construction was accompanied by a gesture with bidirectional movements. These bidirectional gestures were then analyzed in ELAN with regard to the following parameters: articulator, movement type, movement axis, action phase (segments in a stroke; see Hinnell, 2018), hand shape, and hand orientation. The result exhibits some recurrent gestural patterns, each of which has invariant features (e.g. regarding movement type and movement axis) and variant features. While those recurrent patterns are seen as regularity (Mittelberg, 2019) of the multimodal-polysemiotic construction in question and as reflecting a prototype structure (Cienki, 2017), they all preserve schematicity (Mittelberg, 2010, 2018) to the extent that they are the externalization of the SPATIAL OPPOSITION image schema (e.g. UP/DOWN, FRONT/BACK, LEFT/RIGHT), in which the meaning of the linguistic construction “back and forth” is conventionally grounded. I will finally discuss some implications for cognitive semiotics.

Cienki, A. (2017). Utterance construction grammar (UCxG) and the variable multimodality of constructions. Linguistics Vanguard 3(s1).

Hinnell, J. (2018). The multimodal marking of aspect: The case of five periphrastic auxiliary constructions in North American English. Cognitive Linguistics 29(4), 773–806.

Mittelberg, I. (2010). Geometric and image-schematic patterns in gesture space. In V. Evans & P. Chilton (eds.), Language, cognition, and space: The state of the art and new directions, 351–385. London: Equinox.

Mittelberg, I. (2018). Gestures as image schemas and force gestalts: A dynamic-systems approach augmented with motion-capture data analyses. Cognitive Semiotics 11(1).

Mittelberg, I. (2019). Peirce’s universal categories: On their potential for gesture theory and multimodal analysis. Semiotica, 228, 193–222.

Zima, E. & A. Bergs. (2017). Toward a multimodal construction grammar. Special issue. Linguistics Vanguard 3(s1).