Diagrammatic iconicity and morphological complexity: phonesthemes and kinesthemes in speech and gesture
More than 100 years ago, Saussure ( 1966) distinguished grammatical from lexicological types of sign systems. The more a sign system is relatively motivated, the more grammatical it is. Targeting a multimodal approach to grammar and grammaticalization processes raises the crucial questions of how body movements become types that may be meaningful and to what extent typified gestures may combine to form complex structures. Providing and applying a concept of gestural kinesthemes analogous to vocal phonesthemes allows for the conclusion that the same linguistic processes of typification and semanticization become manifest both in spoken language and in the gestural modality (Fricke 2010, 2012, 2014).
Why is the Peircean term ‘diagrammatic iconicity’ crucial for this kind of usage-based approach? And to what extent are phonesthemes and kinesthemes iconic? Contrary to onomatopoeia, like the classic birdcall example cuckoo, a direct similarity relationship between form and meaning is not necessarily given in diagrammatic iconicity. The similarity exists rather between the relations of forms and the relations of meanings. These units are called “submorphemic differentials” or “phonesthemes” (Firth  1957; Bolinger  1975; Zelinsky-Wibbelt 1983). They are characterized as usage-based, intersubjective sound-meaning correlations, as found in the correlation between rhyme and meaning in the English monosyllabics bump, chump, clump, crump, flump, glump, grump, hump etc., which all have the semantic feature ‘heavy’ in common (Bolinger 1975: 219; Zelinsky-Wibbelt 1983: 22).
For the purpose of illustration, let us consider the English example smog following Zelinsky-Wibbelt’s analysis: The phonestheme -og constitutes the rhyme of the word forms fog, bog, clog, hog, jog, log, and slog. The corresponding phonological unit is semanticized with the meaning ‘heavy’ (Zelinsky-Wibbelt 1983). Similarly, the phonestheme sm- at the onset of the word forms smoke, smear, smirch, smirk, smudge, smut and smutch is associated with the meaning ‘dirty’. Combining both phonesthemes results in the phonological form smog (represented by the written word form) on the form level and in its twofold semanticization of ‘heavy’ and ‘dirty’ on the meaning level. Here we have a case of semantic compositionality: Comparing the meaning of smog with the meaning of fog, the meaning of smog may be paraphrased as ‘heavy, burdensome fog that is dirty’ (cf. Zelinsky-Wibbelt 1983).
Providing and applying a concept of gestural kinesthemes analogous to vocal phonesthemes supports and further elaborates the hypothesis of a “rudimentary morphology” in co-speech gestures (Müller 2004) as part of the concept of gesture families and, moreover, allows for the conclusion that the same processes of typification and semanticization become manifest both in spoken language and in the gestural modality (Fricke 2010, 2012, 2014).
Computational approaches to studying iconic co-speech gesture
In this talk, I will present several approaches to study iconic co-speech gesture using computational methods. This will include (1) analyses of crowd-sourced rating data to investigate the multi-functionality of gesture, (2) perceptual experiments with synthetic gesture generate with embodied artificial agents, and (3) computations models of the production of speech and gesture to unravel the underlying cognitive processes and mechanisms.
Constructing Meaning at the Interface between Sign and Gesture
Sign language and gesture share the same visual-gestural modality. Therefore, signers – like speakers – cannot only use co-speech gestures accompanying utterances but also – unlike speakers – systematically integrate manual and non-manual gestures in various ways into utterances. A particularly interesting example for the systematic interaction of gesture and sign language is constructed action (i.e. action role shift), which has been the focus of much debate in recent literature on meaning and modality. In this presentation, I'll discuss various examples of gestural meaning components that are integrated into the proposition expressed by the utterance and show how recent accounts of action role shift (Cormier et al. 2015, Davidson 2015, Maier 2015, Herrmann/Pendzich 2018, Schlenker to appear-a,b) can explain the modality-specific interaction of gesture and sign.
Where is iconicity and why isn’t there more of it?
More and more studies show that spoken and signed languages harbor a considerable degree of iconicity, form-meaning mappings that are motivated by resemblance, such as the word “bang”, which sounds somewhat like an actual bang. In this talk, I will consider iconicity as a graded quantity, with certain words being more or less iconic. Rather than asking the question, “Are languages iconic or arbitrary?”, I will be asking, “Where is iconicity?” Using English iconicity scores from a rating study, I will show that iconicity is ramped up in children’s speech and the child-directed speech of adults. I will then show that iconicity is also heightened in perceptual language, in particular for sound words (“squealing”, “beeping”, “rustling”) and touch words (“rough”, “smooth”, “prickly”). These analyses of the distribution of iconicity within language also help us understand why languages are characterized by a lot of arbitrariness. In particular, I will argue that it is precisely the connection between iconicity and perceptual semantics which restricts its domain of use. To this end, I will report new quantitative evidence which shows that iconicity is inimical to abstraction in language.