SYMBOLIC INTERACTIONS:TOWARDS A COGNITIVE SCIENTIFIC THEORY OF MEANING IN HUMAN TECHNOLOGYINTERACTION

Information technology has perpetuated the role of symbolism in everyday life practice, through its reliance on sign systems for its creation and operation. Increasingly attention has been placed on applying semiotic techniques to analyze user interface design and usability. Surprisingly, although the move towards symbolic interaction has been one of the most striking components of the digital shift, it has proven difficult to build bridges between semiotics and HTI-design thinking. In this article we argue that the problems in linking semiotic analysis of human technology interaction with modern HTIdesign paradigms such as usability or user experience arise from a theoretical gap between the paradigms of semiotics, human cognition and thinking technological psychosemiotics. Consequently, it is necessary to reformulate principal insights of semiotics such as the triangle of reference, by replacing the intuitive concept of thought, with modern philosophical and psychological concepts of human thinking. This allows the unification of usability research based on cognitive research with the analysis of signs in modern semiotics. It is possible to unify the conceptual analysis of signification and semiosis with conceptual and empirical work typical to modern human technology interaction research and design, thus making semiotics an essential tool within the swiftly developing paradigms of interaction design. Indexing terms/


INTRODUCTION
Modern existence is characterized by the computers we use everywhere and every day, from the kitchen to the office, streets, hospitals, shopping malls and subways.Computers are symbol-processing machines and interaction with these machines is based on signs [1,2,3].This quality is only becoming further entrenched by the elimination of physical components such as buttons, knobs and levers.In other words, we are dependent on the recognition, understanding and correct application of symbolic interpretations in order to perform even basic functions, such as to check the calendar.For this reason, semiotics can provide a contributive theory-language and framework for instrumental factors of humantechnology interaction (HTI) research and design [3,4,5].
Signs and their systems form a necessary precondition for all communication practiceinterpersonally, as well as interactively between people and technical artifacts.This is why, systematic examination of how signs and their systems operate in HTI situations is useful, as it reveals the relationships and connections people make between expressed components, such as images, text, sounds and meaningboth physical (function, operation) and meta-physical (values, hierarchies, associations).Due to long traditions in semiotic research, this field of scholarship has a lot to offer in terms of theory language and understanding for HTI research and practice.
In order to integrate semiotics into general HTI research and design discourse, it is essential to consider its functions inside this multidisciplinary field.Intuitively, it is evident that current HTI research can benefit from more frequent implementation of semiotic concepts and methods when investigating interaction issues.This matter has been brought to the fore by scholars such as Clarisse Sieckenius de Souza (semiotic engineeringuser interface design), René Jorna (artificial intelligence and cognitive science) and Mihai Nadin who sought to rigorously analyse and develop the work of Charles Sanders Peirce, in the field of computation design.In the field of cybernetics, Max Bense, one of Nadin's main influences, set forth to explain aesthetics and creative processes through applying rigorous semiotic analysis translated into mathematical formulae [6].
Despite these significant contributions, there is no general agreement and application of semiotic principles describing HTI from both a practical usability, as well as experiential perspective.Thus, what we suggest is that rather than concentrating on applying semiotic analysis to purely usability and design issues, broader concerns may also be analysed through semiotics.The study of signs and the way they operate on a phenomenological level in HTI may additionally be applied toergonomics, user experience (UX) and emotional usability [7,8,9,10,11,12].Additionally, approaches such as crosscultural interaction research, as well as designing for life and life-based design [13,14,15,16] may be included in this scope.This demonstrates HTI design as being a complex and multidisciplinary field, which in itself possesses numerous combinations of signs and sign systems (inter-disciplinary languages).Along with defining the role of semiotics in HTI comes the necessity to reconsider some foundational issues in semiotic thinking.The goal of such foundational considerations or analyses is to gain conceptual clarity when observing the relationship between semiotics and user psychology, as well as other HTI relevant fields of learning [4,17,18].

SIGN FACILITATORS OF SYMBOLIC INTERACTIONS
The function of signs in HCI and HTI is to connect the human mind with design syntax and machine operations.As seen in the development of communication technology such as the telephone (see Figure 1), interactions have shifted from the embodied and intuitive, form follows function (ergonomically the chord remains closest to the base meaning that the mouth piece in the receiver is located at the chord end), to symbolic interactions.This is why there is the need to focus on addressing the matter of how human mentality is connected to signification, and not just from the perspective of the graphic user interface icon design.Interpretation of signs is interdependent on the context (environment, use, social, cultural), the person, and the signs' application.To understand how interpretations of signs are constructed and operate in HTI, we need to know how they operate in human cognition and processes such as perception, categorization, and thinking.It is only after gaining clarity into these issues, that one can ask questions relating to how signs support the human operation of computers.

Fig 1: Telephones -from form follows function to symbolic interactions
Another set of problems is linked to our own theoretical concepts [17].In order to solve the above mentioned questions it is necessary to maintain a sufficiently expressive theory-language.This language must provide interaction researchers and designers with the possibility to problematize and resolve related issues in a sophisticated and elaborated manner.In order to advance the application of semiotic thinking in interaction design and research, the mental metaphysics of signs in the human mind must be explored.This kind of research is based on conceptual discourse, which applies philosophical argumentation and analysis to the explicit and tacit presuppositions of any research field.Here, this is referred to as foundational analysis [17].

Technological psychosemioticsa historical perspective
Interaction processes vary, yet, the basic logic of all technology, digital and non-digital, is the same at a higher conceptual level.Technology has traditionally been created to solve problems, to be operated and used as tools during the course of human living and development.Thus, they are designed with the function in mind, which in turn explains partly about their form.Another component within the syntax of technology is the point of interaction, the user interface, the handles, buttons, levers and now more than ever, icon symbols on touchscreens which (should) indicate how the user should operate the deviceforming a connection between user and function.
In terms of cognitive science, the role of symbols in not just interaction, but information processing, or computation, has roots in traditions stemming from early discussions on artificial intelligence (AI).In these discussions it was supposed that humans, or the human mind was similar to a computer, and when they were interacting with other machines (real computers), they were feeding information into, and obtaining it from a black-box, for instance (see Figure 2).Typical examples of this machine in early cognitive science discourse include the Turing machine [19], the von Neumann [20]machine, and Minsky's [21]M-Machine.They were all conceptual models to explain, not just information technology, but any tool from scissors to cars.Minsky's [21]much cited M-Machine can be used to exemplify this level of thought.

Fig 2: Adaptation of Marvin Minsky's Black Box Machine (1967)
The M-Machine was imagined to be a finite state machine or an automaton [21].Many liken it to the Turing Machine [19,21,22].As Minsky [21] points out, the M-Machine is a description of any technological artifact in existence.It entails all the abstract properties comprised by the machines that people use on a daily basis.This means that when we develop a user interface, we already understand the fundamental logic and basic questions of the interface's development.
The M-Machine has a finite set of distinguishable states in each moment of time.These machine states can be analyzed in terms of input, output and internal states, which are combinations and compositions of physical signs.All these states can be manipulated by either human operations or machine functions.Any two-machine states differ from one another as they entail different signs.Interaction-wise, input and output states are decisive, as these are the points of contact between the machines and human users.Interaction is an activity in which the user inputs a set of signs into the machine and the machine provides feedback for the user through output peripherals such as the screen.The output and its interpretation are influenced by the physical, social or information environment within which it is generated, as is the effect or functional outcome of the M-Machine.Often the users directly follow these effects and use that information to control and steer the technical artifact.The problem within the finite state model is, however, that when moving beyond the behavioral levelfurther than simply matching user interface interaction to functionand towards the psychological level, the combination of the above mentioned factors (i.e.contextual, social etc.) mixed with previously learned knowledge and experience, opens up a universe of possible interpretations or mentally constructed signs.Or, "a field of infinite substitutions" [23].
Signs are crucial in enabling people to communicate explicitly and consciously, or tacitly and ubiquitously with the machines.The feedback is based on signs such as meters or messages [24].Peirce's [25] basic sign theory, entails that signs comprise three core elements: the signifying element (also referred to as sign, symbol or sign-vehicle); the object (that to which is being referred); and the interpretant (the mental representation or meaning of the sign).In turn, Peirce's early semiotic accounts claim that sign-vehicles or representations operate in one of three ways: 1) by sharing a similar quality to the object (icon); 2) by demonstrating a relationship to a specific fact about the object (index); and 3) by connection through an imputed character and arbitrary agreement (symbol) [26].
We encounter examples of all of these sign-vehicles on a daily basis during mobile phone use for example.Pictorial icons in graphic user interfaces, are an interesting example where icon type sign vehicles have transformed into symbols.An example of this is the alarm clock icon, the traditional telephone headset for phone calls, or the Youtube television in iPhone displays (see Figure 3).These icons were iconic in terms of referring to the concrete syntax of objects in which form followed function.Yet, these icons now exist as symbols, particularly to users who have never used the analogue versions of alarm clocks, telephones or televisionsas these now exist as functions accessed via icons on the touchscreen.

Fig 3: Sign-vehicle types -from icon to symbol
On the written level, keywords, programming language, and text in general comprise a relationship between the symbol and the function that is held by arbitrary agreement.
The third type of sign-vehicle, an index, maintains a causal relationship with that to which it refers.Indices in particular may be artificial (designed) or natural.Their form is guided in response to the phenomena they indicate, for example, the meter reading on a speedometer or thermometerthese rise and fall in accordance with physical changes.Natural signs, in turn also reflect physical changes in the environment such as snow indicates cold temperatures, smoke indicates fire.HTI is not limited to human-designed indices, as natural signs are also used in situations such as those where airplane pilots are approaching airports and use landmarks and features such as forests and coastlines to guide their course [25,27].
The examples of signs and sign vehicles mentioned above served a practical purpose, and correspond with interpretations made in the field of interaction design and semiotic engineering [3,4].However, one problem which arises relates to whether the theoretical conceptualization for semiotic processes is effective when conceptualizing and analyzing interaction.That is, theoretical conceptualization of semiotic processes should go beyond the behavioral and functional level, towards how these processes operate on a psychological levelwhat occurs when the human user encounters or constructs (programmes) sign-vehicles (language), how are these processed, and what are the outcomes in addition to smooth operation or usability?Furthermore, it is also essential to consider the best way of building links between other mental-relevant fields of learning and semiotic analysis.This approach we term as technological psychosemiotics.For this reason, mental contents is discussed further on, in order to explain the mental component of these semiotic processes.

Signs and their referents
A basic concept in semiotics is meaning, i.e., the relation of sign to what it stands for or represents [28].Meaning is not a simple concept and there is no clarity as to whether meaning is supposed to refer merely to the relationship between symbol and referent, or additionally to all phenomena which enables meaning-making or semiosis, i.e., the human ability to generate interpretations of symbols, and to draw symbolic connections between random phenomena.The former viewpoint might be called sign-based semiotics, and the latter concept or thought-based semiotics [29].
In terms of sign-based semiotics, the meaning of a symbol arises from its relation to its referent (object).The word "horse" has meaning as it refers to a horse through a linguistic system of arbitrary agreement and conventions.Sign-based semiotic thinking has been explicitly or implicitly prominent in the discipline of traditional behaviorism and formal semiotics, such as game-theoretic, model-theoretic and Montague semantics, in addition to related semiotic discourses [30,31,32,33,34,35,36].A solution typical to sign-based semiotics has been, for example, to identify propositions with the definitions of their truth-conditions in logical semantics [32,37].This has led to the study of word and clause meanings as phenomena that are autonomic and independent of mental activity.
Sign-based semiotics can be used to analyze the meanings of signs in technical artifacts, this exists similarly to areas such as semiotic engineering in which sign recognition and activation are followed by function (user to computer), and vice versa (computer to user) [3,4].Bits and their symbols in a computer's memory maintain direct reference to what they stand for, via means of a programmer's constructed functional interpretationthe interpretation physically represented and fed into the system.A combination of ones and zeros in the output or input and internal memory stores of an M-Machine for example, refer to particular states of affairs.Thus, by using the M-Machine as a semiotic reference, we may observe that a particular combination of signs results in a particular external state and vice versa.
The definition of meaning is mechanistic in sign-based semantics.It is assumed that signs and references are somehow automatically linked to one another.The core problem with sign-based semantics in HTI-design is that it does not account for users' individual interpretations of signs.In formal systems, the meanings of signs must be associated with real world entities, but these conceptualizations exclude discussion on how associations take place.
This neglect of interpretation has its consequences: in truth conditional thinking and other sign semantic approaches it is easy to forget that meaning precedes truth.If we do not know what a proposition or sign means, it is difficult to say whether or not it is true.Therefore, it is possible to define truth conditions only when the meaning is known.The proposition "Mary likes ice-cream" is true, if the Mary the sentence refers to likes icecream.However, if there is another Mary who does not like ice-cream, it is necessary to know which Mary is being referred to.The meaning of "Mary" must be known, before it makes sense to speak about truth conditions.This similarly applies to HTI, whereby technology and various services on offer only exist if they have semantic value, and this value is dependent on familiarity, purpose and motivation.To take Facebook as an example, the term is familiar to millions globally.It is generally known as being something that exists online or through computers, and many families and friends discuss the contents of Facebook (photographs, messages and "status updates").This is one level of meaning.The embodied application, its design, operation and online presence is only encountered when accessing it via an IT device.Even this encounter is not fully embodied until relationships are established between the user, the system and other users.Thus, the user learns how to enter the system and establish a profile and then connect with other userspeople who are generally already known through real world relationships.Without these connections, attachments and motivation, even Facebook is rendered meaningless to the user and only serves to reference the semantic value of the name: face, a part of the human anatomy; or a book (Fig. 4).

Fig 4: The levels of meaningfrom words to system
Another type of problem arising from the application of sign-based semantics to the study of HTI, which is similar to the previous example, relates to misrepresentation.On the programming level, both naïve (or inexperienced) users and programmers can see the same computer interface, but only programmers (professional or amateur) can create computer programs that make sense.Standard interface tools such as keyboards and touchscreens entail different worlds of signification depending on the user's skills and associated mental representations (memories and impressions of the operating logic).A string of letters such as "public void" is an important sign for a C# programmer, but maintains no significance for a layman or novice.From the layman's perspective it is not a sign, because no meaning can be derived from it.Similarly, icons can be of significance to some people and meaningless to the others, even though their sign value remains the same (see Figure 5 of the "meaningless" Adobe CS5 icon pack).

Fig 5: Adobe CS5 "meaningless" icon pack[38]
Sign-based semantics forms thus a basic level discourse for analyzing meaning.This provides a solid basis for technical discourse as one can suppose that the meanings of signs are determined, when they are defined for programs in light of specific user groups.From a practical and technical perspective, it is often taken for granted that the meaning of a user interface icon for example, is housed within the artifact.In other words, the icon's meaning is often mistakenly understood as being the operation to which it refers.Thus, icon meaning equals truth of function.However, as seen in Figure 5 and in reference to Peirce's [25] three types of signifying-vehicles, there is more to this phenomenon of meaning.In particular, it is important to additionally address the problem of abundance in symbols and reliance on arbitrary agreement.As demonstrated in the Adobe icons image, this poses obstacles relating to too much to learn and remember, generating a loss of significance through abundance itself and influx of empty signifiers.
If the focus is on why people cannot use a technology or why they use it mistakenly, explaining meaning in terms of knowledge of icon-function relationships is simply not enough to provide an adequate conceptual basis for HTI research.It is thus essential to consider how the human mind attributes or fails to attribute meanings to symbols.The same sign can mean very different things to different people.In order to explain the differences it is necessary to consider meaning and respectively semiosis as a mental phenomenon.

Signs, concepts and thoughts
Concepts and respectively thoughts give meanings to words and signs.They are thus basic elements in attributing meanings to more complex representations such as sentences or stories [29,39].Assuming that concepts are central in meaning-making processes, sign-based semiotics can be extended by concept and thought-based semiotic thinking [29].Whenever we are dealing with language or any other semiotic system, there is the need to additionally discuss the characteristics of underlying concepts and thoughts.
In these discourses the meaning of a sign does not directly arise from its reference, rather it is mediated by conceptual representations [29,40,41,42].Communicative expression must be presented with the assistance of sign systems.Yet, in order to understand the meaning of specific expressions one must be familiar with the concepts and thoughts to which the signs refer [29].
In terms of an M-Machine, this means that signifying-vehicles for input and output must be interpreted.Interpretation means associating the signs with the respective concepts that can operate as interpretants, which in turn attribute meanings to signs.Thus, an input vector or sign (0000111 or the alarm clock icon) can refer to the concept of a "bell" or alarm and respectively activate auditory output.The causal relationship between the input referring to "bell" and the output of a sound somewhat resembles semiotic causal relationships between signslinguistic, environmental etc.and human behavior.Therefore, often it may be mistakenly understood that the input command for "bell" possesses meaning for the machine (or the M-Machine).Yet, in fact, the mechanical operations that occur in response to the input, i.e. the output of the sound of a bell, possess meaning for the user.Quite similarly, if thinking of the human body as a machine, symptoms or abnormal functioning of the body may be associated with an illness.Thus, the concept of an illness attributes meaning to the symptoms.
Concept-based semiotics has derived much of its current form primarily from research inspired by transformation grammar Chomsky [43,44,45].Many researchers have developed different concept-based and thought-based systems of semiotics [46,47,48,49,50,51,52,53,54,55].Initially, concern was only placed on deep-structures and inborn instincts of thought and systems of projection rules.These were said to illustrate how clause meanings are produced from deepstructures [40,43,52,56].However, as a consequence of concept and thought-based semiotics it has become clearer that language and other symbolic representations are expressions of thought.
According to concept and thought-based semantics, when people are familiar with the relevant concepts of particular signs it is possible to attribute sense-making interpretations.The relationship between signs and references is not simple or straightforward.Rather it is arbitrary and bound by conceptual interpretations.Meaning is thus contingent upon how the sign is bound to concepts.Any sign can be freely bound to any concept, which entails that signs derive meaning through arbitrary agreement instead of objective psychological and socio-cultural processes.
The main problem in concept-based semantics is that the origins of concepts and thoughts remain open.Interpretants are intuitively understood as given.Consequently, some important HTI problems in semiosis are excluded from discourse.Interaction with technical artifacts presupposes expertiseknowledge of the semantic value of sign-vehicles predetermined by the artifacts' developers.
Another problem is posed by the alteration of meanings regarding e.g.icon-function relationships and even meaning imposed by external factors such as cultural environment, social discourse, technological trends and values etc.Its origins rest in the human mind and cultural context, and to explain why people behave differently at various points in time, while varying in their interpretation of the same signs, it is essential to think about the role of the human mind in semiosis.To Democritus the concept of an atom was different to that of Dalton as well as to concepts held by physicists today.Such changes can also be observed in sign coding systems.The interfaces of computers have transformed from text based to iconic and are now are ever more encompassing multi-modal interaction methods such as voice and gesture activation, yet, the basic interaction model of symbolic exchange between person and machine still remains same.The practice of this model of symbolic interaction which runs throughout history must not be confused with the meaning, for the meaning of signs evolves over the centuries, forming new associations and understandings corresponding to changes in culture and society.
The notion of concept and sign-based semiotic thinking provides researchers and designers interested in the semiotics of HTI interpretants with a far more objective analytical tool of signs.However, it does not yet give a clear account regarding the basic cognitive principles and origins of sign representation within the mind.By nature, HTI research presupposes the analysis of interpreters, i.e., the human mind.This presupposes conceptual analyses of the mind.

REPRESENTATION IN THE NEURAL MIND
The core notion of modern psychology is mental representation.A mental representation is information about some state of affairs in the human mind [57,58,59].Representations, or in Peirce's [25] terms interpretants, are signs which stand for something else in the human mind.In contemporary theory of mentality, mental representations are a key concept in terms of explaining the pragmatic relationship between real world phenomena, thought and behavior.For instance, mental representations explain why people have the capacity to use languages [46,54,60].This is why mental representations also provide an explanatory basis for semiotic phenomena and semiosis.Mental representations are different from external representations.A painting or a technical application, for example, exists physically, objectively, explicitly and independently of human mental representations.More than one person has access to these products (material representations).People have the mental capacity to attribute various information contents to environmental objects and events.However, mental representations on the other hand, only exist in the human mind and external signs acquire meaning purely when they are mentally represented.
There are two core aspects of mental representations which have seldom been discussed.Firstly, mental representations rely on the human nervous system, which is a necessary condition for semiosis in the human mind.All mental representations exist as information housed within the human brain.Information exists in brain states, which are formed and maintained via neuralrepresentation vehicles (the neural version of Peirce's [25] sign vehicle).The elasticity of the nervous system enables it to operate as a representation vehicle.Information can exist in different states, which can be attributed varied meanings.Thus, it provides the possibility to represent differences between various meanings.Any two different mental representations obtain their individual meanings through variations in brains states, and associated emotional connections.
Neural semiotics opens up an interesting perspective on the semiotics of HCI as it enables a biosemiotic perspective [61].Biological and non-biological material states can be transformed into representations in an M-Machine via an indicated relationship with an external situation or object.Situations in which biological and non-biological cognitive processes can be seen to alter, particularly in light of context and emotional associations generated by the combination of the context, human mind and technology are readily seen regarding e.g.ubiquitous computing [62].In instances of ubiquitous computing it may be observed that the HTI itself remains constant, i.e., pertaining to the user, the device and the action.However, the contextual change alters the meaning and associations generated during the interaction.This occurs both biologically, i.e. relating to other sensory stimulation (sound, smells, tactile factors, vision and maybe taste), perceived physical threats and levels of awareness regarding the context (in a busy subway station for example and user may need to be aware of other people's positions), and other information, and non-biologically.The non-biological refers to deep thought and higher level cognitive processes which are engaged in the usage of the technology and appreciative processes often referred to as user experience, within the interaction situation [63].
An essential component of the mental is that the material grounding can be organized into separable states, which allow brain states to be associated with signs in a systematic way.Only an adaptive and plastic material organization can serve as the condition for the potential to distinguish and discriminate, which is a condition for all meaning.This basis has been brought out very clearly in classical structuralism [64].Indeed, it is possible to construct the analysis and design of a number of HTI processes based on these kinds of biosemiotic grounds.One can call this kind of thinking brain-computer interaction (BCI) research.BCI interfaces present great possibilities to aid HCI in cases where people have limited to no limb function.BCI is a tangible example of how mental representations (signs), facilitated by brain states, enable the operation of information technology [65,66,67].
Yet, even in light of the evidence that thought, and thus mental signs, are enabled via the nervous system in brain states, there is still much more to explain regarding semiotic interactional processes.Consider, for example, children who initially learn a language e.g.Swedish at an early age, then several years later move to Finland.After living in Finland for several years they notice that they can no longer communicate in Swedish, and that Finnish has become their native language.
By purely studying brain states it is impossible to distinguish whether the new language of the children is Finnish or French, for example.The explanation for a child using one language instead of another, or one code instead of another, cannot be found in the brain but rather in their interaction with their environment.The explanation must be based on the contents of learned information.The role of mental contents in semiosis must be discussed separately.

MENTAL CONTENTS
Mental representations comprise information contents, i.e. mental contents [68,69].Mental contents, or information contents which contribute to meaning generation, pose a secondary aspect of mental representations [57,59,68,70].This is especially important to remember when analyzing all semiotic phenomena such as learning and interaction processes, in addition to considerations for context and environment.The brain cannot explain or predict what human social, physical and information environments are like and for this reason concepts of mental contents are essential when analyzing and working with the problems of technical semiosis.Semiotic thinking, in which the analysis and argumentation of meaning is based on the concepts of mental contents can be called content-based semiotics.M a r c h 2 0 , 2 0 1 4 In content-based semiotics, the meanings of signs are defined by concepts of mental contents.The meaning of any sign must be seen in terms of the mental content it evokes in the minds of the people who use the sign.When we explain how a person has used a sign or responded to it, it is essential to investigate what kind of mental contents has been activated in the mind of the person.Only this knowledge gives a clear understanding of the sign's meaning.The meaning of a word, for example, is embodied in the way it is used [71].The only way of understanding how a word is used, is to indicate the associated mental contents.This means that we gain an understanding of the mental context of the word usage, through ascertaining factors such as memories, associations and emotions attached to the situation to which the person is respondinge.g.their past, present or anticipated interactions with an iPhone 5C, for instance.
Content-based semiotics is related to modern cognitive semiotics.The field of cognitive semiotics has been developed for example in the context of film theory [72], and music [73,74].Nevertheless, content-based thinking extends the traditional cognitive thinking in a number of respects.Firstly, it takes mental contents as its explanatory basis [68,69].Instead of semantics and meaning of representational entities, attention is directed towards the mental contents of specific expressions.This is similar to the relationship between semantics and thoughts behind expressions.While semantics examines linguistic expressions for their ability to reveal thought structure, content-based semiotics focuses specifically on the information content of those expressions [68,69].Secondly, content-based semiotics in HTI accepts emotional and other dynamic phenomena as a premise for analysis and argumentation.Thus, the point in question is not only cognition, rather, the experiential function of the mind as a whole.
Semiotically, a crucial characteristic of mental representations is the connection between the sign and the composition of mental contents a sign activates in a particular situation.If there is no-one to attribute mental contents to a particular sign in a specific situation, the sign is meaningless.To have meaning is to have corresponding mental contents for that to which the sign refers.Only by being able to present the mental contents associated with a sign in a specific situation, can one make sense of and interpret signs.As Wittgenstein [71]argued, the meaning of a sign is how it is used.No sign can have a fixed meaning.Meanings vary from case to case.However, this does not mean that the meanings of signs would be totally arbitrary.Rather this means that signs can have multiple meanings and therefore it is impossible to attribute fixed meanings to signs.For instance, one specific meaning or interpretation of a sign may be appropriate and correct for one situation, yet inappropriate or incorrect in another.
Wittgenstein's thinking was not based on mental representations and therefore the source of multiple meanings rests in the fact that one sign can be associated with many different types of representational contents.The word "system", for example, can have numerous meanings such as computer program, machines, socio-technical organization or even the way people are connected to technologies.This is why mental representations of signs and their mental references differ on the basis of rational semiotics.In HTI this kind of contents-based semiotics plays its own important role.
Mental contents provide keys for solving the problems of misrepresentation.People may misrepresent the meaning of a sign.In a user interface, an icon may be unknown to a user rendering them unable to attribute it with meaning.It may also be that a person allocates false meaning to an icon and thus uses it incorrectly, or for purposes outside the designer's intended usage scope.This kind of phenomena can only be understood by analyzing the contents of mental representations people possess at the time of usage.Misrepresentation arises when people attribute divergent mental contents to the signs they engage with.Thus, content-based semantic argumentation is based on the mental contents of situational representations and associated signs [68,69].When considering technical interaction, it is essential to always investigate the contents of mental representations and their emotional connotations.This is afforded once we understand how people represent the given signs.In other words, in order to understand mental representations and emotional content, one needs to understand how people represent specific signs, and in turn, what mental contents (information) are allocated to these specific signs.
Content-based semiotics provides sufficient grounds for the semiotic analyses of HTI.Sign-based semantics does not discuss concepts and thoughts as explanatory grounds for semiotic phenomena.Thus, its conception of semiosis is mechanical.It provides one necessary condition for HTI-semiotics analyses, i.e. sign and its reference to referent.Without this knowledge it would be impossible to speak about semiosis.However, it is equally essential to consider concepts and thought which explain and interpret the meanings of the signs.Yet, this is not sufficient, because the origins of thoughts remain open.One can get an idea about what is the "right" meaning of a sign, but this is not necessarily its real meaning.The variation of uses between people at different points in time are neglected from discourse and for this reason it makes sense to consider the actual uses in terms of mental contents.

Explicating mental contents
When people use an M-machine, they mentally represent its input, output and internal states.The mental contents of these representations are critical.An M-Machine and its states appeal to people as they are mentally represented.A novice has a different idea about the way technical artifacts work, or the states they are in at a particular moment, in comparison to experts.Understanding the differences presupposes explication of the contents of relevant mental representations.This is possible only by using proper research methods, which are often more psychological and sociological than semiotic.
In HTI research, it is essential to consider knowledge operationally.With respect to mental contents this means that there is some clear method of how mental contents can be investigated.The most obvious method is to ask people about their experiences.They can give verbal responses to questions and interviews, for example.On the grounds of such qualitative information it is possible to understand relevant mental contents implicated in the interactions [69].
Additional methods are numerous.Protocol analyses can give information about how users have understood the meanings of the signs [75].It is also possible to investigate documents such as computer programs, art works, and texts to study how signs such as words or imagery have been used.This kind of knowledge enables explication of the means for examining mental contents, and what types of meanings people attribute to different forms of signification processes.This knowledge allows for the possibility to explain the various ways in which signs are interpreted.Further, it also alludes to the ways in which the respective mental contents are attributed meaning by people in various types of situations.
Nevertheless, explication methods need not be qualitative.It is possible to use multivariate techniques such as factor analyses of multidimensional scaling to investigate the contents of subjects' conceptual systems [76].In surveys, the questions contain contents and in the answers people provide information about how the different items relate to their own mental contents.Therefore, the combinations can be used to understand what kinds of contents (e.g., emotional contents) are clustered into higher level mental representations.
The mental contents associated with the use of signs can thus be investigated by means of methods typical to the social sciences and human research.There is nothing mysterious in investigating the meaning of signs in content-based terms.One only needs to focus on the use of a sign and the relevant mental contents.Accurate definitions of the actions, the actors and the context of the sign usage are needed in order to be able to explicate relevant mental contents and thus the meaning of the signs in any given interaction situation.

Technologies as semiospheres
The origins of mental contents are rooted on the one hand in the human brain's capacity to acquire vast and diverse meanings.Brains have specific areas for processing language and meaning, analyzing sensory information, constructing emotions and making plans [77,78].On the other hand, mental contents depend on physical and environmental information.Through learning, representations and their components such as concepts, schemas or mental models with specific mental contents are established.
There is no real limit to the information environments that people are surrounded by.They can be large such as cultures, countries or professions.They can be small such as family communication with its privately shared experiences.In these endless interaction contexts people create the information contents of their concepts and thoughts.These learned mental contents structure the experiences of signs.
The contexts are not unbound.For example, learning to read medical symptoms and recognizing their sign values is a highly specialized and grounded act.Signs indicate specific conditions, which comprise one or a limited number of possible interpretations.Thus, the contextual function is to act as a basis upon which people organize thoughts and signs into meaningful relationships.This forms domains of discourse or language games [71].The contents of signs are defined in these discourses so that one and the same sign can have different contents in various kinds of discourses.Beginning with Lotman's [79,80] well-known term for describing the relation of sign to its use and environment, one can also say that discourses that define the meaning of a sign are semiospheres.From a content-based perspective, the main environment is in fact information itself.The physical or biological environments are coded in the human mind.Semiosphere is a concept which defines the current language, sign game or discourse and the contents of a sign within it.
An example of a semiosphere is an individual's personal environment.Personal environments entail what people experience in their personal and professional life and how they organize and interpret the signs they come across in their environment.Their skills, their social life, their friends and colleagues as well as their professions span their individual semiosphere.The mental contents in their personal semiospheres allow for the understanding of how they use signs when interacting with technologies.However, a technical semiosphere can be small.For example the signs A+, A+++ etc. form a specific semiosphere telling people the type and the size of a battery.
Signs are a central part of cultural and societal structures.Totemism, signaling, art, iconic signs, mathematics and logic as well as spoken and written language form a central part of human life practice [28,81,82,83].Following the development of computers, the role of symbols and signs has become even more significant.The prominence of these symbol processing machines has brought the importance of understanding the mechanisms of sign systems to the fore [19,84,85,86,87].
Similarly, cultural sign systems and cultural information contents create culturally definable semiospheres.People of Catholic faith are used to organizing their activities around Easter week and its parades and other festivals.They may spend substantial time during the year preparing for this important religious event.Soccer fans assemble in stadiums or gather in front of the TV to see their favorite soccer stars play.In the technical context, people create semiospheres around technological devices and applications.One clear example can be seen regarding the use and form of written language, which has altered with the increase of short message service (SMS) messaging.Other semiospheres can be seen regarding special interest and expert groups, such as specific programming interest groups.In these cases, actions and relevant symbols create semiospheres that can be interpreted in the context of content-based semantic thinking.These are some factors that define the perimeters of relevant mental contents.

INTERACTION AND SEMIOTICS
Signs physically embody intended meanings (or messages) encoded by the sender and technological design, operating as a sign, embodies intended meaning through techno-practical objectives and usage goals [4,5,18,88,89,90,91]. Any technical artifact such as the M-Machine represents something.In interaction, it represents an expected goal state that users aim to achieve through its usage.People steer a boat to reach a desired destination.Steering is based on semiotic interaction of such things as beacons or the direction of waves that relies on the interaction between psychologically analyzable mental representationsdirectional goalsand physical actionphysically manipulating the steering mechanisms to achieve the desired goal, in fulfillment of the mental representation.The role of semiotics is to provide concepts and methods for analyzing the systems of signs used in interaction.The function of empirical psychology is to provide the means for analyzing people's mental representations and sense-making processes in light of their behavior [92,93,94].
Sign systems such as programming language, user interface commands and iconography are multidimensional semiotic systems.The mental contents they activate are attached to cognitive, emotional, and cultural contents.Therefore, design decisions regarding device outlook etc. are loaded with aspects affecting practical efficiency, emotional aspects and references to specific cultural values and traditions [89].All of these components affect the experience of the technology as a whole.Additionally, these factors affect the meaning that designs hold for users, which in turn influences the holistic sentiments gained by users regarding particular designs [63,95].
A content-based semiotic approach is to be used as a means to analyze and explain interaction phenomena on the grounds of mental contents.In this way, the concepts and methods of technical semiotics meets user psychology.The objective of developing a content-based semiotics of HTI is to investigate the ways in which users mentally represent information contents in response to signs, codes and other semiotic systems.To carry out this research it is important to have a clear idea of the dimensions of mental representations people experience, in relation to the corresponding mental contents activated by particular signs.
There are several levels of analysis to consider, and respectively there are also several types of interaction processes.On the ground level, analysis is based on human biology and neural processes.Essentially, this level is the same as neutral semiotics.Brain states have their characteristics and it is possible to register these characteristics by following the electrical activity of the brain [96,97,98].As a result, it is also possible for people to input information into computers purely by thinking as seen in the case of BCI.Similarly, one can also register other biological states by means of modern technology in order to build ubiquitous interfaces [62].Human body temperature, location, speed and posture, for example, can be used as biological signs in interaction.
The second level of analysis can be based on emotions and their related factors.Emotional usability, or user experience, presupposes the analysis of emotional dimensions of mental contents [9,11,95].Key processes are emotional appraisal [99,100,101,102], basic emotions [103]and emotional aspects of cross-cultural factors in HTI semiosis [9].Theorists such as Norman [11,104,105] emphasize the importance of emotions in HTI.For example, it has been shown that when a user experiences positive emotions they are more likely to think creatively, and thus, explore alternative options when faced with unfamiliar situations.The opposite is also shown to occur.When people experience negative emotions, they are more likely to make errors as the result of quick, narrowly focused decisions induced by psycho-physiological effects such as flight and fight [106].
In content-based semiotics, argumentation is based on the explication of the concrete situational representations and associated signs.When considering technical interaction, it is essential to always investigate the contents of the users' involved mental representations and their emotional connotations.This is made possible when we are able to explain how people represent the given signs.In essence, this means articulating the mental contents associated with these signs.These questions are empirical by nature, necessitating them to be investigated by means of empirical methods.Examples of methods used to explicate mental contents can be interviews, focus groups or "think aloud" techniques [10,75,107].In this kind of research, people tell what they think and how they experience signs.This information allows researchers to select relevant information contents, both emotional and cognitive.However, even traditional questionnaires can be used to explicate both emotional and cognitive mental contents [108].The focus of these research efforts is to define what kinds of contents people really mentally allocate to technical signs and symbols.
Mental contents, are facilitated by and are the result of numerous factors from cultural to biological.Our final message is that the information society is by nature semiotic.In this light it is necessary to rethink some basic issues regarding traditional semiotic approaches, and call attention to what we term as semiotic thinking and symbolic interactions.Meaning does not exist in its entirety in that which can be observed through patterns and shared behavioural similarities.Meaning exists in the mind which is contingent upon previous experience (information contents), context and physiology.Semiotic thinking is based on the understanding of mental contents and how they correspond with messages set forth in society, by every day artifacts.Signs and meanings are seen to hold a key position in understanding the human dimensions of the information society.Information exists in signs, which are subsequently interpreted by the human mind and the research in the psychological functions of technological signs can be done inside technological psychosemiotics.Consequently, it is essential to consider signs in the context of human mentality, for this enables interaction researchers and designers to understand the place of semiosis as a mental phenomenon following the principles of mental processes.