When the Web goes visual, it may just take
language right along with it
SEE WHAT I MEAN?
by Douglas Rushkoff, December, 2015
These days, most smartphone keyboards dedicate more resources to emojis than to traditional alphanumeric characters. This development allows the iGeneration to communicate almost exclusively with emoticons and an endless array of seemingly meaningless images, such as identical twin girls wearing bunny ears. One winking smiley face says more for these kids than any message using the Roman alphabet could.
While it can be tempting to lament this development as the decline of written language in the Digital Age, I wonder if we should rather be appreciating emojis, animated GIFs, and the Web’s endless stream of memes as the tip of the iceberg for a coming paradigm shift in how we will communicate with each other. What if those little yellow faces are harbingers of an entirely new visual language?
Remember, every new medium starts off imitating its predecessors until it finds its own form. Early books were written versions of spoken stories. The first TV shows were staged plays performed in front of a camera. And now the Internet is using text, audio, and video as its content while it develops its own.
Indeed, as Web technology has gotten more sophisticated, it has moved steadily away from recreating that earlier media and toward more immersive experiences, like the Oculus Rift VR goggles or the planet-size interactive world of Minecraft. Just as emojis took over instant-messaging platforms, what language will rise to the occasion of 3-D virtual space?
Whatever it is it would have to be real-time, emotive, and universal—as iconic as emoticons yet as dynamic as zooming in and out on a Google map. It can’t just be a vocabulary the way emojis are. Such a new language will have to discover a syntax all its own, capable of expressing not just simple emotions but industrial information like medical data and stock market movements—everything we now do with text and graphics.
“Just as emojis took over instant-messaging platforms,
what language will rise to the occasion of 3-D virtual space?”
From what I’ve been able to gather from my friends at Facebook, Google, and Twitter, every major Silicon Valley player is racing to be the first to own such a system. One firm piling up such patents, Knowledge Visualizaton Systems, originally made its reputation creating interactive visualizations for better understanding brain trauma and climate change. KVS founder Alan Yelsey wants to extend such data visualization into what he calls “dynamic semiotics”—tools that can be used to express ideas and emotions in addition to information.
KVS is investing in research into how shapes and colors convey meaning and if those meanings are universal. Does red always mean danger? Do curves always suggest something organic? Can they translate those universal associations into a visual language that’s flexible enough to grow and spread on its own while remaining objective enough for business and science? If they can, then someday communication could well involve us donning goggles and painting scenes for one another by waving our arms—joining together in an abstract virtual universe. Stock traders will not just see how the NASDAQ is performing, they’ll also be able to share feelings about the numbers and gauge the collective mood. Medical students will not only watch visualizations of cancer as a biological phenomenon, they will also “see” its psychological impact on patients.
This notion of a shared visual landscape—a place where we can express ourselves through colors, shapes, and vibrating emergent forms—may sound like science fiction. Then again, the notion that the whole world would be communicating through handheld touchscreen computers was unimaginable just a couple of decades ago.
The idea that this new medium should get its own language is not just possible; it’s inevitable.
Douglas Rushkoff is a professor of digital economics, a world renowned speaker on technology and change, and a prolific author of such books at Present Shock: When Everything Happens Now. His forthcoming powerhouse book, Throwing Rocks at the Google Bus: How Growth Became the Enemy of Prosperity, will be out shortly.
THE COMMUNICATING COMPLEXITY PROJECT
The Creation of a Universal Dynamic Semiotic Ontological Pattern Language Capable of Best Signifying Systemic Meaning
The Proposition: Those who study cognitive semiotics and linguistics, language, education, computer sciences, art and design, social sciences, neurology and other fields will participate in the cooperative development of a universal computer aided semiotic language that synthesizes and signifies systemics. The involvement of media/cognitive/educational and coding professionals to develop a modern semiotic language capable of processing complexity is a natural and critical evolution. Like the stages of semiotic development in the first 3 years of life, we humans are just at the early stages of our journey, and our actualization of a language for systemics will similarly begin with intentional systemic communications and the onset of various forms of dynamic systemic semiotic communication, leading to a fully realized multimedia narrative form capable of extracting, organizing, processing, synthesizing, storing, retrieving and communicating a higher level multi-variate narrative.
Only recently have we realized the software technology, real-time processing capacity and paradigm shifting potential to conceive and create a new language that provides the engine, tools and processes to effectively communicate complexity, systemics, multi-variate time/space relationships and the overwhelming progression of real time information and real time problems.
We believe the world needs a language that can tame complexity. Fields such as cognitive semiotics, linguistics, neurology, educational psychology, computer sciences and media communications are called upon to work together to address one of the greatest barriers to human progress and beneficent action – the lack of a universal friction free robust cognitively optimized language composed of synthesized systemic knowledge represented by intuitive dynamic semiotic forms that can be introduced and applied by anyone, regardless of inherited languages, cultures or educational levels.
We now endeavor to establish and fund a global project to apply dynamic semiotics and linguistics toward the invention, test and ultimate introduction of a new, robust, scalable semiotic language, toolset and systemic that tames the beast – complexity.
Hypotheses to test:
1) Computer software, hardware and connectivity have reached a threshold that enables humans to utilize dynamic semiotic ontological worlds to represent meaning and objectives, processes and multi-variate relationships, knowledge and perspectives, proofs and uncertainties and participative and inclusive mediated content by using cognitively effective models, worlds, structures, forms and patterns that signify the highest levels of complex knowledge.
2) A dynamic real time generatively coded semiotic ontological modelset would add substantially to systemic knowledge accumulation and synthesis, comprehension, retention and effective multi-variate decision-making.
3) The genetic programming and wiring of human beings, coupled with the interplay among the conscious/subconscious/corporal, can be maximally inspired by an original, creative, appealing, computer aided semiotic world composed in such a way as to be culturally and cognitively accessible (through carefully designed semiotic forms and functions) to high value meaning linked to user objectives and knowledge scaffolds.
Human beings understand their world through the lenses of their inherited programming, their constructed languages, their sensory capacities, their manufactured communications mechanisms, their life experiences and their conscious interpretations and creative manipulations of all of the above.
From the beginning of human evolution until now, humans have adapted and evolved without the benefit of a systemic lens that would enable them to consciously perceive, test and understand the “what is” of the universe as the dynamic multivariate, multi-dimensional, multi-layered, multi-unknown construct that it is. Our existing forms of mass communication rely heavily on written and spoken primary symbols, (words and their accompanying sounds), inherited through centuries of unfocused cultural iteration. This form of language uses words as narrow building blocks that cannot represent a whole view or thought until all of the blocks are in place, and even then, once the wall of words is viewed or heard, the user is limited to a one or two dimensional form incapable of representing complexity, exchanges, interactions, relationships, proofs, alternative views and the inherent inadequacy of current knowledge – ever changing across time, space, state and levels of detail.
For the purposes of this concept paper outline, we are going to characterize human beings as a mutually inclusive dynamic compilation of conscious, subconscious and corporal constructs. This is a systemically informed choice, encouraging the reader to understand a human being as an exchange of matter and energy and bio/chemo/physio functions (corporal), autonomic/genetic programming and hidden inheritances (subconscious), and retained or real time self aware thought processes (consciousness).
Humans are a product of their genetic/inherited programming, their life experience, and their conscious, subconscious and corporal interaction with their life experience. Among the inheritances include a strong predisposition to identify deviations from the anticipated pattern of an environment or situation – across all sensory modalities. Inherited sight and sound processing tends to relegate the common, the every day, the expected, into general background noise, and to promote certain characteristics of sight and sound into immediate and important signal awareness. Unexpected movements and sounds, bright moving colors and lights, sharp piercing noises and specific music constructions, facial features with normative disfigurations or beauteous shapes, deeply signifying gestures and voice configurations, large vs small, loud vs soft, bold vs passive, red vs grey all produce wired human attention and memory because these stimuli all deliver meaning in the form of pattern.
Languages to interact with computers have successfully adopted some advancements in semiotics while languages among humans have remained relatively stable.
The languages humans use to communicate with computing devices have radically progressed. The languages humans use to communicate among each other, and within, have barely changed in over 200 exponential years of realized complexity. Human languages retain the barriers of literacy and manifold distinct versions and manifestations. Since 1970, computing languages have progressed in sophistication to the point that users can interact intuitively with computers without code, and the people that create languages that translate computer functions into desired functions use macro and meta languages, some with visual connectors and symbology, that subsume prior code and exponentially expand the power, prescience and functional capacity and diversity of computer interaction.
At the computer programming level, analogous to the human programming modes of genetic, experiential and self-programming, early laborious linear one- character-at-a-time machine languages have been superseded in practice by a small number of macro/meta languages that use simple forms of systemic representation to command the computer to interact with the coders and end users toward complexity, objectives and sophisticated multi-variate relationships and exchanges of content, form and function. Simple code sets, see Wolfram’s thesis on the deconstruction of everything, can produce generative complexities that “become alive”.
The capacity of human beings to construct meaning through dynamic and passive semiotic representations: symbols, pictures, ceremonies, letters/words/language, expressions, sounds, audio shapes and startles, motion startles-brightness-speed, clothing, jewelry, decorations, performance, art, colors, objects and structures is inextricably linked to our genetic programming which amplifies certain forms and patterns of these media so they become signals rather than noise. Our inherited life experiences enforce and mediate these representations of meaning – making any paradigm shifts in the scale, scope or mechanisms of these meaning conveyances difficult across culture, geography, time and institutional education. Once we inherit a language and linguistics, learn them and use them, they become more and more embedded as code, making it difficult for any new form to disrupt the status quo – what is.
We must remember that all of these forms of cognitive and creative representation were developed locally and incrementally, without explicit objectives, developmental architectures, academic foundations or functional optimizations. The constructs we call “language” and “linguistics” evolved incrementally, locally, without any significant master designer or planner laying out the most effective representations of meaning.
And at this moment, I am communicating to you, the reader, using primitive building block symbols and symbol sets that form learned meanings. There is nothing purely intuitive about this form of communication except for the fact that our sensory and cognitive functionalities enable us to seamlessly process these shapes and focus in on derived meaning, just like when watching television or movies we are transported, integrated, into the media, oblivious to what is around us. Here, the letters become noise, and the totality of shapes and context deliver the signal – autonomically.
Each of us is tuned to a certain spectrum of sensory channels, portals linking ourselves with our world, and ourselves, through which we more strongly perceive, receive, process as signal, store and utilize stimuli. We consciously, subconsciously and even corporally process the meaning and context of everything through our language and all forms of embedded representations of meaning. We think, believe, gut react and communicate within and without according to our language and representations.
The only way to intercept, redirect, disrupt, break or reinvent the totality of our genetic and life experience language programming is through conscious and/or corporal override – deprogramming, reprogramming or inserting new code and new languages into our mechanisms of perception, cognitive processing, context and systemic understanding.
The study of cognitive semiotics and linguistics, along with the study of social sciences and neurological/sensory sciences, provide the blueprint for the invention and ubiquitous adoption of a representational language capable of enabling participative knowledge synthesis and comprehensive systemic understanding.
I now ask you to work with us to organize a global project to hypothesize, prototype, test and ultimately teach a unique representational multimedia language that empowers everyone to better understand and apply complexity across the fabric of society. This effort would consist of applying Cognitive Semiotics and Linguistics to the invention of a visual and auditory computer aided language capable of taming and harnessing complexity. Such a language would ignite a participative revolution in human thought and action, improving lives, saving lives, empowering humans to transcend the severe limitations of words.
For a start, after 5 years of concentrated consideration, here are some of our imagined foundational principles that might contribute to the objective of inventing a process, toolset and language for systemic understanding based upon cognitive semiotics and linguistics:
1) The most effective mechanism for human beings to comprehend multiple dynamic variables and their exchanges and interactions across time and space involves the use of fixed ontological cellular representations filled with a simplified semiotic language (visual and auditory patterns that convey meaning either intuitively or through cognitive learning and synthesis) and its accompanying linguistics and creative expression. Drawing lines to connect variables is inherently ineffective. Having a simplified language composed of colors, shapes, ontological representations and fixed modelsets filled with pattern can be a way forward.
2) Any participative widespread adoption of a mechanism to address complexity requires trust. Trust is best accomplished through a transparent documentation of the objectives, processes, content, sources, research, science, verifications and validations comprising a cumulative mediated visual proof process showing the derivation of everything.
3) Current languages and forms of media must inextricably be linked to the evolution and ultimate foundation of a systemic semiotic language. Any new systemic communications system will not replace existing language structures, but rather serve as a high level representation of the meaning derived through the application of current languages to complexity, a labeling and tagging mechanism for all of the ontological variables, and a complementary language co-existing with current language and data visualization.
4) A new dynamic systemic semiotic language is not just an academic exercise even though it may evolve through academic institutions. A semiotic systemic multimedia language may be capable of surpassing all current forms of communication when it comes to:
2) extracting and communicating verifiable knowledge with alternate perspectives
3) synthesizing systemic understanding
4) understanding and operating any form of human system or enterprise
5) satisfying the objectives of the creator or user in the most effective manner
6) interacting with a computer screen
7) teaching and learning
This is a concept paper outline for your consideration.
Educational Psychologist, USA
Email alan at kvsstudio dot com