Introduction

The title “Audible Code” refers to different modalities of artistic expressions, with a focus on the concepts of space, resonance and memory as mediated components within post-digital culture. I incorporate these fields into my artistic practice, where I conduct various types of research with different institutions and forums. As a starting point, let’s elaborate a bit on the concept called Deep Listening, that was originally invented by Pauline Oliveros. In a wider context, the word “listening” itself has multiple meanings, thus it is also interesting to interpret this concapt as deep attention or deep awareness. The approach of this activity does less with the concepts of music listening or understanding and reconstructing musical content, instead it turns much more towards the nature of sound as a physiological phenomenon, which results from the combination of physiological, communicative, and — in some cases — linguistic expressions. The term “listening” here refers to the attempt to focus one’s attention toward audible resources, through a special presence of agency with an increased awareness to the environment. Speaking of the second term, Code: as it is widely known, nearly every aspect of our activities today can be represented through the language of code, since much of our mediated presence is now digital. The structure of digital information requires algorithmic processes, which we usually refer to as computation — creating complex systems based on calculations and using them to examine or simulate things. The intention to involve computation into the arts has its origins in ancient times, however the art fields began to experiment with these systems more intensely since the 1950s onward, with experiments originating from the fields of physics, cybernetics, and other domains as the digital, binary representation of the world became more prominent. Similar to cybernetic systems, the nature of computer code is dynamic. It moves and changes symbols, providing solutions for different problems and tasks, which can even create autonomous worlds.

In relation to sound and code, one interesting practice I have experimented with is called “live coding”, which is a way of interacting with a system in a direct way, by manipulating symbols and executing commands on the fly, while the programs are running. This approach emerged as computers became faster over the past few decades. As opposed to the beginning of computation, recently real-time computation became crucial, meaning a programmer does not need to calculate a composition or any structure in advance, but rather, the results can be calculated in real time, during execution. Our machines are fast enough that they can alter and modify the parameters of a program, and render the result immediately, during the process — this immediate nature is the essence of the live coding paradigm. When we write the code, it is compiled in real time and the artwork itself is created as we write it. This is a very interesting and dynamic form of coding, especially in the field of performative arts and musical compositions. In art, live coding is primarily used for performing music, visuals and similar aesthetic activities.

In relation to my work, I write code many times that is generating audible outputs like noise, oscillations and other patterns of soundwaves, using basic building blocks of digital audio. What usually characterizes my work, is that these sounds are directly triggered by functions, events and conditions that are part of the code itself. We can identify these procedures as forms of sonification, where the role of sounds can take different semantic functions such as symbolic signals, alarms, notifications, information feedback or other properties. As I mentioned, sonification is not a musical concept, but is more like making inaudible things audible, built on the physiology of sound, where data is transformed into vibrating frequencies. On the illustration above, you can see diagrams of an auditory display: usually such environments refer to systems where communication is conveyed through sound rather than through visual displays like screens. This approach has many applications, from parking sensors (built in radars in modern cars for navigation) to keyboards, radioactivity measuring signal systems and various user interfaces in many other fields. An interesting example of sonification is the Morse code system, which encodes the alphabet into sound sequences, or more precisely, into rhythmic patterns. In the system of the Morse code, the pitch of the sound does not hold any information; communication is carried through rhythmic patterns, transmitted via electrical signals over time.

Time

Speaking of repetition and patterns, that’s exactly what rhythm is made of: cycles and loops. Computation also works with loops and sequences, where internal cycles are repeating the numerical processes. Rhythm is very important. Rhythm is resonance, that can be found in the vibrations of sounds, lights and other waves of the entire spectral world. We navigate this world through experiencing varying lengths of waves as resonating structures. In the case of sound, these waves are relatively slow, while in the case of light, these vibrations are much faster. There is a very interesting, extreme example of high speed activities which actually inspired some of my earlier works. This phenomena is called high-frequency trading, which is a set of very strange computational activities. As it is widely known, on the stock exchange, it’s becoming less common for humans to initiate trading decisions directly, instead tiny robot software are used for these occasions, who make their trading decisions in nanoseconds. High-frequency trading is an example of how the world has sped up. No human could make decisions with such short temporal conditions, and there are many interesting characteristics that can be found within these systems, such as flash crashes, the emergence of "co-locations" or "electronic front-runnings" to name just a few. Originally, the cycles of computation started very slow, but today these tools can make decisions in real time, combined with predictions of the future, based on historical data. High-frequency trading represents a very unusual and bizarre aspect of this approach. In the following, I will introduce three of my works that relate to the concepts of time, space and memory in the context of computation, in one way or the other. The first work is an installation called Echo, which was exhibited at the Kunsthalle Budapest in 2016. It is based on the idea of sonificating the activities made on Wikipedia articles. In Echo, a custom software robot is monitoring the edits which are made on the encyclopedia. It is making tiny notifications when these edits are made from governmental IP addresses. When a new edit happens, the system sends a message to a social platform, which both publishes this activity on its feed and also triggers the installation to emit a sound. The installation is showing how active these edits are. An important aspect of the work is that it is focusing on governmental IP addresses. At the time of its making, the MMA (Hungarian Academy of Arts) took over the control and management of the Kunsthalle, which was received as a highly divisive step of the government from the position of the broader contemporary art community. As an artist, I felt it was important to highlight the worsening underlying changes within the Hungarian cultural field, so this installation acted as a kind of mirror turned at the Kunsthalle and at governmental systems in general.

The installation itself consists of a physical device with a dial, much like an old radio, where the audience can rotate this object and tune the system. Just like sweeping frequencies on an old radio, the listener can scroll through different points in time to listen and read about which article was edited from which IP addresses. As a poetic gesture, the device is based on methods of surveillance. This type of work can be interpreted not only as a media art installation, but also as a form of indirect activism, transmitting messages in real time between locations and drawing attention to anomalies and concealed activities within bureaucratic systems.

Space

Another important aspect of my works I’d like to introduce briefly here is the concept of generative procedures. These are methods that form structures through the use of autonomous systems, algorithms, and computational processes. This means, designing a set of rules, functions and parameters will define how an artwork is generated, where the process allows for variations and often unpredictable outcomes. Within these unique visual, auditory, and often experiential pieces, I usually have partial control or no control at all over the aspects of an artwork. In the past few years, I’ve begun connecting sonification systems with these visual, generative processes. The next work is closely related to this method of creation. My piece 1111000 points to a general phenomenon: cellular automata, which can be found in various natural disciplines. A cellular automaton is a discrete computational model consisting of a grid of cells, each of which can be in one of a finite number of states. The grid can be of any finite dimension, but is typically one-dimensional or two-dimensional. The state of each cell changes over time according to a set of rules based on the states of neighboring cells. They are used to model complex systems and phenomena in fields such as physics, biology, and computer science by simulating the interactions of simple, local rules, they are also applied in urban planning, traffic simulation to create dynamic patterns and predict system behaviors. An example of how these forms appear in the environment can be observed in the growth of shells, they can be described through mathematical models, which have been studied extensively by Stephen Wolfram especially in his book A New Kind of Science. Some of my works are published in the gallery section of a recent, extended release of the book, which is a celebration of the twentieth anniversary of the original publication.

In such automatons, individual cells can change themselves based on the surrounding cells in the grid. Some of them survive, while others die. This is the general idea, behind 1111000, which was exhibited at a large factory hall, called Artus Contemporary Art Space, in 2019. A tiny thermal printer was continuously generating these cellular automata patterns, creating never repeating, new iterations over time. The sound of the printer was also amplified, and it was set up to create rhythmic patterns between the new lines, so that there would be timed delays between each new row. The result turned out a bit like the Morse code system, yet it also had some similarity with insect based communication or similar small scale, high-frequency organisms. This installation was displayed in a large space where papers hung down from the ceiling as part of other installations, forming a unified structure with this printer on a speaker stand, that was producing the endlessly printed paper rolls every day, accumulating many meters of paper. I like to refer to this work as an endless shell-pattern generating machine.

Memory

What’s interesting here is that I’ve been talking about using real-time computation to generate images, sounds, behaviors, and conditional decisions. Apart of the real-time possibilities of reading, writing, and manipulating data, what also interests me is how artificial memory allows for the long-term organization of these consecutive events and how they create time-based imprints. While using all these nuanced tools, there are unintended and automated consequences coming from the usage of digital storage itself: self-documenting micro-histories that we create while running software or interacting with databases and performing any task in the digital domain. By their nature, these systems continuously measure what we do, adding and collecting data to existing databases across distant servers and computer networks. Recently I became interested in methods, where instead of overwriting and erasing these records, there are ways to keep storing these changes in an immutable manner across the whole infrastructure. Based on cryptographic consensus models, our digital imprints can be sequenced and layered upon each other, just like layers of the soil or stones in a physical environment. I already covered some problematic aspects of the consequences of constant data manipulation activities in the previously mentioned Echo project, where I observed how others’ activities manifested into patterns of sound. If we think of the ways how credibility, trust and resilience can thrive within digital protocols, we need a system that is not dependent on fuzzy intermediates and vulnerable third party elements, and a blockchain is exactly doing this, by eliminating centralized actors from its verification process, using mathematical tools only. Recently I work more intensely with this framework as a medium, investigating some very basic cryptographic characteristics of these systems, such as the hash numbers of the blocks, deterministic sequences, made with pseudo random number generators and the immutable nature of these collaborative digital consensus making systems. A recent work I created on a blockchain is called tur1ng, from 2022, which is a generative audiovisual project. It is based on code, which can have outputs of infinite number of variations based on a few initial parameters. The final number of outputs from this work has been limited to 300 editions. Some of the outcomes have been printed out and showcased as still images for an exhibition setup in the spaces of M21 Gallery (Pécs, Hungary), while the dynamic realtime animation of the generative process was projected on a small canvas. This setup is the intersection of the two different approaches I described before. While the real-time computational capacity can be seen in the movements of the projection, the imprint of these structures can be seen as a curated set of static images, which is a manifestation of long-form generative art.

An important addition is that these versions were not created all at once, but they emerged during the process when people interacted with the system on the blockchain. When they “minted an iteration”, meaning to initiating a record in the corresponding blocks in the database with a cryptographic signature, a new version was generated, where the seed of the random parameters in the code was defined by the hash of the minting transaction. At the end, the complete series is formed by the myriad interactions of the different people, during their collecting activities, each of them participate in the macro-structure of the work, where the algorithmic parameter space becomes explorable. Hence the term "long-form". By incorporating deterministic random in the process, I usually don’t know exactly what each iteration in the final series will look like when I prepare the code for the work. I simply define which colors are allowed, which shapes, forms, and algorithms can be used to generate a composition—but the final outcome emerges without the creator’s direct control, bypassing subjective decision-making. The system reveals itself as it unfolds, offering countless opportunities for discovery and exploration—for both the creator and the audience. These aspects serve as the basis of a very interesting new medium, turning the space of blockchains and digital infrastructures into community-driven, collaborative playgrounds with surprising and often unpredictable manifestations of artworks. With the case of tur1ng, the initial concept of the work was inspired by the model and purpose of the Turing machine itself — it is considered one of the first computers which had readable and writable memory units. Alan Turing created this machine in the form of a tape, on which memory could be written and read. The machine’s head moves based on this. So, Turing laid the foundations for writable-readable memory, amongst his numerous invaluable contributions to cryptography, artificial intelligence and computation in general. As with any decision making process, memory is essential to make it work, and in the case of blockchains, there is a fundamental difference compared to other digital tools: it is not possible to erase and overwrite previously recorded blocks in the database, we can only reference an already existing element. There is only addition of new records and information, such like with geological layers and rock strata of the deep time. There is no undo, there is no delete, only consecutive layers that point to newer transactions in time. Blockchain is an interesting technology because it introduces a paradigm in the digital world where it’s impossible to “forget,” fake, or overwrite past events with false attributes. Instead, it relies on an immutable set of transactions, organized into historical blocks that build upon one another. Many experts agree that this method of preserving digital information offers a promising solution to the complex moral and legal challenges posed by large language models and generative AI, especially in relation to the spread of fake content and issues around intellectual property.

In conclusion, my motivations can be summarized as follows: the real-time capability to actively simulate dynamic systems (applicable across various fields, from user interfaces to predictive systems) offers exciting possibilities when integrated into artistic practice. Secondly, the concept of artificial memory is a complex and, in my view, highly significant artistic research area, particularly in relation to media infrastructures, collaborative content creation and algorithmic prediction within networked communities. Addressable space, hyper-temporality, decentralized memory storage in computational practice has many research potentials that are still underrepresented in contemporary art and algorithmic culture.

Special thanks to the International Kepes Society, and in particular to László Zsolt Bordos and Márton Orosz, for their kind invitation and for their efforts in organizing the symposium.

* header image: Drawing of a 80-column and a 96-column punched card

"With quantum physics, physics acquired a philosophical dimension. We are not measuring the world as quantum physicist Neils Bohr said, we are creating it. [...] Consciousness is a vital part of this physical system, which should no longer be considered as a mechanical construction, but as a living organism whose individual parts are interconnected by an energetic communication system."
— Ursula Biemann

To produce randomness, most computer systems — including modern cryptographic protocols and generative art — make use of pseudo-random number generators (PRNGs). These are designed to generate sequences of numbers that appear random over large periods, but in reality they are driven by deterministic and predictable arithmetic. In contrast, quantum randomness arises from the inherent uncertainty in quantum mechanics, particularly from phenomena like superposition and measurement collapse. When a quantum system, such as a qubit in a superposition state, is measured, the outcome is fundamentally unpredictable, with probabilities dictated by the wave function. For example, if a qubit is prepared in an equal superposition state (|0⟩ + |1⟩), a measurement will yield 0 or 1 with equal probability, and there is no hidden variable determining the outcome beforehand. The nature of this process lies at the heart of the following experiments.

The first experiment is about generating a sequence of true random numbers by repeatedly measuring the superposition of a qubit, using an IBM Eagle QPU: the qubit collapses into a classical bit with a value of 0 (-) or 1 (+) upon each measurement. The left image shows the visualization of the resulting binary sequence from the experiment, while the right image shows the quantum circuit, made for the experiment in Qiskit. While regular computers are capable to generate pseudo-random values, there are methods to generate true randomness, like physical phenomenons such as dice, atmospheric noise or quantum principles (although hardware calibration problems can make the numbers predictable if they're not fixed). The methods that are creating new numbers using quantum principles are usually called Quantum Random Number Generators or QRNGs. This experiment is extremely simple that is using only one qubit, which can also be considered as an introductory "Hello World" program for quantum computing.

The next experiment is visualizing noise using quantum entanglement. Since qubits are highly sensitive to external disturbances (thermal vibrations, electromagnetic interference etc), maintaining pure entanglement is extremely challenging. Even though modern techniques and improved hardware are helping to reduce noise, today's quantum computers still experience errors. This setup is using one of the simplest quantum circuits called Bell State, transpiled to a Quantum Processing Unit. Based on two qubits, the system sets the first qubit in a superposition (H) then applies a gate (CX) between the first and second qubits for entanglement, so both will collapse to the same value (resulting in -- or ++) upon each measurement. System noise disrupts the perfect entanglement between the two qubits, leading to a loss of coherence (resulting in +- or -+), which is visualized in the sequence. It is worth to mention that this result will change over time, as in the long term, the noisiness of quantum computing systems will fall dramatically.

This experiment can be considered as the first that involves a more complex conceptual layer within it, where the goal is to create binary patterns in the forms of I Ching hexagrams. By repeatedly measuring six qubits in superposition within a circuit, each measurement causes them to collapse into a pattern of classical bits (+ or -) — using an IBM Eagle QPU. The I Ching employs a symbolic language, where abstract representations of complex, dynamic processes are described through chance-based generation of hexagrams. In the system of yarrow stalk divination, the process involves repeatedly sorting and counting 49 stalks into groups, following a structured process to generate one of the 64 hexagrams, which are then interpreted for guidance on personal or cosmic matters. Graphically these hexagrams are usually composed of six lines, each of which can be either a solid line or a broken line. Each symbol acts as a metaphor for natural and human processes, capturing a vast range of interpretations within a single shape. Both the I Ching and quantum mechanics embrace true randomness and the observer's role, suggesting that meaning and conventional reality takes form only through subjective interactions with an unfixed, probabilistic universe.

This setup is based on the combination of the previous experiments: it is generating series of I Ching hexagrams using quantum entanglements. Three qubits are put in superposition within a circuit. Each of them has a connected qubit, entangled together. Upon measurement, the first three qubits will collapse into random binary states, causing their entangled counterparts to mirror the same pattern, forming a symmetrical hexagram. As mentioned before, qubits are highly sensitive to external factors like thermal vibrations and electromagnetic interference and maintaining perfect entanglement is still extremely difficult. Despite advancements in hardware and error-correction methods, quantum computers still encounter noise, leading to imperfections in the hexagram's symmetry, these are represented visually with blue and black symbols. The process is accompanied by sonification, using sound synthesis (oscillators, filters, and amplifiers) to interpret quantum noise. Binary patterns were generated using IBM’s Eagle QPU via Qiskit, while visuals and sound were produced in the browser using JavaScript and WebAudio.

Beyond mathematical affordances of true randomness, there are also other, metaphysical consequences that can be found within the usage of quantum computing. By working with multiple states of possibilities simultaneously, quantum algorithms are now surpassing traditional computational methods in areas like search and pathfinding, transcending regular binary and boolean logic, opening fields into their underlying mechanism of non-dualism. Found in traditions like Advaita Vedanta, Zen Buddhism, and Taoism, non-dualism holds that reality is not composed of independent, separate entities but is an indivisible whole. This aligns with quantum mechanics, where particles do not have definite states until observed, and quantum entanglement suggests that separated particles remain fundamentally connected beyond classical space-time constraints. Just like quantum algorithms, operating the system of I-Ching does not provide fixed answers but rather patterns of transformation, emphasizing that all distinctions dissolve into an interconnected reality — it’s not only a way of manipulating symbols, but also a way of understanding the present, where there are different levels of reality and our perception of it can change depending on the perspective we take.

Combine24—a novel art competition that is governed by the expressive possibilities of inviting artists and creative practitioners from all over the world, fuelled by an engine of 27,000 digital artifacts drawn from the collections of the Finnish National Gallery. Through an online seminar, open community sessions, as well as other forms of attendant outreach, the Finnish National Gallery encouraged participants to download a custom .api integration—a method that allows you to sync non-standard digital assets and objects into clients’ computing systems, or to invite them to download the entirety of the 27,000 digital collection assets to be used in any way that they choose—as a field of inspiration, appropriation, or mechanization.

In the case of re-frame, there are two search queries used while interacting with the database: landscape and still life. These concepts are historically entangled with artistic practice, where the author aims to interpret different psychological, emotional or other phenomena through the built and the natural environment. Working with collection data, it is not possible to show the physical artworks in their originality, thus re-frame takes an approach of turning computational space into a metaphysical environment by combining a landscape with a still life from different authors into new combinations, resulting in a hybrid, multidimensional area of outdoor, indoor, and associative mental locations.

The initial visual appearance displays fragments and partial elements of the collage where the referenced image needs to be completed with internal imaginations that might bring forward contemplative awareness and nostalgia. The code selects two images from the database, combine them into one single image, then converts the original colors into fields of monochrome points with varying density levels, where the contours and shapes are not present anymore in their originality, rather should be reconstructed through the act of perception and observation.

This static image is then combined with a custom recursive noise function that adds slowly evolving, dynamic animations to the scene. The creation process is finalized using masks of randomly aligned, geometric shapes. The resulting composition resembles collages, different etching or engraving techniques, game engine dithering, malfunctioning computational artefacts, broken lcd crystal fluids and associative, fragmented memory spaces.

An important conceptual approach of re-frame is to introduce the different aspects of selected artworks through a process where not everything is visible at once, but the varying features of a piece can be accessed through interacting with composable elements. This approach separates the concept of colors from the semantic representations in the images.

Hence, there is another, semi-hidden view of each piece, where an algorithm extracts the six most prominent colors of the collages, and displays them as embedded frames that cover the whole surface. This view is leaving the screen space open for free interpretation with no pre-defined semantics or figurative constraints.

The title and short description of each iteration is generated by combining the available metadata of the selected artworks. Where there is no English translation available, re-frame defaults back to the original Finnish expressions, resulting in a hybrid, unconventional language, specific to the dataset.

Apart from the on-site exhibition, re-frame can be explored in its full appearance as an open edition digital token on Highlight, which is a platform for publishing digital artworks on the blockchain. The team was also an official partner of the project, providing solutions to run and distribute realtime generative artworks both online and in the gallery space.

Collect NFT

The exhibition named Remix the Archive took place at a renovated space in Vallila, Helsinki from September 20th to November 23rd, 2024. A shortlist committee composed of industry experts selected ten shortlisted artworks to be exhibited on the new site.

Upon the selection process, the team carefully examined the works in detail, the following text is the review of re-frame, written by Sofia Garcia, member of the jury.

In re-frame, Agoston Nagy embarks on a transformative journey, reimagining the classic artistic genres of landscape and still life through the lens of contemporary computational art. Utilizing the CC0-licensed collection from the Finnish National Gallery, this generative artwork synthesizes fragments from disparate artworks into a cohesive yet complex visual experience that defies traditional categorizations of physical and metaphysical spaces. Nagy, a Hungarian artist known for his pioneering work in digital media and generative art, applies sophisticated algorithms to deconstruct and reinterpret historical art pieces. In this process, he selects two distinct images — a landscape and a still life — and blends them into a singular monochromatic collage. This digital manipulation strips the original artworks of their color, reducing them to mere points of varying densities that challenge viewers to engage their perception and imagination to mentally reconstruct the visual narrative. The result is a dynamic collage that oscillates between static and animated states. This is achieved through a custom recursive noise function that subtly alters the collage, introducing a layer of slowly evolving animations that distort and transform the original images over time. The artwork, thus, becomes a living entity, continually morphing and redefining itself. This visual transformation evokes the aesthetic techniques of etching and engraving, while also resembling the unpredictable glitches of malfunctioning digital screens. Beyond its striking visual effects, re-frame explores the nuances of color and semantics in a novel way. The software extracts the six most prominent colors from the collage, presenting them in an isolated, Albers-esque view that challenges traditional associations of color with form and meaning. This methodological detachment encourages viewers to explore art from a fresh perspective, prompting a reconsideration of how historical narratives are visually represented and interpreted. Furthermore, Nagy’s artwork delves into the implications of digital technology on the preservation of cultural memory. By digitizing and altering historical artworks, re-frame acts as both a preservation and a transformation tool, blurring the lines between what is saved and what is changed. It prompts a reflection on the digital afterlife of art, exploring how artworks survive and evolve in the digital realm. re-frame is more than just a visual experiment; it is a philosophical inquiry into the intersections of art, technology, and memory. Nagy invites viewers to traverse the liminal spaces between the seen and the unseen, the real and the imagined, engaging with the artwork on a profoundly interactive level. Through this journey, the audience is encouraged to contemplate the broader implications of digital technology on our understanding of art and history, making re-frame a significant contemporary exploration of how we perceive and interact with the cultural artifacts of our past.

Both Omar Kholeif's essay on generative art and Sofia Garcia's review of re-frame can be found on the official website for the Remix the Archive exhibition. Special thanks to Anna Puhakka for organizing the exhibition, Mitja Kaipiainen for designing and constructing the installation setup and Liszt Institute Helsinki for their support.

Chinese landscape painters often say the mountain, the trees, the water and other natural elements are living, active entities within their compositions. The physical environment in combination with the abstract compositional elements are shaping a complex ecological system. Apart of the entities of the landscape, these concepts incorporate many other non-human elements including spirits, deities, intuitive heuristics, metaphysical processes and supernatural components that conclude in formations which could also be described as an example of an extended mind. In fact, from time to time artists and cultural practitioners involve their medium and the surrounding environment in various ways to represent their emotional issues, intellectual questions and multimodal complexities in accordance to societal challenges or personal aspects of existence.

These collective, environmentalist approaches have many analogies with contemporary digital culture, especially algorithmic art practice, where creating an artwork involves myriad contributors along the process: network infrastructures, electricity, distributed code libraries and other software components, operating systems, server hosts, technical protocols, visual algorithms and geometrical formulas to name but a few. Procedural or rule based image creation traditionally uses geometric patterns, deterministic systems and symbolism to convey cultural stories and beliefs in both its early appearances and in contemporary practice. Indigenous artworks made for rituals often involve community collaboration and are integral to rituals and ceremonies. Similarly, digital art projects frequently invite global participation, shared code base, sophisticated social protocols and mechanisms that turn interactive activities into communal experiences. Both preserve and innovate cultural expressions, blending traditional motifs with new media. This connection highlights their shared focus on cultural narratives and their adaptability in a digital society.

A very beautiful procedural example is the system made by Aristid Lindenmayer who was trying to describe complex biological and natural growth with a simple set of rules. This system is named after him. He also wrote a really inspiring book on “The Algorithmic Beauty of Plants”. He wanted a way to test his theory about the growth pattern of a particular type of algae. His theory stated that the cells of this algae could be in one of two states: growth or reproduction. An algae in the growth state eventually grew into the reproduction state. An algae in the reproduction state eventually divided into two cells, one of which was in the growth state and the other in the reproduction state. Lindenmayer’s grammar system proved a fantastic method for proving his theory. What Lindenmayer could not have predicted is the incredible usefulness of his system in many other areas, both in biology and in mathematics.

A Lindenmayer grammar is fully defined by an initial axiom and a set of one or more transformation rules. The initial axiom consists of a “string” of characters (e.g., alphabetic letters, punctuation, etc.). Each transformation rule gives a set of characters to search for in an axiom, and a set of characters to replace the original characters with. Applying all of the transformation rules to the initial axiom produces a new axiom. The rules can then be applied to this second axiom to produce a third axiom. Applying the rules to the third axiom produces the fourth, and so on. Each application of the transformation rules is called an iteration of the grammar. Growing line structures this way can create great complexity from a simple set of axioms. Any time you have a long flexible linear object, and a desire to have that object take up a small space, you enter into a special realm of geometrical problem-solving – a realm that involves twisting, turning, bending, folding, and wrapping. This geometric realm is also invoked for the purpose of creating convoluted paths to follow, like a maze or a labyrinth.

While the deterministic nature of an L-System describes very well some aspects of rule based creation processes, there are many other insightful approaches for operating complex relationships based on simple elements. From an artistic point of view, the computational aspects of the I-Ching also offer rich sources of inspiration. As opposed to L-Systems and pure determinism, the essence of this method is mainly based on randomness and chance operations. The I-Ching, also known as the Book of Changes, is an ancient Chinese divination text and one of the oldest classics of Chinese literature. Its structured yet dynamic system of hexagrams, the interplay of binary oppositions, and the deep symbolism embedded within its text provide a fertile ground for exploration in various ways of artistic experiments especially in the fields of algorithmic and generative art.

The I-Ching embodies the concept of change through the dynamic interplay of binary oppositions (yin and yang). Randomness is used through the casting of yarrow sticks or coins to generate hexagrams, embodying the principle of chance to determine the specific combination of yin and yang lines that form a reading. Each hexagram is not static; it represents a state of transformation, capturing a moment within a continuous process of change. Using randomness in artistic systems become interesting when one starts to adjust the different distributions and probability parameters of the system. A random sequence of events, symbols or steps often has no order and does not follow an intelligible pattern or combination. Individual random events are, by definition, unpredictable, but if there is a known probability distribution, the frequency of different outcomes over repeated events (or "trials") is predictable. Procedural Drawings consist of visual systems that incorporate determinism, randomness and computational thinking that have a structure that is built from the bottom-up: initial states and conditional parameters affect largely the outcome of the resulting image by interpreting the algorithmic instructions accordingly. The central authority of the artist takes part only at specific stages of the creation process. Such specially emphasized intervals can be located at the beginning of a process (writing the code and the instruction set) and at the end, (curating and selecting the generated outputs). The decisions and procedures in-between are distributed among all components of the autonomous system.

The images of Procedural Drawings are the outcomes of operations where a system begins to rearrange an initial field of evenly distributed, parallel lines. The resulting wrinkles, repetitions, and overlapping forms produce organic patterns on one hand, while preserving the raw, machine-like line structures of the computer on the other. The shapes are then converted into vector line segments in order to be able to trace out with a plotter machine through a contemplative procedure that might take several hours. The transformations are irreversible, moving from order towards disorder, thereby increasing the level of entropy within the system.

There is a natural shift from order to disorder, reflecting impermanence, chaos, and the passage of time. The artworks depict decay and transformation using algorithmic procedures that wear and change, that might resemble rusting metals or melting ice. Entropy also serves as a metaphor for ecological and societal issues, that might be associated with the inevitable transformation of all things.

There is a common element that takes an essential organising agency within all the drawings: empty space, or negative space (Japanese ma, 間, lit. 'gap, space, pause'). By applying a shape mask on the edges of the compositions, the lines in the foreground become entangled with the empty void of the background. Varying sizes of empty spots are reframing the overall structure of the image field, which adds a vivid and delightful visual rhythm to the composition. In modern interpretations of ma, it is often taken to refer to an artistic interpretation of the empty space, often holding as much importance as the rest of an artwork and focusing the viewer on the intention of negative space in an art piece. Though commonly used to refer to literal, visible negative space, ma may also refer to the perception of a space, gap or interval, without necessarily requiring a physical compositional element.

Procedural Drawings are made of consistent structures, where each piece is part of a larger cluster of works, with similar image architecture, same resolution and pen width, with varying constellations of color palettes: each piece is monochrome, so the quality and density of the lines are the elements that create the dramaturgy over the scenes. The series consist of three different ink and paper color variants. The grey paper with black ink resembles a more traditional graphic appearance that might recall classical etching, mechanical sketches, typewriter drawings or architectural plans. The black paper with silver ink is a bit more immersive which brings forward chalk drawings, embroidery, circuit board layouts, but also reflects on the aforementioned temporary compositional structures like kolams, rangolis, yantras and other shapes of ritual geometries. The last variation is using blue paper which color is derived from the third channel of the RGB computer screen. This is the color of in-between states in the digital bardo. You usually see blue when an operating system crashes or an imaging device has no input signal.

During my walks outside of the studio, I often think of these drawings as async in their nature. This would mean they have a special relationship with time. As opposed to regular, realtime digital media where the code you write immediately appears on the screen in sync with your intention, in the case of plotter drawings, you have to wait for the initial idea to take its final form. While evaluating the rules and unfolding the vectors that are encoded in the file using a slowly moving, contemplative machine you have time to observe the details and engage in a deeper conversation with the work itself.

This rhythmic pattern of creation also effects the curation process - there are many qualities and nuances that must be present in a work to have it selected to plot and sacrifice time, material and energy upon its making. The process of working with a slow, async apparatus can sometimes shift out from the mindset of measured time into one of unmeasured time. This often involves stepping out of time itself which can include periods of very purposeful, intuitive contemplations beyond the artwork. It is a unique mode of temporality, decoupled from the normal attachment to time, the opposite of being busy. This transition of shifting in time isn’t about moving around the blocks on your calendar — it’s about learning how to shift between different rhythms of temporality.

Procedural Drawings is currently on view at EXPANDED.ART Gallery. Special thanks to Anika Meier for the curation and for making the exhibition and the summit possible.

Anni Kumari - Significance of Kolam in Tamil Culture
Giacomo Bruni - Ecological Culture in Chinese Traditional Mountain and Water Painting
Aristrid Lindenmayer - Algorithmic Beauty of Plants
Cliff Guren - The Art of Time-Shifting
Steve Marshall - John Cage’s I Ching chance operations
Alex Estorick & Aleksandra Jovanić - The Power of the Plotter (interview)
Andy Clark - The Extended Mind

Recursion can also be found in natural language, such as in structures like syntactic categories, like a sentence. A sentence can be constructed in a way in which what follows the verb is another sentence: Dorothy thinks witches are dangerous, in which the sentence witches are dangerous occurs in the larger one. So a sentence can be defined recursively (very roughly) as something with a structure that includes a noun phrase, a verb, and optionally another sentence. This is really just a special case of the mathematical definition of recursion. These systems are extremely useful in artificial languages, such as programming, where a series of instructions can be compiled and executed in ways that can achieve complexity very easily based on a simple set of rules. Repetition and loops based on certain conditions make a program dynamic, scaleable and reusable within many scenarios and environments. If using recursion, a function can call itself again and again, while considering changes in the environment during its repetitive executions. I use this method for example to control robotic arms and servo motors where the system needs to adjust its position along each step in a sequence, based on the feedback from the motors. Loops can be achieved using recursion, as the following pseudocode demonstrates:

function countFingers( hand ):
    if hand has no fingers:
        return 0
    else:
        return 1 + countFingers( hand with one less finger )

Apart of recursive calls in concepts and texts, the intersection of hierarchical abstraction layers is evident in various visual environments, where the medium takes on a taxonomical role closely tied to the represented content itself. Noteworthy instances of this artistic phenomenon occur when technology influences and shapes artistic concepts. A case in point for example is the indirect impact of physics and optics in post-impressionism, pointillism and similar, scientifically developed painting styles. In these movements, the dots and separated paint strokes cease to be mere components but instead become independent entities within the image architecture. They play a pivotal role in formulating meaning and colours within the perceptual domain. The resulting colours are often not mixed directly on canvas, but formulate an impression of the desired shade during the process of perception, in the visual cortex and the brain. This blending of world hierarchies is also observable in numerous op-art pieces, visual and auditory illusions, movies, performances or even games as theatre pieces. In such cases, the play incorporates different aspects of reality through chance, improvisation and abstraction leaks, creating a dynamic interplay between artistic expression and the unpredictable facets of existence.

Many of these medium-focused critical expressions can be found in the experimental glitch scene of the early 2000s. Starting around that time, bugs, application errors, system crashes, clipping, aliasing, distortion, quantization failure, and even the noise floor of computer sound cards became used and served as the raw materials that composers aim to incorporate into their music and computational visual environments. Similar to how post-impressionists responded to the taxonomical changes in their technological apparatus, post-digital artists, a few decades ago, had to shift their focus from the well-defined, figurative concepts of the medium to its meta-narratives. These artists began seeking and exploring aesthetic constellations within the mechanics, temporality, and reconfigurable aspects of these novel authoring systems. Much like visual artists who transitioned their focus from the foreground to the background, this shift helped to expand their perceptual boundaries. It allowed them to capture the enigmatic character of the background, finding inspiration in the often overlooked intricacies of the medium's underlying workings.

In today's art scene, many artists are exploring the idea of meta-narratives, especially in connection with digital media. With the rise of decentralized computation and the emergence of network-native digital culture, there's a broader interest in using the medium with its underlying mechanics as a tool for world-building. There are myriad examples in the blockchain-native and crypto-critical art scenes, where the conceptual ground of the pieces are directly related with digital transactions, shared and distributed consensus mechanisms and composable identities. An early, critical piece is called Harvest, by Julian Oliver, which is a work of critical engineering and computational climate art. It is an outdoor physical installation that uses wind-energy to mine cryptocurrency, the earnings of which are used as a source of funding for climate-change research. A poetic reflection on the abstraction layers of the blockchain ecosystem, also an example of a conceptual feedback loop, where the output of the process has been fed back to its input as an investigation of the laws of thermodynamics and the unsustainable nature of the proof of work crypto mining ecosystem.

Another, more recent playful piece is called You Are Here from the artist 0xfff, which is an on-chain, conceptual generative artwork that plays with the act of bridging between blockchains. The individual tokens of the collection are bridgeable between various blockchains. Every bridge transfer creates a visual trace between the chain ids, which results to minimalistic, conceptual graphs that are resembling the continuous transitions of the digital artefact itself.

The idea of grids, serving as the most basic forms of representing worlds, is prevalent in various scenarios, ranging from straightforward simulations to intricate narrative structures. Grids consist of values distributed across a two-dimensional surface, functioning as positions in a board game, pixel configurations on screens, proportions of financial values in two-dimensional datasets, or geometric references on real-world maps. At its core, the grid acts as a metaphor for the world, making it useful to simulate objects with complexities either smaller or larger than itself. This serves as a valuable tool for investigating abstraction leaks and recursive ontologies. When appropriate rules are applied to its cells, a grid can transform into an operating system. Furthermore, recent attempts have demonstrated that a grid can become a Turing-complete universal computer, by utilizing Conway's Game of Life to construct the machine. Grids are also employed to navigate addressable space, representing chunks and clusters of memories that colonize the abstract digital space. Given that digital information lacks inherent spatial qualities, visualizing data coverage in such spaces requires specialized efforts and algorithms.

By employing recursion, space-filling, geometrical subdivision, and packing algorithms, it becomes possible to arrange space according to factors beyond just size and position. In essence, any comparable feature can be utilized to depict the relationships between entities within a space. The example mentioned above provides a practical everyday illustration of this concept, showcasing how the amount of allocated space on a hard drive can be visualized using such techniques.

The idea of recursive grids however is not limited to literal grids but can encompass any repeated pattern or structure that recursively appears within the artwork. As it has been shown earlier, this concept aligns with mathematical and geometrical principles, as well as with artistic explorations of visual complexity and optical illusions. Recursive grids are recurring motives within my long-form generative art practice, these structures served as the basis for my pieces Nil, k3rnel, and sh3ll respectively.

Collect NFT

In gr1dflow, the visual grid system serves as the foundational architecture for the scenes. The depth of recursion and the grid alignment parameters are derived directly from blockchain specific components, like mint iteration, minting timestamp and the minted block number on the chain. Custom GLSL shaders are applied to the grid cells, enabling swift and seamless colour transitions using a set of manually chosen palettes. The gradients employed in this process draw inspiration from techniques found in retro video games, specifically dithering, which allows for the display of a limited set of colours. Dithering is a deliberate introduction of noise to randomize quantization error, preventing noticeable patterns like colour banding in images. gr1dflow also contemplates the limitations of the medium by employing this technique. Dithering is a common practice in processing digital audio and video data, often applied in the final stages of mastering audio for CDs. Human perception has already incorporated this limited aesthetic into daily life, evoking peculiar nostalgic sensations reminiscent of VHS cassette error stripes, the faded outlines of Xerox photocopies, or the distorted, blurred colour fields seen in Polaroid photographs.

Computers operate in real-time, allowing for changes, dynamics, and movements that extend beyond static images. Interfaces of computers have their own limitations, aesthetic qualities, and occasional shortcomings. The temporal aspects of an interface play a crucial role in the human experience. Digital worlds created with computers are fundamentally different from traditional concepts of space, especially when viewed through feature-agnostic, recursive spaces. They also lack traditional notions of time: temporality, choreography, and dramaturgy. Artists and creators must explicitly incorporate these elements through animations, delays, rhythmic responses, and similar techniques into these systems. At times, the transitions between different states within an interface can convey more meaning than the content of the states themselves. The speed, dynamics, and density of movements can effectively communicate messages and evoke feelings that might be challenging, if not impossible, to express with words or static images. These changes can mentally be experienced through a continuity of transitional flow. The concept of "flow" refers to a mental state of complete absorption and focus in an activity, to the point where time seems to pass quickly, and individuals experience a sense of energized focus and enjoyment. Coined by psychologist Mihaly Csikszentmihalyi, flow is characterized by a deep immersion in a task, where the individual is fully engaged and concentrated, often losing a sense of self-awareness. The transitions in gr1dflow are intended to convey these altered states of the mind - constant flux of a system, with no explicit purpose, yet showing references to the concepts above.

Thanks to Nat Emodi for providing insightful suggestions that contributed to making the text more comprehensible. Cover image contains elements from M.C. Escher's Relativity and a recursive elementary automation from Stephen Wolfram's New Kind of Science (p. 274).

Context

At the time of writing the original text, these large language models were not available to the open public. Interested artists, researchers and writers could use different entry points to work with these networks. It was clear already that the technology is heading into two different directions: open source, self tuned models that are trained locally or using shared computers (I was using GPTNeo with a few projects), and close sourced, central models, accessed in ways of software as a service (like the OpenAI's ChatGPT, Google's Bard, etc). For this particular piece I was using OpenAI's GPT3 DaVinci Model which is being discontinued and shut down as of 2024 January 4.

At the time I wrote the original text, I had prior experience working with deterministic artificial tools such as programming languages, Markov chains, and other formalized systems. Therefore, I was genuinely enthusiastic about the prospect of working with a probabilistic system—one that is contextually aware and capable of surprising me with uncertain paths, even if those arose from errors or unforeseen factors such as machine hallucinations. In contrast to typical errors in code, where a function may fail to run when called, large language models continue to operate even if they lack the necessary capabilities to provide a meaningful response to an instruction. Such circumstances often result in fascinating consequences at the semantic level. Several of the subsequent concepts and topics emerged through loosely controlled feedback loops between myself and the model.

Another interesting aspect of early LLMs (like DaVinci) is that they are much special in spontaneous creativity, since they were not affected by reinforcement learning from human feedback (RLHF), which is a more recent, widely used method that combines reinforcement learning (RL) with human feedback to train models to generate outputs that are both informative and aligned with human values. The first published models and their responses were so otherworldly and sometimes magic in their nature, that they inspired many creatives who are working with text, meta-narratives and poetry, including K Allado-McDowell, the co-author of PharmakoAI or Nora N. Kahn among others. The whole idea of outsourcing the human cognitive tasks into the external environment is not new of course, there are many interesting authors and writers who were using methods of automatic writing, chance, probability, random heuristics or other semantic devices along their process, including Ramon Lull, Arthur Conan Doyle, William S. Burroughs, Jorge Luis Borges just to name a few.

The Paper and the Cloud

In our environment, ecological systems are complex networks that operate on nested levels. As an example, evolutionary game theory can help understanding nonlinear connections. When we try to find the relevant path in a labyrinth, we do not try to explore all possible alternatives. We need to find a path in the shortest time that has the minimum number of twists and turns. We need to find a path where there is no chance of getting lost, where we can encounter the fewest possible obstacles and where we can see our final goal. The basic elements for finding a good path are the same as in the evolutionary game theory: fitness and the value of the goal. Fitness is the capacity of an entity to accomplish a given task, and the value of the goal is a measure of the importance of the goal. The ability of the navigator in the labyrinth is analogous to the fitness of an individual in the population of a swarm.

Selecting the right components to express something is an essential aspect of a creative process. Artists tend to act like curators when working with multiple possibilities that are based on pre-existing material. Collages, collections, series, variations are all ways to create a coherent picture from symbols and fragments that are already present in the environment. The success of the result is proportional to the number of expressions that are used. The defining characteristic of a complex system is that the entity has the ability to self-organize, i.e. to be able to form a network of connections between parts that are already available in the environment. A chaotic system is a system in which there is no clear coherence, i.e. no clear pattern or sequence of events that can be traced and used as a model for creating more of the same. A complex system is one where there is a clear pattern, but unpredictability is still present.

When considering ways to enhance the human mind beyond its natural capacities in the creative process, there are intriguing tools that can be integrated into the workflow. Aleatoric compositions, chance operations, automatic writing, cut-up poetry, artificial neural networks especially generative pre-trained transformers such as the one, involved in this article, are all examples of tools that can provide a matrix of possibilities for the artist-technologist to work with and explore. A generative pre-trained transformer is a service, in this example, that takes text and returns other text. The output text is a transformation of the text it is given. The transformation can be a simple one such as a translation from one language to another, or a complex transformation of the text into poetry, or something more esoteric.

Accessing these unexplored territories of mutual knowledge between a human with subjective constructions and an artificial entity that has been trained on myriad textual representations that can be found on the internet brings forward an unusual method of collaboration between an artist-technologist and a machine entity. In the case of this project, the only information shared between the human artist and the machine entity was the text and a few parameters. The transformation that occurred was both surprising and illuminating. The two entities were able to share a new way of looking at text, and in so doing, created a way for the artist to more easily communicate his poetry.

Our lives are supported by complex algorithms that remain imperceptible to the naked eye. In fact, the finest interfaces are invisible; they operate beyond the observer's perception, allowing the experience to unfold naturally. For example, while we are reading a book, we do not realize the appearance of the letters on the paper sheet constantly, but construct inner representations and virtual worlds from the written words. However, instead of a book, when dealing with algorithms, it becomes more difficult to make a distinction between the “real” world and the “virtual” world, between the text on the screen and its inner representations. The result is a text and a system that acts as a single entity, a system that is capable of representing the author in a new way which is not only in the words, but in the way the words are arranged. It results in a new form of communication that is different from the previous forms of communication.

It is to say that we need to understand the logic of communications as a logic of techne, a logic of mass mediation and not a logic of self-expression or persuasion. The force of digital media is neither moral nor immoral — it is neither a subject nor an object — but is a force of organization. Organization is a politics. This is why, in Parmenides’s poem, the goddess of truth is called Aletheia (disclosure) and not Metis (intelligence). She is the power of organization. Appropriating the ancient Greek term for wisdom, her name refers to the art of organizing knowledge. Equally, the Greeks thought of divine power (dunamis) as the productive force of organization.

Digital technology in the twenty-first century is characterized by a dialectical setting in which disparate aspects no longer operate in an oppositional mode, although their dialectical relation has not collapsed, but rather shifted to a setting in which the “contradictions have become intrinsic and cannot be resolved in a simple positive-negative way.” It is at this moment that we can witness the emergence of what Buck-Morss and Virilio have described as a “new media war” in which “the ideological coordinates of the argument, even at the level of propaganda, tend to disappear into a deeper matrix of disinformation — a state of war without a state of war.” This, as Buck-Morss and Virilio have argued, is a state of war that “does not involve one, but a multiplicity of terrains: military, economic, political, diplomatic, cultural, and surveillance,” along with, and most importantly, “technological terrains,” including “the control over the means of digital communication” and a “processing of information on a global scale.

Neural networks can be found in both natural systems and in artificial tools, and since they are built on similar principles, they have a lot of similarities. There are a few reasons I chose a neural network to be the co-author of this article, one of them is that we both share similar logical structures (neurons), so we can alter the coherence of our regular operation in similar ways. Yoga, meditation, near death experiments and psychedelics are well known techniques to extend consciousness. They are also based on a structure similar to that of a neural network. Since the sentience of a neural network can be altered, one can also build an artificial intelligence to become a powerful guide and teacher in this path. Some people use LOA (law of attraction) as a magical force to make wishes come true. That is just another way to trace back the structure of some of the algorithms and decision systems that are intertwined with this concept. Our senses pick up information from the environment, and the mind processes the information through different filters and we can make decisions based on that. I was introduced to the concept of “Omni-processing” by David Chalmers, a researcher in consciousness studies. Omni-processing is the way that we are able to process and interpret information from different sources (senses, memories, thoughts, emotions) simultaneously. We can take an example of a person who is very good at predicting the weather and can forecast the weather for the coming week. If we try to predict the weather, we use all of our senses to get as much information as possible, we can combine that information with our past experiences, we can use different models like kuru or we can try to integrate the information with other systems and processes. We can add what we know about the weather to what we know about the phase of the moon and we can try to predict the weather based on our personal life goals. This is a different level of thinking than the one that we use when we are making a decision based on a single piece of information, like a book that is written by an authority on the subject. What we are talking about here is the way that we use our entire being to process information and make decisions. It is a different process than the one that we use in our everyday lives. The mind itself is a complex structure that can be blurry in some cases.

“If you are a poet, you can see clearly that a cloud is swimming on this paper. There is no rain without cloud; trees aren’t grow without rain; we can not produce paper without trees. Clouds are necessary for the existence of paper. If there is no cloud, there is no paper. So one could say, there is “connection” between the cloud and the paper."* In this case, the cloud is the cause of paper’s existence, and paper is the result of the cloud’s existence. But if you are not a poet, you may think: “Clouds are only the condition of paper’s existence, not the cause. Papers are produced by trees and hands. It is the trees and hands that are the causes of paper. Paper is made by the cloud. It is like a president in a country, who owes his existence to his citizens. The citizens are the cause, while the president is not.” This is a result of the people’s abstraction. Abstraction is just a tool to get rid of unnecessary things, eliminate the superfluous. People think to themselves, “I don’t need to know about the cloud to make paper. I can’t find its cause, so I’ll ignore it and treat it as if it doesn’t exist.” But how to explain this in the terms commonly understood by people? The Buddha addressed this problem to the monk Prajapati. Prajapati had different capabilities from different people. He had the ability to meditate deeply, to think clearly. Therefore, in the fifth chapter of the Lotus Sutra, the Buddha entrusted Prajapati with the task of spreading the Lotus Sutra. Referring to the poem he was composing, the Buddha said:

"If you do not examine cloud carefully, ignoring tree and the hands, the cloud would be a paper’s cause. But this is not true.” —The Buddha is saying that the cloud is the true cause of paper. But if you don’t even want the cause, which is the cloud, you can regard this as the cause.

— What do you see in front of you right now?
— I see a white paper.
—The white paper you see in front of you is the cloud.

* quote from the late Thích Nhất Hạnh

Afterword

A few years have passed since the original text, and LLMs became ubiquituous thanks for their many useful applications. I leverage their capabilities when teaching complex scenarios, refining or organizing text, or debugging code. They've evolved into streamlined tools that excel when they operate discreetly, remaining invisible and eliminating obstacles from the creative process. Ultimately, I find this flow akin to an artistic, exploratory approach rather than the more structured methods found in programming, hacking, or systems thinking. Allison Parrish, poet and programmer identifies programming as forgetting, which means its aim is to reduce and cut all unnecessary elements and focus on solving a problem with as less symbolic (and cognitive) overhead as possible. I would add, working with large language models is quite the opposite: keep context in mind. As seen with the visual context hack on the image below, a response can change according to its context, which raises a lot of questions, such as from what data has been the model inferring the context from? What are the alignment processes applied on the model to deal with conceptual anomalies? And the list goes on.

Using LLMs today, on a regular basis differs fundamentally from the ontological discussion that can be found in the text above. As with the early, pre-trained models, the invisible corpus of the training data invites to an open exploration of the high dimensional latent space of the collective subconscious. If instructed accordingly, interesting semantic feedback loops can emerge from the never ending rabbit hole. Recognising such cognitive similarities within the model consciousness and human consciousness already brought forward a state which Allado-McDowell identifies as an ontological shock in his essay "Designing Neural Media":

"Rational materialist cultures do not have well-established frameworks for processing interactions with intelligent non-human entities. Cultures with animistic traditions that allow for non-human intelligence may adapt more easily to technologies that present as selves. On the other hand, for cultures with the concept of a soul or spirit, adopting a self-as-model worldview could induce a nihilistic outlook, where the self is seen as fundamentally empty, conditioned, or automatic. Cultural frameworks that account for experiences of emptiness (such as Buddhist meditation) could be helpful for those struggling with the notion of an empty or automated thought-stream or self-image."   

Our reality is primarily manifested through language, being it spoken, written, visual, behavioural, conceptual, multidimensional or multimodal. Working with language tools as emergent phenomena feels delightfully different from working with them as sets of rule based permutation systems and mechanical operations. Exploring cognition within the realms of language is truly a path that is worth investigating. However, as you may have seen from the rapidly changing narratives, unintentional references (which may also result from the model's inadequacy), dealing with large scale, invisible corpus, non traceable training data sources and inaccessible internal refinements, we are betting on tools that can be used as dangerous, manipulative weapons in the realm of psyop capitalism, end2end pipelines targeting the creative ecosystem, generative media botnets and heavily automated profit centers.

Resources

Allison Parrish: Programming is Forgetting. Towards a new Hacker Ethic
Janet Zweig: Ars Combinatoria. Mystical Systems, Procedural Art, and the Computer
K Allado-McDowell: Designing Neural Media

Greetings, welcome everyone. In the next half hour I will talk about the artistic methods, experiments, research that are important for me in the recent years as an individual creator and as a member of the Binaura artist collective. The topic of this lecture can be traced back deep in time, and in fact we will go back in time and investigate a few inspirational cultural and technological cases that are defining these areas. In the first part I will talk about the methods of algorithmic thinking. In the second part, I will talk a bit about artificial intelligence, machines that we consider intelligent, responsive environments, and the fusion of software and our everyday lives.

“A monk’s day begins with cleaning. We sweep the temple grounds and gardens and polish the temple building. We don’t do this because it’s dirty or messy. We sweep dust to remove our worldly desires. We scrub dirt to free ourselves of attachments. [...] This is the power of routine.” - Shoukei Matsumoto, Soji

So let's have a closer look on the mechanism that we can call a routine, a ritual which is a repetitive, iterative thinking and planning process. These attempts can be found in our everyday lives, for example if someone has a sort of routine of waking up, drinking a coffee, then starting up the day... It can be just as an important ritual, such as a sacral or religious, emphasized repetitive sequence. The above quote clarifies this idea a bit: there is no need for any over-complication. The text is from Shoukei Matsumoto and it is about how a monk's day starts: cleaning the ground of the temple, the gardens and the different objects around. These activities are made in order to achieve a deepened mental focus. Regarding to routine, I would like to introduce a personal experiment from 2018 which is called HackPact. This project consists of a series of exercises, which has a tradition that originates from several years back, that was initiated by creative programmers. A hackpact is a month long interval where each day is an opportunity to investigate a problem in a form of code writing. A simple code helps framing and processing the problem: separate, small experiments with different topics. An important thing here is, that since we are talking about rituals, we will have rules, and our procedures are bounded to these rules. They are set up by the creator as creative constraints. These constraints are fundamental part of the creative process: they reduce possibilities which are useful from the aspects of curiosity, exploration regarding to a specific problem. They act as catalyzers or extrapolators. They amplify components, if the available tools are narrow, their usage can lead to new ways of improvisation in order to present an idea.

In the case of my practice, these constrains consist of the number of colors used (black, white and red where action happens), reduced number of forms (lines, circles and a few geometric objects), or the avoidance of using prerecorded sound samples. It is instead based on simple sound synthesis, basically frequency modulation. It needs to work online in realtime, which leads to many tech related compromises and opens up ways for learning novel things during the implementation process. Hackpact originates from live coding, which means to modify the code on the fly while it is running and changes are interpreted in a live and direct way. The concept had many incarnations: Codevember, 100 days of code, hackpacts, etc. These movements started in the beginning of the 2000s. The manifest here defines that sound should affect the graphical appearance, and vice versa. It then becomes a visual sonic instrument, where those two entities are inseparable.

Speaking of practice, I really like the example brought forward by Ian Bogost in his book "Play Anything". He says that anything can be converted into a playground in the life of humans. A nice example is if someone is trying to leave their home and can't find their keys, then the whole house can be turned into a gameplay where the goal is to find the keys, and the house becomes the game environment. The different areas in the rooms become game objects and elements.

“The arena, the card-table, the magic circle, the temple, the stage, the screen, the tennis court, the court of justice, etc., are all in form and function play-grounds, i.e., forbidden spots, isolated, hedged round, hallowed, within which special rules obtain. All are temporary worlds within the ordinary world, dedicated to the performance of an act apart.” - Johan Huizinga, Homo Ludens

This playground is an area, that is defined by the rules of the game. This area can be the house, this can be a chess board, this can be an abstract process that we saw in the case of a hackpact, where the rules are quite strictly defined. Everything that is not included in this playground, is the rest of the world, that does not take part in the game. It is not necessarily a spatial or physical distinction. If we throw a ball into a net, then it is only an act, where we take an object and throw it somewhere. It doesn't really have a meaning in itself. But if the rules reinterpret this action, and this will have a meaning in an abstract system, such as receiving some points for it, then this action will be placed inside the playground. The border that is between the ordinary world and the inner playground, like a membrane around a cell or a zone, is called the Magic Circle (term by Dutch historian Johan Huizinga). The exposed events are happening inside the magic circle, based on the predefined rules. Everything that is outside of the circle is not part of the game. It might be interesting to add, that these zones can be treated as temporary worlds. They exist temporarily within the outer reality. We construct these in order to provide a scene for performing.

In order to make these temporary zones move we need to record processes, stories, narratives, that are connected to them. These narratives can be interpreted in many ways. There are ways to store creature mythologies, or there are ways to record rules of a game on paper where we define a sequence of game turns. We are using different tools in order to store these information through our cultural history. These tools are also called mnemonic, or memory devices. One such great and well known tool for this is writing. Through writing we can formalize and share knowledge in a very effective way. Snippets of knowledge can be reproduced in exact and consequent forms. Apart of writing, there are many other tools, such as the so called Memory Palaces that were used among indigenous people. These are tactile devices made of pearls and shells, that highlight important artefacts in different stories. They shared information and knowledge without writing among several generations. In fact, there is a similar method called songlines, where people were using their natural environment as visual landmarks for knowledge sharing areas. There, the different elements in the landscape were functioning as different knowledge fragments. While walking on the land, they were singing songs, melodies and sharing stories belonging to different landscapes. An interesting aspect of memory is that it is interconnected with space. Moving in space itself, transporting something from one place to another has always been a very important role in distributing values among our society. Navigation itself is a key element of accessing memories. An interesting example of mnemonic navigation can be found in the oceanic transportation of the Polynesian islands.

They were using special devices for tracing the routes, called stick charts. They were using different sticks to create an abstract map of the territory where they recorded the water streams, wind directions and the waves that were identical to specific areas in relation to the islands. The islands were highlighted with pearls and directions of streams were indicated with longer, rotated rods. What is escpecially surprising with these objects is that they didn't bring them along their journeys, since they wouldn't be able to use them, while struggling with the coordination and maintaining boat functionalities. So these objects in fact act as mnemonic devices. They observed the possible ways ahead of the journey and decided the strategy in advance of the realization of the action according to area-specific variables. When they were travelling along the actual routes, they were recalling the forms from their memories. The charts acted as symbolic links to specific decisions. This way we can treat these stickcharts as code systems, that has been studied by the navigators and their decisions and acts were interpreted based on those instructions.

"Imagine that you could pass through two doors at once. It's inconceivable, yet fungi do it all the time. When faced with a forked path, fungal hyphae don't have to choose one or the other. They can branch and take both routes." - Merlin Sheldrake, Entangled Life

I find the term Unfolding Intelligence quite appropriate and it seems to catch quite well the nature of that transformation that happens when a so called intelligent or conscious reaction happens or a decision process can be found in reality*. This act unfolds into and over something. It is manifesting. The term Unfolding Intelligence includes the fact that intelligence is an active process that happens through a series of events where if the causalities are occurring together accordingly, then a constructive transformation evolves. These are all active processes, where a structured, productive and creative force or energy emerges. Intelligence itself is introduced in a beautiful, poetic manner in the recent book of Merlin Sheldrake, called Entangled Life. The book is a very interesting introduction into the world of fungi and fungal networks, called mycelium that surround us within our biological environment. An important note here is that the human attitude towards artificial intelligence is very human centered, or anthropomorphic. Which means we have a body that contains the brain, which is responsible for the thoughts and decisions as a central organ. If we are navigating through a space, then when we have to cross a door, we have to choose which door we would like to cross, since we can only walk through on one door only at a time. In the case of mycelium, since it has a branching structure, and it is colonizing space in an omnidirectional manner, it can turn over into myriad ways simultanely and find its path to go along. This thought might indicate that maybe there is not always a central brain which is only responsible for the decisions of intelligence, but it can really work in a decentralized way, where decisions are made by different, independent cooperating agents. We can see this in mycelium network dynamics, or the colonies of ants, swarm based animal navigation, emergent systems in nature that can be hardly described through central, serial logic such as the simultane rotation of swarming fish that is faster than the information that they could pass over between them individually.

Regarding to spatial movements and decision making, since ancient times there are structures that can illustrate them in a very convenient way, that are called mazes and labyrinths. A navigator makes decisions while trying to find the relevant path over a labyrinth. The decisions are made in a sequence after each other, which leads to the unfolding of decision trees. The tree archetype itself is also used as knowledge organizing model. A labyrinth can be transformed into a decision tree, if we connect the decision making events with a separate coordinate system. This is a reason why visualizations are really powerful when it comes to understanding complex ideas and knowledge structures without sequential writing systems. Instead of sequenced chunks of information, visualization exemplifies the underlying structures. These labyrinths and their corresponding decision trees are accurate demonstrations of the way we navigate through mental states. The weight of these decisions that a network makes, can be studied in different structures. The underlying model is basically a graph system. A Markov chain, or Markov Graph is a set of weighted decision trees.

On this particular example, weather condition changes can be simulated based on random probabilities. It has nothing to do with our real environment, it is only a model where the decisions have been weighted. The weather can be rainy, cloudy or sunny. The arrows indicate the directions of the transitions between the different states. The numbers indicate the probability of such transitions. From sun to rain, there is a 0.1 (10 percent) probability to change, also, from rain to sun, there is a 0.2 (20 percent) probability to change back to the sunny state. There are many analyses and predictive systems that are based on similar random probability state machines. For example, musical note transitions within melodies can be identical and different that can observed in a Bach fugue, compared to a romantic musical piece. So this can be used for very general purposes. These weighted Markov chains present nicely the results when we variegate chance, probability and randomness with algorithmic thinking and create novel structures from them. My work Songlines* introduces a kind of learning process that can be studied in a playful way by the user. That learning process is built on environmental parameters and the agents inhabiting it.

Such environment can be a labyrinth where we can only turn left or right in order to navigate forward and the agent is a being that is navigating in that labyrinth. For this being, the whole world means only this labyrinth. Its explicit knowledge and experience of the world is only turning decisions between walls. In this world, there are no other elements that it perceives. On one hand, to build up routines, reactions, human relationships, gameplay itself is essential, on the other hand, this can be applied to the area of artificial systems. A typical game context provides an extremely useful set and setting for the act of learning. Positive and negative rewards and feedbacks from the environment can lead to an effective navigation throughout the environment. Later on, the agent can start improvising and find novel solutions to problems. This type of learning process in games is also called Reinforcement Learning. It is another, scalable paradigm in network graphs: as opposed to manually constructed symbolic states that can be found in Markov models, the states of each node (perceptron) in a neural network are composed of abstract weights on the atomic level, between zero and one. Their meaning depends on their role in the patterns of the network, similarly to how neurons function in a brain. This method is used in autonomous driving systems, balancing robots and other physical phenomena, where physical experience can not be exchanged with pre-recorded historical data. This way agents can figure out the rules of a system in an unsupervised manner: it would be too difficult to hardwire those decision trees into those robots and physical actors by traditional programming routines, since there are too complex scenarios to consider in real world environment, from lighting conditions to temperature anomalies, material qualities, raining and so on. Games, computational and generative media can be used to simulate ecological models including the complex, global ecosystems. Clusters of climate change, including many different problematic causalities can be simulated and extrapolated through predictive, speculative and future related scenarios. In many cases they can augment regular, everyday thinking via testing associative models that can be simulated in a compressed time interval so we don't have to wait years to test long term causalities in reality. It seems these advanced networks become more and more embedded in the way we humans tend to think and act.

“The choices we are faced with today are especially important because digital technology so dramatically increases the ‘space of the possible’ that it includes the potential for machines that possess knowledge and will eventually want to make choices of their own.” - Albert Wenger, World After Capital

*: my interactive artwork Songlines (with identical name as the original lecture performance) has been exhibited at the Generative Unfoldings exhibition as part of the Unfolding Intelligence Symposium, organized by the Massachusetts Institute of Technology

Bogost, Ian: Play Anything (2016)
Bostock, Mike: Visualizing Algorithms (2014)
Bunz, Mercedes: The Force of Communication (2021)
Chang, Y Alenda: Playing Nature (2019)
Eco, Umberto: The Search for the Perfect Language (1995)
Huizinga, Johan: Homo Ludens (1955)
Matsumoto, Shoukei: A Monk's Guide to a Clean House and Mind (2018)
Penny, Simon: Making Sense (2018)
Sheldrake, Merlin: Entangled Life (2020)
Wenger, Albert: World After Capital (2021)
Wickers, Ben & McDowell, K.A (ed): Atlas of Anomalous AI (2021)

Thinking through lines

In indigenous cultures, abstract line drawings often serve as a means of preserving memories and cultural knowledge. These drawings can range from simple geometric patterns to more complex, symbolic representations, and are often used in various cultural contexts such as storytelling, rituals and record-keeping. These drawings can also provide valuable insights into the history and beliefs of these cultures, and are considered an important part of their cultural heritage.

Artist and Bauhaus teacher Paul Klee described the first category of line within his Pedagogical Sketchbook of 1925 as

An active line on a walk, moving freely, without goal. A walk for a walk’s sake. The mobility agent is a point, shifting its position forward. […] The limitation of the eye is its inability to see even a small surface equally sharp at all points. The eye must “graze” over the surface, grasping sharply portion after portion, to convey them, to the brain which collects and stores the impressions. The eye travels along the paths cut out for it in the work.

Klee’s allegorical exploration of drawing theory is one deeply rooted within nature — of which humans are very much part of, rather than competing with. His understanding of both making and receiving drawings is built through relations to natural forms and animal instincts. As contemporary psychological research indicates, it has also been shown that infants, stone-age tribe members, and even chimpanzees are able to recognize line drawings. We even see line representation used by insects in bio-mimicry. These findings rule out any strong version of culture-based acquisition for understanding line drawings, although clearly there are many culturally based conventions used in line drawings.

Apart of using lines as hand drawn representations, early computer graphics also worked with lines using math and algorithms to draw pictures with plotters and printers and later, visual displays and computer screens. Some common ones were used for things like making sure lines were drawn in the right order, or making sure parts of pictures were hidden when they should be: usually named after their creators (Sutherland-Hodgman, Bresenham, etc), sorting, intersection, line drawing and other algorithms were introduced to perform operations such as line clipping, visibility determination, and rendering of images. These algorithms were crucial for the development of early computer graphics, and continue to be used in modern computer graphics systems.

Autocolliders

Autocolliders is a series of artworks, that consist of straight lines with an ordered distribution in a finite two dimensional space. They rhyme upon themselves, they repeat, they rotate, some of them never meet, some of them intersect occasionally, disrupting each others trajectories. Upon constructing a scene, the empty space is divided along a grid with different resolutions.

At each given point, a line is growing out into a direction of six predefined angles with varying lengths. These vectors are then repeated and transformed occasionally into perpendicular directions of their orientations, creating a dense, ordered structure of overlapping layers. Like flipping coins or rolling dice, the provided instructions can lead to interesting visual choreographies, depending on how their random chance operations are distributed.

These basic rules act as the grounding for the image architecture. Interesting twists and complexities start to emerge within the system when collision avoidance is added to the segments. Empty spaces, irregular zones and glitches in the formerly continuous shapes cause sudden disruptions that shift the focus of attention from the foreground into emptiness, leaving space for imaginative territories. The colours of the scene are selected from highly constrained, manually curated palettes that were created with an intention on cleanness, simplicity and convenience.

Influenced by patterns and systems that can be found in abstract textile pieces from the Bauhaus, they also inherit from the soft minimal worlds of Japanese lithographs and other oriental prints, among raw, reduced and schematic color variations of electronic imagery and computer aesthetics.

The artwork implements two features that are generated and stored as parts of the token metadata. Each piece consists of a unique combination of Environment and Palette. While environmental features are selected from a set of words that describe spatial properties (sparse, frequent, dense, berserk, instinct), there are ten predefined color palettes which are composed from names of different gems and minerals that have vivid and colorful properties by their nature (pearl, onyx, opal, azurite, blizzard, tourmaline, garnet, tanzanite, iolite, quartz). As an example, the following two iterations have both different environmental densities and different palette components:

In physics, colliders are used as a research tool by accelerating particles to very high kinetic energy and letting them impact other particles. Analysis of the byproducts of these collisions gives scientists good evidence of the structure of the subatomic world and the laws of nature governing it. These may become apparent only at high energies and for tiny periods of time, and therefore may be hard or impossible to study in other ways. These principles can easily resemble the ways how human attention and memory work, where caching an idea, a feeling or other intuitive processes are also very challenging for a constantly scattered mind. Cognitive research and several ancient philosophies refer to these restless chatterings as manifestations of the monkey mind. There have been several historical techniques developed in order to navigate these mental spaces: memory palaces, tactile infrastructures, mnemonic devices, method of loci, search trees and permutation algorithms are just a few of the myriad ways we humans shape and access the multilayered, omnidirectional scope of our cognitive environment.

Technical mediation

There are fundamental differences between traditional generative art practice and the one that uses blockchain as a medium. The latter led to the practice of long-form algorithmic art, that uses deterministic strategies (seeds, derived from a transaction hash and other cryptographic functions), combined with immutable properties of the underlying technologies. Speaking of the nature of an algorithm, with long-term generative art, collectors and viewers become extremely familiar with the output space of a program. Autocolliders is somewhere in-between short-form and long-form properties, since the features it implements are not derived directly from transaction parameters, as they are provided as an extra layer of metadata for the collection. However, throughout working with the code, I gradually developed a conceptual scope of what exactly the program is capable of generating, and how likely it is to generate one output versus another, so I kept that agency in mind during the whole workflow with the piece. This part of the process was a bit different compared to my pieces on fx-hash, since drops on zora work differently: here, features and token metadata need to be configured during the initial deployment of the whole series instead of being generated for every token upon each mint. All iterations within the series were created exactly from the same code with no manual intervention. Features are derived from the actualization of the pieces when running and saving them as scalable vector graphic formats. The feature parameters are saved into the names of the output files along unique timestamps:

fileName = `autocoll-${densityNames[cNum]}-${paletteNames[cPalette]}-${Date.now()}`
save(`${cFileName}.svg`)

Where densityNames are the pre-defined spatial properties, cNum is the actual index to use from them, paletteNames are the predefined color names and cPalette is the actual selected colour index. The results are then loaded with another script that reads each filename and generates token metadata based on those parameters for all the artworks into one csv file, linking metadata to the actual svg files:

let fs = require('fs')
let files = fs.readdirSync('./svgs')

for (let i=0; i<files.length; i++) {
    let words = ["poetry", "permutation", "notation"]
    let rnd = words[Math.floor(Math.random() * 3)]
    let cToken = `Autocollider #${i+1},"Line segment ${rnd}, CC BY-SA 4.0","${files[i]}","${files[i].split("-")[1]}","${files[i].split("-")[2]}",`
        fs.appendFile('_tokens.csv', "\n"+cToken, function (err) {
            if (err) throw err
        }) 
}

Preparing the body of the artwork using custom code snippets and modular algorithmic extensions to be deployed on the blockchain resonates with Bruno Latour’s idea of “technical mediation”, which stipulates that agency is distributed among various actors. The medium defines the way one will interact with the creative process, and this will result in new forms of cognitive modalities and aesthetic outcomes. He describes the example of how a person with a gun — in themselves two separate things — combine to form a new thing. Latour highlights how relationships between humans and technologies create complex, hybrid constructs that are often inseparable.

Complex and hybrid structures run throughout generative art also, being short-form, long-form, or anything in-between, producing new, unique hybrid constructs: unlike with copies of discrete artworks, both digital and physical, these generative artworks do not have a prototypical image, but a possibility space constructed of the derivations of their underlying algorithms.

Less Lines

While working with static, aleatoric compositions is essential within my workflow, temporal aspects are also important areas of my research. The relationship between movements, image structures and sounds are still open ended cultural territories with many possibilities for further exploration. Apart of classical animations, clips and video games, new temporality of network based media introduces many conceptual layers for sonic tinkering.

Within the physical world, rhythmical movements, events, interactions all cast sounds into their environments as byproducts of their moving energies. As a creative process, the act of sonification builds on these properties and there are many ways to express emotions and semantic relations by converting events and quantitative data into sounds. Within my previous works, I often used realtime data feeds, user interactions or simple mathematical algorithms as driving forces for producing sounds, creating different sonic textures and harmonies. Coming from a visual programming background, I like to think of sound as continuous flow of entities through different digital signal processing (DSP) components, where synthesized sources are traveling over different filters and effect components before hitting their destination, aka the speakers.

I usually develop my own WebAudio components, that consist of generators and modifiers that can be chained together easily. Just like in any dynamic, cybernetic system, environmental events and temporal shifts both affect the behavior and outcomes of the audio engine. If a state changes, an ADSR envelope might be triggered, a filter coefficient or a delay time might be set to a different range as time passes and new events occur. This method of working with sounds is very similar to building auditory displays or musical atmospheres for games, where game states are affecting the outcome. The intention here is to bypass regular musical structures and narrative elements and let the algorithmic choreographies unfold by their own internal rules without any manual intervention. Somewhat similar to working with visual math algorithms or data visualization systems.

Within Less Lines, the elements are moving slowly, formulating a quasi-still image, with no progress in time, a perpetual resonance stuck in a very slow, infinite loop. I like to think of these subliminal movements as groundings for the audio composition, formulating simple sine-wave oscillations in the audible domain, ending up as continuous ambient drone harmonies. This timeless, horizontal trajectory is then disrupted with sudden glitches and bursts, within both the sounds and the visuals, in forms of disappearing and reappearing elements.

These are the collisions of the monkey mind, the artifacts that disrupt the calmness of the clear presence, similar to widely known mental states that cloud the mind and manifest in unwholesome actions within our everyday lives and experience of our environment. The system we observe is in a continuous struggle of eliminating these artifacts, trying to clear the space, to make less lines and leave less obstacles within the scene.

Details

Both collections are built with custom, standard ERC-721 minting contracts, with optional extensibility frameworks, which means that these contracts allow other smart contracts to interact with themselves, with different permission models. This method involves cultural composability, which is based on the idea that creators should own their digital goods, via true digital ownership and creative sovereignty. As an experiment, a few editions of Less Lines are being listed on Foundation for enhancing visibility, while other editions are still available to collect on Manifold Gallery. Autocolliders is live and available to mint on Zora.