The human interest in the manipulation of objects for an aesthetic purpose is very antique. By experimenting with different tools and techniques the old masters studied how material can be shaped into form. Transferring this curiosity to the age of technology, in 2020 we engineered a machine-learning process to observe the evolution of an AI learning how to sculpt a 3D model. While most of our art projects are guided by an open process the aim is to achieve a visual result in the end. With this project it was different. To best experience AI as the creator and to learn from it became our main pursuit.
What happens when AI becomes the creator? How can human creativity and machine-made output feed each other? Curious about co-creating with AI, we initiated the »AI Sculpting« project.
We were guided by the question of how our role as the designer is changing when human creativity and artificial intelligence co-create. In the course of the AI’s sculpting process we were inspired by the unpredictable strategies and outcomes of the reinforcement learning: an experimental approach par excellence, which we guided, observed and visualized.
The Process — Co-creation with AI
When it comes to studying a basic artisan technique humans are mostly motivated by curiosity, while the machine is driven by efficiency — an interesting tension worth exploring. With its underlying setup, this machine-learning process is related to the studies ancient sculptors undertook in the very beginning. Reminiscent of the helpers assisting them in the old workshops, the AI developed different strategies seeking constant improvement on its way to form a given object. By feeding it with different tools, rules and rewards through reinforcement learning, we were steering the process, but not the outcome.
The core aspect of this co-creation process is that we — to an extent — let go of control. We seek an equilibrium of setting a frame, giving instructions and observing the outcome: a simple cube being transformed into an increasingly recognizable shape with every iteration.
The Training – Reinforcement Learning
With the given goal to sculpt a 3D model, the AI was trained through reinforcement learning based on rewards and punishments. The agent, which is defined as a certain machine-learning model, was programmed to seek maximum reward. In the voxel-based environment infinite data and a clear reward structure was provided for the agent to move through. The starting state of the environment was one big cube. Out of this, the agent needed to remove mass to get closer to a predefined target state. With each step, the agent could decide where to go and if to remove a mass of voxels around itself and how. To enable its learning, it was conditioned in a specific way: when extraneous mass was removed it was rewarded and it was given a penalty when mass that ought to be part of the final sculpture was removed. All the technical ML parts were implemented in Unity3D and Unity’s ml-agents library was used for implementing the reinforcement learning.
Detailed information about the setup and the technical pipeline of the training process can be found in this paper.
Trial & Error – Efficiency is the Way
This technical set-up defined the playground for the machine to learn sculpting. Through trial and error the agent created strategies to achieve the desired shape by removing mass from the original cube. By seeking maximum efficiency the sculpting AI was improving rapidly. One very striking outcome was the enormous parallelism in the training process: the AI was able to perform many training sets at the same time, which means a great variety of sculptural output. Observing the evolution of the learning curve including the strategies, behavior and visual output, we started to experiment with different parameters and predefined rules.
In a scenario like this, we collaborate with AI. The outcome is not a reproduction of our creativity, it is unforeseen. We receive unprecedented results, which re-inspire our way of creating, highlighting the importance of exploring the concept of co-creating with AI.
By studying the visual results along the process, our role as a designer became one of a curator: we selected a number of outcomes, which we defined as sculptures to present them as digital artworks. This is where human creativity came to play again: What would we consider as a presentable manipulation of the cube? How would we orchestrate the machine-made output? Which sculptures would we choose and what kind of environment would we create in order to exhibit them?
Visualizing the Process — Traces of Machine Learning
In the search for new ways of creative expression with AI, we experimented by implementing different tools for the agent to choose from. Through the reinforcement learning set-up, this allowed more complexity and resulted in a variety and depth of strategies as well as visual outcomes. We found the use of different tools was vital to the learning process. By regarding the traces of different tools on the surface of the emerging shapes, the sculpting process becomes visible.
Various tools have their own unique fingerprint — from rough to fine and quick to slow — each decision the agent undertakes by choosing a different tool or way of orientation has a different visual outcome and impact.
Limits, rules and aesthetic: this aspect of the process depicts how we as creatives do not generate the end result as a 3D model. Instead we train the AI to follow a human inspired creation process, which it then implements in its own way.
Through the Eyes of the Machine — Changing Perspectives
To explore the concept of co-creation even further, we decided to take on the agent’s view. What did the process look like from the creator’s perspective? Through all steps of the process we found multiple vantage points that could be highlighted. In the attempt to interpret the agent’s decision-making we experimented with visualizing AI data such as confidence, or penalty and reward for individual steps within the 3D environment. By highlighting the agent’s path through the block, for example, we gained another perspective on the creation process itself.
The Conclusion — Co-creation with AI is infinite
What have we learned by observing the machine learning process? With AI as a creator, we have not only gained new inspiration, but also new perspectives on the relationship between man and machine in digital art. Above all, we have also questioned our own role as creators. Rather than leaving creation to AI, we need to find ways to integrate it into the creative process. We take technology as a starting point, a tool, a source of inspiration and a creative partner. The human aspect is quite clear in this: We choose the rules and define the approximate output. However, in the end it was the interplay between our human choices and the agent’s ability to find the best solutions and occasionally surprise us. This made the process rewarding to us and shows the true potential of an AI based co-creation process.
By understanding how the machine works, we can reinterpret it, rethink it, and develop new strategies or systems to implement it, enriching our creativity and the creation process on many levels. In the age of artificial intelligence, creative concepts and ideas become more relevant again as we determine and actively shape the environment and process and thus the impact of this AI evolution.
- Creative Direction: Cedric Kiefer
- Production: Tobias Ziegler
- ML / Reinforcement Learning: Mark Tensen
- Lead 3D Artist: Alexander Hahn
- Code & Design: Norman Wassmuth, Piotr Rymer, Bernd Marbach