AI Sculpting

The human interest in the manipulation of objects for an aesthetic purpose is very antique. By experimenting with different tools and techniques the old masters studied how material can be shaped into form. Transferring this curiosity to the age of technology, in 2020 we engineered a machine-learning process to observe the evolution of an AI learning how to sculpt a 3D model. While most of our art projects are guided by an open process the aim is to achieve a visual result in the end. With this project it was different. To best experience AI as the creator and to learn from it became our main pursuit.

We were guided by the question of how our role as the designer is changing when human creativity and artificial intelligence co-create. In the course of the AI’s sculpting process we were inspired by the unpredictable strategies and outcomes of the reinforcement learning: an experimental approach par excellence, which we guided, observed and visualized.

The Process — Co-creation with AI

When it comes to studying a basic artisan technique humans are mostly motivated by curiosity, while the machine is driven by efficiency — an interesting tension worth exploring. With its underlying setup, this machine-learning process is related to the studies ancient sculptors undertook in the very beginning. Reminiscent of the helpers assisting them in the old workshops, the AI developed different strategies seeking constant improvement on its way to form a given object. By feeding it with different tools, rules and rewards through reinforcement learning, we were steering the process, but not the outcome.

The Training – Reinforcement Learning

With the given goal to sculpt a 3D model, the AI was trained through reinforcement learning based on rewards and punishments. The agent, which is defined as a certain machine-learning model, was programmed to seek maximum reward. In the voxel-based environment infinite data and a clear reward structure was provided for the agent to move through. The starting state of the environment was one big cube. Out of this, the agent needed to remove mass to get closer to a predefined target state. With each step, the agent could decide where to go and if to remove a mass of voxels around itself and how. To enable its learning, it was conditioned in a specific way: when extraneous mass was removed it was rewarded and it was given a penalty when mass that ought to be part of the final sculpture was removed. All the technical ML parts were implemented in Unity3D and Unity’s ml-agents library was used for implementing the reinforcement learning.

Detailed information about the setup and the technical pipeline of the training process can be found in this paper.

Trial & Error – Efficiency is the Way

This technical set-up defined the playground for the machine to learn sculpting. Through trial and error the agent created strategies to achieve the desired shape by removing mass from the original cube. By seeking maximum efficiency the sculpting AI was improving rapidly. One very striking outcome was the enormous parallelism in the training process: the AI was able to perform many training sets at the same time, which means a great variety of sculptural output. Observing the evolution of the learning curve including the strategies, behavior and visual output, we started to experiment with different parameters and predefined rules.

By studying the visual results along the process, our role as a designer became one of a curator: we selected a number of outcomes, which we defined as sculptures to present them as digital artworks. This is where human creativity came to play again: What would we consider as a presentable manipulation of the cube? How would we orchestrate the machine-made output? Which sculptures would we choose and what kind of environment would we create in order to exhibit them?

Visualizing the Process — Traces of Machine Learning

In the search for new ways of creative expression with AI, we experimented by implementing different tools for the agent to choose from. Through the reinforcement learning set-up, this allowed more complexity and resulted in a variety and depth of strategies as well as visual outcomes. We found the use of different tools was vital to the learning process. By regarding the traces of different tools on the surface of the emerging shapes, the sculpting process becomes visible.

Various tools have their own unique fingerprint — from rough to fine and quick to slow — each decision the agent undertakes by choosing a different tool or way of orientation has a different visual outcome and impact.

Through the Eyes of the Machine — Changing Perspectives

To explore the concept of co-creation even further, we decided to take on the agent’s view. What did the process look like from the creator’s perspective? Through all steps of the process we found multiple vantage points that could be highlighted. In the attempt to interpret the agent’s decision-making we experimented with visualizing AI data such as confidence, or penalty and reward for individual steps within the 3D environment. By highlighting the agent’s path through the block, for example, we gained another perspective on the creation process itself.

The Conclusion — Co-creation with AI is infinite

What have we learned by observing the machine learning process? With AI as a creator, we have not only gained new inspiration, but also new perspectives on the relationship between man and machine in digital art. Above all, we have also questioned our own role as creators. Rather than leaving creation to AI, we need to find ways to integrate it into the creative process. We take technology as a starting point, a tool, a source of inspiration and a creative partner. The human aspect is quite clear in this: We choose the rules and define the approximate output. However, in the end it was the interplay between our human choices and the agent’s ability to find the best solutions and occasionally surprise us. This made the process rewarding to us and shows the true potential of an AI based co-creation process.

By understanding how the machine works, we can reinterpret it, rethink it, and develop new strategies or systems to implement it, enriching our creativity and the creation process on many levels. In the age of artificial intelligence, creative concepts and ideas become more relevant again as we determine and actively shape the environment and process and thus the impact of this AI evolution.