Art and the Algorithm: A Computer Program Predicts Paint Preferences

Love the thick brushstrokes and soft color palettes of an Impressionist painting like those of Claude Monet? Or do you prefer the bright colors and abstract shapes of a Rothko? Individual artistic tastes have a certain mystique, but now a new study from Caltech shows that a simple computer program can accurately predict which paintings a person will like.

The new study, published in the journal Nature Human Behaviour, used Amazon’s Mechanical Turk crowdsourcing platform to recruit more than 1,500 volunteers to rate paintings in the genres of Impressionism, Cubism, abstract and color field. The volunteers’ responses were fed into a computer program, and then, after this training period, the computer was able to predict the volunteers’ artistic preferences much better than would happen by chance.

“I used to think that art assessment was personal and subjective, so I was surprised by this result,” says lead author Kiyohito Iigaya, a postdoctoral researcher who works in the laboratory of the Caltech psychology professor John O’Doherty.

The results not only demonstrated that computers can make these predictions, but also led to a new understanding of how people judge art.

“The main point is that we get insight into the mechanism people use to make aesthetic judgments,” O’Doherty says. “That is, people seem to be using elemental image features and combining them. That’s a first step to understanding how the process works.”

In the study, the team programmed the computer to break down a painting’s visual attributes into what they called low-level characteristics – traits like contrast, saturation and hue – as well as high-level features, which require human judgment and include traits such as whether the painting is dynamic or still.

“The computer program then estimates how much a specific feature is taken into account when making a decision on the degree of appreciation for a particular work of art,” says Iigaya. “Low-level and high-level characteristics are combined when making these decisions. Once the computer has estimated this, it can successfully predict a person’s taste for another never-before-seen work of art.”

The researchers also found that the volunteers tended to fall into three general categories: those who like paintings with real objects, such as an impressionist painting; those who love colorful abstract paintings, like a Rothko; and those who love intricate paintings, like Picasso’s cubist portraits. The majority of people belonged to the first category “real life object”. “A lot of people liked impressionist paintings,” says Iigaya.

Additionally, the researchers found that they could also train a deep convolutional neural network (DCNN) to learn how to predict the volunteer’s artistic preferences with a similar level of accuracy. A DCNN is a type of machine learning program, in which a computer receives a series of training images so that it can learn to classify objects, such as cats versus dogs. These neural networks have units that are connected to each other like neurons in a brain. By changing the strength of the connection from one unit to another, the network can “learn”.

In this case, the deep learning approach did not include any of the selected low- or high-level visual features used in the first part of the study, so the computer had to “decide” which features to analyze by himself.

“In deep neural network models, we don’t know exactly how the network is solving a particular task, because the models learn on their own much like real brains do,” Iigaya says. “It can be very mysterious, but when we looked inside the neural network, we could tell it was building the same categories of features that we selected ourselves.” These results suggest the possibility that features used to determine aesthetic preference may arise naturally in brain-like architecture.

“We are now actively investigating whether this is indeed the case by examining people’s brains as they make these same types of decisions,” O’Doherty says.

In another part of the study, the researchers also demonstrated that their simple computer program, which had already been trained on artistic preferences, could accurately predict which photos volunteers would like. They showed volunteers photographs of swimming pools, food and other scenes, and saw results similar to those involving paintings. What’s more, the researchers showed that reversing the order also worked: after first training volunteers in photos, they could use the program to accurately predict the subjects’ artistic preferences.

Although the computer program succeeded in predicting the artistic preferences of the volunteers, the researchers say there is still a lot to learn about the nuances that go into an individual’s tastes.

“There are aspects of unique preferences for a given individual that we failed to explain using this method,” says O’Doherty. “This more idiosyncratic component may relate to semantic characteristics, or the meaning of a painting, past experiences, and other individual personal traits that might influence the evaluation. It may still be possible to identify and to learn these characteristics in a computer model, but to do so will involve a more detailed study of each individual’s preferences in a way that may not generalize across individuals as we have found here.”

Reference: Iigaya K, Yi S, Wahle IA, Tanwisuth K, O’Doherty JP. Aesthetic preference for art can be predicted from a mix of low- and high-level visual characteristics. Nat Hum Behav. 2021;5(6):743-755. do I: 10.1038/s41562-021-01124-6

This article was republished from the following materials. Note: Material may have been edited for length and content. For more information, please contact the quoted source.

Gordon K. Morehouse