Newswise — What did Vincent van Gogh actually paint and draw? Paintings and drawings fade, so researchers from TU Delft are using deep learning to digitally reconstruct works of art and discover what they really looked like. ‘What we see today is not the painting or drawing as it originally was,’ says researcher Jan van der Lubbe.
Snow-Covered Field with a Harrow
One of the paintings the TU Delft researcher Jan van der Lubbe regularly refers to when this subject is discussed, is Snow-Covered Field with a Harrow (after Millet) which can be seen in the Van Gogh Museum. ‘That work’s colours have faded. The edges of the painting, which were protected by the frame, show that the dominant green was originally more purple,’ explains Van der Lubbe. ‘In collaboration with various partners, the museum is trying to digitally reconstruct changes like these. This applies to drawings too. And we're helping them.’
Broad cooperation
How can we find out what colour changes there have been in Van Gogh's work? This question has been occupying the minds of various researchers for a very long time. Within the research project Reassessing Vincent van Gogh (ReViGo), subsidised by the Dutch Research Council (NWO), curators, restorers, art historians and scientists from the Van Gogh Museum and the Cultural Heritage Agency of the Netherlands are collaborating in mapping these changes. This research makes use of both traditional and modern methods of analysis, advanced image analysis as well as machine learning methods based on information technology and artificial intelligence. It is these last two methods that researchers from the Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS) can help with.
CNN
In an article published in Springer’s Machine Vision and Applications earlier this month, EEMCS researchers Jan van der Lubbe, Marco Loog and Yuan Zeng presented a method that uses deep learning to establish what a drawing would have looked like when it was made. ‘Deep learning is a learning method based on artificial neural networks,’ Jan van der Lubbe explains. ‘In the case of the Van Gogh drawings, we use Convolutional Neural Networks (CNN) technology. This technology is used for image analysis particularly and applied, specifically, where there is no explicit physical model of the underlying process. You make a scan of a Van Gogh, take down the RGB values of the pixels you see and then let the technology run a variety of calculations using these values. The calculations eventually yield a new image which is meant to approximate “a younger version” of a Van Gogh drawing. As far as we know, CNN has not been used to digitally reconstruct art works before.’
Voyage of Discovery
The research focuses, amongst other things, on the reconstruction of a Van Gogh drawing. Van der Lubbe explains: ‘You could start off with an explicit model, but this would be like working with a mathematical formula: you feed it something, and it spits out something. Neural networks, however, are implicit. You make observations and then allow a learning method to work out the relationships between various parameters. All this data allows a method like CNN to predict what an old drawing would have looked like years ago. It's a way of going back from the present to the original year of manufacture of the drawing, using knowledge from the past. What we do is actually a voyage of discovery. We want to understand to what extent a reconstruction like this can be arrived at using a learning method.’
Pieces of the puzzle
As members of the broader ReViGo project, computer scientists from Delft are using information gleaned from art history to feed computer models. In 2015, the Dutch Research Council's NWO Wetenschap made a 10-minute broadcast about the joint research project. In this broadcast (from 06.05), Jan van der Lubbe explains how, with the help of various scientific puzzle pieces, he is trying to make a digital reconstruction. Computer scientist Van der Lubbe can also be seen in this episode of De Kennis van Nu.
RSS