Software developed at University of Wisconsin-Stout is helping museums around the world preserve pieces of cultural heritage for future generations by creating 3D models that accurately capture every detail of the color, texture and brightness of objects.
Kintsugi 3D is the creation of Michael Tetzlaff, an assistant professor of computer science, and numerous UW-Stout students. The software has been used by notable institutions worldwide to create models of artifacts in their collections, including the Rijksmuseum in Amsterdam; the Museum of Cultural History in Oslo, Norway; the National Gallery of Art in Washington, D.C; and the Minneapolis Institute of Art (Mia).
“Documenting the art is incredibly important, and we want to get it out in the world however we can,” explained Mia Lead Collections Photographer Charles Walbridge, an avid user of the software. “The Kintsugi project has helped us be able to share more accurate 3D models that we’ve been making in the museum.”
Kintsugi 3D gets its name from the Japanese art of kintsugi, a method of repairing broken pottery with lacquer and precious metals to create unique pieces that celebrate, rather than camouflage, their fractures. Likewise, the software allows photo professionals to capture true-to-life 3D models of sculptures, paintings and other cultural heritage objects.
While existing photogrammetry software – which combines hundreds of photos from different angles to create a 3D model – can effectively record many aspects of an object’s appearance, it falls short in capturing the essence of materials and textures, Tetzlaff said. In particular, photogrammetry software struggles with specularity – the reflectivity of a surface and how it varies – particularly with objects that are shiny or metallic. Kintsugi 3D solves this problem by creating much more true-to-life renderings, capturing the sparkle of a bit of gold leaf or the depth of a brush stroke in oil paint.
“The big thing that our software does is it allows you to process those photographs and then builds a compact representation of what the material of the object looks like so that we can create new, interactive, dynamic 3D renderings where the user can change the viewpoint, change the lighting,” Tetzlaff explained. The resulting 3D models can be rotated by the user, with virtual light playing off the details of the surfaces, whether they are dull or dazzling, giving a much more realistic representation of the item. This is particularly valuable for accurately portraying older objects, whose surfaces may have been damaged or worn unevenly over time.
These models can be shared widely, helping scholars and the public appreciate cultural heritage objects. And they also serve as a kind of insurance, allowing the objects to live on – virtually, at least – if they are damaged or lost.
Software funded with federal, state grants
Tetzlaff said the software grew out of work he did as a doctoral student, and it received a significant boost in 2023 thanks to a $75,000 National Endowment for the Humanities grant received in conjunction with Mia and Cultural Heritage Imaging, a San Francisco-based nonprofit. This and other funding, including from UW-Stout’s Office of Research & Sponsored Programs as well as a recent grant from WiSys, has helped pay for students to work to improve the software.
Luke Denney, a computer science major from Byron, Minnesota, who graduated in May, has put hundreds of hours into Kintsugi 3D over the past two years. In fact, he even gave the software its current name, inspired in part by a “Star Wars” character who repaired his helmet kintsugi-style.
Denney has worked on the software’s loading screen, trying to make it more user-friendly, particularly for those who aren’t programmers themselves. “A big part of this project was user experience design,” Tetzlaff explained. “There was basically working software that I had developed, but it wasn’t very usable. A lot of the research was working with user experience experts and stakeholders in cultural heritage to figure out what works for them.”
Denney shared the work he’s done on Kintsugi 3D at UW-Stout’s Research Day as well as at two other prestigious venues this spring: the National Conference on Undergraduate Research in Pittsburgh and the Universities of Wisconsin’s Research in the Rotunda event in the state Capitol in Madison.
Denney said the project has been invaluable in preparing him for his future career, in part because working on Kintsugi 3D required him to become familiar with a massive codebase. He expects that skill will come in handy when he begins his job later this year at IBM in Rochester, Minnesota.
In addition to navigating thousands of lines of code and gathering user feedback to improve software features, Denney said the project helped him learn to solve problems independently. “Something I focused on more last summer was taking more time before I ask a question,” Denney said. “When I’d run into a roadblock and have the temptation to go ask for help, I thought, ‘Wait, I want to give myself more time to figure this out before asking.’ And most of the time I was successful with that, so that felt really good,” he said.
Denney is just one of more than a dozen UW-Stout students from a variety of disciplines who have contributed to the software. Among them is Chloe Ferguson, master of fine arts in design, Spring Valley.
Ferguson began working as a user experience designer on the program last fall, gathering data to see what features users wanted from the program. Working on the project has given Ferguson a better understanding of the science behind textures and materials, as well as the rewarding opportunity to work with stakeholders on improving the software, she said.
“User experience is all about considering and analyzing the user, striving to make your content more useful and accessible to them,” Ferguson said, noting the skills she is building now will be directly applicable to her future career.
“I feel proud of my work, especially with how directly this program has benefited cultural heritage groups,” she said. “We regularly meet with our stakeholders at the Minneapolis Institute of Art and Cultural Heritage Imaging, which has allowed me to see the impact of my work on these groups firsthand.”
Other students and recent graduates who have worked – or are still working – on the project include master of fine arts in design candidate D. Magda Hannen; game design and development majors Isabel Smith, Elliot Duffy, Lou Brown and Augusto Freitas; and computer science majors Tyler Betanski, Jacob Buelow, Victor Mondragon, Ian Anderson, Zoe Cuthrell, Blane Suess, Isaac Tesch, Nathaniel Willius, Atlas Collins, Kyle Boatwright, Melissa Kosharek, Chris Medina, and Jacob Sletten.
Software draws focus to UW-Stout
Walbridge, of Mia, is one of those who meets frequently with the Kintsugi 3D team to provide feedback. He has been part of Kintsugi 3D and related projects for years, having met Tetzlaff when the latter was a doctoral student at the University of Minnesota working on 3D photography and photogrammetry projects at Mia. “It’s been such a pleasure to continue to work on this with Michael and his team,” said Walbridge, who praised the backing the project has received from UW-Stout.
Walbridge has worked to get other professionals involved in Kintsugi 3D as well, making presentations about the software and how to use it, including a training visit to the Metropolitan Museum of Art in New York in April.
Among the objects from the Mia collection scanned with Kintsugi 3D is a sculpture of 19th-century dancer Loie Fuller by Czech artist Joseph Kratina. In digital photos the bronze sculpture appears to be a yellowish-brown, but it practically comes to life in a three-dimensional scan made with Kintsugi 3D. As a user rotates the image virtually, light plays against the sculpture’s gold patina, and shadows of the dancer’s arms are cast across her voluminous skirt. The software allows users to examine the sculpture in greater detail, giving the impression of walking around it in a well-lit gallery.
“None of the current forms of 3D scanning are able to capture this kind of specularity,” Walbridge explained. The high level of realistic detail provided by the software is a boon to scholars studying such objects online, Walbridge said, and making these models available digitally fits with the ethos, shared by institutions such as Mia, of making cultural heritage freely and publicly available.
The software’s use by professionals at the National Gallery of Art has led to a collaboration between that institution and UW-Stout’s game design program, Tetzlaff added.
“It’s a nice feedback loop where we’re supporting the museums and their work with the software, but then they’re becoming more aware of Stout, and that’s leading to additional opportunities for Stout students,” he said.
