MIT’s GIFT framework improves CAD designs by turning its failures into training data

Creating a realistic 3D model from a simple image remains surprisingly difficult for artificial intelligence. While AI can generate impressive pictures, producing the precise CAD code engineers use to design aircraft parts, cars, and consumer products is far more challenging. Even a small mistake can make a design unusable. Now, researchers from MIT, IBM, and Red Hat have developed a framework that significantly improves the ability of AI systems to turn 2D images into functional CAD programs, producing more accurate designs while reducing inference computation by about 80 percent. This new research work tackles one of the biggest obstacles in AI-driven engineering, which is the shortage of high-quality CAD training data. Existing image-to-CAD systems need vast numbers of examples linking 2D images to working CAD programs, but creating those datasets is expensive and labor-intensive. Instead of relying on more human-created data, the new approach teaches AI using the mistakes it already makes. “We want to obtain data augmentation that is informed by the model itself,” lead study author and AI researcher Giorgio Giannone said. Turning failed designs into training material The team’s framework, called Geometric Inference Feedback Tuning (GIFT), works with vision-language models that convert a 2D image and text description into Python code that can be executed in CAD software to create a 3D object. To understand where a model struggles, GIFT asks it to solve the same CAD problem multiple times in parallel. Some outputs are correct, some are close to the target design, and others are much further away. According to the researchers, these ‘near-misses’ are the most valuable because they reveal exactly what the model does not yet understand. The system executes each generated CAD program and compares the resulting 3D geometry with the target using a standard engineering metric known as Intersection over Union (IoU), which measures how closely two 3D shapes match. Rather than discarding imperfect outputs, GIFT identifies those that are close to correct and uses them to generate new training examples. Giannone explained, “If we sample the model 10 times and it generates 10 correct answers to the same problem, then there is not much for it to learn. We care about the in-between cases, where the model might only solve the problem 50 percent of the time.” Teaching the model how to fix its own errors GIFT improves the model through two complementary techniques. The first, called GIFT-REJECT, keeps alternative CAD programs that generate the correct geometry even if the code differs from the original reference solution. Traditional AI training often assumes there is only one correct CAD program for each object, even though engineers can create the same part in many different ways. By exposing the model to multiple valid solutions, GIFT broadens its understanding of how CAD designs can be constructed. The second method, GIFT-FAIL, focuses on imperfect outputs. The system takes a nearly correct 3D model generated by the AI, converts it back into an image, and pairs that image with the correct CAD program. By doing so, it creates new examples that effectively teach the model how to recover from specific mistakes. More importantly, this process requires no human intervention. The framework automatically identifies errors, creates new training data, and uses that information to improve the model. The researchers also designed GIFT around a technique known as inference-time scaling. Instead of retraining an entire AI model, which can be extremely expensive, the system improves performance during generation. Rather than discarding the extra computation spent searching for better CAD programs, GIFT converts successful and near-successful attempts into new training examples that can later be used to improve the model through fine-tuning. Moreover, users can decide how much computation they want to spend depending on their budget and time constraints. Better CAD designs with a fraction of the computing power When evaluated on the GenCAD-Code dataset, models enhanced with GIFT achieved a 12 percent improvement in geometric accuracy compared with a strong supervised-learning baseline. The framework also reduced the inference computation needed to achieve comparable performance by about 80%. “With GIFT, we started with geometry because with engineering problems, if the geometry of a 3D shape is not correct, nothing else will be correct, but there are many other aspects to consider,” Giannone said. The system’s ability to learn from its own mistakes could help accelerate product development by making AI-generated CAD models more reliable. Engineers could move more quickly from sketches to testable 3D designs, reducing development costs, shortening design cycles, and exploring a wider range of ideas without creating large amounts of new training data. The researchers say this is only the beginning. So far, GIFT focuses on generating correct geometry, but future versions will aim to teach AI how to design parts that are not only accurate in shape but also easier to manufacture and better suited for real-world performance. The study is published on arXiv. Recommended ArticlesRupendra Brahambhatt is an experienced writer, researcher, journalist, and filmmaker. With a B.Sc (Hons.) in Science and PGJMC in Mass Communications, he has been actively working with some of the most innovative brands, news agencies, digital magazines, documentary filmmakers, and nonprofits from different parts of the globe. As an author, he works with a vision to bring forward the right information and encourage a constructive mindset among the masses.