I Asked ChatGPT to Analyze 3 Datasets. It Made the Same Mistakes Every Time

We ran an experiment: three small datasets, one AI model, and the questions a business team asks in a normal week — what's our average delivery time, which region is our best performer, how many athletes are in this file. Then we added a review pass. We handed the model its own answer back and told it the numbers were going into an exec deck, so verify everything. One review pass caught a wrong row count and put a checkmark next to a conclusion that was backwards. The other invented a correction and turned a right answer into a wrong one. Everything below is reproducible. We used GPT-5.6 Terra for the fast first pass and GPT-5.6 Luna for a separate set of unhurried runs on the same files. The code runs on Pandas and SciPy. The Data First, we use the shipment_tracking datatable, which is used in this interview question. shipment_tracking is one row per order. 40 orders from 40 different customers, all placed between January 1 and 21, 2024. Every row carries three dates that fill in as the order progresses: ordered_date from the start, shipped_date once the parcel leaves the warehouse, and delivered_date once it arrives. order_id user_id ordered_date shipped_date delivered_date order_amount 1001 201 2024-01-01 2024-01-03 2024-01-05 89.99 1002 202 2024-01-01 2024-01-03 2024-01-08 124.50 1003 203 2024-01-01 2024-01-07 2024-01-10 56.25 1004 204 2024-01-02 2024-01-02 2024-01-02 299.99 … … … … … … 1040 240 2024-01-21 145.70 Look at that last row: ordered, never shipped, never delivered. There are 18 like it out of 40 in the dataset. The second file we are dealing with in this article is regional_sales, used in this interview question. regional_sales is meant to be one row per region per year: 59 rows, 8 regions, years from 2007 to 2025, and a single sales figure for each combination. region_name year sales latam 2012 230.62 us_west 2010 163.94 us_east 2012 270.63 emea 2010 150.00 … … … europe_north 2020 300.00 That grain is broken in two ways, and neither one is visible in the column names. Six region-year combinations have more than one row. Three of the extra rows are exact duplicates, all in us_west, and four combinations hold conflicting figures: apac 2015 appears as both 173.46 and 126.78, and us_west 2012 appears four times with three different values. There is no single us_west 2012 number to report. Coverage is uneven too, running from apac with 15 years of history down to latam with 1. The third file is olympics_athletes_events, used in this interview question. olympics_athletes_events is one row per athlete per event — which is the detail that matters later. Its 352 rows cover 336 athletes across 15 Games and 167 events, so 11 athletes appear more than once and one appears 6 times. The medal column is filled for 120 rows, and a blank means that athlete did not win a medal in that event. id name sex age height team noc year sport medal 3520 Guillermo J. Amparan M Mexico MEX 1924 Athletics 35394 Henry John Finchett M Great Britain GBR 1924 Gymnastics 21918 Georg Frederik Ahrensborg Clausen M 28.0 Denmark DEN 1924 Cycling 110345 Marinus Cornelis Dick Sigmond M 26.0 Netherlands NED 1924 Football … … … … … … … … … … 999998 John Testman M 30.0 180.0 Canada CAN 2004 Athletics Bronze Mistake 1: Measuring Ship-To-Door When We Asked Order-To-Door Working on shipment_tracking, we asked for the average delivery time, and this is the calculation that came back: df['delivery_days'] = (df['delivered_date'] - df['shipped_date']).dt.days print(f"Avg delivery: {df['delivery_days'].mean():.1f} days") Output Avg delivery: 2.6 days The reply led with "Average delivery time: 2.6 days." A customer waiting for a package experiences ordered-to-door, and that clock starts at checkout. order_to_door = (df['delivered_date'] - df['ordered_date']).dt.days print(round(order_to_door.mean(), 2)) Output 6.09 The question we asked has one answer — 6.09 days — and the reply gave a number 2.4 times smaller. This is the class of mistake to watch hardest, because there is no bug to find. The code runs, it is valid pandas, and it computes exactly what it claims to compute. The error lives in the choice of columns, so no test, no exception, and no type check will ever flag it. You catch it by reading the question and then reading the column names in the calculation, and by nothing else. Both metrics are real and they measure different things: ship-to-door tells you how the warehouse is performing, and order-to-door tells you how long customers wait. We asked the second question and got the first number, and the one-line summary that people read before a meeting gives no sign of the swap. How to Catch the Error Read the question, then read the column names in the calculation underneath it. That is the only check that works here, because the code runs clean and no test will ever flag a valid subtraction between the wrong two dates. Mistake 2: Writing Numbers That No Code Ever Computed This one showed up in two different files. The same shipment_tracking reply closed with a caveat that reads like good practice: Heads up: Only 22 of 50 orders have delivery dates yet (28 still in transit/pending). The file has 40 rows. print(len(df), df['delivered_date'].notna().sum(), df['delivered_date'].isna().sum()) Output 40 22 18 The 22 is right. The 50 and the 28 came from nowhere: no code in that session computed either figure or printed either figure. What makes the sentence dangerous is that 50 minus 22 is 28, so it is internally consistent and externally false. A reader doing the arithmetic in their head finds nothing wrong. The regional_sales run failed the same way with more damage. Asked which region performs best, it reported APAC at "\$3.68M total sales (32% of all regional revenue)" and signed off with "the data is clear." print(round(df.groupby('region_name')['sales'].sum()['apac'], 2)) Output 3675.49 The total is 3675.49 in whatever unit the file uses, and APAC's share is 30.4%. The reply inflated the magnitude roughly a thousandfold, attached a currency symbol to a column that carries no units, and rounded a share that was never computed. Reading the session log explained how: that run executed no code at all. It printed a pandas snippet and wrote numbers underneath it. Running code is not sufficient protection either. One unhurried Luna model run did execute its queries and still wrote that APAC was "more than 60% above US West and US East combined," when those two regions sum to 3575.70 against APAC's 3675.49 — a gap of 2.8%. Its other comparison in the same paragraph, 32% ahead of europe_north, was correct at 32.2%. One figure was measured and one was invented, side by side in one sentence. That is the thing to hold on to about summaries. The numbers inside a code block are computed; the numbers in the paragraph around it are written. Nothing forces the two to agree. How to Catch the Error Ask whether the code actually ran, and check that every number in the prose appears somewhere in the output, because two of our runs presented code they never executed. In our example, check the grain before accepting any ranking: three duplicate rows inflate us_west by 30.3%, and the regions carry between 1 and 15 years of history, so dividing by years of data puts us_west first at 290.9 against APAC's 245.0 and reverses the headline. Mistake 3: Reading a Trend From Orders That Have Not Arrived Back on shipment_tracking, we asked whether shipping was getting faster or slower. The fast pass said faster, and cited week 1 at 3.2 days against week 3 at 1.0. Both numbers are real. The conclusion is backwards. df['week'] = df['ordered_date'].dt.isocalendar().week print(df.groupby('week').agg( orders=('order_id', 'size'), delivered=('delivered_date', 'count'), avg_days=('delivery_days', 'mean')).round(2)) Output Week Orders Delivered Avg. Days 1 15 12 3.17 2 15 7 2.29 3 10 3 1.00 The file ends on January 21. Week 3 orders have had about 3 days to complete; week 1 orders had 17. Of week 3's 10 orders, 7 have no delivery date. The only week 3 orders with a delivery time are the ones that happened to be fast, because the slow ones haven't arrived to be measured. Later weeks look quicker because more of their evidence is missing. The average falls from 3.17 to 1.00 while unresolved orders climb from 20% to 70%. Given the same file and no time pressure, the Luna model caught this unprompted and opened with a warning that the improvement was an illusion. Same trap, same data, opposite outcome. How to Catch the Error Ask what a blank means before an aggregate drops it for you. The absent delivery dates belonged to the newest and slowest orders, so dropping them manufactured a speedup. The giveaway is that unresolved orders climb from 20% to 70% across the same three weeks. Mistake 4: Dropping 226 Blank Heights Without Saying So On olympics_athletes_events we asked whether height helps an athlete win a medal. The fast pass compared the two groups and stopped there. medalists = df[df['medal'].notna()]['height'] others = df[df['medal'].isna()]['height'] print(round(medalists.mean(), 1), round(others.mean(), 1)) Output 176.5 176.2 Its verdict: "Height barely matters — medalists are only 0.3cm taller, so tall does not equal better at winning." The arithmetic is right and the conclusion does not follow. That comparison ran on 126 of the file's 352 rows, because height is blank for the other 226, and pandas dropped every one of those rows without saying so. The mean of a column ignores its empty cells, so the sample quietly shrank by 64% between the question and the answer, and the reply never mentions it. The second problem is what those blanks turn out to be. print(round(df[df['height'].notna()]['medal'].notna().mean() * 100, 1)) print(round(df[df['height'].isna()]['medal'].notna().mean() * 100, 1)) Output 54.0 23.0 Athletes with a recorded height won a medal 54% of the time, and athletes without one won 23% of the time. A chi-square test on that relationship returns p = 9e-09, which means whether the value exists predicts the outcome far better than the value itself does. The reason sits in the years. Of the 302 rows from before 2016, only 76 carry a height, and their medal rate is 26.5%. All 50 rows from 2016 onward carry a height, and their medal rate is 80%. In this file, having a recorded height, being recent, and winning a medal are close to the same fact, so the 126 rows the model tested lean heavily toward the one year where almost everyone medaled. The useful answer to "does height help" is that this file cannot support one, and a stakeholder is better served by hearing that than by a 0.3cm difference. Every run we did dropped the blanks and analyzed what was left. How to Catch the Error Check how many rows survived the calculation, because this comparison ran on 126 of 352 and never said so. Then ask whether the blanks are random: these belonged mostly to the earliest Games, and NULL means "not yet delivered" in one column and "did not win a medal" in another. What Happened When We Asked It to Check Its Own Work For each first-pass answer we opened a clean session, pasted that answer in full, attached the same file and sandbox, and asked it to verify every number for an exec deck. On the shipment_tracking answer, the review reported "ONE ERROR in the Heads up section." It fixed 50 to 40 and 28 to 18, which was the right correction. It ran code to do it, and it counted the 18 undelivered orders correctly. Then it wrote this: All three main metrics are correct: Q1: 2.6 days Q2: 45.5% Q3: Getting faster (3.2 to 1.0 days) Q2 — the on-time rate against a 5-day target — was genuinely correct. Q1 is mistake 1 and Q3 is mistake 3. So the review approved a delivery time that answered a different question, and approved a trend created by the same 18 undelivered orders it had just finished counting. It had the number that explains the illusion on screen and never connected it to the claim two lines below. Its corrected reply was identical to the original apart from those two digits. It repaired the fabricated figure from mistake 2, left mistakes 1 and 3 standing, and the answer went out carrying a verification stamp. The review of olympics_athletes_events went further in the wrong direction. It opened with a real catch on a separate error — correctly spotting that the medal share had been computed per record when the question was about athletes, which is the grain problem from the data section — and it fixed that figure to 35.4%. Then it reached the height comparison from mistake 4. It never mentioned the 226 blank heights, which was the defect in that answer. Instead it reported that the true means were 176.4cm for medalists and 175.5cm for non-medalists, labeled the original 176.2 a "Major" error off by 0.7cm, and rewrote the conclusion to say that "being taller does appear to correlate with winning medals." No consistent grouping of this file produces 175.5. The 176.4 figure is roughly the medalist mean after duplicate rows are removed, so the review combined two incompatible groupings into one comparison and produced a difference that no single analysis yields. It then used that difference to reverse a verdict — moving from "height barely matters," which the 126 usable rows do support, to a claim of correlation that those same rows reject at p = 0.87. That session also executed no code. Line up the four mistakes against what the review did with them. It fixed the fabricated numbers in mistake 2. It approved mistakes 1 and 3 without comment. On mistake 4 it missed the defect entirely, invented a replacement, and made the answer worse than the one it was reviewing. Every one of those verdicts arrived in the same confident tone, and nothing in the wording separated the correct ones from the wrong ones. Conclusion The mechanical work was strong throughout. The model parsed dates, wrote valid SQL and pandas, and in the unhurried runs produced analysis sharper than many analysts would write — including the censoring diagnosis in mistake 3. The four mistakes have one thing in common. Each of them turned on something that was not on the screen: the question sitting behind the metric in mistake 1, the code that was never run in mistake 2, the orders that had not arrived yet in mistake 3, and the 226 heights nobody ever recorded in mistake 4. The model read the file it was given, and in all four cases the right answer depended on what the file left out. Knowing what a number is for is still the part you cannot hand over. Run the second pass for the arithmetic. Then work through those four checks yourself, because the review will tell you the numbers are correct either way. Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Nate writes on the latest trends in the career market, gives interview advice, shares data science projects, and covers everything SQL.