Research paper · 2026-10-21
India AI research · Mithai Shop Challenge
The Diwali Trap: Can AI Forecasting Models Help a Mithai Shop Plan for a Moving Festival?
A simulated Ahmedabad mithai shop shows how a moving Diwali can mislead history-only forecasts; calendar inputs and sell-out aware planning improve decisions in this controlled study.
23N Labs Research Team
Public release · synthetic study
Synthetic data notice. The study uses a simulated Ahmedabad mithai shop, not genuine retailer transactions. Rupee outcomes are simulated under stated assumptions.
The public release compares forecasting and production plans in a controlled simulation. Read the full article, figures and technical appendix below. View the public release and reproduction materials.
Full technical report
⚠️ Synthetic data notice. This study uses a simulated Ahmedabad mithai shop. It does not use genuine shop transaction records. Every sales figure, rupee amount and “profit” in this article is a simulated outcome under stated assumptions. None of it is a saving achieved by, or promised to, any real retailer.
The Diwali Trap
Diwali fell on 20 October 2025 and falls on 8 November 2026, nineteen days later. A sweet shop that plans kaju katli production from last October’s sales will prepare for a rush that has not arrived yet.
We asked three questions about today’s AI forecasting models, including Yug-1 from Birla AI Labs:
- Do they fall into this trap?
- Does giving them the festival calendar help?
- Does a better forecast actually lead to a better production plan?
How we tested it
We built a transparent simulator of a small mithai shop selling kaju katli, motichoor ladoo and gulab jamun (in kg/day, 2019–2026). It includes:
- festival ramps that start weeks before Diwali;
- corporate gift-box orders booked in advance;
- wedding seasons and random bulk orders;
- sell-outs that hide how big demand really was, so past sales understate past demand.
Up to 30 independently randomised shops were simulated. In total we tested 13 forecasting strategies:
| Group | Methods |
|---|---|
| Foundation models, history only | Birla AI Labs Yug-1, Google TimesFM 2.5, Amazon Chronos-Bolt |
| Statistical, history only | seasonal-naive, ETS |
| Calendar-aware regressions | standard, and a variant that accounts for sell-outs |
| Shopkeeper rules | a ×1.10 rule, and a calibrated version |
| Calendar correction | the same festival-calendar correction added to each foundation model and to ETS |
Every forecast was turned into a 56-day production plan under identical prices, costs, shelf lives and capacity. We tested both a plan committed up front and weekly replanning with supplier lead times. We measured regret: the simulated profit lost compared with a plan made with perfect hindsight.
Everything ran on free Kaggle GPUs and a laptop. That came to 241,920 forecasts, with no forecasting failures.
Finding 1: history-only models lean toward last year’s Diwali
In the years when Diwali moved 19 days later (2023 and 2026), models that see only past sales put extra weight on last year’s Diwali dates. Taking each model’s overall bias into account, the extra weight was:
| Method | Extra weight on last year’s dates |
|---|---|
| Seasonal-naive | +45% |
| Yug-1 | +29% |
| Chronos-Bolt | +17% |
| TimesFM 2.5 | +15% |
| Flat ETS (the no-pattern baseline) | +10% |
When Yug-1 forecast a distinct peak, which it did in 48% of cases, the peak came a median of 17 days early. TimesFM and Chronos-Bolt forecast distinct peaks less often (33% and 23%), and those peaks were also early. The tendency that creates the trap also gives Yug-1 the strongest festival surge of any history-only model.
Finding 2: giving models the festival calendar helps every one of them
A simple correction lets any model use the festival calendar. It learns, from past sales only, how demand rises around each festival, then applies that pattern to the known future festival dates. This is an add-on, not a built-in feature of any model. It reduced festival-season forecast error for every model:
- Base simulator (festivals follow the calendar): Yug-1 +10.3 points, TimesFM +6.2, Chronos-Bolt +8.0.
- Hardest scenario (festival shape varies every year, plus unrelated local shocks): +4.7, +2.6 and +3.7 points. Smaller, but still positive.
Finding 3: turning forecasts into decisions matters as much as the model
Every method under-forecast festival demand, because past sell-outs hide true demand. So we gave every method the same training-only correction, learned only from each shop’s own past forecasts and sales. That reshuffled the business ranking. Regret per 56-day product plan, festival season:
| Method | Before correction | After |
|---|---|---|
| Calendar regression (accounts for sell-outs) | ₹63,812 | ₹47,602 |
| Calendar regression | ₹68,886 | ₹50,855 |
| TimesFM 2.5 + calendar | ₹73,043 | ₹52,847 |
| Yug-1 + calendar (average of 5 seeds) | ₹66,275 | ₹52,882 |
| Chronos-Bolt + calendar | ₹78,896 | ₹54,102 |
| Yug-1 (history only, average of 5 seeds) | ₹91,265 | ₹87,051 |
| TimesFM 2.5 (history only) | ₹104,819 | ₹82,656 |
| Chronos-Bolt (history only) | ₹118,008 | ₹91,129 |
For scale, the perfect-hindsight plan earns about ₹327,000 per product plan in the simulation.
What this tells us:
- Accounting for sell-outs was the most valuable single step. After correction, the calendar regression that does so had the lowest regret in the base, varied-festival, shock, combined and weekly-replanning experiments. It did not win every stress test: with a weaker festival, Yug-1 + calendar and TimesFM 2.5 were lower, and under a sudden local shock it tied Yug-1 + calendar.
- After correction, the three AI models with the calendar cannot be separated. Yug-1 + calendar versus TimesFM + calendar: ₹35 per plan, with a 95% interval of −₹1,914 to +₹1,603.
- Before correction, history-only Yug-1 looked like the most profitable AI model. That advantage came mostly from its forecasts being less biased low. After correction, TimesFM 2.5 was ahead by ₹4,395 per plan.
- A simple “make 10% extra” shopkeeper rule beat many uncalibrated models. Once forecasts were bias-corrected, it was overtaken by the calendar-informed methods.
Finding 4: Yug-1’s sampling seed matters
Yug-1 builds its uncertainty range from 100 random sample paths. Changing only the random seed moved the share of days inside its 80% range during festivals between 68% and 80%. It also changed simulated business results noticeably in every experiment. Point accuracy barely moved.
Our first analysis happened to use the seed that was most favourable for Yug-1 in 15 of 16 comparisons, so this final version averages over five seeds.
What it means
- For shops: give your forecasting tool the festival calendar, and correct for days you sold out. How you turn a forecast into a production quantity matters as much as which model you use.
- For model builders: native support for future-known calendar events, seasonality aligned to events rather than dates, and independent random draws per series would directly address what we observed.
- For everyone: this is a controlled simulation. Effects in real shops could be larger, smaller or different. Validation on real, consented retailer data is the essential next step.
Limitations, in brief
- Synthetic data under our assumptions, with festival effects that remain roughly tied to the calendar.
- One planning-policy family, and no tuning for any method.
- Calibration uses censored past sales.
- Foundation models used zero-shot with default settings.
Several analyses were added after our first results, during subsequent review passes. The technical appendix labels each result as pre-specified or post-hoc.
Reproduce it
The code, synthetic-data generator, verified result summaries, inference notebooks and checksums are in this repository. The 367 MB raw synthetic forecast archives remain in the pinned private source; reproduce them using the instructions here. See the technical appendix and README.
23N Labs is independent. This study is not affiliated with or endorsed by Birla AI Labs, Google or Amazon. We thank Birla AI Labs for publishing Yug-1 with clear versioning, which made this independent study possible.
Technical appendix
Technical Appendix — The Mithai Shop Challenge (final, v3)
⚠️ SYNTHETIC RETAIL DEMAND. All data come from a simulated Ahmedabad mithai shop (
src/mithai_core.py). No genuine shop transaction records are used. All rupee amounts are simulated contributions under stated assumptions, not realised or projected savings.
A. Study versions and immutability
| Version | Content | Location | Status |
|---|---|---|---|
| v1 | Original design: 11 methods, 2,160 jobs, committed plan | Pinned private source; result summaries here | Historical 148-entry manifest: 143 original bytes match; five superseded documents are preserved in source history. |
| v2 | Robustness: calendar dependence (R1–R3), calibrated shopkeeper, censoring-aware regression, rolling replanning, Yug-1 seeds on the original jobs | robustness/v2/ and pinned private source |
Historical 50-entry manifest verifies at the source; summaries and code are included here. |
| v3 | Final pass: Yug-1 seeds on every v2 experiment, training-only calibration, hierarchical paired tests, independent verification | robustness/v3/ |
This release. |
B. Pre-specified versus post-hoc (read every result with this table)
| Element | Status |
|---|---|
| Simulator, cut-offs, horizons (28/42/56), methods of v1, WAPE/MAE/wQL/coverage, actual-peak rule, surge window, raw phantom-surge metric, shop-cluster bootstrap, critical-ratio policy, hindsight reference | Pre-specified (before any result) |
| Committed-plan policy as primary; “forecast must have a distinct peak” rule | Changed after a baselines-only dry run, before any foundation-model output |
| Bias-adjusted phantom-surge metric | Post-hoc (after v1 results) |
| R1/R2/R3 scenarios; calibrated shopkeeper; censoring-aware regression; rolling replanning; Yug-1 seeds on the original jobs | Post-hoc (v2, subsequent review pass) |
| Yug-1 seeds on all v2 experiments; training-only bias correction and decision calibration; hierarchical paired tests | Post-hoc (v3, subsequent review pass) |
The headline conclusions in the article rest mainly on post-hoc robustness analyses. They are framed as robustness findings, not confirmatory tests.
C. Data, models and compute
Simulator. 2019–2026, daily, in kg.
- Latent demand and censored observed sales (capped at historical capacity).
- Corporate kaju katli orders booked 7–25 days ahead.
- Wedding seasons and bulk orders.
- Five festivals, with Diwali dates cross-checked against drikpanchang.
Scenarios.
- Base: 30 shops.
- Stress: early gifting, weak festival, +50% local shock; 2026 only; 20 shops each.
- R1: yearly-varying festival shape (lead ×e^N(0,0.35), exponent 0.8–3.5, uplift ×e^N(0,0.35), shift −6…+2 days).
- R2: ~3 festival-independent shocks per year (3–21 days, ×0.55–1.8).
- R3: both. R1–R3 use 20 shops each.
Models (pinned and hash-verified).
| Model | Revision |
|---|---|
| Yug-1 | birlaailabs/yug @ ec53758; package v0.1.1 @ 2ad7da71 |
| TimesFM 2.5 | @ 1d952420; timesfm 3.0.2 @ e51928e2 |
| Chronos-Bolt base | @ 5d9f166d; chronos-forecasting 1.5.3 @ fcd09fe8 |
All three run zero-shot, horizon 56, context ≤ 2,048 days, float32, on a Kaggle Tesla T4. Yug-1 uses its default 100 sample paths with seeds 67067, 1, 2, 3 and 4.
Forecast inventory (raw .npz, counted directly by robustness/v3/src/final_verify.py): 42 files and 241,920 forecast rows, with 0 non-finite and 0 crossed-quantile rows.
| Source | Forecasts |
|---|---|
| v1 foundation models | 12,960 |
| v1 local methods | 10,800 |
| v2 local methods on original jobs | 4,320 |
| v2 local methods on v2 jobs | 65,520 |
| v2 foundation models on v2 jobs | 56,160 |
| v2 Yug-1 seeds on original jobs | 17,280 |
| v3 Yug-1 seeds on v2 jobs | 74,880 |
History identity. Kaggle regenerated every input history, and it hash-matched the local build (v1 d63e862b…, v2 757e8761…). The numpy versions differed: 1.26.4 locally, 2.1.3 on Kaggle.
D. Metrics and decision policy
Accuracy (median forecast against latent demand):
- WAPE: pooled.
- 80% coverage: q0.1–q0.9.
- wQL.
- Peak timing: 7-day sums. Scored only when the actual peak is distinct and the forecast has a distinct peak; the distinct-peak share is always reported alongside.
- Surge error.
- Raw and bias-adjusted phantom surge.
Decisions.
- Target(t) = max(forecast quantile at the critical ratio τ = margin/price, confirmed orders).
- Capacity 1.4× historical, with pre-build.
- FIFO stock, shelf lives of 10, 5 and 4 days.
- Committed 56-day plan, or weekly rolling replanning: 7-day supplier lead time, ingredients 70% of cost, ingredient shelf lives of 28, 14 and 7 days, MRP netting.
- Regret = hindsight reference − policy. The reference is verified feasible under identical constraints; it loses 0.2–0.3% of demand, to capacity at the peaks.
Training-only calibration (v3). Computed per method, per shop and product, from forecast windows that ended before the cut-off:
- bias: b = Σ observed sales ÷ Σ median forecast.
- bias + decision calibration: k = empirical τ-quantile of observed ÷ forecast quantile at τ.
Notes on the calibration:
- The original ×1.10 shopkeeper is calibrated from his point forecast without the 1.10 margin.
- It learns from censored observed sales, which is information a real shop has.
- If no past window exists, the forecast is unchanged; this affects 4% of jobs.
Uncertainty. 95% intervals from 2,000 bootstrap draws. Each draw resamples simulated shops (clusters holding all of a shop’s cut-offs and products) and, for Yug-1, the five sampling seeds. A paired difference whose interval contains 0 is reported as inconclusive. We make no significance claims beyond these intervals.
E. Results (all numbers from robustness/v3/results/final-v3.json, final-v3-addon.json and robustness/v2/results/robustness-v2.json)
E1. Diwali trap, +19-day years (v2 P1)
| Method | Distinct-peak share | Median peak error (days) | Bias-adjusted phantom surge |
|---|---|---|---|
| Yug-1 | 48% | −17 | +29% (5 seeds: +28 to +29%) |
| TimesFM 2.5 | 33% | −16 | +15% |
| Chronos-Bolt | 23% | −19 | +17% |
| Seasonal-naive | 100% | −20 | +45% |
| ETS | 0% | — | +10% |
| Calendar-informed methods | 88–96% | −1 to 0 | +4 to +11% |
E2. Calendar gain, WAPE points, paired (v2 P2)
| Scenario | Yug-1 | TimesFM 2.5 | Chronos-Bolt | ETS |
|---|---|---|---|---|
| Base | +10.3 [9.6, 11.2] | +6.2 [5.7, 6.8] | +8.0 [7.4, 8.6] | +10.4 |
| R1 | +5.6 [4.8, 6.5] | +2.9 [2.1, 3.8] | +4.2 [3.2, 5.3] | +6.7 |
| R2 | +9.4 [8.5, 10.3] | +6.0 [5.4, 6.7] | +7.3 [6.7, 8.0] | +9.6 |
| R3 | +4.7 [3.7, 5.8] | +2.6 [1.9, 3.5] | +3.7 [2.6, 4.8] | +5.9 |
E3. Festival-season regret, ₹ per 56-day product plan, original base jobs (1,080 plans, 30 shops; Yug-1 averaged over 5 seeds)
| Method | Uncalibrated | Bias-corrected | Bias + decision-calibrated |
|---|---|---|---|
| Calendar regression (censoring-aware) | 63,812 [59,497, 68,347] | 47,602 [44,358, 50,949] | 65,797 |
| Calendar regression | 68,886 | 50,855 | 70,975 |
| TimesFM + calendar | 73,043 | 52,847 | 69,768 |
| Yug-1 + calendar | 66,275 [59,251, 73,381] | 52,882 [48,565, 57,620] | 69,791 |
| Chronos + calendar | 78,896 | 54,102 | 72,541 |
| Shopkeeper (×1.10 rule) | 50,880 | 60,564 | 71,086 |
| ETS + calendar | 72,386 | 64,709 | 75,787 |
| Shopkeeper (calibrated) | 71,870 | 65,250 | 71,154 |
| TimesFM 2.5 | 104,819 | 82,656 | 99,708 |
| Yug-1 | 91,265 [83,387, 99,134] | 87,051 | 96,830 |
| Chronos-Bolt | 118,008 | 91,129 | 108,924 |
| Seasonal-naive 52w | 109,445 | 111,298 | 109,853 |
| ETS | 133,411 | 135,663 | 136,939 |
Hindsight reference contribution: ₹327,078 per plan.
Reading the calibration columns.
- Bias correction helps every model-based method.
- The extra decision calibration hurts most methods. Calibrating to censored past sales re-learns the under-production that censoring causes. This is a substantive finding about censored data, not a defect of any model.
E4. Paired differences after bias correction (hierarchical bootstrap; < 0 favours the first method)
| Comparison | Original | R1 | R2 | R3 | Weekly replanning |
|---|---|---|---|---|---|
| Yug-1 + cal − TimesFM + cal | +35 [−1,914, 1,603] inconclusive | +77 inconcl. | −370 inconcl. | −515 inconcl. | −202 inconcl. |
| Yug-1 + cal − Chronos + cal | −1,220 inconcl. | −860 inconcl. | −467 inconcl. | −848 inconcl. | −780 inconcl. |
| Censoring-aware regr. − Yug-1 + cal | −5,280 [−7,302, −3,208] | −4,753 | −4,715 | −3,918 | −4,530 |
| Yug-1 + cal − ETS + cal | −11,828 | −13,199 | −14,688 | −15,989 | −4,056 |
| Yug-1 + cal − calibrated shopkeeper | −12,368 | −18,392 | −14,709 | −19,159 | −13,290 |
In every family, after bias correction, censoring-aware calendar regression has significantly lower regret than every other calendar-informed method. The three calendar-corrected foundation models are mutually inconclusive.
E5. Does Yug-1’s history-only business advantage survive calibration? (original festival season)
| Comparison | Uncalibrated | Bias-corrected | Bias + decision-calibrated |
|---|---|---|---|
| Yug-1 − TimesFM 2.5 | −13,554 (favours Yug-1) | +4,395 (favours TimesFM) | −2,877 |
| Yug-1 − Chronos-Bolt | −26,743 (favours Yug-1) | −4,078 (favours Yug-1) | −12,094 |
The weekly-replanning family behaves similarly. Uncalibrated, Yug-1 − TimesFM is −10,472 [−14,870, −6,690]. With bias plus decision calibration it is +1,484, inconclusive.
Conclusion. Yug-1’s apparent advantage over TimesFM 2.5 was mainly a level-bias effect. It reverses under training-only bias correction. A smaller advantage over Chronos-Bolt remains.
E6. Yug-1 seed sensitivity
- Festival 80% coverage: 68–80% across seeds. Festival WAPE: 41.1–42.6%.
- Seed 67067, used in v1 and v2, was the most favourable seed in 15 of 16 business comparisons (fig F6).
- Plausible mechanism: the v0.1.1
predictcall takes one seed, so every series in a run reuses the same uniform draws. Monte Carlo error is then correlated across the benchmark instead of averaging out.
F. Verification performed (v3)
- Every v1 and v2 manifest entry verifies. The only differences are the five deliberately superseded documents, and each of their v1 versions is preserved.
- WAPE and coverage were recomputed from raw forecasts and regenerated truth, without the evaluation code: 18/18 checks match the published values.
- v2 recomputed every v1 headline point estimate exactly.
- An independently written planning simulator, with constraint assertions, matches the business code.
- Forecasts at the day-0 origin are identical between v1 and v2.
- A rolling-planner bug was found in v2 (perfect hindsight lost 5.5% of demand). It was fixed before any v2 result was used.
- Every number quoted in
article.mdand this appendix is checked against the JSON artifacts byrobustness/v3/src/claims_check.py, which writesfinal/claims-ledger.json.
G. Limitations
- Synthetic data. Effect sizes are assumptions, and festival effects stay roughly calendar-anchored even in R1.
- Censoring dominates the economics. Calibration learns from censored sales.
- Decision policy. One policy family; no tuning of τ or inventory rules; one rolling-planner design.
- Models. Foundation models run zero-shot with defaults. TimesFM ran in eager mode. Native covariate interfaces (TimesFM XReg, Chronos-2) were not tested.
- Post-hoc. Most robustness conclusions are post-hoc (section B).
- Statistics. Bootstrap intervals describe variation across simulated shops and model seeds, not real-world uncertainty.
Sources and data licence
Paper, appendix, original figures and synthetic data: CC BY 4.0. Original code: Apache 2.0. The Office Holidays calendar compilation, cross-checked with Drik Panchang, retains upstream terms and is outside the 23N grants. Model weights are not redistributed.
Mithai Shop Challenge public release · Original article · Original technical appendix · Full rights and attribution notice · Festival calendar source and cross-check.
Copyright 2026 23N Labs Research Team. Original paper, appendix, figures and synthetic data are under CC BY 4.0; original code is under Apache 2.0. This website is a format adaptation of the cited public release. The third-party factual calendar compilation comes from Office Holidays, with a Drik Panchang cross-check. Its upstream terms remain in force; no calendar publisher page text or images are reproduced here. Model weights and raw forecast archives are not included.
AI tools assisted coding, analysis and manuscript preparation; independent human review is not asserted. The study is independent and is not affiliated with or endorsed by Birla AI Labs, Google or Amazon. No venue or DOI is assigned.
Code & recorded outputs
Source files and saved checks are provided for inspection. The website does not execute research code. Recorded PASS results do not replace independent review.