Andreea Vasilescu

At Microsoft, we run AB testing in pre-production to catch regressions before release. But figuring out whether an incident was a true positive, a false alarm, or a missed regression used to mean hours of manual detective work.

In this talk, I’ll share how we built ALFA (Automated Loop Feedback Analysis), a system that uses Large Language Models (LLMs) to close the feedback loop between incidents in production and gaps in pre-production.

You’ll discover:

• How LLMs extract structured signals (builds, error codes, event tables) from messy incident reports.
• Why reasoning-based binary classification beats rule-based heuristics for identifying missed regressions.
• How we use LLMs for root-cause analysis — spotting coverage gaps, signal gaps, and process gaps — and automatically generating recommendations.