Case Study 05
AppleSupport AI
AI Customer Support Agent — Research Pipeline
AppleSupport AI is a support-agent research pipeline on real AppleSupport tweets — not a chatbot demo. Verified phases 1–3: thoughtvector/customer-support-on-twitter loading, reply-graph reconstruction, 10-class taxonomy, baselines, MiniLM classifiers and a 200-row golden set. Retrieval, generation, escalation policy and LLM judging are planned phases 4–9, not implemented.
01 · Overview
Overview
- Reconstruct real threads, classify needs, evaluate with a blind golden set.
- Flat tweet rows become reply-graph conversations before modelling.
- ML research on 2.8M-scale data with AppleSupport focus.
- Gated steps: validation gates, leakage-free splits, CV-on-train selection.
02 · The Problem
The Problem
- Raw CSV (tweet_id, author_id, inbound, created_at, text, response ids) is not conversations.
- Naive splits leak threads across train/test.
- Keyword models miss paraphrase; agent claims need measured foundations.
03 · The Solution
The Solution
- Chunked loader (200k chunks, IDs as strings) + thread collection + topic EDA.
- Reply-graph DFS with pair extraction + 7-point validation gate.
- 10-class taxonomy; keyword/majority/TF-IDF baselines; frozen MiniLM classifiers; blind 200-row golden set.
04 · How It Works
How It Works
01 · 1. Load-
chunked twcs.csv, schema validation, stats, threads.
02 · 2. Clean+
unescape, strip RT/URLs/mentions, keep case/typos; report.
03 · 3. Reconstruct+
reply-graph DFS, author filter, frames, pairs.
04 · 4. Validate+
7-point gate, fail loudly.
05 · 5. Label blind+
turn-1 openers, hint-free worksheet, 70/15/15 seed 42 (n=200).
06 · 6. Baseline+
16-config grid, 5-fold CV, test n=29.
07 · 7. Embed+
frozen MiniLM 384-d + LogReg/centroid, confidence 0.60.
Select a stage to expand it. Every stage maps to code in the repository.
05 · System Architecture
System Architecture
raw twcs.csv + processed pairs/conversations (git-ignored).
config (0.60/0.65/top-5) · data · intents · empty phase-4+ stubs.
author IDs, prep, EDA (11 topics, 6 PNGs), golden, baselines, embedding.
golden CSV/JSON, splits, isolation manifest, versioned results.
40-cell executed notebook, report, decision log.
06 · Core Features
Core Features
Reply-graph reconstruction with cycle handling.
Reply-graph reconstruction with cycle handling.
10-class taxonomy with escalation hints.
10-class taxonomy with escalation hints.
Honest baselines: keyword 0.48, majority 0.66, TF-IDF 0.62 accuracy (n=29; train 1.0).
Honest baselines: keyword 0.48, majority 0.66, TF-IDF 0.62 accuracy (n=29; train 1.0).
Embedding: MiniLM+LogReg 0.6552 acc / 0.3032 macro-F1; centroid 0.5517/0.3894.
Embedding: MiniLM+LogReg 0.6552 acc / 0.3032 macro-F1; centroid 0.5517/0.3894.
Blind golden set: 200 rows, single labeler (stated limit).
Blind golden set: 200 rows, single labeler (stated limit).
07 · Technical Implementation
Technical Implementation
- Python; pandas/numpy, sklearn, sentence-transformers (CPU), matplotlib/seaborn, dotenv.
- No FAISS/Streamlit/OpenAI/LangChain in requirements — placeholders only.
- APPLE_SUPPORT_AUTHOR_ID required; thresholds parsed but not consumed beyond classifiers.
08 · My Contribution
My Contribution
- Chunked loading, cleaning, reconstruction, validation gate.
- Blind golden pipeline, conversation splits, CV selection, versioned results.
- Taxonomy, grids, frozen-encoder classifiers, error analysis.
- Notebook, report, decision log marking 4–9 planned.
09 · Tech Stack
Tech Stack
- Python · pandas · sklearn · sentence-transformers.
- Python · pandas · sklearn · sentence-transformers.
- Data
- thoughtvector tweets (~2.8M) · reply graphs.
- Eval
- 200-row golden · stratified splits · 5-fold CV.
- Not built
- RAG · escalation · generation · LLM judge.
10 · Challenges & Decisions
Challenges & Decisions
- Flat → threads → DFS + partial roots + gate → loud failures.
- Leakage → conversation splits + manifest → honest tests.
- Rare classes + single labeler → report macro-F1 + state limits.
11 · Result
Result
- Phases 1–3 delivered with measured metrics and artefacts.
- Explicitly not built: retrieval, generation, escalation, LLM judge, human-eval study.
12 · Links