PROACTIVECOACH
Improving Proactive AI Assistance
with Hierarchical
Procedural Understanding
1 Sungkyunkwan University 2 Nanyang Technological University
* Corresponding authors
What to say. When to say it. At the right level of detail.
IN BRIEF
Assistance that follows
the task, and the user.
Proactive AI assistants observe ongoing activity, decide whether to provide new guidance or remain silent, and adapt the level of detail to the user’s expertise and needs. Existing datasets focus on detection-based responses or fixed-granularity guidance, limiting evidence about fine-grained progress, broader procedural context, and when a guided event is complete.
The ProactiveCoach suite provides ProactiveCoach-Instruct for training and ProactiveCoachBench for evaluation. Phase–step–action guidance and level-specific timing connect fine-grained progress with broader procedural context. The benchmark evaluates appropriate guidance at the right time at each level, including adaptation when the requested level changes during an ongoing task.
ProactiveCoach jointly fine-tunes a single vision-language model to generate phase-, step-, and action-level guidance. An adaptive guidance system combines this model with a lightweight router that selects the level requested by the user, without additional fine-tuning of the guide model.
02 / GUIDANCE GRANULARITY
One task.
Three ways to help.
An illustration of phase, step, and action guidance.
USER GOALMake aglio e olio pasta
Phase Make the sauce
Step Infuse olive oil with garlic
Action Add sliced garlic
CHOOSE THE USER'S REQUEST
PHASE-LEVEL GUIDANCE
“Next, make the sauce for the pasta.”
The guidance level changes with the user's needs. Response timing is defined separately at each level.
ACTUAL DATA / PROACTIVECOACHBENCH
Follow a procedure.
Inspect every guidance moment.
Watch five HoloAssist examples from the beginning, with synchronized phase, step, and action annotations. Select an event to inspect its guidance and timing.
Loading examples…
USER QUERY
Annotations available.
Video unavailable for this case.
You can still explore the annotations. Reload to retry the video.
Recording details
- Dataset
- Recording ID
- Video interval
Latest annotated guidance
Updates at the recorded guidance timestamps.
Step · ground truth
Action · ground truth
Event spans & guidance timing
Event executionGuidance timestamp
/
Swipe the timeline horizontally for detail. Use the event list below for precise selection.
03 / THE DATA
From procedural video
to structured guidance.
11,119annotated samples
7,232first-person videos
8source datasets
3guidance levels
ProactiveCoach-Instruct
10,007 samples
Hierarchically structured guidance with level-specific timing for learning what to say and when to respond.
ProactiveCoachBench
1,112 samples
Multi-level proactive guidance evaluation, including adaptation when the requested guidance level changes.
Sources: Ego4D Goal-Step · Ego-Exo4D · EPIC-KITCHENS · CaptainCook4D · EgoExoLearn · HD-EPIC · WTaG · HoloAssist
04 / EVALUATION
Better guidance.
Across procedural levels.
Selected Qwen3-VL-4B results from Table 2.
Avg. is the mean of sF1, gF1, and PQS. Higher is better.
Base model With our training
ProactiveCoachBench evaluates phase, step, and action guidance; EgoProactive evaluates step-level guidance. Evaluation uses recent video and ground-truth dialogue history.
Hierarchy helps beyond step-only supervision.
Step-level Avg. on ProactiveCoachBench, Table 3. Gains are absolute score points.
| Backbone | Step-only | Hierarchical | Gain |
|---|---|---|---|
| Qwen3-VL-4B | 49.32 | 58.96 | +9.64 |
| Qwen3-VL-8B | 53.07 | 59.37 | +6.30 |
| Qwen3.5-4B | 50.06 | 59.30 | +9.24 |
| Qwen3.5-9B | 51.18 | 58.81 | +7.63 |
sF1
Matches guidance by timing and semantic content.
gF1
Measures response versus silence decisions.
PQS
Combines timing decisions with guidance quality.
05 / EXPLORE FURTHER