Execution of the eight real-world evaluation tasks.
Eight unseen tasks across three difficulty levels.
Evaluation is conducted at three difficulty levels: Easy, where a single executable task is present; Medium, where multiple executable tasks require discrimination; and Hard, where multiple valid execution modes exist.
| Level | Method | Weighing | Wipe | Tea | Drawer | Soap | Fruit | Chopsticks | Shelf | Average |
|---|---|---|---|---|---|---|---|---|---|---|
| Easy | ||||||||||
| Fast-WAM | 0.90 | 0.87 | 0.83 | 0.80 | 0.63 | 0.90 | 0.73 | 0.77 | 0.80 | |
| π0.5 | 0.87 | 0.97 | 0.80 | 0.60 | 0.73 | 0.60 | 0.53 | 0.67 | 0.72 | |
| SimpleICL (ours) | 0.93 | 0.76 | 0.73 | 0.83 | 0.73 | 0.86 | 0.80 | 0.83 | 0.81 | |
| Medium | ||||||||||
| Fast-WAM | 0.83 | 0.50 | 0.63 | 0.43 | 0.60 | 0.83 | 0.63 | 0.73 | 0.65 | |
| π0.5 | 0.63 | 0.46 | 0.37 | 0.43 | 0.57 | 0.47 | 0.23 | 0.57 | 0.47 | |
| SimpleICL (ours) | 0.83 | 0.66 | 0.60 | 0.80 | 0.67 | 0.63 | 0.73 | 0.63 | 0.70 | |
| Hard | ||||||||||
| Fast-WAM | 0.40 | 0.60 | 0.27 | 0.53 | 0.33 | 0.57 | 0.27 | 0.37 | 0.42 | |
| π0.5 | 0.53 | 0.33 | 0.20 | 0.17 | 0.47 | 0.40 | 0.10 | 0.30 | 0.31 | |
| SimpleICL (ours) | 0.80 | 0.67 | 0.57 | 0.67 | 0.63 | 0.70 | 0.76 | 0.60 | 0.68 | |
Real-world evaluation on eight unseen tasks. Success rates (%) are reported for each method. Wipe = wiping a plate; Tea = serving tea; Drawer = pulling a drawer; Soap = pressing a soap dispenser; Fruit = placing fruit; Chopsticks = organizing chopsticks and bowls; Shelf = organizing items on a shelf.