# Ablation Evaluation Results - SWE-Bench Lite

## Summary Table

| Experiment | Step | File F1 | Module F1 | Entity F1 | Success Rate | Avg Steps | Avg Tools | Output Format |
|------------|------|---------|-----------|-----------|--------------|-----------|-----------|---------------|
| Baseline-14B | 10 | 0.3534 | 0.2323 | 0.1690 | 50/50 (100.0%) | 4.80 | 4.70 | ✓ Tool (44) |
| Baseline-14B | 20 | 0.3190 | 0.1977 | 0.1394 | 50/50 (100.0%) | 4.98 | 4.80 | ✓ Tool (40) |
| Baseline-14B | 30 | 0.3288 | 0.1887 | 0.1033 | 50/50 (100.0%) | 3.54 | 3.48 | ✓ Tool (46) |
| Baseline-14B | 40 | 0.4310 | 0.2933 | 0.1913 | 50/50 (100.0%) | 3.70 | 3.68 | ✓ Tool (49) |
| Baseline-14B | 50 | 0.4767 | 0.2367 | 0.1400 | 50/50 (100.0%) | 5.16 | 5.14 | ✓ Tool (47) |
| Baseline-14B | 60 | 0.5067 | 0.3567 | 0.2167 | 50/50 (100.0%) | 5.34 | 5.34 | ✓ Tool (47) |
| Baseline-14B | 70 | 0.5400 | 0.3933 | 0.2333 | 50/50 (100.0%) | 4.46 | 4.46 | ✓ Tool (49) |
| Baseline-14B | 80 | 0.4900 | 0.3633 | 0.2133 | 50/50 (100.0%) | 5.20 | 5.20 | ✓ Tool (47) |
| Baseline-14B | 90 | 0.4733 | 0.2213 | 0.1613 | 50/50 (100.0%) | 5.60 | 5.60 | ✓ Tool (46) |
| Baseline-14B | 100 | 0.4067 | 0.2633 | 0.1767 | 50/50 (100.0%) | 8.34 | 8.34 | ✓ Tool (26) |
| Baseline-14B | 110 | 0.4500 | 0.2878 | 0.2000 | 50/50 (100.0%) | 7.14 | 7.14 | ✓ Tool (41) |
| Baseline-14B | 120 | 0.4800 | 0.3233 | 0.1967 | 50/50 (100.0%) | 6.22 | 6.22 | ✓ Tool (46) |
| Baseline-14B | 130 | 0.5600 | 0.3533 | 0.2333 | 50/50 (100.0%) | 5.78 | 5.78 | ✓ Tool (48) |
| Baseline-14B | 140 | 0.5667 | 0.3467 | 0.2067 | 50/50 (100.0%) | 5.26 | 5.26 | ✓ Tool (50) |
| Baseline-14B | 150 | 0.5200 | 0.3500 | 0.2167 | 50/50 (100.0%) | 5.96 | 5.96 | ✓ Tool (45) |
| Baseline-14B | 160 | 0.5067 | 0.3800 | 0.2747 | 50/50 (100.0%) | 5.60 | 5.58 | ✓ Tool (47) |
| Baseline-14B | 170 | 0.4333 | 0.2867 | 0.1900 | 50/50 (100.0%) | 7.84 | 7.84 | ✓ Tool (33) |
| Baseline-14B | 180 | 0.5000 | 0.4093 | 0.2643 | 50/50 (100.0%) | 7.06 | 7.06 | ✓ Tool (39) |
| Baseline-14B | 190 | 0.4933 | 0.3600 | 0.2200 | 50/50 (100.0%) | 6.22 | 6.22 | ✓ Tool (44) |
| Baseline-14B | 200 | 0.5567 | 0.4667 | 0.3130 | 50/50 (100.0%) | 5.28 | 5.28 | ✓ Tool (48) |
| Baseline-14B | 210 | 0.5167 | 0.4300 | 0.2713 | 50/50 (100.0%) | 6.00 | 6.00 | ✓ Tool (43) |
| Baseline-14B | 220 | 0.5500 | 0.5213 | 0.3180 | 50/50 (100.0%) | 6.22 | 6.22 | ✓ Tool (40) |
| Baseline-14B | 230 | 0.5200 | 0.4560 | 0.2981 | 50/50 (100.0%) | 6.06 | 6.06 | ✓ Tool (44) |
| Baseline-14B | 240 | 0.5433 | 0.4700 | 0.3167 | 50/50 (100.0%) | 5.76 | 5.76 | ✓ Tool (47) |
| Baseline-14B | 250 | 0.5467 | 0.4627 | 0.3360 | 50/50 (100.0%) | 5.26 | 5.26 | ✓ Tool (46) |
| Baseline-14B | 260 | 0.5400 | 0.4133 | 0.3333 | 50/50 (100.0%) | 5.26 | 5.26 | ✓ Tool (47) |
| Baseline-14B | 270 | 0.5200 | 0.4267 | 0.3200 | 50/50 (100.0%) | 5.38 | 5.38 | ✓ Tool (47) |
| Baseline-14B | 280 | 0.5233 | 0.4600 | 0.3190 | 50/50 (100.0%) | 5.28 | 5.28 | ✓ Tool (46) |
| Baseline-14B | 290 | 0.4800 | 0.4433 | 0.2980 | 50/50 (100.0%) | 5.04 | 5.04 | ✓ Tool (48) |
| Baseline-14B | 300 | 0.5067 | 0.4467 | 0.3000 | 50/50 (100.0%) | 4.90 | 4.90 | ✓ Tool (48) |
| Baseline-14B | 310 | 0.4967 | 0.4600 | 0.3133 | 50/50 (100.0%) | 5.22 | 5.22 | ✓ Tool (46) |
| Baseline-14B | 320 | 0.5000 | 0.4533 | 0.3100 | 50/50 (100.0%) | 4.88 | 4.88 | ✓ Tool (48) |
| Baseline-14B | 330 | 0.5067 | 0.4467 | 0.3200 | 50/50 (100.0%) | 4.86 | 4.86 | ✓ Tool (47) |
| Baseline-14B | 340 | 0.5200 | 0.4667 | 0.3267 | 50/50 (100.0%) | 5.16 | 5.16 | ✓ Tool (46) |
| Baseline-14B | 350 | 0.5267 | 0.4600 | 0.3400 | 50/50 (100.0%) | 4.88 | 4.88 | ✓ Tool (47) |
| Baseline-14B | 360 | 0.5233 | 0.4867 | 0.3400 | 50/50 (100.0%) | 4.88 | 4.86 | ✓ Tool (46) |
| Baseline-14B | 370 | 0.5133 | 0.4433 | 0.3100 | 50/50 (100.0%) | 5.14 | 5.14 | ✓ Tool (47) |
| Baseline-14B | 380 | 0.4933 | 0.4333 | 0.3033 | 50/50 (100.0%) | 5.14 | 5.14 | ✓ Tool (47) |
| Baseline-14B | 390 | 0.5400 | 0.5033 | 0.3467 | 50/50 (100.0%) | 4.66 | 4.66 | ✓ Tool (48) |
| Baseline-14B | 400 | 0.5000 | 0.4633 | 0.3067 | 50/50 (100.0%) | 4.80 | 4.78 | ✓ Tool (46) |
| Baseline-14B | 410 | 0.5300 | 0.4733 | 0.3333 | 50/50 (100.0%) | 5.02 | 5.02 | ✓ Tool (48) |
| Baseline-14B | 420 | 0.5433 | 0.4867 | 0.3467 | 50/50 (100.0%) | 4.90 | 4.90 | ✓ Tool (48) |
| Baseline-14B | 430 | 0.5533 | 0.4933 | 0.3500 | 50/50 (100.0%) | 4.90 | 4.90 | ✓ Tool (47) |
| Baseline-14B | 440 | 0.5467 | 0.4800 | 0.3400 | 50/50 (100.0%) | 5.14 | 5.14 | ✓ Tool (47) |
| Baseline-14B | 450 | 0.5100 | 0.4600 | 0.3400 | 50/50 (100.0%) | 5.14 | 5.14 | ✓ Tool (47) |
| Baseline-14B | 460 | 0.5367 | 0.5067 | 0.3533 | 50/50 (100.0%) | 4.86 | 4.86 | ✓ Tool (48) |
| Baseline-14B | 470 | 0.5100 | 0.4800 | 0.3400 | 50/50 (100.0%) | 4.96 | 4.94 | ✓ Tool (46) |
| Baseline-14B | 480 | 0.5433 | 0.5167 | 0.3600 | 50/50 (100.0%) | 5.04 | 5.04 | ✓ Tool (47) |
| Baseline-14B | 490 | 0.5433 | 0.4467 | 0.3257 | 50/50 (100.0%) | 4.86 | 4.86 | ✓ Tool (47) |
| Baseline-4B-2507-sonibugfix-withrg | 25 | 0.4133 | 0.2567 | 0.1133 | 50/50 (100.0%) | 4.00 | 9.64 | ✓ Tool (48) |
| Baseline-4B-2507-sonibugfix-withrg | 50 | 0.5033 | 0.3467 | 0.1500 | 50/50 (100.0%) | 4.58 | 10.98 | ✓ Tool (49) |
| Baseline-4B-2507-sonibugfix-withrg | 75 | 0.5667 | 0.3933 | 0.1733 | 50/50 (100.0%) | 4.00 | 9.96 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 100 | 0.5600 | 0.4400 | 0.1800 | 50/50 (100.0%) | 4.00 | 10.02 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 125 | 0.5000 | 0.3600 | 0.2267 | 50/50 (100.0%) | 4.06 | 10.14 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 150 | 0.4267 | 0.3133 | 0.2000 | 50/50 (100.0%) | 4.02 | 10.06 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 175 | 0.5133 | 0.4067 | 0.2333 | 50/50 (100.0%) | 4.00 | 9.96 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 200 | 0.7358 | 0.5454 | 0.3541 | 271/271 (100.0%) | 4.17 | 10.49 | ✓ Tool (269) |
| Baseline-4B-2507-sonibugfix-withrg | 200 | 0.6429 | 0.4755 | 0.2646 | 49/49 (100.0%) | 4.14 | 10.43 | ✓ Tool (49) |
| Baseline-4B-2507-sonibugfix-withrg | 225 | 0.6200 | 0.4733 | 0.2933 | 50/50 (100.0%) | 3.98 | 9.94 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 250 | 0.6133 | 0.4767 | 0.2967 | 50/50 (100.0%) | 3.98 | 8.94 | ✓ Tool (50) |
| Baseline-4B-2507-sonibugfix-withrg | 275 | 0.6633 | 0.5567 | 0.3567 | 50/50 (100.0%) | 4.04 | 7.70 | ✓ Tool (49) |
| Baseline-4B-2507-sonibugfix-withrg | 300 | 0.7100 | 0.5812 | 0.4402 | 273/273 (100.0%) | 4.03 | 9.02 | ✓ Tool (273) |
| Baseline-4B-2507-sonibugfix-withrg | 300 | 0.6224 | 0.5816 | 0.4469 | 49/49 (100.0%) | 4.00 | 8.94 | ✓ Tool (49) |
| a10_grpo_norm_by_std | 25 | 0.4300 | 0.2333 | 0.1213 | 50/50 (100.0%) | 3.74 | 9.46 | ✓ Tool (50) |
| a10_grpo_norm_by_std | 50 | 0.4947 | 0.3280 | 0.1513 | 50/50 (100.0%) | 4.02 | 8.98 | ✓ Tool (50) |
| a10_grpo_norm_by_std | 75 | 0.5033 | 0.3600 | 0.2200 | 50/50 (100.0%) | 4.08 | 9.24 | ✓ Tool (50) |
| a10_grpo_norm_by_std | 100 | 0.5400 | 0.3900 | 0.2333 | 50/50 (100.0%) | 3.96 | 7.62 | ✓ Tool (49) |
| a10_grpo_norm_by_std | 125 | 0.4600 | 0.3533 | 0.2058 | 50/50 (100.0%) | 4.00 | 9.44 | ✓ Tool (50) |
| a10_grpo_norm_by_std | 150 | 0.4267 | 0.2200 | 0.1333 | 50/50 (100.0%) | 3.04 | 6.04 | ✓ Tool (31) |
| a10_grpo_norm_by_std | 175 | 0.4600 | 0.3233 | 0.2133 | 50/50 (100.0%) | 3.78 | 6.48 | ✓ Tool (49) |
| a10_grpo_norm_by_std | 200 | 0.5200 | 0.3933 | 0.2147 | 50/50 (100.0%) | 4.00 | 6.30 | ✓ Tool (43) |
| a10_grpo_norm_by_std | 225 | 0.4600 | 0.3133 | 0.2133 | 50/50 (100.0%) | 4.18 | 5.94 | ✓ Tool (40) |
| a10_grpo_norm_by_std | 250 | 0.5100 | 0.3933 | 0.2333 | 50/50 (100.0%) | 4.32 | 6.00 | ✓ Tool (40) |
| a11_vanilla_grpo | 25 | 0.4847 | 0.2913 | 0.1200 | 50/50 (100.0%) | 4.28 | 10.80 | ✓ Tool (50) |
| a11_vanilla_grpo | 50 | 0.5900 | 0.3733 | 0.2067 | 50/50 (100.0%) | 3.40 | 5.96 | ✓ Tool (49) |
| a11_vanilla_grpo | 75 | 0.6267 | 0.4667 | 0.3267 | 50/50 (100.0%) | 4.00 | 6.00 | ✓ Tool (50) |
| a11_vanilla_grpo | 100 | 0.5300 | 0.3133 | 0.1533 | 50/50 (100.0%) | 4.44 | 6.52 | ✓ Tool (50) |
| a11_vanilla_grpo | 125 | 0.5367 | 0.4567 | 0.3067 | 50/50 (100.0%) | 4.00 | 6.80 | ✓ Tool (50) |
| a11_vanilla_grpo | 150 | 0.5867 | 0.4867 | 0.3467 | 50/50 (100.0%) | 4.00 | 7.08 | ✓ Tool (50) |
| a11_vanilla_grpo | 175 | 0.6000 | 0.4933 | 0.3333 | 50/50 (100.0%) | 4.04 | 7.06 | ✓ Tool (50) |
| a11_vanilla_grpo | 200 | 0.6898 | 0.5365 | 0.3607 | 274/274 (100.0%) | 4.02 | 7.07 | ✓ Tool (273) |
| a1_baseline_4b | 25 | 0.4480 | 0.2933 | 0.1590 | 50/50 (100.0%) | 5.46 | 11.84 | ✓ Tool (49) |
| a4_length_norm_sqrt | 50 | 0.5100 | 0.2300 | 0.0900 | 50/50 (100.0%) | 4.74 | 9.56 | ✓ Tool (50) |
| a4_length_norm_sqrt | 100 | 0.4767 | 0.2633 | 0.1700 | 50/50 (100.0%) | 5.08 | 11.40 | ✓ Tool (50) |
| a4_length_norm_sqrt | 150 | 0.4767 | 0.3667 | 0.2233 | 50/50 (100.0%) | 4.32 | 10.24 | ✓ Tool (50) |
| a4_length_norm_sqrt | 200 | 0.5000 | 0.3200 | 0.1667 | 50/50 (100.0%) | 3.98 | 8.62 | ✓ Tool (50) |
| a4_length_norm_sqrt | 250 | 0.5467 | 0.4333 | 0.2233 | 50/50 (100.0%) | 4.10 | 9.94 | ✓ Tool (50) |
| a4_length_norm_sqrt | 300 | 0.5267 | 0.3933 | 0.2533 | 50/50 (100.0%) | 4.10 | 9.90 | ✓ Tool (50) |
| a4_length_norm_sqrt | 350 | 0.4333 | 0.3333 | 0.2467 | 50/50 (100.0%) | 4.00 | 8.82 | ✓ Tool (50) |
| a4_length_norm_sqrt | 400 | 0.5467 | 0.4267 | 0.2867 | 50/50 (100.0%) | 4.04 | 8.46 | ✓ Tool (50) |
| a4_length_norm_sqrt | 450 | 0.5600 | 0.4733 | 0.3000 | 50/50 (100.0%) | 4.02 | 8.48 | ✓ Tool (50) |
| a5_sapo_on_sqrt | 25 | 0.4780 | 0.1900 | 0.0833 | 50/50 (100.0%) | 4.70 | 9.06 | ✓ Tool (44) |
| a5_sapo_on_sqrt | 50 | 0.4622 | 0.2467 | 0.1400 | 50/50 (100.0%) | 5.06 | 9.66 | ✓ Tool (46) |
| a5_sapo_on_sqrt | 75 | 0.5333 | 0.2900 | 0.1613 | 50/50 (100.0%) | 5.40 | 11.28 | ✓ Tool (49) |
| a5_sapo_on_sqrt | 100 | 0.4367 | 0.2700 | 0.1367 | 50/50 (100.0%) | 5.08 | 9.64 | ✓ Tool (50) |
| a5_sapo_on_sqrt | 125 | 0.4747 | 0.3233 | 0.1733 | 50/50 (100.0%) | 4.62 | 9.76 | ✓ Tool (49) |
| a5_sapo_on_sqrt | 175 | 0.5067 | 0.2900 | 0.1767 | 50/50 (100.0%) | 4.96 | 9.82 | ✓ Tool (50) |
| a5_sapo_on_sqrt | 200 | 0.5500 | 0.3700 | 0.1800 | 50/50 (100.0%) | 4.86 | 9.98 | ✓ Tool (49) |
| a6_eps_clip_02 | 25 | 0.4733 | 0.1780 | 0.0833 | 50/50 (100.0%) | 4.32 | 9.22 | ✓ Tool (48) |
| a6_eps_clip_02 | 50 | 0.4633 | 0.2713 | 0.1747 | 50/50 (100.0%) | 4.44 | 9.80 | ✓ Tool (49) |
| a6_eps_clip_02 | 75 | 0.4280 | 0.2167 | 0.1140 | 50/50 (100.0%) | 5.22 | 12.26 | ✓ Tool (50) |
| a6_eps_clip_02 | 100 | 0.4733 | 0.3100 | 0.1547 | 50/50 (100.0%) | 3.94 | 9.10 | ✓ Tool (50) |
| a6_eps_clip_02 | 125 | 0.4900 | 0.3100 | 0.1900 | 50/50 (100.0%) | 4.34 | 10.70 | ✓ Tool (50) |
| a6_eps_clip_02 | 150 | 0.4733 | 0.3000 | 0.1813 | 50/50 (100.0%) | 4.48 | 10.62 | ✓ Tool (50) |
| a6_eps_clip_02 | 175 | 0.4867 | 0.3267 | 0.1600 | 50/50 (100.0%) | 4.22 | 10.16 | ✓ Tool (49) |
| a6_eps_clip_02 | 200 | 0.5100 | 0.3267 | 0.1533 | 50/50 (100.0%) | 4.14 | 9.98 | ✓ Tool (50) |
| a8_dr_grpo_gspo | 25 | 0.3767 | 0.2033 | 0.1033 | 50/50 (100.0%) | 3.82 | 8.32 | ✓ Tool (50) |
| a8_dr_grpo_gspo | 50 | 0.5567 | 0.2933 | 0.1833 | 50/50 (100.0%) | 4.66 | 10.18 | ✓ Tool (49) |
| a8_dr_grpo_gspo | 75 | 0.5700 | 0.4067 | 0.2313 | 50/50 (100.0%) | 3.62 | 8.86 | ✓ Tool (50) |
| a8_dr_grpo_gspo | 100 | 0.5100 | 0.3633 | 0.2167 | 50/50 (100.0%) | 3.02 | 7.14 | ✓ Tool (48) |
| a8_dr_grpo_gspo | 125 | 0.5133 | 0.3600 | 0.2133 | 50/50 (100.0%) | 3.14 | 7.60 | ✓ Tool (46) |
| a8_dr_grpo_gspo | 150 | 0.5033 | 0.4600 | 0.2933 | 50/50 (100.0%) | 3.94 | 9.82 | ✓ Tool (50) |
| a8_dr_grpo_gspo | 175 | 0.5800 | 0.5027 | 0.3653 | 50/50 (100.0%) | 6.12 | 17.24 | ✓ Tool (50) |
| a8_dr_grpo_gspo | 200 | 0.5200 | 0.3867 | 0.2667 | 50/50 (100.0%) | 4.00 | 7.70 | ✓ Tool (49) |
| baseline | qwen3-4b | 0.3400 | 0.1067 | 0.0533 | 50/50 (100.0%) | 5.66 | 8.62 | ✓ Tool (26) |
| baseline-4b-300test | qwen3-4b | 0.7184 | 0.5821 | 0.4376 | 274/274 (100.0%) | 4.02 | 8.99 | ✓ Tool (273) |

## Evaluation Details

- **BASE_MODEL**: `Qwen3-4B-Instruct-2507`
- **Dataset**: SWE-Bench Lite (274 instances)
- **N-limit**: 300
- **Max Iterations**: 10
- **Num Workers**: 16
- **Instance Timeout**: 60s
- **Evaluation Date**: 2026-02-27 10:15:35


## Improvement Over Baseline (Qwen3-4B-Instruct-2507)

Baseline: File F1=0.3400, Module F1=0.1067, Entity F1=0.0533

| Experiment | Step | Δ File F1 | Δ Module F1 | Δ Entity F1 |
|------------|------|-----------|--------------|-------------|
| Baseline-14B | 10 | +0.0134 | +0.1256 | +0.1157 |
| Baseline-14B | 20 | -0.0210 | +0.0910 | +0.0861 |
| Baseline-14B | 30 | -0.0112 | +0.0820 | +0.0500 |
| Baseline-14B | 40 | +0.0910 | +0.1866 | +0.1380 |
| Baseline-14B | 50 | +0.1367 | +0.1300 | +0.0867 |
| Baseline-14B | 60 | +0.1667 | +0.2500 | +0.1634 |
| Baseline-14B | 70 | +0.2000 | +0.2866 | +0.1800 |
| Baseline-14B | 80 | +0.1500 | +0.2566 | +0.1600 |
| Baseline-14B | 90 | +0.1333 | +0.1146 | +0.1080 |
| Baseline-14B | 100 | +0.0667 | +0.1566 | +0.1234 |
| Baseline-14B | 110 | +0.1100 | +0.1811 | +0.1467 |
| Baseline-14B | 120 | +0.1400 | +0.2166 | +0.1434 |
| Baseline-14B | 130 | +0.2200 | +0.2466 | +0.1800 |
| Baseline-14B | 140 | +0.2267 | +0.2400 | +0.1534 |
| Baseline-14B | 150 | +0.1800 | +0.2433 | +0.1634 |
| Baseline-14B | 160 | +0.1667 | +0.2733 | +0.2214 |
| Baseline-14B | 170 | +0.0933 | +0.1800 | +0.1367 |
| Baseline-14B | 180 | +0.1600 | +0.3026 | +0.2110 |
| Baseline-14B | 190 | +0.1533 | +0.2533 | +0.1667 |
| Baseline-14B | 200 | +0.2167 | +0.3600 | +0.2597 |
| Baseline-14B | 210 | +0.1767 | +0.3233 | +0.2180 |
| Baseline-14B | 220 | +0.2100 | +0.4146 | +0.2647 |
| Baseline-14B | 230 | +0.1800 | +0.3493 | +0.2448 |
| Baseline-14B | 240 | +0.2033 | +0.3633 | +0.2634 |
| Baseline-14B | 250 | +0.2067 | +0.3560 | +0.2827 |
| Baseline-14B | 260 | +0.2000 | +0.3066 | +0.2800 |
| Baseline-14B | 270 | +0.1800 | +0.3200 | +0.2667 |
| Baseline-14B | 280 | +0.1833 | +0.3533 | +0.2657 |
| Baseline-14B | 290 | +0.1400 | +0.3366 | +0.2447 |
| Baseline-14B | 300 | +0.1667 | +0.3400 | +0.2467 |
| Baseline-14B | 310 | +0.1567 | +0.3533 | +0.2600 |
| Baseline-14B | 320 | +0.1600 | +0.3466 | +0.2567 |
| Baseline-14B | 330 | +0.1667 | +0.3400 | +0.2667 |
| Baseline-14B | 340 | +0.1800 | +0.3600 | +0.2734 |
| Baseline-14B | 350 | +0.1867 | +0.3533 | +0.2867 |
| Baseline-14B | 360 | +0.1833 | +0.3800 | +0.2867 |
| Baseline-14B | 370 | +0.1733 | +0.3366 | +0.2567 |
| Baseline-14B | 380 | +0.1533 | +0.3266 | +0.2500 |
| Baseline-14B | 390 | +0.2000 | +0.3966 | +0.2934 |
| Baseline-14B | 400 | +0.1600 | +0.3566 | +0.2534 |
| Baseline-14B | 410 | +0.1900 | +0.3666 | +0.2800 |
| Baseline-14B | 420 | +0.2033 | +0.3800 | +0.2934 |
| Baseline-14B | 430 | +0.2133 | +0.3866 | +0.2967 |
| Baseline-14B | 440 | +0.2067 | +0.3733 | +0.2867 |
| Baseline-14B | 450 | +0.1700 | +0.3533 | +0.2867 |
| Baseline-14B | 460 | +0.1967 | +0.4000 | +0.3000 |
| Baseline-14B | 470 | +0.1700 | +0.3733 | +0.2867 |
| Baseline-14B | 480 | +0.2033 | +0.4100 | +0.3067 |
| Baseline-14B | 490 | +0.2033 | +0.3400 | +0.2724 |
| Baseline-4B-2507-sonibugfix-withrg | 25 | +0.0733 | +0.1500 | +0.0600 |
| Baseline-4B-2507-sonibugfix-withrg | 50 | +0.1633 | +0.2400 | +0.0967 |
| Baseline-4B-2507-sonibugfix-withrg | 75 | +0.2267 | +0.2866 | +0.1200 |
| Baseline-4B-2507-sonibugfix-withrg | 100 | +0.2200 | +0.3333 | +0.1267 |
| Baseline-4B-2507-sonibugfix-withrg | 125 | +0.1600 | +0.2533 | +0.1734 |
| Baseline-4B-2507-sonibugfix-withrg | 150 | +0.0867 | +0.2066 | +0.1467 |
| Baseline-4B-2507-sonibugfix-withrg | 175 | +0.1733 | +0.3000 | +0.1800 |
| Baseline-4B-2507-sonibugfix-withrg | 200 | +0.3958 | +0.4387 | +0.3008 |
| Baseline-4B-2507-sonibugfix-withrg | 200 | +0.3029 | +0.3688 | +0.2113 |
| Baseline-4B-2507-sonibugfix-withrg | 225 | +0.2800 | +0.3666 | +0.2400 |
| Baseline-4B-2507-sonibugfix-withrg | 250 | +0.2733 | +0.3700 | +0.2434 |
| Baseline-4B-2507-sonibugfix-withrg | 275 | +0.3233 | +0.4500 | +0.3034 |
| Baseline-4B-2507-sonibugfix-withrg | 300 | +0.3700 | +0.4745 | +0.3869 |
| Baseline-4B-2507-sonibugfix-withrg | 300 | +0.2824 | +0.4749 | +0.3936 |
| a10_grpo_norm_by_std | 25 | +0.0900 | +0.1266 | +0.0680 |
| a10_grpo_norm_by_std | 50 | +0.1547 | +0.2213 | +0.0980 |
| a10_grpo_norm_by_std | 75 | +0.1633 | +0.2533 | +0.1667 |
| a10_grpo_norm_by_std | 100 | +0.2000 | +0.2833 | +0.1800 |
| a10_grpo_norm_by_std | 125 | +0.1200 | +0.2466 | +0.1525 |
| a10_grpo_norm_by_std | 150 | +0.0867 | +0.1133 | +0.0800 |
| a10_grpo_norm_by_std | 175 | +0.1200 | +0.2166 | +0.1600 |
| a10_grpo_norm_by_std | 200 | +0.1800 | +0.2866 | +0.1614 |
| a10_grpo_norm_by_std | 225 | +0.1200 | +0.2066 | +0.1600 |
| a10_grpo_norm_by_std | 250 | +0.1700 | +0.2866 | +0.1800 |
| a11_vanilla_grpo | 25 | +0.1447 | +0.1846 | +0.0667 |
| a11_vanilla_grpo | 50 | +0.2500 | +0.2666 | +0.1534 |
| a11_vanilla_grpo | 75 | +0.2867 | +0.3600 | +0.2734 |
| a11_vanilla_grpo | 100 | +0.1900 | +0.2066 | +0.1000 |
| a11_vanilla_grpo | 125 | +0.1967 | +0.3500 | +0.2534 |
| a11_vanilla_grpo | 150 | +0.2467 | +0.3800 | +0.2934 |
| a11_vanilla_grpo | 175 | +0.2600 | +0.3866 | +0.2800 |
| a11_vanilla_grpo | 200 | +0.3498 | +0.4298 | +0.3074 |
| a1_baseline_4b | 25 | +0.1080 | +0.1866 | +0.1057 |
| a4_length_norm_sqrt | 50 | +0.1700 | +0.1233 | +0.0367 |
| a4_length_norm_sqrt | 100 | +0.1367 | +0.1566 | +0.1167 |
| a4_length_norm_sqrt | 150 | +0.1367 | +0.2600 | +0.1700 |
| a4_length_norm_sqrt | 200 | +0.1600 | +0.2133 | +0.1134 |
| a4_length_norm_sqrt | 250 | +0.2067 | +0.3266 | +0.1700 |
| a4_length_norm_sqrt | 300 | +0.1867 | +0.2866 | +0.2000 |
| a4_length_norm_sqrt | 350 | +0.0933 | +0.2266 | +0.1934 |
| a4_length_norm_sqrt | 400 | +0.2067 | +0.3200 | +0.2334 |
| a4_length_norm_sqrt | 450 | +0.2200 | +0.3666 | +0.2467 |
| a5_sapo_on_sqrt | 25 | +0.1380 | +0.0833 | +0.0300 |
| a5_sapo_on_sqrt | 50 | +0.1222 | +0.1400 | +0.0867 |
| a5_sapo_on_sqrt | 75 | +0.1933 | +0.1833 | +0.1080 |
| a5_sapo_on_sqrt | 100 | +0.0967 | +0.1633 | +0.0834 |
| a5_sapo_on_sqrt | 125 | +0.1347 | +0.2166 | +0.1200 |
| a5_sapo_on_sqrt | 175 | +0.1667 | +0.1833 | +0.1234 |
| a5_sapo_on_sqrt | 200 | +0.2100 | +0.2633 | +0.1267 |
| a6_eps_clip_02 | 25 | +0.1333 | +0.0713 | +0.0300 |
| a6_eps_clip_02 | 50 | +0.1233 | +0.1646 | +0.1214 |
| a6_eps_clip_02 | 75 | +0.0880 | +0.1100 | +0.0607 |
| a6_eps_clip_02 | 100 | +0.1333 | +0.2033 | +0.1014 |
| a6_eps_clip_02 | 125 | +0.1500 | +0.2033 | +0.1367 |
| a6_eps_clip_02 | 150 | +0.1333 | +0.1933 | +0.1280 |
| a6_eps_clip_02 | 175 | +0.1467 | +0.2200 | +0.1067 |
| a6_eps_clip_02 | 200 | +0.1700 | +0.2200 | +0.1000 |
| a8_dr_grpo_gspo | 25 | +0.0367 | +0.0966 | +0.0500 |
| a8_dr_grpo_gspo | 50 | +0.2167 | +0.1866 | +0.1300 |
| a8_dr_grpo_gspo | 75 | +0.2300 | +0.3000 | +0.1780 |
| a8_dr_grpo_gspo | 100 | +0.1700 | +0.2566 | +0.1634 |
| a8_dr_grpo_gspo | 125 | +0.1733 | +0.2533 | +0.1600 |
| a8_dr_grpo_gspo | 150 | +0.1633 | +0.3533 | +0.2400 |
| a8_dr_grpo_gspo | 175 | +0.2400 | +0.3960 | +0.3120 |
| a8_dr_grpo_gspo | 200 | +0.1800 | +0.2800 | +0.2134 |
| baseline | qwen3-4b | +0.0000 | +0.0000 | -0.0000 |
| baseline-4b-300test | qwen3-4b | +0.3784 | +0.4754 | +0.3843 |

## Best Results by Metric

### File Localization F1
- **Experiment**: Baseline-4B-2507-sonibugfix-withrg (step 200)
- **Score**: 0.7358
- **Success Rate**: 271/271

### Module Localization F1
- **Experiment**: baseline-4b-300test (step qwen3-4b)
- **Score**: 0.5821
- **Success Rate**: 274/274

### Entity Localization F1
- **Experiment**: Baseline-4B-2507-sonibugfix-withrg (step 300)
- **Score**: 0.4469
- **Success Rate**: 49/49

## Learning Curves

### Baseline-14B

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 10 | 0.3534 | 0.2323 | 0.1690 |  |
| 20 | 0.3190 | 0.1977 | 0.1394 | -0.0344 |
| 30 | 0.3288 | 0.1887 | 0.1033 | +0.0098 |
| 40 | 0.4310 | 0.2933 | 0.1913 | +0.1022 |
| 50 | 0.4767 | 0.2367 | 0.1400 | +0.0457 |
| 60 | 0.5067 | 0.3567 | 0.2167 | +0.0300 |
| 70 | 0.5400 | 0.3933 | 0.2333 | +0.0333 |
| 80 | 0.4900 | 0.3633 | 0.2133 | -0.0500 |
| 90 | 0.4733 | 0.2213 | 0.1613 | -0.0167 |
| 100 | 0.4067 | 0.2633 | 0.1767 | -0.0666 |
| 110 | 0.4500 | 0.2878 | 0.2000 | +0.0433 |
| 120 | 0.4800 | 0.3233 | 0.1967 | +0.0300 |
| 130 | 0.5600 | 0.3533 | 0.2333 | +0.0800 |
| 140 | 0.5667 | 0.3467 | 0.2067 | +0.0067 |
| 150 | 0.5200 | 0.3500 | 0.2167 | -0.0467 |
| 160 | 0.5067 | 0.3800 | 0.2747 | -0.0133 |
| 170 | 0.4333 | 0.2867 | 0.1900 | -0.0734 |
| 180 | 0.5000 | 0.4093 | 0.2643 | +0.0667 |
| 190 | 0.4933 | 0.3600 | 0.2200 | -0.0067 |
| 200 | 0.5567 | 0.4667 | 0.3130 | +0.0634 |
| 210 | 0.5167 | 0.4300 | 0.2713 | -0.0400 |
| 220 | 0.5500 | 0.5213 | 0.3180 | +0.0333 |
| 230 | 0.5200 | 0.4560 | 0.2981 | -0.0300 |
| 240 | 0.5433 | 0.4700 | 0.3167 | +0.0233 |
| 250 | 0.5467 | 0.4627 | 0.3360 | +0.0034 |
| 260 | 0.5400 | 0.4133 | 0.3333 | -0.0067 |
| 270 | 0.5200 | 0.4267 | 0.3200 | -0.0200 |
| 280 | 0.5233 | 0.4600 | 0.3190 | +0.0033 |
| 290 | 0.4800 | 0.4433 | 0.2980 | -0.0433 |
| 300 | 0.5067 | 0.4467 | 0.3000 | +0.0267 |
| 310 | 0.4967 | 0.4600 | 0.3133 | -0.0100 |
| 320 | 0.5000 | 0.4533 | 0.3100 | +0.0033 |
| 330 | 0.5067 | 0.4467 | 0.3200 | +0.0067 |
| 340 | 0.5200 | 0.4667 | 0.3267 | +0.0133 |
| 350 | 0.5267 | 0.4600 | 0.3400 | +0.0067 |
| 360 | 0.5233 | 0.4867 | 0.3400 | -0.0034 |
| 370 | 0.5133 | 0.4433 | 0.3100 | -0.0100 |
| 380 | 0.4933 | 0.4333 | 0.3033 | -0.0200 |
| 390 | 0.5400 | 0.5033 | 0.3467 | +0.0467 |
| 400 | 0.5000 | 0.4633 | 0.3067 | -0.0400 |
| 410 | 0.5300 | 0.4733 | 0.3333 | +0.0300 |
| 420 | 0.5433 | 0.4867 | 0.3467 | +0.0133 |
| 430 | 0.5533 | 0.4933 | 0.3500 | +0.0100 |
| 440 | 0.5467 | 0.4800 | 0.3400 | -0.0066 |
| 450 | 0.5100 | 0.4600 | 0.3400 | -0.0367 |
| 460 | 0.5367 | 0.5067 | 0.3533 | +0.0267 |
| 470 | 0.5100 | 0.4800 | 0.3400 | -0.0267 |
| 480 | 0.5433 | 0.5167 | 0.3600 | +0.0333 |
| 490 | 0.5433 | 0.4467 | 0.3257 | +0.0000 |

### Baseline-4B-2507-sonibugfix-withrg

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.4133 | 0.2567 | 0.1133 |  |
| 50 | 0.5033 | 0.3467 | 0.1500 | +0.0900 |
| 75 | 0.5667 | 0.3933 | 0.1733 | +0.0634 |
| 100 | 0.5600 | 0.4400 | 0.1800 | -0.0067 |
| 125 | 0.5000 | 0.3600 | 0.2267 | -0.0600 |
| 150 | 0.4267 | 0.3133 | 0.2000 | -0.0733 |
| 175 | 0.5133 | 0.4067 | 0.2333 | +0.0866 |
| 200 | 0.7358 | 0.5454 | 0.3541 | +0.2225 |
| 200 | 0.6429 | 0.4755 | 0.2646 | -0.0929 |
| 225 | 0.6200 | 0.4733 | 0.2933 | -0.0229 |
| 250 | 0.6133 | 0.4767 | 0.2967 | -0.0067 |
| 275 | 0.6633 | 0.5567 | 0.3567 | +0.0500 |
| 300 | 0.7100 | 0.5812 | 0.4402 | +0.0467 |
| 300 | 0.6224 | 0.5816 | 0.4469 | -0.0876 |

### a10_grpo_norm_by_std

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.4300 | 0.2333 | 0.1213 |  |
| 50 | 0.4947 | 0.3280 | 0.1513 | +0.0647 |
| 75 | 0.5033 | 0.3600 | 0.2200 | +0.0086 |
| 100 | 0.5400 | 0.3900 | 0.2333 | +0.0367 |
| 125 | 0.4600 | 0.3533 | 0.2058 | -0.0800 |
| 150 | 0.4267 | 0.2200 | 0.1333 | -0.0333 |
| 175 | 0.4600 | 0.3233 | 0.2133 | +0.0333 |
| 200 | 0.5200 | 0.3933 | 0.2147 | +0.0600 |
| 225 | 0.4600 | 0.3133 | 0.2133 | -0.0600 |
| 250 | 0.5100 | 0.3933 | 0.2333 | +0.0500 |

### a11_vanilla_grpo

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.4847 | 0.2913 | 0.1200 |  |
| 50 | 0.5900 | 0.3733 | 0.2067 | +0.1053 |
| 75 | 0.6267 | 0.4667 | 0.3267 | +0.0367 |
| 100 | 0.5300 | 0.3133 | 0.1533 | -0.0967 |
| 125 | 0.5367 | 0.4567 | 0.3067 | +0.0067 |
| 150 | 0.5867 | 0.4867 | 0.3467 | +0.0500 |
| 175 | 0.6000 | 0.4933 | 0.3333 | +0.0133 |
| 200 | 0.6898 | 0.5365 | 0.3607 | +0.0898 |

### a1_baseline_4b

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.4480 | 0.2933 | 0.1590 |  |

### a4_length_norm_sqrt

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 50 | 0.5100 | 0.2300 | 0.0900 |  |
| 100 | 0.4767 | 0.2633 | 0.1700 | -0.0333 |
| 150 | 0.4767 | 0.3667 | 0.2233 | +0.0000 |
| 200 | 0.5000 | 0.3200 | 0.1667 | +0.0233 |
| 250 | 0.5467 | 0.4333 | 0.2233 | +0.0467 |
| 300 | 0.5267 | 0.3933 | 0.2533 | -0.0200 |
| 350 | 0.4333 | 0.3333 | 0.2467 | -0.0934 |
| 400 | 0.5467 | 0.4267 | 0.2867 | +0.1134 |
| 450 | 0.5600 | 0.4733 | 0.3000 | +0.0133 |

### a5_sapo_on_sqrt

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.4780 | 0.1900 | 0.0833 |  |
| 50 | 0.4622 | 0.2467 | 0.1400 | -0.0158 |
| 75 | 0.5333 | 0.2900 | 0.1613 | +0.0711 |
| 100 | 0.4367 | 0.2700 | 0.1367 | -0.0966 |
| 125 | 0.4747 | 0.3233 | 0.1733 | +0.0380 |
| 175 | 0.5067 | 0.2900 | 0.1767 | +0.0320 |
| 200 | 0.5500 | 0.3700 | 0.1800 | +0.0433 |

### a6_eps_clip_02

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.4733 | 0.1780 | 0.0833 |  |
| 50 | 0.4633 | 0.2713 | 0.1747 | -0.0100 |
| 75 | 0.4280 | 0.2167 | 0.1140 | -0.0353 |
| 100 | 0.4733 | 0.3100 | 0.1547 | +0.0453 |
| 125 | 0.4900 | 0.3100 | 0.1900 | +0.0167 |
| 150 | 0.4733 | 0.3000 | 0.1813 | -0.0167 |
| 175 | 0.4867 | 0.3267 | 0.1600 | +0.0134 |
| 200 | 0.5100 | 0.3267 | 0.1533 | +0.0233 |

### a8_dr_grpo_gspo

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| 25 | 0.3767 | 0.2033 | 0.1033 |  |
| 50 | 0.5567 | 0.2933 | 0.1833 | +0.1800 |
| 75 | 0.5700 | 0.4067 | 0.2313 | +0.0133 |
| 100 | 0.5100 | 0.3633 | 0.2167 | -0.0600 |
| 125 | 0.5133 | 0.3600 | 0.2133 | +0.0033 |
| 150 | 0.5033 | 0.4600 | 0.2933 | -0.0100 |
| 175 | 0.5800 | 0.5027 | 0.3653 | +0.0767 |
| 200 | 0.5200 | 0.3867 | 0.2667 | -0.0600 |

### baseline

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| qwen3-4b | 0.3400 | 0.1067 | 0.0533 |  |

### baseline-4b-300test

| Step | File F1 | Module F1 | Entity F1 | Δ File |
|------|---------|-----------|-----------|--------|
| qwen3-4b | 0.7184 | 0.5821 | 0.4376 |  |