How do I report on skill selection?

Table of Contents

The instrument records every skill the agent chooses, through a PreToolUse hook on the Skill tool. The hook and its rationale live in Claude Code Settings; this recipe is how you read what it collected.

1. Question

How good is our skill selection, and how do I record that a skill was chosen wrongly?

2. Answer

2.1. Read the report

./compass.sh skills report

The window defaults to 14 days. Pass --since 6h, --since 30d or --since all.

The report shows the number of selections, the correction rate, the range of catalogue sizes the agent chose from, the causes ranked, the split between attenuated and unattenuated sessions, and the most corrected and most chosen skills.

2.2. Record a correction

Selection is recorded automatically; whether it was wrong is a judgement, so it is recorded by hand, right after the correction happens: 

./compass.sh skills correct --cause description --chose compass-pr-merge

This marks the most recent selection. Pass --selection <id> to correct an earlier one. The cause is one word from a fixed set:

Cause Meaning
description The chosen skill's description misled; another skill fit better
catalogue-size Too many skills were visible to pick well
level The skill was wrong for the acting System
none Corrected, with no cause worth naming

2.3. Where the log lives

Append-only JSONL in the git common directory, so every worktree in the fleet appends to one file and the baseline accrues across checkouts rather than restarting on each env provision. compass skills report prints the path when the log is empty.

3. Script

projects/ores.compass/src/compass_skills.py holds the command and the event shapes. record_skill_selection_hook.py is the hook, tangled from doc/llm/claude_code_settings.org.

4. Tested by

projects/ores.compass/tests/test_compass_skills.py and projects/ores.compass/tests/test_record_skill_selection_hook.py.

5. See also

Emacs 29.3 (Org mode 9.6.15)