AI Engineer Coach: the feedback loop prompt engineering needs

Prompt engineering advice is everywhere. The missing piece is feedback on your prompts after they meet a real codebase, a real agent, and a real deadline. That is why AI Engineer Coach caught my attention. It does not try to write another magic prompt for you. It reads the local logs from the coding assistants you already use and turns your habits into something you can inspect, measure, and improve.

The result is closer to a training dashboard for agentic engineering than a chat feature. It shows where your sessions go off the rails, where you are repeating yourself, and which project context is helping—or hurting—the agent. That makes it a genuinely useful tool for getting better at prompting over time.

The value is not a score for its own sake. It is shortening the loop between a weak prompting habit and the next better prompt you try.

What AI Engineer Coach actually does

AI Engineer Coach is an open-source VS Code extension and dashboard. It analyzes local AI coding-session logs, then groups the results into four useful questions:

• What happened? The dashboard, timeline, coding-moment gallery, and workspace filters make it easier to review how you used an agent across days and projects.
• What did the agent produce? The Output view breaks down generated code by language, workspace, model, and harness.
• Which habits are costing me? Its Anti-Patterns area applies 45 editable rules across prompt quality, session hygiene, code review, tool use, and context management.
• What should I change next? Findings include severity, concrete actions, and example prompts; the learning and skill features turn recurring work into deliberate practice.

Those features matter together. A chat transcript can tell you what you typed. A useful coach needs to help you connect that behavior to an outcome, spot a recurring pattern, and then give you a small experiment for the next session.

Why this can make you better at prompting

Most weak prompts do not fail because they are missing a clever phrase. They fail because the task, constraints, available context, success conditions, or review loop are underspecified. In day-to-day agent work, the cost shows up as a vague request, a wandering session, several cancellations, and finally a large patch that still needs to be untangled.

The project makes that pattern visible. Its rule set includes checks for lazy prompting, low constraint usage, no file context, repeated prompts, session drift, excessive file context, instruction bloat, and skipping plan or specification structure. None of these labels is a universal verdict—rules are heuristics, not a measure of engineering ability—but they are excellent prompts for reflection.

For example, suppose the coach keeps flagging low constraint usage. Instead of asking an agent, “add authentication,” I would turn that into a reusable opening brief:

Implement the session-refresh endpoint in the existing auth module.

Constraints:
- Preserve the current cookie and error-response conventions.
- Do not change the database schema.
- Add focused tests for expired and revoked refresh tokens.
- Before editing, identify the relevant files and give me a short plan.

Done means the existing auth test suite passes and the new cases are covered.

That is not a rigid prompt template. It is a better contract: scope, constraints, evidence, and a definition of done. The next time the same issue appears in the dashboard, you can compare whether the new operating habit reduced the friction. This is how prompt engineering stops being a collection of tips and becomes a practice.

More than prompt wording: context engineering

The most interesting part of the project is its Context Health work. It scores context health, builds a workspace context map, checks agentic readiness, and audits instruction files. That puts the focus in the right place: a good prompt can only do so much if the agent has a chaotic repository, a missing AGENTS.md, or a stale pile of instructions.

In practice, I would use the tool in two loops:

1. Personal loop: Review the highest-severity prompt and session findings once a week. Pick one behavior to change for the next few sessions—such as planning before editing, naming the relevant files, or stating test expectations.
2. Project loop: Use Context Health to clean up the durable context your agents depend on: repository instructions, local setup, architecture notes, and reusable skills. Then check whether the pattern of missed context improves.

There is also a Skill Finder that looks for repeated prompt patterns and surfaces matching community skills. That is a practical bridge from “I keep typing this” to “this should be a reusable workflow with guardrails.”

What it supports today

The published extension is designed for VS Code 1.115 or later. It can also run as a canvas in the GitHub Copilot app. The current code explicitly collects local session sources from GitHub Copilot in VS Code, plus external logs for Claude Code, Codex CLI, and OpenCode. The dashboard can filter the data by workspace and harness, so someone who switches tools does not have to treat their prompting history as separate silos.

The GitHub Copilot app canvas supports the log-driven views—Dashboard, Timeline, Coding Moments, Output, Patterns, and Anti-Patterns. Features that need VS Code’s local language-model API, including Skill Finder, Learning Center, Level Up, and the AI portion of Context Health, remain VS Code-only in that mode.

One setup caveat: this is not a Marketplace extension at the time of writing. You build a .vsix from the repository with npm ci and npm run package, then install it in VS Code. That is a little more work, but it also makes the project easy to inspect and contribute to. It is MIT licensed and described by its maintainers as a community effort by Microsoft employees, not an official Microsoft product.

A privacy story worth noticing

Session logs can reveal source paths, prompts, and work habits. The project’s default model is reassuring: the extension is read-only, analyzes logs locally, and does not include proprietary telemetry. Some optional features—such as the rule compiler, Skill Finder, and context review—use VS Code’s built-in Copilot language-model API only when you explicitly invoke them.

That does not eliminate the normal responsibility to review what your coding assistant and its provider retain. It does mean the analytics layer is not automatically creating a second cloud destination for your session history.

How I would use it without gaming the score

I would resist the temptation to optimize every metric. Long sessions are sometimes appropriate. A concise prompt is not automatically better. And no static rule can decide whether a given repository really needed an agent plan or an extra file of context.

Instead, I would treat the dashboard as a retrospective partner. Each week, choose one recurring anti-pattern, read the provided action and example prompt, and make a small change. Keep the change if it produces clearer plans, fewer restarts, safer diffs, or easier reviews. Drop it if it creates ceremony without value.

That is the promise of AI Engineer Coach: it gives prompt engineering a feedback loop. Not “write the perfect prompt once,” but observe your real work, make a sharper request next time, and turn what works into durable project context and skills.

If you work with Claude Code, Codex, OpenCode, or GitHub Copilot, I think it is well worth a test drive. Start with a week of history, look for the one finding that feels uncomfortably familiar, and let that become your next deliberate practice.