Agent Effectiveness

Agent Effectiveness measures how much faster your engineering organization ships with Factory, and connects agent spend to the work it produces.

Most teams that adopt AI coding agents can tell that work is moving faster, but they have a hard time proving it. Survey-based estimates of "how long would this have taken before AI" are unreliable, and existing delivery metrics show how fast work moves without showing how much of that speed came from agents.

Agent Effectiveness answers that question from data your organization already produces: sessions in Factory, and issues, projects, and pull requests in the tools your teams already use.

How it works

Agent Effectiveness reads from your connected project management, issue-tracking, and source control integrations, then links agent activity to the work those tools track. As agent spend is applied, cycle times on projects, issues, and pull requests change measurably, and teams use the recovered time either to take on more work or to raise the bar on the work they already have.

Analysis happens at the organization level and requires no per-repository configuration.

What it measures

Throughput

How cycle times on projects, issues, and pull requests change as agent spend is applied, and which projects, pods, and users the change is concentrated in.

Output

Where effort is going, using session intents to classify work and compare the split of engineer time and Factory Standard Credits against your targets.

Attribution

The link between an individual session, the work stream it belongs to, and the issues, projects, and artifacts it produced.

See the Effectiveness dashboard for how each view is presented in the Factory App.

Session intents

A session intent is a classification of what a session was for, such as feature development, maintenance, bug fixing, or exploration. Intents let you compare planned allocation against actual allocation during the quarter rather than after it.

When the mix drifts from your targets, the Output view surfaces the drift while there is still time to act on it.

Attribution

Attribution connects a session to the pull requests, tickets, and documents it worked on. Factory records those links when it classifies the session, so an organization-level number can be traced back to the specific work behind it.

The same data is available through the Analytics API, so you can build your own ROI reporting from the agent spend behind each pull request, ticket, project, or session; see Advanced Analytics in the Analytics API.

How session classification works

Session classification powers Agent Effectiveness. When an organization turns on Advanced Analytics, Factory classifies each session by running a small set of rubrics over windows of it. Factory will refine, add, and retire rubrics over time.

Intent and completion

What the session set out to do (feature work, bug fixing, maintenance, or exploration), whether it got there, and which pull requests, tickets, and documents it worked on. It powers the Output and Attribution views, which Owners and Managers see in Analytics.

Friction

Moments that made a session worse: errors that blocked progress, the agent going in circles, the user repeating themselves or undoing the agent's work, or visible frustration. The normal write-test-fix loop and one-off retries don't count. Each moment gets a severity and a cause, such as model provider errors, permission gates, tool bugs, billing limits, the user's environment, the agent's own behavior, or third-party outages.

Expertise

How familiar the user was with the task in that window, and how many planning and execution decisions the user made versus the agent. It is adapted from Anthropic's research on agentic coding and expertise, and leaves out occupation and dollar-value estimates. Expertise is not shown in any customer UI or API.

Privacy and controls

• Opt-in. Agent Effectiveness is off by default and available on Enterprise plans only. An Owner or Manager turns it on with the Advanced Analytics enterprise control. For organizations that haven't enabled it, Factory doesn't classify sessions.
• Cloud sync. Factory-managed session classification requires cloud session sync to be on at the organization level to classify new sessions.
• Model policy. Classifier selection is checked against your organization's effective model policy, and it may use a different model from your chat session. Custom (BYOK) classifiers use a separate inference route and are checked against the applicable custom-model and endpoint permissions.
• Model training. Under Factory's enterprise agreements, customer data isn't used to train models.

Other usage and adoption metrics

Agent Effectiveness focuses on delivery speed and the link between spend and shipped work. Adjacent measures are documented separately:

Requirements

Agent Effectiveness is available for Enterprise organizations. It requires organization-level integrations plus the Advanced Analytics enterprise control. See Enable Agent Effectiveness for the full setup.