Glossary: Measuring the impact of GitHub Copilot

The GitHub Copilot Impact Glossary

This glossary lists a number of measures that an organization can use to help evaluate the impact of Copilot for their organization. Learn:

• Definitions and caveats related to measures
• When (which stage) to use a measure
• How to access a measure (source data location)

For more information on GitHub’s approach to measuring Copilot Impact, check out our GitHub Copilot Learning Pathway.

Remember 😀: The relevance and usefulness of system-level speed and quality measures vary across organizations. Any Copilot impact beyond coding is indirect, and the downstream possibilities are endless. Thus, we must be careful when searching for causality in speed and quality measures.

The metrics provided below are representative of processes that often improve with Copilot, but the degree of impact will vary. Based on the strengths and weaknesses of your organization, the impact may be more or less. It's less important to achieve a benchmark than to observe neutral-to-positive improvement. Engineering leaders should only be concerned if a metric regresses. In that case, pause to determine the root cause before proceeding.

Tips:

• GitHub recommends looking for improvements in each measure over time rather than setting fixed targets.
• We suggest aligning measurement frequency with your development cadence (e.g. sprint cycles).
• You may choose to collate your data points into existing BI or reporting tools for ease of analysis and review over time for trends. Alternatively, use a spreadsheet to track measures over time.
• As you add developers to Copilot you may see a decrease in some measures as those developers build capability and confidence in using Copilot as part of their daily flow. Consider analyzing measurements according to onboarding groups (set of developers provided with a Copilot seat).
• You may choose to monitor some DevOps metrics over the evaluation and adoption stages for early indicators of likely system- impact areas. But particularly in the evaluation and adoptions stages, we recommend focusing on happiness and engagement measures. Once developers are using Copilot effectively, system level measures are more likely to improve.

Category legend:

Developer Happiness - 😃
Developer Engagement - 👩‍💻
Speed - 🏎️
Quality - ✨

Measure	Definition	Stage	Source data location
Satisfaction using Copilot - 😃	Number of Copilot users who value Copilot in their workflow.	All	Copilot Impact Survey: • How would you feel if you could no longer use Copilot?
Benefits of using Copilot - 👩‍💻	Range of areas in which users leverage Copilot and the range/extent of benefits experienced.	All	Copilot Impact Survey: • When using Copilot, I… • When using Copilot, my team… • In using Copilot in the last week… • I use GitHub Copilot to…
Challenges using Copilot - 👩‍💻	Number of users impacted by potential adoption barriers.	All	Copilot Impact Survey: • What challenges have you encountered during your Copilot trial?
Enablement provided - 👩‍💻	Number of users with a seat who have received enablement resources/training sessions.	All	Copilot Impact Survey: • Which of the following GitHub Copilot enablement materials have you received?
Average Daily Active Users per Month - Completions - 👩‍💻	Users who have received 1+ suggestions on a given day.	All	Copilot Metrics API: • Take average of Active Users - Completions from API, over 28 days
Average Daily Active Users - Chat - 👩‍💻	Users who have 1+ chats on a given day.	All	Copilot Metrics API: • Take average of Active Users - Chats from API, over 28 days
Average Daily Active Users - Total - 👩‍💻	Unique users who have received and accepted 1+ suggestions OR participated in 1+ chats on a given day.	All	Copilot Metrics API: • Take average of Active Users - Total from API, over 28 days
Suggestions Delivered - Completions - 👩‍💻	Number of suggestions per day displayed in the code editor for Copilot completions in IDE.	All	Copilot Metrics API: • Number of Suggestions
Number of Acceptances - Completions - 👩‍💻	Number of suggestions accepted (total of both partial and full) per day.	All	Copilot Metrics API: • Number of Chat/code acceptances
Lines of Code Accepted - Completions - 👩‍💻	Lines of code accepted (total of both partial and full) accepted per day	All	Copilot Metrics API: • Lines of Code Accepted - Completions
Total Acceptance rate - Completions - 👩‍💻	Suggestions Accepted (Combined: Full and Partial) / Total Number of Suggestions	All	Manual calculation (based on Copilot Metrics API): • Number of Suggestions & • Number of Acceptances
New seats added in billing cycle - 👩‍💻	Additional seats purchased for organizational members during this billing cycle	Adopt, Optimize, Sustain	Billing and plans (Copilot) (organizational settings) • New seats added during this billing cycle
Dormant users - 👩‍💻	Number of users who last used Copilot more than ‘n’ days ago	Adopt, Optimize, Sustain	Copilot Access Management (organizational settings): • Sort by last used
Total completed pull requests - 🏎️	Total number of pull requests completed in a given week	Adopt, Optimize, Sustain	GitHub PRs list (in a repo) can be filtered on closed PRs for a time period
Pull requests per developer - 🏎️	Number of pull requests / code contributors	Adopt, Optimize, Sustain	Manual calculation: • GitHub PRs list (in a repo) can be filtered on closed PRs for a time period & • Your organization's data on developers contributing to repo
Time to merge - 🏎️	Average time from PR opened to merge	Adopt, Optimize, Sustain	GitHub: calculation requires consideration of your organization's GitHub set-up
PR lead time - 🏎️	Average time from PR opened to deploy	Adopt, Optimize, Sustain	GitHub Actions: calculation requires consideration of your organization's GitHub set-up
Code Review Time - 🏎️	Average code review time per pull request (time between PR opened to PR closed)	Adopt, Optimize, Sustain	GitHub: calculation requires consideration of your organization's GitHub set-up
Pull request merge rate - ✨	Total number of successful builds completed in a given week.	Adopt, Optimize, Sustain	GitHub: GitHub PRs list (in a repo): The number of PRs merged / The number of PRs submitted
Total successful builds - ✨	The percentage of pull requests successfully merged	Adopt, Optimize, Sustain	GitHub Actions: calculation requires consideration of your organization's GitHub set-up
Change failure rate - ✨	Percentage of code deployments that caused a failure in production	Adopt, Optimize, Sustain	Calculation requires consideration of your organization's GitHub set-up and integration with third-party apps
Open security vulnerabilities - ✨	Number of unresolved vulnerabilities as identified by GHAS	Adopt, Optimize, Sustain	Organizational-level security insights
CI success rate - ✨	Number of successful CI runs inclusive of test automation / Number of attempted CI runs	Adopt, Optimize, Sustain	Calculation requires consideration of your organization's GitHub set-up and integration with third-party apps