2 years ago

AI Code Monitor Beta V3.10: ROI of AI

Here's the latest video release notes: https://youtu.be/QNRl-pP0sYA

In the current release, we have implemented the ROI of AI functionality, which is explained below:

The basic point here is to enable Engineering leadership teams to show how much more productive the team is as a result of using Generative AI tools, how much more effective they are currently and what percentage of the potential have they gained?

Reason one for this functionality is to help, guide and provide teaching and coaching opportunities for the Engineering teams.

Reason two is to give updates to the rest of the organization, who, of course, are very keen to understand how Gen AI coding is affecting the Engineering team

How it works: Measuring ROI of AI happens at the repository group / product level. That's because there's so much nuance for each team, it doesn't really make sense to set measures for the Engineering organization as a whole. There's two parts to how the ROI of AI works.

> One, there are settings that Engineering leaders can tune based on the specifics of their situation.

> And then second, the AI Code Monitor detects how much Gen AI there is and comes up with a conclusion based on the settings from the Engineering leader.

On part one, Engineering leaders select three things:
1. How much time are Engineers spending in coding. If they're only spending 30% of their time coding, even no matter how good the AI tool is it's there's only going to be a 30% increase in potential impact.
2. What is the primary Engineering focus of that product, such as starting a new code base, or within an existing code bas -, adding features and functionality, or tech debt reduction. There are five choices to select from. Sema has default expected productivity improvement, based on research, which is available to review via a link to a blog post within the product. Then, Engineering leaders can override the defaults based on the specifics of their situation.
3. Decide how much Gen AI is appropriate, given the unique circumstances of that team and what they're working on. If the team uses Gen AI to that maximum desired level, then they are getting the full benefit of whatever the potential productivity gain is.

In this example, for sample product "Crow-Proof of Concept":
1. The Engineering team spends 30% of their time coding
2. GenAI could generate 30% productivity improvement
3. The Engineering team reaches its maximum benefit of GenAI when 30% of the code originates with GenAI.

The results are:

1. The team is now 3% more productive as a result of Gen AI usage
2. They have captured 36% of the potential from using Gen AI.

2 years ago

AI Code Monitor Beta V3.9: Historical, Cumulative & Usage in time period charts; Dynamic scaling of charts; Sort & Export Developers

Here's the latest video release notes: https://youtu.be/jfgKHWRKHss

In the current release, we have implemented a few customer-centric features, which are explained below:

1) Historical chart display on Dashboard: The user can go a month back from the day of setting up on AI Code Monitor tool and view the charts on the Dashboard to view the GenAI usage% - Overall, Pure & Blended

2) Display 6 charts on Dashboard: The user can now view 3 charts for Cumulative Usage, displaying Cumulative line chart for Overall, Pure & Blended AND view 3 charts for Usage in time period, displaying Cumulative line chart for Overall, Pure & Blended

3) Dynamic scale to change the y-axis on the charts on Dashboard, allowing the user to change the scale and compare the stats on the charts

4) Developer Group Page-Export to csv: Ability to Export all the data on this page on CSV for further analysis. This enables you to build a sprint review report as well, if needed.

5) Developer Group Page-Sort option: At Developer group level, user has an option to Sort by Name, GenAI, NotGenAI, PureAI, Blended. With this option, user can view the names of Developers based on these sorting parameters (Ascending & Descending). If the user wants to know which Developers have highest GenAI% usage or which Developers have highest Pure GenAI% usage, then they can sort and see the list of developers.

6) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

AI Code Monitor Beta V3.8: Status check, Date picker, Pagination, Link to merge developers

Here's the latest video release notes: https://youtu.be/OS0ltMvfpe8

In the current release, we have implemented a few customer-centric features, which are explained below:

1) Status check in Settings area: When enabled, status checks of pull requests that don’t meet the rules will be marked as “failed” in GitHub. You can keep this disabled as well.

2) Date picker at Repository group: You can now select the date range to view the charts displaying the Overall, Pure & Blended stats based on the date range selected. So, in case you are doing your Sprint retrospective, you could select the last 2 weeks and evaluate the GenAI usage at your Repository group levels.

3) Date picker at Dashboard: You can select the date range and view the GenAI charts for Overall, Pure & Blended based on the date range selected. This shows the stats at the Organization level. User can select:

Last 7 days
Last 14 days (This is default)
Last 30 days
Last 90 days
Last 365 days
All time

4) Increase pagination from 20 to 50: This will help a Dev Manager or tech lead view more developer names under each Developer group

5) Add a link to Merge Developers page: This will help a Dev manager or Tech lead easily navigate to merging the duplicate developer names

6) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

AICM Tool as of July 2024: Quick recap of Product Features & Benefits

Here's the latest video of the Long version Recap of Product Features & Benefits of AICM tool: https://youtu.be/xfmg2RKcE6w (3mins 26 sec)

Here's the latest video of the Short version Recap of Product Features & Benefits of AICM tool: https://youtu.be/OC42J6t3bPM (1min 50 sec)

Here's the latest video of the Shortest 30-second Quick Overview of AICM tool: https://youtu.be/RnjUfE4HaqI (34 sec)

Sign Up > Connect with GitHub or AZDO
View Dashboard > Overall, Pure, Blended
View Dashboard > Rules, Traffic Light
View Dashboard > Chart
Repositories > Group
Repositories > NotGenAI, GenAI (Pure, Blended)
Repositories > Rules, Traffic Light, Chart
Repositories > % Code attested
Repositories > Individual repositories
Repositories > Repository Detail Page
Repository Detail Page > Folder, Files, Lines of Code, Chunk Labels
Repository Detail Page > Search, Filter
Repository Detail Page > Author
Repository Detail Page > Attestation, ToolTip
PR Scan Review Page > GitHub PR page, GenAI Detector, More Details
PR Scan Review Page > GenAI Composition for PR, Folder, Files, Lines of Code, Chunk Labels
PR Scan Review Page > Search, Filter
PR Scan Review Page > Author
PR Scan Review Page > Attestation, Tool Tip
Settings > Members
Settings > Rules
Settings > Repository Groups

# Quick Overview of the AICM SaaS Tool:

1) Sign-Up and Set Rules

2) View GenAI usage stats at Organization level, PR Code Review level and Repository level

3) Attest the code by Accepting or Overriding the label at PR Code review level or on the Repository level, thus making a permanent record of GenAI code labels in the codebase

2 years ago

AI Code Monitor Beta V3.7: Chunk count, Expanded File tree, Charts at Repo level, Attestation% at Repo level & Repo group level

Here's the latest video release notes: https://youtu.be/OUwCdLwdXVs

In the current release, we have implemented a few customer-centric features, which are explained below.

We have done some simple improvements to enhance the user experience on the AICM tool, which include:

1) Favicon: With the implementation of the Sema favicon, the user can easily identify the Sema browser amongst all open browsers

2) ToolTip: When the user is attempting to do the Attestation on Repository detail page or the PR Code review detail page, then these helpful tooltips come handy to inform what does the 'Tick' and 'Cross' symbol mean.

3) Improvement on Hyperlink at Repository level: In continuation of providing a good user experience on Repositories page, all the repositories that have Zero analyzed files are now listed at the bottom without any hyperlink, hence indicating the user that no code-level information is available for this repository. The GenAI% and NotGenAI% are displayed as "N/A".

In this release, we have implemented some new features which include:

4) Easily find files that have selected labels on Repo detail Page & PR scan detail page: When the user is on the PR Code review detail page, the file tree is fully expanded by default for the user to easily view which files have the chunks related to this PR. Based on the selected filters (NotEvaluated, PureAI, NotGenAI, Blended), the count of chunks are shown against each file, making it intuitive which file to click to view the related chunks. For e.g. if you are interested to view only the 'PureAI' chunks, then you can select the filter as 'PureAI' and the files that have the 'PureAI' chunks will be listed with a count>0. This makes it efficient for the user to browse across the files and click only on those files. The same feature is available at the Repository detail page, with the only difference being that the whole file tree doesn't show expanded by default. This is done to load the page faster.

5) Display charts for repository group: The Dashboard has the charts to display the GenAI statistics at the organization level. Similarly, the charts are now displayed at the Repository group level. When you expand a Repository group, you will see the stats of that Repository in charts and the associated respotitories will be listed below that.

6) Display the percentage of attested code per group and repository: When the user is at the Repository group level and at Repository level, they can easily find out what percentage of the lines of code have been attested. This is a great identifier for them to know where they need to take action w.r.t. Attestation. The equation to calculate the Attestation% is as follows: numerator = lines of code that is attested. denominator = lines of code that can be attested.

7) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

Frequently Asked Questions (FAQs) on AI Code Monitor Tool

What is Pure GenAI? How do you calculate it?
What is Blended GenAI?
What is Overall?
What is NotGenAI%?
What does this mean? "97.11% of code evaluated"
What do you mean by "2% code attested"?
What do you mean by "Not evaluated: 24 files"?
What do you mean by "Python: 25 files"?
What do you mean by "N/A" under GenAI% and NotGenAI%?
What is Repository Group?
What is Attestation?
What is the 'Tick' icon and 'Cross' icon on Attestation?
What is "Status checks" under Settings>Oraganization Information?
What do you mean by "Apply at organization level"?
What is GBOM (GenAI Bill of Materials)?

------------ ------------ ------------ ------------ ------------

1. What is Pure GenAI? How do you calculate it?

Percentage of chunk of code labeled as GenAI, where the whole chunk belongs to single commit. In order to calculate Pure GenAI%, we first detect that code is GenAIOriginated. If yes, we check if there is a second commit. If there is a second commit, we mark it as Blended GenAI. If there is no second commit, we mark it as Pure GenAI.

2. What is Blended GenAI?

Percentage of chunk of code labeled as GenAI, where the whole chunk belongs to multiple commits. In order to calculate Blended GenAI%, we first detect that code is GenAIOriginated. If yes, we check if there is a second commit. If there is a second commit, we mark it as Blended GenAI.

3. What is Overall?

Total percentage of GenAI in the codebase. It is sum of Pure GenAI% and Blended GenAI%

4. What is NotGenAI%?

Total percentage of human-generated code. Example: Code generated by:

Developer at the organization
External developer (such as Open Source code, or Stack Overflow-copied code)
Auto-generated code (such as code provided alongside frameworks)

5. What does this mean? "97.11% of code evaluated"

Ticket CAW-840 is work-in-progress to clarify this definition

6. What do you mean by "2% code attested"?

This % is number of lines that has been attested divided by the number of lines that can be attested.

Not all code is attestable yet. For e.g. 946/47314 = 2%

7. What do you mean by "Not evaluated: 24 files"?

This is the count of files that has not been evaluated. The definition of 'Not Evaluated' is work-in-progress under ticket CAW-840

8. What do you mean by "Python: 25 files"?

This shows that there are 25 files of type 'Python'

9. What do you mean by "N/A" under GenAI% and NotGenAI%?

The % number under GenAI% and NotGenAI% represents the % of code that is GenAI and NotGenAI respectively. When that particular repository has Analysed files = Zero, then no calculation has been performed and hence it displays "N/A" i.e. Not Applicable.

10. What is Repository Group?

A user of AICM tool can create groups of repositories by giving a Repository Group Name and selecting a Usage category. This is useful in grouping repositories that you want to monitor against similar rules and get an average metric at a Repository group level.

11. What is Attestation?

Teams within the organization can create a permanent record of GenAI code composition, we call this “attestation”. They can do this by confirming or overriding the AI / Not AI label. So far, we have implemented Attestation at 2 levels: 1. Repository detail page level 2. Code review level

Refer this link for more information: https://announcekit.co/semasoftware/release-notes/ai-code-monitor-beta-v3.4-attestation-of-genai-code-labels-4yEm9q

12. What is the 'Tick' icon and 'Cross' icon on Attestation?

Tick - Attest by Agreeing with the label - When the user clicks on this icon, they are attesting that the label on that chunk is correct

Cross - Attest by Overriding the label - When the user clicks on this icon, they are attesting that the label on that chunk is incorrect and they override that by selecting another label and have an option to write a comment.

13. What is "Status checks" under Settings>Oraganization Information?

When this option is selected under 'Settings > Organization Information', then the Pull Request page on GitHub/Azure DevOps for that organization will show another check called 'Sema GenAI Detetector'

14. What do you mean by "Apply at organization level" for Rules under Settings?

When this option is selected, it means that this Rule is applicable to the whole Organization, including all repository groups and developer groups.

15. What is GBOM (GenAI Bill of Materials)?

If you are a user of AI Code Monitor tool and you want to get a detailed view of the GenAI usage for the developers in your organization, then you can download the GBOM file. For more information please refer: https://announcekit.co/semasoftware/release-notes/view-developer-usage-in-detail-using-gbom-download-how-to-video-1AJ70I

2 years ago

AI Code Monitor Beta V3.6: Display Author name at code level and display "N/A" at stats level where needed

Here's the latest video release notes: https://youtu.be/q98NJ3xHu7o

In the current release, we have implemented a few customer-centric features, which are explained below.

1) Display developer names next to the lines of code: The senior leaders in the organization like VPs / Engineering Managers can view the names of the developers next to the lines of code on both Repository detail page and on the PR scan detail page. This will help them reach out to the respective developers who committed the code.

2) Display "N/A" stats on repositories and groups that have no files analyzed: When the users are viewing the GenAI usage stats at the repository and repository group level, then they will see 'N/A' in the stats in case there were Zero analyzed files, which is an accurate representation of what has been analyzed and being used for calculation of the stats.

3) Under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

How to find GenAI usage stats

This post includes the steps to view the GenAI code usage in :

1) GitHub Copilot

2) Amazon CodeWhisperer

3) Sema AI Code Monitor

#1 To see the level of GenAI code use in GitHub Copilot, you can follow these steps:

1a) GitHub Copilot Dashboard

The QuickStart guide can be accessed here: https://docs.github.com/en/copilot/quickstart

GitHub provides a dashboard for Copilot users that includes usage statistics. This dashboard can show you how much of your code is being generated by Copilot. To access it:

Open Visual Studio Code (VS Code) or any other supported IDE where you have GitHub Copilot installed.
Click on the GitHub Copilot icon in the sidebar or navigate to the Copilot settings.
Look for a section labeled "Copilot Statistics" or similar. This section will display metrics such as the percentage of code suggestions accepted, the number of suggestions made, and other relevant usage statistics.

1b) GitHub Copilot Insights

GitHub Copilot Insights is a feature that provides detailed analytics on how Copilot is being used in your projects. This can include:

The percentage of code generated by Copilot.
The types of suggestions being made.
The acceptance rate of Copilot's suggestions.

To access Copilot Insights:

Go to the GitHub Copilot website or the settings within your IDE.
Navigate to the "Insights" or "Analytics" section.

#2 To see the level of GenAI code use in Amazon CodeWhisperer, you can utilize several methods and tools provided by AWS:

2a) CodeWhisperer Usage Analyzer

The QuickStart guide can be accessed here: https://docs.aws.amazon.com/codewhisperer/latest/userguide/getting-started.html

AWS provides a specific tool called the Amazon CodeWhisperer Usage Analyzer. This tool helps organizations analyze the usage of CodeWhisperer by collecting metrics on code suggestions and security scans. Here’s how you can use it:

Deployment: You need to deploy the usage analyzer in your AWS account. This involves cloning the repository, installing dependencies, and bootstrapping the AWS Cloud Developer Kit (CDK).
Metrics Collection: Once deployed, the analyzer will publish metrics into a data pipeline, making the information available for querying via Amazon Athena.
Queries: You can write queries to report on various usage metrics, such as the number of unique users, popular programming languages, and the number of security scans performed.

These queries help you understand how extensively CodeWhisperer is being used within your organization

2b) Manual Review and Insights

Similar to GitHub Copilot, you can manually review the codebase to identify the extent of CodeWhisperer’s contributions. This can be done by:

Reviewing commit messages and comments that indicate CodeWhisperer’s involvement.
Using code review tools to track changes and identify patterns in the code that suggest CodeWhisperer’s assistance.

#3 To see the level of GenAI code use in AI Code Monitor, you can follow these steps:

3a) GitHub Cloud-Based

Step1) Signup using this link: https://app.semasoftware.com/signup/

Step2) Connect your GitHub or AzureDevOps repositories

Step3) Go to Dashboard to view the GenAI usage stats

3b) Other Version Control Systems

If you need any other connection, you may contact us.

2 years ago

AI Code Monitor Beta V3.5: Connect Azure DevOps, Repository Detail page & Code review PR detail page features

Here's the latest video release notes: https://youtu.be/mrdyp1QcvAc

In the current release, we have implemented a few customer-centric features, which are explained below.

1) Users now have the option to connect with Azure DevOps. Previously, they just had the option to connect with GitHub.

Other related How-to-video to help you connect with Azure DevOps is shared below:

https://announcekit.co/semasoftware/release-notes/connect-azure-devops-on-ai-code-monitor-tool-how-to-instructions-1CYfL2

2) As part of simplification of the user journey for the customers, we have removed the repository group information from the Dashboard and given its own placeholder. The user can directly click on the repository group link left-hand-pane to view the GenAI composition for various repository groups and the underlying repositories. In addition to that, if the user wants to view the details of that repository at the chunk & code level, they can click on any repository name, and they will be taken to the Repository detail page.

3) Additional user-friendly features on the Repository detail page. If the user wants to search for a specific folder or file, they can easily search that name. Also, they have the option to filter the chunk based on the labels: NotGenAI, NotEvaluated, PureAI, and Blended. This is useful in going to the specific chunk or code they are interested in.

4) All the above features are available on the Code review PR detail page as well, providing users with consistent user journeys.

5) You will also see a separate link on the left-hand-pane called the "Developer Group Beta". Upon clicking, the user will be taken to the Developer group page, where all information related to the developer group and their associated developers will be displayed. This page is going to evolve to provide more actionable insights. So, keep tuned.

6) Under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

Developer attribution: How it is calculated now on AI Code Monitor Tool

Video: https://youtu.be/FC7FweUYO2s

The AI Code Monitor tool is in Beta. Regarding Developer attribution, while the math is correct given a certain AI/ Not AI composition, the formula we picked first is one of several possible ways to think about developers’ code and GenAI.

For context, it’s very important to us that we help our clients blend, i.e. modify GenAI code to avoid security risk, code quality risk, diligence-at-exit risk (several major PE firms and strategists are now scanning for Blended usage).

So, if code was generated at Time1 by Developer1 and was not Blended initially, and then the code was modified at Time2 by Developer2, then we want the system to give credit for the code ultimately being Blended.

The current version of developer attribution works as follows:

Developer percentages refer to the amount of code in a moment in time. We get the whole codebase for a certain commit (the last of each day). Then we analyze chunks to set GenAI labels. For each line of each chunk, we attribute that line to the LAST developer that committed to them.

For example:

Let's say Developer1 pushes a function, which is PureAI. It has 30 lines. Developer1 gets 30 lines PureAI attributed to them.

Next day, Developer2 modifies 20 of those 30 lines. Now that entire chunk is labelled as Blended, but Developer2 has 20 lines Blended, and Developer1 has 10 lines Blended, even though Developer1 didn’t do anything on the next day.