Here's the latest video of the Long version Recap of Product Features & Benefits of AICM tool: https://youtu.be/xfmg2RKcE6w (3mins 26 sec)

Here's the latest video of the Short version Recap of Product Features & Benefits of AICM tool: https://youtu.be/OC42J6t3bPM (1min 50 sec)

Here's the latest video of the Shortest 30-second Quick Overview of AICM tool: https://youtu.be/RnjUfE4HaqI (34 sec)

• Sign Up > Connect with GitHub or AZDO
• View Dashboard > Overall, Pure, Blended 
  
• View Dashboard > Rules, Traffic Light
• View Dashboard > Chart
  
• Repositories > Group
• Repositories > NotGenAI, GenAI (Pure, Blended)
• Repositories > Rules, Traffic Light, Chart
• Repositories > % Code attested
• Repositories > Individual repositories
• Repositories > Repository Detail Page
• Repository Detail Page > Folder, Files, Lines of Code, Chunk Labels
• Repository Detail Page > Search, Filter
• Repository Detail Page > Author
• Repository Detail Page > Attestation, ToolTip
• PR Scan Review Page > GitHub PR page, GenAI Detector, More Details
  
• PR Scan Review Page > GenAI Composition for PR, Folder, Files, Lines of Code, Chunk Labels
• PR Scan Review Page > Search, Filter
• PR Scan Review Page > Author
• PR Scan Review Page > Attestation, Tool Tip
• Settings > Members
• Settings > Rules
• Settings > Repository Groups

# Quick Overview of the AICM SaaS Tool:

1) Sign-Up and Set Rules

2) View GenAI usage stats at Organization level, PR Code Review level and Repository level

3) Attest the code by Accepting or Overriding the label at PR Code review level or on the Repository level, thus making a permanent record  of GenAI code labels in the codebase

Here's the latest video release notes: https://youtu.be/OUwCdLwdXVs 

In the current release, we have implemented a few customer-centric features, which are explained below.

2) ToolTip: When the user is attempting to do the Attestation on Repository detail page or the PR Code review detail page, then these helpful tooltips come handy to inform what does the 'Tick' and 'Cross' symbol mean.

3) Improvement on Hyperlink at Repository level: In continuation of providing a good user experience on Repositories page, all the repositories that have Zero analyzed files are now listed at the bottom without any hyperlink, hence indicating the user that no code-level information is available for this repository. The GenAI% and NotGenAI% are displayed as "N/A".

In this release, we have implemented some new features which include:

4) Easily find files that have selected labels on Repo detail Page & PR scan detail page: When the user is on the PR Code review detail page, the file tree is fully expanded by default for the user to easily view which files have the chunks related to this PR. Based on the selected filters (NotEvaluated, PureAI, NotGenAI, Blended), the count of chunks are shown against each file, making it intuitive which file to click to view the related chunks. For e.g. if you are interested to view only the 'PureAI' chunks, then you can select the filter as 'PureAI' and the files that have the 'PureAI' chunks will be listed with a count>0. This makes it efficient for the user to browse across the files and click only on those files. The same feature is available at the Repository detail page, with the only difference being that the whole file tree doesn't show expanded by default. This is done to load the page faster.

5) Display charts for repository group: The Dashboard has the charts to display the GenAI statistics at the organization level. Similarly, the charts are now displayed at the Repository group level. When you expand a Repository group, you will see the stats of that Repository in charts and the associated respotitories will be listed below that.

6) Display the percentage of attested code per group and repository: When the user is at the Repository group level and at Repository level, they can easily find out what percentage of the lines of code have been attested. This is a great identifier for them to know where they need to take action w.r.t. Attestation. The equation to calculate the Attestation% is as follows: numerator = lines of code that is attested. denominator = lines of code that can be attested.

7) As always, under the hood we made improvements in accuracy, including more varied human data in training set

1. What is Pure GenAI? How do you calculate it?
2. What is Blended GenAI?
3. What is Overall?
4. What is NotGenAI%?
5. What does this mean? "97.11% of code evaluated"
6. What do you mean by "2% code attested"?
7. What do you mean by "Not evaluated: 24 files"?
8. What do you mean by "Python: 25 files"?
9. What do you mean by "N/A" under GenAI% and NotGenAI%?
10. What is Repository Group?
11. What is Attestation?
12. What is the 'Tick' icon and 'Cross' icon on Attestation?
13. What is "Status checks" under Settings>Oraganization Information?
14. What do you mean by "Apply at organization level"?
15. What is GBOM (GenAI Bill of Materials)?

------------    ------------    ------------    ------------    ------------    

1. What is Pure GenAI? How do you calculate it? 

Percentage of chunk of code labeled as GenAI, where the whole chunk belongs to single commit. In order to calculate Pure GenAI%, we first detect that code is GenAIOriginated. If yes, we check if there is a second commit. If there is a second commit, we mark it as Blended GenAI. If there is no second commit, we mark it as Pure GenAI.

2. What is Blended GenAI? 

Percentage of chunk of code labeled as GenAI, where the whole chunk belongs to multiple commits. In order to calculate Blended GenAI%, we first detect that code is GenAIOriginated. If yes, we check if there is a second commit. If there is a second commit, we mark it as Blended GenAI.

3. What is Overall?

Total percentage of GenAI in the codebase. It is sum of Pure GenAI% and Blended GenAI%

4. What is NotGenAI%?

Total percentage of human-generated code. Example: Code generated by:

• Developer at the organization
• External developer (such as Open Source code, or Stack Overflow-copied code) 
• Auto-generated code (such as code provided alongside frameworks) 

5. What does this mean? "97.11% of code evaluated"

Ticket CAW-840 is work-in-progress to clarify this definition

6. What do you mean by "2% code attested"?

This % is number of lines that has been attested divided by the number of lines that can be attested. 

Not all code is attestable yet. For e.g. 946/47314 = 2%

7. What do you mean by "Not evaluated: 24 files"?

This is the count of files that has not been evaluated. The definition of 'Not Evaluated' is work-in-progress under ticket CAW-840

8. What do you mean by "Python: 25 files"?

This shows that there are 25 files of type 'Python'

9. What do you mean by "N/A" under GenAI% and NotGenAI%?

The % number under GenAI% and NotGenAI% represents the % of code that is GenAI and NotGenAI respectively. When that particular repository has Analysed files = Zero, then no calculation has been performed and hence it displays "N/A" i.e. Not Applicable.

10. What is Repository Group?

A user of AICM tool can create groups of repositories by giving a Repository Group Name and selecting a Usage category. This is useful in grouping repositories that you want to monitor against similar rules and get an average metric at a Repository group level.

11. What is Attestation?

Teams within the organization can create a permanent record of GenAI code composition, we call this “attestation”. They can do this by confirming or overriding the AI / Not AI label. So far, we have implemented Attestation at 2 levels: 1. Repository detail page level  2. Code review level 

Refer this link for more information: https://announcekit.co/semasoftware/release-notes/ai-code-monitor-beta-v3.4-attestation-of-genai-code-labels-4yEm9q 

12. What is the 'Tick' icon and 'Cross' icon on Attestation?

Tick - Attest by Agreeing with the label - When the user clicks on this icon, they are attesting that the label on that chunk is correct

Cross - Attest by Overriding the label - When the user clicks on this icon, they are attesting that the label on that chunk is incorrect and they override that by selecting another label and have an option to write a comment.

When this option is selected under 'Settings > Organization Information', then the Pull Request page on GitHub/Azure DevOps for that organization will show another check called 'Sema GenAI Detetector'

When this option is selected, it means that this Rule is applicable to the whole Organization, including all repository groups and developer groups.

If you are a user of AI Code Monitor tool and you want to get a detailed view of the GenAI usage for the developers in your organization, then you can download the GBOM file. For more information please refer: https://announcekit.co/semasoftware/release-notes/view-developer-usage-in-detail-using-gbom-download-how-to-video-1AJ70I

Here's the latest video release notes: https://youtu.be/q98NJ3xHu7o

In the current release, we have implemented a few customer-centric features, which are explained below.

1) Display developer names next to the lines of code: The senior leaders in the organization like VPs / Engineering Managers can view the names of the developers next to the lines of code on both Repository detail page and on the PR scan detail page. This will help them reach out to the respective developers who committed the code.

2) Display "N/A" stats on repositories and groups that have no files analyzed: When the users are viewing the GenAI usage stats at the repository and repository group level, then they will see 'N/A' in the stats in case there were Zero analyzed files, which is an accurate representation of what has been analyzed and being used for calculation of the stats.

This post includes the steps to view the GenAI code usage in :

1) GitHub Copilot

2) Amazon CodeWhisperer 

3) Sema AI Code Monitor

#1 To see the level of GenAI code use in GitHub Copilot, you can follow these steps:

1a) GitHub Copilot Dashboard

The QuickStart guide can be accessed here: https://docs.github.com/en/copilot/quickstart

GitHub provides a dashboard for Copilot users that includes usage statistics. This dashboard can show you how much of your code is being generated by Copilot. To access it:

• Open Visual Studio Code (VS Code) or any other supported IDE where you have GitHub Copilot installed.
• Click on the GitHub Copilot icon in the sidebar or navigate to the Copilot settings.
• Look for a section labeled "Copilot Statistics" or similar. This section will display metrics such as the percentage of code suggestions accepted, the number of suggestions made, and other relevant usage statistics.

1b) GitHub Copilot Insights

GitHub Copilot Insights is a feature that provides detailed analytics on how Copilot is being used in your projects. This can include:

• The percentage of code generated by Copilot.
• The types of suggestions being made.
• The acceptance rate of Copilot's suggestions.

To access Copilot Insights:

• Go to the GitHub Copilot website or the settings within your IDE.
• Navigate to the "Insights" or "Analytics" section.

#2 To see the level of GenAI code use in Amazon CodeWhisperer, you can utilize several methods and tools provided by AWS:

2a) CodeWhisperer Usage Analyzer

The QuickStart guide can be accessed here: https://docs.aws.amazon.com/codewhisperer/latest/userguide/getting-started.html

AWS provides a specific tool called the Amazon CodeWhisperer Usage Analyzer. This tool helps organizations analyze the usage of CodeWhisperer by collecting metrics on code suggestions and security scans. Here’s how you can use it:

• Deployment: You need to deploy the usage analyzer in your AWS account. This involves cloning the repository, installing dependencies, and bootstrapping the AWS Cloud Developer Kit (CDK).
• Metrics Collection: Once deployed, the analyzer will publish metrics into a data pipeline, making the information available for querying via Amazon Athena.
• Queries: You can write queries to report on various usage metrics, such as the number of unique users, popular programming languages, and the number of security scans performed.

These queries help you understand how extensively CodeWhisperer is being used within your organization

2b) Manual Review and Insights

Similar to GitHub Copilot, you can manually review the codebase to identify the extent of CodeWhisperer’s contributions. This can be done by:

• Reviewing commit messages and comments that indicate CodeWhisperer’s involvement.
• Using code review tools to track changes and identify patterns in the code that suggest CodeWhisperer’s assistance.

#3 To see the level of GenAI code use in AI Code Monitor, you can follow these steps:

3a) GitHub Cloud-Based 

Step1) Signup using this link: https://app.semasoftware.com/signup/

Step3) Go to Dashboard to view the GenAI usage stats

3b) Other Version Control Systems 

If you need any other connection, you may contact us.

Here's the latest video release notes: https://youtu.be/mrdyp1QcvAc

In the current release, we have implemented a few customer-centric features, which are explained below.

1) Users now have the option to connect with Azure DevOps. Previously, they just had the option to connect with GitHub.

Other related How-to-video to help you connect with Azure DevOps is shared below:

https://announcekit.co/semasoftware/release-notes/connect-azure-devops-on-ai-code-monitor-tool-how-to-instructions-1CYfL2

2) As part of simplification of the user journey for the customers, we have removed the repository group information from the Dashboard and given its own placeholder. The user can directly click on the repository group link left-hand-pane to view the GenAI composition for various repository groups and the underlying repositories. In addition to that, if the user wants to view the details of that repository at the chunk & code level, they can click on any repository name, and they will be taken to the Repository detail page.

3) Additional user-friendly features on the Repository detail page. If the user wants to search for a specific folder or file, they can easily search that name. Also, they have the option to filter the chunk based on the labels: NotGenAI, NotEvaluated, PureAI, and Blended. This is useful in going to the specific chunk or code they are interested in.

4) All the above features are available on the Code review PR detail page as well, providing users with consistent user journeys.

6) Under the hood we made improvements in accuracy, including more varied human data in training set

Video: https://youtu.be/FC7FweUYO2s 

The AI Code Monitor tool is in Beta. Regarding Developer attribution, while the math is correct given a certain AI/ Not AI composition, the formula we picked first is one of several possible ways to think about developers’ code and GenAI. 

So, if code was generated at Time1 by Developer1 and was not Blended initially, and then the code was modified at Time2 by Developer2, then we want the system to give credit for the code ultimately being Blended.

The current version of developer attribution works as follows:

Developer percentages refer to the amount of code in a moment in time. We get the whole codebase for a certain commit (the last of each day). Then we analyze chunks to set GenAI labels. For each line of each chunk, we attribute that line to the LAST developer that committed to them.

For example:

Let's say Developer1 pushes a function, which is PureAI. It has 30 lines.  Developer1 gets 30 lines PureAI attributed to them. 

Next day, Developer2 modifies 20 of those 30 lines. Now that entire chunk is labelled as Blended, but Developer2 has 20 lines Blended, and Developer1 has 10 lines Blended, even though Developer1 didn’t do anything on the next day. 

Video: https://youtu.be/PRwIHvcFA0k

If you are a user of AI Code Monitor tool and you want to get a detailed view of the GenAI usage for the developers in your organization, then you can download the GBOM file (See the columns 'earliest_committer' and 'all_committers') from the Dashboard and view the following details:

• repository_group
• repository
• file_path
• start_line
• end_line
• num_lines
• not_genai_lines
• pure_ai_lines
• blended_lines
• commit_sha
• commit_date
• language
• earliest_committer
• all_committers
• ranges__start|end|label|author|commit_sha|commit_date

 *GBOM = GenAI Bill of Materials

Sometimes, the GBOM file could be big in size (depending upon your organization and repositories). In that case, you will see a message (see below) on the Dashboard. After a short while, you will receive an email (see below) with the option to download the GBOM file. Upon clicking the link, the csv file will be downloaded on your system.

Video: https://youtu.be/bwmdfDXNi7c

Development / Engineering team members of your organization can manage the GenAI code composition statistics during the Code review phase of the SDLC. When they have opened a Pull Request in GitHub, the GenAI code composition for that Pull Request is visible to them on the Pull Request details page and then on the AICM Tool's Pull Request details page. They can agree with the GenAI code labels provided by the AICM tool or they can override and re-label it based on their knowledge.

On one hand, this allows them to verify and rectify the GenAI code composition labels for this PR, while on the other hand allows them to approve/reject a Pull Request based on GenAI code usage.

Follow these steps to do 'Attestation':

Step1. Go to the GitHub or AzureDevOps tools to look for Pull Requests

Step2. Go to the 'Checks' page where all otehr checks are listed. Here you will find Sema GenAI Detector

Step3. Click on 'Sema GenAI Detector' link to view the GenAI Composition for this PR

Step4. You can further click on 'More Details' link to go to the PR detail page on AICM tool to view the GenAI Composition for this PR and the code assocaietd with this PR. Once you are on this page, you can take the following actions w.r.t. 'Attestation'.

1. Agree – By clicking the 'Tick' icon, you are Attesting that you agree with the GenAI code labelling.
2. Disagree [add comments (optional)] – By clicking the 'Cross' icon, you are Attesting that you do not agree with the GenAI code labelling, and you select from the given options of Blended, PureGenAI, NotGenAI – that you think is the correct label.
3. Do nothing - This doesn't count as attestation.

After doing Attestation, the chunk will show the name of the person who did the Attestation and the timestamp of when this was done. You also have an option to Edit.

Note: When you attest a chunk in a PR, the PR GenAI stats moves based on the action taken by the most recent user. 

  

Stats before Attestation

Stats after Attestation: 

Attestation actions:

1) Agree with one chunk labelled as NotGenAI

2) Relabeled another NotGenAI chunk as Blended - because of this the stats changed at PR level

Old Stats: Overall = 4.83 %, Pure = 0.76 , Blended = 4.07 %

New Stats: Overall = 5.85 %, Pure = 0.77 , Blended = 5.08 %  (Blended and Overall increased significantly)

Here's the latest video release notes: https://youtu.be/4Wo9_Xko1k0

In the current release, we have implemented Attestation at 2 levels: 1. Repository detail page level  2. Code review level 

Teams can create a permanent record of GenAI code composition, we call this “attestation”, by confirming or overriding the AI / Not AI label.

1. Attestation at Repository detail page level: As a user of the AICM tool, when you login, you can go to the Repository detail page and view the GenAI code stats. You can manage the GenAI code composition labels displayed on the Repository detail page. You can agree with the GenAI Code labels provided by the AICM tool or you can override and re-label it based on your knowledge. This allows you to verify and (if needed) rectify the GenAI code composition labels for your organization.

2. Attestation at Code Review level: Development / Engineering team members of your organization can manage the GenAI code composition statistics during the Code review phase of the SDLC. When they have opened a Pull Request in GitHub, the GenAI code composition for that Pull Request is visible to them on the Pull Request details page and then on the AICM Tool's Pull Request details page. They can agree with the GenAI Code labels provided by the AICM tool or they can override and re-label it based on their knowledge. This allows them to verify and rectify the GenAI code composition labels for this PR, much before the code has been committed to production, hence, providing more flexibility in managing the GenAI code labels.

You can take the following actions w.r.t. 'Attestation'.

1. Agree – By clicking the 'Tick' icon, you are Attesting that you agree with the GenAI code labelling.
2. Disagree [add comments (optional)] – By clicking the 'Cross' icon, you are Attesting that you do not agree with the GenAI code labelling, and you select from the given options of Blended, PureGenAI, NotGenAI – that you think is the correct label.
3. Do nothing - This doesn't count as attestation.

Note: When you attest a chunk in a PR, the PR GenAI stats moves based on the action taken by the most recent user.