a year ago

AI Code Monitor Beta V3.14: ROI of Cost of AI Tools - at Repo groups, Rolled-up ROI number at Organization level

Here's the latest video release notes -- https://youtu.be/8ojwiXXV68E

In the current release, we have implemented ROI of Cost of AI Tools - at Repo groups, Rolled-up ROI number at Organization level

1) V1.3 ROI Calculation: ROI of Cost of AI Tools

Measuring ROI of Cost of AI tools happens at the repository group / product level and at Organization level. The Technical lead or CTO of an organization can look at the 'Total Impact' to view this information along with the in the saved time (hours) and cost ($) that has happened as a result of GenAI usage in the various repository groups.

Depending upon the Cost of AI tools per developer, number of developers in each Repository group, and the Cost saved, the Technical leads can see the impact in terms of ROI of cost of AI tools. To view this calculation, you will have to set 'Per Developer Monthly cost of GenAI tools' in the Settings page.

Capture the field in Settings >Organization level:

Display ROI of Cost of GenAI Tools at Repo Group1:

Display ROI of Cost of GenAI Tools at Repo Group2:

Display ROI of Cost of GenAI Tools at Repo Group3:

Display ROI of Cost of GenAI Tools at Organization level:

2 years ago

AI Code Monitor Beta V3.13: ROI of AI - Time and Cost saved at Repo groups, Rolled-up ROI numbers at Organization level

Here's the latest video release notes: https://youtu.be/Tzj1_9ZX41E

In the current release, we have implemented a few customer-centric features, which are explained below:

1) V1.1 ROI Calculation: Saved Time & Cost

Measuring ROI of AI happens at the repository group / product level. The Technical lead or CTO of an organization can look at the 'Total Impact' in the saved time (hours) and cost ($) that has happened as a result of GenAI usage in the various repository groups. Depending upon the productivity achievement and number of developers in each Repository group, the Technical leads can see the impact in terms of cost & time saved. To view this calculation, you will have to set 'Number of Developers' and the 'Average developer yearly cost ($)' in the Settings page. At the moment, we are assuming 1500 working hours as the number of yearly hours worked by a Developer.

Demonstration: In this example, for sample product "Front End Code": The Productivity Achievement is 12%

The results are:

1. Time saved annually are 3600 hours

2. Cost saved annually are $12,000

2) V1.2 ROI of AI (Rolled-up stats on Home page)

The CTO and other senior executives can view the rolled-up number for the 3 stats:
- Productivity Achievement%
- Potential Productivity Captured%
- Total Impact (Time and Cost savings)
This displays the weighted average of the Productivity achievement% and weighted average of Potential Productivity Captured% across all repository groups of this organization, while the Total Impact is total of the Hours saved and Cost saved across all repository groups of this organization

Demonstration: In this example, for sample product "Front End Code" and "Testing repos group2": The Productivity Achievement% are available for the different Repo Groups ; the Potential Productivity Captured% are available for the different Repo Groups, and the Time saved & Cost saved are available for the different Repo Groups

The results are:

1) Organization-level Productivity Achievement% is 9%

2) Organization-level Potential Productivity Captured% is 67%

3) Organization-level Total Impact (Time and Cost savings) are 5117 hours and $17,058

2 years ago

AI Code Monitor Beta V3.12: Dashboard banner, SignUp journey messages, Marketing consent, CTO Dashboard pages in AICM Tool

Here's the latest video release notes: https://youtu.be/1ZBC0RCYPYE

In the current release, we have implemented a few customer-centric features, which are explained below:

1) Display the Dashboard Banner for release notes and downtime. This allows the backend team to share dynamic information and messages to the user. This is time-specific and event-specific i.e. it will disappear after a certain time as it was posted to draw attention for a specific event.

A screenshot of a computer
Description automatically generated

2) Display System messages when the user has just SignedUp, providing with a self-serve journey by displaying messages When the Scan has ‘Just’ started and NO repository scan has been completed; When the Scan is in progress and full repository scan has NOT been completed; and When the Scan has finished.

3) Marketing Express Consent is available on SignUp page. When the user is on the SignUp page, we explicitly take their consent for receiving any marketing related content from AICM Too. Those users who didn't tick this box, will not receive any marketing emails.

4) Display the CTO Dashboard pages under the AICM Tool Left-hand-Panel including Executive Summary, Product Detail and Compliance Detail. These pages will now display the information on Code Quality, Process, Team, Code Security, Cyber Security and Open Source.

5) As always, under the hood we made improvements in accuracy, including more varied human data in training set.

2 years ago

AI Code Monitor Beta V3.11: Date range affects GenAI stats, View Attestation progress, Organization-wide rules, Inactive user message

Here's the latest video release notes: https://youtu.be/TbpAiftFl_M

In the current release, we have implemented a few customer-centric features, which are explained below:

1) The date range affects the GenAI stats: When I am on the Repository Group page, and I change the date range, not only the charts but also the GenAI usage stats change based on the date range. This allows you to monitor, track & compare the GenAI stats for different time periods.

2) The Repository detail page and PR page now have a feature to view the Attestation progress. As a user, you have the option to view the attestation progress bar that tells you the progress of how many chunks you have attested and how many are left. Apart from that, you also have the option to do bulk attestation. As a user, if you Agree with the labels on the chunks, then you have an option click on “Agree to all” button.

3) On the SignUp page, we wanted to understand better what are the User needs of signing up for the AICM tool, so that we could serve them better. We have provided a multichoice selection of "Why is it important for you to measure GenAI in your code?" which will help us capture the user need.

4) To provide clarity of Organization-wide rules set at Repository group level, we are displaying this information explicitly on the Settings page. This is to make it clear that even if there are no Repository-level rules set, even then the Traffic lights might appear on the Repo groups because of Organization-wide rules.

5) Display message to Inactive users on the Login page to help them navigate to next steps

6) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

AI Code Monitor Beta V3.10: ROI of AI

Here's the latest video release notes: https://youtu.be/QNRl-pP0sYA

In the current release, we have implemented the ROI of AI functionality, which is explained below:

The basic point here is to enable Engineering leadership teams to show how much more productive the team is as a result of using Generative AI tools, how much more effective they are currently and what percentage of the potential have they gained?

Reason one for this functionality is to help, guide and provide teaching and coaching opportunities for the Engineering teams.

Reason two is to give updates to the rest of the organization, who, of course, are very keen to understand how Gen AI coding is affecting the Engineering team

How it works: Measuring ROI of AI happens at the repository group / product level. That's because there's so much nuance for each team, it doesn't really make sense to set measures for the Engineering organization as a whole. There's two parts to how the ROI of AI works.

> One, there are settings that Engineering leaders can tune based on the specifics of their situation.

> And then second, the AI Code Monitor detects how much Gen AI there is and comes up with a conclusion based on the settings from the Engineering leader.

On part one, Engineering leaders select three things:
1. How much time are Engineers spending in coding. If they're only spending 30% of their time coding, even no matter how good the AI tool is it's there's only going to be a 30% increase in potential impact.
2. What is the primary Engineering focus of that product, such as starting a new code base, or within an existing code bas -, adding features and functionality, or tech debt reduction. There are five choices to select from. Sema has default expected productivity improvement, based on research, which is available to review via a link to a blog post within the product. Then, Engineering leaders can override the defaults based on the specifics of their situation.
3. Decide how much Gen AI is appropriate, given the unique circumstances of that team and what they're working on. If the team uses Gen AI to that maximum desired level, then they are getting the full benefit of whatever the potential productivity gain is.

In this example, for sample product "Crow-Proof of Concept":
1. The Engineering team spends 30% of their time coding
2. GenAI could generate 30% productivity improvement
3. The Engineering team reaches its maximum benefit of GenAI when 30% of the code originates with GenAI.

The results are:

1. The team is now 3% more productive as a result of Gen AI usage
2. They have captured 36% of the potential from using Gen AI.

2 years ago

AI Code Monitor Beta V3.9: Historical, Cumulative & Usage in time period charts; Dynamic scaling of charts; Sort & Export Developers

Here's the latest video release notes: https://youtu.be/jfgKHWRKHss

In the current release, we have implemented a few customer-centric features, which are explained below:

1) Historical chart display on Dashboard: The user can go a month back from the day of setting up on AI Code Monitor tool and view the charts on the Dashboard to view the GenAI usage% - Overall, Pure & Blended

2) Display 6 charts on Dashboard: The user can now view 3 charts for Cumulative Usage, displaying Cumulative line chart for Overall, Pure & Blended AND view 3 charts for Usage in time period, displaying Cumulative line chart for Overall, Pure & Blended

3) Dynamic scale to change the y-axis on the charts on Dashboard, allowing the user to change the scale and compare the stats on the charts

4) Developer Group Page-Export to csv: Ability to Export all the data on this page on CSV for further analysis. This enables you to build a sprint review report as well, if needed.

5) Developer Group Page-Sort option: At Developer group level, user has an option to Sort by Name, GenAI, NotGenAI, PureAI, Blended. With this option, user can view the names of Developers based on these sorting parameters (Ascending & Descending). If the user wants to know which Developers have highest GenAI% usage or which Developers have highest Pure GenAI% usage, then they can sort and see the list of developers.

6) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

AI Code Monitor Beta V3.8: Status check, Date picker, Pagination, Link to merge developers

Here's the latest video release notes: https://youtu.be/OS0ltMvfpe8

In the current release, we have implemented a few customer-centric features, which are explained below:

1) Status check in Settings area: When enabled, status checks of pull requests that don’t meet the rules will be marked as “failed” in GitHub. You can keep this disabled as well.

2) Date picker at Repository group: You can now select the date range to view the charts displaying the Overall, Pure & Blended stats based on the date range selected. So, in case you are doing your Sprint retrospective, you could select the last 2 weeks and evaluate the GenAI usage at your Repository group levels.

3) Date picker at Dashboard: You can select the date range and view the GenAI charts for Overall, Pure & Blended based on the date range selected. This shows the stats at the Organization level. User can select:

Last 7 days
Last 14 days (This is default)
Last 30 days
Last 90 days
Last 365 days
All time

4) Increase pagination from 20 to 50: This will help a Dev Manager or tech lead view more developer names under each Developer group

5) Add a link to Merge Developers page: This will help a Dev manager or Tech lead easily navigate to merging the duplicate developer names

6) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

AICM Tool as of July 2024: Quick recap of Product Features & Benefits

Here's the latest video of the Long version Recap of Product Features & Benefits of AICM tool: https://youtu.be/xfmg2RKcE6w (3mins 26 sec)

Here's the latest video of the Short version Recap of Product Features & Benefits of AICM tool: https://youtu.be/OC42J6t3bPM (1min 50 sec)

Here's the latest video of the Shortest 30-second Quick Overview of AICM tool: https://youtu.be/RnjUfE4HaqI (34 sec)

Sign Up > Connect with GitHub or AZDO
View Dashboard > Overall, Pure, Blended
View Dashboard > Rules, Traffic Light
View Dashboard > Chart
Repositories > Group
Repositories > NotGenAI, GenAI (Pure, Blended)
Repositories > Rules, Traffic Light, Chart
Repositories > % Code attested
Repositories > Individual repositories
Repositories > Repository Detail Page
Repository Detail Page > Folder, Files, Lines of Code, Chunk Labels
Repository Detail Page > Search, Filter
Repository Detail Page > Author
Repository Detail Page > Attestation, ToolTip
PR Scan Review Page > GitHub PR page, GenAI Detector, More Details
PR Scan Review Page > GenAI Composition for PR, Folder, Files, Lines of Code, Chunk Labels
PR Scan Review Page > Search, Filter
PR Scan Review Page > Author
PR Scan Review Page > Attestation, Tool Tip
Settings > Members
Settings > Rules
Settings > Repository Groups

# Quick Overview of the AICM SaaS Tool:

1) Sign-Up and Set Rules

2) View GenAI usage stats at Organization level, PR Code Review level and Repository level

3) Attest the code by Accepting or Overriding the label at PR Code review level or on the Repository level, thus making a permanent record of GenAI code labels in the codebase

2 years ago

AI Code Monitor Beta V3.7: Chunk count, Expanded File tree, Charts at Repo level, Attestation% at Repo level & Repo group level

Here's the latest video release notes: https://youtu.be/OUwCdLwdXVs

In the current release, we have implemented a few customer-centric features, which are explained below.

We have done some simple improvements to enhance the user experience on the AICM tool, which include:

1) Favicon: With the implementation of the Sema favicon, the user can easily identify the Sema browser amongst all open browsers

2) ToolTip: When the user is attempting to do the Attestation on Repository detail page or the PR Code review detail page, then these helpful tooltips come handy to inform what does the 'Tick' and 'Cross' symbol mean.

3) Improvement on Hyperlink at Repository level: In continuation of providing a good user experience on Repositories page, all the repositories that have Zero analyzed files are now listed at the bottom without any hyperlink, hence indicating the user that no code-level information is available for this repository. The GenAI% and NotGenAI% are displayed as "N/A".

In this release, we have implemented some new features which include:

4) Easily find files that have selected labels on Repo detail Page & PR scan detail page: When the user is on the PR Code review detail page, the file tree is fully expanded by default for the user to easily view which files have the chunks related to this PR. Based on the selected filters (NotEvaluated, PureAI, NotGenAI, Blended), the count of chunks are shown against each file, making it intuitive which file to click to view the related chunks. For e.g. if you are interested to view only the 'PureAI' chunks, then you can select the filter as 'PureAI' and the files that have the 'PureAI' chunks will be listed with a count>0. This makes it efficient for the user to browse across the files and click only on those files. The same feature is available at the Repository detail page, with the only difference being that the whole file tree doesn't show expanded by default. This is done to load the page faster.

5) Display charts for repository group: The Dashboard has the charts to display the GenAI statistics at the organization level. Similarly, the charts are now displayed at the Repository group level. When you expand a Repository group, you will see the stats of that Repository in charts and the associated respotitories will be listed below that.

6) Display the percentage of attested code per group and repository: When the user is at the Repository group level and at Repository level, they can easily find out what percentage of the lines of code have been attested. This is a great identifier for them to know where they need to take action w.r.t. Attestation. The equation to calculate the Attestation% is as follows: numerator = lines of code that is attested. denominator = lines of code that can be attested.

7) As always, under the hood we made improvements in accuracy, including more varied human data in training set

2 years ago

Frequently Asked Questions (FAQs) on AI Code Monitor Tool

What is Pure GenAI? How do you calculate it?
What is Blended GenAI?
What is Overall?
What is NotGenAI%?
What does this mean? "97.11% of code evaluated"
What do you mean by "2% code attested"?
What do you mean by "Not evaluated: 24 files"?
What do you mean by "Python: 25 files"?
What do you mean by "N/A" under GenAI% and NotGenAI%?
What is Repository Group?
What is Attestation?
What is the 'Tick' icon and 'Cross' icon on Attestation?
What is "Status checks" under Settings>Oraganization Information?
What do you mean by "Apply at organization level"?
What is GBOM (GenAI Bill of Materials)?

------------ ------------ ------------ ------------ ------------

1. What is Pure GenAI? How do you calculate it?

Percentage of chunk of code labeled as GenAI, where the whole chunk belongs to single commit. In order to calculate Pure GenAI%, we first detect that code is GenAIOriginated. If yes, we check if there is a second commit. If there is a second commit, we mark it as Blended GenAI. If there is no second commit, we mark it as Pure GenAI.

2. What is Blended GenAI?

Percentage of chunk of code labeled as GenAI, where the whole chunk belongs to multiple commits. In order to calculate Blended GenAI%, we first detect that code is GenAIOriginated. If yes, we check if there is a second commit. If there is a second commit, we mark it as Blended GenAI.

3. What is Overall?

Total percentage of GenAI in the codebase. It is sum of Pure GenAI% and Blended GenAI%

4. What is NotGenAI%?

Total percentage of human-generated code. Example: Code generated by:

Developer at the organization
External developer (such as Open Source code, or Stack Overflow-copied code)
Auto-generated code (such as code provided alongside frameworks)

5. What does this mean? "97.11% of code evaluated"

Ticket CAW-840 is work-in-progress to clarify this definition

6. What do you mean by "2% code attested"?

This % is number of lines that has been attested divided by the number of lines that can be attested.

Not all code is attestable yet. For e.g. 946/47314 = 2%

7. What do you mean by "Not evaluated: 24 files"?

This is the count of files that has not been evaluated. The definition of 'Not Evaluated' is work-in-progress under ticket CAW-840

8. What do you mean by "Python: 25 files"?

This shows that there are 25 files of type 'Python'

9. What do you mean by "N/A" under GenAI% and NotGenAI%?

The % number under GenAI% and NotGenAI% represents the % of code that is GenAI and NotGenAI respectively. When that particular repository has Analysed files = Zero, then no calculation has been performed and hence it displays "N/A" i.e. Not Applicable.

10. What is Repository Group?

A user of AICM tool can create groups of repositories by giving a Repository Group Name and selecting a Usage category. This is useful in grouping repositories that you want to monitor against similar rules and get an average metric at a Repository group level.

11. What is Attestation?

Teams within the organization can create a permanent record of GenAI code composition, we call this “attestation”. They can do this by confirming or overriding the AI / Not AI label. So far, we have implemented Attestation at 2 levels: 1. Repository detail page level 2. Code review level

Refer this link for more information: https://announcekit.co/semasoftware/release-notes/ai-code-monitor-beta-v3.4-attestation-of-genai-code-labels-4yEm9q

12. What is the 'Tick' icon and 'Cross' icon on Attestation?

Tick - Attest by Agreeing with the label - When the user clicks on this icon, they are attesting that the label on that chunk is correct

Cross - Attest by Overriding the label - When the user clicks on this icon, they are attesting that the label on that chunk is incorrect and they override that by selecting another label and have an option to write a comment.

13. What is "Status checks" under Settings>Oraganization Information?

When this option is selected under 'Settings > Organization Information', then the Pull Request page on GitHub/Azure DevOps for that organization will show another check called 'Sema GenAI Detetector'

14. What do you mean by "Apply at organization level" for Rules under Settings?

When this option is selected, it means that this Rule is applicable to the whole Organization, including all repository groups and developer groups.

15. What is GBOM (GenAI Bill of Materials)?

If you are a user of AI Code Monitor tool and you want to get a detailed view of the GenAI usage for the developers in your organization, then you can download the GBOM file. For more information please refer: https://announcekit.co/semasoftware/release-notes/view-developer-usage-in-detail-using-gbom-download-how-to-video-1AJ70I