Here's the latest video release notes: https://youtu.be/PmKHKrjdKQ4

In the current release, we have implemented the following items.

1. Self-serve for GitHub users of AICM tool

1.1) Any prospective user with GitHub can start a free trial of the AICM tool on their own (without any external help). They can click on 'Start Free Trial' button on the marketing site - https://www.semasoftware.com/ai-code-monitor. 

1.2)Upon clicking on 'Start Free Trial', they will be taken to a marketing page explaining in detail about the AICM free trial (https://www.semasoftware.com/ai-code-monitor-free-trial). 

1.3) Upon clicking the 'Start Free Trial' on this page, the user is taken to the Sign Up page.

1.4) Upon successful Sign Up, user is prompted to enter further details like Organization name & Industry (Optional).

1.5) In the next step, the user is taken to the AICM tool Dashboard asking them to connect to GitHub

-----------------------------------------------------------------------------------------------------------------------

2. Preset rules by Sema

These are the predefined rules created by sema in the backend. Any new organization will have these rules available to them under Settings > Rules area. The user can apply these rules at Organization level and/or Repository group level and/or Developer group level. The user can also edit these rules to adapt it to their situation and needs. On one hand this allows the user to understand and adapt the rules to suit their needs and on the other hand it is a time-saving feature to utilize predefined rules.

3.  NotEvaluated calculation on Dashboard

The message blurb at the top of the Dashboard clearly mentions the % of code that has been evaluated

The VS Code Widget for the AI Code Monitor is now available for free download.

Demo video: https://youtu.be/KQuEyhWx2i4

It is currently available for Python only.

The VS Code Widget helps developers understand how much GenAI code is in their codebase, and what type of GenAI code.

Copy-pasted GenAI code, or "Pure" GenAI, can be riskier than GenAI code that's been modified by developers, "Blended" GenAI.

And knowing the amount of GenAI in the code helps developers stay in line with team and company best practices for GenAI usage.
 

Here's the latest video release notes: https://youtu.be/nqVNAAefqTg

The AI Code Monitor helps companies answer three questions:

1. GenAI Code Composition.How much GenAI code do we have, and is it in line with our organizational goals and requirements? 

2. Are developers using GenAI to its full potential?

3. What risks from AI usage should we aware of?

In the current release:

1. We extend our rules engine to apply to Organization-level GenAI standards. In this case the codebase as a whole has less GenAI than desired (target 5%, actual 4.36%).

2. We enabled groups of developers and allowed rules to be extended to developer groups. Here, in house junior developers have a different, lower threshold for GenAI usage, than Senior developers, because they are still learning coding practices.

Here's the latest video release notes: https://youtu.be/QSgaE1jlt34.

The big addition is the ability to group repositories into groups:

And then to produce group-specific "traffic light" indicators to show whether a codebase is in line with the organization's standards for GenAI usage.

Video release notes: here.

Developer-Level Metrics: the dashboard now shows how much each developer is using GenAI, and if they are using GenAI, how much is Pure (copy and paste from the GenAI tool) vs. Blended (modified by the developer).

Expanded GBOM: there are now three tabs to the GBOM: 
1- unchanged: GenAI usage at the function level (subset of file)
2- new: GenAI usage at the repository level
3- new: GenAI usage by developer 


Coming soon: VSCode Widget.

To make it easier for developers to get the most out of GenAI tools whilethe company stays in compliance, we're rolling out a VSCode widget where they can track their usage. See the notification in the bottom right.

Video Release notes here: https://www.youtube.com/watch?v=gX6yWlsvQrE

Repo-level view: Now the AI Code Monitor shows Repository- level detail on AI-Generated code composition. These three repos, for example, include code that was made with 77.51%-82.56% GenAI-incorporated code.

This matters because the security, legal and compliance standards will vary by codebase usage. It is a much lower risk to have internal tooling have pure Gen AI, for example, since it will not need copyright protection and will likely be less susceptible to security attacks.


Expanded GBOM (tm): Now the Generative AI Bill of Materials (GBOM (tm)) includes whether the code was AI or human-generated, along with the confidence level, repository and file path, and size. In this example, the first file, lambada_function.py, size 477 Bytes, has a >99% likelihood of being purely human-generated, while the larger Generate_SQL file has a 99% likelihood of including GenAI code.

For example, the EU AI Act, which was approved by Eurpoean Parliament on June 14, 2023, but is not yet final, may in its final form cover companies with Headquarters, Developers, or Customers in the  EU. It will cover companies of all industries. The EU AI Act is a Compliance Standard.

The EU AI Act has three Components. One such component is greater scrutiny for Gen AI uses in certain high-risk systems.

Video release notes: https://youtu.be/-ybsV6BCUz4


Introducing the AI Code Monitor

We've added the AI Code Monitor to the CTO Dashboard to help companies manage the risks and capture the benefits of code created in full or part by Generative AI tools like ChatGPT and GitHub CoPilot.

You can read more about the benefits and risks from Gen AI code here. 

In the sample codebase analyzed, GenAI tools assisted with 25% of the code.

- Per the German Copyright Act, companies won’t get copyright protection on their code unless the Gen AI code is sufficiently blended.  

- Per the State government's LfDI AI Data Protection Discussion Paper from last week, there may be an additional scrutiny of companies that use Gen AI code to evaluate their employees' work product. This likely applies to companies with a custom-trained model.


Other improvements to the CTO Dashboard

The product now supports integration with BitBucket along with GitHub, and integrates data from the two version control systems seamlessly.

Multiple code quality metrics are now available from Codacy.

We're up to five integrations now:

Explainer video: https://youtu.be/PV-p2kLT4I0

Example Use Case- working with third-party development shops.

Here's an example of how users are using the Dashboard to keep third-party development shops in alignment. 

In particular, several organizations did not have the awareness they were looking for on who was working on their code, and how much staff turnover and activity level there is.   

Using the dashboard they can see the individual developers working on the product:

Too, the dashboard shows the trends in development activity (commits to master branch) to observe unusual variation:

At Sema we are passionate believers in looking at the data -- but also passionate about being curious rather than judgmental. We strongly advise the same when it comes to coding metrics, as software development is a craft, not a competition. Asking questions to get the context is essential.

In this particular case, what might look like a decline in productivity was actually an organization's push to release v1 of a new product in August/ September, followed by smaller, less frequent, more complicated refinements. This was expected and desired behavior.  

New Feature 1- Open Source metrics.

We added metrics from the 5th and 6th components (module) of our codebase health framework so that now there is a comprehensive picture.

Module #5 is Open Source risk -- the legal risk coming from using Open Source code with restrictive licenses like Copyleft. You can read more about this risk here. This data comes from Snyk and we'll be adding other Software Composition Analysis tools in the months ahead.

New Feature 2- Code quality metrics.

Module #6 of 6 is Code Quality. Here we added four charts from Codacy, too we'll be adding other code quality tools in the months ahead.

- Number of warnings about code style, security, error proneness, and performance 

- Duplicate code

- Code complexity

- Testing coverage

Here's an example:

You'll notice that we are considering Codacy's security warnings to be part of code quality, not code security.

Why? It is our view that organizations need an advanced security tool such as Snyk (but not just Snyk) to capture true security risk. So code quality tooling that also adds security risk-- Sonar is in this example, too-- is tracked in code quality.

Here and in so many places in the dashboard, we are opinionated about how to think about codebase health. We know you won't always agree-- but we commit to being transparent in our reasoning and approach, and to being sincerely open to feedback. 

New Feature 3- Full Codebase Health score.

As noted above, now that Open Source and Code Quality risk are included, we have representation from all six components of Codebase Health included in the overall Codebase Health Score.

As a result, companies who use GitHub, Codacy, Snyk and iRADAR can get a full Codebase Health score that updates in real time for quarterly reviews, sprint debriefs, even weekly check-ins with the team.

Not using all of those tools? The Codebase Health score doesn't "ding" you for not having a certain tool, it simply excludes those metrics from the numerator and the denominator.

Here's Beta version 1.3 release notes for the CTO Dashboard.

We're feeling frisky and skipped v1.2. ;0

Explainer Video

https://youtu.be/aWD1VB1fNT4

Part 1 of 3 -- More Metrics

We added a metric on developer retention, part of the Team section -- the most important single measure of codebase health. 

And we added a Cyber Security metric, this one a rollup of thirteen risk vectors such as incomplete email setup and breached credentials.

You can use the Executive Summary page standalone for your daily and weekly check-ins and prep for Board meetings, then dive into the detail on Product and Compliance as needed.

Part 3 of 3 -- Adding Another Engineering tool-- Codacy, for code quality and security

You can now connect Codacy through its API.

Happy Monday, all.

Here's what's new in v1.1 of the CTO Dashboard Beta.

Thanks to the Beta users who gave us quite helpful (and candid!!) feedback on v1.0. We have incorporated those improvements and are ready to add a few more users.

If your organization uses GitHub and at least one of Snyk for Code Security or Wati's i-RADAR for Cyber Security, and you'd like to join the beta, let us know.


In these release notes:

1. Repository summarization to make it easier to keep track of many repositories. 

2. New Executive Summary functionality.

3. A preview of the future of the Executive Summary.



You can also watch a 7.5 minute explainer video: link here.

1. Repository summarization to make it easier to keep track of many repositories. 

When organizations have fewer than 10 repos, we've found it's sensible to see all of those repos at once on a chart. Beyond 10, it becomes too much of an "eye chart." 

So now if there are 10 or more repos, you see the process charts showing the total activity, not by repo.

2. New Executive Summary functionality.

We are building this product to make it easier for CTOs to share a contextualized summary of their codebase -- whether it's at standup, at the end of a sprint, with the C-Suite or their Board.

To that end, we're introducing an Executive Summary, which summarizes the key metrics of codebase health and risk into overall scores, including a single 1-100 "Sema Score."

In the first iteration, we are summarizing the results of one metric, specifically a code security metric of high-risk SAST and DAST warnings, relative to codebases of similar sizes and stages.

In our Sandbox codebase example, the number of high-risk security warnings is in the lowest quartile of its comparison group, so the Sema Score gets the maximum 100 points. Trust us it won't always be 100, and shouldn't be -- scores of 30-70 reflect reasonable investments in non-functional requirements.

3. A preview of the future of the Executive Summary.

This is our second product, we've been working on collecting Executive-ready Engineering metrics for the last six years.

With our first product, we can capture information about Code Quality, Development Process, Development Team, Open Source Legal Risk, Code Security, and Cyber Security at once-- using our code scanning tools. WIth the CTO Dashboard, you'll be able to use your own existing tools to see the same kind of executive results.

Here's a view of a report from our first product so you can see what's coming for the Executive Summary.