Here's the latest video release notes: https://youtu.be/QSgaE1jlt34.

The big addition is the ability to group repositories into groups:

And then to produce group-specific "traffic light" indicators to show whether a codebase is in line with the organization's standards for GenAI usage.

Video release notes: here.

Developer-Level Metrics: the dashboard now shows how much each developer is using GenAI, and if they are using GenAI, how much is Pure (copy and paste from the GenAI tool) vs. Blended (modified by the developer).

Expanded GBOM: there are now three tabs to the GBOM: 
1- unchanged: GenAI usage at the function level (subset of file)
2- new: GenAI usage at the repository level
3- new: GenAI usage by developer 


Coming soon: VSCode Widget.

To make it easier for developers to get the most out of GenAI tools whilethe company stays in compliance, we're rolling out a VSCode widget where they can track their usage. See the notification in the bottom right.

Video Release notes here: https://www.youtube.com/watch?v=gX6yWlsvQrE

Repo-level view: Now the AI Code Monitor shows Repository- level detail on AI-Generated code composition. These three repos, for example, include code that was made with 77.51%-82.56% GenAI-incorporated code.

This matters because the security, legal and compliance standards will vary by codebase usage. It is a much lower risk to have internal tooling have pure Gen AI, for example, since it will not need copyright protection and will likely be less susceptible to security attacks.


Expanded GBOM (tm): Now the Generative AI Bill of Materials (GBOM (tm)) includes whether the code was AI or human-generated, along with the confidence level, repository and file path, and size. In this example, the first file, lambada_function.py, size 477 Bytes, has a >99% likelihood of being purely human-generated, while the larger Generate_SQL file has a 99% likelihood of including GenAI code.

For example, the EU AI Act, which was approved by Eurpoean Parliament on June 14, 2023, but is not yet final, may in its final form cover companies with Headquarters, Developers, or Customers in the  EU. It will cover companies of all industries. The EU AI Act is a Compliance Standard.

The EU AI Act has three Components. One such component is greater scrutiny for Gen AI uses in certain high-risk systems.

Video release notes: https://youtu.be/-ybsV6BCUz4


Introducing the AI Code Monitor

We've added the AI Code Monitor to the CTO Dashboard to help companies manage the risks and capture the benefits of code created in full or part by Generative AI tools like ChatGPT and GitHub CoPilot.

You can read more about the benefits and risks from Gen AI code here. 

In the sample codebase analyzed, GenAI tools assisted with 25% of the code.

- Per the German Copyright Act, companies won’t get copyright protection on their code unless the Gen AI code is sufficiently blended.  

- Per the State government's LfDI AI Data Protection Discussion Paper from last week, there may be an additional scrutiny of companies that use Gen AI code to evaluate their employees' work product. This likely applies to companies with a custom-trained model.


Other improvements to the CTO Dashboard

The product now supports integration with BitBucket along with GitHub, and integrates data from the two version control systems seamlessly.

Multiple code quality metrics are now available from Codacy.

We're up to five integrations now:

Explainer video: https://youtu.be/PV-p2kLT4I0

Example Use Case- working with third-party development shops.

Here's an example of how users are using the Dashboard to keep third-party development shops in alignment. 

In particular, several organizations did not have the awareness they were looking for on who was working on their code, and how much staff turnover and activity level there is.   

Using the dashboard they can see the individual developers working on the product:

Too, the dashboard shows the trends in development activity (commits to master branch) to observe unusual variation:

At Sema we are passionate believers in looking at the data -- but also passionate about being curious rather than judgmental. We strongly advise the same when it comes to coding metrics, as software development is a craft, not a competition. Asking questions to get the context is essential.

In this particular case, what might look like a decline in productivity was actually an organization's push to release v1 of a new product in August/ September, followed by smaller, less frequent, more complicated refinements. This was expected and desired behavior.  

New Feature 1- Open Source metrics.

We added metrics from the 5th and 6th components (module) of our codebase health framework so that now there is a comprehensive picture.

Module #5 is Open Source risk -- the legal risk coming from using Open Source code with restrictive licenses like Copyleft. You can read more about this risk here. This data comes from Snyk and we'll be adding other Software Composition Analysis tools in the months ahead.

New Feature 2- Code quality metrics.

Module #6 of 6 is Code Quality. Here we added four charts from Codacy, too we'll be adding other code quality tools in the months ahead.

- Number of warnings about code style, security, error proneness, and performance 

- Duplicate code

- Code complexity

- Testing coverage

Here's an example:

You'll notice that we are considering Codacy's security warnings to be part of code quality, not code security.

Why? It is our view that organizations need an advanced security tool such as Snyk (but not just Snyk) to capture true security risk. So code quality tooling that also adds security risk-- Sonar is in this example, too-- is tracked in code quality.

Here and in so many places in the dashboard, we are opinionated about how to think about codebase health. We know you won't always agree-- but we commit to being transparent in our reasoning and approach, and to being sincerely open to feedback. 

New Feature 3- Full Codebase Health score.

As noted above, now that Open Source and Code Quality risk are included, we have representation from all six components of Codebase Health included in the overall Codebase Health Score.

As a result, companies who use GitHub, Codacy, Snyk and iRADAR can get a full Codebase Health score that updates in real time for quarterly reviews, sprint debriefs, even weekly check-ins with the team.

Not using all of those tools? The Codebase Health score doesn't "ding" you for not having a certain tool, it simply excludes those metrics from the numerator and the denominator.

Here's Beta version 1.3 release notes for the CTO Dashboard.

We're feeling frisky and skipped v1.2. ;0

Explainer Video

https://youtu.be/aWD1VB1fNT4

Part 1 of 3 -- More Metrics

We added a metric on developer retention, part of the Team section -- the most important single measure of codebase health. 

And we added a Cyber Security metric, this one a rollup of thirteen risk vectors such as incomplete email setup and breached credentials.

You can use the Executive Summary page standalone for your daily and weekly check-ins and prep for Board meetings, then dive into the detail on Product and Compliance as needed.

Part 3 of 3 -- Adding Another Engineering tool-- Codacy, for code quality and security

You can now connect Codacy through its API.

Happy Monday, all.

Here's what's new in v1.1 of the CTO Dashboard Beta.

Thanks to the Beta users who gave us quite helpful (and candid!!) feedback on v1.0. We have incorporated those improvements and are ready to add a few more users.

If your organization uses GitHub and at least one of Snyk for Code Security or Wati's i-RADAR for Cyber Security, and you'd like to join the beta, let us know.


In these release notes:

1. Repository summarization to make it easier to keep track of many repositories. 

2. New Executive Summary functionality.

3. A preview of the future of the Executive Summary.



You can also watch a 7.5 minute explainer video: link here.

1. Repository summarization to make it easier to keep track of many repositories. 

When organizations have fewer than 10 repos, we've found it's sensible to see all of those repos at once on a chart. Beyond 10, it becomes too much of an "eye chart." 

So now if there are 10 or more repos, you see the process charts showing the total activity, not by repo.

2. New Executive Summary functionality.

We are building this product to make it easier for CTOs to share a contextualized summary of their codebase -- whether it's at standup, at the end of a sprint, with the C-Suite or their Board.

To that end, we're introducing an Executive Summary, which summarizes the key metrics of codebase health and risk into overall scores, including a single 1-100 "Sema Score."

In the first iteration, we are summarizing the results of one metric, specifically a code security metric of high-risk SAST and DAST warnings, relative to codebases of similar sizes and stages.

In our Sandbox codebase example, the number of high-risk security warnings is in the lowest quartile of its comparison group, so the Sema Score gets the maximum 100 points. Trust us it won't always be 100, and shouldn't be -- scores of 30-70 reflect reasonable investments in non-functional requirements.

3. A preview of the future of the Executive Summary.

This is our second product, we've been working on collecting Executive-ready Engineering metrics for the last six years.

With our first product, we can capture information about Code Quality, Development Process, Development Team, Open Source Legal Risk, Code Security, and Cyber Security at once-- using our code scanning tools. WIth the CTO Dashboard, you'll be able to use your own existing tools to see the same kind of executive results.

Here's a view of a report from our first product so you can see what's coming for the Executive Summary.

All,

I’m very pleased to share that the CTO Dashboard is now available in Closed Beta = by invitation only.

If your organization uses GitHub AND (Snyk for Code Security OR WATI’s i-RADAR for Cyber Security), and you would like to participate in the Closed Beta, please Reply.

Details below.

Matt

The CTO Dashboard is designed to support CTOs and the other most senior leaders of technology teams in two ways:

1.    Delight the C-Suite and Board of Directors: make it easy to provide reports to the CEO, the Leadership Team, and the Board about the state, trends, risks and investment needs for the codebase.

2.    Coach Engineering Managers: help focus CTOs’ direct reports on the highest importance areas of the codebase to maintain, invest, or improve.

Key features of the current version of the Beta include: 

1.    Assessment of the codebase across: 

  a.    the Development Process including developer activity trends,

  b.    Code Security including SAST and CVE warnings, and

  c.    Cyber Security assessment of the organization’s domains and subdomains.

2.    Benchmarking of your results against a database of billions of lines of code and companies worth over $1 Trillion

3.    Cost and timing recommendations—what to fix, when, and how much it will cost

4.    Set up in minutes by connecting native API integrations (more to come growing)

5.    Two permission types: an Owner who can see the Insights and set up the API connections, and Settings Editors who can only handle settings.

We’d love to get your input shaping the next generation of CTO tooling.