Health-checks are peculiar things

Healthchecks is a monitoring technique with a special place flavor: healthchecks are fired off at regular, frequent intervals (sometimes every 10 seconds, sometimes 1 minute) by orchestration platforms and monitoring tools. Most healthchecks are HTTP-based, and the returned HTTP response is checks based on the status code and (sometimes) content. But really, the only healthchecks a person needs to know about, are those that fail, which usually lead to containers being torn down and other disruptive infrastructure changes.

Issues with health-checks in Lumigo

Given that Lumigo's pricing model is based on the amount of requests we process, the large amount of successful healthchecks that every container workload undergoes leads to undesirable consumption of quota, for data that is effectively not useful. Moreover, successful healthchecks lead to noise in the Explore and Transactions view, degrading the overall experience.

Cutting the Gordian knot

Luckily, one can often spot recognize HTTP requests that are healthchecks pretty easily! Both AWS ELB health-checks, as well as Kubernetes ones (including EKS), come in with specific User-Agent headers. Lumigo now automatically drops in the data processing pipeline in the Lumigo platform all the spans that:

1. Carry the User-Agent HTTP header with values that are known to be health-checks, specifically ELB-HealthChecker/* (AWS ELB, often used with Amazon ECS) and kube-probe/<kubelet_version> (Kubernetes, including Amazon EKS)
2. Return an HTTP status code that denotes a successful response (a.k.a.: `2xx` like `200 OK`, `201 Accepted`, etc.). This is because if a Health-check fails (e.g. returning HTTP status code 500), usually something bad is about to happen to your containers :-)

What do you need to do on your end?

Nothing. It just works with every version of tracers we released so far for containers and all HTTP OpenTelemetry instrumentations we have ever seen. Enjoy :-)

P.S.: Matching health-checks by path (e.g., /health) sounds like a good solution on paper, but in practice it leads to very annoying false-positives (i.e., HTTP calls that are NOT related with health-checks). Moreover, healthcheck paths are configurable, and practitioners do make use for that configurability, which would lead to false negatives (health checks we let through). User-Agent headers, on the other hand, are far less often changed by healthcheck systems. User-agent matching, on the other hand, is usually rather reliable for this use-case.

Since we launched Amazon ECS support earlier this summer, we have come across many user workloads that behave like batch jobs (which, incidentally, we see often scheduled via AWS Batch and, occasionally, via AWS Step Functions). Rather than relying on long-running processes that receive request over HTTP, these workloads execute jobs pulled from the Amazon SQS or sometimes the process environment, perform computation involving databases, other services and messaging queues, and then terminate.

The most intuitive representation for such transactions consist of a "root" span representing the "main" method, with the outgoing requests to databases, messaging queues and other service nested directly under the "main" span. And this is how Lumigo will now represent these workloads, provided that you use the OpenTelemetry API to create the "root" span.

Lumigo now also supports the case where the distributed trace starts with an outgoing request, but given there is no common parent span, multiple such ongoing requests will each result in a separate transaction.

Enjoy this improved support for your containerized workloads and let us know what you think about it!

P.S. If you want a hand using the OpenTelemetry API to create root spans, we are happy to help! Let us know through the support channel, and we'll gladly arrange a call to help you out. It's usually just 5 minutes coding, and then pushing it to your environment to validate :-)

P.P.S.S. Lumigo now also shows Elastic Load Balancers that serve HTTP requests issued by containerized workflows.

We extended Lumigo's support of OpenTelemetry resource attribute semantic conventions to cover the `cloud.platform` key and the Amazon ECS semantic conventions.

Our goal with implementing the OpenTelemetry semantic conventions is to enable you to get the best integration with Lumigo for your workloads even when not using the Lumigo OpenTelemetry distributions, but rather using upstream OpenTelemetry SDKs.

The Amazon ECS semantic conventions are not yet widely implemented in OpenTelemetry SDKs, but we are very much looking forward to contributing them to the community!

For more information about the OpenTelemetry semantic conventions supported by Lumigo, refer to the OpenTelemetry Supported Semantic Conventions documentation.

A new UI brings a best-in-class visual experience to your System Map, all with the same features you love, including filtering, to give you a detailed bird's eye view of your whole stack.

Search for pages, functions, and product documentation — all with keyboard shortcut to maximize your efficiency.

Hit Ctrl + K for Windows and Cmd + K for macOS to search for pages, functions, and saved documents.

See the line of code where the exception was thrown on the STACK TRACE section. This important information can help you figure out where things went wrong or how various subroutines work together during execution.

When will I see the stack trace? At the moment the stack trace is available for traced Python functions.

We’ve just released a new user interface designed to improve the overall user experience including new time picker, and better support for Safari and Firefox. 

We’ve revamped Slack notifications to ensure teams quickly get the information they need when issues occur. You can now see the details of the issue at a glance, and investigate the issue in lumigo with one click. 

For more information read the documentation here.

We've added a Function Duration widget to the dashboard to help you easily identify slow functions. 

Check it out:

To help you resolve issues faster we've updated the Issues & Alert page.

• Perform actions on a bulk of issues.
• Customize the issues table to match your needs.
• View execution tags distribution.

For more information read the documentation here.