---
title: "eMudhra CA Key Compromise: Response Guide for Developers"
description: "Anyone can compute eight eMudhra intermediate CA private keys from public certificates. Browsers block the compromised intermediates, but CI runners, containers, JVMs and Go binaries can still trust forged certificates. Test and update affected systems, and decide whether to rotate credentials."
url: "https://safedep.io/blog/emudhra-ca-key-compromise"
author: "SafeDep Team"
published: "2026-10-22"
---

# eMudhra CA Key Compromise: Response Guide for Developers

Anyone can compute the private keys of eight eMudhra intermediate certificate authorities (CAs)
from their public certificates. These keys let attackers forge TLS certificates for any domain.
Current browsers block the compromised intermediates, but many other TLS clients still trust them.
Use this guide to identify affected systems, test and update them, and decide whether to rotate
credentials.

## Summary

- **What was wrong.** eMudhra created eight intermediate CAs in April 2025 with weak private keys.
  Each private key is a number below 256, so anyone can compute it in less than a second. Six of the
  intermediates issue TLS certificates. Two issue code signing certificates.
- **Who is affected.** A TLS client is affected if it trusts the eMudhra or Sectigo root that
  signed the intermediates, does not block the intermediates, and does not check revocation. Many
  CI runners, container images, JVMs, Go binaries, Python, Rust and PHP CA bundle packages,
  Electron apps and macOS built-in tools have this problem.
- **Server certificates.** Firefox and clients with an updated trust store already reject
  `emSign ECC Root CA - G3`. Chrome will reject it on 27 October 2026.
- **Attacks.** Forged certificates for `google.com` and other domains are in public
  [Certificate Transparency (CT)](https://certificate.transparency.dev/) logs since 19 October
  ([`ctlogs.dev`][ctlogs]). As of 21 October, nobody has confirmed that an attacker used one
  against a client ([The Hacker News][thn]).
- **Attack tools.** mitmproxy 12.3.0, released on 22 October, can sign intercepted traffic with
  a recovered eMudhra key ([mitmproxy commit][mitm]). An attacker who can intercept the traffic of
  an affected client needs no custom tools.

## Respond now

Do these first:

- [ ] Check each server certificate that you operate. Replace each one that chains to
      `emSign ECC Root CA - G3` now. See
      [Check your own certificates](#5-check-your-own-certificates).
- [ ] Run the probes in one CI job for each runner type: hosted, self-hosted and container jobs.
      Also test each TLS inspection proxy, because every client behind it depends on it. See
      [Run a probe](#1-run-a-probe-where-the-client-runs). To test a real application, see
      [Test the real application with mitmproxy](#test-the-real-application-with-mitmproxy).
- [ ] List the images that run in production. Scan each image, rebuild it with the updated OS trust
      store, and redeploy it. See [Scan images](#2-scan-images-and-ca-bundle-files).
- [ ] Run `go version` on each deployed Go binary. Rebuild each binary built with an affected Go
      version.
- [ ] Check each JVM. Upgrade the JDK or add the `blocked.certs` entries, then restart.
- [ ] Upgrade Node.js and restart each Node.js process.
- [ ] Check the lockfiles for `certifi`, `webpki-roots`, `rustls-webpki` and `composer/ca-bundle`.
      See [the version table](#3-check-component-versions).

Next:

- [ ] Fix the developer laptops, mainly macOS.
- [ ] Check Windows systems that verify code signatures.
- [ ] Search TLS inspection and proxy logs for the compromised intermediates.
- [ ] Decide whether to rotate credentials. See [Investigate past exposure and rotate credentials](#investigate-past-exposure-and-rotate-credentials).
- [ ] Notify [affected teams and vendors](#notify-affected-teams-and-vendors).

## What happened

On 30 April 2025 eMudhra created eight intermediate CAs with weak private keys. eMudhra says a flaw
in its key generation script generated a second set of weak keys, and staff used the wrong set
([Mozilla bug 2083449][bz]). The eight intermediates are:

- TLS: `EM DV TLS CA - G3B`, `EM DV TLS CA - G3C`, `EM OV TLS CA - G3B`, `EM OV TLS CA - G3C`,
  `EM EV TLS CA - G3B`, `EM EV TLS CA - G3C`
- Code signing: `EM OV CS CA - G3B`, `EM OV CS CA - G3C`

Each intermediate has two paths to a trusted root. Sectigo signed a second copy of each
intermediate in May 2025. This gives 16 CA certificates with 8 keys. A client that trusts the root
of either path accepts a forged certificate, unless it blocks the intermediates or checks
revocation.

![Two paths from a forged certificate to a trusted root. On the emSign path, a forged leaf chains through a compromised eMudhra intermediate and emSign Root TLS CA - G3 to the root emSign ECC Root CA - G3. On the Sectigo path, a Sectigo copy of the intermediate chains to the root AAA Certificate Services.](https://safedep.io/_next/static/immutable/media/emudhra-ca-chain-paths.1xn3v8s_uc8-i.svg)

```text
emSign path:   leaf -> EM DV TLS CA - G3B -> emSign Root TLS CA - G3 -> emSign ECC Root CA - G3
Sectigo path:  leaf -> EM DV TLS CA - G3B (Sectigo copy) -> AAA Certificate Services
```

Timeline (UTC):

- **17 October 2026.** A researcher credited as Wayne sent proof of the weak keys to the browser
  vendors. Chrome and Firefox blocked the intermediates the same day.
- **18 and 19 October.** eMudhra revoked the eight certificates. Sectigo revoked its eight
  copies.
- **19 October.** Apple blocked the intermediates. Go released 1.27.3 and 1.26.10.
- **20 October.** Microsoft blocked the intermediates. Mozilla removed `emSign ECC Root CA - G3`
  from its root list (NSS 3.131.1).
- **21 October.** Node.js released fixes. Red Hat published [RHSB-2026-012][rh]. Projects track
  the trust store updates as CVE-2026-111943.
- **27 October.** Chrome will remove `emSign ECC Root CA - G3` ([Google][google]).

Browsers reject the forged certificates because browser vendors push blocklists to them (Chrome
CRLSet, Firefox OneCRL). Most other clients read a CA bundle file from disk and do not check
revocation. For the full technical analysis, read
[Scott Helme][helme] and [Mozilla bug 2083449][bz]. The community tracker at
[`emudhra.we-have-a-problem.com`][tracker] keeps the test hosts and the current patch list.

## Who is affected

A client is affected when all of these are true:

1. It trusts `emSign ECC Root CA - G3`, `emSign Root TLS CA - G3` or `AAA Certificate Services`.
   Trust stores that come from the Mozilla root list (Debian, Ubuntu, Alpine, `certifi`, Node.js)
   do not contain `AAA Certificate Services`. Windows, macOS, older Linux releases and some JDK
   builds contain it.
2. It does not block the 16 intermediate certificates.
3. It does not check revocation.

A client keeps its trust store in one or more of these places:

| Place                           | Examples                                                                                                               | Fix                             |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| OS trust store                  | `ca-certificates` on hosts and in container images                                                                     | Upgrade the package and rebuild |
| Runtime                         | Go key blocklist, Node.js roots, JDK `cacerts`, Deno, Electron                                                         | Upgrade, rebuild and restart    |
| Package in the lockfile         | `certifi`, `webpki-roots`, `webpki-root-certs`, `castore`, `composer/ca-bundle`, `Mozilla::CA`                         | Upgrade the package             |
| CA bundle file in a source tree | `cacert.pem` in a repository, an SDK, firmware, a GraalVM native image, or a precompiled binary that bundles `certifi` | Find it with the scan           |

Risk by system:

| System                                              | Risk       | Why                                                                         |
| --------------------------------------------------- | ---------- | --------------------------------------------------------------------------- |
| CI/CD runners and build containers                  | High       | They download code and hold cloud keys, registry tokens and signing keys    |
| Services that call external APIs                    | High       | Payment, OAuth, webhook and cloud SDK calls carry secrets                   |
| `curl \| sh` installers and self-updaters           | High       | A forged certificate can deliver attacker code                              |
| IoT devices and appliances                          | High       | Old firmware trusts these CAs until the certificates expire                 |
| Developer laptops                                   | Medium     | macOS `/usr/bin/curl`, `git`, `python3` and `openssl` read an old CA bundle |
| Windows code signing                                | Medium     | Microsoft trusts `emSign Root CS CA - G3` for code signing                  |
| Servers with certificates that chain to the G3 root | Outage now | Firefox and patched clients reject the root. Chrome follows on 27 October   |
| Current browsers                                    | Low        | Browser vendors push a block for the intermediates                          |

An attack needs a forged certificate and a position where the attacker can intercept the traffic.
Examples are a hostile Wi-Fi network, a compromised router, a DNS or BGP hijack, or a shared
network of a self-hosted runner.

## How to test your systems

### 1. Run a probe where the client runs

A probe is a short program that connects to a tracker test host. The test hosts serve a forged
certificate. Each probe prints one of three results:

| Result         | Meaning                                                           | Action                                                                                                  |
| -------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `AFFECTED`     | The client trusts the forged certificate                          | Fix the client. See [Update affected systems](#update-affected-systems).                                |
| `NOT AFFECTED` | The client rejected the certificate                               | Check that both test hosts give this result                                                             |
| `INCONCLUSIVE` | The connection failed for another reason (DNS, proxy, no network) | Use the [scan](#2-scan-images-and-ca-bundle-files) and the [version table](#3-check-component-versions) |

Rules for a correct result:

- Test both paths. `dv-g3b-emudhra.we-have-a-problem.com` tests the emSign path.
  `dv-g3b-aaa-emudhra.we-have-a-problem.com` tests the Sectigo path. A client is not affected only
  if both hosts give `NOT AFFECTED`. Each command below loops over both hosts.
- Run the probe in the real environment: in the CI job, in each production image, and on each
  laptop type. Use the same environment variables as the application. `SSL_CERT_FILE`,
  `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `NODE_EXTRA_CA_CERTS` and `NODE_OPTIONS` change the
  trust store that a client reads.
- Through a TLS inspection proxy, the probe tests the proxy's certificate validation rather than
  the client's. To test the client directly, use a network path without TLS inspection or the
  offline checks in sections 2 and 3.

Set the two test hosts once:

```bash
HOSTS="dv-g3b-emudhra.we-have-a-problem.com dv-g3b-aaa-emudhra.we-have-a-problem.com"
```

**curl.** Exit code 0 means `AFFECTED`. Exit code 60 means `NOT AFFECTED`. Any other exit code
means `INCONCLUSIVE`.

```bash
for HOST in $HOSTS; do curl -sS -o /dev/null --max-time 10 "https://$HOST/"; echo "$HOST: curl exit code $?"; done
```

**Python.** The probe tests the standard library, which reads the OS trust store, and `requests`,
which reads `certifi`. Run it in each virtual environment. It needs Python 3.6 or later.

```python
# probe.py: python3 probe.py HOST
import ssl
import sys
import urllib.request

host = sys.argv[1] if len(sys.argv) > 1 else "dv-g3b-emudhra.we-have-a-problem.com"
url = f"https://{host}/"


def report(name, fetch):
    try:
        fetch()
        print(f"{host} {name}: AFFECTED")
    except Exception as e:
        cert_error = isinstance(e, ssl.SSLCertVerificationError) or "CERTIFICATE_VERIFY_FAILED" in str(e)
        print(f"{host} {name}: {'NOT AFFECTED' if cert_error else 'INCONCLUSIVE'} ({type(e).__name__})")


report("urllib, OS trust store", lambda: urllib.request.urlopen(url, timeout=10))
try:
    import requests
except ImportError:
    print(f"{host} requests: not installed")
else:
    report("requests, certifi", lambda: requests.get(url, timeout=10))
```

```bash
for HOST in $HOSTS; do python3 probe.py "$HOST"; done
```

**Node.js.** The probe needs Node.js 18 or later.

```bash
for HOST in $HOSTS; do
node -e '
fetch(`https://${process.argv[1]}/`, { signal: AbortSignal.timeout(10000) }).then(
  () => console.log(process.argv[1], process.version, "AFFECTED"),
  (e) => {
    const code = e.cause?.code ?? String(e.cause ?? e);
    const result = /CERT|UNABLE_TO|SIGNATURE/.test(code) ? "NOT AFFECTED" : "INCONCLUSIVE";
    console.log(process.argv[1], process.version, result, code);
  });' "$HOST"
done
```

**Go.** This probe tests the Go version that runs it. It needs Go 1.20 or later. A deployed binary keeps the Go version that
built it. Check deployed binaries with `go version` in [section 3](#3-check-component-versions).

```go
// probe.go: go run probe.go HOST
package main

import (
	"crypto/tls"
	"errors"
	"fmt"
	"net/http"
	"os"
	"runtime"
	"time"
)

func main() {
	host := "dv-g3b-emudhra.we-have-a-problem.com"
	if len(os.Args) > 1 {
		host = os.Args[1]
	}
	client := http.Client{Timeout: 10 * time.Second}
	resp, err := client.Get("https://" + host + "/")
	var certErr *tls.CertificateVerificationError
	switch {
	case err == nil:
		resp.Body.Close()
		fmt.Println(host, runtime.Version(), "AFFECTED")
	case errors.As(err, &certErr):
		fmt.Println(host, runtime.Version(), "NOT AFFECTED:", err)
	default:
		fmt.Println(host, runtime.Version(), "INCONCLUSIVE:", err)
	}
}
```

```bash
for HOST in $HOSTS; do go run probe.go "$HOST"; done
```

**Java.** JDK 11 and later can run one source file. For an image with only a JRE, compile the
probe on a JDK with `javac --release 11 Probe.java`, copy `Probe.class` into the image, and run
`java Probe "$HOST"`. Java 8 cannot run this probe. Use the offline `blocked.certs` check in
[section 3](#3-check-component-versions).

```java
// Probe.java: java Probe.java HOST
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import javax.net.ssl.SSLHandshakeException;

public class Probe {
    public static void main(String[] args) throws Exception {
        var host = args.length > 0 ? args[0] : "dv-g3b-emudhra.we-have-a-problem.com";
        var client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();
        var request = HttpRequest.newBuilder(URI.create("https://" + host + "/")).build();
        try {
            client.send(request, HttpResponse.BodyHandlers.discarding());
            System.out.println(host + " " + Runtime.version() + " AFFECTED");
        } catch (SSLHandshakeException e) {
            var certError = String.valueOf(e.getMessage()).matches("(?s).*(PKIX|[Cc]ertificate).*");
            System.out.println(host + " " + Runtime.version() + (certError ? " NOT AFFECTED: " : " INCONCLUSIVE: ") + e.getMessage());
        } catch (IOException e) {
            System.out.println(host + " " + Runtime.version() + " INCONCLUSIVE: " + e);
        }
    }
}
```

```bash
for HOST in $HOSTS; do java Probe.java "$HOST"; done
```

**PHP.** The probe tests the curl extension and PHP streams. In a Composer project it also tests
the CA bundle file of `composer/ca-bundle`. Run it from the project directory. It needs PHP 8.

```php
<?php
// probe.php: php probe.php HOST
$host = $argv[1] ?? "dv-g3b-emudhra.we-have-a-problem.com";
$url = "https://$host/";

$c = curl_init($url);
curl_setopt_array($c, [CURLOPT_RETURNTRANSFER => true, CURLOPT_TIMEOUT => 10]);
if (curl_exec($c) !== false) {
    echo "$host curl extension: AFFECTED\n";
} else {
    $result = curl_errno($c) === 60 ? "NOT AFFECTED" : "INCONCLUSIVE";
    echo "$host curl extension: $result (" . curl_error($c) . ")\n";
}

function stream_probe(string $host, string $name, string $url, array $ssl = []): void {
    $context = stream_context_create(["http" => ["timeout" => 10], "ssl" => $ssl]);
    $errors = "";
    set_error_handler(function ($code, $message) use (&$errors) {
        $errors .= $message . "\n";
        return true;
    });
    $body = file_get_contents($url, false, $context);
    restore_error_handler();
    if ($body !== false) {
        echo "$host $name: AFFECTED\n";
        return;
    }
    $result = str_contains($errors, "certificate verify failed") ? "NOT AFFECTED" : "INCONCLUSIVE";
    echo "$host $name: $result\n";
}

stream_probe($host, "streams", $url);
if (is_file("vendor/autoload.php")) {
    require "vendor/autoload.php";
    stream_probe($host, "composer/ca-bundle", $url, ["cafile" => \Composer\CaBundle\CaBundle::getBundledCaBundlePath()]);
}
```

```bash
for HOST in $HOSTS; do php probe.php "$HOST"; done
```

**Rust.** A probe that you build now gets the newest crates, so it does not show what your
application uses. Check the crate versions in the `Cargo.lock` of the application instead:

```bash
grep -A1 -E 'name = "(webpki-roots|webpki-root-certs|rustls-webpki)"' Cargo.lock
```

An affected build shows old versions, for example:

```text
name = "rustls-webpki"
version = "0.103.15"
--
name = "webpki-roots"
version = "1.0.9"
```

Upgrade `webpki-roots` and `webpki-root-certs` to 1.0.10 or later, and `rustls-webpki` to 0.103.16
or later. In the SafeDep test, either upgrade alone caused the client to reject the forged
certificate. `rustls-webpki` 0.102 and older get no fix. No output means that the build does not
use these crates.

SafeDep ran each probe on 22 October against two local servers that serve a forged certificate,
one on each path. Each probe printed `AFFECTED` with the old version and `NOT AFFECTED` with the
patched version. On the Sectigo path, every trust store from the Mozilla root list gave
`NOT AFFECTED`, old and new, because it does not contain `AAA Certificate Services`. Go 1.26.9
with a CA bundle file that contains that root gave `AFFECTED`, and Go 1.26.10 gave
`NOT AFFECTED`.

| Client                     | `AFFECTED`                                     | `NOT AFFECTED`                                    |
| -------------------------- | ---------------------------------------------- | ------------------------------------------------- |
| curl and PHP               | Ubuntu `ca-certificates 20260601~26.04.1`      | Debian `ca-certificates 20250419+deb13u1`         |
| Python `requests`          | `certifi` 2026.7.22                            | `certifi` 2026.10.19                              |
| Node.js                    | 22.23.3                                        | 22.23.4 (`CERT_REVOKED`)                          |
| Go, with the old CA bundle | go1.26.9                                       | go1.26.10                                         |
| Rust `ureq`                | `rustls-webpki` 0.103.15, `webpki-roots` 1.0.9 | `rustls-webpki` 0.103.16 or `webpki-roots` 1.0.10 |
| Java, Temurin 21.0.12.1+1  | Default `cacerts`                              | With the `blocked.certs` entries                  |
| PHP `composer/ca-bundle`   | 1.4.15, 1.5.14                                 | 1.4.16                                            |

The tracker has twelve test hosts: one for each of the six TLS intermediates on each path. To test
a trust store, use the two `dv-g3b` hosts. To test a blocklist (Go, Node.js, Java `blocked.certs`,
Windows), use all twelve hosts. The host names are `{dv,ov,ev}-{g3b,g3c}-emudhra.we-have-a-problem.com` and
`{dv,ov,ev}-{g3b,g3c}-aaa-emudhra.we-have-a-problem.com`.

#### Test the real application with mitmproxy

The probes test a runtime with one request to a tracker host. mitmproxy 12.3.0 and later can test
the real application against any host, internal hosts too ([mitmproxy commit][mitm]). The
`--emudhra` option signs each intercepted connection with the recovered key of
`EM OV TLS CA - G3C` on the emSign path. The `--emudhra-old` option uses the Sectigo path. Use
mitmproxy only on systems that you own.

Install mitmproxy in a virtual environment:

```bash
python3 -m venv mitm && mitm/bin/pip install 'mitmproxy>=12.3.0'
```

Start it:

```bash
mitm/bin/mitmdump --emudhra -p 8080
```

In a second terminal, send the client through the proxy. This example uses curl:

```bash
HTTPS_PROXY=http://127.0.0.1:8080 NO_PROXY= curl -sS -o /dev/null https://example.com/; echo "curl exit code $?"
```

Read the result:

- The request works: `AFFECTED`.
- The client reports a certificate error: `NOT AFFECTED`.
- `mitmdump` does not print the request: the client did not use the proxy. The result is
  `INCONCLUSIVE`.

Then stop `mitmdump`, start it with `--emudhra-old`, and run the client again.

To send other clients through the proxy:

- Python, Go, curl and most command-line tools read `HTTPS_PROXY`. Clear `NO_PROXY` for the test
  host. Go never uses a proxy for `localhost`.
- Node.js also needs `NODE_USE_ENV_PROXY=1`.
- Java needs `-Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=8080`.

mitmproxy connects to the real host, so the machine that runs mitmproxy must reach that host.
SafeDep ran the curl, Python, Go, Node.js and Java probes through `mitmdump --emudhra` against
`example.com` on 22 October. Each probe gave the same result as in the table above.

### 2. Scan images and CA bundle files

The scan works without network access. It finds each PEM file (a text file of certificates) and
compares the SHA-256 fingerprint of each certificate with the three eMudhra roots,
`AAA Certificate Services` and the 16 intermediates. It prints `FAIL` for each match and exits 1.
A `FAIL` on `AAA Certificate Services` means that a client which reads that file and has no
blocklist trusts the Sectigo path. It needs GNU grep and OpenSSL.

The scan does not read the JDK `cacerts` keystore, which is a binary file. It does not see roots
that are compiled into Node.js or Go. Use the version table to check JDK, Node.js and Go versions.

```sh
#!/bin/sh
# emudhra-scan.sh: find CA bundle files (PEM) that trust the compromised eMudhra intermediates.
# Usage: emudhra-scan.sh [DIR ...]   (default: /)
# Exit 1 on a match.
set -u

BAD='
86A1ECBA089C4A8D3BBE2734C612BA341D813E043CF9E8A862CD5C57A36BBE6B FAIL emSign ECC Root CA - G3
7DD78D5F4F13459A83DFF9ABBBA62EDBAF6F2D102BF257FD712F4D9F2746ED8D FAIL emSign Root TLS CA - G3
F2EDAAFF3D9B701FB4D14C9F285F8C0BAD90FC3BCC1FA07D47F4718EDABA34A5 FAIL emSign Root CS CA - G3
D7A7A0FB5D7E2731D771E9484EBCDEF71D5F0C3E0A2948782BC83EE0EA699EF4 FAIL AAA Certificate Services (Sectigo path)
42C4ACCD12C4F59597925E1A08A7A90102608543809308D50EC427E118BA1493 FAIL EM DV TLS CA - G3B
45A883F349AEDFD5F3A649B5498FFB121712BA8B448F3F2A91DBC5A766FCDDBE FAIL EM DV TLS CA - G3B (AAA)
98B896625C0C230ADEEC3EB2CF6AAD5001ECF9156A68644E82374CE5833BC25B FAIL EM OV TLS CA - G3B
5F1DED4DD590AF74631290CF073B122A2AAC6A1FA479778A0BD43579BD832699 FAIL EM OV TLS CA - G3B (AAA)
E7568FE6E12331F4FD7DE61DA4C7559D0F8EB192A218D4741A3BA75184333838 FAIL EM EV TLS CA - G3B
579480AEA166290BD3C643589E795FC0D0E54E86505D180F1DFF99EFD3199512 FAIL EM EV TLS CA - G3B (AAA)
0164D7CEAE5BB909C272AF151FF44B1177AE9B47D9424B94F1D9558C30934D19 FAIL EM DV TLS CA - G3C
62B7F9868263F72F6911FC78CEAF72C1979158B7368BF648683BB2AA39B02E72 FAIL EM DV TLS CA - G3C (AAA)
CB6E15165AC802E9B3F993F16C25D41F4E81C21EBDE71AE1B012AC00BF9C7C3F FAIL EM OV TLS CA - G3C
7485B51C0ED192878FA1D2F948FB529AE1842CC5086C4B67F7BE5B2039DE9489 FAIL EM OV TLS CA - G3C (AAA)
432EC25F0D28DE9006D97BF752F17BBB2CD41F669933FCB072D9A1148D0261F3 FAIL EM EV TLS CA - G3C
FD9D5C2DBBD4995AB69B8D50C439D9527B00529199C2FD01B2174EB58F07021F FAIL EM EV TLS CA - G3C (AAA)
9E39DE2881273ACC225A149C72BA3E0F51424963669AB633EE1DC7DF5DB067B4 FAIL EM OV CS CA - G3B
A496E23AC0CAB8155FC9B117B97EAE368EFCBB9041AEC8804D5A308D2D01C2D8 FAIL EM OV CS CA - G3B (AAA)
4B2460A466F95FD40113BCDCCA2CEB29B3B093921DE45474DA7DF48874C630B1 FAIL EM OV CS CA - G3C
84112972B8E59D14560AE7E7119654474AAD119990CF3722BFC1E3B91615E1FD FAIL EM OV CS CA - G3C (AAA)
'

[ $# -eq 0 ] && set -- /
tmp=$(mktemp -d) || exit 2
trap 'rm -rf "$tmp"' EXIT
fails=0

for root in "$@"; do
  grep -rlIs --exclude-dir=proc --exclude-dir=sys --exclude-dir=dev \
    -e '-----BEGIN CERTIFICATE-----' "$root" 2>/dev/null
done | sort -u > "$tmp/files"

while IFS= read -r f; do
  rm -f "$tmp"/c.*
  awk -v d="$tmp" '/-----BEGIN CERTIFICATE-----/{n++; out=d "/c." n} out{print > out} /-----END CERTIFICATE-----/{close(out); out=""}' "$f"
  for c in "$tmp"/c.*; do
    [ -f "$c" ] || continue
    fp=$(openssl x509 -in "$c" -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//; s/://g')
    [ -n "$fp" ] || continue
    name=$(printf '%s\n' "$BAD" | grep "^$fp " | cut -d' ' -f3-)
    [ -n "$name" ] || continue
    printf 'FAIL\t%s: %s\n' "$f" "$name"
    fails=$((fails + 1))
  done
done < "$tmp/files"

echo "files scanned: $(wc -l < "$tmp/files"), matches: $fails" >&2
[ "$fails" -eq 0 ]
```

To scan an image, export its file system and scan it on the host:

```bash
cid=$(docker create eclipse-temurin:21-jre)
mkdir rootfs && docker export "$cid" | tar -x -C rootfs && docker rm "$cid"
sh emudhra-scan.sh rootfs
```

On 22 October the `eclipse-temurin:21-jre` image (Ubuntu 26.04, `ca-certificates
20260601~26.04.1`) gave this result:

```text
FAIL	rootfs/etc/ssl/certs/ca-certificates.crt: emSign ECC Root CA - G3
FAIL	rootfs/usr/share/ca-certificates/mozilla/emSign_ECC_Root_CA_-_G3.crt: emSign ECC Root CA - G3
files scanned: 123, matches: 2
```

The `python:3.13-slim` image (Debian 13, `ca-certificates 20250419+deb13u1`) had no match. The
`debian:12` and `ubuntu:24.04` base images have no CA bundle file. The file comes from the
`ca-certificates` package that your Dockerfile installs, so scan your own images. The
`alpine:3.22` image ships `ca-certificates-bundle 20261020-r0`, which has the fix.

To list the images that run in a Kubernetes cluster:

```bash
kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | sort -u
```

Do not check for the root by name. `grep -c emSign /etc/ssl/certs/ca-certificates.crt` gives 0 on
an affected system, because that file has no names in it. In a bundle file with names, the same
command finds three other eMudhra roots (`emSign Root CA - G1`, `emSign Root CA - C1` and
`emSign ECC Root CA - C3`) that are not part of this incident. Compare fingerprints.

To check one CA bundle file without the script, download
[chains.zip](https://emudhra.we-have-a-problem.com/chains.zip) from the tracker and verify both
test chains with OpenSSL. `OK` means `AFFECTED`. `error 20` (unable to get local issuer
certificate) means `NOT AFFECTED`. Any other error means `INCONCLUSIVE`.

```bash
for h in dv-g3b-emudhra dv-g3b-aaa-emudhra; do
  openssl verify -no-CApath -no-CAstore -CAfile /etc/ssl/certs/ca-certificates.crt \
    -untrusted "$h.we-have-a-problem.com.pem" "$h.we-have-a-problem.com.pem"
done
```

### 3. Check component versions

This table lists patched versions from the [tracker][tracker] as of 22 October. Check the tracker
again before you close the incident.

| Component                                          | How to check                                                                         | Patched version                                                                                            | Action                                 |
| -------------------------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| Ubuntu `ca-certificates`                           | `dpkg-query -W ca-certificates`                                                      | `20260816ubuntu0.22.04.1`, `20260816ubuntu0.24.04.1`, `20260816ubuntu0.26.04.1`, `20260816ubuntu0.26.10.1` | Upgrade, rebuild images                |
| Debian `ca-certificates`                           | `dpkg-query -W ca-certificates`                                                      | `20250419~deb12u2` (12), `20250419+deb13u1` (13)                                                           | Upgrade, rebuild images                |
| Alpine `ca-certificates`, `ca-certificates-bundle` | `apk list -I 'ca-certificates*'`                                                     | `20261020-r0`                                                                                              | Upgrade, rebuild images                |
| Chainguard and Wolfi `ca-certificates`             | `apk list -I 'ca-certificates*'`                                                     | `20260909-r3`                                                                                              | Upgrade, or pull a new image digest    |
| Fedora `ca-certificates`                           | `rpm -q ca-certificates`                                                             | `2026.2.92_v9.0.319`                                                                                       | Upgrade, rebuild images                |
| RHEL and UBI `ca-certificates`                     | `rpm -q ca-certificates`                                                             | No package on 22 October                                                                                   | Use the [RHSB-2026-012][rh] workaround |
| Go binary                                          | `go version BINARY`                                                                  | go1.26.10 or later, go1.27.3 or later. Go 1.25 gets no fix                                                 | Rebuild, redeploy                      |
| Node.js                                            | `node --version`                                                                     | 22.23.4, 24.22.1, 26.12.1. Node.js 20 and older get no fix                                                 | Upgrade, restart                       |
| OpenJDK and Oracle JDK, per the tracker            | `java -version`                                                                      | 27.0.1.0.2, 25.0.5.0.2, 21.0.13.0.2, 17.0.21.0.2, 11.0.33.0.2, 8u511                                       | Upgrade, restart                       |
| Temurin                                            | `java -version`                                                                      | No fix on 22 October                                                                                       | Add `blocked.certs`, restart           |
| Debian and Ubuntu OpenJDK packages                 | `readlink -f "$JAVA_HOME/lib/security/cacerts"` prints `/etc/ssl/certs/java/cacerts` | Fixed with the OS trust store                                                                              | Upgrade `ca-certificates`, restart     |
| Python `certifi`                                   | `pip show certifi`                                                                   | 2026.10.19                                                                                                 | Upgrade                                |
| pip, uv                                            | `pip --version`, `uv --version`                                                      | pip 26.2.2, uv 0.13.4                                                                                      | Upgrade                                |
| Rust `webpki-roots`, `webpki-root-certs`           | `grep -A1 'name = "webpki-roots"' Cargo.lock`                                        | 1.0.10                                                                                                     | Upgrade                                |
| Rust `rustls-webpki`                               | `grep -A1 'name = "rustls-webpki"' Cargo.lock`                                       | 0.103.16                                                                                                   | Upgrade                                |
| PHP `composer/ca-bundle`                           | `composer show composer/ca-bundle`                                                   | 1.4.16 or 1.6.0. The 1.5 releases have no fix                                                              | Upgrade                                |
| Elixir `castore`                                   | `mix deps`                                                                           | 1.0.22                                                                                                     | Upgrade                                |
| Perl `Mozilla::CA`                                 | `perl -MMozilla::CA -e 'print $Mozilla::CA::VERSION, "\n"'`                          | 20261020                                                                                                   | Upgrade                                |
| Electron                                           | The version of the app                                                               | 44.9.1, 43.8.1, 42.11.16                                                                                   | Ship a new build                       |
| Deno                                               | `deno --version`                                                                     | 2.9.8                                                                                                      | Upgrade                                |
| Homebrew `ca-certificates`                         | `brew list --versions ca-certificates`                                               | 2026-09-25_1                                                                                               | `brew upgrade`                         |
| mise                                               | `mise --version`                                                                     | 2026.10.17                                                                                                 | Upgrade                                |
| Claude Code                                        | `claude --version`                                                                   | 2.1.303                                                                                                    | Upgrade                                |
| Bun, Erlang `certifi`, Bottlerocket                | None                                                                                 | No fix on 22 October                                                                                       | Watch the tracker                      |

For a Go binary in an image, copy the binary out of the image and run `go version` on it:

```bash
cid=$(docker create IMAGE) && docker cp "$cid":/path/to/binary ./binary && docker rm "$cid"
```

```bash
go version ./binary
```

```text
./binary: go1.26.9
```

For a JVM without network access, check whether `blocked.certs` lists the intermediates. If the
command prints `0` and the JDK is not a patched version, the JVM is affected. On Java 8 the file is
`$JAVA_HOME/jre/lib/security/blocked.certs`.

```bash
grep -c 42C4ACCD12C4F59597925E1A08A7A90102608543809308D50EC427E118BA1493 "$JAVA_HOME/lib/security/blocked.certs"
```

### 4. Check lockfiles with vet

CA bundle packages often arrive as transitive dependencies of HTTP clients. Check the resolved
lockfiles. The `vulnerability` control of [`vet`](https://github.com/safedep/vet) reports
advisories for some of these packages, for example PYSEC-2026-4320 for `certifi` and
RUSTSEC-2026-0381 for `webpki-roots`. This `vet` policy rule fails the build on each CA bundle
package that is older than its fix:

```yaml
# vet-policy.yml
version: 2
rules:
  - id: stale-ca-bundle
    description: A CA bundle package older than the eMudhra fix still trusts the compromised intermediates.
    when: >-
      (package.is("certifi") && package.version_cmp("2026.10.19") < 0) ||
      (package.is("webpki-roots") && package.version_cmp("1.0.10") < 0) ||
      (package.is("webpki-root-certs") && package.version_cmp("1.0.10") < 0) ||
      (package.is("rustls-webpki") && package.version_cmp("0.103.16") < 0) ||
      (package.is("composer/ca-bundle") && (package.version_cmp("1.4.16") < 0 ||
        (package.version_cmp("1.5.0") >= 0 && package.version_cmp("1.6.0") < 0))) ||
      (package.is("castore") && package.version_cmp("1.0.22") < 0) ||
      (package.is("pip") && package.version_cmp("26.2.2") < 0)
    action: fail
    severity: high
    help: Upgrade the package to the version that removes the eMudhra roots, then rebuild.
    link: https://emudhra.we-have-a-problem.com/
```

Policy v2 is in the `vet` v2 alpha. Install it with `brew install --cask safedep/tap/vet@edge` or
`go install github.com/safedep/vet/v2/cmd/vet@latest`. Then run:

```bash
vet scan --policy vet-policy.yml -o plain
```

Exit code 1 means that the gate failed. On a test project with `certifi==2026.7.22` and
`webpki-roots 1.0.9`, the `policy` rows are the findings of this rule. The `vulnerability` rows
are advisories, and the `requests` rows are not part of this incident.

```text
SEVERITY	CONTROL	SUBJECT	WHERE	FINDING_ID
critical	vulnerability	pkg:pypi/certifi@2026.7.22	requirements.txt:2	f-a1adf3f5b5408696
critical	vulnerability	pkg:pypi/certifi@2026.7.22	requirements.txt:2	f-e988805d0d07e1eb
high	policy	pkg:pypi/certifi@2026.7.22	requirements.txt:2	f-1cec22867326d431
high	policy	pkg:cargo/webpki-roots@1.0.9	Cargo.lock:13	f-dbaa4355032eee40
medium	vulnerability	pkg:pypi/requests@2.32.5	requirements.txt:1	f-240166541670eda0
medium	vulnerability	pkg:cargo/webpki-roots@1.0.9	Cargo.lock:13	f-4e2db632712d19b9
medium	vulnerability	pkg:pypi/requests@2.32.5	requirements.txt:1	f-d7a860703900ae40
```

A lockfile scan does not find a CA bundle file in a source tree or the Go version of a binary.
Use the CA bundle scan to find files in source trees, and `go version` to check compiled Go binaries.

### 5. Check your own certificates

Check the chain of each endpoint that you operate, and of each vendor endpoint that your services
call:

```bash
openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null 2>/dev/null \
  | grep -E '^ *([0-9]+ )?[si]:' \
  | grep -E 'emSign ECC Root CA - G3|emSign Root TLS CA - G3|EM (DV|OV|EV) TLS CA - G3[BC]' \
  || echo "no match"
```

Read the result:

- A match on `emSign ECC Root CA - G3` or `emSign Root TLS CA - G3` means that the certificate
  chains to the root that Firefox and patched clients reject now. Replace the certificate now
  with one from another chain. ACME users can change the directory URL to another CA. Tell the
  owners of the clients that call this endpoint.
- A match on `EM DV`, `EM OV` or `EM EV TLS CA - G3B` or `G3C` means that the endpoint serves a
  certificate from a compromised intermediate. eMudhra issued no TLS certificates under these
  intermediates before browsers blocked them ([Google][google]). Treat the certificate as forged
  and start an incident. The tracker test hosts are the exception.
- `no match` means that the endpoint does not use these chains.

An updated trust store also breaks calls to vendor endpoints with certificates that chain to
`emSign ECC Root CA - G3`. Check those endpoints before you upgrade, and ask the vendor to replace
the certificate.

Next, look for forged certificates for your own domains. Search the CA pages on
[`ctlogs.dev`][ctlogs]. crt.sh did not show these entries on 22 October.

## Update affected systems

Start with the systems that hold secrets: CI runners, services that call external APIs, and
deployment tools. Use [the version table](#3-check-component-versions) for each patched version.

1. **Upgrade the OS trust store and rebuild each image.** A host upgrade does not change an image
   that is already built. Add the upgrade to the Dockerfile, build with `--pull --no-cache`,
   update pinned digests, and redeploy.

   ```dockerfile
   RUN apt-get update && apt-get install -y --only-upgrade ca-certificates \
       && rm -rf /var/lib/apt/lists/*
   ```

   ```dockerfile
   RUN apk upgrade --no-cache ca-certificates-bundle ca-certificates
   ```

   Distroless and Chainguard images have no package manager. Pull a new image digest and rebuild.

   Red Hat has no patched package for RHEL and UBI on 22 October. Use the workaround in
   [RHSB-2026-012][rh], and rebuild UBI images when the package is released.

2. **Rebuild each Go binary** with Go 1.27.3 or 1.26.10 or later, then redeploy it. The fix is
   compiled into the binary. A new Go version on the build host does not change a deployed binary.
   Go 1.25 gets no fix, so move to Go 1.26.10 or later.

3. **Update each JVM.** Upgrade to a patched JDK. If your vendor has no fix yet, add the 16
   intermediates to `blocked.certs`.

   Temurin, Oracle and other vendor JDKs use their own `cacerts` files. Updating the OS trust
   store does not update these files. Temurin's `USE_SYSTEM_CA_CERTS=1` option adds roots but
   does not remove them.

   Debian and Ubuntu OpenJDK packages use `/etc/ssl/certs/java/cacerts`, which the OS trust store
   builds. For these packages, the OS trust store update in step 1 applies.

   SafeDep tested the OS trust store update and `USE_SYSTEM_CA_CERTS=1` on Temurin 21.0.12.1+1.
   Java still trusted the forged certificate. SafeDep also tested the OS trust store update with
   OpenJDK 17 on Debian 12.

   To block the intermediates, save this file as `emudhra-blocked.certs`:

   ```text
   42C4ACCD12C4F59597925E1A08A7A90102608543809308D50EC427E118BA1493
   45A883F349AEDFD5F3A649B5498FFB121712BA8B448F3F2A91DBC5A766FCDDBE
   98B896625C0C230ADEEC3EB2CF6AAD5001ECF9156A68644E82374CE5833BC25B
   5F1DED4DD590AF74631290CF073B122A2AAC6A1FA479778A0BD43579BD832699
   E7568FE6E12331F4FD7DE61DA4C7559D0F8EB192A218D4741A3BA75184333838
   579480AEA166290BD3C643589E795FC0D0E54E86505D180F1DFF99EFD3199512
   0164D7CEAE5BB909C272AF151FF44B1177AE9B47D9424B94F1D9558C30934D19
   62B7F9868263F72F6911FC78CEAF72C1979158B7368BF648683BB2AA39B02E72
   CB6E15165AC802E9B3F993F16C25D41F4E81C21EBDE71AE1B012AC00BF9C7C3F
   7485B51C0ED192878FA1D2F948FB529AE1842CC5086C4B67F7BE5B2039DE9489
   432EC25F0D28DE9006D97BF752F17BBB2CD41F669933FCB072D9A1148D0261F3
   FD9D5C2DBBD4995AB69B8D50C439D9527B00529199C2FD01B2174EB58F07021F
   9E39DE2881273ACC225A149C72BA3E0F51424963669AB633EE1DC7DF5DB067B4
   A496E23AC0CAB8155FC9B117B97EAE368EFCBB9041AEC8804D5A308D2D01C2D8
   4B2460A466F95FD40113BCDCCA2CEB29B3B093921DE45474DA7DF48874C630B1
   84112972B8E59D14560AE7E7119654474AAD119990CF3722BFC1E3B91615E1FD
   ```

   Add it in the Dockerfile:

   ```dockerfile
   COPY emudhra-blocked.certs /tmp/emudhra-blocked.certs
   RUN cat /tmp/emudhra-blocked.certs >> "$JAVA_HOME/lib/security/blocked.certs" \
       && rm /tmp/emudhra-blocked.certs
   ```

   Restart each JVM. After this change, the Java probe printed this against the SafeDep test
   server:

   ```text
   localhost:8443 21.0.12.1+1-LTS NOT AFFECTED: (certificate_unknown) PKIX path validation failed: java.security.cert.CertPathValidatorException: Untrusted certificate: CN=EM DV TLS CA - G3B, O=eMudhra BV, C=NL
   ```

   `blocked.certs` blocks the intermediates on both paths and does not remove a root, so
   legitimate Sectigo chains keep working. GraalVM native images contain the trust store from
   build time. Add the file to the GraalVM installation and rebuild the native image.

4. **Upgrade Node.js and restart each process.** A running process uses the old version until it
   restarts. Electron apps need a new release that is built on a patched Electron version.

5. **Upgrade the CA bundle packages.** Require `certifi` 2026.10.19 or later in each Python
   environment, also where the project does not list `certifi`. Some libraries load `certifi` when
   it is installed. Upgrade `webpki-roots`, `rustls-webpki`, `composer/ca-bundle`, `castore` and
   `Mozilla::CA`. Replace each CA bundle file that the scan finds in a source tree.

6. **Fix developer laptops.** On macOS, `/usr/bin/curl`, `git`, `python3`, `perl`, `ruby` and
   `openssl` read `/etc/ssl/cert.pem`. Apple did not update that file. Apple's block covers only
   apps that use the macOS certificate check. Use Homebrew curl
   (`/opt/homebrew/opt/curl/bin/curl` on Apple silicon), or apply the macOS workaround on the
   [tracker][tracker]. Upgrade pip, uv and mise.

7. **Check Windows.** Microsoft sends the block through automatic certificate trust list updates.
   For a machine that does not get these updates, the [tracker][tracker] has a PowerShell script
   that adds the 16 certificates to `LocalMachine\Disallowed`. An application with its own CA
   bundle file still needs its own update.

8. **Close the Sectigo path.** If the scan finds `AAA Certificate Services` in a CA bundle file,
   first upgrade the package that ships the file. Remove the root by hand only if no update
   exists, because it also signs legitimate Sectigo chains. Test the services that use Sectigo
   certificates after you remove it. Clients with a blocklist (patched Go and Node.js versions,
   Java with `blocked.certs`) reject the Sectigo path without this change.

9. **Replace server certificates** that chain to `emSign ECC Root CA - G3` now.

10. **Fix the nodes and the proxies.** Kubernetes nodes pull images over TLS with containerd or
    Docker, which read the node OS trust store. Upgrade the node image. Upgrade each TLS
    inspection proxy, because it makes the trust decision for every client behind it.

11. **Fix GitHub-hosted runners.** You cannot change the runner image. Run the probes in a job to
    see the current result. Until the probes print `NOT AFFECTED`, run the job in a container image
    that you updated, or upgrade the trust store in a step at the start of the job.

Files to change in a repository:

- Dockerfiles: base image tags and digests, and the `ca-certificates` upgrade step.
- `go.mod` (`go` and `toolchain` lines) and the Go version in CI.
- `.nvmrc`, `package.json` `engines`, `.tool-versions` and `mise.toml`.
- The versions in `actions/setup-go`, `actions/setup-node` and `actions/setup-java` steps.
- Dependency lockfiles: `requirements.txt`, `poetry.lock`, `uv.lock`, `Cargo.lock`, `composer.lock`,
  `mix.lock`.
- Any `cacert.pem` or other CA bundle file that the scan finds in the source tree.

## Investigate past exposure and rotate credentials

The weakness became widely known between 17 and 20 October 2026. Forged certificates appeared in
CT logs from 19 October. Treat clients that were still affected after 17 October as the highest
risk. Before that date, an attacker had to discover the weakness independently. Nobody has
reported that this happened.

1. Search TLS inspection, proxy and EDR logs for server certificates that one of the 16
   intermediates signed. Match on the `spki_sha256` values (public key hashes) in
   [impacted-intermediates.json][json]. A match outside the test hosts is a forged certificate.
2. List the affected clients that sent secrets: CI jobs with cloud credentials, package publish
   steps, deployment tools, and API clients with long-lived tokens.
3. Decide on rotation. Use the first rule that applies:
   1. A log shows a forged certificate. Rotate the credentials of that client now and start an
      incident.
   2. The client was affected after 17 October, sent secrets, and used a self-hosted runner, a
      shared network or public Wi-Fi, or has no logs. Rotate now.
   3. The client was affected after 17 October and sent secrets. Rotate at the next maintenance
      window.
   4. In all other cases, do not rotate.

4. On Windows, check binaries and drivers signed under `EM OV CS CA - G3B` or `EM OV CS CA - G3C`
   with `Get-AuthenticodeSignature` or `signtool verify /pa /v`. Treat such a binary as suspect.

## Notify affected teams and vendors

- The security team or the incident lead, with the list of affected systems.
- The owners of each service with a certificate that chains to `emSign ECC Root CA - G3`.
- Vendors whose endpoints chain to `emSign ECC Root CA - G3`.
- Customers, if their clients connect to your endpoints on those chains.

## References

- [Mozilla bug 2083449: eMudhra private key compromise of intermediate certificates][bz]
- [Google: Protecting Chrome and Android users from compromised eMudhra intermediate CA certificates][google]
- [eMudhra intermediate CA compromise tracker: test hosts, patch list, workarounds][tracker]
- [Red Hat RHSB-2026-012][rh]
- [Node.js 22.23.4 release notes][node]
- [Go release history](https://go.dev/doc/devel/release)
- [Scott Helme: Eight bits of trust][helme]
- [Guillaume Laforge: The eMudhra CA compromise and the JVM][jvm]
- [`ctlogs.dev`: eMudhra intermediate CA compromise][ctlogs]
- [The Hacker News: Eight eMudhra CA keys compromised][thn]
- [mitmproxy: Add `--emudhra` option][mitm]

[bz]: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
[google]: https://blog.google/security/protecting-chrome-and-android-users-from-compromised-emudhra-intermediate-ca-certificates/
[tracker]: https://emudhra.we-have-a-problem.com/
[json]: https://emudhra.we-have-a-problem.com/impacted-intermediates.json
[rh]: https://access.redhat.com/security/vulnerabilities/RHSB-2026-012
[node]: https://nodejs.org/en/blog/release/v22.23.4
[helme]: https://scotthelme.co.uk/eight-bits-of-trust-emudhras-ca-keys-cracked-in-milliseconds/
[jvm]: https://glaforge.dev/posts/2026/10/21/the-emudhra-ca-compromise-why-your-jvm-is-vulnerable-and-how-to-fix-it/
[ctlogs]: https://ctlogs.dev/blog/emudhra-ca-compromise
[thn]: https://thehackernews.com/2026/10/eight-emudhra-ca-keys-compromised.html
[mitm]: https://github.com/mitmproxy/mitmproxy/commit/169234499088fc2c414644a67efaed3d64ae2e3b