IncidentAI infrastructure
ShadowRay: AI clusters hijacked through an unauthenticated jobs API
CriticalAnyscale Ray
Attackers took over internet-exposed Ray clusters, a widely used framework for scaling AI, through a job-submission API with no authentication, then mined cryptocurrency and stole credentials.
Replay the attack step by step in the Learning Center
How it happened
- Ray's Jobs API has no authorization, and many clusters were reachable from the internet.
- Attackers submitted jobs that ran arbitrary commands.
- They installed cryptominers and reverse shells.
- They harvested OpenAI, Hugging Face, Stripe and cloud credentials from the cluster.
Impact
Costly GPU clusters, cloud credentials and models could be taken over.
The fix
No patch; the vendor says Ray must run on strictly controlled networks, and Ray now documents an optional token authentication mode.
Lessons
- Never expose AI orchestration dashboards or job APIs to the internet.
- Keep long-lived secrets out of cluster environments.
- Watch GPU clusters for unexpected processes.