ShortlistFox
← Back to results

Machine Learning Infrastructure Engineer

true-talent-solutions-partners

San Francisco

  • $200,000–$400,000 per year
  • Onsite
  • Full Time
Apply

About the role

About the Role

Join an AI team building large physics foundation models to predict real-world systems and identify effective actions. You will develop the distributed infrastructure that supports model training and inference, with weather data as a key application area.

What You'll Do

  • Design, deploy, and maintain distributed machine learning training and inference clusters.

  • Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the ML lifecycle.

  • Evaluate training approaches, including parallelization methods and numerical precision trade-offs across model sizes.

  • Profile and debug low-level GPU operations to improve performance.

  • Track relevant research and apply promising ideas to infrastructure and workflows.

What We're Looking For

  • Two to ten years of experience building large-scale infrastructure for pretrained or foundation models, including compute clusters, data infrastructure, inference, and distributed training.

  • Hands-on experience optimizing large-scale training, data, or inference workloads, and using distributed training frameworks such as FSDP or DeepSpeed.

  • Experience with low-level GPU performance analysis and debugging, including CUDA and JAX.

  • Familiarity with Python, C++, Linux, NVIDIA GPUs, cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.

  • Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications.

  • Comfort working across the ML lifecycle in a fast-paced environment, including analyzing experiments and ablation studies.

Compensation & Benefits

Base salary is $200,000 to $400,000 annually. Visa sponsorship is not available.

Location

On-site in San Francisco, California. Relocation support is provided.