ShortlistFox
← Back to results

Data Platform Engineer Principal_999

AZGROUPPROD

Minneapolis, MN

  • $130,000–$170,000 per year
Apply

About the role

What you do

Platform Architecture & Engineering

  • Design and implement Databricks architecture: workspace topology, Unity Catalog metastore design, cluster policies, compute pools, Delta Lake optimization (Z-ordering, OPTIMIZE/VACUUM scheduling, liquid clustering), and job orchestration strategy (Workflows, multi-task jobs, Delta Live Tables)
  • Architect Azure data landing zones: subscription/resource group hierarchy, hub-spoke networking, private endpoints, VNet integration for Databricks (VNet injection/secure cluster connectivity), ADLS Gen2 hierarchical namespace design, and Azure Data Factory/Synapse pipeline orchestration
  • Design Snowflake account-level architecture: multi-cluster warehouse strategy, resource monitor hierarchies, query performance tuning, data sharing/clean rooms, Snowpipe/Streams/Tasks for ingestion, and cost attribution models across business units
  • Define reference architectures and golden patterns for new data platform use cases; act as technical authority in architecture review boards

Access Management & Security Engineering

  • Design fine-grained authorization models: Unity Catalog row/column-level security and dynamic views, Snowflake RBAC hierarchies with functional/access roles separation, Azure RBAC + PIM for just-in-time privileged access
  • Engineer identity federation across platforms: Entra ID SCIM provisioning into Databricks/Snowflake, OAuth/SAML SSO, service principal and managed identity lifecycle management, secrets management (Azure Key Vault, Databricks secret scopes)
  • Lead security hardening: private connectivity (Private Link/Service Endpoints), network security group design, customer-managed encryption keys (CMK), audit log pipeline design (Unity Catalog audit logs, Snowflake ACCOUNT_USAGE, Azure Activity Logs) feeding into SIEM
  • Own technical response to security findings and support DORA/GDPR compliance evidence generation, including data residency and data lineage requirements

Cost & Performance Engineering

  • Build cost observability tooling/dashboards correlating Azure cost management, Databricks DBU consumption, and Snowflake credit usage against business units and workloads
  • Engineer compute efficiency: cluster autoscaling policies, spot/low-priority instance strategies, Photon acceleration adoption, warehouse auto-suspend tuning, query profiling and optimization (pruning, caching, clustering keys)
  • Implement FinOps automation: policy-as-code guardrails (cluster size limits, idle timeout enforcement), chargeback/showback models, anomaly detection on spend

Reliability Engineering & Operations

  • Define and operate SLOs/SLIs for platform availability and job reliability; build observability stacks (metrics, logs, traces) using Azure Monitor, Log Analytics, and Databricks system tables
  • Design disaster recovery architecture: cross-region replication for Delta Lake/ADLS, Snowflake database replication and failover groups, RTO/RPO-driven DR runbooks and tested failover procedures
  • Build CI/CD pipelines for infrastructure and platform configuration using Terraform (modules for Databricks/Azure/Snowflake providers), GitHub Actions/Azure DevOps, with automated drift detection and policy validation (OPA/Conftest, Checkov)
  • Lead incident response for Sev1/2 platform incidents; drive blameless postmortems and reliability improvements

Technical Leadership & Governance

  • Mentor junior/mid-level cloud administrators and data engineers on platform best practices
  • Define data platform engineering standards, IaC module libraries, and reusable automation frameworks
  • Partner with data engineering, security, and enterprise architecture to shape platform roadmap
  • Represent the data platform team in architecture governance and change advisory processes

 

What you bring

  • 5+ years in cloud infrastructure/platform engineering, with 3+ years focused on data platforms
  • Deep hands-on expertise with Databricks: Unity Catalog, Delta Lake internals, cluster/job architecture, performance tuning at scale
  • Advanced Microsoft Azure engineering experience: networking (VNets, Private Link, ExpressRoute), IAM/Entra ID, ADLS Gen2, Azure Data Factory/Synapse
  • Advanced Snowflake engineering experience: warehouse/query optimization, RBAC design, data sharing, Snowpipe/Streams/Tasks
  • Strong Infrastructure-as-Code expertise: Terraform (module authoring, state management, provider-level configuration for Databricks/Azure/Snowflake)
  • Strong scripting/automation: Python (API integrations, automation frameworks), PowerShell, Bash
  • Solid understanding of distributed systems, query engines (Spark internals), and data lake/lakehouse architecture patterns
  • Experience designing for compliance in regulated environments (DORA, GDPR, data residency/lineage requirements)
  • Proven incident response and reliability engineering experience (SLO design, postmortems, DR testing)

Preferred Qualifications

  • Advanced certifications: Databricks Certified Data Engineer Professional, SnowPro Advanced Architect, Azure Solutions Architect Expert
  • Experience in financial services/insurance with hands-on DORA/GDPR implementation
  • Experience with policy-as-code tooling (OPA, Sentinel, Checkov) and GitOps workflows
  • Familiarity with data lineage/catalog tools (Unity Catalog, Purview) and observability platforms (Azure Monitor, Datadog)
  • Experience building internal developer platforms or self-service data infrastructure tooling

 

What we offer

  • We offer a hybrid work model which recognizes the value of striking a balance between in-person collaboration and remote working
  • We believe in rewarding performance and our compensation and benefits package includes health insurance, company bonus scheme, 401(K) with company match, company paid holidays, paid time off, paid volunteer days, tuition reimbursement, paid parental leave, an employee shares program and employee discounts From career development and digital learning programs to international career mobility, we offer lifelong learning for our employees worldwide and an environment where innovation, delivery and empowerment are fostered.
  • The annualized base pay range for this role is $130,000 - 170,000. The annual base salary range represents a nationwide market range. The actual salary for this position will be determined by a number of factors, including the scope, complexity and location of the role, and the skills, education, training, credentials and experience of the candidate. The base pay is just one component of the ATA total compensation package. As part of our comprehensive compensation and highly rated benefits programs, ATA also offers eligibility for an incentive-based annual bonus.

 

 

About Allianz Technology

Allianz Group is one of the most trusted insurance and asset management companies in the world. Caring for our employees, their ambitions, dreams and challenges is what makes us a unique employer. 

 

We are united by a shared commitment: to put our customers first and at the centre of everything we do. Their needs inspire our thinking and guide our actions.

 

Together, we can build an environment where everyone feels empowered and confident to explore, grow and shape a better future – for our customers and for the world around us. At Allianz, we stand for unity: we believe that a united world is a more prosperous world, and we are dedicated to consistently advocating for equal opportunities for all. The foundation for this is our inclusive workplace, where people and performance both matter, and where integrity, fairness, inclusion and trust are at the heart of our culture. 

 

We therefore welcome applications regardless of ethnicity or cultural background, age, gender, nationality, religion, social class, disability or sexual orientation, or any other characteristics protected under applicable local laws and regulations.

 

Join us. Let’s care for tomorrow.

You.IT