Data Engineer

This role is for one of Weekday’s clients

Salary range: Rs 1000000 - Rs 1500000 (ie INR 10 - 15 LPA)

Min Experience: 4+ years
Location: Pune, Bangalore
JobType: full-time

We are looking for a skilled Data Engineer with hands-on experience in data engineering, web scraping, system integrations, and data migration. The role involves building automated data extraction solutions to retrieve information from healthcare vendor portals, transforming and validating the extracted data, and migrating it into target systems through REST APIs.

The ideal candidate should have strong Python or Node.js/TypeScript programming skills, practical experience with browser automation tools, and the ability to build reusable, scalable data connectors. The candidate must also be comfortable interacting with US-based customers, managing multiple migrations under tight deadlines, and handling sensitive patient information in compliance with HIPAA requirements.

Requirements

Key Responsibilities

1. Data Extraction and Web Scraping

• Design, develop, and maintain automated crawlers to access healthcare vendor portals such as WebPT, Prompt, Raintree, Clinicient, and Jane.
• Extract and retrieve structured information, including patient records, insurance details, appointments, provider information, documents, and account balances.
• Handle portal-specific challenges such as multi-factor authentication (MFA), session timeouts, pagination, rate limits, and frequent UI or layout changes.
• Build reliable browser automation workflows using Playwright, Puppeteer, Selenium, or equivalent tools.

2. Data Transformation and Validation

• Map vendor-specific fields to the target data model and ensure accurate data transformation.
• Clean, deduplicate, validate, and normalise extracted datasets.
• Identify unmapped fields, inconsistencies, and missing information, and flag them for resolution.
• Use SQL and data-processing libraries such as Pandas to perform data wrangling and validation.

3. Data Migration and API Integration

• Load extracted and transformed data into target systems through REST APIs.
• Ensure data is migrated in the correct sequence with appropriate authentication, batching, pagination, and retry mechanisms.
• Implement structured logging, error handling, reconciliation, and rollback processes to maintain data integrity.
• Perform test migrations, production go-live cutovers, and post-migration validation.

4. Reusable Connector Development

• Convert one-time scripts into reusable, configurable, and scalable data connectors.
• Develop standardised frameworks that reduce the time and effort required to onboard new healthcare vendors.
• Improve automation, reliability, and maintainability across data extraction and migration workflows.

5. Customer Coordination and Migration Execution

• Work closely with customers and Onboarding Managers to obtain portal credentials, understand migration requirements, and finalise data scope.
• Coordinate customer sign-offs before production go-live.
• Manage test data loads, migration cutovers, and reconciliation activities.
• Troubleshoot and resolve post-go-live data issues promptly.
• Manage multiple migration projects simultaneously while meeting fixed delivery timelines.
• Communicate effectively with non-technical US-based customers in English.

6. Data Security and Compliance

• Handle sensitive patient information securely throughout extraction, processing, storage, and migration.
• Follow HIPAA requirements and applicable data protection practices.
• Ensure appropriate access controls and secure handling of customer credentials and healthcare data.

Must-Have Skills

• Python
• Agentic AI
• Web Scraping
• Data Engineering and Data Migration
• Browser Automation
• REST API Integration
• SQL and Data Processing

Good-to-Have Skills

• Pandas
• Playwright
• Puppeteer
• Selenium
• Node.js/TypeScript
• Workflow Orchestration: Airflow, Prefect, or Temporal
• AWS and Docker
• Large Language Models (LLMs) for automated field mapping and document parsing

Qualifications and Experience

• 4–7 years of experience in data engineering, system integrations, or data migration.
• Strong programming skills in Python or Node.js/TypeScript, with experience developing production-ready scripts and pipelines.
• Hands-on experience with browser automation tools such as Playwright, Puppeteer, or Selenium, including login workflows, MFA, and dynamic web pages.
• Strong understanding of SQL, data transformation, and data-wrangling techniques.
• Experience integrating with REST APIs at scale, including authentication, pagination, batching, and retry handling.
• Proven ability to develop reusable automation frameworks and configurable connectors rather than relying solely on one-off scripts.
• Strong problem-solving skills and the ability to manage multiple data migration projects under tight deadlines.
• Excellent English communication skills and confidence interacting with non-technical customers in the US.

Preferred Experience

• Exposure to US healthcare data and Electronic Health Record (EHR) or Practice Management (PM) systems, preferably in Physical Therapy (PT), Occupational Therapy (OT), or Speech-Language Pathology (SLP).
• Familiarity with healthcare data standards and terminology, including HL7, FHIR, X12 (837/835), CPT, and ICD-10 codes.