Senior Data Engineers - Remote - Urgent Need

Position : Senior Data Engineers - Remote Location : San Diego, CA Duration : 06 Months Total Hours/week : 40.00 1st Shift Client : Medical Devices Company Job Category : Professional Level of Experience : Senior Level Employment Type : Contract on W2 (Need US Citizens or GC Holders or GC EAD or OPT or EAD or CPT) Job Description Temp to perm • As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization’s data strategy using software engineering best practices. • Beyond technical expertise, you will also serve as a change leader, guiding teams through adopting new tools, technologies, and workflows to improve data management and processing. • This position requires extensive hands-on data system design and coding experience, as well as the development of modern data pipelines (AWS Step functions, Prefect, Airflow, Luigi, Python, Spark, SQL) and associated code in AWS. • You will work closely with stakeholders across the business to understand their data needs, ensure scalability, and foster a culture of innovation and learning within the data engineering team and beyond. Key Responsibilities • Be responsible for the overall architecture of a specific module within a product (e.g., Data-ingestion, near-real-time-data-processor, etc.), perform design and assist implementation considering system characteristics to produce optimal performance, reliability and maintainability. • Provide technical guidance to team members, ensuring they are working towards the product's architectural goals. • Create and manage RFCs (Request for Comments) and ADRs (Architecture Decision Records), Design notes and technical documentation for your module, following the architecture governance processes. • Lead a team of data engineers, providing mentorship, setting priorities, and ensuring alignment with business goals. • Architect, design, and build scalable data pipelines for processing large volumes of structured and unstructured data from various sources. • Collaborate with software engineers, architects, and product teams to design and implement systems that enable real-time and batch data processing at scale. • Be the go-to person for PySpark-based solutions, ensuring optimal performance and reliability for distributed data processing. • Ensure that data engineering systems adhere to the best data security, privacy, and governance practices in line with industry standards. • Perform code reviews for the product, ensuring adherence to company coding standards and best practices. • Develop and implement monitoring and alerting systems to ensure timely detection and resolution of data pipeline failures and performance bottlenecks. • Act as a champion for new technologies, helping ease transitions and addressing concerns or resistance from team members. Ideal Candidate • Experience leading a data engineering team with a strong focus on software engineering principles such as KISS, DRY, YAGNI etc. • Candidate MUST have experience in owning large, complex system architecture and hands-on experience designing and implementing data pipelines across large-scale systems. • Experience implementing and optimizing data pipelines with AWS is a must. • Production delivery experience in Cloud-based PaaS Big Data related technologies (EMR, Snowflake, Data bricks etc.) • Experienced in multiple Cloud PaaS persistence technologies, and in-depth knowledge of cloud- based ETL offerings and orchestration technologies (AWS Step Function, Airflow etc.) • Experienced in stream-based and batch processing, applying modern technologies • Working experience with distributed file systems (S3, HDFC, ADLS), table formats (HUDI, Iceberg), and various open file formats (JSON, Parquet, Csv, etc.) • Strong programming experience in PySpark, SQL, Python, etc. • Database design skills including normalization/de-normalization and data warehouse design • Knowledge and understanding of relevant legal and regulatory requirements, such as SOX, PCI, HIPAA, Data Protection • Experience in the healthcare industry, a plus • A collaborative and informative mentality is a must! Toolset • AWS, preferably AWS certified Data Engineer and AWS certified Solutions Architect. • Proficiency in at least one programming language C#, GoLang, JavaScript or ReactJs • Spark / Python / SQL • Snowflake/ Databricks / Synapse / MS SQL Server • ETL / Orchestration Tools (Step Function, DBT etc.) • ML / Notebooks Education And Experience Required • Bachelors or Master’s in Computer Science, Information Systems, or an engineering field or relevant experience. • 10+ years of related experience in developing data solutions and data movement. This role can be REMOTE