Senior Data Engineer
Emploi Inde
Titre du poste : Senior Data Engineer
Entreprise : Tredence
Description du poste : Job Description:Job Responsibilities:
- Design and implement highly scalable, distributed data pipelines using Spark and Delta
Lake.
- Collaborate with the team to integrate Spark workflows into orchestration tools like
Airflow.
- Implement best practices for job scheduling, resource allocation, and failure recovery in
Apache Spark.
- Conduct performance benchmarking and recommend improvements for Apache Spark
tuning based on workload patterns.
- Monitor cluster health and troubleshoot issues to maintain high availability and stability.
- Work closely with the data engineering team to define data models and transformations
using DBT Core and Spark SQL.Skills:Required Skills:
- hands-on experience in building Big Data enterprise grade data platforms.
- Proven track record of optimizing Apache Spark clusters for large-scale workloads and
complex job patterns.
- Strong proficiency in Apache Spark Core, Spark SQL, and PySpark.
- Experience with Delta Lake for managing data lakes and supporting ACID transactions.
- Familiarity with Debezium with Apache Kafka, Airflow, and DBT Core is a plus.
- Strong understanding of distributed systems, memory management, and resource
allocation strategies.
- Solid experience with on-premises infrastructure and ability to collaborate with system
administrators for cluster setup and tuning.
- Problem-solving mindset and ability to work independently.
About Company:Welcome to Tredence! Since our founding in 2013, we have been dedicated to transforming data into actionable insights, helping over 50 Fortune 500 clients win in their respective industries. With our headquarters in San Jose and a presence in 5 countries, our mission is to be the world’s most indispensable analytics partner. At Tredence, we blend deep domain expertise with advanced AI and data science to drive unparalleled business value. We are excited to have you join us on this innovative journey.
Salaire attendu :
Localisation : Bangalore, Karnataka
Date du poste : Fri, 06 Jun 2025 06:46:32 GMT
Postulez dès maintenant !
