Impetus is hiring for Sr. Data Engineer role for Gurgaon/Bangalore location. We are looking that candidate should have strong experience in Bigdata, Hadoop, Hive, HDFS, Spark, Pyspark and in Python. And should have experience in any sort of cloud like AWS, GCP, Azure. We are looking candidate who can join us in 0-30 days at max. If you are interested & meeting above criteria, please do share your resume at vaishali.tyagi@impetus.com.
About the Role
The Sr. Data Engineer will be responsible for designing and implementing solutions for problems arising out of large-scale data processing, ensuring end-to-end ownership of all tasks being aligned including development, testing, deployment, and support.
Roles & Responsibilities
Experience in working on Spark framework, good understanding of core concepts, optimizations, and best practices.
Person should know RDDS, difference between RDDS & Data set & Data frames & fault tolerance & how it works in RDD.
Should know DAGS & transformations in spark & should be aware of different types of transformations.
Person should know different drivers & executor & task in spark.
Person should have good experience or some knowledge into any cloud platform – GCP, AWS, Azure, Databrick, Snowflakes.
Candidate should have strong experience in data warehouses, data lakes, Analytics/Reporting.
Should know complete lifecycle - ingestion, transformation, serving/consumption.
Good hands-on experience in writing code in PySpark, should understand design principles and OOPS & strong in Python or Java or Scala.
Good experience in writing complex queries to derive business critical insights.
Hands-on experience on Stream data processing.
Understanding of Data Lake vs Data Warehousing concept.
Knowledge on Machine learning would be an added advantage.
Experience in NoSQL Technologies – MongoDB, Dynamo DB.
Good understanding of test-driven development.
Flexible to learn new technologies.
Design and implement solutions for problems arising out of large-scale data processing.
Attend/drive various architectural, design and status calls with multiple stakeholders.
Ensure end-to-end ownership of all tasks being aligned including development, testing, deployment and support.
Design, build & maintain efficient, reusable & reliable code.
Test implementation, troubleshoot & correct problems.
Capable of working as an individual contributor and within team too.
Ensure high quality software development with complete documentation and traceability.
Fulfil organizational responsibilities (sharing knowledge & experience with other teams/groups).
Required Skills
Experience in working on Spark framework.
Good understanding of core concepts, optimizations, and best practices.
Strong experience in data warehouses, data lakes, Analytics/Reporting.
Good hands-on experience in writing code in PySpark.
Strong in Python or Java or Scala.