PI2 Data Engineer_42_1/2/3

Job description

 

Position title – Data Engineer

 

Job purpose :

Data Engineer in the IT Advisory Services team to be part of the delivery of Data/Digital based projects for our customers across the globe.

 

Technical Responsibilities:

·  Big data Engineer with 6 to 10 years of IT experience including 4 to 7  years in Big Data and Analytics field, developing E2E Data pipelines to perform batch and Real - Time/Stream analytics on structured and unstructured data.
·  Developing, constructing, testing and maintaining architectures for data lakes, data pipeline, data warehouses and large-scale data processing systems .
·  Expertise in building Data Lake, Data Pipeline on Cloudera or Hortonworks.
·  Experience in Big Data Analytics and design in Hadoop ecosystem using Spark/Spark Streaming, Kafka
·  Solid understanding of data processing patterns, distributed computing and in building applications for real-time and batch analytics.
·  Strong programming skills in design and implementation using Python, Scala and Java
·  Hands on Data Ingress, Egress, Processing using Kafka or Data Bricks Streaming.
·  Good knowledge on NoSQL Data bases such as HBase, Cassandra, Redis and usingSpark streaming for real time stream processing of data into the cluster.
·  Experience with multiple Hadoop file formats like Avro, Parquet, ORC, and JSON etc.

We are looking for the candidates with the following:

·  BE/B Tech/MCA with a sound industry experience of 6to 10 years

 

Mandatory skills:

·  Spark, Scala, Spark  Streaming, Pyspark
·  Kafka, Hbase, Cassandra, Mongodb,
·  Cloudera/Horton works Hadoop Distribution
·  Any RDBMS

Preferred skills:

·  Data Warehousing, ETL, Hive, Flink, AWS/Azure
·  Experience working on CMMI / Agile / SAFE methodologies