Big Data Engineer

Job Description

• 5+ year relevant experience in data management ideally in high volume and real time data processing Essential Skills * Experiencewith Python Spark / Spark Streaming for ETL/ELT development * Experiencewith data modelling and data query tools like Hive SparkSQL and ANSI SQL * Experience with relational databases such as MS SQL Server No SQL databases. Like Mongo Cassandra * Apply structures like Parquet Avro or any column family schemas on the data to speed up analytics * Understanding of agile development methods and leading software development teams in an Agile/Lean environment * Strong communication skills and the ability to effectively discuss at the appropriate level of technical detail Preferred Skills * Experience with Azure Big Data Stack - HDInsight Data Factory Events Hub Service Bus * Build test release and monitor the HTTP/RESTful web service APIs with CI/CD practice * Experience in designing and developing Distributed Application * Experience with source code controlled environments like GIT Role : - Configure data mappings & transformations to orchestrate data integration & validation - Implement ETL process for integration of data from disparate sources leveraging Big Data Stack - Design build & test data analytics processes using spark & SQL - Identification analysis & resolution of production & development bugs -Work in Agile / SCRUM mode forimplementation -Test driven development - Provide subject matter expertise - Peer reviews of work developed by other data engineers within the team Please refer to the Job description above

Job Summary