Operation Engineer - Big Data Technologies - Hadoop/python

Job Description

• Responsibilities : - Managing Production Cluster of 1500 nodes + clusters with 500+ nodes. - Providing Hadoop as service to customers of different teams. - System Engineer managing a 30 petabyte Hadoop cluster with responsibilities including - Complete infrastructure maintenance by oncall support and development. - System automations - Python/Bash and Ansible. - Performance and availability management to keep up SLAs. - Reparing and performing the installation of Hadoop software - Adding and configuring the nodes - Monitoring Cluster Health and Troubleshooting - Management of the metadata databases - Manage and review Backups - Moving data efficiently between clusters using Distributed Copy - Restore in case of project-specific requirements - Rebalancing of the HDFS - Regularly scheduling statistics runs and automating them - Quota administration notification in case of space problems - Moving/shifting nodes/roles/services to other nodes - Performing cluster/node migrations (new hardware and/or new OS version and/or new Hadoop version) with OS or Hadoop tools or manually - Processing the events of Hadoop logs taking measures correcting errors and involving the relevant teams if necessary - Tuning by making changes to settings (e.g. HBase Hive etc. Required skills :- Operation Engineer or Lead with 3-7 years of experience in Big Data Technologies- Exp with Hadoop ecosystem such as HDFS HBase Hive Yarn Spark Zookeeper Kafka Storm- 3+ years- experience in a systems administration engineering or operations.- Experience configuring managing or troubleshooting systems of scale (cloud virtualization distributed networks colocation..)- Linux Operating Systems skills- Bachelors Degree or Equivalent work experience- Experience in other database like MySql Redis Hazelcast- Exp in any of the scripting languages Python Perl Bash- Linux admin with exp in distributed cluster operations- Experience in managing Hadoop clusters- Attitude towards automation. Experience in workflow and operations automation- Strong analytical skills- Troubleshooting- Problem solving skills- Contribution in open source projects is a plusAny certificate is a must (Linux Certification Big Data Certification Certified in Red Hat System Administration Apache Hadoop Certification) .

Job Summary