Quantexa Data Engineer

Role Overview

We are seeking a talented and experienced Data Engineer with strong expertise in Quantexa, Hadoop, Scala, Apache Spark, Elasticsearch, OpenShift Container Platform (OCP), and DevOps practices .

The successful candidate will be responsible for designing, developing, optimizing, and maintaining scalable big data solutions using Apache Spark, Scala, Hadoop, and Elasticsearch . The role involves collaborating with cross-functional teams to build efficient data processing pipelines and search applications.

Knowledge and experience in the Compliance / AML domain will be an added advantage.

Key Responsibilities

• Design, develop, and implement Spark/Scala applications and data processing pipelines for large volumes of structured and unstructured data.
• Implement data transformation, aggregation, enrichment, and computation processes to support analytics and machine learning initiatives.
• Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions.
• Integrate Elasticsearch with Spark for efficient data indexing, querying, and retrieval.
• Implement transformations and aggregations using Spark RDDs, DataFrames, Datasets, and Spark SQL .
• Develop scalable, reliable, and fault-tolerant Spark applications following industry best practices and coding standards.
• Optimize and tune Spark jobs and Elasticsearch queries to improve performance, scalability, and resource utilization.
• Monitor job performance, identify bottlenecks, troubleshoot issues, and implement appropriate optimizations.
• Troubleshoot and resolve issues related to data processing, data quality, Spark performance, and Elasticsearch integration .
• Ensure data quality, consistency, accuracy, and integrity throughout the data processing lifecycle.
• Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration technologies.
• Optimize data engineering workflows for containerized environments and efficient resource utilization.
• Collaborate with DevOps teams to streamline deployments and implement CI/CD pipelines .
• Implement data governance, data lineage, and metadata management practices to ensure data accuracy, traceability, and compliance.
• Implement monitoring and logging mechanisms to ensure the health, availability, and performance of data infrastructure.
• Monitor and optimize end-to-end data pipeline performance and implement required enhancements.
• Document data engineering processes, workflows, architecture, and infrastructure configurations for knowledge sharing and future reference.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, Information Technology, or a related field .
• Quantexa Certified Data Engineer / Data Architect with hands-on experience and strong proficiency in the Quantexa platform.
• Proven experience as a Data Engineer working with Hadoop, Spark, and large-scale data processing technologies.
• Strong proficiency in Scala and familiarity with functional programming concepts.
• In-depth understanding of Apache Spark architecture, RDDs, DataFrames, Datasets, and Spark SQL .
• Strong expertise in Hadoop ecosystem technologies , including HDFS, Hive, Pig, and related tools.
• Hands-on experience with Elasticsearch , including data indexing, search applications, data modeling, indexing strategies, and query optimization.
• Experience with OpenShift Container Platform (OCP) and Kubernetes-based container orchestration.
• Strong programming skills in Scala, Python, Java, and/or Spark .
• Good understanding of DevOps practices, CI/CD pipelines, and infrastructure automation .
• Experience with tools such as Docker, Jenkins, Ansible, and Bitbucket .
• Experience with distributed computing, parallel processing, and large-scale datasets .
• Strong experience in performance tuning and optimization of Spark applications and Elasticsearch queries.
• Experience with Git and collaborative software development workflows.
• Strong analytical and problem-solving skills with the ability to troubleshoot complex technical issues.
• Excellent communication and collaboration skills with the ability to work effectively with cross-functional teams.
• Experience with Grafana, Prometheus, and Splunk will be an added advantage.
• Exposure to cloud platforms such as AWS, Azure, or GCP and their data services will be a plus.
• Knowledge or experience in the Compliance / AML domain will be an added advantage.