Data Engineer

Our rapidly growing client is seeking a skilled Data Engineer to lead the design and development of their data pipelines and infrastructure. This is a unique opportunity for someone who is passionate about building robust data systems, working with cutting-edge technologies, and solving complex data challenges.

The Data Engineer will take ownership of an existing production system that collects and transforms data from client file uploads, several third-party web APIs, and a few internal databases. The system is hosted in Azure and currently uses Databricks/Spark to automate the data pipeline, ultimately feeding into an Azure Cosmos DB application database. No system is perfect, and the Data Engineer will be expected to review the design of the existing system and recommend improvements.

Key Responsibilities:

• Design and implement scalable data pipelines using Azure data services such as Azure Data Factory, Azure Databricks, and Azure Data Lake Store
• Develop and maintain efficient data processing workflows using Databricks/Spark
• Collaborate regularly with software engineers, data scientists, and consumers of the data
• Establish data governance and data quality best practices for our data lake and data warehouse.
• Design and maintain data architecture models and data flow diagrams.
• Manage the deployment and operation of data pipelines and ensure reliability, availability, and scalability.

Requirements:

• Bachelor's degree in computer science, data science, or a related field
• At least 5 years of experience as a Data Engineer or a related role
• Strong experience with Azure data services (e.g., Azure Data Factory, Azure Databricks, Azure Data Lake Store, etc.)
• In-depth knowledge of Databricks/Spark and the ability to develop efficient data processing workflows.
• Solid understanding of data governance, data quality, and data management best practices
• Experience designing and implementing data architectures and data flow diagrams.
• Strong programming skills in Python, Scala, or Java
• Experience with cloud-based data warehousing technologies

Our client is a dynamic organization with a culture of innovation and a passion for technology. They offer a competitive salary and a comprehensive benefits package, including health insurance, retirement plans, and paid time off. If you're excited about the opportunity to lead data engineering initiatives, work with cutting-edge technologies, and solve complex data challenges, please submit your resume.