Data Engineer
-
Location
New York
-
Sector:
-
Job type:
-
Salary:
Negotiable
-
Contact:
James Lesniak
-
Contact email:
j.lesniak@ioassociates.com
-
Job ref:
BBBH142720_1721855320
-
Duration:
6
Position Title: Data Engineer
Location: New York City; Hybrid work arrangement, 2 days per week onsite
Job Type: Contract (Minimum 6 months)
Job Description:
We are looking for a proficient Software/Data Engineer specializing in Directed Acyclic Graphs (DAGs) and dependency graph orchestration. The ideal candidate will contribute significantly to implementing step control functions on Snowflake for a critical risk system development initiative. This role involves the design, development, and maintenance of robust data pipelines and orchestration frameworks to ensure efficient and reliable data processing workflows.
Primary Responsibilities:
- Design and deploy scalable and efficient data pipelines using DAG-driven orchestration tools.
- Develop and manage step control functions and workflows on Snowflake or equivalent cloud platforms.
- Collaborate closely with data scientists, analysts, and stakeholders to translate data processing requirements into technical solutions.
- Optimize and monitor data workflows for performance, reliability, and scalability.
- Implement best practices for data pipeline development, including testing, version control, and documentation.
- Resolve issues related to data workflows and pipelines through effective troubleshooting.
- Stay abreast of advancements in data engineering, cloud platforms, and orchestration tools.
Required Qualifications:
- Bachelor's degree in Computer Science, Engineering, or a related field; Master's degree preferred.
- 5+ years of experience in software/data engineering, focusing on data pipeline development and orchestration.
- Proficiency in DAG-based orchestration tools like Apache Airflow, Prefect, or Luigi.
- Strong hands-on experience with Snowflake, including Snowpipe, tasks, and streams.
- Expertise in cloud platforms such as AWS, GCP, or Azure.
- Proficiency in SQL and programming languages such as Python, Java, or Scala.
- Deep understanding of ETL/ELT processes and data integration techniques.
- Experience with version control systems (e.g., Git) and CI/CD pipelines.
- Excellent problem-solving abilities and attention to detail.
- Strong communication and collaboration skills.
Preferred Qualifications:
- Experience with Risk and trading/Realtime systems is advantageous.
- Familiarity with additional data processing frameworks/tools like Apache Spark, dbt, or Kafka.
- Knowledge of containerization and orchestration tools (e.g., Docker, Kubernetes).
- Understanding of data warehousing concepts and big data technologies.
- Certification in cloud platforms (e.g., AWS Certified Solutions Architect, Google Professional Data Engineer).
Technology Stack:
- DAG Orchestration: Apache Airflow, Prefect, Luigi
- Cloud Platforms: Snowflake, AWS, GCP, Azure
- Programming Languages: Python, Java, Scala
- Data Processing: SQL, Apache Spark, dbt
- Containerization:Docker, Kubernetes
- Version Control: Git
- CI/CD: Jenkins, GitLab CI/CD, CircleCI
![](https://counter.adcourier.com/amxlc25pYWsuMDUxNDkuMTI1NzVAcGFub3JhbWljLmFwbGl0cmFrLmNvbQ.gif)