Data Engineer

  • Location

    New York

  • Sector:

  • Job type:

    Temporary

  • Salary:

    Negotiable

  • Contact:

    James Lesniak

  • Contact email:

    j.lesniak@ioassociates.com

  • Job ref:

    BBBH142720_1721855320

  • Duration:

    6

Position Title: Data Engineer

Location: New York City; Hybrid work arrangement, 2 days per week onsite

Job Type: Contract (Minimum 6 months)

Job Description:
We are looking for a proficient Software/Data Engineer specializing in Directed Acyclic Graphs (DAGs) and dependency graph orchestration. The ideal candidate will contribute significantly to implementing step control functions on Snowflake for a critical risk system development initiative. This role involves the design, development, and maintenance of robust data pipelines and orchestration frameworks to ensure efficient and reliable data processing workflows.

Primary Responsibilities:
- Design and deploy scalable and efficient data pipelines using DAG-driven orchestration tools.
- Develop and manage step control functions and workflows on Snowflake or equivalent cloud platforms.
- Collaborate closely with data scientists, analysts, and stakeholders to translate data processing requirements into technical solutions.
- Optimize and monitor data workflows for performance, reliability, and scalability.
- Implement best practices for data pipeline development, including testing, version control, and documentation.
- Resolve issues related to data workflows and pipelines through effective troubleshooting.
- Stay abreast of advancements in data engineering, cloud platforms, and orchestration tools.

Required Qualifications:
- Bachelor's degree in Computer Science, Engineering, or a related field; Master's degree preferred.
- 5+ years of experience in software/data engineering, focusing on data pipeline development and orchestration.
- Proficiency in DAG-based orchestration tools like Apache Airflow, Prefect, or Luigi.
- Strong hands-on experience with Snowflake, including Snowpipe, tasks, and streams.
- Expertise in cloud platforms such as AWS, GCP, or Azure.
- Proficiency in SQL and programming languages such as Python, Java, or Scala.
- Deep understanding of ETL/ELT processes and data integration techniques.
- Experience with version control systems (e.g., Git) and CI/CD pipelines.
- Excellent problem-solving abilities and attention to detail.
- Strong communication and collaboration skills.

Preferred Qualifications:
- Experience with Risk and trading/Realtime systems is advantageous.
- Familiarity with additional data processing frameworks/tools like Apache Spark, dbt, or Kafka.
- Knowledge of containerization and orchestration tools (e.g., Docker, Kubernetes).
- Understanding of data warehousing concepts and big data technologies.
- Certification in cloud platforms (e.g., AWS Certified Solutions Architect, Google Professional Data Engineer).

Technology Stack:
- DAG Orchestration: Apache Airflow, Prefect, Luigi
- Cloud Platforms: Snowflake, AWS, GCP, Azure
- Programming Languages: Python, Java, Scala
- Data Processing: SQL, Apache Spark, dbt
- Containerization:Docker, Kubernetes
- Version Control: Git
- CI/CD: Jenkins, GitLab CI/CD, CircleCI

Back job search