Software Engineer - ETL/Data Engineering

Role Overview

We are seeking a Software Engineer to support an enterprise IT program. The selected candidate will be joining an existing team supporting extract, transform, load (ETL) workflows for intelligence operations.

Position Summary

The Sponsor supports analysts through the provision of large datasets, methodologies, and data visualizations to address pressing intelligence questions. The Sponsor requires support to ingest, clean, store, and analyze data. The Sponsor needs experienced support in extract, transform, load (ETL) workflows. The work includes engaging regularly with data scientists, analysts, and managers.

Key Responsibilities

• Load large datasets into the Sponsor's on-premises and Cloud environments
• Develop and maintain ingestion algorithms and schemas for large datasets
• Analyze new large-volume datasets to optimize the data ingest processes
• Support the creation of Apache NiFi schemas for new data loads
• Develop software tools that efficiently preprocess, modify, aggregate, load, index, and archive large data collections into clusters in near real-time
• Ensure proper access controls are implemented
• Generate metrics to track data ingest statistics to maintain data integrity and provenance
• Document the data-flows according to standards set by Sponsor

Mandatory Skills & Requirements

Technical Requirements

1. Education/Experience - Demonstrated professional experience in Computer Science, Computer Engineering, Systems Engineering, or closely related discipline
2. AWS Cloud Services - Demonstrated professional experience with AWS cloud services, including long-term storage options, and cloud-based database services
3. Databricks - Demonstrated experience working with Databricks
4. SQL Database Management - Demonstrated experience understanding SQL database structures and mapping them between different SQL databases
5. Apache NiFi - Demonstrated professional experience working with Apache NiFi
6. Big Data Platforms - Demonstrated professional experience working with large data and high performance compute clusters such as Hadoop or similar
7. API Development - Demonstrated experience with API development techniques
8. ETL Processes - Demonstrated experience developing and deploying ETL processes for large data sets
9. Scripting - Demonstrated experience creating operating system level scripts to perform ETL operations on SQL databases
10. Version Control - Demonstrated professional experience with version control systems, preferably Git
11. Software Testing - Demonstrated experience testing the development of software solutions for the extraction, transformation, and loading of data using the most efficient languages for the task such as NiFi, Python, and SQL
12. Parallel Processing - Demonstrated experience implementing multiprocessing data-flows to parallelize ingest operations

Desired Skills & Qualifications

1. Sponsor Environment - Demonstrated experience with the Sponsor's data environment
2. Collaboration Skills - Demonstrated experience exhibiting strong coordination and collaboration skills
3. Full-Stack Integration - Demonstrated experience working with full-stack developers to deploy applications that leverage large data sets
4. Technical Communication - Demonstrated experience communicating technical concepts to non-technical audiences