Big Data Engineer (Live Anywhere)
Big Data Engineer
Join a dynamic team optimizing a modern, Lakehouse-focused analytics platform. This position blends strategic business insight with hands-on technical execution. You will collaborate with cross-functional partners to gather requirements, design pragmatic architectures, and deploy scalable streaming and ingestion pipelines end-to-end.
Responsibilities:
-
Design, build, and maintain production ETL/ELT pipelines using Databricks, Apache Spark (PySpark/Spark SQL), and Delta Lake.
-
Implement Delta Live Tables (DLT) for streaming/batch workflows and set up data governance via Unity Catalog.
-
Integrate high-volume event streams using Apache Kafka and leverage AWS cloud infrastructure (S3, IAM).
-
Cleanse, transform, and curate large structured and semi-structured datasets to support BI reporting and machine learning needs.
-
Enforce data quality, operational observability, and engineering best practices in an Agile environment.
Qualifications:
-
5+ years in Data Engineering; strong proficiency in SQL, Spark, and Databricks.
-
Hands-on experience with Delta Lake, Delta Live Tables, Unity Catalog, Kafka, and AWS analytics services.
-
Background in Agile delivery and translating complex requirements into reliable models.
-
Preferred: Exposure to Snowflake, NoSQL databases (e.g., Cassandra), or enterprise messaging systems.
Location & Work Schedule:
This is a 6+ month auto-renewing contract. Open to 100% remote work for non-local candidates willing to work EST hours. Atlanta-based candidates will work a hybrid schedule (3 days in-office).
Through Visionaire Partners, full-time W2 contractors gain access to a full benefits package (for contractors, spouses, domestic partners, and dependents), including a 401(k) with up to 4% match, medical, dental, vision, life insurance, short/long-term disability, critical illness, hospital indemnity, accident coverage, and Medical/Dependent Care FSAs.
#LI-Hybrid