Sr. Data Engineer - Boston, MA

Job Overview: We are seeking a highly skilled and motivated Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Python, SQL, and AWS, with mandatory experience in PySpark. Knowledge of Databricks is a significant advantage, and an understanding of ECS and Lambda will be essential for success in this role. The Senior Data Engineer will play a crucial role in designing, developing, and maintaining our data infrastructure, ensuring data pipelines are efficient, scalable, and reliable. Responsibilities: Data Pipeline Development: Design and implement scalable data pipelines using Python, SQL, and PySpark to process large volumes of data efficiently. AWS Infrastructure Management: Manage and optimize AWS services such as S3, Redshift, ECS, and Lambda to support data processing and storage needs. Databricks Integration: Leverage Databricks to enhance data processing capabilities, including ETL (Extract, Transform, Load) processes and advanced analytics. Performance Optimization: Continuously monitor and optimize data pipelines to ensure high performance and reliability. Collaboration: Work closely with data scientists, analysts, and other engineers to understand data requirements and deliver solutions that meet business needs. Security and Compliance: Ensure all data processing activities comply with security and data protection regulations. Mandatory Skills: Should have 5+ years of experience as Data Engineer Proficiency in Python, SQL, and AWS services. Extensive experience with PySpark for big data processing. Understanding of AWS ECS and Lambda. Familiarity with data warehousing solutions like Redshift or Snowflake. Knowledge of data modeling, ETL processes, and data lakes. Good to Have Skills: Exposure to Databricks for data processing and analytics. (This is a big plus)