Our client who is a leader in the Financial Services space has an exciting opportunity for a Senior AWS Data Engineer.
Minimum Qualifications
- Bachelor’s Degree in Computer Science, Information Technology, or other relevant fields.
- Has experience in any of the following AWS Athena, AWS Glue, Pyspark, AWS DynamoDB, AWS Redshift, AWS Lambda and AWS Step Functions.
- Proficient in SQL, Python and PySpark
- Proficient in utilizing a cloud platform and services
- Knowledge of software engineering best practices across the development lifecycle, including agile methodologies, coding standards, code reviews, source management, build processes, testing, devops and operations.
Additional Experience
- A successful history of manipulating, processing and extracting value from large, disconnected data sets.
- Experience supporting and working with cross-functional teams in a dynamic environment.
- Experience in a Data Engineering roles.
- Experience with the following is a must: AWS Glue, PySpark, SQL
- Experience with data pipeline and workflow management tools like AWS StepFunctions and Control-M would be beneficial but not required.
- Experience with AWS cloud services: EC2, EMR, RDS, DynamoDB would be beneficial but not required.
Experience
- Advanced Data Engineering knowledge and experience working with modern data practices.
- Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets.
- Experience working with distributed systems as it pertains to data storage and computing.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, meta data, dependency, and workload management.
- Proficiency in SQL with expertise in writing and optimizing complex queries
Responsibilities
- Design and implement large scale enterprise data solutions by using a combination of the of the following technologies – AWS Glue, AWS Step-functions, AWS RedShift, AWS Lambda, AWS Athena, AWS Lakeformation, Spark, Python.
- Analyze, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS and 3rd party services.
- Design and build production data pipelines from ingestion to integration within a big data architecture, using PySpark, Python and SQL.
- Design, implement and support an analytical data infrastructure providing ad-hoc access to large datasets and computing power.
- Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using SQL and AWS big data technologies.
- Continual research of the latest big data and visualization technologies to provide new capabilities and increase efficiency.
- Collaborate with other tech teams to implement advanced analytics algorithms that exploit our rich datasets for statistical analysis, prediction, clustering and machine learning.
- Help continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers.
Desired Skills:
- Systems Analysis
- Complex Problem Solving
- Programming/configuration
- Critical Thinking
- Time Management