We are the biggest datalake in our Company. Besides batch ingestion we do support near real-time data ingestion and distribution using Kafka. We plan to move to a new streaming ingest platform, using container technologies and focus more on being an event based datalake. We have build a generic, metadata driven, data pipeline on top of Nifi. You will be joining a team of experts, where you can learn, share and will have fun.
- Build the solutions required for optimalloading of data from a wide variety of datasources
- Create and maintain, with the team, an optimaldata pipeline architecture
- Assemble large, complex data sets that meetfunctional / non-functional requirements
- Identify, design, and implement internalprocess improvements
- Keep data separated and secure acrossinternational boundaries through multipledata centers
- Work with data and analytics experts to strivefor greater functionality in our data systems
- Work with stakeholders including the Data andDesign teams to assist with data-relatedtechnical issues and support their datainfrastructure needs
- You have experience with Nifi, especially writing custom processors in Groovy,
- You have advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases,
- You have great development skills in Groovy, Scala or Python
- You have 5+ years of experience in a Data and/or Software Engineer role,
- You are communicative person,
- You have very good analitical skills and can solve complex problems,
- You can complete tasks and achieves results in an efficient, timely and high-quality manner.
- Experience with HDFS/Hive and generally with Hadoop
- Spring/springboot based microservices creation,
- Bash and/or Ansible knowledge,
- Experience in the financial services area.
- Experience with stream-processing using Flink or Kafka,
- Experience in building and optimizing big scale data pipelines, architectures and data sets - in both batch and real-time data integration,
- Working knowledge of message queuing, stream processing, and highly scalable BigData data stores,
- Experience working in a DevOps environment - CI/CD, Azure DevOps, Test automation, Dockereld.
Do ciekawego projektu w branży AI/ML poszukujemy osób na stanowisko: Big Data Engineer
Użyj naszego kreatora CV, w którym znajdziesz gotowe szablony
do uzupełnienia, wskazówki oraz przykłady. Stwórz swoje CV teraz.
Pomagamy znaleźć wymarzoną pracę
Szukasz pracy? Znajdź pasującą ofertę i aplikuj!Przejdź do ofert pracy