We are looking for a Big Data Engineer that will work on the collecting, storing and processing of huge sets of data. The primary focus will be on choosing and developing the optimal solutions to use for these purposes.
Responsibilities:
- Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
- Implementing ETL processes
- Monitoring performance and advising any necessary infrastructure changes
- Defining data retention policies
Skills and Qualifications:
- Experience with C#/Java object oriented design and programming.
- Proficient understanding of distributed computing principles
- Ability to solve any ongoing issues with operating the Frameworks and systems.
- Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming.
- Experience with integration of data from multiple data sources
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Knowledge of various ETL techniques and frameworks, such as Flume
- Experience with various messaging systems, such as Kafka or RabbitMQ
- Good understanding of Lambda Architecture, along with its advantages and drawbacks