Sqoop is an engine which facilitates the import and export of data from traditional data stores such as Oracle and MySQL. It easily pulls data out of HDFS and maps to tables in an SQL store. Sqoop is pre-integrated into Oozie allowing job output to be uploaded into a data store upon job completion.
Oozie is fast becoming the engine of choice for managing Hadoop data workflows. Oozie is fully integrated into the Hadoop eco system and makes the process of managing very complex sequences of data processing jobs with inter-dependencies a manageable task. We have extensive experience of building out workflows using Oozie and are able to develop customisations, such as new actions and monitoring tools to meet your business needs.
What is Flume?
Flume is an agent based platform for transporting data from source to a sink. Sources can be anything from flat files to data transported over a network socket such as syslog packets. The Flume architecture supports both scalability and reliability of data ingestion. Reliability can be used to ensure message delivery and retransmission on the event of failure, furthermore it can be implemented to provide very high degrees of Fault tolerance to node and communication failure.
We have implemented Flume for clients and are happy to help determine an appropriate solution for your needs. This includes architecture, design and development of custom plugins to read data from your custom sources such as JMS or MQ.
Great Big Data is a specialist UK consultancy primarily focused on Cloud based data processing platforms, in particular Hadoop and its related toolsets. We provide an end to end consultancy from project inception, hardware specification to implementation and go live. Our consultants are familiar with all aspects of integration to your data sources and have worked extensively with Flume to provide scalable and reliable data ingestion. We also provide outbound exports to databases using Sqoop and/or custom software against HDFS to move data out of Hadoop into Enterprise Data Warehouses for example.
We have particular expertise in Telecoms, however we are easily able to transfer our expertise to your particular domain.