It is quite a pain to setup everything.
Here are some links which helped me significantly:
Use java 1.7 with newest hadoop/hdfs/hive 2.0.0
To create metastore – go to $HIVE_HOME/bin and run:
schematool -initSchema -dbType derby
Derby is java in-memory database. This option will not allow you to run simultaneously Hive metastore (required for Presto) and Hive itself and so consider using mysql for metastore.
Then install presto going through instructions on prestodb.io
So to use presto – you need to shutdown Hive CLI and start metastore service from same directory where your derby is being set with schematool. To start metastore:
hive --service metastore
To check which components of Hive/HDFS are running on machine, run:
To start datanode:
Create 2 aliases in ~/.bashrc to start/stop hadoop/hdfs:
I decided to try hadoop for some huge files processing.
Basically, I’m doing some testing for one of the kaggle problems and needed to process 2-8G files in some way which requires a lot of CPU power.
I decided to try Amazon EMR with their pre-configured hadoop machines.
EMR is actually very good, but I have found for myself to have 1 special cluster running all the time for tests – to check jobs before submitting large files to big clusters to save time on testing on a small inputs beforehand.
Discovered that Hive is not probably the best choice for you if you have a lot of logic or very complex queries to run.
For myself I’m using custom jar clusters only.
How do I make a test before submitting job to big cluster? Connect to master machine and run:
hadoop jar myjar.jar input-files-from-s3
How to check what is the status of jobs you are running?
1. Look at monitoring status on Amazon screens
2. Portforward to Hadoop web interface and look there – recommended way:
ssh -i your-ssh-key.pem -L <br />9100:amazon-public-ip-for-master-node:9100 <br />hadoop@amazon-public-ip-for-master-node
And then – just open http://localhost:9100 in browser to see hadoop web-console.