pig demo from Mat Kelcey on Vimeo.
based on a talk i gave at work recently
Posts Tagged ‘hadoop’
a pig screencast
Sunday, January 17th, 2010e11.2 aggregating tweets by time of day
Saturday, October 24th, 2009for v3 lets aggregate by time of the day, should make for an interesting animation
browsing the data there are lots of other lat longs in data, not just iPhone: and ÜT: there are also one tagged with Coppó:, Pre:, etc perhaps should just try to take anything that looks like a lat long.
furthermore lets switch to a bigger dataset again, 4.7e6 tweets from Oct 13 07:00 thru Oct 19 17:00,
i’ve been streaming all my tweets ( as previously discussed ) and been storing them in a directory json_stream
here are the steps…
e11.1 from bash scripts to hadoop
Sunday, October 18th, 2009let’s rewrite v1 using hadoop tooling, code is on github
we’ll run hadoop in non distributed standalone mode. in this mode everything runs in a single jvm so it’s nice and simple to dev against.
step 1: extract the locations strings from the json stream
in v1 it was
bzcat sample.bz2 | ./extract_locations.pl > locations
using the the awesome hadoop streaming interface it’s not too different. this interface allows you to specify any app as the mapper or reducer. the main difference is that it works on directories not just files.
for the mapper we’ll use exactly the same script as before; extract_locations.pl and since there is no reduce component of this job so we use an “identity” script, ie cat, as the reduce phase.
mkdir json_stream bzcat sample.bz2 | gzip - > json_stream/input.gz # hadoop supports gzip out of the bound but not bzip2 :( export HADOOP_STREAMING_JAR=$HADOOP_HOME/contrib/streaming/hadoop-*-streaming.jar hadoop jar $HADOOP_STREAMING_JAR \ -mapper ./extract_locations.pl -reducer /bin/cat \ -input json_stream -output locations
this gives us the locations in a single file locations/part-0000
e10.3 twitter crawl progress
Tuesday, September 29th, 2009since the twitter api is rate limited it’s quite slow to crawl twitter and after a most of a week i’ve still only managed to get info on 8,000 users. i probably should subscribe to get a 20,000 an hr limit instead of the 150 i’m on now. i’ll just let it chug along in the background of my pvr.
while the crawl has been going on i’ve been trying some things on the data to decide what to do with it.
i’ve managed to write a version of pagerank using pig which has been very interesting. (for those who haven’t seen it before pig is a query language that sits on top of hadoop’s mapreduce). my initial feel for pig is that it’s pretty awesome. it was much quicker to write this script than to write the statistically improbable phrases. in fact i’m reinspired to have another crack at the sip stuff using pig. my final result wasn’t great for the performance of hadoop and after some great feedback on the hadoop mailing list i’ve got a number of other things to try including writing my joins in pig.
anyways, here’s my pagerank in pig
e10.0 introducing tgraph
Saturday, September 19th, 2009so e9 sip is on hold for a bit while i kick off e10 tgraph. was looking for another problem to try hadoop with and came across a classic graph one, pagerank. a well understood algorithm like page rank will be a great chance to try pig, the query language that sits on top of hadoop mapreduce.
so we need a graph to work on. my first thoughts were using one of the wikipedia linkage dumps but it feels a bit sterile. instead it’s a good excuse to do a little crawl of the following graph of twitter.
this will also be a chance to try to document a project via a blog. skorks‘ incessant blog rambling has convinced me to give it a go.
first hadoop experiment
Wednesday, September 16th, 2009just finished my first hadoop experiment.
not fantastic results but heaps of of feedback from hadoop mailing group
more results coming soon
