so rather than write no blog post here’s mats-list-of-cool-machine-learning-books in the order i think you should consider reading them…

1) “programming collective intelligence” by toby segaran

if you know nothing about machine learning and haven’t done maths since high school then this is the book for you. it’s a fantastically accesible introduction to the field. includes almost no theory and explains algorithms using actual python implementations.

2) “data mining” by witten and frank

this book covers quite a bit more than programming c.i. while still being extremely practical (ie very few formula). about a fifth of the book is dedicated to weka, a machine learning workbench which was written by the authors. apart from the weka section this book has no code. i made a little screencast on weka awhile back if you’re after a summary.

3) “introduction to data mining” by tan, steinbach and kumar

covers almost the same material as the witten/frank text but delves a little bit deeper and with more rigour. includes no code (none of the books do from now on) with algorithms described by formula. has a number of appendices on linear algebra, probability, statistics etc so that you can read up if you’re a bit rusty or new to the fields (the witten/frank text lack these). some people might argue having both of these books is a waste since they cover so much of the same ground but i’ve always found multiple explanations from different authors to be a great way to help understand a topic. i read the witten/frank text first and am glad i did but if i could only keep one i’d keep this one.

intermission

at this point you’ve probably got enough mental firepower to handle some of the uni level machine learning course notes that are floating about online.

if you’re keen to get a better foundation of the maths side of things it’d be worth working through andrew ng’s lecture series on machine learning. (20 hours of a second year stanford course on machine learning)

i also found andrew moore’s lecture slides really great. (they do though require a reasonable understanding of the basics)

4) “foundations of statistical natural language processing” by manning and schutze

not a machine learning book as such but great for learning to deal with one of the most common types of data around; text. since most of machine learning theory is about maths (ie numbers) this is awesome in helping to understanding how to deal with text in a mathematical context.

5) “introduction to machine learning” by ethem alpaydin

covers generally the same sort of topics as the data mining books but with much more rigour and theory (derivations, proofs, etc). i think this is a good thing though since understanding how things work at a low level gives you the ability to tweak and modify as required. loads more formulas but again with appendixs that introduce the basics in enough detail to get by.

6) “all of statistics” by larry wasserman

by this stage you’ll probably have an appreciation of how important statistics is for this domain and it might be worth foccussing on it for a bit. personally i found this book to be a great read and though i’ve only read certain sections in depth i’m looking forward to when i get a chance to work through it cover to cover

7) “the elements of statistical learning” by hastie, tibshirani and friedman.

with a bit more stats under your belt you might have a chance of getting through this one; the most complex of the lot. this book is absolutely beautifully presented and now that it’s FREE to download you’ve got no reason not to have a crack at it. a remarkable piece of work and one i’ve yet to get through fully cover to cover, it’s quite hardcore and right on the border of my level of understanding ( which makes it perfect for me :P )

ps. books i haven’t read that are in the mail

“machine learning” by tom mitchell

have been wanting to read this one for awhile, i’m a big fan of tom mitchell, but couldn’t justify the cost however just found out the other day the paperback is a third of the price of the hardback i was looking at!! the book’s in the mail

“pattern recognition and machine learning” by chris bishop

all of a sudden seemed like everyone was reading this but me so it was time to jump on the bandwagon

brutally short intro to weka from Mat Kelcey on Vimeo.

so here’s a little experiment

1. get the people i follow on twitter
2. look up the words that “represent” them according to the infochimps word bag api
3. build a similiarity matrix based on the common use of those terms
4. plot the connectivity for the top 30 or so pairings

it’s basically partitioned into three groups…

1. veztek (my boss john) and smcinnes (steve from the lonely planet community team) in the top right
2. a big clump of nosqlness with mongodb – hbase – jpatanooga – kevinweil in the bottom left
3. everyone else

an interesting enough result given the time taken; the codes on github

#worldcup tweet viz1 from Mat Kelcey on Vimeo.

here’s a simple visualisation of the use of official country codes (eg #aus) in a week’s worth of tweets from the search stream for #worldcup.

rate is about 2hours of tweets per sec. orb size denotes relative frequency of that country code. edges denote that those two countries feature a lot in the same tweets. movement is based on gravitational like attraction along edges.

the quiet period at about 0:17 is a twitter outage :)

here’s the original processing applet version with a bit more discussion

> x = c(3,1,4,1,5,9,2,6,5,3,5,8)
> ra_x = filter(x, rep(1,3)/3)
> ra_x
Time Series:
Start = 1
End = 12
Frequency = 1
 [1]       NA 2.666667 2.000000 3.333333 5.000000 5.333333 5.666667 4.333333
 [9] 4.666667 4.333333 5.333333       NA

anyways, here’s the first few days analysis based the use of official country tags (eg #aus) in the search stream for #worldcup.

tomorrow i might look in more detail at one of the games, wondering how many variants of ‘goooooooal’ i’ll find :D

i’m finding streaming related algorithms quite interesting lately and think are the way forward in terms of dealing with large amounts of constant data. it’s just not feasible to use algorithms that expect you to have all the data at any given time; it forces you to reprocess all the data you’ve ever seen as you get new examples. my thinking is the best solutions are the ones that build models of the data and fold in new examples in batches. anyways, i’m getting off topic already.

tf/icf as presented in the paper is a variant on the classic tf/idf for term weighting but instead of requiring all weighting in all docs to be recalculated as a new document comes along (as tf/idf strictly does) it instead just approximates based on what has been seen before.

so how does it do? actually quite well, here’s my experimental breakdown

brutally short demo of ggobi from Mat Kelcey on Vimeo.

note: non embedded version has higher res at full screen

a sensible place to start when looking for trends is to consider single terms but what if though we ended up with three equally trending terms ‘happy’, ‘new’ and ‘year’? it’s pretty obvious that the actual trend is ‘happy new year’ but what is the best way to express this as a single trend in an algorithmic sense?

one approach i’ve been playing with is to collect unigrams, bigrams and trigrams (1,2,3 term ‘phrases’) and consider the cases where the terms overlap. basically if ‘happy new year’ is trending then, in some sense, we can ignore trends for ‘happy new’, ‘new year’, ‘happy’, ‘new’ and ‘year’. but does this result in to many false positives? would we miss ‘happy’ as a trend if lots of people were chirpy about the change of year (as they usually are, on new years eve)

rather than outright ignore we could somehow reduce the weighting by removing the double counting.

eg if we had 3 trends; (free beer,11), (free,12) & (beer,25)
we can take 11 (from the 2gram) off both 1grams to give (free beer,11), (free,1) & (beer,14)
showing that ‘beer’, outside of the phrase ‘free beer’, is perhaps a trend in itself (as it should be)

this feels like it might work but would be non trivial (read: fun) to implement

another slightly different problem is around the handling of retweeting. my experiments have shown a huge amount of the ‘trends’ found are related to retweets, which is fine in itself, but it gives quite strange trends since the retweeted portion of the text is usually quite long.

for example; say lots of people are retweeting something and, as some people do, are adding various bits and pieces at the beginning and end; eg ‘RT @bob omg i just found a peanut’ or ‘omg i just found a peanut; via @bob lucky him!!’

if we’re considering bigrams (which i am in my current implementation) we end up with an odd selection of trends such as ‘just found’, ‘a peanut’, ‘omg i’, ‘found a’, ‘i just’ and in these cases it’d be great to be able to just stitch them together into the common retweeted element ‘omg i just found a peanut’.

we could ’solve’ this problem by not just considering 1,2 and 3 grams but considering _all_ possible n-grams for each tweet and employing the technique we spoke of above of reducing the counts. it’d almost be feasible, since tweets are never that long, but feels uber clumsy and i’d hate to see the order statistic of that algorithm ;)

this seems more like a stitching problem of some kind; eg if we have 4 grams ‘omg i just found’, ‘i just found a’, ‘just found a peanut’ perhaps we can identify the non trivial overlap and stitch them together (?)

not sure, there are a number of things to try. was hoping that brain dumping some of this would help me see the light but nothing obvious jumps out :(

the main approach will be

1. maintain a relation with one record per ngram we want to monitoring for trending
2. fold 1 hours worth of new data at a time into the model
3. check the entries for the latest hour for any trends

the full version is on github. read on for a line by line walkthrough

the ruby impl used the simplest approach possible for calculating mean and stddev; maintain a record of all the values seen so far and recalculate for each new value.

for our pig version we’ll take a fixed space approach. rather than keep all the values for each time series it turns out we can get away with storing just 3…

1. n: the number of values
2. m: the current mean of all values
3. ms: the current mean of the squares of all values

the idea is that the mean_n+1 = ( n * mean_n + new value ) / n+1
a similar function holds that derives the standard deviation_n+1 from n, the mean_n and the mean of the squares_n

let’s go over the pig script one command a time.

we’ll assume we’ve already run it 6 times and we’re now folding in the 7th hour

the first thing is to load the existing version of the model, in this case stored in the file ‘model.006′
it contains everything we need for checking the trending for each ngram

> raw_model = load 'model.006' as (key:chararray, n:int, m:double, ms:double);

> describe raw_model;
raw_model: {key: chararray, n: int, m: double, ms: double}

> dump raw_model;
(a b,6,1.3333333333333333,2.0)
(a a,3,1.3333333333333333,2.0)
(a c,4,1.25,1.75)
(a d,1,2.0,4.0)
(b a,3,1.0,1.0)
(b d,1,2.0,4.0)
(b c,6,1.5,2.5)
(d c,1,2.0,4.0)
(c a,4,1.0,1.0)
(d e,1,1.0,1.0)
(c d,4,2.0,4.0)
(d a,2,1.0,1.0)

next we tag each entry from the loaded model with a zero frequency. we’ll see later how this makes it easier to fold in the new data.

> model = foreach raw_model generate key, n, m, ms, 0 as f;

> describe model;
model: {key: chararray, n: int, m: double, ms: double, f: int}

> dump model;
(a b,6,1.3333333333333333,2.0,0)
(a a,3,1.3333333333333333,2.0,0)
(a c,4,1.25,1.75,0)
(a d,1,2.0,4.0,0)
(b a,3,1.0,1.0,0)
(b d,1,2.0,4.0,0)
(b c,6,1.5,2.5,0)
(d c,1,2.0,4.0,0)
(c a,4,1.0,1.0,0)
(d e,1,1.0,1.0,0)
(c d,4,2.0,4.0,0)
(d a,2,1.0,1.0,0)

now that we’ve loaded the existing version of the model we can load the next hour of data, in this case contained in ‘chunks/006′.

> next_chunk = load 'chunks/006';

> dump next_chunk;
(a b a b)
(c d a b)
(a b c)
(a d d d)

from the text we want to get the frequency of the ngrams.
the breaking apart of each line into its 2-grams is handled by a simple ruby script; ngram.rb

> define ngramer `ngram.rb` ship('ngram.rb');
> ngrams = stream next_chunk through ngramer as (key:chararray);

> describe ngrams;
ngrams: {key: chararray}

> dump ngrams;
(a b)
(b a)
(a b)
(c d)
(d a)
(a b)
(a b)
(b c)
(a d)
(d d)
(d d)

calculating the frequencies of the ngrams is a simple two step process of first grouping by the key…

> ngrams_grouped = group ngrams by key;

> describe ngrams_grouped;
ngrams_grouped: {group: chararray, ngrams: {key: chararray}}

> dump ngrams_grouped;
(a b,{(a b),(a b),(a b),(a b)})
(a d,{(a d)})
(b a,{(b a)})
(b c,{(b c)})
(c d,{(c d)})
(d a,{(d a)})
(d d,{(d d),(d,d)})

…and then generating the key, frequency pairs

> ngram_freq = foreach ngrams_grouped generate group as key, SIZE(ngrams) as f;

> describe ngram_freq;
ngram_freq: {key: chararray, f: long}

> dump ngram_freq;
(a b,4L)
(a d,1L)
(b a,1L)
(b c,1L)
(c d,1L)
(d a,1L)
(d d,2L)

from this we know all the distinct 2grams that are contained in the next chunk we’re analysing
for each of these 2grams one of two things is true;

1. either the ngram has been seen before (thus it has an entry in the model)
2. this is the first time we’ve seen it, in which case we need to add a new entry to the model

the easiest way i’ve worked out in pig to handle this is to generate a ’seed’ model just for this chunk and fold it into the real model but unioning the relations

(i’ve been using pig 0.3 to keep in line with the current version of elastic map reduce but it might be easier with the various extra joins that are in later versions of pig)

so first we generate the ’seed’ relation…

> seed_values = foreach ngram_freq generate key, 0 as n, 0.0 as m, 0.0 as ms, f;

> describe seed_values;
seed_values: {key: chararray, n: int, m: double, ms: double, f: long}

> dump seed_values;
(a b,0,0.0,0.0,4L)
(a d,0,0.0,0.0,1L)
(b a,0,0.0,0.0,1L)
(b c,0,0.0,0.0,1L)
(c d,0,0.0,0.0,1L)
(d a,0,0.0,0.0,1L)
(d d,0,0.0,0.0,2L)

…and fold it in with a 3 step process; unioning with the original model, grouping and collapsing

first the union…

> model_plus_seed = union model, seed_values;

> describe model_plus_seed;
model_plus_seed: {key: chararray, n: int, m: double, ms: double, f: long}

> dump model_plus_seed;
(a b,0,0.0,0.0,4L)
(a b,6,1.3333333333333333,2.0,0L)
(a d,0,0.0,0.0,1L)
(a a,3,1.3333333333333333,2.0,0L)
(b a,0,0.0,0.0,1L)
(a c,4,1.25,1.75,0L)
(b c,0,0.0,0.0,1L)
(a d,1,2.0,4.0,0L)
(c d,0,0.0,0.0,1L)
(b a,3,1.0,1.0,0L)
(d a,0,0.0,0.0,1L)
(b d,1,2.0,4.0,0L)
(d d,0,0.0,0.0,2L)
(b c,6,1.5,2.5,0L)
(d c,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(d e,1,1.0,1.0,0L)
(c d,4,2.0,4.0,0L)
(d a,2,1.0,1.0,0L)

then the grouping…

> model_plus_seed2 = group model_plus_seed by key;

> describe model_plus_seed2 = group model_plus_seed by key;;
model_plus_seed2: {group: chararray, model_plus_seed: {key: chararray, n: int, m: double, ms: double, f: long}}

> dump model_plus_seed2;
(a a,{(a a,3,1.3333333333333333,2.0,0L)})
(a b,{(a b,0,0.0,0.0,4L),(a b,6,1.3333333333333333,2.0,0L)})
(a c,{(a c,4,1.25,1.75,0L)})
(a d,{(a d,0,0.0,0.0,1L),(a d,1,2.0,4.0,0L)})
(b a,{(b a,0,0.0,0.0,1L),(b a,3,1.0,1.0,0L)})
(b c,{(b c,0,0.0,0.0,1L),(b c,6,1.5,2.5,0L)})
(b d,{(b d,1,2.0,4.0,0L)})
(c a,{(c a,4,1.0,1.0,0L)})
(c d,{(c d,0,0.0,0.0,1L),(c d,4,2.0,4.0,0L)})
(d a,{(d a,0,0.0,0.0,1L),(d a,2,1.0,1.0,0L)})
(d c,{(d c,1,2.0,4.0,0L)})
(d d,{(d d,0,0.0,0.0,2L)})
(d e,{(d e,1,1.0,1.0,0L)})

and finally the collapsing using MAX…

> model_n =
     foreach model_plus_seed2 generate
        group as key,
        MAX(model_plus_seed.n) as n,
        MAX(model_plus_seed.m) as m,
        MAX(model_plus_seed.ms) as ms,
        MAX(model_plus_seed.f) as f;

> describe model_n;
model_n: {key: chararray, n: int, m: double, ms: double, f: long}

> dump model_n;
(a a,3,1.3333333333333333,2.0,0L)
(a b,6,1.3333333333333333,2.0,4L)
(a c,4,1.25,1.75,0L)
(a d,1,2.0,4.0,1L)
(b a,3,1.0,1.0,1L)
(b c,6,1.5,2.5,1L)
(b d,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(c d,4,2.0,4.0,1L)
(d a,2,1.0,1.0,1L)
(d c,1,2.0,4.0,0L)
(d d,0,0.0,0.0,2L)
(d e,1,1.0,1.0,0L)

at this stage we have the original model weaved in with the new data but still need to update the mean and square of means for the values from the latest hour.

we can do this by first seperating out the values we need to update based on whether the frequency is non zero
(recall non zero frequencies represent ngrams from the latest hour)

> split model_n into to_update if f>0, not_to_update if f==0;

> dump to_update;
(a b,6,1.3333333333333333,2.0,4L)
(a d,1,2.0,4.0,1L)
(b a,3,1.0,1.0,1L)
(b c,6,1.5,2.5,1L)
(c d,4,2.0,4.0,1L)
(d a,2,1.0,1.0,1L)
(d d,0,0.0,0.0,2L)

> dump not_to_update;
(a a,3,1.3333333333333333,2.0,0L)
(a c,4,1.25,1.75,0L)
(b d,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(d c,1,2.0,4.0,0L)
(d e,1,1.0,1.0,0L)

we can now update the mean and std deviations based on the new frequency values

> updated =
     foreach to_update {
         m2  = ((n*m)+f)/(n+1);
         ms2 = ((n*ms)+(f*f))/(n+1);
         generate key, n+1 as n, m2 as m, ms2 as ms, f;
     }

> describe updated;
updated: {key: chararray, n: int, m: double, ms: double, f: long}

> dump updated;
(a b,7,1.7142857142857142,4.0,4L)
(a d,2,1.5,2.5,1L)
(b a,4,1.0,1.0,1L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c d,5,1.8,3.4,1L)
(d a,3,1.0,1.0,1L)
(d d,1,2.0,4.0,2L)

these new rows, along with the rows we didn’t update, can be stored as the model at time n+1 ready for the next hours chunk

> to_store = union model_n1, not_to_update;
> store to_store into 'model.007';

> dump to_store;
(a b,7,1.7142857142857142,4.0,4L)
(a a,3,1.3333333333333333,2.0,0L)
(a d,2,1.5,2.5,1L)
(a c,4,1.25,1.75,0L)
(b a,4,1.0,1.0,1L)
(b d,1,2.0,4.0,0L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c a,4,1.0,1.0,0L)
(c d,5,1.8,3.4,1L)
(d c,1,2.0,4.0,0L)
(d a,3,1.0,1.0,1L)
(d e,1,1.0,1.0,0L)
(d d,1,2.0,4.0,2L)

now that we’ve updated the model we can start making the trending check!

first step is to filter out entries that correspond to ngrams we are seeing for the first time
( an new item can’t be trending )

> requiring_trending_check = filter model_n1 by n>1;

> dump requiring_trending_check;
(a b,7,1.7142857142857142,4.0,4L)
(a d,2,1.5,2.5,1L)
(b a,4,1.0,1.0,1L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c d,5,1.8,3.4,1L)
(d a,3,1.0,1.0,1L)

and finally we can make the trending calculation!
we can calculate the minimum trending value, based on mean + twice std dev…

> calc_min_trending =
     foreach requiring_trending_check {
        sd_lhs = n * ms;
        sd_rhs = n * (m*m);
        sd = org.apache.pig.piggybank.evaluation.math.SQRT((sd_lhs-sd_rhs)/n);
        min_trend_value = m + (2*sd);
        generate key, f, m as mean, sd as std_dev,
                 min_trend_value as min_trend_value,
                 f / min_trend_value as percent_over_trend;
    }

> describe calc_min_trending;
calc_min_trending: {key: chararray, f: long, mean: double, std_dev: double, min_trend_value: double, percent_over_trend: double}

> dump calc_min_trending;
(a b,4L,1.7142857142857142,1.0301575072754257,3.7746007288365657,1.059714732061981)
(a d,1L,1.5,0.5,2.5,0.4)
(b a,1L,1.0,0.0,1.0,1.0)
(b c,1L,1.4285714285714286,0.4948716593053934,2.4183147471822153,0.4135111036167584)
(c d,1L,1.8,0.4,2.6,0.3846153846153848)
(d a,1L,1.0,0.0,1.0,1.0)

… and any entries with a frequency over the min trending value are deemed trending!
( for this example it’s only the one )

> trending = filter calc_min_trending by percent_over_trend > 1;

> describe trending;
trending: {key: chararray, f: long, mean: double, std_dev: double, min_trend_value: double, percent_over_trend: double}

> dump trending;
(a b,4L,1.7142857142857142,1.0301575072754257,3.7746007288365657,1.059714732061981)

as a normalisation step i’ve been playing with also factoring in the frequency itself,
haven’t come to a conclusion on whether this is a better metric or not…

> trending2 =
     foreach trending {
        normalised_trend_value = org.apache.pig.piggybank.evaluation.math.LOG10(f) * percent_over_trend;
        generate key, min_trend_value, percent_over_trend, normalised_trend_value as normalised_trend_value;
     }

> describe trending2;
trending2: {key: chararray, min_trend_value: double, percent_over_trend: double, normalised_trend_value: double}

> dump trending2;
(a b,3.7746007288365657,1.059714732061981,0.6380118423953504)

and finally store the top trending values for processing!

> trending_sorted = order trending2 by normalised_trend_value desc;
> top_50 = limit trending_sorted 50;
> store trending_sorted into 'trending.model.006;