brutally short intro to weka from Mat Kelcey on Vimeo.

so here’s a little experiment

1. get the people i follow on twitter
2. look up the words that “represent” them according to the infochimps word bag api
3. build a similiarity matrix based on the common use of those terms
4. plot the connectivity for the top 30 or so pairings

it’s basically partitioned into three groups…

1. veztek (my boss john) and smcinnes (steve from the lonely planet community team) in the top right
2. a big clump of nosqlness with mongodb – hbase – jpatanooga – kevinweil in the bottom left
3. everyone else

an interesting enough result given the time taken; the codes on github

#worldcup tweet viz1 from Mat Kelcey on Vimeo.

here’s a simple visualisation of the use of official country codes (eg #aus) in a week’s worth of tweets from the search stream for #worldcup.

rate is about 2hours of tweets per sec. orb size denotes relative frequency of that country code. edges denote that those two countries feature a lot in the same tweets. movement is based on gravitational like attraction along edges.

the quiet period at about 0:17 is a twitter outage :)

here’s the original processing applet version with a bit more discussion

> x = c(3,1,4,1,5,9,2,6,5,3,5,8)
> ra_x = filter(x, rep(1,3)/3)
> ra_x
Time Series:
Start = 1
End = 12
Frequency = 1
 [1]       NA 2.666667 2.000000 3.333333 5.000000 5.333333 5.666667 4.333333
 [9] 4.666667 4.333333 5.333333       NA

anyways, here’s the first few days analysis based the use of official country tags (eg #aus) in the search stream for #worldcup.

tomorrow i might look in more detail at one of the games, wondering how many variants of ‘goooooooal’ i’ll find :D

i’m finding streaming related algorithms quite interesting lately and think are the way forward in terms of dealing with large amounts of constant data. it’s just not feasible to use algorithms that expect you to have all the data at any given time; it forces you to reprocess all the data you’ve ever seen as you get new examples. my thinking is the best solutions are the ones that build models of the data and fold in new examples in batches. anyways, i’m getting off topic already.

tf/icf as presented in the paper is a variant on the classic tf/idf for term weighting but instead of requiring all weighting in all docs to be recalculated as a new document comes along (as tf/idf strictly does) it instead just approximates based on what has been seen before.

so how does it do? actually quite well, here’s my experimental breakdown

brutally short demo of ggobi from Mat Kelcey on Vimeo.

note: non embedded version has higher res at full screen

a sensible place to start when looking for trends is to consider single terms but what if though we ended up with three equally trending terms ‘happy’, ‘new’ and ‘year’? it’s pretty obvious that the actual trend is ‘happy new year’ but what is the best way to express this as a single trend in an algorithmic sense?

one approach i’ve been playing with is to collect unigrams, bigrams and trigrams (1,2,3 term ‘phrases’) and consider the cases where the terms overlap. basically if ‘happy new year’ is trending then, in some sense, we can ignore trends for ‘happy new’, ‘new year’, ‘happy’, ‘new’ and ‘year’. but does this result in to many false positives? would we miss ‘happy’ as a trend if lots of people were chirpy about the change of year (as they usually are, on new years eve)

rather than outright ignore we could somehow reduce the weighting by removing the double counting.

eg if we had 3 trends; (free beer,11), (free,12) & (beer,25)
we can take 11 (from the 2gram) off both 1grams to give (free beer,11), (free,1) & (beer,14)
showing that ‘beer’, outside of the phrase ‘free beer’, is perhaps a trend in itself (as it should be)

this feels like it might work but would be non trivial (read: fun) to implement

another slightly different problem is around the handling of retweeting. my experiments have shown a huge amount of the ‘trends’ found are related to retweets, which is fine in itself, but it gives quite strange trends since the retweeted portion of the text is usually quite long.

for example; say lots of people are retweeting something and, as some people do, are adding various bits and pieces at the beginning and end; eg ‘RT @bob omg i just found a peanut’ or ‘omg i just found a peanut; via @bob lucky him!!’

if we’re considering bigrams (which i am in my current implementation) we end up with an odd selection of trends such as ‘just found’, ‘a peanut’, ‘omg i’, ‘found a’, ‘i just’ and in these cases it’d be great to be able to just stitch them together into the common retweeted element ‘omg i just found a peanut’.

we could ’solve’ this problem by not just considering 1,2 and 3 grams but considering _all_ possible n-grams for each tweet and employing the technique we spoke of above of reducing the counts. it’d almost be feasible, since tweets are never that long, but feels uber clumsy and i’d hate to see the order statistic of that algorithm ;)

this seems more like a stitching problem of some kind; eg if we have 4 grams ‘omg i just found’, ‘i just found a’, ‘just found a peanut’ perhaps we can identify the non trivial overlap and stitch them together (?)

not sure, there are a number of things to try. was hoping that brain dumping some of this would help me see the light but nothing obvious jumps out :(

the main approach will be

1. maintain a relation with one record per ngram we want to monitoring for trending
2. fold 1 hours worth of new data at a time into the model
3. check the entries for the latest hour for any trends

the full version is on github. read on for a line by line walkthrough

the ruby impl used the simplest approach possible for calculating mean and stddev; maintain a record of all the values seen so far and recalculate for each new value.

for our pig version we’ll take a fixed space approach. rather than keep all the values for each time series it turns out we can get away with storing just 3…

1. n: the number of values
2. m: the current mean of all values
3. ms: the current mean of the squares of all values

the idea is that the mean_n+1 = ( n * mean_n + new value ) / n+1
a similar function holds that derives the standard deviation_n+1 from n, the mean_n and the mean of the squares_n

let’s go over the pig script one command a time.

we’ll assume we’ve already run it 6 times and we’re now folding in the 7th hour

the first thing is to load the existing version of the model, in this case stored in the file ‘model.006′
it contains everything we need for checking the trending for each ngram

> raw_model = load 'model.006' as (key:chararray, n:int, m:double, ms:double);

> describe raw_model;
raw_model: {key: chararray, n: int, m: double, ms: double}

> dump raw_model;
(a b,6,1.3333333333333333,2.0)
(a a,3,1.3333333333333333,2.0)
(a c,4,1.25,1.75)
(a d,1,2.0,4.0)
(b a,3,1.0,1.0)
(b d,1,2.0,4.0)
(b c,6,1.5,2.5)
(d c,1,2.0,4.0)
(c a,4,1.0,1.0)
(d e,1,1.0,1.0)
(c d,4,2.0,4.0)
(d a,2,1.0,1.0)

next we tag each entry from the loaded model with a zero frequency. we’ll see later how this makes it easier to fold in the new data.

> model = foreach raw_model generate key, n, m, ms, 0 as f;

> describe model;
model: {key: chararray, n: int, m: double, ms: double, f: int}

> dump model;
(a b,6,1.3333333333333333,2.0,0)
(a a,3,1.3333333333333333,2.0,0)
(a c,4,1.25,1.75,0)
(a d,1,2.0,4.0,0)
(b a,3,1.0,1.0,0)
(b d,1,2.0,4.0,0)
(b c,6,1.5,2.5,0)
(d c,1,2.0,4.0,0)
(c a,4,1.0,1.0,0)
(d e,1,1.0,1.0,0)
(c d,4,2.0,4.0,0)
(d a,2,1.0,1.0,0)

now that we’ve loaded the existing version of the model we can load the next hour of data, in this case contained in ‘chunks/006′.

> next_chunk = load 'chunks/006';

> dump next_chunk;
(a b a b)
(c d a b)
(a b c)
(a d d d)

from the text we want to get the frequency of the ngrams.
the breaking apart of each line into its 2-grams is handled by a simple ruby script; ngram.rb

> define ngramer `ngram.rb` ship('ngram.rb');
> ngrams = stream next_chunk through ngramer as (key:chararray);

> describe ngrams;
ngrams: {key: chararray}

> dump ngrams;
(a b)
(b a)
(a b)
(c d)
(d a)
(a b)
(a b)
(b c)
(a d)
(d d)
(d d)

calculating the frequencies of the ngrams is a simple two step process of first grouping by the key…

> ngrams_grouped = group ngrams by key;

> describe ngrams_grouped;
ngrams_grouped: {group: chararray, ngrams: {key: chararray}}

> dump ngrams_grouped;
(a b,{(a b),(a b),(a b),(a b)})
(a d,{(a d)})
(b a,{(b a)})
(b c,{(b c)})
(c d,{(c d)})
(d a,{(d a)})
(d d,{(d d),(d,d)})

…and then generating the key, frequency pairs

> ngram_freq = foreach ngrams_grouped generate group as key, SIZE(ngrams) as f;

> describe ngram_freq;
ngram_freq: {key: chararray, f: long}

> dump ngram_freq;
(a b,4L)
(a d,1L)
(b a,1L)
(b c,1L)
(c d,1L)
(d a,1L)
(d d,2L)

from this we know all the distinct 2grams that are contained in the next chunk we’re analysing
for each of these 2grams one of two things is true;

1. either the ngram has been seen before (thus it has an entry in the model)
2. this is the first time we’ve seen it, in which case we need to add a new entry to the model

the easiest way i’ve worked out in pig to handle this is to generate a ’seed’ model just for this chunk and fold it into the real model but unioning the relations

(i’ve been using pig 0.3 to keep in line with the current version of elastic map reduce but it might be easier with the various extra joins that are in later versions of pig)

so first we generate the ’seed’ relation…

> seed_values = foreach ngram_freq generate key, 0 as n, 0.0 as m, 0.0 as ms, f;

> describe seed_values;
seed_values: {key: chararray, n: int, m: double, ms: double, f: long}

> dump seed_values;
(a b,0,0.0,0.0,4L)
(a d,0,0.0,0.0,1L)
(b a,0,0.0,0.0,1L)
(b c,0,0.0,0.0,1L)
(c d,0,0.0,0.0,1L)
(d a,0,0.0,0.0,1L)
(d d,0,0.0,0.0,2L)

…and fold it in with a 3 step process; unioning with the original model, grouping and collapsing

first the union…

> model_plus_seed = union model, seed_values;

> describe model_plus_seed;
model_plus_seed: {key: chararray, n: int, m: double, ms: double, f: long}

> dump model_plus_seed;
(a b,0,0.0,0.0,4L)
(a b,6,1.3333333333333333,2.0,0L)
(a d,0,0.0,0.0,1L)
(a a,3,1.3333333333333333,2.0,0L)
(b a,0,0.0,0.0,1L)
(a c,4,1.25,1.75,0L)
(b c,0,0.0,0.0,1L)
(a d,1,2.0,4.0,0L)
(c d,0,0.0,0.0,1L)
(b a,3,1.0,1.0,0L)
(d a,0,0.0,0.0,1L)
(b d,1,2.0,4.0,0L)
(d d,0,0.0,0.0,2L)
(b c,6,1.5,2.5,0L)
(d c,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(d e,1,1.0,1.0,0L)
(c d,4,2.0,4.0,0L)
(d a,2,1.0,1.0,0L)

then the grouping…

> model_plus_seed2 = group model_plus_seed by key;

> describe model_plus_seed2 = group model_plus_seed by key;;
model_plus_seed2: {group: chararray, model_plus_seed: {key: chararray, n: int, m: double, ms: double, f: long}}

> dump model_plus_seed2;
(a a,{(a a,3,1.3333333333333333,2.0,0L)})
(a b,{(a b,0,0.0,0.0,4L),(a b,6,1.3333333333333333,2.0,0L)})
(a c,{(a c,4,1.25,1.75,0L)})
(a d,{(a d,0,0.0,0.0,1L),(a d,1,2.0,4.0,0L)})
(b a,{(b a,0,0.0,0.0,1L),(b a,3,1.0,1.0,0L)})
(b c,{(b c,0,0.0,0.0,1L),(b c,6,1.5,2.5,0L)})
(b d,{(b d,1,2.0,4.0,0L)})
(c a,{(c a,4,1.0,1.0,0L)})
(c d,{(c d,0,0.0,0.0,1L),(c d,4,2.0,4.0,0L)})
(d a,{(d a,0,0.0,0.0,1L),(d a,2,1.0,1.0,0L)})
(d c,{(d c,1,2.0,4.0,0L)})
(d d,{(d d,0,0.0,0.0,2L)})
(d e,{(d e,1,1.0,1.0,0L)})

and finally the collapsing using MAX…

> model_n =
     foreach model_plus_seed2 generate
        group as key,
        MAX(model_plus_seed.n) as n,
        MAX(model_plus_seed.m) as m,
        MAX(model_plus_seed.ms) as ms,
        MAX(model_plus_seed.f) as f;

> describe model_n;
model_n: {key: chararray, n: int, m: double, ms: double, f: long}

> dump model_n;
(a a,3,1.3333333333333333,2.0,0L)
(a b,6,1.3333333333333333,2.0,4L)
(a c,4,1.25,1.75,0L)
(a d,1,2.0,4.0,1L)
(b a,3,1.0,1.0,1L)
(b c,6,1.5,2.5,1L)
(b d,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(c d,4,2.0,4.0,1L)
(d a,2,1.0,1.0,1L)
(d c,1,2.0,4.0,0L)
(d d,0,0.0,0.0,2L)
(d e,1,1.0,1.0,0L)

at this stage we have the original model weaved in with the new data but still need to update the mean and square of means for the values from the latest hour.

we can do this by first seperating out the values we need to update based on whether the frequency is non zero
(recall non zero frequencies represent ngrams from the latest hour)

> split model_n into to_update if f>0, not_to_update if f==0;

> dump to_update;
(a b,6,1.3333333333333333,2.0,4L)
(a d,1,2.0,4.0,1L)
(b a,3,1.0,1.0,1L)
(b c,6,1.5,2.5,1L)
(c d,4,2.0,4.0,1L)
(d a,2,1.0,1.0,1L)
(d d,0,0.0,0.0,2L)

> dump not_to_update;
(a a,3,1.3333333333333333,2.0,0L)
(a c,4,1.25,1.75,0L)
(b d,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(d c,1,2.0,4.0,0L)
(d e,1,1.0,1.0,0L)

we can now update the mean and std deviations based on the new frequency values

> updated =
     foreach to_update {
         m2  = ((n*m)+f)/(n+1);
         ms2 = ((n*ms)+(f*f))/(n+1);
         generate key, n+1 as n, m2 as m, ms2 as ms, f;
     }

> describe updated;
updated: {key: chararray, n: int, m: double, ms: double, f: long}

> dump updated;
(a b,7,1.7142857142857142,4.0,4L)
(a d,2,1.5,2.5,1L)
(b a,4,1.0,1.0,1L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c d,5,1.8,3.4,1L)
(d a,3,1.0,1.0,1L)
(d d,1,2.0,4.0,2L)

these new rows, along with the rows we didn’t update, can be stored as the model at time n+1 ready for the next hours chunk

> to_store = union model_n1, not_to_update;
> store to_store into 'model.007';

> dump to_store;
(a b,7,1.7142857142857142,4.0,4L)
(a a,3,1.3333333333333333,2.0,0L)
(a d,2,1.5,2.5,1L)
(a c,4,1.25,1.75,0L)
(b a,4,1.0,1.0,1L)
(b d,1,2.0,4.0,0L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c a,4,1.0,1.0,0L)
(c d,5,1.8,3.4,1L)
(d c,1,2.0,4.0,0L)
(d a,3,1.0,1.0,1L)
(d e,1,1.0,1.0,0L)
(d d,1,2.0,4.0,2L)

now that we’ve updated the model we can start making the trending check!

first step is to filter out entries that correspond to ngrams we are seeing for the first time
( an new item can’t be trending )

> requiring_trending_check = filter model_n1 by n>1;

> dump requiring_trending_check;
(a b,7,1.7142857142857142,4.0,4L)
(a d,2,1.5,2.5,1L)
(b a,4,1.0,1.0,1L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c d,5,1.8,3.4,1L)
(d a,3,1.0,1.0,1L)

and finally we can make the trending calculation!
we can calculate the minimum trending value, based on mean + twice std dev…

> calc_min_trending =
     foreach requiring_trending_check {
        sd_lhs = n * ms;
        sd_rhs = n * (m*m);
        sd = org.apache.pig.piggybank.evaluation.math.SQRT((sd_lhs-sd_rhs)/n);
        min_trend_value = m + (2*sd);
        generate key, f, m as mean, sd as std_dev,
                 min_trend_value as min_trend_value,
                 f / min_trend_value as percent_over_trend;
    }

> describe calc_min_trending;
calc_min_trending: {key: chararray, f: long, mean: double, std_dev: double, min_trend_value: double, percent_over_trend: double}

> dump calc_min_trending;
(a b,4L,1.7142857142857142,1.0301575072754257,3.7746007288365657,1.059714732061981)
(a d,1L,1.5,0.5,2.5,0.4)
(b a,1L,1.0,0.0,1.0,1.0)
(b c,1L,1.4285714285714286,0.4948716593053934,2.4183147471822153,0.4135111036167584)
(c d,1L,1.8,0.4,2.6,0.3846153846153848)
(d a,1L,1.0,0.0,1.0,1.0)

… and any entries with a frequency over the min trending value are deemed trending!
( for this example it’s only the one )

> trending = filter calc_min_trending by percent_over_trend > 1;

> describe trending;
trending: {key: chararray, f: long, mean: double, std_dev: double, min_trend_value: double, percent_over_trend: double}

> dump trending;
(a b,4L,1.7142857142857142,1.0301575072754257,3.7746007288365657,1.059714732061981)

as a normalisation step i’ve been playing with also factoring in the frequency itself,
haven’t come to a conclusion on whether this is a better metric or not…

> trending2 =
     foreach trending {
        normalised_trend_value = org.apache.pig.piggybank.evaluation.math.LOG10(f) * percent_over_trend;
        generate key, min_trend_value, percent_over_trend, normalised_trend_value as normalised_trend_value;
     }

> describe trending2;
trending2: {key: chararray, min_trend_value: double, percent_over_trend: double, normalised_trend_value: double}

> dump trending2;
(a b,3.7746007288365657,1.059714732061981,0.6380118423953504)

and finally store the top trending values for processing!

> trending_sorted = order trending2 by normalised_trend_value desc;
> top_50 = limit trending_sorted 50;
> store trending_sorted into 'trending.model.006;

what does it mean for a topic to be ‘trending’? consider the following time series (430e3 tweets containing cheese collected over a month period bucketed into hourly timeslots)

without a formal definition we can just look at this and say that the series was trending where there was a spike just before time 600. as a start then let’s just define a trend as a value that was greater than was ‘expected’.

how can we calculate trending?

one really nice simple algorithm for detecting a trend is to say a value, v, is trending if v > mean + 3 * standard deviation of the data seen so far. (thanks @peteskomoroch for the suggestion, works a treat)

let’s consider the same time series as before but this time with some overlaid data;
green – the mean
red – minimum trend value ( = mean + 3 * std dev )
blue – instances where the value > minimum trend value

here’s a zoom in on the last 200 values

this works surprisingly well, the mean gives a solid expectation of the value with the standard deviation covering the daily periodic nature of the data.

it’s not perfect though as this system only ever allows a trend around the peaks of the cycle.

for example consider the troughs which have a frequency value around 250. if we had a value in one of those timeslot’s that was 1000, ie four times what was expected given that time of day, it would not be considered trending since the value has to be over 1500

facet by hour

one way to handle this is to not have a single time series but instead maintain 24 time series, one for each hour of the day.

faceting in this way gives the following trending

and though this doesn’t present any cases of trends at a trough we can see it was prettttty close a number of times.

facet by ngram

one other interesting way to facet, and the main purpose of this project, is to maintain a seperate time series for each ngram in the tweet.

the top 10 2-grams in my dataset are…

let’s look at the time series for a few of them.

#4 grilled cheese

we get an interesting result from the very first spike at around 225… poor fangirl @rachelljonas spent 10 minutes tweeting like crazy trying to get the attention of @nickjonas (some popstar i’ve never heard of) and bumped up ‘grilled cheese’ for a single timeslot (here’s her attempt to get his attention…)

this raises an interesting point about spam and should possibly my first pre processing data cleaning step. how should we disregard too many tweets from a single user in a timeslot?

the next spike at around 375 shows potentially my first true trending topic, a sudden increase in the discussion of making grilled cheese. the data has no dups so looks like it was just grilled cheese time!

#5 cream cheese

one major spike at about 376, looking at the data. there might have been a competition being run relating to #gno #bagelfuls ?

#412 goats cheese

nothing uber interesting with the ‘goats cheese’ time series but it does illustrate an interesting point. for all the examples we’ve looked at so far each timeslot of an hour has included as least one entry for the 2gram. by the time we’re getting to the less frequent ngrams we see as many timeslots with a zero frequency as with a non zero frequency.

interestingly if you only consider the cases where the frequency values are non zero i think you get a better sense of where the values are trending.

this also turns out to make things easier to process :)

#1483 apple juice

with ‘apple juice’, an even less frequent 2gram, the effect is even more noticable if you ignore the zero frequency cases.

so with two ways of faceting the data, either timeslots or ngrams, the next step is porting the algorithm to pig so we can run it at scale, write up coming soon!

( code ( in a pretty raw form ) available at github )