so here’s a little experiment

1. get the people i follow on twitter
2. look up the words that “represent” them according to the infochimps word bag api
3. build a similiarity matrix based on the common use of those terms
4. plot the connectivity for the top 30 or so pairings

it’s basically partitioned into three groups…

1. veztek (my boss john) and smcinnes (steve from the lonely planet community team) in the top right
2. a big clump of nosqlness with mongodb – hbase – jpatanooga – kevinweil in the bottom left
3. everyone else

an interesting enough result given the time taken; the codes on github

#worldcup tweet viz1 from Mat Kelcey on Vimeo.

here’s a simple visualisation of the use of official country codes (eg #aus) in a week’s worth of tweets from the search stream for #worldcup.

rate is about 2hours of tweets per sec. orb size denotes relative frequency of that country code. edges denote that those two countries feature a lot in the same tweets. movement is based on gravitational like attraction along edges.

the quiet period at about 0:17 is a twitter outage :)

here’s the original processing applet version with a bit more discussion

anyways, here’s the first few days analysis based the use of official country tags (eg #aus) in the search stream for #worldcup.

tomorrow i might look in more detail at one of the games, wondering how many variants of ‘goooooooal’ i’ll find :D

the main approach will be

1. maintain a relation with one record per ngram we want to monitoring for trending
2. fold 1 hours worth of new data at a time into the model
3. check the entries for the latest hour for any trends

the full version is on github. read on for a line by line walkthrough

the ruby impl used the simplest approach possible for calculating mean and stddev; maintain a record of all the values seen so far and recalculate for each new value.

for our pig version we’ll take a fixed space approach. rather than keep all the values for each time series it turns out we can get away with storing just 3…

1. n: the number of values
2. m: the current mean of all values
3. ms: the current mean of the squares of all values

the idea is that the mean_n+1 = ( n * mean_n + new value ) / n+1
a similar function holds that derives the standard deviation_n+1 from n, the mean_n and the mean of the squares_n

let’s go over the pig script one command a time.

we’ll assume we’ve already run it 6 times and we’re now folding in the 7th hour

the first thing is to load the existing version of the model, in this case stored in the file ‘model.006′
it contains everything we need for checking the trending for each ngram

> raw_model = load 'model.006' as (key:chararray, n:int, m:double, ms:double);

> describe raw_model;
raw_model: {key: chararray, n: int, m: double, ms: double}

> dump raw_model;
(a b,6,1.3333333333333333,2.0)
(a a,3,1.3333333333333333,2.0)
(a c,4,1.25,1.75)
(a d,1,2.0,4.0)
(b a,3,1.0,1.0)
(b d,1,2.0,4.0)
(b c,6,1.5,2.5)
(d c,1,2.0,4.0)
(c a,4,1.0,1.0)
(d e,1,1.0,1.0)
(c d,4,2.0,4.0)
(d a,2,1.0,1.0)

next we tag each entry from the loaded model with a zero frequency. we’ll see later how this makes it easier to fold in the new data.

> model = foreach raw_model generate key, n, m, ms, 0 as f;

> describe model;
model: {key: chararray, n: int, m: double, ms: double, f: int}

> dump model;
(a b,6,1.3333333333333333,2.0,0)
(a a,3,1.3333333333333333,2.0,0)
(a c,4,1.25,1.75,0)
(a d,1,2.0,4.0,0)
(b a,3,1.0,1.0,0)
(b d,1,2.0,4.0,0)
(b c,6,1.5,2.5,0)
(d c,1,2.0,4.0,0)
(c a,4,1.0,1.0,0)
(d e,1,1.0,1.0,0)
(c d,4,2.0,4.0,0)
(d a,2,1.0,1.0,0)

now that we’ve loaded the existing version of the model we can load the next hour of data, in this case contained in ‘chunks/006′.

> next_chunk = load 'chunks/006';

> dump next_chunk;
(a b a b)
(c d a b)
(a b c)
(a d d d)

from the text we want to get the frequency of the ngrams.
the breaking apart of each line into its 2-grams is handled by a simple ruby script; ngram.rb

> define ngramer `ngram.rb` ship('ngram.rb');
> ngrams = stream next_chunk through ngramer as (key:chararray);

> describe ngrams;
ngrams: {key: chararray}

> dump ngrams;
(a b)
(b a)
(a b)
(c d)
(d a)
(a b)
(a b)
(b c)
(a d)
(d d)
(d d)

calculating the frequencies of the ngrams is a simple two step process of first grouping by the key…

> ngrams_grouped = group ngrams by key;

> describe ngrams_grouped;
ngrams_grouped: {group: chararray, ngrams: {key: chararray}}

> dump ngrams_grouped;
(a b,{(a b),(a b),(a b),(a b)})
(a d,{(a d)})
(b a,{(b a)})
(b c,{(b c)})
(c d,{(c d)})
(d a,{(d a)})
(d d,{(d d),(d,d)})

…and then generating the key, frequency pairs

> ngram_freq = foreach ngrams_grouped generate group as key, SIZE(ngrams) as f;

> describe ngram_freq;
ngram_freq: {key: chararray, f: long}

> dump ngram_freq;
(a b,4L)
(a d,1L)
(b a,1L)
(b c,1L)
(c d,1L)
(d a,1L)
(d d,2L)

from this we know all the distinct 2grams that are contained in the next chunk we’re analysing
for each of these 2grams one of two things is true;

1. either the ngram has been seen before (thus it has an entry in the model)
2. this is the first time we’ve seen it, in which case we need to add a new entry to the model

the easiest way i’ve worked out in pig to handle this is to generate a ’seed’ model just for this chunk and fold it into the real model but unioning the relations

(i’ve been using pig 0.3 to keep in line with the current version of elastic map reduce but it might be easier with the various extra joins that are in later versions of pig)

so first we generate the ’seed’ relation…

> seed_values = foreach ngram_freq generate key, 0 as n, 0.0 as m, 0.0 as ms, f;

> describe seed_values;
seed_values: {key: chararray, n: int, m: double, ms: double, f: long}

> dump seed_values;
(a b,0,0.0,0.0,4L)
(a d,0,0.0,0.0,1L)
(b a,0,0.0,0.0,1L)
(b c,0,0.0,0.0,1L)
(c d,0,0.0,0.0,1L)
(d a,0,0.0,0.0,1L)
(d d,0,0.0,0.0,2L)

…and fold it in with a 3 step process; unioning with the original model, grouping and collapsing

first the union…

> model_plus_seed = union model, seed_values;

> describe model_plus_seed;
model_plus_seed: {key: chararray, n: int, m: double, ms: double, f: long}

> dump model_plus_seed;
(a b,0,0.0,0.0,4L)
(a b,6,1.3333333333333333,2.0,0L)
(a d,0,0.0,0.0,1L)
(a a,3,1.3333333333333333,2.0,0L)
(b a,0,0.0,0.0,1L)
(a c,4,1.25,1.75,0L)
(b c,0,0.0,0.0,1L)
(a d,1,2.0,4.0,0L)
(c d,0,0.0,0.0,1L)
(b a,3,1.0,1.0,0L)
(d a,0,0.0,0.0,1L)
(b d,1,2.0,4.0,0L)
(d d,0,0.0,0.0,2L)
(b c,6,1.5,2.5,0L)
(d c,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(d e,1,1.0,1.0,0L)
(c d,4,2.0,4.0,0L)
(d a,2,1.0,1.0,0L)

then the grouping…

> model_plus_seed2 = group model_plus_seed by key;

> describe model_plus_seed2 = group model_plus_seed by key;;
model_plus_seed2: {group: chararray, model_plus_seed: {key: chararray, n: int, m: double, ms: double, f: long}}

> dump model_plus_seed2;
(a a,{(a a,3,1.3333333333333333,2.0,0L)})
(a b,{(a b,0,0.0,0.0,4L),(a b,6,1.3333333333333333,2.0,0L)})
(a c,{(a c,4,1.25,1.75,0L)})
(a d,{(a d,0,0.0,0.0,1L),(a d,1,2.0,4.0,0L)})
(b a,{(b a,0,0.0,0.0,1L),(b a,3,1.0,1.0,0L)})
(b c,{(b c,0,0.0,0.0,1L),(b c,6,1.5,2.5,0L)})
(b d,{(b d,1,2.0,4.0,0L)})
(c a,{(c a,4,1.0,1.0,0L)})
(c d,{(c d,0,0.0,0.0,1L),(c d,4,2.0,4.0,0L)})
(d a,{(d a,0,0.0,0.0,1L),(d a,2,1.0,1.0,0L)})
(d c,{(d c,1,2.0,4.0,0L)})
(d d,{(d d,0,0.0,0.0,2L)})
(d e,{(d e,1,1.0,1.0,0L)})

and finally the collapsing using MAX…

> model_n =
     foreach model_plus_seed2 generate
        group as key,
        MAX(model_plus_seed.n) as n,
        MAX(model_plus_seed.m) as m,
        MAX(model_plus_seed.ms) as ms,
        MAX(model_plus_seed.f) as f;

> describe model_n;
model_n: {key: chararray, n: int, m: double, ms: double, f: long}

> dump model_n;
(a a,3,1.3333333333333333,2.0,0L)
(a b,6,1.3333333333333333,2.0,4L)
(a c,4,1.25,1.75,0L)
(a d,1,2.0,4.0,1L)
(b a,3,1.0,1.0,1L)
(b c,6,1.5,2.5,1L)
(b d,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(c d,4,2.0,4.0,1L)
(d a,2,1.0,1.0,1L)
(d c,1,2.0,4.0,0L)
(d d,0,0.0,0.0,2L)
(d e,1,1.0,1.0,0L)

at this stage we have the original model weaved in with the new data but still need to update the mean and square of means for the values from the latest hour.

we can do this by first seperating out the values we need to update based on whether the frequency is non zero
(recall non zero frequencies represent ngrams from the latest hour)

> split model_n into to_update if f>0, not_to_update if f==0;

> dump to_update;
(a b,6,1.3333333333333333,2.0,4L)
(a d,1,2.0,4.0,1L)
(b a,3,1.0,1.0,1L)
(b c,6,1.5,2.5,1L)
(c d,4,2.0,4.0,1L)
(d a,2,1.0,1.0,1L)
(d d,0,0.0,0.0,2L)

> dump not_to_update;
(a a,3,1.3333333333333333,2.0,0L)
(a c,4,1.25,1.75,0L)
(b d,1,2.0,4.0,0L)
(c a,4,1.0,1.0,0L)
(d c,1,2.0,4.0,0L)
(d e,1,1.0,1.0,0L)

we can now update the mean and std deviations based on the new frequency values

> updated =
     foreach to_update {
         m2  = ((n*m)+f)/(n+1);
         ms2 = ((n*ms)+(f*f))/(n+1);
         generate key, n+1 as n, m2 as m, ms2 as ms, f;
     }

> describe updated;
updated: {key: chararray, n: int, m: double, ms: double, f: long}

> dump updated;
(a b,7,1.7142857142857142,4.0,4L)
(a d,2,1.5,2.5,1L)
(b a,4,1.0,1.0,1L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c d,5,1.8,3.4,1L)
(d a,3,1.0,1.0,1L)
(d d,1,2.0,4.0,2L)

these new rows, along with the rows we didn’t update, can be stored as the model at time n+1 ready for the next hours chunk

> to_store = union model_n1, not_to_update;
> store to_store into 'model.007';

> dump to_store;
(a b,7,1.7142857142857142,4.0,4L)
(a a,3,1.3333333333333333,2.0,0L)
(a d,2,1.5,2.5,1L)
(a c,4,1.25,1.75,0L)
(b a,4,1.0,1.0,1L)
(b d,1,2.0,4.0,0L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c a,4,1.0,1.0,0L)
(c d,5,1.8,3.4,1L)
(d c,1,2.0,4.0,0L)
(d a,3,1.0,1.0,1L)
(d e,1,1.0,1.0,0L)
(d d,1,2.0,4.0,2L)

now that we’ve updated the model we can start making the trending check!

first step is to filter out entries that correspond to ngrams we are seeing for the first time
( an new item can’t be trending )

> requiring_trending_check = filter model_n1 by n>1;

> dump requiring_trending_check;
(a b,7,1.7142857142857142,4.0,4L)
(a d,2,1.5,2.5,1L)
(b a,4,1.0,1.0,1L)
(b c,7,1.4285714285714286,2.2857142857142856,1L)
(c d,5,1.8,3.4,1L)
(d a,3,1.0,1.0,1L)

and finally we can make the trending calculation!
we can calculate the minimum trending value, based on mean + twice std dev…

> calc_min_trending =
     foreach requiring_trending_check {
        sd_lhs = n * ms;
        sd_rhs = n * (m*m);
        sd = org.apache.pig.piggybank.evaluation.math.SQRT((sd_lhs-sd_rhs)/n);
        min_trend_value = m + (2*sd);
        generate key, f, m as mean, sd as std_dev,
                 min_trend_value as min_trend_value,
                 f / min_trend_value as percent_over_trend;
    }

> describe calc_min_trending;
calc_min_trending: {key: chararray, f: long, mean: double, std_dev: double, min_trend_value: double, percent_over_trend: double}

> dump calc_min_trending;
(a b,4L,1.7142857142857142,1.0301575072754257,3.7746007288365657,1.059714732061981)
(a d,1L,1.5,0.5,2.5,0.4)
(b a,1L,1.0,0.0,1.0,1.0)
(b c,1L,1.4285714285714286,0.4948716593053934,2.4183147471822153,0.4135111036167584)
(c d,1L,1.8,0.4,2.6,0.3846153846153848)
(d a,1L,1.0,0.0,1.0,1.0)

… and any entries with a frequency over the min trending value are deemed trending!
( for this example it’s only the one )

> trending = filter calc_min_trending by percent_over_trend > 1;

> describe trending;
trending: {key: chararray, f: long, mean: double, std_dev: double, min_trend_value: double, percent_over_trend: double}

> dump trending;
(a b,4L,1.7142857142857142,1.0301575072754257,3.7746007288365657,1.059714732061981)

as a normalisation step i’ve been playing with also factoring in the frequency itself,
haven’t come to a conclusion on whether this is a better metric or not…

> trending2 =
     foreach trending {
        normalised_trend_value = org.apache.pig.piggybank.evaluation.math.LOG10(f) * percent_over_trend;
        generate key, min_trend_value, percent_over_trend, normalised_trend_value as normalised_trend_value;
     }

> describe trending2;
trending2: {key: chararray, min_trend_value: double, percent_over_trend: double, normalised_trend_value: double}

> dump trending2;
(a b,3.7746007288365657,1.059714732061981,0.6380118423953504)

and finally store the top trending values for processing!

> trending_sorted = order trending2 by normalised_trend_value desc;
> top_50 = limit trending_sorted 50;
> store trending_sorted into 'trending.model.006;

what does it mean for a topic to be ‘trending’? consider the following time series (430e3 tweets containing cheese collected over a month period bucketed into hourly timeslots)

without a formal definition we can just look at this and say that the series was trending where there was a spike just before time 600. as a start then let’s just define a trend as a value that was greater than was ‘expected’.

how can we calculate trending?

one really nice simple algorithm for detecting a trend is to say a value, v, is trending if v > mean + 3 * standard deviation of the data seen so far. (thanks @peteskomoroch for the suggestion, works a treat)

let’s consider the same time series as before but this time with some overlaid data;
green – the mean
red – minimum trend value ( = mean + 3 * std dev )
blue – instances where the value > minimum trend value

here’s a zoom in on the last 200 values

this works surprisingly well, the mean gives a solid expectation of the value with the standard deviation covering the daily periodic nature of the data.

it’s not perfect though as this system only ever allows a trend around the peaks of the cycle.

for example consider the troughs which have a frequency value around 250. if we had a value in one of those timeslot’s that was 1000, ie four times what was expected given that time of day, it would not be considered trending since the value has to be over 1500

facet by hour

one way to handle this is to not have a single time series but instead maintain 24 time series, one for each hour of the day.

faceting in this way gives the following trending

and though this doesn’t present any cases of trends at a trough we can see it was prettttty close a number of times.

facet by ngram

one other interesting way to facet, and the main purpose of this project, is to maintain a seperate time series for each ngram in the tweet.

the top 10 2-grams in my dataset are…

let’s look at the time series for a few of them.

#4 grilled cheese

we get an interesting result from the very first spike at around 225… poor fangirl @rachelljonas spent 10 minutes tweeting like crazy trying to get the attention of @nickjonas (some popstar i’ve never heard of) and bumped up ‘grilled cheese’ for a single timeslot (here’s her attempt to get his attention…)

this raises an interesting point about spam and should possibly my first pre processing data cleaning step. how should we disregard too many tweets from a single user in a timeslot?

the next spike at around 375 shows potentially my first true trending topic, a sudden increase in the discussion of making grilled cheese. the data has no dups so looks like it was just grilled cheese time!

#5 cream cheese

one major spike at about 376, looking at the data. there might have been a competition being run relating to #gno #bagelfuls ?

#412 goats cheese

nothing uber interesting with the ‘goats cheese’ time series but it does illustrate an interesting point. for all the examples we’ve looked at so far each timeslot of an hour has included as least one entry for the 2gram. by the time we’re getting to the less frequent ngrams we see as many timeslots with a zero frequency as with a non zero frequency.

interestingly if you only consider the cases where the frequency values are non zero i think you get a better sense of where the values are trending.

this also turns out to make things easier to process :)

#1483 apple juice

with ‘apple juice’, an even less frequent 2gram, the effect is even more noticable if you ignore the zero frequency cases.

so with two ways of faceting the data, either timeslots or ngrams, the next step is porting the algorithm to pig so we can run it at scale, write up coming soon!

( code ( in a pretty raw form ) available at github )

first i build a friend graph for 100 people ‘around’ me (taken from a crawl i did last year). by ‘friend’ i mean that if alice follows bob then bob also follows alice.

here the graph, some things to note though; it was an unfinished crawl (can a crawl of twitter EVER be finished) and was done october last year so is a bit out of date.

and here is the dendrogram decomposition

some interesting clusterings come out..

right at the bottom we have a small clique (ie everyone following everyone else) of people i’ve known from when i was in sydney

this small group connects to the group i’m in; tinybuddha down to evanbottcher; which roughly describes the group of people i’ve met in melbourne.

the order of the single breakaways in the melbourne group is pretty arbitrary. i get quite different ordering if i run the decomposition multiple times due to the random tie breaking involved. i could either run the decomposition multiple times and work out some kind of averaging or choose another more granular way of deciding how to break ties.

the next connector after syndey and melbourne are unified is deanemorrow a coworker when i was at distra. this one sticks out for me as being the biggest flaw in the clustering since it would have made more sense to have him placed near distra at the bottom.

another interesting clique is near me..

it has four thoughtworkers; mark, gill, debs and evan and one sensiser; korny. did korny perhaps work for thoughtworks in a previous life ;)

another interesting note is there exists a path from me to peter norvig (who is too busy for twitter it seems) but only because of the huge connector nodes that exist in twitter. an example in this case is TUAW who follow 30,000+ people and have even more followers. these nodes cause a bit of noise in the system since they are slightly false representations of what a ‘friend’ means in my mind. not sure how to take these numbers into account…

things to do…

• the biggest oversimplification in this system is how i break ties for deciding which edge to cut out next if multiple exist with the same betweenness. currently it chooses the one that would make the most even sized break (based on smallest standard deviation of the connected components). though this is good for breaking a group into even sizes it’s bad since it favours breaking a single element off a large group. this is what has caused the ‘laddering’ we see in the melbourne group.
• the shortest path algorithm used to calculate edge betweenness is stochastic and if multiple shortest paths exist only one of them is chosen. it’d be better if all were considered with a weighting scheme.
• it might be better to consider vertex betweenness instead of edge betweenness since one person could exist in multiple groups. if i started down this path though i think i’d rather just rewrite the lot using something like  the clique percolation method

all the code is on github

i could label a heap of tweets myself as being positive, neutral or negative but instead this seems to be the perfect job for mechanical turk

so i put up 100 ‘cream cheese’ tweets on mechanical turk, asked for 3 opinions per tweet and offered $0.01 per opinion. took under 30 minutes to get back all 300 opinions and only cost $4.50 ($3 for the work, $1.50 admin fee)

the results are interesting in themselves…

mostly they are consistent;

for example all three sentiments for bagels and cream cheese for breakfast. very original were neutral

and all three sentiments for very few things are as good as a warm nyc bagel with cream cheese first thing in the am were positive.

but occasionally they aren’t consistent;

the tweet developing a recipe for orange cream cheese swirled cardamom brownies… that’s too long a name. hmm… suggestions? had one positive, one neutral and one negative

interestingly there was no case of a tweet having all opinions being negative; even bad idea. dont eat bagel with mixed berry cream cheese, right after u washed ur mouth with listerine. . ended up with two negatives and one positive (?)

hmmmm

i’ve got to say mongodb is such an easy tool for working with json data. these 20 odd lines insert a text json tweet stream into mongo. so simple, why can’t all code be this easy…

sometimes it’s really important ground breaking world changing stuff…
but most of the time it’s ridiculous waste of time stuff like ‘i ate some cheese’

in fact how much do people actually tweet about cheese?
and when they do, what are the most important cheese related topics?

lets gather some data…

using the twitter search api this is dead simple. here’s a hacktastic script which does a polling search for cheese. start it up and bake at 200deg C, i mean, run overnight.

bash> ./collect_cheese.rb >> cheese.out

in the morning we’ve got some tweets about cheese, yay!

UPDATE!

even easier than a hacktastic script is the new filter streaming api

bash> curl -s -u user:pasword http://stream.twitter.com/1/statuses/filter.json?track=cheese

yay!

let’s poke around, but first some l33t hax0r bash aliases for the sake of brevity

alias t='tail'
alias h='head'
alias s='sort'
alias u='uniq'
alias g='grep'

let’s start with a sample, the first 10 tweets…

bash> ./parse_cheese_out.rb < cheese.out | h
Pasta with pesto and cheese. Some watermelon but alas did not get to the salad. At least not yet. http://twitpic.com/pf1rr
guess imma have a steak n cheese since a certain sum1 stunntin in bring me chines
@souljaboytellem Cheese and Bread XD
Great tips for awesome Pizza. Who loves Pizza. Cheese Matters http://bit.ly/42xkZS
Hey bring that cheese.
Milwaukee airport officially 10x better than Lambert. Not surprisingly, a large cheese selection. Kind of a cheese motif in the shops here.
http://twitpic.com/pf1pl - The Cheese Cake is served.
LA has amazing restaurants. had the best grilled cheese (w/sharp cheddar, gruyere and dijon mustard) at comme ca for lunch with @maxwanger
Yummy pepperoni and bleu cheese pizza consumed. Next up a glass of wine. Yay weekend!
Applebees take-out flow aka Dinner for 1- Grilled Chicken/Shrimp and cheddar jack mac & cheese.

to get a context on what words are being used lets find the most frequent words, ie 1-grams (where an n-gram is a phrase of n terms) in the first 10 tweets.
we’ll sanitize by removing weird characters, downcasing, removing urls, etc

bash> ./parse_cheese_out.rb < cheese.out | h | ./sanitise.rb | ./cheese_grams.rb 1 | s | u -c | s -n | t
 2 in
 2 of
 2 with
 3 for
 3 not
 3 pizza
 4 the
 5 a
 5 and
11 cheese

(this says ‘cheese’ was mentioned 11 times while ‘with’ was mentioned twice)

observations;

1. it’s no surprise that cheese is the most frequent since it was our search term.
2. the most frequent words that aren’t cheese, ‘and’, ‘a’ and ‘the’, are classic english constructs.
3. the next most frequent term is ‘pizza’. is pizza the biggest cheese use? well, in these 10 tweets perhaps…

next lets consider the frequency of bi-grams, that is 2 word phrases.

bash> ./parse_cheese_out.rb < cheese.out | h | ./sanitise.rb | ./cheese_grams.rb 2 | s | u -c | s -n | t
1 to the
1 up a
1 watermelon but
1 who loves
1 wine yay
1 with maxwanger
1 with pesto
1 w sharp
1 yay weekend
1 yummy pepperoni

this result is not too interesting, or surprising. seems there are no common two word tuples across the first 10 tweets. fair enough.

how about bigrams across 100 tweets instead of 10?

bash> ./parse_cheese_out.rb < cheese.out | h -100 | ./sanitise.rb | ./cheese_grams.rb 2 | s | u -c | s -n | t
 4 mac and
 4 nacho cheese
 5 & cheese
 5 cheese and
 5 grilled cheese
 5 i love
 6 cheese cake
 6 hot pocket
 6 with cheese
14 and cheese

now we’re getting somewhere!

more (arguable) observations

1. when people are making something with cheese they say ‘X and cheese’ more than ‘cheese and X’ ( from the raw frequencies )
2. people like cheese more than they dislike it; ( the only sentiment is ‘love’ )
3. ‘with’ is a reasonable synonym for ‘and’ ( since they are the two most frequent bigrams ending in cheese )

one that jumps out at me is ‘mac and’. mac ? is this macaroni?

let’s try 3grams…

bash> ./parse_cheese_out.rb < cheese.out | h -100 | ./sanitise.rb | ./cheese_grams.rb 3 | s | u -c | s -n | t
3 mac & cheese
3 not so delicious
3 pocket next time
3 pocket not so
3 so delicious as
3 sounds better go
3 the ham and
3 wine and cheese
3 you like cheese
4 mac and cheese

hmmm. looks like mac is macaroni.

from wikipedia… “Macaroni and cheese (also referred to as macaroni cheese in the United Kingdom and mac ‘n’ cheese in parts of the United States and Canada)”
so macaroni cheese is pretty popular! at least in the 3gram space, at this time of day, which is late afternoon in the united states timezone

does this pattern continue as we consider more tweets? say 1000?

bash> ./parse_cheese_out.rb < cheese.out | h -1000 | ./sanitise.rb | ./cheese_grams.rb 3 | s | u -c | s -n | t
 7 macaroni and cheese
 7 philly cheese steak
 8 on the moon
 9 steak and cheese
10 ham and cheese
11 wine and cheese
28 chuck e cheese
33 mac and cheese
33 mac & cheese
48 mac n cheese

yep. how about 10,000 ?

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | ./cheese_grams.rb 3 | s | u -c | s -n | t
 74 wine and cheese
 77 a grilled cheese
 78 egg and cheese
 78 ham and cheese
 81 with cream cheese
 89 at chuck e
194 mac & cheese
294 mac and cheese
301 mac n cheese
337 chuck e cheese

bam! chuck e cheese, one of the bastions of children’s health food in america, takes over!

there is an overlap between ‘at chuck e’ and ‘chuck e cheese’ so let’s try 4 grams…

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | ./cheese_grams.rb 4 | s | u -c | s -n | t
27 mac n cheese and
27 water on the moon
28 bacon egg & cheese
28 mac and cheese and
29 call of duty modern
29 duty modern warfare 2
29 of duty modern warfare
32 bacon egg and cheese
58 to chuck e cheese
82 at chuck e cheese

observations…

1. more people tweet when they’re at chucke cheese than when they’re planning to go there.
2. ‘call of duty modern warfare 2′ wtf1?
3. ‘water on the moon’ wtf2?

let’s have a look at call of duty first.

bash> ./parse_cheese_out.rb < cheese.out | ./sanitise.rb | g 'call of duty' | h | s | u -c
1 i unlocked the royale with cheese achievement on call of duty modern warfare 2
1 making stromboli with tomoto cheese and spinach then going to best buy to get the new call of duty
8 unlocked royale with cheese in call of duty modern warfare 2 xboxtweet

so looks like one of those auto-tweet things (do they have a name yet?) people playing this on xbox have the option
to tweet about how awesomely awesome they are when they unlock some secret level.

how best to get rid of them? let’s try exact deduping of a tweet and see if they ‘go away’

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | wc -l
10000
(sanity)
bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | sort | uniq | wc -l
9773

so it’s getting rid of 230 odd tweets, not too destructive..

what about wtf2, ie ‘water on the moon’ ?

bash> ./parse_cheese_out.rb < cheese.out | g -i 'water on the moon' | h
Water on the moon... what?!? No cheese?
Water on the moon...wow. Did they find any cheese? I hope not, cause then there'll be some big ass moon rats.
Water on the moon? They probably just didn't realize this "water" is actually cheese.
RT @jdickerson: The significant amount of water on the moon means, of course, that the cheese is mozzarella.
so the moons not made of cheese? google's logo is changed to celebrate the discovery of water on the moon
NASA finds water on the moon. "Moon River" composed in 1961. So, why is NASA surprised? Anyway, can't make cheese without water, DOH.
"Significant water on the Moon" - so it looks like grey Swiss cheese, but with a consistency closer to mozarella rather than Parmesan...
Forgive my lack of awe at the news of water on the moon. I mean, used to think it was cheese. Water's not as cool as cheese.
RT @Grundy: Forgive my lack of awe at the news of water on the moon. I mean, used to think it was cheese. Water's not as cool as cheese.
RT @berkun: Of course there is water on the moon. The moon is made of cheese, and cheese has water in it.

ahha, i see. comical reactions to nasa recent lcross impact report.

hypothesis

1. if i had more data and did a sliding window analysis of say a few days then ‘water on the moon’ would appear and disappear as a trending topic when dealing with cheese
2. people would be tweeting less about chuck e cheese in the middle of week. need more data to check this one…

let’s try an hugmongous 10gram check

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | s | u | ./cheese_grams.rb 10 | s | u -c | s -n | t
8 is something that doesnt matter unless you are a cheese
8 mad ur man because he wont pull his dick out
8 man because he wont pull his dick out in chucky
8 ur man because he wont pull his dick out in
9 dick out in chucky cheese so u can give him
9 he wont pull his dick out in chucky cheese so
9 his dick out in chucky cheese so u can give
9 out in chucky cheese so u can give him some
9 pull his dick out in chucky cheese so u can
9 wont pull his dick out in chucky cheese so u

this time. really. wtf?
( i can’t believe i’m actually going to do this next grep… )

bash> ./parse_cheese_out.rb < cheese.out | g -i 'pull his dick out' | h
RT @KevinHart4real #youKnowurahoeif U get mad @ur man becuz he won't pull his dick out n chucky cheese so u can give him head! Boom..oh my
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
RT :@KevinHart4real #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
#youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head! (via @KevinHart4real)
RT @KevinHart4real #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some hea ...
#youKnowurahoeif you get mad at cho boyfriend cuz he won't pull his dick out in chucky cheese so u can give him some head! BOOOOOOM !!!!!
RT @KevinHart4real #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head! Bo
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!

so retweets are a bit adhoc and it seems that they aren’t pulled out as part of the deduping

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | s | u | g 'pull his dick out'
gangstressb youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head boooom
ha rt kevinhart4real youknowurahoeif u get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head
ha rt kevinhart4real youknowurahoeif u get mad ur man cause he wont pull his dick out n chucky cheese so u can give him sum head boom
kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head ewww
rt kevinhart4real youknowurahoeif u get mad ur man becuz he wont pull his dick out n chucky cheese so u can give him head boom oh my
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some hea
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head bo
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head boooom
youknowurahoeif you get mad at cho boyfriend cuz he wont pull his dick out in chucky cheese so u can give him some head boooooom
youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head via kevinhart4real

observations:

1. people retweet in a quite adhoc fashion ( i guess this is why twitter has recently introduced the retweet feature? )
2. ppl abbrev however they can 2 get a tweet 2 140 chars. all the tweets have ’so u can’ instead of ’so you can’ so i’m assuming that was in the original tweet. but ‘because’ has been shortened to ’cause’ or even ‘cuz’ so that peoples additional insightful commentary (such as ‘ewww’ or ‘booom’) could be appended… more statistically synonym building oppurtunities!
3. people left it as ‘chucky cheese’ instead correcting it back to ‘chuck e cheese’

anyways seems like removing all retweets is arguable. it would depend on what you’re trying to analyse for…
let’s give it a crack anyways and use a simple retweet check (ie starts with ‘rt ‘)

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 10 | s | u -c | s -n | t
5 pull his dick out in chucky cheese so u can
5 savory pies ham and cheese breakfast quiche submitted by bobkat2000
5 the early bird may get the worm but the second
5 wont pull his dick out in chucky cheese so u
6 and savory pies ham and cheese breakfast quiche submitted by
6 clue thin slices of stuffed with cheese and ham and
6 get the worm but the second mouse gets the cheese
6 quiche and savory pies ham and cheese breakfast quiche submitted
6 slices of stuffed with cheese and ham and then sauteed
6 thin slices of stuffed with cheese and ham and then

looks like people are retweeting some kind of recipe…

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | s | u | g 'thin slices of stuffed'
mystery phrase is v a b u clue thin slices of stuffed with cheese and ham and then sauteed
mystery phrase is v a c o b eu clue thin slices of stuffed with cheese and ham and then sauteed
mystery phrase is vea c do b eu clue thin slices of stuffed with cheese and ham and then sauteed
mystery phrase is vea cordo b eu clue thin slices of stuffed with cheese and ham and then sauteed
mystery phrase is veal cordo b eu clue thin slices of stuffed with cheese and ham and then sauteed
mystery phrase is veal cordo bleu clue thin slices of stuffed with cheese and ham and then sauteed

??? maybe it’s not a retweet? mystery phrase? is this some kind of game???
seems so, check out http://twitter.com/tweet_words

this type of duplication would require a more complex deduping; perhaps shingling or simhash bucketing. for another day….

so in summary the final 1 to 4 grams are….

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 1 | s | u -c | s -n | t
1336 is
1465 my
1597 of
1619 with
2065 to
3157 a
3161 i
3408 the
4175 and
10213 cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 2 | s | u -c | s -n | t
310 of cheese
359 the cheese
362 e cheese
375 chuck e
402 grilled cheese
409 n cheese
431 & cheese
445 cream cheese
796 cheese and
1018 and cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 3 | s | u -c | s -n | t
72 wine and cheese
75 ham and cheese
77 egg and cheese
81 a grilled cheese
84 with cream cheese
94 at chuck e
203 mac & cheese
300 mac n cheese
305 mac and cheese
342 chuck e cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 4 | s | u -c | s -n | t
21 with cream cheese frosting
22 chuck e cheese with
23 and mac and cheese
24 a grilled cheese sandwich
25 bacon egg & cheese
28 mac n cheese and
31 mac and cheese and
32 bacon egg and cheese
57 to chuck e cheese
87 at chuck e cheese

todos:

1. collect some data and compare use over a weekend compared to a weekday compared to the entire week
2. employ a more advanced deduping algorithm, i’ve been looking for an excuse to try some modifications to my sketching algorithms

to be continued, who knew cheese could be so much fun!!!

it compares 4.7e6 tweets with any location vs  320e3 tweets with identifiable lat lons

some interesting observations with unanswered questions…

1. the ebb and flow is not just a result of the time of day for high twitter traffic areas. the reduction between 06:00 and 10:00 comes close to zero. this is false, there is never a worldwide time when internet traffic hits zero. does twitter turn down it’s gatdenhose for capacity reasons?
2. the number of tweets with lat lons are correlated to those without EXCEPT past 17:00 where the lat lon cases drop drastically. have a couple of ideas banging around my head why this is the case but nothing concrete. any ideas?

speaking of correlation here’s a scatterplot of tweets with lat lons vs without. we can see that time period uncorrelatedness that occurs past 17:00 as a quite obvious cluster.

and here is the R code for these graphs