brain of mat kelcey...

people tweet about all sorts of stuff.

sometimes it's really important ground breaking world changing stuff... but most of the time it's ridiculous waste of time stuff like 'i ate some cheese'

in fact how much do people actually tweet about cheese? and when they do, what are the most important cheese related topics?

lets gather some data...

bash> curl -s -u user:pasword http://stream.twitter.com/1/statuses/filter.json?track=cheese

let's poke around, but first some l33t hax0r bash aliases for the sake of brevity

alias t='tail'
   alias h='head'
   alias s='sort'
   alias u='uniq'
   alias g='grep'

bash> ./parse_cheese_out.rb < cheese.out | h
   Pasta with pesto and cheese. Some watermelon but alas did not get to the salad. At least not yet. http://twitpic.com/pf1rr
   guess imma have a steak n cheese since a certain sum1 stunntin in bring me chines
   @souljaboytellem Cheese and Bread XD
   Great tips for awesome Pizza. Who loves Pizza. Cheese Matters http://bit.ly/42xkZS
   Hey bring that cheese.
   Milwaukee airport officially 10x better than Lambert. Not surprisingly, a large cheese selection. Kind of a cheese motif in the shops here.
   http://twitpic.com/pf1pl - The Cheese Cake is served.
   LA has amazing restaurants. had the best grilled cheese (w/sharp cheddar, gruyere and dijon mustard) at comme ca for lunch with @maxwanger
   Yummy pepperoni and bleu cheese pizza consumed. Next up a glass of wine. Yay weekend!
   Applebees take-out flow aka Dinner for 1- Grilled Chicken/Shrimp and cheddar jack mac & cheese.

bash> ./parse_cheese_out.rb < cheese.out | h | ./sanitise.rb | ./cheese_grams.rb 1 | s | u -c | s -n | t
    2 in
    2 of
    2 with
    3 for
    3 not
    3 pizza
    4 the
    5 a
    5 and
   11 cheese

observations;

1. it's no surprise that cheese is the most frequent since it was our search term.
2. the most frequent words that aren't cheese, 'and', 'a' and 'the', are classic english constructs.
3. the next most frequent term is 'pizza'. is pizza the biggest cheese use? well, in these 10 tweets perhaps...

bash> ./parse_cheese_out.rb < cheese.out | h | ./sanitise.rb | ./cheese_grams.rb 2 | s | u -c | s -n | t
   1 to the
   1 up a
   1 watermelon but
   1 who loves
   1 wine yay
   1 with maxwanger
   1 with pesto
   1 w sharp
   1 yay weekend
   1 yummy pepperoni

how about bigrams across 100 tweets instead of 10?

bash> ./parse_cheese_out.rb < cheese.out | h -100 | ./sanitise.rb | ./cheese_grams.rb 2 | s | u -c | s -n | t
    4 mac and
    4 nacho cheese
    5 & cheese
    5 cheese and
    5 grilled cheese
    5 i love
    6 cheese cake
    6 hot pocket
    6 with cheese
   14 and cheese

more (arguable) observations

1. when people are making something with cheese they say 'X and cheese' more than 'cheese and X' ( from the raw frequencies )
2. people like cheese more than they dislike it; ( the only sentiment is 'love' )
3. 'with' is a reasonable synonym for 'and' ( since they are the two most frequent bigrams ending in cheese )

let's try 3grams...

bash> ./parse_cheese_out.rb < cheese.out | h -100 | ./sanitise.rb | ./cheese_grams.rb 3 | s | u -c | s -n | t
   3 mac & cheese
   3 not so delicious
   3 pocket next time
   3 pocket not so
   3 so delicious as
   3 sounds better go
   3 the ham and
   3 wine and cheese
   3 you like cheese
   4 mac and cheese

from wikipedia... "Macaroni and cheese (also referred to as macaroni cheese in the United Kingdom and mac 'n' cheese in parts of the United States and Canada)" so macaroni cheese is pretty popular! at least in the 3gram space, at this time of day, which is late afternoon in the united states timezone

does this pattern continue as we consider more tweets? say 1000?

bash> ./parse_cheese_out.rb < cheese.out | h -1000 | ./sanitise.rb | ./cheese_grams.rb 3 | s | u -c | s -n | t
    7 macaroni and cheese
    7 philly cheese steak
    8 on the moon
    9 steak and cheese
   10 ham and cheese
   11 wine and cheese
   28 chuck e cheese
   33 mac and cheese
   33 mac & cheese
   48 mac n cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | ./cheese_grams.rb 3 | s | u -c | s -n | t
    74 wine and cheese
    77 a grilled cheese
    78 egg and cheese
    78 ham and cheese
    81 with cream cheese
    89 at chuck e
   194 mac & cheese
   294 mac and cheese
   301 mac n cheese
   337 chuck e cheese

there is an overlap between 'at chuck e' and 'chuck e cheese' so let's try 4 grams...

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | ./cheese_grams.rb 4 | s | u -c | s -n | t
   27 mac n cheese and
   27 water on the moon
   28 bacon egg & cheese
   28 mac and cheese and
   29 call of duty modern
   29 duty modern warfare 2
   29 of duty modern warfare
   32 bacon egg and cheese
   58 to chuck e cheese
   82 at chuck e cheese

1. more people tweet when they're at chucke cheese than when they're planning to go there.
2. 'call of duty modern warfare 2' wtf1?
3. 'water on the moon' wtf2?

bash> ./parse_cheese_out.rb < cheese.out | ./sanitise.rb | g 'call of duty' | h | s | u -c
   1 i unlocked the royale with cheese achievement on call of duty modern warfare 2
   1 making stromboli with tomoto cheese and spinach then going to best buy to get the new call of duty
   8 unlocked royale with cheese in call of duty modern warfare 2 xboxtweet

how best to get rid of them? let's try exact deduping of a tweet and see if they 'go away'

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | wc -l
   10000
   (sanity)
   bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | sort | uniq | wc -l
   9773

what about wtf2, ie 'water on the moon' ?

bash> ./parse_cheese_out.rb < cheese.out | g -i 'water on the moon' | h
   Water on the moon... what?!? No cheese?
   Water on the moon...wow. Did they find any cheese? I hope not, cause then there'll be some big ass moon rats.
   Water on the moon? They probably just didn't realize this "water" is actually cheese.
   RT @jdickerson: The significant amount of water on the moon means, of course, that the cheese is mozzarella.
   so the moons not made of cheese? google's logo is changed to celebrate the discovery of water on the moon
   NASA finds water on the moon. "Moon River" composed in 1961. So, why is NASA surprised? Anyway, can't make cheese without water, DOH.
   "Significant water on the Moon" - so it looks like grey Swiss cheese, but with a consistency closer to mozarella rather than Parmesan...
   Forgive my lack of awe at the news of water on the moon. I mean, used to think it was cheese. Water's not as cool as cheese.
   RT @Grundy: Forgive my lack of awe at the news of water on the moon. I mean, used to think it was cheese. Water's not as cool as cheese.
   RT @berkun: Of course there is water on the moon. The moon is made of cheese, and cheese has water in it.

hypothesis

1. if i had more data and did a sliding window analysis of say a few days then 'water on the moon' would appear and disappear as a trending topic when dealing with cheese
2. people would be tweeting less about chuck e cheese in the middle of week. need more data to check this one...

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | s | u | ./cheese_grams.rb 10 | s | u -c | s -n | t
   8 is something that doesnt matter unless you are a cheese
   8 mad ur man because he wont pull his dick out
   8 man because he wont pull his dick out in chucky
   8 ur man because he wont pull his dick out in
   9 dick out in chucky cheese so u can give him
   9 he wont pull his dick out in chucky cheese so
   9 his dick out in chucky cheese so u can give
   9 out in chucky cheese so u can give him some
   9 pull his dick out in chucky cheese so u can
   9 wont pull his dick out in chucky cheese so u

bash> ./parse_cheese_out.rb < cheese.out | g -i 'pull his dick out' | h
RT @KevinHart4real #youKnowurahoeif U get mad @ur man becuz he won't pull his dick out n chucky cheese so u can give him head! Boom..oh my
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
RT :@KevinHart4real #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
#youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head! (via @KevinHart4real)
RT @KevinHart4real #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some hea ...
#youKnowurahoeif you get mad at cho boyfriend cuz he won't pull his dick out in chucky cheese so u can give him some head! BOOOOOOM !!!!!
RT @KevinHart4real #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head! Bo
RT @KevinHart4real: #youKnowurahoeif you get mad @ ur man because he won't pull his dick out in chucky cheese so u can give him some head!

so retweets are a bit adhoc and it seems that they aren't pulled out as part of the deduping

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | s | u | g 'pull his dick out'
gangstressb youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head boooom
ha rt kevinhart4real youknowurahoeif u get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head
ha rt kevinhart4real youknowurahoeif u get mad ur man cause he wont pull his dick out n chucky cheese so u can give him sum head boom
kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head ewww
rt kevinhart4real youknowurahoeif u get mad ur man becuz he wont pull his dick out n chucky cheese so u can give him head boom oh my
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some hea
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head bo
rt kevinhart4real youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head boooom
youknowurahoeif you get mad at cho boyfriend cuz he wont pull his dick out in chucky cheese so u can give him some head boooooom
youknowurahoeif you get mad ur man because he wont pull his dick out in chucky cheese so u can give him some head via kevinhart4real

observations:

1. people retweet in a quite adhoc fashion ( i guess this is why twitter has recently introduced the retweet feature? )
2. ppl abbrev however they can 2 get a tweet 2 140 chars. all the tweets have 'so u can' instead of 'so you can' so i'm assuming that was in the original tweet. but 'because' has been shortened to 'cause' or even 'cuz' so that peoples additional insightful commentary (such as 'ewww' or 'booom') could be appended... more statistically synonym building oppurtunities!
3. people left it as 'chucky cheese' instead correcting it back to 'chuck e cheese'

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 10 | s | u -c | s -n | t
   5 pull his dick out in chucky cheese so u can
   5 savory pies ham and cheese breakfast quiche submitted by bobkat2000
   5 the early bird may get the worm but the second
   5 wont pull his dick out in chucky cheese so u
   6 and savory pies ham and cheese breakfast quiche submitted by
   6 clue thin slices of stuffed with cheese and ham and
   6 get the worm but the second mouse gets the cheese
   6 quiche and savory pies ham and cheese breakfast quiche submitted
   6 slices of stuffed with cheese and ham and then sauteed
   6 thin slices of stuffed with cheese and ham and then

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | s | u | g 'thin slices of stuffed'
   mystery phrase is v a b u clue thin slices of stuffed with cheese and ham and then sauteed
   mystery phrase is v a c o b eu clue thin slices of stuffed with cheese and ham and then sauteed
   mystery phrase is vea c do b eu clue thin slices of stuffed with cheese and ham and then sauteed
   mystery phrase is vea cordo b eu clue thin slices of stuffed with cheese and ham and then sauteed
   mystery phrase is veal cordo b eu clue thin slices of stuffed with cheese and ham and then sauteed
   mystery phrase is veal cordo bleu clue thin slices of stuffed with cheese and ham and then sauteed

this type of duplication would require a more complex deduping; perhaps shingling or simhash bucketing. for another day....

so in summary the final 1 to 4 grams are....

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 1 | s | u -c | s -n | t
   1336 is
   1465 my
   1597 of
   1619 with
   2065 to
   3157 a
   3161 i
   3408 the
   4175 and
   10213 cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 2 | s | u -c | s -n | t
   310 of cheese
   359 the cheese
   362 e cheese
   375 chuck e
   402 grilled cheese
   409 n cheese
   431 & cheese
   445 cream cheese
   796 cheese and
   1018 and cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 3 | s | u -c | s -n | t
   72 wine and cheese
   75 ham and cheese
   77 egg and cheese
   81 a grilled cheese
   84 with cream cheese
   94 at chuck e
   203 mac & cheese
   300 mac n cheese
   305 mac and cheese
   342 chuck e cheese

bash> ./parse_cheese_out.rb < cheese.out | h -10000 | ./sanitise.rb | g -v "^rt " | s | u | ./cheese_grams.rb 4 | s | u -c | s -n | t
   21 with cream cheese frosting
   22 chuck e cheese with
   23 and mac and cheese
   24 a grilled cheese sandwich
   25 bacon egg & cheese
   28 mac n cheese and
   31 mac and cheese and
   32 bacon egg and cheese
   57 to chuck e cheese
   87 at chuck e cheese

1. collect some data and compare use over a weekend compared to a weekday compared to the entire week
2. employ a more advanced deduping algorithm, i've been looking for an excuse to try some modifications to my sketching algorithms