brain of mat kelcey...

i've been doing some reading for my statistical synonyms project and have uncovered a heap of cool papers. most of them are around an idea (from the 1950's!) called the distributional hypothesis that simply states that words that appear in similar contexts often have similar meanings.

the coolest paper so far is 'Web-Scale Distributional Similarity and Entity Set Expansion' by Pantel,Crestan,Borkovsky et al which has introduced me to an area of research i didn't really know existed; entity set expansion.

entity set expansion is a bit like thesaurus building for proper nouns; given a seed set of related items can you expand the set to include other semantically similiar items?

an example might be brands of japanese motorbikes. starting with 'yamaha' and 'kawasaki' we might expect the set to be expanded to include 'honda'

i started hacking around in pig but today switched back to ruby for slightly quicker prototyping. who knows, i might give piglet a go!

the code is on github