This post was planned at the end of last year with some weekend hours on nltk and intention of comparing with Lingpipe/Gate. Unfortunately the latter exercise never finished.
In general I have gone ahead and cleaned up some adjectives, articles, preposition etc which do not add any value in analysis. Idea was to see if it is possible to find out more about a person from tweets and. This was done at end of 2009 but got around to writing an analysis just now.
What are the pair of words which get most mention?
| Tweetid |
Word pair |
| Gulpanag & |
please visit; visit http://gulpanag.net; Col Shamsher; thank you:; happy birthday; Sub Himalayan; Shamsher Singh; Whatever happened; welcome aboard; long term; highly motivated; #skoda #yeti; cotton farmers; district #Fatehgarhsaheb; films please; #ndtvtechlife awards; sub committee; right wing; Looking forward; last night;
#mulitpolar #balanceofpower; ‘Chad Gaya; @pavanduggal: #indiancyberlaw; @viveksmisra HTC; Berlin wall; Gaya upar; Hindi-vear; Hot Tub; Rajya Sabha |
| Iamsrk * |
insha allah; dont know; looking forward; red chillies; feel awful; low fat; human being.; take care; six pack; sleep now.; really nice.; access version; good enuff.; happy birthday; really nice; many things; keep awake; mani sirs; medicinal properties; pradeep dhoot; push ups |
| Shashitharoor |
Lok Sabha; Rajya Sabha; Chandran Tharoor; copy jacob@tharoor.in; 140 characters; Fareed Zakaria; last night; old friend; don’t know; Irfan Pathan; Passport Seva; yrs ago; Shihab Thangal; Foreign Minister; Indian community; Rashtrapati Bhavan; Seva Kendras; next year; Kapil Sibal; Indira Gandhi; Excellent mtg; Indian Ocean; haven’t seen;
latest story; Tharoor Foundation; long day; roster duty; Youth Congress; pls write; foreign policy |
| Virsanghvi # |
via @addthis; Many thanks; Rude Food; @addthis Rude; @addthis Counterpoint; Bal Thackeray; Parallax View; couldn’t agree; Vir Sanghvi:;; Manu Sharma; Shiv Sena; Raj Thackeray;Amar Singh; agree entirely; good one; foreign policy; @addthis Pursuits; Discovery Travel; Buck Stops; don’t think; next time; Fair enough.; Air India; Shashi Tharoor; Indira Gandhi; Custom Made; much. Glad; upset stomachs; good idea. |
| Sardesairajdeep~ |
cnn ibn; write in.; shiv sena; amar singh; don’t miss; per cent; karan thapar; devils advocate; shashi tharoor; aam admi; sri lanka; lalit modi; news docu; ibn lokmat; prime time; real heroes.; three idiots; womens bill; big story:; must watch; looks like; breaking news:; big story; must confess; indo-pak talks; udhav thackeray; mani shankar; phone tapping |
| Rajivmakhni |
gadget guru; Kareena Kapoor; ndtv 24×7; Steve Waugh; Xperia X10; ndtv profit; iPhone 3GS; notion ink; size zero; next week; Gadget Guru; much appreciated; NDTV 24×7.; Windows Phone; #ndtvtechlife awards; number portability; sea link; NDTV 24×7; Newsnet today; quick review; @ankitv @abhishektelang; proud moment; ink adam; awards jury; ndtv india; newsnet today; Apple iPad; 100 invites; @RahulDX @ankitv; super thin |
| Lintool |
#hadoop #mapreduce; Maryland: http://bit.ly/caJi3h; looking forward; @brandynwhite talking; compute vision;
#Hadoop #MapReduce; vision applications; language processing; natural language; object creation; #hadoop #MapReduce; #mapreduce HUG; “explode” mapper; @cloudera @philz42; Marginal Relevance; Maximal Marginal; [pwsim algo]; interactive Maximal; key space; scale natural; 8GB input; FAST 2010; large scale; @kevinweil
@abdur; input key; seem like; #Hadoop #MapReduce:; @deliprao @ChrisDiehl; it’s like |
| Jboner$ |
Looking forward; Scala Days; available: http://bit.ly/9HzbUn; ‘receive’ method.; @RayRoestenburg: Blogged; Message Routing:; Patterns’: http://is.gd/c5itt; Routing: Part; art: http://bit.ly/civmvk; highly concurrent; main diff; last night; Great talk; Actor’s ‘receive’; Actor’s logic; Stability Patterns’:; That’s true.; @remeniuk Akka; Akka HotSwap; real state; @pavlobaron @mknittig; @ManoMarks thanks; #Akka Message; ; @djspiewak Multiverse; good idea.; much better; would love |
| Tunkuv |
Carl Levin; David Frum; Daily Beast; Charles Leerhsen; Hurt Locker; Parties http://www.thedailybeast.com/blogs-and-stories/2010-04-15/at-last-the-truth-about-tea-partiers/; Tea Parties; all-volunteer army;
Paul Stevens; dear old; foreign policy; illegal immigration; turns readers; vituperative morons; online medium; John Batchelor; New York; Milton Friedman; doesn’t mean; #healthcare reform; John Paul; can’t decide; Show tonight |
| Abdur% |
Red Dwarf; Anyone else; Looking forward; next time; pretty cool; Happy Birthday; Indian food; new years; pretty sure; taxi magic; Pretty cool; sit next; cant wait; late night; Jack Falstaff; Steve Blank; Stinson Beach; Web 2.0; fresh wasabi; pie chart; last night; San Francisco; Wine tasting; alarm clock; dim sum; kitty hospital; Great dinner; screwing us.; rental cars; |
&removed- SJOBA Sub
*removed – kwon doh; lahe wah; minal lahe; nasrun minal; taekwon; wah fatahun; artemis fowl; dash hulk; hulk kai;
#removed – @addthis Parallax
~removed – cnn ibn:; cnn ibn.;
%removed-”The Art
$removed- I’ll add; I’m sure; blog post
Strawman Analysis :
Abdur/Lintool sort of belong to one extreme of folks analyzed. They could not be more different. Abdur – coming from twitter-posts very little information processing/analysis stuff and it is mostly about food, wine and personal stuff. Lintool clearly comes out as the hadoop-mapreduce man. Abdur also comes out harassed by united, taxi.
Jboner undoubtedly provides credential to his scala /akka background. He is passionate about akka, scala and his friends @debashishg, @viktorlang, @sbtourist. Akka goes nicely with scalability, availability, actor, concurreny. He also points out lot of presentations and slide decks in general.
Tunkuv comes out swinging as conservative who is hooked to cricket and still has patience to go through cricinfo and blast them for shoddy writing. Being a conservative he also has views on the healthcare and policies of the us administration. Off course tea party is the slogan of the opposition so makes an appearance.
Rajiv Makhni is no doubt “the gadget man”. He is also part of the media crowd who put “breadcrumbs” to their tv shows, writings or in general fellow folks. There is lot of link back to parent firm – NDTV and its various shows that is. He pretty much looks obsessed with phones, devices like ipad/tablet, dimensions and features. Good thing about Rajiv is that he does most of the hard work and in general not immediately vowed by apple, Microsoft. His latest coverage about “other” brands of phones is genuine attempt to provide voice to alternative (micromax).
Rajdeep Sardesai too talks about his channel, upcoming show(requests to watch with timings and all) and then some comments on current happenings across amar singh, thackreys, shashitharoor(man did the media guys milk him?). He also points out Sagarika’s articles from time to time. This is apparent in urls that he points too. I have not shared them here but extraction of urls and crawl to them provided the base for that conclusion.
Virsanghvi is unique. Virsanghvi has hands full with pointers to his articles via @addthis. He does cover wider array of topics from Padma Bhushan fiasco , headley, china/Pakistan and love of thackrey/Pawar family. I wished I could do sentiment analysis of those tweets most of which are full of sarcasm. He made it a point during Tharoor gate to pin Tharoor followers with his unique insights on folly of being Tharoor and his followers. He followed that with one page writeup in of the magazines basically saying Tharoor was pretty much product of vanity. He also made it personal wrt Mumbai vs Bombay and gave two hoots to the fact that he or his sort who speak French or Italian in 2 day visit that they go to those countries but consider it below themselves to learn a local language(may be he will pick up a cue from Aamir Khan who is learning it or Rajnikant who learnt/mastered local language). In case of food HOMP on “ndtv good times” guys take his happiness in terms of genuine freshness.
ShashiTharoor – the man who was brought down by twitter unfortunately was a judicious user of tweeter. He posted his meetings, addresses and visits to his constituency, state on tweets. One can get a glimpse of his work just through his tweets. Man was transparent. He did have cricket on his radar though. Unlike media folks he did not pre-judge and pass comments or forwarded to his blog/writing etc. Only time he opened about personal life was about his son and his achievements just like any other father would. His ministerial work too gets prominence though he did keep his boundary wrt to foreign affair. One can clearly see man had India on his mind(this is without stemming so could be really lot more). He too comes out pretty savvy re-tweeter. But his focus is clear – India, Kerala, meetings, visits and foreign affairs. Cricket does popup but just engulfed by all the meetings, briefings and mentions about generally issues that is tackling or good things he sees.
Shahrukh comes out as genuine person who wants to connect with his fans and shares tips to youngsters about good/bad habits and need to focus on work. He is extremely dedicated to his kids and family and that comes out pretty strongly. His faith also makes appearance through his invocation of insha allah phrase(basically as/if god wishes). His sleep pattern makes an appearance and one tweet will be about the need of sleep in early morning when he goes to sleep. Exercise, healthy food and fitness make way on the tweets at times. His firm red chillies, his team kkr make brief seasonal appearance. He is also the only one who dwells on his past – mother and father – again re-affirming his attachments. His language also comes out pretty dominated with verbs and action reflecting his persona? Shahrukh as mentioned earlier comes out swinging for family(kids), friends(karan), cities(Mumbai, Delhi) and his present disposition (Kolkata knight riders or movie ra.one).
Gulpanag comes across as refreshing change with interest in riding, biking, automobiles and across issues of the time. This ranges from “army scam”, “farmer plight”, “girl child”, eco friendliness and in general fitness (running,bike).
There is gentleman whose tweet analysis I have not done in detail here. But Pankaj Pachauri comes across as genuine person with nuanced interests across varied subjects. His tweets and his programmes are both high quality. He is very few of the journalist who presents facts and analyzes situation of price increase, local manufacturing, Africa mining, gas price increase, education cost and relation to enterprenuer ship. But unfortunately as it would happen his shows are mistimed and like the regional newspapers and good folks there he is ignored.
The word collocation as the above exercise is called can be easily combined with actual word frequency across tweets. In this case I have not taken trouble to normalize the data with synonyms, homonyms or polynyms. I just intersected across corpus of wordnet and brown. There is slight difference between frequencies of regular text tokenization and custom tokenization. For our purpose we will ignore regular text tokenization. Name and place extraction via alchemy, evri did not work very well, nor did sentiment analysis for singular tweets. The other challenge was lot of new “sms/tweeter” words like hv, 4, lrnt, mv etc. These words need to be in dictionary replaced with actual words. Another idea would be – do common factor search across a bunch of folks (via # tags first and then other simpler things)
NLTK is useful kit to do quick 2 hrs work. It has its quirks and I have not exploited its whole NLP side of the world. One of the days J.
Tweets downloaded in Dec 2009 and appended in May 2010.
Twitter api – http://code.google.com/p/python-twitter/ (python 2.6)
How many times word was tweeted (rough and not complete for space)
| Tweet Id |
Word frequency |
| Gulpanag |
@gulpanag -> 39
please -> 38
http://gulpanag.net -> 37
don’t -> 35
about -> 32
visit -> 30
think -> 26
being -> 25
should -> 25
india -> 24
check -> 22
thanks -> 44
women -> 21
twitter -> 18
world -> 17
always -> 16
happy -> 16
welcome -> 16
great -> 15
lol!! -> 14
films -> 13
indian -> 13
bombay -> 12
chandigarh -> 12
#fail -> 11
#fatehgarhsaheb -> 11
#iphone -> 10
#punjab -> 10
phone -> 10
really -> 10
riding -> 10 |
| Iamsrk |
kids -> 101
thanx -> 70
life -> 69
think -> 69
sleep -> 61
should -> 53
great -> 48
being -> 47
really -> 45
happy -> 42
films -> 36
world -> 36
write -> 36
always -> 35
ra.one -> 35
shoot -> 35
people -> 31
start -> 31
everyone -> 30
never -> 30
still -> 29
better -> 26
friend -> 26
without -> 25
friends -> 24
later -> 23
reading -> 23
thought -> 23
doing -> 22
sometimes -> 22
time. -> 22
believe -> 21
film. -> 21
karan -> 21
looking -> 21
@kkriders -> 20
family -> 20
kolkata -> 20
mumbai -> 20
didnt -> 19
please -> 19
sleep. -> 19
sorry -> 19
them. -> 19
things -> 19
anyone -> 18
insha -> 18
thing -> 18
today -> 18
twitter -> 18
allah -> 17
early -> 17
makes -> 17
match -> 17
right -> 17
trying -> 17
watch -> 17
which -> 17 |
| ShashiTharoor |
indian -> 124
india -> 164
kerala -> 113
@shashitharoor -> 82
delhi -> 64
great -> 64
tweet -> 60
visit -> 55
addressed -> 46
minister -> 45
write -> 44
dinner -> 43
foreign -> 43
always -> 42
people -> 42
world -> 41
today -> 40
cricket -> 39
public -> 37
state -> 36
sorry, -> 35
spoke -> 35
official -> 34
speech -> 32
congress -> 29
excellent -> 29
lunch -> 28
visited -> 27
office@tharoor.in -> 26 |
| VirSanghvi |
thanks -> 349
@addthis -> 133
think -> 69
agree -> 64
indian -> 45
counterpoint -> 43
india -> 43
liked -> 36
thackeray -> 30
point -> 24
enjoyed -> 23
guess -> 23
media -> 23
really -> 22
against -> 21
pakistan -> 21
still -> 19
@pritishnandy -> 18
always -> 18
pilots -> 18
@vinkaycee -> 17
delhi -> 17
times -> 17
legal -> 16
parallax -> 16
sorry -> 16
26/11 -> 15
@thyagu2009 -> 15
bombay -> 15
fight -> 15
india’s -> 15
support -> 15
state -> 14
watching -> 14
@gulpanag -> 13
entirely -> 13
foreign -> 13
government -> 13
great -> 13
police -> 13
politicians -> 13
absolutely -> 12
action -> 12
anyone -> 12
certainly -> 12
channels -> 12
headley -> 12
padma -> 12
problem -> 12
public -> 12
twitter -> 12
china -> 11
happy -> 11 |
| Sardesairajdeep |
india -> 107
indian -> 53
watch -> 53
tonight -> 47
story -> 43
mumbai -> 41
today -> 41
cricket -> 36
great -> 33
tharoor -> 33
write -> 33
special -> 31
breaking -> 30
hockey -> 30
political -> 29
channel -> 28
years -> 28
world -> 25
media -> 24
debate -> 23
against -> 22
sachin -> 22
still -> 21
11.30 -> 20
report -> 20
truly -> 20
guess -> 19
needs -> 19
three -> 18
watching -> 18
indo-pak -> 17
terror -> 17
twitter -> 17
ibnlive.com -> 16
justice -> 16
power -> 16
prices -> 16
sunday -> 16
indians -> 15
journalism -> 15
stories -> 15
womens -> 15
karan -> 14
killed -> 14
pakistan -> 14
singh -> 14
thought -> 14
because -> 13
delhi -> 13
india’s -> 13
parliament -> 13
party -> 13
rahul -> 13
violence -> 13
attack -> 12
budget -> 12
chief -> 12 |
| RajivMakhni |
cellguru -> 68
gadget -> 44
phone -> 43
newsnet -> 39
today -> 38
about -> 33
thank -> 32
great -> 31
review -> 26
first -> 25
6.30pm -> 24
iphone -> 24
kareena -> 23
phones -> 23
india -> 21
watch -> 21
coming -> 20
kapoor -> 18
apple -> 17
steve -> 17
10.30pm -> 16
mobile -> 16
price -> 16
people -> 15
12.30 -> 13
twitter -> 13
24×7. -> 12
android -> 12
nokia -> 12
pretty -> 12
profit -> 12
@vikramchandra -> 11
market -> 11
better -> 10
content -> 10
notion -> 10
samsung -> 10
tablet -> 10
details -> 9
phone, -> 9
shows -> 9
start -> 9
xperia -> 9
amazing -> 8
garmin -> 8
microsoft -> 8
questions -> 8
quite -> 8
8.30pm -> 7 |
| Lintool |
#hadoop -> 63
#mapreduce -> 57
about -> 19
dryadlinq -> 11
@kevinweil -> 9
@abdur -> 8
@deliprao -> 7
@ian_soboroff -> 7
paper -> 7
@brandynwhite -> 5
parallel -> 5
processing -> 5
university -> 5
vision -> 5 |
| Jboner |
#akka -> 54
scala -> 23
great -> 21
#scala -> 12
clojure -> 12
concurrency -> 11
presentation -> 9
better -> 8
cool. -> 8
doing -> 8
looking -> 8
availability -> 6
awesome. -> 6
forward -> 6
interesting -> 6
looks -> 6
support -> 6
actors. -> 5
after -> 5
scalability -> 5
slides -> 5
stability -> 5
starting -> 5
still -> 5 |
| Tunkuv |
#cricket -> 14
#goldman -> 11
#healthcare -> 11
piece -> 11
@telegraphnews -> 9
about -> 8
david -> 8
indian -> 8
#india -> 6
beast -> 6
cricket -> 6
right -> 6
@guardianbooks -> 5
comment -> 5
human -> 5
interview -> 5 |
| Abdur |
great -> 81
twitter -> 91
about -> 76
thanks -> 59
should -> 46
happy -> 40
@kevinweil -> 39
people -> 37
going -> 34
anyone -> 27
reading -> 25
think -> 25
there -> 23
pretty -> 22
still -> 22
watching -> 22
@goldman -> 21
because -> 21
check -> 20
today -> 19 |
Which people get lot of mention through tweets?
| Twitter id |
Top mentions |
| Gulpanag |
@vkaul, @nitinsgr, @nithinkd, @angadc, @achitnis, @reallybuffalo, @sonaliranade, @rwac48, @arifone , @madhulata, @maheshmurthy, @ssarbjit, @acorn…@sherbir |
| iamsrk |
@kkriders, karan |
| virsanghvi |
@addthis, @vikram_sood, @pritishnandy, @vinkaycee, @thyagu2009, @itssotweet, @kanchangupta, @gulpanag |
| Sardesairajdeep |
@imangy; @visaraj; @arunraveen; @RohanBhade; @swapsdailydose;
@jinglebells27; @sidharth_madhav; @Varunrd; @St_Hill; @aurodip;
@MirzaSania; @jemin_p; @santheepnair; @bhogleharsha; |
| Rajivmakhni |
@vikramchandra, @ankitv, @gulpanag, @sachin_malhotra, @achitnis, @mariagorettiz |
| shashitharoor |
@shashitharoor, @23jacob, @ashwinsid: @ramgandhi52; @jaipurprince:;
@karmadude @chrisbrogan;@cricketwallah; @arpitamgupta:
@josephseb: @arungiri; @khalidalkhalifa:; @PARVEZ89:; |
| Jboner |
@jboner, @djspiewak, @debasishg, @viktorklang, @sbtourist, @pavlobaron |
| Lintool |
@kevinweil, @abdur, @deliprao, @ian_soboroff, @brandynwhite, |
| abdur |
@kevinweil, @goldman , @gregpass, @elizabeth, @jayvirdy, @jess, @evan,@pankaj |
| tunkuv |
@telegraphnews, @guardianbooks, @prempanicker; @saliltripathi; @ultrabrown; |
Is the user tweeter savvy and uses hash tags/lists?
| Tweeter id |
Top # tags |
| Abdur |
#sgu; #justreturnzero; #bestConfLunchOfAllTime; #chirp; #whereisbiz; #SS.; #sfcabbies; #sfcabsareslow; #spoiledbynorthercalweather; #copeyesightfail; #notquittingdayjobforstandup; #conspiracytheories; #2; #boycotunited; #tsafail; #istlecture; #tokyoiswaycool; #SantaAbuse; #ihatedish #faileddonotcalllist #endphonemarketers;
#zeroinboxfail; #mylaptopisscrewed; #3; #mustfix;#goodfoodforthought; #foo09; #lifeleasons; #Retarded; #ssm09; #sfheatfail; #fixfuckedsfcabcompanies; #TED; #7; #SQLputdowns; #cikm2008; #; #ceas |
| Gulpanag |
#quote; #shatabdi; #CWG; #Chandigarh;#amplifier; #trek; #Fatimabhutto; #ipad;
#Hockey; #t20; #delhi; #triathlon; #race; #Womenempowerment; #BRICK; #lyrics;
#jetlag; #thiruvananthapuram; #RoyalEnfield; #Kasauli; #MAC; #iPhone; #Twitter; #iPhone;
#Delhi; #AlQaeda #jem #LET;#roadrunning; #strike; #HT; #Mumbai ; #respect; #Airtel;
#fail; #sin; #sine; #indianexpress; #;#kasab; #newlylearntfact; #orkutification; #olaytotaleffects; #Ladakh; #raiddehimalya; #NationalGeographic; #migraine;
#wolframalpha; #LED; #Pvr;#reliance; #iPhone; #blackberry; #Hot; #tatasky; #Surya;#ipl; #wonderwhy; #ndtvtechlife; ; #unitedindiapak; #kindness; #siemens; #stalker; #democracy; #china; #harleydavidson; #indiancyberlaw; #KingOfGoodTimes; #harassment #; #childmarriage; #zeroaccuracy; #capitalstagnation; #africa.; #reva; #hockey |
| iamsrk |
na |
| Jboner |
#Scala; #Akka; #NodeJS; #MongoDB; #AMQP; #REST; #UnitTesting;# #javaone; #Terrastore; #jax2010; #clojure; #playframework; #camel; #github; #assembla; #maven |
| Lintool |
#SIGIR2010; #WWW2010; #MapReduce; #Hadoop ;#1; #2; #cloudcomputing; #nlp;
#LHC; #pig ;#goog;#aws; |
| rajivmakhni |
#nowwatching; #iPhone; #BBM #Appworld; #tech-fairs; #ndtvtechlife; #MycelebList; #ndtvgreenathon; |
| Shashitharoor |
#TWITronym; #India; #HiFlyers2009; #TED; #US; #nuclear; #NMST;#awesomeindianthings; #TEDIndia; #beatcancer; #Nobel; #Gandhi; #uighurs; #Jet2Kerala |
| tunkuv |
#iPad; #DavidHockney; #WorldT20; #cricket; #Scotus; #Euro #pakistan; #Beatles; #leadersdebate; #GordonBrown; #Goldman; #UKelections; #Arizona; #Belgium; #SpongebobSquarepants #Heimlich; #Britain; #WallStreet; #Facebook; #Samaranch; #TeaParty; #NBC; #Pulitzer; |
| virsanghvi |
na |
Code for analysis could be shared at git little later or shared on request.
or at least I could not make enough sense of the policy doc. I could not get a response from them and waited for reasonable time before posting it.
.
