Event – Event hosters Vinayak Hegde(Data platform owner), Sharad Agarwal (ex-yahoo,hadoop,yarn committer,  present platform head at InMobi) were punctual, humble and kind. Vinayak and Sharad provided needed time checks, context and hoped to continue the effort with help from community. Turnout was varied right from recent newhires to people with multiple decades in industry. It had lot of yahoo (no-surprise) folks, Nokia (100 node cluster), Huawei (apparenly built a HA and have deployed cluster of x nodes), NetApp (Bejoy and team with y node cluster) , Mu-sigma(evaluating and using various pieces of hadoop).  Joydeep (ex facebook, hive creator) came around to see/meetup folks. I was looking for raptor folks from Sungard though.

Sessions

Sonal on crux - This talk had two pieces – how crux allows uses api based interaction with mapping and reporting of data inside HBase. Her intention was to get people to contribute and help out build the other moving pieces.Crux at present goes directly rather than in between translation from SQL like query language. She explained how one needs to design backend carefully to ensure efficient/performant data access is done. Crux allows composite keys, filters  but eod secondary indexes and like need to be thought about by the system designer. Since Crux is just a reporting tool – it can only do so much(idea is to be nice to get/range operations – how much a db guy likes these operators – seek vs scan(killl)). Kudos to her for getting something out and talking about it.

Sharad‘s talk on next generation Hadoop clarified the present constraints and hence the goals of .23 world. HA – Restart of Namenode vs DataNode, Scalability of NameNode – sheer footprint of everything it needs to keep track of and respond to, Need to support alternative parallelizable algo-with no force fitting into MR. His talk was succint and filled with great depth. Idea of containers getting resource fullfillment from ResourceManager to get themselves created via NodeManager and then spawning off AppMaster to look after the applifecycle independently is the key.   Application lifecycle of either MR/Iterative/MPI could be managed independently and Resources can still be managed centrally. Important takeaway for older installations – no change. Their world remains the same.  Data Affinity based container spawning is possible – this looks interesting from perspective of reducing network io.

Yoda talk by Gaurav from InMobi was about the inhouse Datawarehousing/Reporting tool they built with few resources over short period of time. It was slick.  He explained the pragmatism to do custom development compared to using Hive or other tools. One of the important pivot for decision making was documentation, community support and “in-ordinate” spawning of Jobs without taking into picture metadata and layout of data. Looks like good solution to their issues and allows them evolution according to their needs.  It is niether designed as generic framework nor does it aim to be one.  This honesty from the data framework team was refreshing as they were not trying to boil the ocean and focused on their constraints (lack of massive clusters) , needs of analysts (inhouse/publishers).  It would have been great to see how they choose plans for execution, is it cost based or .., which operators they push up or down and on what basis – if it based on metadata of the data – how do they keep updating it?

From my perspective – it was also great meeting up folks like abhinasha from bizosys – thanks buddy for the beer and leading the assault on food counter.  For old person like me  - I am still looking at easier ways to adopt as end user.

- SQL DSL front end (for loading PIG is ok but presence of sqoop, scribe is explosion of choices – lot of time is spent in evaluation)

- debugging the performance of a given Query , how many combiners, partitioners, which operator gets mapped to how many jobs and how it takes care of affinity to data location (ideally less I get to know, much productive I will be), Relational DBs have made me lazier & biased , Way to extract only given amount of data.

- Monitoring/Prioritizing and concurrent access for read/write are what will get us relational folks in easier way to that world.

The day MapR, Hadapt combine & provide statspack(ish), DMVish, monitoring support, is the day real revolution will become. (For record I have not had time to look at much appreciated cloudera distribution).

There are sort of two parts of this post
1. Why Framework of anykind for js based development
2. List of Interesting frameworks/libraries/tools in no particular order

So let us see what is the issue we are talking about. Let us say server side is going to be an endpoint (native app listener)
or a controller (whether hosted natively for native app-thick client app  or zend/cakePHP/asp.net mvc/spring/seam etc), or a thrift/wcf endpoint we need

1. Logic to render the view in client browser/container or  at server via view (controller endpoint of either kind)
2. Logic for serializing/deserializing, validation (basic vs domain specific),  routing
3. Clean difference between layout & data

Options we have
1. Use of server-mvc heavy framework with js only used for  ajaxified/json-strigified talking back to ajax endpoint
2. Use of mixed environment where client side mvc is used to organize  various moving parts and use server end points for backend interaction
3. Create thin server with validation and thicker client side mvc

If we go down the path of jquery or similar tool to do client side  interaction, we have potential of large amount of code getting developed
over period of time. Jquery/ExtJS also has lot of “browser” specific code and  this needs to be weaned out.

So one needs to choose a set of helper libraries/best practices to achieve the goal.

Factors
a. Size of the framework/goals of the framework
b. Dependencies required
c. Updates and support
d. Ease of testing – selenium support – at least for the main logic
e. Tools support – perf/testing, profiler
f. Familiarity & comfort with patterns – pure mvc vs mvvm
g. Duplicate code at server/client side

Library/Framework requiring no introduction
1. jquery/prototype/dojo/mootools/ExtJS -
Use base on comfort factor of the team.  No further explanations reqd.
Restrict to specific usage and use dependency management system to load
right modules.

MVC Frameworks
1. SproutCore
Apologies I have tried but not very successful at picking this up in  actual project – my bad – it is on my todo list one of these years.

http://sproutcore.com/

2. javascriptMVC – is a real heavyweight framework in real terms and does a “lot” of things. I am still a journeyman here but looks like in terms of  above 7 points this library comes closest to what can be used. It does take  time to grow on you. One thing which will comes as surprise is its dependency on java for compression, funcunit tests & code generators(I guess).

http://javascriptmvc.com/

3. Spine
Another mvc framework which looks like backbone but is more minimalistic. It is certainly lighter than javascriptMVC but both have different audiences. Key piece is the way inheritence implemented – it uses native prototype  based inheritence. I am at present not comfortable with its model creating/requiring those IDs and its persistence back to real db. Becuase they are mentioned in the main docs – some folks think of them as main actors in spine usage. Another important piece is spine can be much better integrated with backend rest endpoint.  One needs to be careful while using it’s ajax calls – as by default everything is asynchronous. Compared to javascriptMVC I find the idioms it follows to organize/extend very organic. (I could not formulate a better sentence to convey what it means to see x.controller.create {}) . Spine  has advantage in terms of api for html5 localstorage & async operation. Folks well versed in backbone will find it simpler to adopt. One thing I do not like in terms of simplicity – init after every create(partly bec of prototypal inheritence).

http://maccman.github.com/spine/

4. Backbone
Mentioning backbone after spine could be sacriligeous for some. But here is  another beautiful framework bit loaded towards model(eod data is god). It can be used with jquery or zepto.  It’s views are more of view-controllers(my own term). It gives a structure
without tying models to the end dom – this is a big plus as you move away from jquery(trying to do many things there). Backbone uses
underscore for handling collections of models.

http://documentcloud.github.com/backbone/

5. Cappucino – Another framework out of my league, ObjectiveJ is as seductive as ObjectiveC. Serious lack of motivation has prevented me
from doing anything with it. I hear Mac only developers like the familiarity  – but again it is second hand information. Biggest advantage is no need to learn html or use it, css or manipulate  DOM. Cross compilation gets new meaning.

http://cappuccino.org/

6. Google Web toolkit – EOD write code in java and use libraries to  maximize productivity. Has lot of tools – unfortunately I have not
used them personally except bits and pieces.

http://code.google.com/webtoolkit/

Not a MVC framework but a great library to know and work with
1.Node **
Node.js is an evented I/O framework(more of library plus whole ecosystem)  for V8(the default engine).Easily the most happening event oriented server  side library for js. LinkedIn has claimed fantastic performance increase over ror  (but hey they were using ruby – not fair to compare if you are not using 1.9 at least)

http://nodejs.org/

Windows support – Node 0.5.4 supports windows build via VS using GYP http://blog.nodejs.org/2011/08/12/node-v0-5-4/

Miscellaneous libraries

1. Store.js – Resolve cross browser localstorage issues. Uses localStorage, globalStorage, and userData behavior.

https://github.com/marcuswestin/store.js

2. underscore – It gives you useful functions you might normally have in your own utililties js file like each, map, isEmpty, isElement and more. Also  described as “It’s the tie to go along with jQuery’s tux”. Everything is available through _ – no really . Functional programming grokkers will like this tool.

http://documentcloud.github.com/underscore/docs/underscore.html

3. requirejs – Handle dependencies in clean manner.  Use it to minimize loading time, organize dependencies in clear fashion. LabJs too have something similar.

http://requirejs.org/

4. socket.io -  When you need to do “realtime” communication while websockets support  evolves. Really wished present browsers could be reused rather than  large re-install with new browser. It is not fun to test for regression  for all apps in large organization. This is where socket.io comes in good. It has support for realtime goes from sophisticated(shiny-new-browser) to legacy(IE5.5/safari3). It uses range of technologies to provide graceful degradation for experience across available browsers and devices.

http://socket.io/#how-to-use

Code to browse – https://github.com/LearnBoost/socket.io/blob/master/support/node-websocket-client/lib/websocket.js
Line 442 onwards is where one should concentrate for understanding key stuff.

Various transports supported

https://github.com/LearnBoost/socket.io/tree/master/lib/transports

Silverlight support-http://groups.google.com/group/socket_io/browse_thread/thread/6f6d8575ae887829
Great thing for SL team to implement as plugin.

5. coffeescript
Another code generator for javascript – love child of python/ruby. I  personally stay away from more abstractions.Good stuff – all variables
are in local lexical scope, no pollution of global scope http://jashkenas.github.com/coffee-script/

6. LabJS – Another asynchronous script loader, allows as much parallel  load limited by browser settings. One can easily specify which
scripts have execution order dependencies and LABjs will ensure proper  load order. Small size comes in handy.

http://labjs.com/

7. Zepto – Mobile/jquery friendly framework for webkit browsers. Main reason  to suggest this would be that other frameworks have lot of code for browsers not available on client devices (IE on iOS/Android as of today). In spirit  of doing the right thing – suggested doing custom work or use Zepto kind of framework. It uses mobile webkit features and is very small – 2k, has support for major gestures.

http://github.com/madrobby/zepto

8. CacheProvider – from twitter’s Dustin Diaz – allows you to cache  data both in memory and using localStorage if it’s available.

http://www.dustindiaz.com/javascript-cache-provider

9. NowJS – One more node goodness to keep track of. Node has become the tsunami of  all good things. Aim of NowJS is to make real-time applications easier to develop by synchronising functions and variables in a namespace  between clients and a server.

http://nowjs.com/

10. jsFiddle – Docs http://doc.jsfiddle.net/

http://jsfiddle.net/MNGpr/

Templates
This is a topic which ignites passion as people cling to them like their shadow. Do you notice no mention of server side templates – it all
depends on the choice of infra you want to make.

1. Moustache -http://mustache.github.com/,http://icanhazjs.com/
2. Jquery Templates – http://api.jquery.com/category/plugins/templates/
Inspired by John resig’s template plugin
3a. jqote – http://aefxx.com/jquery-plugins/jqote/ ( No like <![CDATA[… absolutely not)
3b. render – https://github.com/atduskgreg/srender/
4. closure template – http://code.google.com/closure/templates/

Engines

1. Rhino
Reminds of script# effort and is basically java for javascript  (compiler & interpreter). Allows javascript to interact on server side
with java objects (pojo/bean…). I am not a big fan of mixing these  two worlds. Let them talk to each other at an end point and use common
serialization mechanism. Much touted DynamicInvoke feature is here  in Java 7 but it’s usage in future for Rhino -last I checked – months
before this writeup was written is not yet there.

2. SpiderMonkey – Compiler and Interpreter with mark and sweep gc  written in c. Used at Mozilla. The famous piece is the
Tracing JIT  -https://developer.mozilla.org/En/SpiderMonkey/Internals/Tracing_JIT

3. V8 – Complete VM/runtime. Since it has become part of Node it has  spread its goodness allaround.

http://code.google.com/apis/v8

I have no direct experience in usage of V8/SpiderMonkey.

Rhino I have evaluated for usage and that is how I know miniscule piece of it.

I am not making a particular OS as the requirement as I think any
programmer worth his salt can pick up and decide on usage based on his constraints.

** Node.js will be available from joyent on Windows pretty soon

[updated : 11th July 2011] : Urls of site/people/projects

[updated : 11th July 2011] : Renaming and few corrections/disclaimer.

This is not complete view as I could not attend other fabulous sessions around membase, memcache,choosing right php framework and gearman. But I did hear great things about them, hoping to catch them on videos.

Good things-

1. Event website,  content, payment process, organizers

Conceptualization of event was validated with potential speakers, their talks and votes of interest. Kiran took pains to look at public profile and github of speakers. A very nicely designed  website for potential attendees was the first temptation – really a blowback to vendor backed agency cookie cutter output. Although not all talks were focused on php or deep php , the theme connected cloud hosted php website.

Smooth checkout/payment integration was another plus. Wished there was a button, come later details will be preserved, as I struggled to justify staying out for full day away from little kiddo and other stuff as i entered details multiple times.

Another good thing was no over hype of changing the world kind of thing. A bunch of professionals who wanted to provide a reasonable venue for exchange of thoughts. More importantly no marketing only types or speaking and no work types projecting themselves as gift to humanity. This was apparent in most of the speakers especially zynga, facebook, capillary, mobstac & some yahoo folks.(after the event impression meeting/exceeding the expectation).

2. Event venue – Dharmaram college inside christ college with good parking and plenty of trees was idyllic setup. College had really good quality projectors in place. Food, tea, biscuits were good- how can I say anything bad about food which provides pineapple gojju . Restrooms were plenty and clean. Water was available when needed. Only challenge was Internet, which I personally thought was a good riddance.  Then there was event photographer Kushal Das omnipresent with his gadgetry.

3. Content & people

My expectations were  around learning best practices across the technologies. It was wonderful to see pythonistas around prepping for pycon. There were lot of offline discussions around chef, puppet, ganglia, capstriano. I saw folks coding up in both django and a template I could not make out plus pure lamp stack or just plain cgi.

Flipkart continous deployment

I personally liked flipkart’s sessions – real people sharing real insights, best practices with humility previously not seen elsewhere. Flipkart’s sharing of how they do continous deployment – major 6 + releases everyday – I thought I saw 30 , although speaker could have done with double shot of coffee , a person who knows his stuff  but sort of shy in speaking up. There are people with less than ounce of his depth who create unfortunately much perception of depth – they can talk without single shipping application.

Anyway coming back to content – main take away was use of debian packages to do deployment. They provision bunch of new machines after netboot to deploy right os, puppet to do configuration. Then finally debian packages are used to deploy the  code base, dependencies etc. Easy to roll back, audit. Some times somethings are  still done by hand(schema change). They can do rolling deployments most of the times. They do use very heavy cache and solr for search. This sessiom focussed on how every  commit results in build, deployment. They replicate as much of production environment into dev, something we push many customers at least for db size.

45 minutes were too short into get details of what tests they run, how they do perf testing, how they profile, use monitoring ( gomez/keynote) vs pingdom etc). Or dig into usage of logs to raise alerts, or dynamic monitoring of resource usage across php, jvm, nginx, ha proxy or varnish, network components. As they shared later in another session they have home made load balance running linux in ha mode with multiple network cards to allow 3 networks down. Amol  Amod (thanks Mekin) clarified lot of questions wrt deployment, again very humble down to earth folks.

Flipkart performance tips

This talk by Siddarth was worth its time in Gold. Flipkart uses php minus apache front ended by ha proxy, varnish and nginx. I am assuming they use memcache. Explanation of various options and why they chose what they had was well detailed.

I think/assume usage of solr takes off whole lot of load and thrifted services are invoked only for real work with db. Since they do not store any state anywhere, near horizontal scale is possible at web, services layer. They have interesting component in java as threaded service/deamon which runs locally on every web server. This component  is the one which does async, parallel calls, does logging etc. Imho this is really the guts and choke point of app and I really could not get the point of this abstraction as they could talk from php to thrift service(except parallel calls). Availability of this component, failover was something time did not permit to get into. Caching, performance tools across proxy,cache(static-dynamic), web server, php, jvm(any particular gc setting?), solr,cache, db was not delved in deep as time was less. It would would been awesome to get into discussions around why of templates or no templates, models in php vs abstracted behind service layer. Similar discussion around use of jquery vs other frameworks, usage of cdn would have been great. One thing I did not see was use of any queue or analytics(not really part of the topic so okay) or nosql, functional lang ( was expecting some scala vs clojure adoption) discussion. Very-2 pragmatic and generous. Almost too generous for sharing pretty much their core. Even today it is difficult to find how amazon does its website scaling inspite all talk of openness etc.  Flipkart’s another good thing was that they did not boast because they use a “tech/lang”.

Practo and usage of beanstalkd

This particular talk by practo co-founder Abhinav lal went into how he made the choice in favor of beanstalkd over rabbitmq or expensive message buses. He rightly called his usage as job queue where he pushes priority based messages into tube and has workers to process those messages. He uses ttr and persistence ( not clear whether it is automatically purged  after some time) to pull messages and process. Usage of supervisord to scale out, monitor the worker process was a neat trick he shared. Looked like he used php on worker as well as client side so serialization of data was no big deal. Overall beanstalkd came out as decent job queue and excelling at it. There is no fancy publish/subscribe, esb, jms infra on top of it. Again a great speaker.  I did bump into beanstalkd via delayedjob earlier. So good validation of “scale” vs “usage”. Again a great value for money discussion.

Failure as an option by Vijay from zynga

This was the talk delivered by a person who had guts to say things as they are. Specifically around cloud – unbound resources, elasticity, provisioning etc….basically burst lot of myths… importantly he had imp advice – it is not possible to know/control all unknowns. So at least control your destiny where you can. He adviced to have few reserved or passive instances to overcome the tyranny of ‘avalanche ‘ when cloud provider faces downtime, all clients try to provision more resources. He added other ideas around why/how to failover completely for x userbase vs half baked response to everybody. Now this requires lot of work and thinking at lot of layers – it would be ideal to apologize only to people whose service went down rather than fail whale for everybody. His suggestion of usage of ha proxy intelligently, not failing over all load of 5 machines to remaining 5 ( out of 10)if those 5 machines are at 50% + usage already. This would start cycle of provisioning and more load on already stressed infra.

Ravi pratap‘s talk turned out to be  ec101 rather than being true to title of scaling. He I am sure could have done justice to that content too as he is great speaker with lot of hands on work. His explanations were crisp and he shared one imp thing thing- they have their own ami as expected. Again time was too short for him to get into details of scaling as covering basics took longer time. But would expect at some other location he can share his tips.

Eucalyptus talk was big let down unfortunately as it would have been easier to start of with demo rather than explain cloud again…certainly not cool to run down valid competition as malware.

Python tool fabric as deployment option by capillary (Nigel Babu) was completely new subject for me.

Thrift talk by capillary(Piyush Goel) on how they use it for communication from php to java service  was good tutorial. Main adv over protocol buffers was lack of rpc in protobuf.

Again thanks to @hasgeek for conjuring up goood meet. http://phpcloud.hasgeek.in

Conducted on 9th july 2011

केव्हातरी पहाटे उलटून रात्र गेली
मिटले चुकून डोळे हरवून रात्र गेली

कळले मला न केव्हा सुटली मिठी जराशी
कळले मला न केव्हा निसटून रात्र गेली

सांगू तरी कसे मी वय कोवळे उन्हाचे?
उसवून श्वास माझा फसवून रात्र गेली!

उरले उरात काही आवाज चांदण्याचे..
आकाश तारकांचे उचलून रात्र गेली!

स्मरल्या मला न तेव्हा माझ्याच गीतपंक्ती
मग ओळ शेवटाची सुचवून रात्र गेली!

आता कुशीत नाही ती चंद्रकोर माझी..
हलकेच कूस माझी बदलून रात्र गेली

अजुनी सुगंध येई दुलईस मोगऱ्याचा..
गजरा कसा फुलांचा विसरुन रात्र गेली?

शिशिरामधल्या   उत्तर्रात्त्री

1. Deliverance 1971 – Awesome photography, editing with well rounded gritty characters. Movie becomes the river it talks about, sometimes meandering and others like its rapids but finally arrested by the mighty dam.

2. Americanization of Emily 1964 – Wonderful chirpy ode to life and peace. Some of the best quotes are at – http://www.imdb.com/title/tt0057840/quotes. Movie is filled with witty dialogues. James Garner’s dog robber portrayal is one of the best performances in terms of holding together viewer interest.

3. There will be blood 2007 – Never has a man looked so menacing on the screen after Jack Nicholson. Nope I do not care much about masks and gory makeup, it is the chilling closeup of man projecting evil. Daniel day lewis just overshadows everything I have seen from Jack N. One of the best movies ever – right up there with Coen brother’s other work. The cinematography captures the hopes and futility of being an oil enterprenuer. Daniel’s performance will be benchmark for many generations to come.

Update – 2010-Nov 4 -

4. Aura 2006 - An Argentinean movie shot entirely in patagonia region has stunning visuals. Supposedly a 2nd and last movie by the director. It is very lucid in its flow and its shot of trees swaying in forest is pure poetry. Only thing jarring was somewhat muffled sound recording – was it deliberate – because there is sudden change in movie flow/sound when the epileptic character starts going into fits. Anyway movie has heist premise which is possible but  can it be pulled off in new conditions? Again story telling of higher quality than usual fare. Another way of joining various threads than flashback/non linear story telling. Acting by the characters too is very fluid -nothing seems forced/dramatic at anytime.

Fabiàn Bielinsky (director) had another wonderful movie – Nine queens (I guess remake – criminal was not such a good film though).

Where is the information – Right now on FB.

How can one get it

It should be simple, it is public data. You just get all the posts of Delhi traffic police spend few minutes in formulating a regular expression to extract the number plates. And follow it up by co-relating it to time it was reported and time it was “posted as action taken”. You can do fancier things by extracting entities and posting them back on map. But to start with access to reports and corresponding action taken post will be enough.

Challenge – FB’s dev policy unfortunately does not allow it or at least I could not make enough sense of the policy doc. I could not get a response from them and waited for reasonable time  before posting it.

Conclusion

Twitter is better friend here.

Search firms should be allowed to gather posts of “recognized” public  entities – in this case delhi traffic police. There need not be any access to its network of people who are “friends” with it or “like it”.

I can see the challenge though where an article refers to older incident thus affecting the ability to cluster. Another challenge would be an article summarizing the attacks on RTI applicants over years but these will be very few anyway.

Apparently google predict account/apache mahout should enable this. What is involved?

1. Identifying groups/clusters

2. Cluster identification on common name, date, location (extract date of crawl, date of publishing of article, source, name of person involved in attack or death. Name extraction can be accomplished by phrase analysis of sentence mentioning the two words nearby each other and looking up subject. Similar thing that needs to be done in Delhi traffice police tweets. Location extraction also is part of earlier process. If trusted sources provide the data – a pattern can be assumed – Location. Date. “Phrase describing the incident” – followed by lot of extraneous information.

3. Disambiguating based on exclusive notifcation rather than summary/article on scores of similar incidents

Trusted source information – rather than looking at all results – search should be directed across 2-3 places and information extracted from them. reuters/bbc/pti/xinhua.

Must mention – YQL/Yahoo pipes are like dynamic languages – instant gratification and visible work.

BTW must mention – youtube’s “featured video” suggests “cannibalism video from bbc” while watching videos on Vitthal (God).  Could not explain to mom – exactly what was the science behind that one .

Screwed up "featured video"

Words which come together :

Traffic obstruction; broken down.; may contact; specific suggestions/complaints; broke down.; area specific; Raja Garden; Range

having…; DHAULA KUAN; red light; truck broke; concerned DCP/Traffic; given below:…; DTC bus; one truck; HTV broke; bus broke; flyover towards; website http://delhitrafficpolice.nic.in; Traffic heavy;  floor Bus; towards Gurgaon; towards Ashram; 1700 Hrs; Akshardham mandir; Budha Jayanti; CHOWKI NO.; G.C. Dwivedi; Jal Board

Word frequency

traffic                   ->                       157
obstruction               ->                       122
towards                   ->                        99
down.                     ->                        63
broken                    ->                        46
truck                     ->                        44
broke                     ->                        42
there                     ->                        31
flyover                   ->                        30
heavy                     ->                        17
nagar                     ->                        12
chowk                     ->                        11
contact                   ->                        11
garden                    ->                        11
specific                  ->                        11
suggestions/complaints,   ->                        11
container                 ->                        10
dhaula                    ->                        10
wazirabad                 ->                        10
ashram                    ->                         9
delhi                     ->                         9
vihar                     ->                         9
naraina                   ->                         8
cleared                   ->                         7
gurgaon                   ->                         7
hospital                  ->                         7
airport                   ->                         6
between                   ->                         6
having…                 ->                         6
mayapuri                  ->                         6
range                     ->                         6
staff                     ->                         6
aiims                     ->                         5
below:…                 ->                         5
break                     ->                         5
cleared.                  ->                         5
concerned                 ->                         5
dcp/traffic               ->                         5
e-mail                    ->                         5
flyover.                  ->                         5
given                     ->                         5
lajpat                    ->                         5
light                     ->                         5

Tweets with “red light” (shows I need to use new api to get time and complete text with help of tweet id) – 7 matches

has been cleared at Kalindi kunj red light   Traffic obstruction at kalandi kun
ffic obstruction at kalandi kunj red light due to one truck broken down   Traff
rk    Traffic heavy at Wazirabad red light from Burari towards ISBT due to heav
ffic obstruction at SHASTRI PARK red light due to a truck broken down  http //b
J  Traffic obstruction at ASHRAM RED light due to a LGV broke down  http //bit
both carriageway at Subroto park red light  http //bit ly/d8Wilc  Traffic jam a
Rohini West Metro station due to red light point fault  http //bit ly/c0fXIg  T

Tweets with “obstruction” 25 of 122 matches:
rgaon  has been removed   Traffic obstruction at MADHUBAN CHOWK TOWARDS PEERA G
GAS TANKER broken down   Traffic obstruction between naraina flyover to mayapu
ue to a HTV broken down   Traffic obstruction at Subroto Park towards Gurgaon a
crane has been directed   Traffic obstruction at Ladosarai T-point towards Khan
oved traffic normalized   Traffic obstruction at Ladosarai T-point towards Khan
due to a tree uprooted   Traffic obstruction from Naraina to mayapuri has been
oved traffic normalized   Traffic obstruction at Majnu ka tilla towards Wazirab
irabad  has been removed  Traffic obstruction from Naraina to mayapuri due to a
a container broken down   Traffic obstruction at Majnu ka tilla towards Wazirab
a container broken down   Traffic obstruction has been cleared at Dhaula Kuan
n  21 19 PM 07-July-2010  Traffic obstruction at RTR flyover underpass towards
irport has been cleared   Traffic obstruction at  RTR vacant vihar  IIT towards
rport  has been cleared   Traffic obstruction at savitri flyover towards Nehru
u place has been removed  Traffic obstruction at RTR flyover underpass towards
damper has broken down   Traffic obstruction at  RTR vacant vihar  IIT towards
damper has broken down   Traffic obstruction at savitri flyover towards Nehru
ntainer has broken down   Traffic obstruction at Birtanya choke towards Punjabi
ue to one truck has broken down   obstruction has been cleared at Wazirabad fly
ed at Wazirabad flyover   Traffic obstruction at Wazerabad flyover due to one t
e truck has broken down   Traffic obstruction at Raja Garden chowk from Tilak n
Tanker has broken down   Traffic obstruction has been cleared at Kalindi kunj
Kalindi kunj red light   Traffic obstruction at kalandi kunj red light due to
over construction work    Traffic obstruction at NAGIA PARK  CHOWKI NO  2 SHAKT

Challenge is normalized names (no spelling differences etc) and then ability to pull their geolocation from one of the apis(mostly google) and plot them back with heat map kind of thing to point out bottlneck across a day.

Looks like Naraina, Dhaula kuan, RTR flyover,Wazirabad red light, Kalindi kunj red light ,Raja Garden,NAGIA PARK are choke points. More detailed analysis to follow on another day with locations and kind of obstruction, accidents and hopefully geolocations on map.

Finally must get out the blogger crawler for solr what else. And also finish getting data about dtp’s fb postings of people who are booked and for hopefully for what offence. This data can be far easily posted by delhi police themselves in form of odata for future usage rather than just fb posts or tweets.

Overall Delhi police must be congratulated for following up on citizen requests and posting action back to fb. Nothing expensive stuff to maintain or buy except mobile phones equipped with fb/twitter app. Sure there is some nic app which is doing the storage of offenders etc. But interaction through twitter/fb is outright simple.

In general I have gone ahead and cleaned up some adjectives, articles, preposition etc which do not add any value in analysis. Idea was to see if it is possible to find out more about a person from tweets and. This was done at end of 2009 but got around to writing an analysis just now.

What are the pair of words which get most mention?

&removed- SJOBA Sub

*removed – kwon doh; lahe wah; minal lahe; nasrun minal; taekwon; wah fatahun; artemis fowl; dash hulk; hulk kai;

#removed – @addthis Parallax

~removed – cnn ibn:; cnn ibn.;

%removed-”The Art

$removed- I’ll add; I’m sure; blog post

Strawman Analysis :

Abdur/Lintool sort of belong to one extreme of folks analyzed. They could not be more different. Abdur – coming from twitter-posts very little information processing/analysis stuff and it is mostly about food, wine and personal stuff. Lintool clearly comes out as the hadoop-mapreduce man. Abdur also comes out harassed by united, taxi.

Jboner undoubtedly provides credential to his scala /akka background.  He is passionate about akka, scala and his friends @debashishg, @viktorlang, @sbtourist.  Akka goes nicely with scalability, availability, actor, concurreny.  He also points out lot of presentations and slide decks in general.

Tunkuv comes out swinging as conservative who is hooked to cricket and still has patience to go through cricinfo and blast them for shoddy writing.  Being a conservative he also has views on the healthcare and policies of the us administration. Off course tea party is the slogan of the opposition so makes an appearance.

Rajiv Makhni is no doubt “the gadget man”. He is also part of the media crowd who put “breadcrumbs” to their tv shows, writings or in general fellow folks. There is lot of link back to parent firm – NDTV and its various shows that is.  He pretty much looks obsessed with phones, devices like ipad/tablet, dimensions and features.  Good thing about Rajiv is that he does most of the hard work and in general not immediately vowed by apple, Microsoft. His latest coverage about “other” brands of phones is genuine attempt to provide voice to alternative (micromax).

Rajdeep Sardesai too talks about his channel, upcoming show(requests to watch with timings and all)  and then some comments on current happenings across amar singh, thackreys, shashitharoor(man did the media guys milk him?). He also points out Sagarika’s articles from time to time. This is apparent in urls that he points too. I have not shared them here but extraction of urls and crawl to them provided the base for that conclusion.

Virsanghvi is unique. Virsanghvi has hands full with pointers to his articles via @addthis. He does cover wider array of topics from Padma Bhushan fiasco , headley, china/Pakistan and love of thackrey/Pawar family. I wished I could do sentiment analysis of those tweets most of which are full of sarcasm.  He made it a point during  Tharoor gate to pin Tharoor followers with his unique insights on folly of being Tharoor and his followers. He followed that with one page writeup in of the magazines basically saying Tharoor was pretty much product of vanity. He also made it personal wrt Mumbai vs Bombay and gave two hoots to the fact that he or his sort who speak French or Italian in 2 day visit that they go to those countries but consider it below themselves to learn a local language(may be he will pick up a cue from Aamir Khan who is learning it or Rajnikant who learnt/mastered local language). In case of food  HOMP on “ndtv good times” guys take his happiness in terms of genuine freshness.

ShashiTharoor – the man who was brought down by twitter unfortunately was a judicious user of tweeter. He posted his meetings, addresses and visits to his constituency, state on tweets. One can get a glimpse of his work just through his tweets. Man was transparent. He did have cricket on his radar though. Unlike media folks he did not pre-judge and pass comments or forwarded to his blog/writing etc. Only time he opened about personal life was about his son and his achievements just like any other father would. His ministerial work too gets prominence though he did keep his boundary wrt to foreign affair. One can clearly see man had India on his mind(this is without stemming so could be really lot more).  He too comes out pretty savvy re-tweeter. But his focus is clear – India, Kerala, meetings, visits and foreign affairs.  Cricket does popup but just engulfed by all the meetings, briefings and mentions about generally issues that is tackling or good things he sees.

Shahrukh comes out as genuine person who wants to connect with his fans and shares tips to youngsters about good/bad habits and need to focus on work. He is extremely dedicated to his kids and family and that comes out pretty strongly. His faith also makes appearance through his invocation of insha allah phrase(basically as/if god wishes). His sleep pattern makes an appearance and one tweet will be about the need of sleep in early morning when he goes to sleep. Exercise, healthy food and fitness make way on the tweets at times. His firm red chillies, his team kkr make brief seasonal appearance. He is also the only one who dwells on his past – mother and father – again re-affirming his attachments.  His language also comes out pretty dominated with verbs and action reflecting his persona? Shahrukh as mentioned earlier comes out swinging for family(kids), friends(karan), cities(Mumbai, Delhi) and his present  disposition (Kolkata knight riders or movie ra.one).

Gulpanag comes across as refreshing change with interest in riding, biking, automobiles and across issues of the time. This ranges from “army scam”, “farmer plight”, “girl child”, eco friendliness and in general fitness (running,bike).

There is gentleman whose tweet analysis I have not done in detail here. But Pankaj Pachauri comes across as genuine person with nuanced interests across varied subjects. His tweets and his programmes are both high quality. He is very few of the journalist who presents facts and analyzes situation of price increase, local manufacturing,  Africa mining, gas price increase, education cost and relation to enterprenuer ship. But unfortunately as it would happen his shows are mistimed and like the regional newspapers and good folks there he is ignored.

The word collocation as the above exercise is called can be easily combined with actual word frequency across tweets. In this case I have not taken trouble to normalize the data with synonyms, homonyms or polynyms. I just intersected across corpus of wordnet and brown. There is slight difference between frequencies of regular text tokenization and custom tokenization. For our purpose we will ignore regular text tokenization.  Name and place extraction via alchemy, evri did not work very well, nor did sentiment analysis for singular tweets. The other challenge was lot of new “sms/tweeter” words like hv, 4, lrnt, mv etc. These words need to be in dictionary replaced with actual words. Another idea would be – do common factor search across a bunch of folks (via # tags first and then other simpler things)

NLTK is useful kit to do quick 2 hrs work. It has its quirks and I have not exploited its whole NLP side of the  world. One of the days J.

Tweets downloaded in Dec 2009 and appended in May 2010.

Twitter api – http://code.google.com/p/python-twitter/ (python 2.6)

How many times word was tweeted (rough and not complete for space)

Which people get lot of mention through tweets?

Is the user tweeter savvy and uses hash tags/lists?

Code for analysis could be shared at git little later or shared on request.