What do ISVs trying to bring their solutions to Cloud want?

Easy to understand Billing model  

Make it easy to reason about the billing model, simpler than what is  exposed to “pay per use”. I need to use it every day. It should just work without surprise. Do not expose the – “you looked at me – y $, you asked for that z$”. Please provide reliable API that I can utilize for creating SaaS applications.

Tell me about your maintenance cycles (please) –

For end customers using a  solution, downtime communication is essential. Ideally 24*7 operation is required but we can craft a solution which can deliver minimum viable  option at lower cost.

MultiTenancy

It means a documentDB/Aurora or Search should have ability to create “tiers” for free/shared instances where I can club in folks for my “freemium tier” without paying production amounts. As it is very low margin business let us find ways to make it simpler. This is little bit different from me creating a shard instance.

Support for MultiCloud Libraries/stacks

We need support for jcloud, fog, libcloud across Provisioning, monitoring, billing of all possible assets.  We understand it will not be a odbc standard but something more workable. Provide deeper integration into chef/puppet/ansible/salt with better templating than promoting custom “provider models”.  Thanks for integrating with github…push it as alternative to store assets. So that config (testing/deployment) etc everything is coded up and stored in github or something similar.  Thanks for support for docker, coreos.

What azure is supported only for blobs in one of them somewhere(libcloud)? No powershell is awesome but not everyone’s favourite piping tool.

Win-Win

I bring you x $, you provide me 0.20%x. No really – make the partnership work with real people rather than english.  Let us find a way to make the adoption faster.  Help us unseat the existing partner brokers who are deadweight – whose deployment/AMC (people/cost) models are a challenge in pure cloud model. That air cover we talked about needs to be about partners, partners, partners. Help unlock the cio-tech-team ice. It is not about x% discounts on the platform.  Focus on that annual sign up stuff for certain software licenses will not open door to growing pie.

Here is shout out to Vijay who joined MongoDB and he correctly  points out “lack of lever” with both customer and seller – there is no  complexity. http://andvijaysays.com/2014/03/25/are-we-there-yet-cant-wait-to-start-my-new-adventure/

In cloud based setup simplicity is much more stark.

Support

Real support in terms of what does not work rather than “green my  scorecard” – so just use it(shove down my throat). Own up the support  issues and help bring down my costs and increase your spread. Get folks who understand both business and technology(people outside use different from what you sell). Let us know at what is coming down which can potentially make us commodity. Be honest about it.

No unless I explicitly tell you don’t push a service. I will pick unique services based on their strength, honestly I will. Love completely hands off 99.99 % Sql Azure where I get backup, HA all in great price. Wished that infra was available for others to host stuff like DB.

 Make it supportable

Other OS is as useful and widely deployed so tools for picking up monitoring information should become better. IIS is a great tool but so are nginx , apache and their friends ha-proxy, squid, varnish. Make “separation/divorce” easier. Easier to withdraw data, easier to withdraw configuration settings – UX should reflect what is possible through powershell, cli and at worst language specific rest bindings. Preferably a language which runs on all platforms.

New age media challenges – muzzling the opposing views

TL;DR;

Organizations like Twitter, FB (social media) or Search organizations need to share what is the way they decide what is right/wrong on their sites beyond legal words. How they decide which view of which participant is muzzled. 

In https://www.youtube.com/watch?v=gYN6uybDKzY Twitter CEO Dick Costolo says people have to assume information will be available to all. Emphasis on word assume.

In https://www.youtube.com/watch?v=J-y8TcHT8Lg Twitter co-founder Jack Dorsey
recounts his becoming the entrepreneur. He exhorts people  to join the movement and question everything.

Earlier populace had to depend on media – printed media to take the views of people
to the leaders and vice versa. Unfortunately like the incestuous relationship of
auditors and companies in private world – lot of give and take was done and watchers
became the mouthpieces. Overtime interest groups realized they need to control the
media to shape viewpoints and pushing of their agenda. Now we have overt politically biased media houses catering to their captive audiences.

Social media birth and evolution helped cement itself as one tool for people to
exchange ideas, information and possibly form opinions. Sadly it also came with tools
to analyze what is being said and ways to block the “opposing” view by simple
“block/report”.

Corporations, ruling entities could easily circumvent or block an unpleasant
question.

Challenge is tool like twitter has not made lot of things transparent. It is like the
chinese firewall but controlled by few people sitting somewhere in CA. Just like
uber, AirBnB we have little commitment or understanding of issues and claim to
disruption without iota of responsibility.

There was move to get old-media folks as editors? or advisors in some of the social
media organizations. Ideas like protecting the source of information, ideas like
allowing questioning not hate filled agenda – who decides what gets on timeline Who
makes these decisions? An algorithm ? People – Who are those ? What are their
political, religion, institutional biases ? Good way to see these biases is to
compare an Al-jazeera and guardian , BBC, ABC, Fox News, MSNBC, Xinhua, Google news  for an event in Gaza, Europe-Russia events, China or India.

For events which are called terrorist events – a certain section will paint it as
“suspected gunmen”. Some organizations will put a religious tone by including larger
context and attaching religious imagery with words, groups, faith adherence. Or
sometimes there is complete blackout of news as in some “controlled” countries.

Tools like Google news twitter, facebook and others need to come clean on
- what is the ranking for feed- really what is it that you decide our world is -
whether a search engine, timeline or the wall . Are you providing governments,
organizations way to control what we see/hear even before it comes online or muzzle.
– what is ignored , what is given more weight
– what is blocked – at least a notification that you have been blocked without
disclosing , in case of search results – just how do you decide to show what is on those pages. what got ignored/blacklisted.
– how is unfolding of “non-popular” but obscure important stories, events, views
done? Is there a metric here for people to follow?

This is to avoid biased coverage like the printed media does because of any
affiliations (owner – fox/aljazeera or network18 here locally).

What does this mean?
As originally said we will need to be ready to withstand opposing and unpleasant
viewpoint. And let laws which are less stringent than french laws for questioning
others be more prevalent. This has geopolitical connotation – earlier media could be
controlled easily by not allowing airwaves or print media or import of books. Sadly
digital world is much more easily controllable and its disappearance is much more
silent. Your search results can disappear, your tweet could be muzzled or facebook

This also means the role of PR/Media advisors and tools which do topic and sentiment
analysis(however broken) needs to become “auditable” across organizations with laws
backing up.

The tough challenge is digital media allows photographs, videos and other assets to
be put online which have much more shocking impact on people watching them. They are also considered powerful propaganda material which organizations, governments want to control.

Examples a sadist organization like IS using it to recruit, influence a
section of people. These organizations balance out “negatives” with “posts of
positive” actions – “helping the neighbourhood etc”.

The reason government carry out muzzling is to either favor curries for the ruling or
the perception of being right. This could have deep festering origins – China still
seething from opium trade or indignities of Nanking. India not liking the questions
around favor to near-dear ones of the ruling section or certain actions of police or
investigation agency somewhere. Or worst to control the opinion or questioning
itself.

Other stronger reason is throughout our history we have had specialists who claim know
economics, foreign policies and certain people control political agendas. Only
certain agencies and people are considered competent to know and take actions on
certain things.
For instance I personally think it was brave of American folks to question methods of
its intelligence agency against snowden and other revelations. Not every country
either has the guts or desire to explore those depths because of perceived guilt or
affront to pedestal status of being right. Sadly other countries and people who are
saying “we said so” – have much more corrupt and unaccounted actions. See Turkey or
Saudi Arabia or for that matter developed country’s surveillance and treatment of
prisoners (political/ideological/war) or any other UN country. Because war and intelligence and interwined and latter is important for lot of things. Some of the police organizations in other countries are more tougher and have unspeakable tactics than compared to the agency which was admonished. But that fact was never bought out by mainstream media or the digital folks.

Folks adept at misusing will do so and have potential to misguiding populace over
religion, language and perceive impact of abortion/gay marriage on local customs.
End of the day it Us-Vs-Them is the end tool in politics, corporations or local
communities and these tools should not become pawns for these purposes. There are
countries which chose to import few things, viewpoints and want to control others. It
is amazing to see Obama who is leftist and has unleashed more drone based attacks and
ended few wars painted more unpatriotic in that country. Almost every right-side
everywhere across considers themselves more patriotic and leftist/liberals are
apologists.

New tools focus on dissemination of information but this control right now neither
rests with governments (at least not explicitly) or people who are fed these. We do
not have oversight of good editors who decide serendipity, local context, issue
weight/counter opinion. Everything is instant – trends for today, popular now and
immediately just like that incidents are pushed off the main screen. Although
language constructs prevent semantic or topic based search. The dominance of few
firms in each country and region prevents healthy conversation and next steps to open
them up for everybody.
I heard locally – local city folks do not want agencies or themselves on social media
as they need not be answerable or keep countering the viewpoints. It is easier to
control physical news a/v, print media by buying them out or dumping few ads. It is easier to overcome digital media by not being on them.

Sadly we need not choose this future as we see some good possibilities of traffic police on
social media.

But beyond this a common man needs a way to know why his voice was muzzled. Context
– I asked
@PMOIndia it is time to have swachpeople first and speak less with more
action.
@PMOIndia about use of very colorful and respectful language threatening
mothers, sisters and death by ruling party MLA against a Medical officer for
reinstating his “corrupt” relative.

– I asked @economist About their language use for response to Boko Haram. Verbatim
text – “the group has been boosted by the impotent reaction of regional governments”.

In former case I duly expect cells of the ruling party, PR teams finding “offensive
viewpoints, questions” to be reported and blocked. This is very similar to PR teams of corporates who have to contain the “percieved damage” and “move on”. Which they duly did by “saying suspension is enough” but no police action required for person who has done this earlier too.
In latter case a respected news organization which should ideally have just expressed
regret over the language and moved on as “macho” responses are more acceptable across the cultures and has different connotations in perpetrators and victims. Offcourse in new scheme of things person with help of software decided my question was neither worth answering nor thinking but effectively requires banning. Sadly Twitter abetted it.

Sadly twitter failed me in both places, it did not bother telling me my tweet was
“blocked/banned” or just wiped off. It lost its credibility in objective evaluation
and rather let a machine algorithm take precedence. I am sure a celebrity like
Appelbaum’s views will not be muzzled (just a guess) but some obscure person
somewhere is an ok target.

We are back to owners and listeners and the incestuous relationships of auditors and
their clients. Our owners across media , corporations and governments have found new
ways of aligning their mutual interests. This unfortunately technology can’t overcome
by “RideWithX” or “IndiaWithxx” tags and self congratulating themselves. We have
darker future where information can be taken off without a trace and viewpoints
created with bunch of hired hands.

Azure Linux THP

You should read the compatibility of your application with THP.(transparent huge pages)

How do you find its status

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled  
grep -i --color huge /proc/meminfo
sudo sysctl -a | grep hugepage

at present you will see cat /sys/kernel/mm/transparent_hugepage/enabled telling it is enabled [always]. Other commands are other ways to see the usage.

How do you modify it? 

1. Edit /etc/rc.local or better yet /etc/sysctl.conf  . WRT rc.local add

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

2. Add “transparent_hugepage=never” to the kernel boot line in the “/etc/grub.conf” file.

 

Oracle – does not like THP.

Mongo – does not like THP and prefers 4k pages.

Cassandra – There was a thread on twitter and the google group wrt THP.  Looks like suggestion is to disable it.

Hadoop does not like THP.

Splunk does not like THP.

MySql does not like THP.

Postgres does not like THP.

What does it do – here are the details. 

Azure Linux tip – swappiness

In general public clouds do not have swap file for linux instances (ymmv).

How to detect swapfile is present

1. grep -i –color swap /proc/meminfo
2. swapon -s
3. free -m

You will get confirmation no swap is setup.

If you check for swappiness via cat /proc/sys/vm/swappiness though you will see swapping of default 60 :). Question on your mind will be where it is doing the swapping.

What should you do ? In general no swapping is good thing, so setting that swappiness to 0 is good thing with default installation. In case you require swapfile(which you will – if you care about latest kernel changes), Add a swap file based off local disk(sdb1 on the /mnt mostly or ssd ) on the guest (do not add azure storage) for the instance.

How to modify swappiness  (for a web or file server)
– echo 5  | sudo tee /proc/sys/vm/swappiness or
– sudo sysctl vm.swappiness= 5
– To persist this setting through reboots it is better to edit the /etc/sysctl.conf and ensure add the swapfile to fstab.

No swapping is good for lucene workloads(solr/elasticsearch), databases (cassandra/mongo/mysql/postgres etc) but for stability reasons at high constantly peaked machines- it is good to have local disk/ssd as help

How to allocate swapfile  usually you will do it on local disk – use df -ah to get mount name)
– Allocate swapfile – sudo fallocate -l 4G /mnt/swapfile (ensure size is double the memory size)
– Ensure root has access – sudo chmod 600 /mnt/swapfile
– sudo mkswap /mnt/swapfile
– verify free -m
– add to fstab – sudo nano /etc/fstab
**** add line *** /mnt/swapfile none swap sw 0 0

To switch off swapping completely On Linux systems, you can disable swap temporarily by running: sudo swapoff -a. To disable it permanently, you will need to edit the /etc/fstab file and comment out any lines that contain the word swap.

Why to swap if nobody likes swapping and it is not 90s – For safety.  From kernel version 3.5-rc1 and above, a swappiness of 0 will cause the OOM killer to kill the process instead of allowing swapping. (ref – http://java.dzone.com/articles/OOM-relation-to-swappiness )

While you are at all of this do notice – df /dev/shm and see what you can do about it. Do you want to use it?

Azure throttling errors

Most of the cloud services provide elasticity  creating illusion of unlimited resources. But many  times hosted services need to push back requests to  provide good governance.

Azure does a good job providing information about this throttling in various ways across services. One  of the 1st service was SQLAzure which provided error
to help the client to retry. Slowly now all services are providing information when they are throttled. Depending on whether you access native API or REST endpoint you get this information in different ways.  I am hoping slowly comprehensive information from services and underlying resources like network, cpu and memory starts percolating like storage so that client, monitoring systems can manage workloads.

Azure DocumentDB provides throttling error and also the time after which to retry.
(HTTP error 429 ) . It definitely is ahead of other services for providing this exclusive information.

Azure Storage on other hand provides information to the native client so that it can back off retry. It also pushes this information into metrics. A great paper exists which provides information about Azure transactions and capacity.

SQL Azure Throttling    – was one of the 1st services to provide throttling information to due to crud/memory operations(45168,45169,40615,40550,40549,40551,40554,40552,40553).

Azure Search throttling provides HTTP error 429/503 so that client can take proper action.

Azure Scheduler provides HTTP Status 503 as it gets busy and expects client to retry.

Azure Queue, Service Bus Queue both send back 503 which REST clients can take advantage of.

Biztalk services  provides “Server is busy. Please try again”

Over # of years we always request customers to exploit Azure and one of the ways is to actually is to work with hosted services and plan workloads by catching these kind of errors. Some of the customers like SQLAzure’s throttling so much they wished they want some of those soft/hard throttling errors in on-premise database.

Most of the Azure services do not charge when quotas are hit or throttling is done. Idea for the client is to back off and try again. I hope though the “monitoring” becomes better say for example in case of biztalk services – a client should be able to query the “busy-ness” since it has to try after systems becomes less busy. SQlAzure’s retry logic has been the well codified and understood over years.

Just in case you wonder other public cloud services too have throttling?  Public cloud services are shared infrastructure and implement throttling for governance. Throttling is implemented and it is exposed in different ways. DynamoDB for example - has 400 series of error codes with specifically LimitExceededException,  ProvisionedThroughputExceededException, ThrottlingException as an example. Almost every service has 400 series of errors with Throttling as specific exception.

Who owns our past

This is a very old draft of a thought. Finally after 10 years it is time to publish much smaller version.

As Search engines dominate the digitization an old fear comes back to haunt. What will be the story that a future generation will hear or read. Will they ever get time and exposure to explore different points of view? Will set of points of view dominate the narration?

I have always wondered how historians tell the story in detail by looking at artefacts and what is left over physically. Sometimes they also reference  written stuff and interpret it to mean something. I have also wondered whether the person who wrote something did not wilfully change the narrative(bible to every other written word)? What about people who did not document anything? How many documents also record honestly everything that happened without bias of any kind.What about people where a new religion or new people erased everything from past. In digital world – search makes it frighteningly easy to obliterate your identity, information about community (news/views/existence).  Since the veracity of information is always suspect we need to better alternative or at least ensure national archives are physical and open up their versions of narrative.

Over last century newspapers have played an important role in dissemination of information and helping people formulate opinions. They do tend to focus on immediate, tactical. Long term investigations and viewpoints come out in magazines or obscure books or worse academic journals. Nobody has time to read through everything. Will future generation have enough information or they will get curated data?

It is in our interest for pushing search engines , social networks, aggregators, applications to share their ranking for relevance, not showing links, withdrawn due to dispute, withdrawn due to laws (government x).

Left Facebook, planning to leave LinkedIn too…just infitismely small statistic

It is quite sometime since I left FB, I keep connected with friends via phone calls or physical meeting. Other friends are checked on by better half’s account occasionally. When FB announced the “experiment” results  – that was it for me.

I am skeptical of news aggregators, search sites and now social networking sites. I
was personally appalled by folks pushing in exaggerated versions of themselves form
their work life.

Lack of transparency
I do not know why certain news items appear in google news and others never make
it. There is neither explanation nor transparency. Over time there will be
dominance of only one “viewpoint” depending on the ownership of these places. It
scares the hell out of me.

I do not get much job offers anyway on linkedIn – and their pulse is very skewed to
interests which do not align to mine. The flow of stories, updates too seems less
on freshness and more on “linked to” etc..I do not have reason to follow these. I
rarely have conversations here.

Only thing which works is Prismatic – I wished Twitter would buy them. They bring
in stuff I am interested in, allow me to explore at my pace rather than twitter –
where stuff is lost after that instant or predefined window controlled by Twitter.
I am hoping there is more of context – more of “permanency” rather than immediate
day or 2. I wished there was better search, right now lot of stuff locked into
individual networks. Twitter’s search has to improve. Please. Just don’t focus on twitter stream, there is bigger world outside.

And yes I am keeping my offline/online conversation tool email handy. I will also
scare people away by putting random stuff on the blog. :) .