Making Sense of Climate Data using ArcGIS and Tableau software

 * This post is the replication of the Making Sense of My Data (Climate Data) which i wrote in LinkedIn

As I promised in my previous article, I am back with more to say about Geo-Data science. This time its about visualization of the data that we have extracted. Being independent researcher(Climate Change) and guy who loves data, i would love to show how beautiful(rather worrying for climate scientist) visualization of data could be using Tableau software. As i told in previous post that we have extracted 15 years of LST,NDVI and Snow extent data. Although input was EOS-HDF files of MODIS satellite, we got CSV file as final output. If you need to know how we got CSV, please refer https://www.linkedin.com/pulse/geo-data-science-shivaprakash-yaragal.

Data that we extracted as elevation, class and time as bins. So there could lot of trend analysis using such data. Data can be aggregated, grouped , regrouped based on elevation. Its basically multidimensional data. Tableau software is one of the most powerful that market can provide for us. For students , Tableau provide 1 year free subscription of this software. Thanks to my dear friend Savan Kumar for introducing me to Tableau a year back. Tutorial for Tableau is also free. If you are not student you can still get Tableau public. Given below is the tutorial for Tableau public(https://public.tableau.com/en-us/s/resources).

Having talked about Tableau , lets move on to what we came out with. As this project involves other researchers also , i should restrict myself for not putting all the data. For purpose of this article there are enough visualization to put through . From our data we have done trend analysis , which are quite interesting for Climate Change researcher and as well as data science guy. Below is one such graph. I have reduced granularity in these so that people could get broad picture of whats happening. In the below picture its month wise(Ex: all January month of 15 years) trend at elevation of 4000m to 4500m. This is just for 4 months. We have done it for elevation between 2000m to 6000m having bins of 500m . This data is for 15 years. Month data is from grouped average data. Its quite interesting to see that at 4000m to 4500m there is decreasing in the temperature during Winter and starting of summer. There is much more to infer if NDVI , SNOW extent at same elevation is analyzed(which we have done). I leave inference to readers.

Below picture is beautiful visualization of how snow and no-snow pixels will vary month wise. Each line represents single year. So we have 15 line showing how snow and no-snow pixels vary with Georgian calender. Snow graph clearly shows in which month we have high snow pixel(starting from December and decreasing in March). It also show low number of pixel in summer months. If granularity is increased there is lot much infer upon. We will not go into it. These could be more beautified like bride if you know how to makeup ;-) ;-)

I love multidimensional data, visualization of these is big task for analytics industry. I need to appreciate what Tableau can do with this kind of data. Below image is one of the most lovely visualization( of-course this graph is from me only). she is mine from core ;-) :-) she was my first love , my first love of climate data ;-) ;-) It because of this beautiful graph , i started loving visualization. Lot things can be inferred from it. Its a scatter plot of no of snow pixel vs no-snow pixel categorized by months given by color and size. Size is reversed in order to high the negative aspect, which may be low numerical value, low average etc. Basically its reverse symbolism. Giving big symbol to numerically smaller entity to highlight that entity. From this graph it's inferred that its only in month of June,July, and august there are large number of pixels that dont follow the usual trend.

The graph which is at the top ie at the starting of the article is derived from snow data. It shows variation of 15 years data. All our effort is in just that graph , its as simple. I love that even tough not much can be inferred. As people say "Devil is in details" , all the other graphs are derived from detailed and granular data rather than aggregated data.

I will explain another 2 graph (tree map) which are quite good.These are tree map. These give distribution of data from left to right along with varying size of cells. The red color tree map is about 10 top hottest months of the study area. Blue color tree map shows 10 coldest months. Its feels awesome to come out with such tree map with just click of button using Tableau.

From these Tree maps its clear that June 2009, was hottest(of-course 291.059 Kelvin is total average of large geographical area spread over large area). And February 2008 was the coldest with 267.010. Please note that i havent given you the elevation of the data ;-) ;-) . I leave it to the imagination of the readers. There are numerous such visualization that we are able to do. I am not great climate guy or data mining guy, but what i know is : "what,how and where" to extract data and how to visualize data. My concept is simple "Any one can be come Data science guy". 

I have been doing or done other data visualization as well, I will put that(PM2.5 visualization of Industrial city) in coming days, till then happy visualization. And please ignore any mistakes ;-) ;-) . If any suggestion or talk on the these or any other thing please do contact me on LinkedIn or mail me @ shivaprakash.ssy@gmail.com .

Geo Data Science

I have always wondered why and how people establish new nomenclature in industry. Geo-spatial industry is not new, neither is data science. I was one of the few lucky guys who tested both at the same time way back in 2011-12, while i was still trainee with Tata Consultancy Services. Although i was trained as Java developer, but was given opportunity to work on data mapping using Informatica. Before i could know its part of BI, i had one of myu feet in GIS in maintaining spatial repository for GIS application . It was a accidental entry as non in the existing team were interested in working on GIS.

Between 2012-15 it was altogether different experience. It was all about understanding sociology, political science, anthropology, history, environment etc. This has given me strong belief that complexity exist in problem solving, reductionist attitude doesn't provides solution but clarity of the complexity could. Clarity of complexity is just have high resolution data of large area. I always dreamt of working in a domain that integrates all that i have learnt. It has been 1 year in my master in GIS, we have been working on 2 projects on Climate change

• Detecting Greening effect in Uttarakhand area of India.
• AOD study of cities of India

Both of these projects are first of its kind in terms of scale both spatially and temporally. We use both level 2 and level 3 data of MODIS Terra Satellite. Task in both were similar conceptually

• compile 15 years of temporal data with 500m spatial resolution and 16 days temporal resolution
• 3 parameters were used in Uttarakhand project and 3 other parameters in AOD project

For both project we applied different methodology but we used same approach of data science which included following steps

1. Data preparation
2. Data extraction
3. Data visualizations
4. Data interpretation

For Data preparation and Data extraction we used Ubuntu server and bash programming. Huge amount of data was downloaded from LPDAAC. In GIS or in Geo-spatial domain Data preparation also involves

• creating vector data which include point and polygon on the study area. This included creating point cloud of 150 cities.
• data preparation also included downloading or ftping huge amount of data from the servers

Data Extraction

We used ArcGIS 10.4.1 software to extract data . As we had large number of files to process we used python scripting. In Uttarkhand project we developed series of code which performed not only better then MRT tool- given by LPDAAC(USGS, NASA)- but also was running on the local machine. All was possible by using ArcPy packages in ArcGIS packages for data processing. This processed data which was in the image format was using ArcGIS itself which is Geo-visualization software as well. But we were able to create lot classified images which were extracted to .csv file. And these images were in order of thousands. Working on thousands of image was huge challenges.

.csv that were generated out of the classified images were fed into Tableau 9.1(Now Tableau 10) for visualization. I am happy to announce that we finally found Greening effect in Himalayas of Uttarakhand. We had another huge task of creating bins fro 3 parameters ---> NDVI, Temperature and Snow Extent & Days. These bins were made on bases of elevation. Bin width was 500meters. It was computationally heavy as each image of single parameter needed to be classified in 10 bins. This data was as previous said was converted into .csv file. All was done using python code in ArcGIS.
I am also happy to tell viewers that there is increase in NDVI pixels at elevation above 5000meters in Himalayan mountains of Uttarakhand. This is clearly visible from temporal graphs that were out come of the Tableau Visualization. This gives us confidence to put another aspect of Data Science ie
Data Visualization 

As i explained we have used
Data Preparation--->Data Extraction---->Data Visualization
Interpretation is our result of Green effect, the question of weather and how these parameters can be interrelated and to which extent.

Same logic was followed while carrying out Aerosol Optical Depth(AOD) for more than 150 cities of India. It included

Data Preparation

• using ArcGIS software
• FME desktop
• Ubuntu Server

Data Extraction

• Python script(ArcPy)

Data Visualization

• Tableau
• ArcMap

Data Interpretation: We were able to prove following

• there exist co-relation between temperature and AOD. This correlation is geographical correlation.
• AOD of most Indian cities except some South Indian cities is greater than 0.3 for most of the month of the Year
• Brown cloud (AOD) exist all along the Indo-Gangetic plain staring from Punjab and ending up in West Bengal. Hence most cities that comes under this region dont have clear air to breath.

We are conducting similar projects which i will update in coming months. From above projects its clear that Data Science methods are and has been part of Geo-Data or Geo-spatial data. Geo Data Science is inherent part of GIS or Geo-spatial world.

GI

Geographical Intelligence 

Concept of AI (Artificial Intelligence)seems to be old one. Turing Test was hypothesised way back in 1960s. GI is similar concept to it. AIs and GIs were to be reality . people seems to know how, but few knew when. Human conscious is function of time. Machine consciousness is function of both computing, search and time. We have reached a critical juncture where first 2 are met, but its about time now. We have AIs now which can pass Turing test in one component like speech etc. We till now haven't been able to disprove Moore law , but I hope world is close to at least challenge it. World is waiting for next big thing. There are things out there, but its gradual reorganization of these related elements that can create AI . AI could just be abstraction of human consciousness, but what about abstraction of nature. Will we be able to create abstraction of this?. We really don't know this. What form it may take?!!!, no one knows. What it should be called?!!!;, A God, or Natural Intelligence, GI ie geographical intelligence etc . its very intuitive way of answering the question of "What's Next after AI". If u ask me I would prefer a cup of coffee with GI if at all its in human form .smile emoticon smile emoticon smile emoticon

 smile emoticon 

Democracy : A Jumla

Idea of Nation State is active result of social contract. Institution are the base of this idea. Nation States are built for society, not for person, community, religion or family or particular group. If some one or some organisation think that they were, are and will be the savior of the nation State than its mockery of Nation State , which is sovereign. Nation State is and should be always be pursued to be under constant danger both from within and outside. Institutions of Nation State are threat to the very idea of Nation State if they are unchecked. Hence fathers of modern democracy(US) introduced checks(Jeffersonian Democracy). Nation State is built for society as whole , but with time power tends to concentrate in hands of very few; be it in hands of politician, executive, be it in judiciary or oligarchs. The greatest threat to Democracy are Demagogues and Un/under checked  executive. Most of dictatorial tendency , as world history has noted , is from Executive. Hence its necessary to protect society from the Executive who have sweeping power be its President, be its PM , CM or secretaries of any country. 

Dualistic Democracy emerged to this new challenge. Its also true that for South Asia, Democracy has always been idealistic "Jumla"(Hindi word for Simile). Its true that Nation Builders never built Nation State to concentrate power. Distributing and diffusing power were meridian ideas of Democratic Nation State. But with time ideologies have got degenerated and Nation States now revolves not around citizens but around power. This is due to the acts of those who rule in the name of people. Athenian Democracy was the only Democracy which was near to what Democracy is all about; all others seems to be "Jumla". South Asian democracies tend to have become "Jumla". Demagogue, messiah, family, group etc are player in this "Jumla". Dualistic Democracy  is one step to check this "Jumla". Having said that, we cant fully rely on this action as , this effort is also by the people who decide "In the name of people". Most of us have become part of "Boiling frog Syndrome". First Gods ruled us, we never asked; then Kings came who ruled as sons of god; then nobles and aristocrats ruled us in our name; now neo-aristocrats (be it particular family or particular group or mother group or hand few of people at executive level) rule us, all in the name of people. These people build cult at the expense of State, institution and citizen. This has always been tactic of demagogues and power centrist executive be its PM,CM or others, of any nations.

Yesterdays (13 th May 2015) judgement of Supreme court(Indian) that tax payer money must not be used for govt advertisement which has photos other than President, PM and CJI, is a-well coming decision. we seems to be moving towards actual democracy; Democracy by and for Citizens. Ask yourself, when Nation State has asked consent of yours, to change the frame work. Ask your self are we following "Rule of Law" or "Rule by Law". From Boiling frog syndrome it can be asserted that we are in "Rule by Law" and "Law" in India or most countries is created by handful of Parliamentarians and by Bureaucracy ; who both are not foolproof. System will never be perfect; most of time its "JUMLA" , Democracy by Judiciary(Dualistic Democracy) is small corrective mechanism to make this "Jumla" to walk the talk. It important that we need to move towards "Rule of Law" . Judiciary need to be strengthen , because historically no one has stopped executive to degenerate into dictatorial messiah. We as citizens are far more supreme than "Rulers in others Name", because it "We The People" who have adopted constitution Not the "Who Rule in the Name of Others".


By
Shivaprakash

Health4All

Public Health Policy

“The government is responsible for the people's health but the people themselves are the government"

 Dr Bamgboye Afolabi. 

How can I disagree with Dr on above statement. Its true that we have far more generalist in the health care system , be its secretary level or minister etc. Public health system is a complex system. Programs must be meticulously designed. In developing country like India we start to fail from planning itself. And loop hole are so numerous in number, some time i feel programs are designed in order to siphon off money. Very small amount reaches to the needy once. In India the biggest problem is poverty, next to it is corruption. So its not shocking that India has poor population equal to population of USA. 

When designing public health system, question comes is of financing. who has to pay the bills. Who it should be financed ; is it through public or private or both. This is question which has been debated from the start. But its also true that every thing is not free, some one has to pay some where. However ,it happens some times that ,Not all people has to pay, but certain amount of population have to pay and remaining will get benefit from it. Its unrealistic to have 100% universal health program without paying. And arguing that poor people cant pay and those will be left out ,is passive thinking. Differentiated paying could work. Govt must pay for poor and must allow other to pay for them-self , off course it should be minimal. We have seen far more cases of "Tragedy of common". Health care system must not be made one. Same boot doesnt fit all. Certain disease which have spillover effect ,hence needs a system where every one can be made to pay same or some times nil. But in some circumstances differential paying could be brought in . People must be given option in health care services; some will opt private; some who cant pay, will and must be serviced by Govt. Its about right to differencial choice. Person with ability must not be stopped from paying more if he wish to pay. Mosquito may not make difference while biting because its mosquito but our treatment is in human world not in mosquito's world. Here differentiation exist and if i wish to apply for it, i must be allowed.

Motivation, Spending and Health 

Not spending on health sector is bad idea. I dont agree with core idea of "not increase cost". Our spending on public health is at abysmal level(1.2% of GDP). As health and education are state subject, its state responsibility to increase spending. If motivation was the only factor then soviet union would have collapsed. We have far ignored the incentive part. Incentivisation of health sector is good, salaries has to go up. Bureaucratic red tape must be at higher level of administration not at lower level.

It must also be noted that front-line worker join system not by motivation but for living , hence it has to be kept that way. Their lives must be made worth while by providing incentives, be its their children education or salary or health insurance. Its might be shocking to put forward that in most state govt, govt dont even provide health insurance to Health Department worker. Its laughable, however some state have recently started providing insurance. We also have departmental co ordination problem. Departmental work of one fall in the other; example Women and Child welfare dept has same mandate as Health department in some programs. As i have seen , the former dept more like dead, and coordination in pathetic . Most programs are not implemented. All are cooked books except in TB program and polio program. Most PHC are under employed with ANMs and ASHA workers. There is systemic problem, and Not spending more is part of that problem. For sustainable health system we need both ideological motivation as well as incentive based motivation along with systemic reforms. We have had Civil service reforms but why cant we have Public Health System Reforms.

Obsession with GDP???

Obsession with GDP???

It is the fact that world including developing world is follow GDP as measure . I dont call its an obsession. GDP itself is new concept. National GDP is altogether post 1945 phenomenon. Its good that we had a measure. But developing measure for a Nation State was a thought process. It seem now, its non representative which is true now but not then. Coming up with index that is satisfiable is a huge thing. world walked till GDP now, it will move forward which could include all value related thing (even though these are hard to quantify). We have HDI, GII, Happiness Index, Gini coefficient, Failed state ranking, Press freedom Index and other smaller bigger and smaller index. These are getting incorporated in literature . Its not that developing country dont want to follow other index. Nepal have happiness index , way back from 1970s.


Index needs credibility, UN index are never debatable but Failed state index is. And Most of Developing Nation state see other Index as conspiracy of External power, so as a Nation State its very hard to come to terms on these other Index. Its known fact that Gender inequality, other inequality, HDI, violence, human right violation is rampant in countries of south Asia, Africa and other developing countries. Nation state take these index as challenge to there own claim. Take the exam of Human Right Violation during Eelam war(2009) in Srilanka or Maoist war in India, both Nation State dont except report on these. Its hard as of now to move away from GDP, which is a reality not the choice. But all these are nation specific Index centred at Nation State. We need to move beyond Nation State Indexes. How is question for Globalist to think.


And obsession of GDP hasnt hindered healthcare as for  as India is considered; It has actually acted like positive feedback mechanism. Spending of Indian State wrt GDP is less compared to developed and some developing countries. It has given civil society a measure to ask for more spending which is the need of time. Yes it could have been done better but its not worst. Time to time intelligentsia discusses HDI,GII, Gini, etc which puts politics and govt on toes to do something good to be in electoral politics. Food Security Act (http://en.wikipedia.org/wiki/National_Food_Security_Act,_2013) , Education for all, Health for all(coming in few years) are output of those political and electoral debate.

Regards

Shivaprakash

We ,The Children of Aristotle

We ,The Children of Aristotle

“Let us study Aristotle : The Despot, The perfect Elitist; Philosophical son of Plato: The Tyrant”, This was how i got introduced to the greatest philosopher of ancient Greek world. Initially it intrigued me; a mere thought about such an introduction was kind of blasphemy for me. This was an year back, after turning some pages of ancient history of both know and not so known world , answer seems to be settling for me now; Yes there is certain truth why my philosopher called Aristotle The Perfect Elitist and Plato , a Tyrant. May be it depends upon how you read him; not one , not twice but any time you read. At time, he appears to be “A defender of Monarch”, “advocate of universal monarchy”, at some time he seems to be “Radical Republican”, at some he seems to be “naturalist”, at some point he is advocate of slavery and at others he denounce slavery. Aristotle is not single character , he is like Ravana with many heads.

A man of notorious secrecy and reluctance, he was just referred as “The philosopher” or “The Master of those who know”. He was an unchallenged man of his time; even today he is! After reading his work, i wonder what he would have to say about our world. Would he reject us or give a conning smile of acceptance. He was the first philosopher who said ,“we are the 'zoon politikon': political animal”. For him man alone ,among the animal, had speech or logos(reason). Human was the animal who used logos to distinguish pain and pleasure, who took advantages of harm, who measured just and unjust. It was very particular to humans. It was humans' logos that certain moral categorisation:a sort of moral constitution, to pick things that are harmless, just to him itself, was developed; he called it a family. Family developed into tribe, tribe developed into community, community into village , and then city and then city state. Its natural development .

Polis is for Aristotle a natural entity , which allows humans to achieve their ends and perfect them. Participation in city is necessary for human excellence. A person who is without city is either above humanity(god) or below humanity(beast). However he makes it clear that , naturality is not biological but sociological. He say man(sorry for sexual bias) is political animal not because we have some biological impulse or drive to participate in politics, but , we have power of reasoning. Its reasoning that makes us “political”. We participate because we have reasoning capacity. Connecting logos ie power of speech or reason to politics ,was the greatest outcome of his argument. The power of speech as he calls ,helped us to develop common moral language, a common concept of just and unjust, common shared ideas etc. This is what made the city. Its also brought with it power of love, affection , friendship, sympathy and all the qualities that make us human.

So far he is great , but he starts to narrow down from here. For him this city is a particular city , the polis will be a small city. A polis of closed society. A society that helps us to get our perfection. To achieve these perfection we need to be held together by bonds , bonds of trust, bonds of friendship etc. For him, we cant trust all people, trust can only be extended to small circle of friends or fellow citizens. Only a small city built on trust can be governed justly and in political sense. The empire can only be ruled despotically. Hence imperialism must go in hand with despotism. Think of this statement. These were the first seeds of imperial despotism sown by Aristotle himself.

Man is a political animal and city is by natural but city cant be universal state. It can be something that cant encompass the whole humanity, a world state or world government will be a dream. World state will and can never be as a small , self governing city state. In city, Loyalty is prime virtue , citizen has to be loyal to city. A good citizen of one city cant be good citizen of another. City will exist, and will have enemies and friends; inside city there will test of loyalty and disloyalty, a test of citizen ship and non citizen. Think of this with respect to modern conflict involving nations like India and Pakistan. A City cant be friends with all cities , there will be wars . Wars are unjust but have to be fought.

Inside each city, perhaps during war, citizens have to participate in the offices of city. Freedom will be there but only through political participation, ie only when there is political responsibility(compare this statement with statement related to casting of vote ). Freedom doesn't mean living as you like , but freedom will be in the form of restriction and awareness . Awareness that, not all is permitted(think this wrt PDA, public display of affection ). This society will promote moderation, restraint and self control(think this with respect to dictact on what women should ware in public). In many angle he seems , atleast for me a critique of freedom.

In this city state , according to Aristotle , in equality is basic rule. Its a human rule, hence there exist hierarchy in the citizenship. Those who cant be citizens , they can be owned , because only citizens cant be owned in citizen and all other things can be owned. Hence non citizens can be brought as slaves. As inequality is natural and basic rule ; slavery can be deduced as natural. Distinction between master and slave is natural (think of this argument; think of slavery that exist in world till recently). For me he seems to be the most anti-democrartic philosopher that i have encountered till date. He who is equivalent to Indian Manu. But he also points out that slavery was controversial topic even at that time. Was he speaking in defence of slavery?Perhaps some say yes he was, and some disagree with a note that you better read him once again. He does oppose slavery that is built on unjust wars. He says these type of salvery are not natural , slave-master relationship is relation of power. But it can be argued that when polity is about upbringing and its not biological, which he himself said. How can , hierarchy of intelligence which defines hierarchy of citizenship, be natural and hence how can slavery be called a natural thing?

In all , Aristotle seems to regard , all men are not free. Hence all don't have equal rights to participate in ruling (as citizens) and being ruled as citizens of a state. He seems to advocate that discipline and self restraint is necessary for an educated mind . This education should be for closed society. A group of closely linked citizen , who can govern the city(think of modern politician, bureaucrats, corporates , film industry). To be specific, A well educated and well trained aristocrat. Best regime will be an aristocratic republic or monarchy ,where a group of educated elites governs the city or nation for the good of all. Its seems atleast for me ,a very elitist proposition. But think of this, Is is not easy for cricketer son to be a cricketer; Is it not easy for Bhatts or kapoors to enter film industry; Think of political parties: are all becoming group or family run entities; Think about IPL teams and there owners, running show in the name of all Indians ; Think about how officers in Delhi or Bangalore take decision in the name of “Good for all”. Think how we accept our own social hierarchy. Think how we choose our friends and our life partners. Think of our own leaning towards people. Its not hard to imagine how Aristocratic we are. If Aristotle would visit this planet now , he would say ,“YOU ARE ALL MY CHILDREN”.

Dedicated to those who taught me.

Shivaprakash