Sunday, 19 November 2017
Nehru, The City Designer
Wednesday, 15 November 2017
Should We Need To Hate Nehru
History of modern India wouldn't be complete without Nehru. Why should we forget Nehru. Should we forget because he introduced Hindu Marriage and Succession act, which transformed Hindu society as we know it today. Even though conservative forces which are in power today opposed it vehemently. Or Should we forget him because he followed socialist model of economic development in line with "Bombay plan". Or should we forget because he didn't allow India to slip into revolutionary communism when our neighbours either slipped into dictatorship or communism. Or should we forget him because he stood against imperialism/ west and asked for complete independence not only of India but also of African and Asian countries.
Should we forget him because he gave idea of self development and non-alignment when every country wanted to align post WW2. Should we forget him because he vehemently opposed Gandhian softness towards British in matter of Dominian status to India.
Should we forget him because he declared Purna Swaraj. Or should we hate him because he along with Patel didn't accept balkanisation plan of India. Or Should we hate him for not listening to Patel on Kashmir matter and listening to Mountbatten instead ,which brought all current mess.
Or Should we forget him because he supported Ambedkar who designed modern marvel in law , "The Indian Constitution" , when every constitution born in the world during that period, has crumbled.
Or should we hate him because he listened to field Marshal Karriyappa about capability of Indian army to intervene in the neighbours conflict. Or should we hate him for being optimistic and idealistic towards young communist China which behaved like North Korea.
Or should we hate him because he didn't become authoritarian even though most world leader he meet were. Or Should we forget him because he created states based on language or should we hate him because he taught how to be secular even though it was not part of constitution. Or should we hate him because he was not militarily aggressive towards our smaller neighbours.
Thursday, 4 May 2017
Drought Analytics of Karnataka Using MODIS Satellite Level 2 Data(2001-2015) using ArcPy python package and Tableau Public
Humans have been changing the face of the earth since they emerged out out of Africa. Most of the recent changes has been happening since 1750s. Be it industrial revolution, be it green revolution or white revolution ; all has happened in this short period of time. However post 1950s the change has accelerated in such as way that its being called as Anthropocene : The Age of Humans.
The Age of Humans also brought with the complication of guiding the whole mankind through the problem that came along with Anthropocene. Drought is one of the biggest challenge the mankind has, is and will be facing. There are lot of index to define different type of droughts; this would be a topic for some other day. In the present article i am going to come out with simple index on drought and how to extract this index using MODIS Satellite Level 2 data for entire India using ArcGIS tools and visualize data using Tableau Public.
In previous article i had explained point extraction of data on MODIS level 3 data. Here we would be concentrating on MODIS Level 2 data. Thanks to our work at NIIT University's GIS Lab, we are able to extract and process MODIS Level 2 data for entire India. I am also happy to announce that NIIT University GIS Lab has now capabilities to work on Level 2 data of 4 MODIS products, in large scale running in 100s of GBs. Existing index of Drought is a simple ratio of NDVI and LST(In Kelvin). It does have its own flaws but its simplest and easiest way to conduct preliminary analysis on huge Spatio-temporal data. As this project has other stack holder I would abstain from disclosing the full methodology. I would rather discuss this in brief.
Method:
As previously said we have capability to extract all years data of MODIS Level 2 satellite data for whole India(South Asia) or any part of the world. Resolution of this product varies from 250meters to 500 meters and 1km. NDVI product that has been derived for whole India, has resolution of 500 meters. LST which is in Kelvin has 1 km resolution. This data is 8 days(LST) and NDVI (16 days) composite data. We have codes to convert 8 days LST to 16 days LST and NDVI 500m data to 1 Km resolution. Hence end product will be 1km resolution and 16 days composite data. Drought Index will be calculated pixel wise. Has its index, we intend to know how many number of pixel are present in the defined range of the Index. Drought Index has both Minimum and Maximum, all other data will vary between these. Minimum would be -9 and maximum would be 36. This index is just number but this is power full number that could give a lot of insight in the drought.
If index is less which means NDVI is less and LST is high, and if index is more means NDVI is high and LST is less. Every other things would vary between these interpretation. We have created numerical class for putting all values of pixels index less that 0 in one class and then defining 7 class with the gap of 5. In all 8 classes Class 0 to Class 7. This is still not standard, more research with more parameters could be added to come out with brand new index. Here we are interested in having capability to extract data and analysis district wise using raster data, rather than on Index itself. We have also developed scripts that can take shape file(GIS) of whole state or nation or region containing 1 level of subdivision and extract the raster data based on the subdivision. This extracted data would be in the form of CSV file. The data extracted is just simple conversion of attribute table of each raster file that has been clipped based of shape file given. This is done using ArcPy a python package provided by ESRI. CSV file is fed into Tableau software of both GIS and non-GIS visualization. I am glad to tell readers that this integration of Arcpy and Tableau software for analyzing Raster data(MODIS) is uniquely developed by us in NIIT GIS lab. This method uses processing power of ArcPy at ArcGIS server and visualization power of Tableau.
Link 1
Dry Areas(here u can interact with data by changing years and months)
Link 1: Please Click above Link to see interactive graphs
Our data processing is done for whole India. Just extraction of data from drought raster file took more than 24 hr of processing for entire India. I can tell that i have district wise data for all district of India in this Drought Index. In coming day LST for all districts would be generated. Here in Tableau public I have presented data for my state ie Karnataka state in Republic Of India. Same can be done for any part of the world. Tableau public provides unique opportunity to share data publicly. Given above figure (links) are off Tableau public . These are interactive graphs users can change the selection given in the Tableau. Tableau public is free. Tableau also comes with GIS feature which i have tried to showcase. Lot of interpretation can be drawn out of these graphs. Figure 1(Link 1) will give the spatio-temporal variation of 15 years of Drought Index for all district of Karnataka, for all months and all years. User could filter the visualization. Figure 2(link2) gives plotting. Figure 3 (Link3 ) gives GIS visulaiztaion of Dry spell for entire data. Dry spell data is for post-Rabi season. Northern part of Karnataka state is prominently effected by drought like situations. Reader, I would encourage you to use filters to see how this Index changes year wise and month wise.
Link 2: DroughtAnalytics of Karnataka Using MODIS Satellite Data 2001-2015
There could be lot of discussion on just visualization but typing constrain. But i promise reader that I would come back with more such interesting ESDA(Earth Science Data Analytics) works. Next time it would be LST for districts of karnataka or could be at sub-division level or could be at panchayat level. I have scripts which could work till Panchayat level, even i would be eager to work on this but only processing capability is blocking the move at Panchayat level. Anyway till next time happy learning , Happy ESDA.
Link 3 Here reader can select the graph based upon District and Month. Interpretation of the graph will give rain fall pattern of each districts of Karnataka
Readers please do contact me if you which to derive data for your own area. I would more than happy to help you. mail me at shivaprakash.ssy@gmail.com or contact me via LinkedIn. I would also entourage readers to download Tableau Workbook of these analytics and you can work on my data. We would also provide assistance for those who want to extract data for their region.
Sunday, 2 April 2017
Extraction and Visualization of MODIS level 3 AOD data using ArcGIS and Tableau
It has been a month since I wrote "Making Sense of My Climatic Data". In order to spread the word that "Any one can become Data science guy", I need to keep writing about the work that i have done or doing as part of my projects. The present post is about a project that we(Arun Kale, Manoj Divakaran and Me ) took under the guidance of Dr Parul Srivastav at NIIT University. This project was about investigation of CO(Carbon Monoxide), NO2(Nitrogen Dioxide) concentration over industrial cities near NIIT University using Remotely sensed data of MODIS sensor on board Terra satellite. We also looked into AOD over these industrial cities. Here we have used level 3 data which could be downloaded from on of the NASA's website (https://neo.sci.gsfc.nasa.gov/).
Main objective of this project was to extract data from CO, NO2 and AOD over Neemrana City(RICCO industrial Area), Bawal and Behror. I followed Earth Science Data Analytics methodology but with my own taste to it. I included ArcGIS and Tableau software for the project. ArcGIS Desktop was used to extract the data from the TIFF files and convert the extraction into CSV file which was then fed to Tableau software for Visualization in the form of temporal graph.

- Data in TIFF formate has to be downloaded. Here we have taken monthly data for 15 years form 2001 to 2015.
- A point shape file representing cities was prepared.Using ArcMap tool (Extract Multivalues To Point), we used point shape file to extract the values present at that X,Y in TIFF files.
- This gave us 3 shape files. 1 for CO, 1 for NO2 and another for AOD. Attribute table of these shape files contained data, this table should be extracted to table using Table to Table or Table to excel conversation tool in ArcMap. The output of this conversion in either CSV file or Excel file.
- This csv or excel file is input to the Tableau which is used for visualization of the data. However there is little modification to done to the csv or Excel files. In order to get temporal visualization curve, time period must be in a single column. However the output of the extraction doesnt give that format. so the whole extracted data as to be transposed using Transpose function in the Excel.
- And extraction of data would create a column of extracted values for all points with column name same as the file name so name convention of the file must be such that it must define month and year of the file, so that this information could be used for temporal visualization using Tableau.
Earth Science data analytics(ESDA) doesnt mention at what stage there should be data cleaning, or extraction. This could be adapted based on project. once data preparation, data extraction and data cleaning is done, its time to Visualization and analysis of the extracted data.


Carbon Monoxide(CO) is also important measure for industrial area. Below are the graph of 3 cities.



Even in this(2005 to 2015) data there is clear trend which shows that till 2009-2010 the curve remained steady only to increase from 2010. This is consistent even in other cities.

- Highest AOD values of 3 cities (RICCO Neemrana, Behror, and Bawal) before 2010 have been in month of June and July. This is in consistence with the environment as Rajasthan experiences sand storms and strong winds carry dust in these time. But this trend changes from 2010. The highest recorded AOD from 2010 has been in months of November. This is consistence in 2010,2012,2013,2015. This is due to smog that gets created during the winter, which starts from November. Burning of the crops in Punjab and Haryana during November creates particulate matter that are main reason for the smog in this area along with addition of air born particles from the Industries. This gives clear indication that from 2010 there has been huge increase in pollution levels relative to the previous years.
- Nitrogen Dioxide in RICCO Neemrana, Behror and Bawal area is on rise from 2005. However the slope till 2009 is very small however after 2009 there is rapid rise in Nitrogen Dioxide levels, which is clear indication of addition of Nitrogen Dioxide from industrial sources. Nitrogen Dioxide values also increase during winter months due to temperature inversion.
- Carbon Monoxide in RICCO Neemrana, Behror, and Bawal area shows constant decrease trend from 2001. Highest values in most year has been in month of May. This decreasing trend and peaks in May needs to studied further.
Friday, 3 March 2017
Making Sense of Climate Data using ArcGIS and Tableau software




Friday, 7 October 2016
Geo Data Science
Between 2012-15 it was altogether different experience. It was all about understanding sociology, political science, anthropology, history, environment etc. This has given me strong belief that complexity exist in problem solving, reductionist attitude doesn't provides solution but clarity of the complexity could. Clarity of complexity is just have high resolution data of large area. I always dreamt of working in a domain that integrates all that i have learnt. It has been 1 year in my master in GIS, we have been working on 2 projects on Climate change
- Detecting Greening effect in Uttarakhand area of India.
- AOD study of cities of India
- compile 15 years of temporal data with 500m spatial resolution and 16 days temporal resolution
- 3 parameters were used in Uttarakhand project and 3 other parameters in AOD project
- Data preparation
- Data extraction
- Data visualizations
- Data interpretation
- creating vector data which include point and polygon on the study area. This included creating point cloud of 150 cities.
- data preparation also included downloading or ftping huge amount of data from the servers
We used ArcGIS 10.4.1 software to extract data . As we had large number of files to process we used python scripting. In Uttarkhand project we developed series of code which performed not only better then MRT tool- given by LPDAAC(USGS, NASA)- but also was running on the local machine. All was possible by using ArcPy packages in ArcGIS packages for data processing. This processed data which was in the image format was using ArcGIS itself which is Geo-visualization software as well. But we were able to create lot classified images which were extracted to .csv file. And these images were in order of thousands. Working on thousands of image was huge challenges.
.csv that were generated out of the classified images were fed into Tableau 9.1(Now Tableau 10) for visualization. I am happy to announce that we finally found Greening effect in Himalayas of Uttarakhand. We had another huge task of creating bins fro 3 parameters ---> NDVI, Temperature and Snow Extent & Days. These bins were made on bases of elevation. Bin width was 500meters. It was computationally heavy as each image of single parameter needed to be classified in 10 bins. This data was as previous said was converted into .csv file. All was done using python code in ArcGIS.
I am also happy to tell viewers that there is increase in NDVI pixels at elevation above 5000meters in Himalayan mountains of Uttarakhand. This is clearly visible from temporal graphs that were out come of the Tableau Visualization. This gives us confidence to put another aspect of Data Science ie
Data Visualization
As i explained we have used
Data Preparation--->Data Extraction---->Data Visualization
Interpretation is our result of Green effect, the question of weather and how these parameters can be interrelated and to which extent.
Same logic was followed while carrying out Aerosol Optical Depth(AOD) for more than 150 cities of India. It included
Data Preparation
- using ArcGIS software
- FME desktop
- Ubuntu Server
- Python script(ArcPy)
- Tableau
- ArcMap
- there exist co-relation between temperature and AOD. This correlation is geographical correlation.
- AOD of most Indian cities except some South Indian cities is greater than 0.3 for most of the month of the Year
- Brown cloud (AOD) exist all along the Indo-Gangetic plain staring from Punjab and ending up in West Bengal. Hence most cities that comes under this region dont have clear air to breath.
Monday, 18 January 2016
GI
Geographical Intelligence
Concept of AI (Artificial Intelligence)seems to be old one. Turing Test was hypothesised way back in 1960s. GI is similar concept to it. AIs and GIs were to be reality . people seems to know how, but few knew when. Human conscious is function of time. Machine consciousness is function of both computing, search and time. We have reached a critical juncture where first 2 are met, but its about time now. We have AIs now which can pass Turing test in one component like speech etc. We till now haven't been able to disprove Moore law , but I hope world is close to at least challenge it. World is waiting for next big thing. There are things out there, but its gradual reorganization of these related elements that can create AI . AI could just be abstraction of human consciousness, but what about abstraction of nature. Will we be able to create abstraction of this?. We really don't know this. What form it may take?!!!, no one knows. What it should be called?!!!;, A God, or Natural Intelligence, GI ie geographical intelligence etc . its very intuitive way of answering the question of "What's Next after AI". If u ask me I would prefer a cup of coffee with GI if at all its in human form .smile emoticon smile emoticon smile emoticon
smile emoticon Wednesday, 13 May 2015
Democracy : A Jumla
Idea of Nation State is active result of social contract. Institution are the base of this idea. Nation States are built for society, not for person, community, religion or family or particular group. If some one or some organisation think that they were, are and will be the savior of the nation State than its mockery of Nation State , which is sovereign. Nation State is and should be always be pursued to be under constant danger both from within and outside. Institutions of Nation State are threat to the very idea of Nation State if they are unchecked. Hence fathers of modern democracy(US) introduced checks(Jeffersonian Democracy). Nation State is built for society as whole , but with time power tends to concentrate in hands of very few; be it in hands of politician, executive, be it in judiciary or oligarchs. The greatest threat to Democracy are Demagogues and Un/under checked executive. Most of dictatorial tendency , as world history has noted , is from Executive. Hence its necessary to protect society from the Executive who have sweeping power be its President, be its PM , CM or secretaries of any country.
Dualistic Democracy emerged to this new challenge. Its also true that for South Asia, Democracy has always been idealistic "Jumla"(Hindi word for Simile). Its true that Nation Builders never built Nation State to concentrate power. Distributing and diffusing power were meridian ideas of Democratic Nation State. But with time ideologies have got degenerated and Nation States now revolves not around citizens but around power. This is due to the acts of those who rule in the name of people. Athenian Democracy was the only Democracy which was near to what Democracy is all about; all others seems to be "Jumla". South Asian democracies tend to have become "Jumla". Demagogue, messiah, family, group etc are player in this "Jumla". Dualistic Democracy is one step to check this "Jumla". Having said that, we cant fully rely on this action as , this effort is also by the people who decide "In the name of people". Most of us have become part of "Boiling frog Syndrome". First Gods ruled us, we never asked; then Kings came who ruled as sons of god; then nobles and aristocrats ruled us in our name; now neo-aristocrats (be it particular family or particular group or mother group or hand few of people at executive level) rule us, all in the name of people. These people build cult at the expense of State, institution and citizen. This has always been tactic of demagogues and power centrist executive be its PM,CM or others, of any nations.
Yesterdays (13 th May 2015) judgement of Supreme court(Indian) that tax payer money must not be used for govt advertisement which has photos other than President, PM and CJI, is a-well coming decision. we seems to be moving towards actual democracy; Democracy by and for Citizens. Ask yourself, when Nation State has asked consent of yours, to change the frame work. Ask your self are we following "Rule of Law" or "Rule by Law". From Boiling frog syndrome it can be asserted that we are in "Rule by Law" and "Law" in India or most countries is created by handful of Parliamentarians and by Bureaucracy ; who both are not foolproof. System will never be perfect; most of time its "JUMLA" , Democracy by Judiciary(Dualistic Democracy) is small corrective mechanism to make this "Jumla" to walk the talk. It important that we need to move towards "Rule of Law" . Judiciary need to be strengthen , because historically no one has stopped executive to degenerate into dictatorial messiah. We as citizens are far more supreme than "Rulers in others Name", because it "We The People" who have adopted constitution Not the "Who Rule in the Name of Others".
By
Shivaprakash
Health4All
Friday, 20 March 2015
Obsession with GDP???
It is the fact that world including developing world is follow GDP as measure . I dont call its an obsession. GDP itself is new concept. National GDP is altogether post 1945 phenomenon. Its good that we had a measure. But developing measure for a Nation State was a thought process. It seem now, its non representative which is true now but not then. Coming up with index that is satisfiable is a huge thing. world walked till GDP now, it will move forward which could include all value related thing (even though these are hard to quantify). We have HDI, GII, Happiness Index, Gini coefficient, Failed state ranking, Press freedom Index and other smaller bigger and smaller index. These are getting incorporated in literature . Its not that developing country dont want to follow other index. Nepal have happiness index , way back from 1970s.
Index needs credibility, UN index are never debatable but Failed state index is. And Most of Developing Nation state see other Index as conspiracy of External power, so as a Nation State its very hard to come to terms on these other Index. Its known fact that Gender inequality, other inequality, HDI, violence, human right violation is rampant in countries of south Asia, Africa and other developing countries. Nation state take these index as challenge to there own claim. Take the exam of Human Right Violation during Eelam war(2009) in Srilanka or Maoist war in India, both Nation State dont except report on these. Its hard as of now to move away from GDP, which is a reality not the choice. But all these are nation specific Index centred at Nation State. We need to move beyond Nation State Indexes. How is question for Globalist to think.
And obsession of GDP hasnt hindered healthcare as for as India is considered; It has actually acted like positive feedback mechanism. Spending of Indian State wrt GDP is less compared to developed and some developing countries. It has given civil society a measure to ask for more spending which is the need of time. Yes it could have been done better but its not worst. Time to time intelligentsia discusses HDI,GII, Gini, etc which puts politics and govt on toes to do something good to be in electoral politics. Food Security Act (http://en.wikipedia.org/wiki/National_Food_Security_Act,_2013) , Education for all, Health for all(coming in few years) are output of those political and electoral debate.
Monday, 9 February 2015
We ,The Children of Aristotle
Tuesday, 2 December 2014
Sustainable Development
Yes Economic model have failed; these were the very same model that made Developed world the "DEVELOPED World". These were the same models that made US a super power. These were the same model that made EU a prosperous region. Back Then , these were expectable for every one. But thing take time , time to show up glitch in the system . One of the first such glitch was ozone hole. The same economics has not include most part of world till 1990s. World realised lately about disparity, destruction and duality of economics. But is it evil then; No , its not.
SDG has in its plan: ending of poverty, education, inclusion, equity, health, agriculture , urban development, energy , governance .... all these need economics but with the taste of sustaining it for long time. Yes , for this sustaining character of development we need to change economics . Development must have to come under frame work of planetary boundaries. It needs a touch of nature. Its easy to provide Economics to few people. But distributing it among huge number is the greatest challenge. Its classic case of "scarcity" , which in itself is Economics. If we were in a system that provided us unlimited supply of resource than The same old Economics would have been hailed as victor; But world is not plentyful in-front of human desire . Economics was victor for West but not any more. West has realised that its not in hypothetical world of unlimited resource. We in developing economies has seen things changing from : 1 hr daily electricity, with once in 15 days water from municipality, from no phone , no laptop in 1997(as for as i remember at my place)... to 24 hr electricity, once in 4 day s water, 3 smart phone,4 normal mobile, a land line, 2 laptops, a desktop in the same place by 2014. This is gift of the very Economics which is being blamed. My people enjoy this development which world has started to scrutinise, we need it ; but not with same face. We need development that last for longer time, say till 2050 or till my grand or great grand children or score of generation to come . We need Economics that has Sustainability and inclusiveness as its core values .
Regards
Shivaprakash
Saturday, 25 October 2014
Obligation To Protect
Political legitimacy is one of the supporting concept that is linked with Intervention and sovereignty. Political legitimacy is double edge sword; a sword that cuts both ruling class and the ruled one. How ?...Ask Bush administration!!!! Bush Administration would have said "If its used against ruling, it leads to regime change and If its against ruled ,it leads to state oppression". Problem of legitimacy come into play whenever the state is weak; ie institution strength is low and Rule of law is under question. Effective governance in such a territory is nil. In Hobbesian term its in "state of nature". In this globalised world, there is disincentive, in allowing a state to be in “State of Nature”. Every country acknowledges the fact that intervention is necessary in "State of Nature". Woodrow Wilson was one of the first to advocate internationalism as base of intervention. League of Nation was and UN is the product of this thinking. However when and why to intervene has been the debate which has been going on for a century now.
Afghan war :
- Afghan war was a war for greater stability of the region.
- Afghan war emphasized obligation that world has towards the people of the region and the world, and also to protect world against future attacks from the terror state.
- Its is also the case of super state having obligation to intervene which i would refer as “Obligation To Protect”.
- The War was to change the regime and to prevent WMD(Weapons of Mass Destruction) from falling into hands of terrorist.
- Case presented was to prevent proliferation of WMD.
- Iraq case is classic case of “Right to protect”. Iraq war highlighted the right of a country to intervene.
Let us take recent example :
- Syrian civil war has displaced 1.5
million refugees in the region; making it hotbed for various
criminal activities and internal hub for terrorism. Al Jazeera in
2012-13 did report the growth of these crimes and growth of
terrorism under shadow of civil war. Unintended consequence of not
intervening earlier gave birth of IS in 2014 ,Now IS is the greatest
challenge to the world .
- Absence of state in Somalia has lead
to dangerous growth of Piracy at sea which is a threat to shipping lane of
the world. Its also terror exporting base in the region. Kenya, Ethopia are effected by this.
- Afghanistan's failure is disincentive to the region. Its
terror base of the region; exporting terror to India, China and
other regional countries.
- Case can also be made
if a rouge state(North Korea) has proven weapons of mass destruction and if there
is high probability that it will use it on the neighbouring state
unprovoked.
- The Nation becomes state less and plunge into civil war with no effective government creating huge humanitarian crisis.
- State become a terror hub and active exporter of terrorism.
- State having WMD and their is high probability that it uses.