Extraction and Visualization of MODIS level 3 AOD data using ArcGIS and Tableau

It has been a month since I wrote "Making Sense of My Climatic Data". In order to spread the word that "Any one can become Data science guy", I need to keep writing about the work that i have done or doing as part of my projects. The present post is about a project that we(Arun Kale, Manoj Divakaran and Me ) took under the guidance of Dr Parul Srivastav at NIIT University. This project was about investigation of CO(Carbon Monoxide), NO2(Nitrogen Dioxide) concentration over industrial cities near NIIT University using Remotely sensed data of MODIS sensor on board Terra satellite. We also looked into AOD over these industrial cities. Here we have used level 3 data which could be downloaded from on of the NASA's website (https://neo.sci.gsfc.nasa.gov/).
Main objective of this project was to extract data from CO, NO2 and AOD over Neemrana City(RICCO industrial Area), Bawal and Behror. I followed Earth Science Data Analytics methodology but with my own taste to it. I included ArcGIS and Tableau software for the project. ArcGIS Desktop was used to extract the data from the TIFF files and convert the extraction into CSV file which was then fed to Tableau software for Visualization in the form of temporal graph.

Figure show the methodology that was followed. Its a simple methodology. Following are the steps

1. Data in TIFF formate has to be downloaded. Here we have taken monthly data for 15 years form 2001 to 2015.
2. A point shape file representing cities was prepared.Using ArcMap tool (Extract Multivalues To Point), we used point shape file to extract the values present at that X,Y in TIFF files.
3. This gave us 3 shape files. 1 for CO, 1 for NO2 and another for AOD. Attribute table of these shape files contained data, this table should be extracted to table using Table to Table or Table to excel conversation tool in ArcMap. The output of this conversion in either CSV file or Excel file.
4. This csv or excel file is input to the Tableau which is used for visualization of the data. However there is little modification to done to the csv or Excel files. In order to get temporal visualization curve, time period must be in a single column. However the output of the extraction doesnt give that format. so the whole extracted data as to be transposed using Transpose function in the Excel.
5. And extraction of data would create a column of extracted values for all points with column name same as the file name so name convention of the file must be such that it must define month and year of the file, so that this information could be used for temporal visualization using Tableau.

2 Things that need to kept in mind is file naming convention and transposing of the data. Once the data is ready; Tableau will takecare of the rest. Above method involved extraction by ArcGIS, there is no data preparation involved but downloading TIFF data and renaming using python script could be considered as Data preparation. Data cleaning is not necessary as level 3 data doesnt have irregular data. if 0 or NoData exist in data , please verify before using it into tableau. For AOD, the no-data would be extracted as 0 as No-Data's gray-scale would be 0. Hence, if 0 value exist in the data , it should be converted into No-Data. This could be done both at Image level , which would be before extraction and at excel level after extraction.
Earth Science data analytics(ESDA) doesnt mention at what stage there should be data cleaning, or extraction. This could be adapted based on project. once data preparation, data extraction and data cleaning is done, its time to Visualization and analysis of the extracted data.

This graph is temporal visualization of AOD over RICCO area near Neemrana City. There are lot of spikes which are high value of AOD, which is naturally in the month of November, June and July.

June and July are due to intense sand storm which increase the AOD of the area and November and December is due to temperature inversion in the area compounded by burning of straws in the fields. There is change in recent trend, as high values of AOD have started getting consistently in the month of November and December only after 2009 , from when industries are being operationalized in the area . This need to be investigated further. This could be highlighted using Tree map of some of the highest AOD and their months. These graphs are obtained for Behror and Bawal also.
Carbon Monoxide(CO) is also important measure for industrial area. Below are the graph of 3 cities.

There is clear declining trend in both RICCO Neemrana and as well as Bawal. Even Behror also shows the similar trend. Why there is decreasing trends needs to be further investigated. Another interesting thing came out of these data was CO would be highest in the summer months ie in the Month of April, May and June. which varies between 120 ppbv to 160 ppbv.

While evaluating NO2 concentration which is directly linked with industrialization ie as industries increase there will be increase of NO2 gas. This, we got clearly from the temporal graph of NO2 over these cities. Even though data was available only from 2005.

Even in this(2005 to 2015) data there is clear trend which shows that till 2009-2010 the curve remained steady only to increase from 2010. This is consistent even in other cities.

From Tree map that been generated for Neemrana RICCO has most of the highest values of NO2 from the year of 2011 to 2015. Only a few of highest values are from year before 2010.From the above data visualization we can draw some conclusions.

• Highest AOD values of 3 cities (RICCO Neemrana, Behror, and Bawal) before 2010 have been in month of June and July. This is in consistence with the environment as Rajasthan experiences sand storms and strong winds carry dust in these time. But this trend changes from 2010. The highest recorded AOD from 2010 has been in months of November. This is consistence in 2010,2012,2013,2015. This is due to smog that gets created during the winter, which starts from November. Burning of the crops in Punjab and Haryana during November creates particulate matter that are main reason for the smog in this area along with addition of air born particles from the Industries. This gives clear indication that from 2010 there has been huge increase in pollution levels relative to the previous years.
• Nitrogen Dioxide in RICCO Neemrana, Behror and Bawal area is on rise from 2005. However the slope till 2009 is very small however after 2009 there is rapid rise in Nitrogen Dioxide levels, which is clear indication of addition of Nitrogen Dioxide from industrial sources. Nitrogen Dioxide values also increase during winter months due to temperature inversion.
• Carbon Monoxide in RICCO Neemrana, Behror, and Bawal area shows constant decrease trend from 2001. Highest values in most year has been in month of May. This decreasing trend and peaks in May needs to studied further.

These type of data extraction and visualization could be done using this simple methodology. This study could be extended to all the cities in order to know trends of AOD, NO2 and CO. I need to thank Manoj Diwakaran and Arun Kale for generating these graphs and data. And if reader has doubt please mail me at shivaprakash.ssy@gmail.com, I would be more than happy to help you people. Happy learning and happy ESDA.

Making Sense of Climate Data using ArcGIS and Tableau software

 * This post is the replication of the Making Sense of My Data (Climate Data) which i wrote in LinkedIn

As I promised in my previous article, I am back with more to say about Geo-Data science. This time its about visualization of the data that we have extracted. Being independent researcher(Climate Change) and guy who loves data, i would love to show how beautiful(rather worrying for climate scientist) visualization of data could be using Tableau software. As i told in previous post that we have extracted 15 years of LST,NDVI and Snow extent data. Although input was EOS-HDF files of MODIS satellite, we got CSV file as final output. If you need to know how we got CSV, please refer https://www.linkedin.com/pulse/geo-data-science-shivaprakash-yaragal.

Data that we extracted as elevation, class and time as bins. So there could lot of trend analysis using such data. Data can be aggregated, grouped , regrouped based on elevation. Its basically multidimensional data. Tableau software is one of the most powerful that market can provide for us. For students , Tableau provide 1 year free subscription of this software. Thanks to my dear friend Savan Kumar for introducing me to Tableau a year back. Tutorial for Tableau is also free. If you are not student you can still get Tableau public. Given below is the tutorial for Tableau public(https://public.tableau.com/en-us/s/resources).

Having talked about Tableau , lets move on to what we came out with. As this project involves other researchers also , i should restrict myself for not putting all the data. For purpose of this article there are enough visualization to put through . From our data we have done trend analysis , which are quite interesting for Climate Change researcher and as well as data science guy. Below is one such graph. I have reduced granularity in these so that people could get broad picture of whats happening. In the below picture its month wise(Ex: all January month of 15 years) trend at elevation of 4000m to 4500m. This is just for 4 months. We have done it for elevation between 2000m to 6000m having bins of 500m . This data is for 15 years. Month data is from grouped average data. Its quite interesting to see that at 4000m to 4500m there is decreasing in the temperature during Winter and starting of summer. There is much more to infer if NDVI , SNOW extent at same elevation is analyzed(which we have done). I leave inference to readers.

Below picture is beautiful visualization of how snow and no-snow pixels will vary month wise. Each line represents single year. So we have 15 line showing how snow and no-snow pixels vary with Georgian calender. Snow graph clearly shows in which month we have high snow pixel(starting from December and decreasing in March). It also show low number of pixel in summer months. If granularity is increased there is lot much infer upon. We will not go into it. These could be more beautified like bride if you know how to makeup ;-) ;-)

I love multidimensional data, visualization of these is big task for analytics industry. I need to appreciate what Tableau can do with this kind of data. Below image is one of the most lovely visualization( of-course this graph is from me only). she is mine from core ;-) :-) she was my first love , my first love of climate data ;-) ;-) It because of this beautiful graph , i started loving visualization. Lot things can be inferred from it. Its a scatter plot of no of snow pixel vs no-snow pixel categorized by months given by color and size. Size is reversed in order to high the negative aspect, which may be low numerical value, low average etc. Basically its reverse symbolism. Giving big symbol to numerically smaller entity to highlight that entity. From this graph it's inferred that its only in month of June,July, and august there are large number of pixels that dont follow the usual trend.

The graph which is at the top ie at the starting of the article is derived from snow data. It shows variation of 15 years data. All our effort is in just that graph , its as simple. I love that even tough not much can be inferred. As people say "Devil is in details" , all the other graphs are derived from detailed and granular data rather than aggregated data.

I will explain another 2 graph (tree map) which are quite good.These are tree map. These give distribution of data from left to right along with varying size of cells. The red color tree map is about 10 top hottest months of the study area. Blue color tree map shows 10 coldest months. Its feels awesome to come out with such tree map with just click of button using Tableau.

From these Tree maps its clear that June 2009, was hottest(of-course 291.059 Kelvin is total average of large geographical area spread over large area). And February 2008 was the coldest with 267.010. Please note that i havent given you the elevation of the data ;-) ;-) . I leave it to the imagination of the readers. There are numerous such visualization that we are able to do. I am not great climate guy or data mining guy, but what i know is : "what,how and where" to extract data and how to visualize data. My concept is simple "Any one can be come Data science guy". 

I have been doing or done other data visualization as well, I will put that(PM2.5 visualization of Industrial city) in coming days, till then happy visualization. And please ignore any mistakes ;-) ;-) . If any suggestion or talk on the these or any other thing please do contact me on LinkedIn or mail me @ shivaprakash.ssy@gmail.com .

Geo Data Science

I have always wondered why and how people establish new nomenclature in industry. Geo-spatial industry is not new, neither is data science. I was one of the few lucky guys who tested both at the same time way back in 2011-12, while i was still trainee with Tata Consultancy Services. Although i was trained as Java developer, but was given opportunity to work on data mapping using Informatica. Before i could know its part of BI, i had one of myu feet in GIS in maintaining spatial repository for GIS application . It was a accidental entry as non in the existing team were interested in working on GIS.

Between 2012-15 it was altogether different experience. It was all about understanding sociology, political science, anthropology, history, environment etc. This has given me strong belief that complexity exist in problem solving, reductionist attitude doesn't provides solution but clarity of the complexity could. Clarity of complexity is just have high resolution data of large area. I always dreamt of working in a domain that integrates all that i have learnt. It has been 1 year in my master in GIS, we have been working on 2 projects on Climate change

• Detecting Greening effect in Uttarakhand area of India.
• AOD study of cities of India

Both of these projects are first of its kind in terms of scale both spatially and temporally. We use both level 2 and level 3 data of MODIS Terra Satellite. Task in both were similar conceptually

• compile 15 years of temporal data with 500m spatial resolution and 16 days temporal resolution
• 3 parameters were used in Uttarakhand project and 3 other parameters in AOD project

For both project we applied different methodology but we used same approach of data science which included following steps

1. Data preparation
2. Data extraction
3. Data visualizations
4. Data interpretation

For Data preparation and Data extraction we used Ubuntu server and bash programming. Huge amount of data was downloaded from LPDAAC. In GIS or in Geo-spatial domain Data preparation also involves

• creating vector data which include point and polygon on the study area. This included creating point cloud of 150 cities.
• data preparation also included downloading or ftping huge amount of data from the servers

Data Extraction

We used ArcGIS 10.4.1 software to extract data . As we had large number of files to process we used python scripting. In Uttarkhand project we developed series of code which performed not only better then MRT tool- given by LPDAAC(USGS, NASA)- but also was running on the local machine. All was possible by using ArcPy packages in ArcGIS packages for data processing. This processed data which was in the image format was using ArcGIS itself which is Geo-visualization software as well. But we were able to create lot classified images which were extracted to .csv file. And these images were in order of thousands. Working on thousands of image was huge challenges.

.csv that were generated out of the classified images were fed into Tableau 9.1(Now Tableau 10) for visualization. I am happy to announce that we finally found Greening effect in Himalayas of Uttarakhand. We had another huge task of creating bins fro 3 parameters ---> NDVI, Temperature and Snow Extent & Days. These bins were made on bases of elevation. Bin width was 500meters. It was computationally heavy as each image of single parameter needed to be classified in 10 bins. This data was as previous said was converted into .csv file. All was done using python code in ArcGIS.
I am also happy to tell viewers that there is increase in NDVI pixels at elevation above 5000meters in Himalayan mountains of Uttarakhand. This is clearly visible from temporal graphs that were out come of the Tableau Visualization. This gives us confidence to put another aspect of Data Science ie
Data Visualization 

As i explained we have used
Data Preparation--->Data Extraction---->Data Visualization
Interpretation is our result of Green effect, the question of weather and how these parameters can be interrelated and to which extent.

Same logic was followed while carrying out Aerosol Optical Depth(AOD) for more than 150 cities of India. It included

Data Preparation

• using ArcGIS software
• FME desktop
• Ubuntu Server

Data Extraction

• Python script(ArcPy)

Data Visualization

• Tableau
• ArcMap

Data Interpretation: We were able to prove following

• there exist co-relation between temperature and AOD. This correlation is geographical correlation.
• AOD of most Indian cities except some South Indian cities is greater than 0.3 for most of the month of the Year
• Brown cloud (AOD) exist all along the Indo-Gangetic plain staring from Punjab and ending up in West Bengal. Hence most cities that comes under this region dont have clear air to breath.

We are conducting similar projects which i will update in coming months. From above projects its clear that Data Science methods are and has been part of Geo-Data or Geo-spatial data. Geo Data Science is inherent part of GIS or Geo-spatial world.

GI

Geographical Intelligence 

Concept of AI (Artificial Intelligence)seems to be old one. Turing Test was hypothesised way back in 1960s. GI is similar concept to it. AIs and GIs were to be reality . people seems to know how, but few knew when. Human conscious is function of time. Machine consciousness is function of both computing, search and time. We have reached a critical juncture where first 2 are met, but its about time now. We have AIs now which can pass Turing test in one component like speech etc. We till now haven't been able to disprove Moore law , but I hope world is close to at least challenge it. World is waiting for next big thing. There are things out there, but its gradual reorganization of these related elements that can create AI . AI could just be abstraction of human consciousness, but what about abstraction of nature. Will we be able to create abstraction of this?. We really don't know this. What form it may take?!!!, no one knows. What it should be called?!!!;, A God, or Natural Intelligence, GI ie geographical intelligence etc . its very intuitive way of answering the question of "What's Next after AI". If u ask me I would prefer a cup of coffee with GI if at all its in human form .smile emoticon smile emoticon smile emoticon

 smile emoticon 

Democracy : A Jumla

Idea of Nation State is active result of social contract. Institution are the base of this idea. Nation States are built for society, not for person, community, religion or family or particular group. If some one or some organisation think that they were, are and will be the savior of the nation State than its mockery of Nation State , which is sovereign. Nation State is and should be always be pursued to be under constant danger both from within and outside. Institutions of Nation State are threat to the very idea of Nation State if they are unchecked. Hence fathers of modern democracy(US) introduced checks(Jeffersonian Democracy). Nation State is built for society as whole , but with time power tends to concentrate in hands of very few; be it in hands of politician, executive, be it in judiciary or oligarchs. The greatest threat to Democracy are Demagogues and Un/under checked  executive. Most of dictatorial tendency , as world history has noted , is from Executive. Hence its necessary to protect society from the Executive who have sweeping power be its President, be its PM , CM or secretaries of any country. 

Dualistic Democracy emerged to this new challenge. Its also true that for South Asia, Democracy has always been idealistic "Jumla"(Hindi word for Simile). Its true that Nation Builders never built Nation State to concentrate power. Distributing and diffusing power were meridian ideas of Democratic Nation State. But with time ideologies have got degenerated and Nation States now revolves not around citizens but around power. This is due to the acts of those who rule in the name of people. Athenian Democracy was the only Democracy which was near to what Democracy is all about; all others seems to be "Jumla". South Asian democracies tend to have become "Jumla". Demagogue, messiah, family, group etc are player in this "Jumla". Dualistic Democracy  is one step to check this "Jumla". Having said that, we cant fully rely on this action as , this effort is also by the people who decide "In the name of people". Most of us have become part of "Boiling frog Syndrome". First Gods ruled us, we never asked; then Kings came who ruled as sons of god; then nobles and aristocrats ruled us in our name; now neo-aristocrats (be it particular family or particular group or mother group or hand few of people at executive level) rule us, all in the name of people. These people build cult at the expense of State, institution and citizen. This has always been tactic of demagogues and power centrist executive be its PM,CM or others, of any nations.

Yesterdays (13 th May 2015) judgement of Supreme court(Indian) that tax payer money must not be used for govt advertisement which has photos other than President, PM and CJI, is a-well coming decision. we seems to be moving towards actual democracy; Democracy by and for Citizens. Ask yourself, when Nation State has asked consent of yours, to change the frame work. Ask your self are we following "Rule of Law" or "Rule by Law". From Boiling frog syndrome it can be asserted that we are in "Rule by Law" and "Law" in India or most countries is created by handful of Parliamentarians and by Bureaucracy ; who both are not foolproof. System will never be perfect; most of time its "JUMLA" , Democracy by Judiciary(Dualistic Democracy) is small corrective mechanism to make this "Jumla" to walk the talk. It important that we need to move towards "Rule of Law" . Judiciary need to be strengthen , because historically no one has stopped executive to degenerate into dictatorial messiah. We as citizens are far more supreme than "Rulers in others Name", because it "We The People" who have adopted constitution Not the "Who Rule in the Name of Others".


By
Shivaprakash

Health4All

Public Health Policy

“The government is responsible for the people's health but the people themselves are the government"

 Dr Bamgboye Afolabi. 

How can I disagree with Dr on above statement. Its true that we have far more generalist in the health care system , be its secretary level or minister etc. Public health system is a complex system. Programs must be meticulously designed. In developing country like India we start to fail from planning itself. And loop hole are so numerous in number, some time i feel programs are designed in order to siphon off money. Very small amount reaches to the needy once. In India the biggest problem is poverty, next to it is corruption. So its not shocking that India has poor population equal to population of USA. 

When designing public health system, question comes is of financing. who has to pay the bills. Who it should be financed ; is it through public or private or both. This is question which has been debated from the start. But its also true that every thing is not free, some one has to pay some where. However ,it happens some times that ,Not all people has to pay, but certain amount of population have to pay and remaining will get benefit from it. Its unrealistic to have 100% universal health program without paying. And arguing that poor people cant pay and those will be left out ,is passive thinking. Differentiated paying could work. Govt must pay for poor and must allow other to pay for them-self , off course it should be minimal. We have seen far more cases of "Tragedy of common". Health care system must not be made one. Same boot doesnt fit all. Certain disease which have spillover effect ,hence needs a system where every one can be made to pay same or some times nil. But in some circumstances differential paying could be brought in . People must be given option in health care services; some will opt private; some who cant pay, will and must be serviced by Govt. Its about right to differencial choice. Person with ability must not be stopped from paying more if he wish to pay. Mosquito may not make difference while biting because its mosquito but our treatment is in human world not in mosquito's world. Here differentiation exist and if i wish to apply for it, i must be allowed.

Motivation, Spending and Health 

Not spending on health sector is bad idea. I dont agree with core idea of "not increase cost". Our spending on public health is at abysmal level(1.2% of GDP). As health and education are state subject, its state responsibility to increase spending. If motivation was the only factor then soviet union would have collapsed. We have far ignored the incentive part. Incentivisation of health sector is good, salaries has to go up. Bureaucratic red tape must be at higher level of administration not at lower level.

It must also be noted that front-line worker join system not by motivation but for living , hence it has to be kept that way. Their lives must be made worth while by providing incentives, be its their children education or salary or health insurance. Its might be shocking to put forward that in most state govt, govt dont even provide health insurance to Health Department worker. Its laughable, however some state have recently started providing insurance. We also have departmental co ordination problem. Departmental work of one fall in the other; example Women and Child welfare dept has same mandate as Health department in some programs. As i have seen , the former dept more like dead, and coordination in pathetic . Most programs are not implemented. All are cooked books except in TB program and polio program. Most PHC are under employed with ANMs and ASHA workers. There is systemic problem, and Not spending more is part of that problem. For sustainable health system we need both ideological motivation as well as incentive based motivation along with systemic reforms. We have had Civil service reforms but why cant we have Public Health System Reforms.

Obsession with GDP???

Obsession with GDP???

It is the fact that world including developing world is follow GDP as measure . I dont call its an obsession. GDP itself is new concept. National GDP is altogether post 1945 phenomenon. Its good that we had a measure. But developing measure for a Nation State was a thought process. It seem now, its non representative which is true now but not then. Coming up with index that is satisfiable is a huge thing. world walked till GDP now, it will move forward which could include all value related thing (even though these are hard to quantify). We have HDI, GII, Happiness Index, Gini coefficient, Failed state ranking, Press freedom Index and other smaller bigger and smaller index. These are getting incorporated in literature . Its not that developing country dont want to follow other index. Nepal have happiness index , way back from 1970s.


Index needs credibility, UN index are never debatable but Failed state index is. And Most of Developing Nation state see other Index as conspiracy of External power, so as a Nation State its very hard to come to terms on these other Index. Its known fact that Gender inequality, other inequality, HDI, violence, human right violation is rampant in countries of south Asia, Africa and other developing countries. Nation state take these index as challenge to there own claim. Take the exam of Human Right Violation during Eelam war(2009) in Srilanka or Maoist war in India, both Nation State dont except report on these. Its hard as of now to move away from GDP, which is a reality not the choice. But all these are nation specific Index centred at Nation State. We need to move beyond Nation State Indexes. How is question for Globalist to think.


And obsession of GDP hasnt hindered healthcare as for  as India is considered; It has actually acted like positive feedback mechanism. Spending of Indian State wrt GDP is less compared to developed and some developing countries. It has given civil society a measure to ask for more spending which is the need of time. Yes it could have been done better but its not worst. Time to time intelligentsia discusses HDI,GII, Gini, etc which puts politics and govt on toes to do something good to be in electoral politics. Food Security Act (http://en.wikipedia.org/wiki/National_Food_Security_Act,_2013) , Education for all, Health for all(coming in few years) are output of those political and electoral debate.

Regards

Shivaprakash