Diversity Calculator
The motivation for this software came several years ago when I responded to a request on the Society for Conservation GIS list serve. The issue was the difficulty in summarizing the diversity of a set of species locations as point data over a set of study area polygons.
A simple spatial join in ArcMap could summarize averages and sums of numeric point data but did not have the ability to summarize categorical data. I wrote an ArcGIS plug-in that would do what was needed and promptly forgot about it.
Several years later, I was contacted by someone who had been using this plug-in for several years and I was told that it no longer worked with the new version of ArcGIS. For reasons described below in the project background section which, for the sake of brevity, I won’t go into here, I decided to implement this solution as a web application this time around.
Loading your data
The basic premise is this. We have a set of points containing species observations. One point for each observation. And we want to summarize the diversity of species in those points. We also have a set of project area polygons representing the boundaries of the area that we want to summarize species within. The polygon layer has to have a field with a unique identifier, and the point layer must have a field containing the name of the species.
This application allows you to load your data onto the map. This data is loaded from your local file system and does not get uploaded to a server anywhere. It needs to be in GeoJSON format, I am not aware of any method for loading shapefiles from the local file system. They could be loaded to the server and put in the map but that would involve uploading potentially sensitive data to a server and managing that data on the server and you would have to zip the files up anyhow. It’s actually easier to convert your data to GeoJSON text files. The easiest way to do this is to open your data in QGIS and save it as GeoJSON. QGIS can read almost any data format under the sun and its free to download so this should not present any difficulties. You also need to project it to WGS84, but that is trivial to do as part of the GeoJSON export process in QGIS. A simple demonstration can be seen below.
I don’t know why ESRI doesn’t allow direct export to GeoJSON, but at least as of version 10.4 it was not possible. I have seen some add-ins that could do it and if you want to try that you are welcome to but honestly, it will likely be much easier to simply download QGIS. You should have it anyway because it can do a LOT of things that you would need to pay thousands of dollars to ESRI for. It is free and it works great so, in my opinion you should at least get familiar enough with it to load up some data. Read more about QGIS here.
Selecting your data and attributes
Once you have a GeoJSON text file of your polygon project boundary data and another of your species occurrence points you are ready to begin.
The first step is to open https://millermountain.com/diversity and load those two files onto the map. You can use the 
button located in the upper right of the map to open an open file dialog box and navigate to and select the two GeoJSON files. Or if you prefer, at least load the point data and you can generate a hexagon grid as your project area layer in case you are interested in simply calculating how diversity varies across space.
Once the data is loaded click the 
button in the upper left of your screen to open the Diversity Calculator side bar. You’ll need to either a) select the Polygon Layer from the list of available layers or 2) Make a grid of hex polygons using the Make Gird Layer button. This will fill the current map view with hexagons.
Next you’ll need to select the point layer containing species occurrence, and enter the name of the attribute containing the species name.
Calculating diversity and using the results
When you are happy with the specified inputs, simply click the Calculate Diversity button. This will produce both a results layer which duplicates the original polygon layer but includes the diversity information in the popup and the same information in tabular format at the bottom of the sidebar. This layer is color coded from red to green with red being the lowest diversity and green being the highest. You can copy and paste that table directly into an excel spreadsheet if you want to manipulate it in Excel.
The Show Results GeoJSON button will replace the results table with the actual geojson data of the results layer. You can simply copy and paste this text into any text editor and save it with a .geojson extension and then load it into QGIS and if you need to save it as a shapefile.
The results include
- sample size, which is the number of points found in each project area polygon.
- Richness – simply the number of unique species found in the project area.
- Shannon’s H’ – Calculated from the species count data according to the formula
- Simpsons diversity index – calculated ccording the the formula
- Simpsons diversity index with a small smaple size correction – calculated according to the formula
- A summary of the count data for each species found in the project area.
Project Background
I want to preface this section by saying it is just my observations and opinions. It should not be taken as sour grapes against ESRI, which I think is a great company that has done a lot of good for the conservation community and they deserve our gratitude. Still, I think that they could and should put a bit more effort into backward compatibility when they introduce new software, or at least make it easier for developers to set up multiple versions of ArcGIS. They are pushing small independent consultants away from their products and they should be made aware of that.
This is a big issue for me as a consultant working with ESRI products. Clients generally don’t like being told that they have to pay extra money to maintain a product that they paid for years ago and I don’t like working for free. In the case of this Diversity Calculator add-in, the initial work was something that only took me a couple hours and was not a big deal. But upgrading it after several years, would have taken me far longer than it did to write the code in the first place.
The root of the problem is two-fold. First, ESRI is totally dependent on Microsoft for the codebase of ArcMap. When it was first release as ArcView 8.0 in 2000, they bought into the Microsoft development environment hook, line, and sinker. At the time ArcGIS was the largest implementation of Microsoft’s COM technology. Even larger than Microsoft Office. And when Microsoft released new versions of their .NET framework, ESRI had to follow suit. And when ESRI released new versions of ArcMap that were dependent on new versions of their .NET framework they wouldn’t work with plug-ins that were developed on old versions of the .NET framework. And that was when my add-ins would break and my phone would start ringing.
Confounding the problem was ESRI’s refusal to allow multiple versions of their software to run on the same computer or under the same license. My clients were hesitant to upgrade to new versions of ESRI software when it was fresh because so often it took several releases for them to get the bugs worked out and my clients did not like being ESRI’s guinea pigs. They needed stable software that worked and were not interested in ESRI’s shiny new objects with all their bells and whistles. As a result they were generally at least one version behind the latest and often more than that. Upgrading ArcGIS had been such a headache in the past that nobody wanted to do it until absolutely necessary and then only during a slow period when there would be time to adjust. And its even more difficult when you have multiple clients and need to have them all working on the same version of ArcMap, or else I would need to buy not just another license at $1500 but an entire new computer to run it on. Because my clients were a version or two behind, I needed my licenses to be several versions behind as well and couldn’t upgrade just for a small project that I had originally done for free.
Further confounding the issue is that, because of ESRI’s dependence on the Microsoft environment, when they move to a new version of ArcGIS, developers also need to move to a new version of Microsoft’s Visual Studio. The process for setting up a development environment that can communicate between ArcGIS and Visual Studio is not trivial and once you have one working, nobody wants to change anything. It can take hours to set up and if you have to revert back to the old version, success is not guaranteed. For this reason ESRI developers are even more hesitant than their clients to upgrade to new versions of their software.
This, combined with their refusal or inability to implement multi-user editing solutions at a reasonable cost, is the reason that I and many other small consultants have been moving towards and pushing our clients towards open-source solutions such as QGIS and web-based solutions. It’s not even the cost of the initial license as much as it is the constant hassle of upgrading and the inability to have multiple versions. And when you move towards multi-user and web-based solutions the costs of doing things “the ESRI way” skyrocket exponentially and its difficult to pass those costs onto your clients in a competitive market.
Thus I learned to do things using open-source technology including QGIS, PostGIS, and Leaflet. Now when I get a request from a client I only have to worry about how to code it up and not whether or not it’s possible to do it with ESRI products and how much that will cost.
Change Log
07/17/2017 – Creates selection box for field selection based on geoJSON properties.
07/17/2017 – Button to download the results table into an Excel spreadsheet.
Simply, THANK YOU!!!!!