What is web GIS?

When most people think about web GIS, they think about publishing a map or a data set for the world to see.  That’s certainly part of it, but only a small part.  For many years when I would try to sell my employers on the concept of web GIS they would reply “We don’t want everyone to see our proprietary information.”  After digging in and learning more on my own I began to realize that there was much more to web GIS than publishing content.

For the purposes of this post, web GIS is any method of accessing, analyzing, and displaying geospatial data using standard web technologies. This would include standard client side technologies such as HTML, CSS, and JavaScript. On the server-side it could include several languages used for interacting with a database. The most common one would be PHP, but it could also include Java, Python, Microsoft Asp.NET, Perl, Ruby, and even more JavaScript using Node.JS.

The advantages of interacting with GIS through the web are numerous

• Your GIS data can be accessed on any computer from anywhere there is an internet connection. This includes from mobile devices if they have a cellular data connection.
• Accessing your data via the web doesn’t require any software licensing fees, although depending on the method you choose there may still be software costs involved.
• Web GIS tends to be more stable in the face of changing technologies. It’s not that web technologies never change as much as it is that they put a lot more effort into backward compatibility. This means that your solutions are more likely to be running in 5 years than they would be with desktop GIS. This is more likely to be true with basic open-source web technologies than commercial solutions.

What can you do with web GIS?

The short answer is, almost anything you can do with desktop GIS is possible with web GIS. The reality, however, is that some things are better suited for web GIS than others. Web GIS should not be thought of as a replacement for desktop GIS. It is better suited as a tool for addressing specific problems rather than general ones.  Some examples of good uses for web GIS are:

• Publishing your map and/or data to the general public. This is the most common use, and what most people think of when they think of web GIS.
• Providing secure portals for your clients. In 15 years as a GIS specialist for environmental consulting firms, I spent a tremendous amount of time packaging up data products that we had created and sending them to clients so that they could view them in their own GIS systems. In many cases, it would have been far more efficient to provide the client with a secure log-in to a web GIS application so that they could view or even possibly edit that data via the internet.
• Providing limited GIS access to light users within your organization who don’t require a full desktop license. This might include web-accessible viewers for project managers to view the data collected by field crews.  It might also include lightweight data creation and editing capabilities for field crews.
• Mobile mapping and data collection. You can build a web page that is designed for mobile devices and that acts like a native app. This will allow field personnel to navigate in the field and collect data that will be instantaneously accessible to project managers in the office. If no data connection is available, it is relatively simple to store data on the device and update the central database when a connection is available.
• Developing analytical tools that are accessible to everyone, from anywhere there is an internet connection. Web based analytical tools are less dependent on the vagaries of commercial software upgrade cycles. The last thing any developer wants is to have a client come back in two years and tell them that their tool stopped working because the software it was developed for just released a new version.
• Citizen Science or crowd-sourced data collection. If you are relying on the public for data collection you want them to be able to supply that information via a web site rather than purchase expensive software or devices.

Examples

In 15 years, I have never needed to simply publish a map or data to the general public as part of my work for environmental consulting companies. This is one reason why many small and medium sized companies tend to overlook the power of web GIS. I have, however, had many occasions where web GIS solutions would have been ideal if I had the knowledge to implement them. I have also been able to implement a few of these in the past several years as I have gained the required knowledge.

Secure client portals

I have only created one secure client portal. This was for a mosquito control project where the client, a county government, wanted real-time access to the data field crews were collecting so that they could monitor the work and perform QA/QC. It was relatively easy to implement as we already had that information available for our office staff.  I simply created a reduced version of what the office staff had available and used the existing log-in system for security.  Its far easier to remove capabilities than to add them so this project only took a few hours to implement.

There are other examples from my work where a secure client portal would have been advantageous. One was a stormwater monitoring project for a 60 mile long transmission line upgrade.  Our field staff worked with the construction crew to recommend best management practices (BMP’s) for minimizing stormwater run-off control along the entire corridor. They also monitored each site after rainfall events to evaluate their success or failure.  Of course all of the BMPs had to be mapped

Our client wanted real-time access to the BMP recommendations so they could order materials and notify construction personnel. This was 10 years ago and in a remote area with no internet connection. Our field staff would collect location data with GPS in the field and every night they would have to download the data, post-correct it, make a map, print the map, and fax it to the client with an estimate of materials required, then they would also enter the information into an Access database for tracking purposes. This took a large amount of time after a long day in the field.  Today I would implement a solution where all our field staff need to do would be collect the data in the field on a mobile device which would automatically update a central database.  The client would have a secure portal from which they could view the data, print maps, and generate material reports.

I was very close to implementing another project before oil prices crashed. Our client was an oil and gas company.  Our field staff would survey proposed project locations for potential environmental constraints that would impact our client’s operations.  These included raptor nests, wetlands, endangered species habitat, etc. Development was fairly dense in this area. A raptor nest that was located for one project often would impact several others as well. The cost to the client of having a drill-rig sit un-used for a day was in the neighborhood of $100,000 so they wanted immediate notification of any findings that would impact their operations.  It was a fast-paced, complex, project that required a lot of coordination between field crews, project managers, and our clients.

My proposal would have provided a secure portal for the client so that they could see all of our data in real-time as it was being collected in the field and receive automatic e-mail notifications of any finding that impacted active projects. It would also have allowed the client to draw an area on the screen and generate a report of potential constraints for that area.  It would not have eliminated the need for our field crews to survey the area because our coverage was not complete and constantly changing, however it would have provided valuable information to the client for planning purposes.

GIS access

For the mosquito control project described above I developed a system in which the two project managers had full access to the GIS data, including real time access to the data that the field personnel were collecting.  They also had access to a suite of reports that they needed to manage the project, QA/QC tools, and even the ability to see where their field crews were and where they had been.  This project also had three people working in a lab who needed the ability to view the data, update contact information for field sites, and do limited QA/QC.  The lab personnel, however, were not provided the ability to edit the data that the field crew were collecting or to add and delete sites, or manage the field personnel.

Web GIS made it easy to control exactly which features were available to which personnel.  Two additional benefits were that:

• The web GIS interfaces were available from any computer, anywhere that had an internet connection. Even tablets and phones if designed for smaller screens.  This allowed project managers to keep tabs on their projects when traveling or if they were too sick to come to the office.
• None of these interfaces consumed a desktop GIS license which saved the company a considerable amount of money.

A desktop GIS program such as ArcMap is incredibly flexible and powerful.  With that flexibility and power, however, comes a bit of complexity. In my experience, any organization has a wide range of personnel with an equally wide range of GIS knowledge and interests.  Some people love using GIS and are eager to learn more, others simply see it as a tool, and others still view it as a source of constant frustration and would prefer not using it any more than absolutely necessary.  The feedback we often got from our field staff was “Why can’t we just have something like Google Maps but with our data on it?”

Web GIS provides the ability for an organization to provide those employees with less interest and/or ability with a simpler, project specific web interface that allows them to view the data, edit attributes, add new features, etc.  They greatly appreciated the simple Google Maps type interface, especially for field work.  The power-users in the organization can still access the full power and flexibility of desktop GIS.

Had the technology been available earlier, web GIS would have saved the companies I worked for considerable time and money on other projects as well.  One project involved mapping wetlands over the entire Navajo Reservation. An area larger than several states. We utilized students from Dinė College to help with the initial phases of mapping and installing the tools I had developed to digitize in ArcMap on the university computers was challenging.  In addition, we struggled with multi-user access issues and a web GIS approach would have solved both those problems.

In the oil and gas project I described above we had four project managers.  Two of them were knowledgeable and comfortable with ArcMap, but the other two simply wanted to be able to see their project areas and the environmental constraint data. They were constantly asking why they couldn’t just see the data in Google Earth. A web GIS approach would have provided that ability and eased a lot of frustration and expense on everyone’s part.

Mobile data access and data collection

Web browsers on mobile devices today are pretty amazing.  Any of the web GIS portals I described above could be viewed on a phone or tablet.  The user experience, however, of trying to use a web site designed for a large screen on the small screen of a phone is challenging.  The opposite is also true.  In addition, the touch interfaces and GPS capabilities of phones and tablets provide some issues and opportunities not present in desktop computers.  It is also important to consider the possibility of a lack of internet access on mobile devices, particularly in remote areas.

For these reasons, even though the basic web technologies are the same, it is preferable to design mobile specific interfaces to your data. For our mosquito control project, the need for our field crews to enter data in the field was what drove our company to use a web GIS approach for this project.  We had contracts with 18 different municipalities and each had slightly different rules and regulations. We needed data collection software that ensured that the rules were followed as there were legal ramifications and a rigorous QA/QC process, so standard GPS data collection devices/software were not sufficient. We had previously handled data collection with a customized version of ArcPad on windows mobile devices.  This worked well for 5 years but the devices had outlived their useful life-spans and a new approach was called for.

We decided to go with a web GIS approach running on Android tablets. The hardware was much cheaper and we no longer had to pay for ArcPad licensing. That savings made it feasible to develop our own software which did exactly what we needed and nothing more. We needed the ability for our field personnel to see on the map, which sites were due for an inspection, and the results of previous inspections. We needed our field personnel to be able to view the site access information as most sites were on private land. Most importantly, we needed the software to do real-time validation to reduce the possibility of treatment application errors and the rules were different for each of the 18 municipalities we worked for. We also wanted the ability for office staff to send messages to the field staff to notify them of recently added “hot-spot” requests and for the field crew to notify the office staff of changes to site access.  And we wanted the office staff to be able to see where the crew members were and who was working on any given day in order to assign tasks appropriately.  In addition, cellular data coverage was spotty in some parts of our area so we needed the application to work even if a cellular data signal was not available.  Designing our own custom software with web GIS technology gave us the ability to meet all these requirements in a cost-effective manner.

Developing web-based analytical tools

I’ve developed many custom tools for people when no solution was available in standard desktop GIS applications. As powerful and flexible as desktop GIS software such as ArcMap and QGIS are, they simply can’t provide a solution to every analytical need out of the box. Most of my career the solutions that I developed were in the form of an ArcGIS add-ins. ESRI has made great improvements since version 10 in the ability to distribute these add-ins, however they were still dependent on Microsoft’s Visual Studio and their .NET framework.  When ESRI released new versions of ArcGIS they were built on the latest version of Microsoft .NET framework and add-ins created with older versions would no longer work.

On one occasion, I developed a tool to calculate species diversity as a favor in response to a post on a conservation list-serve that I am involved with. I didn’t charge anything for this, it only took me an hour or two and I knew it would save the client many hours of mundane editing and it was a cause that I wanted to support. Several years later they contacted me and told me that the tool no longer worked in the latest version of ArcMap that they had installed.  I did not even remember creating the tool and didn’t even have the source code any longer.  To make matters worse, ESRI doesn’t allow multiple versions of ArcMap on a single computer and I was elbows deep in a project using the previous version of ArcMap so I didn’t even have access to the recently released versions of either ArcMap or Visual Studio.

It’s very frustrating for both developers and those who use the tools that the developer creates to be at the mercy of the ESRI software update cycle. This may be less of a problem with tools created with Python, I’m not sure. It’s less of a problem with QGIS as there are no issues with having multiple versions on one computer.  Some people, however, don’t like installing new software on their computer, even if it’s free. Some organizations even have strict restrictions about installing unauthorized software. And of course, some people are hesitant to try new software because of the learning curve, so even if you develop a QGIS plug-in, people may be hesitant to use it.

If you can implement your tool with web GIS technologies, these problems disappear. Your tool will be available to anyone with an internet connection whether they use QGIS, ArcGIS, Manifold, Mapinfo or any other GIS system. It will be available to people whether they are using Windows, MacIntosh, Linux, or a mobile device. Most people don’t view going to a new web page as learning new software and they aren’t actually installing anything on their computer so the psychological and institutional barriers disappear as well. Of course, not every tool is suitable for web GIS. Tools that require outputting a new dataset are more difficult to implement (although not impossible). Tools that simply take input and produce an output table, report, or result set to display on a map on the other hand are generally easier.

Citizen Science or crowd-sourced data collection

One of the first projects I was approached with when I began looking into web GIS involved a small (one-person) non-profit working to increase awareness of black bear activity in the small mountain town I was living in. Every day this person would gather reports of bear sightings in town and make a map that would appear in the local daily newspaper. They were wondering what it would take to create a web site where citizens could add sightings themselves.  Due to my training as a wildlife biologist, I was excited to implement this project as I saw a lot of advantages to having this information available in digital form to track changes in bear activity by time of day, season, in response to annual influences such as drought and cold snaps that would affect natural food availability, and long term changes resulting from management activities and/or population trends.

Today I could implement a project like this in a few hours using open-source software with no cost other than a few dollars per month for web-hosting. Ten years ago, however, both my knowledge and available technology were far less advanced. I had taken several courses on creating web maps with the ESRI JavaScript API, but the ability to edit data via web interfaces was new and I did not have access to ArcServer. ArcGIS online was still in its infancy and did not have anywhere near the capabilities that it does today. It would have taken thousands of dollars in software and probably many weeks of my time to learn what I needed to implement this project and I was unable to convince my boss that the long-term benefits justified the cost to him.

How web GIS works

Programming for the web, whether it’s for public consumption or for restricted viewing via an organizations intranet, is fundamentally different than programming for a single-user computer. The difference is in the client-server architecture on which the web is built. Client-server architecture essentially means that there are many clients that interact with a single database on a server.

The web evolved in a bottom up fashion with little top down organization or control. Technologies that were adopted as standards were done so largely because they had the largest user-base. PHP, for instance, was developed by Rasmus Lerdorf for his own use on his personal web site. He never intended it to be a programming language, let alone to interact with databases. Nevertheless, PHP is currently used in over 80% of web pages that have a server side database component. This bottom-up approach is the reason that there are so many different technologies and why programming on the client uses a completely different language than programming on the server (unless you are programming on the server with Node.JS). This conglomeration of different technologies is often frustrating to people who are just beginning to learn web programming.

Client-side web GIS

Programming on the client has largely standardized on three general technologies. HTML provides structure for the web page and includes content. CSS allows the structure and content included in the HTML to be styled into an attractive appearance. JavaScript is a full-fledged programming language that allows the programmer to perform calculations, respond to events, and generally allows the web-browser to do things other than simply display content. In terms of the web, the client is a web browser. All modern web browsers can interpret HTML, CSS, and JavaScript. There are also a myriad of general purpose libraries available that simplify using the three basic technologies. Bootstrap is a very popular CSS library and jQuery is a very popular JavaScript library.

There are also libraries available for specific purposes, such as mapping. Google Maps provides an API that allows you to interact with google maps by adding your own data, for example. ESRI also provides a JavaScript API that can include ESRI base layers and feature data from ArcServer. Leaflet and OpenLayers are open-source JavaScript libraries that each have specific advantages and disadvantages.  In general leaflet is a simple, lightweight library that depends on community provided plug-ins for more advanced capabilities. This allows the programmer to pick and choose exactly which features they want to include. Open Layers is a more complete and fuller-featured library.

No matter which mapping library you choose, the first thing most GIS users will want to do is put their own data on it. If you have access to ArcGIS Server, the ESRI JavaScript API makes it easy to include that data on your map. OpenLayers works well with the open-source WMS and WFS services. All four of these can display data stored in GeoJSON format (or retrieved from a database on the server and converted to GeoJSON). GeoJSON is a text based standard for storing vector GIS data that works particularly well with JavaScript and PHP. You can use JavaScript to interact with your own data, for instance by filtering it, searching for specific features, viewing feature attributes, zooming and panning, etc. There is also an excellent JavaScript library called Turf.js that provides spatial analysis capability in the client. Turf.js also takes data in the form of GeoJSON.

If all you need to do is to display static data on a map you can get by with only client-side technologies. You can store your data in GeoJSON format and load them onto the map, filter them by attributes, and perform spatial analysis on them. If your data is static and doesn’t change much, you don’t need the complexity of a database server.

Server-side Web GIS

If your data is changing rapidly and you want the user to see the most current data, or if you want to give the user the ability to add and/or modify the data, you need to store that data in a database on the server. You will also need to learn about server-side technologies and probably a different programming language as well to interact with the database. PHP is the most common server-side programming language, but others include Java, Ruby, Python, C# (using Microsofts ASP.NET technology). You can also write server side code in JavaScript using Node.JS. You can manage and interact with the same data on the server from most desk-top GIS software as well.

ESRI has developed a database extension called ArcSDE that adds geospatial functionality to several commercial and open-source databases, including SQLServer, Oracle, and PostgreSQL. PostGIS is by far, the most common spatial database extension in the open source world. PostGIS works only with the PostgreSQL database, although most open-source databases have some level of spatial capability.  Spatially enabled databases generally add three primary functions.

• Spatial data storage – Generally, spatial data (coordinates) are stored in binary format as a BLOB field, while attributes are stored simply as text and numeric fields in the database.
• Spatial functions – Most spatial databases include a set of custom functions that provide the ability to convert between data types and spatial reference as well as perform spatial analysis. PostGIS has over 1000 of these custom functions.
• Spatial indexes – Enterprise databases are highly optimized for indexing and spatial databases are likewise optimized for spatial indexing. This makes searching and geoprocessing more efficient.

Your web page can interact with the database using SQL commands. Structured Query Language, or SQL allows you to create, modify, delete, and retrieve data stored in a database. While entire courses can and have been written about SQL, the basics are pretty simple and you can do a LOT with the basics.  If you’ve ever used visual query builders in ArcGIS or QGIS to select by attributes then you’ve used SQL. They are just visual interfaces to build a SQL where clause.  PostGIS extends SQL with its custom spatial functions so you include spatial criteria in your queries, such as retrieve all the lines that are within 200 feet of a point or all the points that intersect a polygon.

Most of your interaction with the database in your web page will come in the form of SQL statements to retrieve a certain set of data. That data needs to be turned into a form that your mapping API can understand. Server side programming languages such as PHP can convert the data you retrieved into GeoJSON format that you can then send back to the client to display on the map.

You can also set up a service to interact with the database. This is what ArcGIS Server does. In the open-source world GeoServer and MapServer can do the same with PostGIS data and other standard database formats. Personally, I have not yet found anything that I could not do with direct SQL queries to the database and thus have not seen a need to add an additional layer of complexity to my applications.  If you want to provide other people with the ability to include your data on their own maps without giving them access to your database you can accomplish this by publishing your data as a web service using GeoServer, MapServer, or ArcGIS Server.  If you have raster data that you want to serve on a web page you may also benefit from GeoServer or a similar tool.

How to get started

Non-programming approaches

There are essentially two approaches to getting started with web GIS. ESRI has been pushing its “non-programming” approach, especially with ArcGIS Online. They also offer a suite of mobile apps that primarily act on data accessed as services by ArcGIS Server or hosted on ArcGIS Online. They allow you to author a map in ArcGIS desktop and publish it to ArcGIS Online or ArcGIS server. If you are an ESRI user and have access to their products, there is a lot that you can do without programming. I can’t tell you much more than that as I gave up on ESRI products several years ago for several reasons.  As a programmer, I found it far easier to develop my own applications than to try to figure out their licensing and user account approaches and I just couldn’t justify passing their costs on to my clients.

QGIS also has the ability to author static web maps. The QGIS2Web plug-in will take a QGIS map document and “publish” to the web, although the process is far simpler and more limited that the ESRI approach. What it actually does is create a web application using either Leaflet or OpenLayers for the map canvas. The data on the map is converted to GeoJSON files, symbology is preserved as much as possible, and even pop-ups that you create in QGIS are converted to the web map. Then you can simply upload the HTML, CSS, and data files to your company’s web page.  This works great as a starting point for publishing data.  If your company has a secure client log-in system set up already, you can easily add a static web map to their access area. The down side is that you are only serving static data and thus you do not have the ability to edit it.

There is also an open-source project called GeoNode that seems to have similar functionality to ArcGIS Online. I have not used GeoNode, I don’t know much about it, but it allows you to set up user accounts and host GIS data much as you would with ArcGIS Online. It also allows editing of hosted data. Perhaps one of my readers has experience using GeoNode that they can share.

Programming Approaches

As appealing as a “non-programming” approach to building web maps sounds they have limitations. Perhaps, as a programmer I am biased. I like to understand how things work under the hood. I get really frustrated when I am using software that can’t do something simple that I need it to do and that then requires me to do hours of monotonous work by hand when I know it could be automated.  I hate inefficiency. I very much prefer writing my own programs to do exactly what I want and nothing more, and that I can fix myself if there are problems, or if project requirements change.

Most of the projects I have worked on have been sufficiently complicated that “non-programming” approaches were not enough. It’s not a knock on ESRI and other companies who put a lot of effort into making things flexible enough that non-programmers can use them productively. It’s just that the real-world is complicated and there is no way to anticipate all the possible things that a web GIS application may be called upon to do. On the mosquito control project that I described above, I had to hard code in rules specifying which treatments should be applied and how much, depending on the inputs to several previous attributes. There’s simply no way to anticipate those things in a generalized non-programming approach.  The alternative is to not include those rules and not validate the data at the time of entry. Doing this, however, usually means more problems with poor-quality data and more time spent with QA/QC at the end of the project and nobody enjoys doing that.

Having made the jump into learning the basic technologies required for web GIS I can generally put together a basic web map or mobile data collection application, such as could be done with a “non-programming” approach in a few hours. And then if the client asks me if I can add a specific feature I can usually say “yes”. And I can usually add that feature far quicker than I can investigate if its available without programming and how to implement it.  And when, as is usually the case, I find out that its going to require a custom modification it is far easier to implement than to try to figure out somebody else’s code in order to modify it. Most importantly, if the project is not going to have a huge number of users, I can offer to host the site myself. At least I don’t have to tell my client that they need to pay $500 per year for an ArcGIS on-line subscription for each user who wants access to their own data.

Making that jump was not easy. I took a lot of web development courses. I went down a lot of rabbit-holes chasing the latest and greatest technologies. I learned a lot from all of them, but I eventually concluded that a solid foundation in the basic technologies of HTML, CSS, JavaScript, PHP, and SQL was the best approach. With those languages, there is little that you cannot do with web GIS. Depending on your application, you may be able to do some things more efficiently with Angular.js, React, Node, MongoDB or other more advanced technologies but you have to learn to walk before you can learn to fly.

Learning any one of those languages is not that difficult. There are many resources available for all of them. Books, on-line courses, etc. What was difficult was really understanding how they all work together to form a web application. In particular, there are some specific differences between developing a web GIS application and a standard non-mapping web application.

• Some parts of JavaScript, such as objects and JSON, are critical to understanding web GIS applications but they are not generally prioritized in standard web development courses.
• Most web development courses use MySQL as the backend database. This makes sense as it is simple and very common on web hosting services. Most web GIS applications, however, use PostgreSQL as the backend database because of the availability of PostGIS. You’re on your own if you want to install PostgreSQL and modify the code to take advantage of PostGIS.
• Because web GIS applications rely on a JavaScript mapping library, it makes sense for them to be built around JavaScript and use AJAX to communicate with the server. With this approach PHP is only used to respond to AJAX requests. Many PHP courses spend an inordinate amount of time teaching you to build PHP applications that are not suitable for mapping applications.
• Many PHP courses also teach methods for interaction with the database that are inherently insecure. While it’s OK to learn these methods for simplicity’s sake, it’s not OK, in my opinion, to end the course without teaching students more secure methods. Learning about security issues and how to address them took me quite a bit of time, even after I knew how to “make things work” in an insecure manner.

Warning: Mild sales pitch ahead

For this reason and many others, I put together a course on web development for GIS applications. Basically, I made the course that I wish had been available when I started learning about web GIS. It includes an introduction to all the core technologies. None of the sections on the core technologies are a thorough treatment, rather my purpose was to provide enough information to understand how they all work together, particularly in the context of a web GIS application. I also introduce some standard libraries such as Bootstrap and jQuery that will make your life much easier as a web developer. Most importantly I introduce web GIS specific libraries and technologies such as Leaflet, Turf.js, PostGIS and GeoJSON, and explain how to make them all work together in a web GIS application. This course is intended to be background material for more advanced courses on web GIS, including a full course on client-side programming with Leaflet and client side mobile application development as well as courses on server-side programming for web GIS applications  and spatial databases.  More information on these courses is available on the courses page of this blog.

If you are interested in learning more about developing web GIS applications on your own, I hope that you will consider these courses and find them useful. They have been released on the Udemy platform which means that they are available anywhere, anytime, for life.  All of my courses on Web GIS will be available for $12.99 through November 26, 2021 using the coupon code WEBMNTH.