Open Data in Software Development

How software improves when high-quality open data is available for free.

No time to read the whole story? 😁 Here are the main take-aways — aka my personal hypotheses — from this blogpost:

Open Data in Software Development

  • Easy and fast access to Open Data is critical for software development
  • Open Data is to be preferred over manually created test data which is time-consuming and does not reflect reality

Software Development with Open Data

  • Open Data leads to more inclusion and diversity in the development process as it avoids overfitting
  • Open Data enables innovation and more efficient development cycles

Feel free to challenge these hypotheses in the comments below 👇

Manhattan Skyscraper Explorer by Raluca Nicola — using Open Data published by the city of New York.

About a year ago, I was invited to an Open Data Beer in Zurich where I spoke about how we use Open Data at the Esri R&D Center Zurich. I finally found some time to work on this write-up and looking forward to hearing your thoughts about this and how you are using Open Data in your daily work 😊

Open Data

Only a small subset is made available as Open Data, mostly data captured and maintained by government, universities and research institutes, as well as non-profit organizations. Even some private companies work on making Open Data accessible to the public as an example of Google’s research database shows that can query over 25 Mio. freely available datasets. There are several global initiatives to drive the accessibility of government data — technically it belongs to the people anyway as they’ve paid for it with their taxes 😎. But Open Data is not just data that is for free — it also needs to be machine-readable, well documented, and up-to-date. If you are interested in learning more, you can read more about the requirements for data to become Open Data here.

To achieve the goals of sustainable development, critical data must be open and available for reuse by anyone, anywhere, anytime. — Tim Berners-Lee

Frankly speaking, Zurich is an Open Data paradise 😍. Already as a student, I made use of the amazing data made available by both the City of Zurich and the Canton of Zurich. For example, in one of our semester projects we built a fun location-based game called Pioneer — clearly inspired by the board game “Settlers of Catan” — where we used Open Data to display sources of water, wood, and metal connected to real-world objects in the city such as fountains or trees. It was extremely easy to find, download, and use the data for our mobile app.

In my experience now as a Software Developer, the lack of available high-quality data tends to, unfortunately, stop exciting and innovative projects from going further — especially when exploring the machine learning universe of endless possibilities. We also have a strong focus on geospatial data which is even rarer to be found as it typically results in higher maintenance.

BUT I really don’t want to complain 😁 because we still get to work with incredible data that helps us all along the software development life cycle.

Software Development

Classical software architecture (MVC) with data as the foundation.

But data is not only the foundation of most software products, it is also used throughout the software development life cycle to increase the quality of software. Of course, data is being used throughout the whole life cycle. In the following I will be showing you different examples from three stages of the software development life cycle where we are frequently using Open Data:

  • Open Data for Analysis
  • Open Data for Implementation
  • Open Data for Testing & Integration
Software Development Life Cycle and highlighted stages at where we frequently use Open Data.

I believe that the way software is pre-dominantly developed today — in an agile way— has made access to Open Data more important than it used to be. Less time available for each stage of the development cycle and a lower level of specialization of the people involved require easy, fast, and open access to relevant datasets.

Open Data for Analysis

The outcomes of this phase are usually POC’s — proof of concepts. Many design iterations lead to a visual and vivid prototype where everybody involved can imagine the final feature’s look & feel. At the same time, the feasibility of the implementation has been explored in order to make assumptions about workload, technical challenges, dependencies, etc.

Example: Interactive data exploration is a great way to make users better understand their data, for example in a city environment. In a 3D space, exploration can be quite complex and the main challenge is to keep the visual clutter low while presenting all the necessary information to the user. The below image shows a prototype that we have built to explore this use case further. It uses the openly available, amazingly detailed 3D buildings of the city of Zurich to map space use onto it and display it throughout the city.

c-through: A proof of concept for interactive 3D data exploration.

If test data is needed in this phase, it is often mocked — mainly due to time constraints. It is either created manually from scratch (often overfitting the use case) or generated automatically with some assumptions in mind. Sometimes this results in rather unrealistic data — as you can imagine 😆 — with smaller or larger implications.

When developers create their own test data, it can result in unrealistic data entries.

✔️ Why is it important to use Open Data for this stage in the software development process? So that early (design) decisions can be made with realistic information at hand. Realistic test data can show you immediately what you will have to deal with later on for implementation. Additionally, realistic data will prevent you from overfitting your data to the use case.

✔️ And what type of Open Data is needed for this stage? Datasets that are easy to understand and parse, and easy to embed and use in prototypes. If it takes you two days to understand the schema of the dataset & another three days to download and convert the data to the right format, then you probably won’t end up using it for your prototype.

Open Data for Implementation

When implementing new features, we often think and take into account edge cases — rare occurrences of what users would want to do, or how data could look like (or both at the same time — jackpot⭐). Even though edge cases tend to be rare, they still happen & it is our responsibility as developers to catch them, handle them elegantly, and most of all: make sure they don’t crash the app.

Example: To formalize the relationships between different datasets, we made a preliminary assumption that a parcel (lot) geometry as shown below would map to one underlying zoning geometry — in 99% of the cases. When we started working with real zoning and parcel data, as shown here by the city of Zurich, it quickly turned out that parcels with multiple intersecting zoning boundaries are actually not that rare.

Not-so-rare edge case: a parcel on two different zoning boundaries.

Coming up with all possible edge cases is a challenging exercise as we often cannot even imagine how data can look like until we have seen it. Edge cases (which are not always so rare) often only show up when you start working with real data. If you start testing your application with real-world data only after implementation (or after releasing it to your users*), you will likely miss certain edge cases.

When it comes to data, we also need to make an effort to protect the user from creating (or importing) invalid data. Catching edge cases early & being able to handle them thoughtfully (and not in a rush for a hotfix after release) will spare you a lot of trouble.

✔️ Why is it important to use Open Data for this stage in the software development process? In order to account for many different occurrences of data entries and to catch edge cases during implementation — and not after.

✔️ And what type of Open Data is needed for this stage? Rich and diverse datasets that can provide many use cases, e.g. time series or datasets from different locations.

*side note: there is of course validity in conducting beta testing or A/B testing with users before officially releasing a (new) version of your product. We have made good experiences collaborating with early-adopters before releasing the first version of our product ArcGIS Urban in order to account for a variety of user requirements and test it out with different datasets.

Open Data for Testing & Integration

Example: 3D web applications — running in a browser environment — are often limited by memory constraints. 3D data can be highly detailed, especially when showing complex geometries as well as realistic textures, and therefore fill up browser memory quickly. For our web applications that display multiple three-dimensional datasets at the same time and over relatively large areas, we needed to make sure that we are aware of the limits that the browser can handle for display. Finding those limits allowed us to first restrict certain functionality, e.g. to a maximum feature count, which still enabled most of the use cases — but not all of them. Then, we continuously worked to expand these limits, e.g. by using tile-based data streaming, using different levels of details when displaying data, by introducing new data formats (e.g. i3s), etc. Over time, the limits could be set higher and eventually even be removed completely.

Large volume and diverse data in a 3D scene for realistic performance testing.

Testing is done to ensure the quality of the software, but also to find areas of improvement. Innovation often sparks from discovering something new — a new use case or a new requirement. I believe that automatically generated and potentially overfitting datasets do not leave a lot of room for improvement. Realistic data on the other hand can show you new ways to help solve problems that your users are dealing with.

✔️ Why is it important to use Open Data for this stage in the software development process? To find the limits of the software, to expose it to diverse data, to ensure quality, and to spark innovation.

✔️ And what type of Open Data is needed for this stage? Large and rich datasets, complex geometries, the same type of data in different locations, and sometimes even “unpolished” data can be helpful.

Final Thoughts

I strongly believe that software development can highly benefit from Open Data — and vice-versa. When thinking about the use cases of openly available data, software development is maybe not the first to pop up in people’s minds. However, also in research, this use case has recently been picked up (see quote & article below) and analyzed in order to improve the formats and accessibility of Open Data — specifically taking software development as a use case into account.

The way open data resources of varied type and volume are used by software applications remains only partly known. —”Towards Increased Understanding of Open Data Use for Software Development” by Maciej Grzenda & Jaroslaw Legierski, in Information Systems Frontiers (2019)

Therefore, I hope that I was able to successfully explain here how we are using Open Data, and show that this is a relationship worth exploring. In hopes that more and more Open Data will become available in the future and software development can continuously be improved by making use of it. Thanks for reading along 😍

I am looking forward to hearing your thoughts and experiences about the use of Open Data in software development 🙌 comment below or reach out to me if you’d like to continue the conversation!

Software Engineer @ Esri #arcgisurban. Co-Managing Director @ Girls in Tech Switzerland. Self-employed yoga teacher.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store