Use Mapbo to Create a Gpx Uploader

Data STORIES | 3D MAP VISUALIZATION | KNIME ANALYTICS PLATFORM

KNIME & GPX: how to build a 3D map with mapbox.com

Using no code tools to procedure massive geoloc information

Equally first published on Restless Engineer

If you lot have read my previous commodity, you know that I beloved visualizing and working with geolocated data. What could be the reason? Maybe my natural predisposition to travel and explore. Or is it easier to obtain data of this type? I have no idea.

Anyway, lately I have been exploring how to create a 3D map to visualize information from routes in GPX format. There is more than and more than geopositioned information, and being able to visualize it quickly and easily while having it embedded in a web page is the best way to analyze it. Merely how to approach the whole process without programming?

The initial premise is uncomplicated: geolocalized information available to be processed with no-code tools, and an easy style to visualize it that can exist integrated into a WordPress-based website.

Claiming

As you may know, I have a special involvement in the Camino de Santiago. Combining engineering and the Camino gives me the chance to help my fellow pilgrims by showing them data that would otherwise exist difficult to obtain. Or just looking at it in a different style.

Objective: flying over the peaks of the Pyrenees forth the first stages of the Camino de Santiago through Aragon.

In this example, the challenge, based on the previous premise, was as follows:

• Using public information from the CNIG nearly all available routes of the Camino de Santiago in the Iberian Peninsula, visualize them embedded in an commodity of my web folio.
• Provide extra added value to GPS positions. I was virtually certain that the files would have some elevation inconsistency. This is i of the near interesting features for pilgrims who have to behave a haversack.
• Utilise no-lawmaking tools and share it in a mode that anyone could replicate and ameliorate it. I am not an expert developer and information technology is a personal project. I am more than interested in the technical viability than the perfection of the result.

Starting data

As I said earlier, the starting data is all the GPX files available on the CNIG portal (Spanish National Center of Geographic Information). This data has been provided by the Federación Española de Asociaciones de Amigos del Camino de Santiago (FEAACS). Data is available in both GPX and KML formats. I chose GPX because I found it easier to extract the data to an intermediate JSON file.

Afterward downloading each of the 1,015 private files corresponding to all the stages of all the variants of all the Caminos, I unified them into a single database of GPS points using the open source, no-lawmaking tool for data science par excellence: KNIME Analytics Platform.

Here is the detail of the GPX parser workflow that I implemented with KNIME.

GPX Parser component to encapsulate all the logic.

First, I ingested all files using the List Files/Folders node. And so, I extracted the path information from all GPX files, read them every bit text-based files and catechumen them to XML. Next, I converted them to JSON (I am more comfortable processing JSON than XPaths) to prepare the strings for the extraction of information about the routes and stages: the RegEx Extractor, String Manipulation and String to Number nodes.

Inside the GPX Parser metanode.

I am sure that more advanced KNIME users will notice a better manner of doing this. Go out a annotate and give me the opportunity to learn :-).

During this ETL process, I noticed that some files were missing or even data within some of those files. This is when a tool like KNIME helps yous tremendously to alloy data from unlike sources. In any case, I already had 398,000 GPS points to begin with.

Missing elevation data

During the data blending process, I realized that there were stages and caminos that did not have elevation information. To be honest, extracting just the GPS information and displaying it didn't seem sufficient simply I also didn't want to incur the expense of an API like Google'south to go the tiptop.

Later on some enquiry, I found an API service (open-elevation.com) that I could use to automate those queries and get the data. Past existence careful with the calls, I was able to download the data for over 105,000 points that had previously wrong or missing elevation information in less than 4 minutes. The steps would exist every bit follows:

1. Identify points that accept no elevation data or have an error.
2. Filter this data to generate the JSON that I will utilize as a request torso in a Post call to the open-elevation.com service.
3. Generate a loop to request a maximum of i,000 points per phone call. Be kind my friend.
4. Generate the JSON with the breadth and longitude data in the required format. The nodes in this case would be: the Columns to JSON and JSON Row Combiner.
5. Now nosotros have a JSON table with the arguments for the API POST. In this way, we volition only need to use a Mail service Asking node with a delay of 2,000 ms.

Workflow for GPS position elevation requests.

Displaying elevation information visually

Before moving on to the side by side challenge, i.eastward. integrating mapbox.com service, let'southward cheque that the information obtained is, at least, consistent.

To do this, start I unified the 398,000 points and used KNIME's quick display capabilities. I tested both the Wikimedia Map and Bing Map (OSM Map View node).

Elevation data visualized on Bing Map using the OSM Map View node.

If we look at areas such as Cruz de Ferro (León) or the areas of Picos de Europa, we tin can clearly see how the tonality changes, indicating greater terrain elevation.

Checked… so far so adept.

Mapbox (3D map): connecting to the service

After processing and cleaning the data, I used the Mapbox'due south map service to display all the information with its 3D engine. AGILE mode: ON. Let's start by importing all the points in the Mapbox editor (Mapbox Studio) and check if we tin integrate it in a web article.

1. Create a new mode with the Terrain 3D display to show the information of the points that we processed.
2. Create a new tileset by importing the generated CSV file with the following information (Table Manipulator node): camino ID, stage ID, description of the Camino, description of the phase, latitude, longitude and elevation.
3. We associate the tileset with the style and so adjust the display according to the acme. Hither is a screenshot.

Adjusting the display of information in Mapbox Studio.

Now we but have to publish information technology and embed it in our website. Afterwards searching for how to practise information technology, the truth is that information technology was simpler than it seemed. Here you have the code to integrate it in your WordPress page or post.

CSS, mapbox integration and zoom/rotation controls.

Calculation some interaction…

… nigh individual points.

This is the merely point where I added some code to improve the user interface by showing a pop-up with the data of the point when yous click on whatsoever of them. All y'all accept to practise is insert your ain [accessToken], [user] and [mode].

Yes, you are correct, I had to include some JavaScript code in a no-code. My error. It won't happen again.

Result

You lot tin can see the issue (and play with it) on my Camino de Santiago website: All the Caminos in 3D.

Just an example of how can yous explore the data in an interactive style.

Here is likewise the complete workflow and then that you can replicate it. I uploaded it on the KNIME Hub for you to download it for free.

KNIME workflow for processing the GPX files of all the Pilgrim Roads of Santiago.

Summary

• Database of GPX files with the stages of the National Middle of Geographic Data (CNIG). We can never exist grateful enough to the Spanish Federation of Associations of Friends of the Camino de Santiago (FEAACS) for the piece of work of dissemination and compilation that they practise. Information technology is absolutely fundamental to build all Caminos in 3D.
• KNIME Analytics Platform. Equally y'all tin can see, I have used it mainly for the process of ingesting, transforming and updating information earlier uploading it to data visualization services. On this occasion, the focus was on updating the top information to generate the profiles of the selected stages, variants and Caminos.
• Open up Summit API. Equally they describe information technology themselves: "Open-Pinnacle is a gratuitous, open source alternative to the Google Acme API and other like offerings." Same #nocode philosophy: admission to your API service with Mail nodes and JSON Row Combiner to generate the request for those runway points that had no associated elevation data. Actually easy.
• Mapbox. Possibly one of the best map services that already in its free version offers a lot of functionalities. Their Mapbox Studio suite is a small wonder for generating maps. It as well enables 3D visualization with an impressive degree of particular. To load the data I used a simple CSV file with the information of the points.

When new GPX files become available, I already have a workflow for the automatic processing and increased map coverage.

Future Improvements

Here are some improvements that I would similar to work on with a petty more than time:

• Include the modules to practise the loading and updating in the system automatically using the API provided past Mapbox.
• Automatically connect to the download surface area of the Castilian Federation of Associations of Friends of the Camino de Santiago (FEAACS) to increase the number of supported routes: the European ones.

Go out me a comment if you would like me to include information that may exist of your interest or to explain a signal that may not exist clear.

chamberlaincoth1990.blogspot.com

Source: https://medium.com/low-code-for-advanced-data-science/knime-gpx-how-to-build-a-3d-map-with-mapbox-com-6b68cbc1b304

Share this post