Posted 07 December 2024

I recently came back to Condor virtual soaring after several years away, and also started using XCSoar on a tablet as an auxiliary navigation tool. One challenge in doing this is getting the scenery (.XCM) files associated with the various Condor sceneries. Some years ago ‘Folken’ (folken@kabelsalat.ch) created the https://mapgen.xcsoar.org/ web app to facilitate this process. The website accepts a map name, email address, and map bounds information and produces the corresponding .XCM file, which can then be dropped into XCSoar for navigation support – neat! Map bounds can be defined three ways – as manually entered max/min lat/lon values, as a rectangle on a dynamic world map, or as a waypoint file (.CUP or .DAT).

Unfortunately, as I and several others have found, the web app doesn’t actually support waypoint file bounds; it produces a ‘unsupported waypoint file’ error whenever a waypoint file is submitted. The developer has been unwilling/unable to work the problem due to other demands on his time, so I decided to take a shot at finding/fixing the problem.

First attempt: ‘backend code’ assessment: https://github.com/paynterf/XCSoarMapGenDebug

Last February (Feb 2024) I looked through the github repository, and because I am totally clueless regarding modern (or any age, for that matter) web app development, I decided to concentrate on the ‘backend code’ to either find the problem(s) or exclude this code as the culprit. To do this I created the above repo, and eventually worked my way through most of the backend code, without finding any issues – oh well.

Current attempt: Build and run the web app (ugh!):

After ignoring this problem again for almost a full year, I decided to take another shot at this. Folken’s website has a Readme that details the process of setting up a web server on a Debian linux box, and since I happen to have an old laptop with Debian installed, I decided to give this a whirl. The Readme describes how to use an ‘Ansible Playbook‘ to build and provision an XCMapGen web app. I tried this several times, and to say I got ‘a bit disoriented’ would be the understatement of the century. After having to reload Debian on my laptop several times, I reached out to Folken for help. Amazingly, he actually answered and was (and is) quite helpful. He told me he had gotten away from Ansible and was now using Docker for website build and provisioning. In addition, he gave me detailed steps on how to use Docker to bring up an XCSoarMapGen website on my Debian laptop. Here’s his email back to me:

Of course I had never heard of Docker (or Flask, or Cherrypy, or Ansible or…..), so I was in for some serious research work. Eventually I got the XCSoarMapGen website running at ‘localhost’ on my Debian linux laptop, and for me that was quite an achievement – from web ignoramus to web genius in 234 easy steps! 😎

After some more help from Folken, I got to the point where I could watch the activity between the web page and the backend code, but eventually I decided that this wasn’t getting me anywhere – I really needed to be able to run the backend code in debug mode, but of course I had no idea how to do that. After lots more inet searches for ‘Debugging web applications’ and similar, I found (and worked through) a number of tutorials, like Marcel Demper’s “Debugging Python in Docker using VSCode” and Debugging flask application within a docker container using VSCode, but I was never able to actually get my localhost XCSoarMapGen app to run under debug control – rats!

However, what I did get from the tutorials was the fact that the tutorials both referred to Flask (a software tool I’d never heard of before), while Folken’s XCSoarMapGen app uses CherryPy (another software tool I’d never heard of before). So this sent me off on another wild-goose chase through the internet to learn about debugging with Cherrypy.

After another hundred years or so of web searches and non-working tutorials, I never figured out how to actively debug a Cherrypy app, but I did figure out how to insert print (actually cherrypy.log() statements into the code and see the results by looking at the ‘error.log’ file produced by (I think) Cherrypy. Here’s the ‘parse_seeyou_waypoints()’ function in the code:

And here’s some output from ./error.log:

So, the run/debug/edit/run cycle is:

• Make changes to the source code, insert/edit cherrypy.log() statements
• Rebuild the affected files and restart the XCSoarMapGen website at localhost:9090 on my lilnux box with ‘sudo docker-compose up –build’
• Reconnect to the error log output with ‘sudo docker exec -it mapgen_mapgen-frontend_1 bash’ followed by (at the #prompt) ‘tail -f ./error.log’

Here’s the startup command and resulting output:

And here’s the commands (in a separate terminal) to start the logging output:

So now I can run the web app at localhost:9090 on my linux box, and watch the action via cherrypy.log() statements – cool!

After a while I had narrowed down my search to the ‘parse_seeyou_waypoints(lines, bounds=None)’ function in the ‘seeyou_reader.py’ file (repeated here for convenience).

Here’s the email I sent off to Folken:

After sleeping on this for a while, I realized there were two other mysteries associated with this file.

• I noticed the waypoint printout from the temporary for loop at the start of the function showed the letter ‘b’ prepended on each waypoint line, and that isn’t what’s actually in the waypoint file. I have no idea how that letter got added, or even if it is real (could be some artifact of the way linux handles string output) – it’s a mystery.
• I realized there was a disconnect between the syntax of the call to ‘parse_seeyou_waypoints()’ in ‘parser.py’ and the actual definition of the function in ‘seeyou_reader.py’. In parser.py the call is ‘return parse_seeyou_waypoints(file)’, but in seeyou_reader.py the definition is ‘def parse_seeyou_waypoints(lines, bounds=None):’ So parser.py is sending a file object, but seeyou_reader.py is expecting a ‘lines’ object (presumably the list of lines read in from the waypoint (.CUP) file).

Just to make sure I wasn’t blowing smoke, I did a ‘git grep parse_seeyou_waypoints’ in the repo directory and got the following:

This shows there is exactly one call to parse_seeyou_waypoint(), so it’s clear that what is being sent is not what is expected. I’m not sure that this problem is the reason that waypoint files aren’t being processed, but is sure is a good bet.

So, what to do? It looks like both the parse_winpilot_waypoints() and parse_seeyou_waypoints functions are called with a ‘file’ parameter, and expect a ‘lines’ parameter, so it’s probably best to do the file->lines conversion in parse_waypoint_file(). Just as a sidenote, I noticed that ‘parse_winpilot_waypoints()’ doesn’t include a ‘bounds = None’ default parameter – wonder why?

08 December 2024 Update:

Well, I ran around in circles for quite a while, trying to get my head around the issue of ‘mixed signals’ – where the call from parser.py is: shows

but the definition of parse_seeyou_waypoints() in seeyou_reader.py is

To add to the mystery, it is apparently OK to treat the ‘lines’ argument as a list of strings, so:

and this loop correctly prints out all the lines in the selected .CUP file (albeit with a leading ‘b’ that I can’t yet explain). So clearly there is some Python magic going on where a ‘file’ object gets turned into a ‘lines’ object on-the-fly!

09 December 2024 Update:

To try and clear up the ambiguity between ‘file’ and ‘lines’, I placed the following code into the ‘parse_waypoint_file(filename, file = none)’ function definition in parser.py

When I refreshed the web app, I immediately got the printout of all the lines in the file, even though the ‘Waypoint File:’ box showed ‘no file selected’. I suspect this is because the file information from the last selection is cached.

I closed and re-opened the web site (‘x’ed out the tab and then retyped ‘localhost:9090’ in a new tab), and then re-started the web app. This time the only readout was ‘At the top of the index.html function, with params = {}’.

Then I entered a map name and my email address clicked on the ‘Browse’ button and selected the ‘Slovenia.cup’ file. This did not trigger the code in parser.py.

Then I clicked on the ‘Generate’ button and this triggered the ‘lines’ display. Then I entered ‘F5’ to regenerate the page, and the line list printout was triggered again. So, I think it’s clear that the site chaches entry data.

OK, I think I might have gotten somewhere; so the line I added in parser.py/parse_waypoint_file(filename, file=None):

Works as expected, and loads the ‘lines’ object with a list of lines in the file. In addition, I could now change the call to ‘parse_seeyou_waypoints’ as follows:

Thereby (at least for me) getting rid of the headache I experienced every time I looked at the disconnect between the way parse_seeyou_waypoints was called from parse_waypoint_file and the way it is defined in seeyou_reader.py. When I run this configuration, I get the full waypoint list printout in both parse_waypoint_file() and parse_seeyou_waypoints() – yay!

However, we are still left with the original problem, which is that .CUP (and probably .DAT) files aren’t getting processed properly. I am now starting to believe that the ‘b’ character prepended on all the lines read in from the waypoint file is actually there. If that were in fact the case, that might well explain why the processing loop quits after line 1 (or maybe after line 2 – not entirely sure). When viewed in a normal text viewer like Notepad++ in windows, or in https://filehelper.com/view, I see:

but when I use ‘cherrypy.log(‘line%s: %s’ % (wpnum, line)) in a loop to display the lines in a linux terminal window, I get:

10 December 2024 Update:

So, the problem with the leading ‘b’ turned out to be an issue with the binary-to-ascii decoder used in cherrypy.log() statements. Because no decoder was specified, I got no decoding – hence the leading ‘b’. Once I bought a clue from yet another post to StackExchange, I prefixed any log statements with something like

The ‘ISO-8859-2’ decoder was used because some waypoints use eastern European accent marks.

At the end of the day today, I had worked my way through a number of other minor and not-so-minor problems (most caused by stupidity on my part), and arrived at the point where the code is properly processing the Slovenia3.cup file, at least as far as extracting field elements from the lines, as shown below:

As shown in the printout for row 26, the eastern European accent marks are being handled properly. The ‘parse_seeyou_waypoints()’ function responsible for this is shown below: Note that not all of this function is enabled yet – that’s next!

11 December 2024 Update:

I uncommented the rest of the parse_seeyou_waypoints(lines, bounds=None) function, and except for an oddity regarding the default parameter ‘bounds’, it all seemed to function OK. When parse_waypoint_file() calls parse_seeyou_waypoints() it doesn’t use the ‘bounds’ argument, so it is set to ‘None’ at the start of that function. And, there is nothing in the function to initialize ‘bounds’ to anything, so I’m not sure why it was included in the first place. The ‘bounds’ object is referenced twice, as follows:

but since ‘bounds’ is never initialized and is always ‘None’, these two ‘if’ statements always fail. This is definitely fishy – maybe the original coder had something in mind that never got included?

As I understand things so far, the purpose of ‘parse_seeyou_waypoints(()’ is to create a list of waypoint objects and return it to the calling function with

Since the calling code in ‘parse_waypoint_file()’ is:

I think this means that whatever calls ‘parse_waypoint_file()’ receives the now-filled waypoint list. This appears to be the calling code in ‘server.py’:

So server.py calls parse_waypoint_file with the filename and the file object (pointer?). parse_waypoint_file extracts a list of lines from the file, and then passes that list to (for the .CUP file case) to parse_seeyou_waypoints(), and gets a waypoint list back, which is then calls the ‘.get_bounds()’ method in the waypoint list class. The ‘get_bounds() method is declared in the ‘WaypointList’ class as follows:

When instrumented as shown above to print out the final min/max lat/lons, I got the following:

Comparing the above figures with the actual Slovenia map in Condor2, they cover the measured map extents quite nicely – yay!

12 December 2024 Update:

While looking through the code in server.py, I ran across the following ‘if’ statement:

The intent of the above boolean expression is to warn the user that the extension of the selected file is neither ‘.DAT’ nor ‘.CUP’. So, if the extension is .DAT then the expression is false immediately, and therefore the warning is not emitted. If the extension isn’t .DAT, then the second half of the expression is evaluated, and it is true only if the file extension is NOT .CUP. So, it works but it sure is confusing. A much better way of writing this would be

At the end of the day I had things running pretty well, and I added a line in server.py to print out the min/max lat/lon values calculated from parsing the ‘slovenia3.cup’ file, resulting in the following output:

18 December 2024 Update:

At this point, I have the website working to the point where it can successfully parse either a .CUP file or a .DAT file and print the bounds in the ‘error’ field on the web page. In order to get to this place, I actually wrote a small python script to convert back and forth between .CUP and .DAT formats, so that I could test MapGen using the same exact waypoints in both formats. The calculated bounds, of course, should also be identical.

Here’s the bounds output from Slovenia3.dat: