Introducing the new 3d Terrain Viewer

[Boyd]

The new app is now online. Kind of a difficult birth, there are many complex things happening at the same time in this app that depend on each other. While everything worked fine on my development server, everything was much slower running live on the internet. This caused lots of errors, with data not fully loading. Have tweaked everything and it seems to work reliably for me now - let me know if you have problems. It is very slow to open a map now, but that can probably be improved after I study things and learn where the bottlenecks are. Anyway, here's a very quick introduction - I have been at the computer way too long today and am going out for a break after posting this.

This quarry near Pottstown makes a good demo of what the app can do...

https://boydsmaps.com/terra/#40.362.../0/1000/577/-1000/40/3/3d/shader10/ecm/-2/z17

Getting closer to the Pines, this is Arney's Mount

https://boydsmaps.com/terra/#40.006...9/0/1000/592/-1014/75/3.5/3d/ortho/ech/-2/z17

And here's @pinelandpaddler 's "mystery canal", which is a good example of the custom shader in the flatlands

https://boydsmaps.com/terra/#39.633...0/0/1000/600/-1000/45/2/2d/shader20/ech/0/z17

These are all direct links to the new app, I like to think of it as a "destination" instead of a "starting place". There are a couple basic tools for switching maps within the terrain viewer, but the main idea is that you will find locations in the main boydsmaps app, then go to the terrain viewer to look more closely. So, you would start by choosing any location on the map, then clicking the cube button in the toolbar.

This will bring up a settings menu that you can customize to open the terrain viewer. These will only be the initial settings, they can all be changed in the viewer. But it takes so long to open a map, it helps to have things setup the way you like at the beginning. You can just click the open button here and leave everything set to the default for now. Later, when you're more familiar with the software, you can set preferences.

Hopefully this is enough to get everyone started... really, you just need to play with all the controls and see what happens. Give it a try and let me know what you think. I'm gonna get a pizza and have a beer!

 

[bobpbx]

That is awesome Boyd. Really. I'm looking at a piece of land I was on that has ravines. What do you think the dark choppy line represents?

 

[bobpbx]

I can rotate the view in 3D, but not 2D.

 

[Boyd]

I locked out rotation in 2d, didn't see the point? If you want to do that, just use 3d view and tilt straight down.

The line is caused by errors in the terrain data. I wrote a bit about this in the help file. In the Pines, I suggest using the high error correction setting, it will fix quite a few problems like that (what setting were you using?) I spent awhile tweaking the high setting just for the pines, in fact.

Update 7/2/23: The elevation data has all been re-processed to remove the errors, so the error correction settings are no longer needed

 

[bobpbx]

I had jazzed it up pretty high just to emphasize them. I was just curious. As for the 2D, I don't see any way to switch from 3D to 2D other than the buttons under the compass. Oddly, when I rotate in 3D to make the railroad in the foreground parallel to the bottom of my screen, and then hit the 2D button, it rotates it as shown below:

Seen in 3D:

Then I hit the 2D button and I get:

 

[bobpbx]

Never mind, I found the tilt function in help. Duh.

 

[Boyd]

Something odd is going on in your top screenshot... there are four crosshairs and there should only be one. That is a bug I have seen before, possibly caused by switching map types or sizes a bunch of times. Have also seen it load more than one 3d surface at a time (you can see this in 3d mode when you tilt the map up). When you see this kind of issue, you might want to hit the re-load button in your browser. This is some pretty complex code with lots of opportunities for bugs. It's going to take a bit of a "shakedown cruise" for me to resolve some of these issues...

I had jazzed it up pretty high just to emphasize them.

I think you are talking about vertical exaggeration there?

To be clear, I was suggesting that you set error correction to high. It should do a better job of fixing the kind of gaps in the screenshot you posted earlier. The high setting usually doesn't work very well in mountainous areas, but in flat terrain like the pines, it is pretty good. That assumes you are using the 1.5 mile map size however, if you use the larger size (lower resolution) maps, the high setting might not work as well.

If you are mainly looking at terrain in the Pines, you can just set high error correction as the default in the menu when you click the cube button to open the viewer.

Update: error correction is no longer needed (now we're perfect!) and the settings have been removed.

 

[Boyd]

there are four crosshairs and there should only be one.

OK, found that bug. Everytime you change maps in the viewer, it adds another crosshairs! That is a straightforward fix, but going to wait until later today when I have some more time. Will also try to speed up map loading and will roll it all into an app update later today. In the meantime, using the reload button in your browser should fix the problem.

 

[Boyd]

FWIW, finally tried testing the terrain viewer on my iPad 6th generation running iPadOS 16.5. After going through the map loading process, it crashed with a message about repeated errors. This did not surprise me, as I didn't really expect it to work on an iPad or a phone. But you might have better luck on a newer iPad (my 6th generation iPad dates back to 2018).

If you are able to get the site working on any kind of mobile device, I'd be interested to hear about it. AFAIK, it's just not compatible due to the demands it makes on computing resources.

 

[Boyd]

That is a straightforward fix, but going to wait until later today when I have some more time. Will also try to speed up map loading and will roll it all into an app update later today

The update is online now. I didn't do any timing, but think it loads much faster now. Instead of just waiting a long time to be sure all the data has loaded, the new code can tell when the data finishes loading and finish up immediately. I can probably tweak this for even better performance, but will wait and see if anyone is having problems before making more changes.

 

[Boyd]

The line is caused by errors in the terrain data. I wrote a bit about this in the help file. In the Pines, I suggest using the high error correction setting, it will fix quite a few problems like that

These errors have bothered me ever since I got the terrain viewer working, but I couldn't find the cause. Now that the viewer is finished, I dug deep and it seems the errors are being created in the final step of preparing the elevation data. The weird thing is that it only occurs at certain specific elevations and I just couldn't see the pattern... until now. If this bears out, I'll re-process my whole elevation dataset (around 300gb / 6 million tiles IIRC) and perhaps we we'll no longer need error correction.

Here's the existing Apple Pie Hill data with no error correction.

My error correction code improves this, but there are still problems.

Here's the re-processed elevation data. Note the scale on each - with error correction turned off, it shows a minimum elevation of 31 ft, which is clearly an error. With error correction turned on, the minimum is 57 ft. But the "real" minimum elevation is 89 feet, as shown below. This changes the map color, making it more blue.

 

[pinelandpaddler]

And here's @pinelandpaddler 's "mystery canal", which is a good example of the custom shader in the flatlands

https://boydsmaps.com/terra/#39.633...0/0/1000/600/-1000/45/2/2d/shader20/ech/0/z17

View attachment 20016

Loving this! Thank you!

The shading nicely highlights the fact that Rabbit Island used to be the high ground at the interior of an old river meander. Lots of cool fluvial geomorphologic history going on at the Forks.

 

[bobpbx]

Lots of cool fluvial geomorphologic history going on at the Forks.

Stop talking dirty.

 

[Boyd]

If this bears out, I'll re-process my whole elevation dataset (around 300gb / 6 million tiles IIRC) and perhaps we we'll no longer need error correction.

Have started updating the individual components of the dataset but it's a big job that will take a number of days. They look good so far on my development server at home with error correction turned off. Will start updating the public version tomorrow, one section at a time. The first will be the highest resolution maps (1.5 mile map size) for New Jersey and Long Island.

 

[bobpbx]

Have started updating the individual components of the dataset but it's a big job that will take a number of days. They look good so far on my development server at home with error correction turned off. Will start updating the public version tomorrow, one section at a time. The first will be the highest resolution maps (1.5 mile map size) for New Jersey and Long Island.

How much storage do you have?

 

[Boyd]

2 terabytes on the internet server. 8 terabytes of SSD on my computer at home, plus another 25 tb of network and external hard drives, plus 4tb SSD on a media server! And - would you believe it - I am still running out of space.

 

[Boyd]

Will start updating the public version tomorrow, one section at a time.

This update is now in progress. The high resolution (1.5 mile map size) data for the Pines is now live. So if you're using the terrain viewer with these maps in the Pines, you should turn OFF error correction.

 

[Boyd]

After a few false starts, the full elevation data update should finish tonight or tomorrow. If anyone's interested, here's what the problem was. Elevation data is encoded as colors in special maps that are hidden from the user. By reading the red, green and blue (RGB) values at the current map location, a simple calculation provides the elevation. The hard part is creating the special maps and maintaining the color accuracy of each pixel throughout multiple processing steps. This was failing at the final step but I finally found a work-around.

The interesting thing is that the errors only occured at very specific elevations and it took awhile to figure out what the pattern was. You can clearly see it in this "before" screenshot - the errors look just like contour lines!

And they basically are contour lines, spaced every 25.6 meters. The full explanation is too complicated, but note that elevation is encoded in 1/10 meter increments and 256 divided by 10 is 25.6. Here's the re-processed data, without the errors. Due to the size of the map, I can't check every location, but so far it's looking good!

After I finish the update and check everything out, I plan to remove the error correction settings. In the meantime, be sure to turn off error correction when viewing the 1.5 mi and 3 mi maps.

 

[Boyd]

All the map sizes in the terrain viewer have now been updated and should (theoretically) be error free. Here's a view of the Core Pines at the 48-mile scale, for example

https://boydsmaps.com/terra/#39.713...2/0/2000/972/-2000/45/6/2d/shader20/eco/0/z12

Still some more work to do, this same data is also used for the 3d Mid Atlantic LIDAR map in the main app and it also provides the data for the elevation readout. After I finish up the low-resolution data for those purposes, I'll update the terrain viewer code to remove the error correction settings. Will post about that later, but if you use the terrain viewer in the meantime, error correction should always be turned off.

 

[Boyd]

Well, I think everything is finally done! The error correction menu is gone now and the code has been updated to work more efficiently. Here's an interesting spot called Strawberry Hill between Lambertville and Washington Crossing. The bottom of that quarry is 120 feet below sea level and only about 1000 feet from the Delaware River.

https://boydsmaps.com/terra/#40.324...-58/0/1324/789/-1296/45/2/3d/shader20/0/2/z17

Another nice thing about this big update is that it greatly improves the quality of the 3d view in the main boydsmaps app as well. It should also improve the accuracy of the elevation display in the top left corner.

https://boydsmaps.com/#16.00/40.324596/-74.910416/mbx3dmidatl/97.80/58.40

Let me know if you encounter any problems. If things don't work properly, it is most likely a browser cache issue due to all the code and data updates. Clearing the cache should fix that.

Too many hours at the computer today - heading out on a walk for awhile now.