boydsmaps in the post-NJGIN era

[Boyd]

Note: This new thread includes posts that were split from the trouble report thread. To summarize: we no longer offer maps from the NJGIN servers, due to compatibility issues. However, many of their maps are now available directly from boydsmaps. These local versions cover the entire state of NJ but the newer aerials (2000, 2007, 2012, 2015 and 2020) only go up to level 17 where the NJGIN originals went to level 19. The astronomical amount of storage required for level 19 maps makes it impossible to offer that locally. However, if you use the Share > NJPB menu, it will open the same map (if available) in NJPB maps where you can zoom all the way in.
______________________________________________________________

It appears that none of the NJ State (NJGIN) maps are working this morning. Not sure when this problem began, I got an e-mail from another member about it a couple hours ago. Their servers appear to be fine, you can access their maps here with NJPB Maps and I can directly access their servers from home with some other software.

Wasn't so sure at first, but the more I look, it appears they are intentionally blocking boydsmaps access - getting an access control error - this is related to something called Cross Origin Resource Sharing (CORS) which is a way that a site can control whether other sites can link to their data. Interesting that they are blocking boydsmaps and not NJPB Maps. Had a look on the NJGIN website and don't see any policy changes or access restrictions to the use of their data. Maybe this is part of DOGE?

Anyway, sorry for this inconvenience, I don't know whether it's intentional or not. Will do some more trouble-shooting in case I missed something. But I have no relation with NJGIN and not interested in establishing one. If they want to block access to boydsmaps, that's certainly their privilege. If this outage continues, will just remove their maps from the menu. Sorry....

 

[Boyd]

Well, the NJGIN maps are still being blocked. Funny, I just updated the help file a few days ago and pointed out the difference between locally-hosted and remotely-hosted maps, explaining that boydsmaps will be focussing mainly on locally-hosted maps in the future. Did not plan to phase out the NJGIN maps yet, but it appears that decision may have been made for me.

Will give this another day or so, then will remove the NJGIN maps from the menu over the weekend if this is actually a permanent change and not just some glitch. I don't want a bunch of maps on the menu that no longer work. Just like removing the HERE and MapBox maps, boydsmaps will be fine without them and once I change the site code, they won't be coming back.

 

[Boyd]

FWIW, I remembered that Mapbox actually had an example of using an NJGIN map with their API. This used to work fine, but now it's also broken (only shows a basemap, should be the 2015 aerials). So, it seems I'm not the only one being blocked. Weird thing is, they haven't blocked NJPB Maps.

 

[stiltzkin]

FWIW, I remembered that Mapbox actually had an example of using an NJGIN map with their API. This used to work fine, but now it's also broken (only shows a basemap, should be the 2015 aerials). So, it seems I'm not the only one being blocked. Weird thing is, they haven't blocked NJPB Maps.

View attachment 24158

Yes, I was going to say the same thing. I do not think you are being specifically blocked. This looks like a misconfiguration on their web server. Specifically, the required header access-control-allow-origin is not being set, which could happen for a number of different reasons.

Even the viewer on newjersey.maps.arcgis.com shows CORS problems, and obviously this site is allowed. I wouldn't take this as a sign that you need to remove the maps from your site.

 

[Boyd]

Thanks for checking that out, I appreciate it and you could be right. But why doesn't it affect NJPB ? Maybe because NJPB uses the Google API and I don't? Sorry, the decision is final and the NJGIN maps will be removed later today. Even if this is a mistake, it just highlights the problems with maps over which I have no control. Performance is often poor too, compared to my local maps that load very quickly. I'm just not interested in contacting somebody at nj.gov who won't understand what I'm talking about and will bounce me around to somebody else who probably won't. It may be related to arcGIS who hosts the maps and not even NJGIN.

I am sorry - just like HERE and Mapbox, these full-resolution NJGIN aerials are very useful. Too bad, but I have already moved on. It's certainly unfortunate, and if this means boydsmaps no longer meets your needs, I'll understand. But, unlike Mapbox/HERE, there's a solution to this problem - just click Share > View this location on NJPB Maps. That will open the comparable NJGIN map here at NJPB. Right now, it's kind of "dumb" as it will always open the Google map. But I'm revising the code today, so that it will open the same map (if available). For example: if you're viewing the 2000's BW aerials at boydsmaps, it will open the NJ 2002 infrared aerials at NJPB. This will let you browse the maps quickly at boydsmaps, then when you want to zoom in farther, NJPB Maps is only two clicks away.

I'll make another post when the site changes are completed later today (tomorrow at the latest). Am also expanding the 2007 and 2012 aerials to cover all of NJ (they currently only cover the Pines), this should help with the transition. The 2012 expansion is all downloaded, rendering the maps now (a long process). Am still downloading North Jersey files for the 2007 aerials, they will also be online before long. After these are completed, will probably expand the 2020 aerials as well.

But yeah - this will change the nature of boydsmaps - for the better IMO. And the site is more popular than ever, have set some new records for the number of tiles/amount of data downloaded this week. People are coming from throughout the Northeast US region, not just NJ, which makes me think I'm doing something right.

In summary: sorry if this is a disappointment, I can understand that. I was happy enough with the status quo until this happened. But I'm not going to have a long menu of maps that don't work, I don't want to wait until they decide to fix this (assuming it's not intentional) and definitely not getting into a dialog with NJGIN about why I'm "entitled" to use their maps.

 

[stiltzkin]

But why doesn't it affect NJPB ? Maybe because NJPB uses the Google API and I don't?

Because you are setting Sec-Fetch-Mode: cors in the requests you are making to nj.maps.gov.

The NJPB maps site is setting Sec-Fetch-Mode: no-cors, so it works.

Essentially, your site asks for strict CORS checks and NJPB does not.

You can likely resolve this entire issue by changing a single request header in your site code. But it sounds like your mind is made up.

Too bad, but I have already moved on. It's certainly unfortunate, and if this means boydsmaps no longer meets your needs, I'll understand.

Yes, unfortunately it means I won't use the site very much anymore, as it was my primary use case. Thanks for providing these layers up until now, anyway.

 

[Boyd]

Yeah, was thinking about this after my earlier post. With the MapTalks API that I use, there are two modes. If you want to use the 3d mode (like I do) that requires webGL which strictly enforces CORS. I'm fine with that - not going to fundamentally change boydsmaps and remove 3d view because of the way NJGIN has re-configured their server. But the situation is kind of funny....

You're sitting in your living room with the front door unlocked. I knock and you say "go away", which I do. Google knocks and then charges right in, saying "hey... it wasn't locked".

But seriously, was thinking about this well before the current NJGIN problem. When it comes to a "mission", I don't really aspire to be the place to access imagery from NJGIN servers. I'd rather to be known as the place with unique, high performance maps you can't find anywhere else (or at least, not for free with no ads). I was happy to include the NJGIN products for all these years, but I don't want to deal with this kind of issue anymore, it's hard enough just to fix my own mistakes.

My new focus may not align well with everyones needs and that's unfortunate, but this is where I'm heading. I discontinued the HERE and MapBox maps a year ago and site usage has grown significantly since then. Getting a nice mix of users throughout the Northeast - but still a large number from NJ. I think it will continue to grow, but could certainly be wrong. When you aren't selling anything, does it even matter how popular you are? And if everyone goes away, that's not such a bad outcome.... If I shut boydsmaps down, I could buy one of your fancy smokers every year and that would be much cheaper than my annual server lease.

 

[Boyd]

The changes to the site are ready, just want to double-check my code a bit before uploading. I have also extended the 2012 aerials to cover the entire state and the new map tiles are installed. Just finishing up an expanded version of the 2007 aerials with coverage of the whole state too. When that's completed (this evening), I'll activate all the changes. Will make another post at that time.

Also finishing the download of expanded 2020 aerials, which should be finished tomorrow. I'll probably continue, and when completed, boydsmaps will have full NJ coverage from 2007, 2012, 2015, 2019 and 2020 in addition to the larger midatlantic aerials from the 1980's, 1990's and 2000's.

 

[Boyd]

The site has now been updated. Unfortunately, I'd expect this to cause problems due to cached data. Clearing you cache should fix these issues, here's a link that may help.

How to hard refresh your browser Chrome, Firefox, Safari

Hard refreshing your browser clears your cache and forces your browser to download a new copy of the website. This is extremely helpful when you developing a website and need to see new changes.

fabricdigital.co.nz

If you've installed the app on the home screen of your phone, that can be more problematic. It's probably easiest to just delete the icon from your home screen and re-install from your browser. This only takes a few seconds, you will find instructions in the help file.

If you have a custom favorite maps menu, the NJGIN maps will be automatically removed from it. Any old links to NJGIN maps should automatically substitute a locally-hosted map now. This needs more testing, but I think it should work. As mentioned above, you can still access many of the NJGIN maps using the share function. For example, if you're viewing the 2007 aerials, click the share button, then click NJPB Maps. On a phone, you need to press the menu button first, then the share button.

That will open the 2007 aerials in NJPB maps, where you can zoom in farther if needed (seems very slow tonight)

With this update, I also added a feature I've wanted for awhile. On a large computer screen, I like to leave the map menu open so I can "channel surf" through the maps. The problem with that is, if you zoom out and move too far away, you can move outside of the map coverage areas. The solution was to close the menu and open it again, but that was awkward. Now, if you leave the map menu open, it will update automatically when you move out of a region. You can see this at work if you go to this link, open the map menu and leave it open. Now drag the map to the West to somewhere around Lancaster, PA. When you release the mouse, the menu should automatically update to remove all the maps of the Pines. Now drag the map up to New York City. The menu should update again when you stop moving the map.

https://boydsmaps.com/#8.00/39.904998/-74.746425/usopentopo/0.00/0.00

 

[Boyd]

Update: I have 2020 aerials for the full state ready to go, finishing up full 2015 aerials and downloading full 2019 aerials (servers are slow today). As with the others aerials, resolution is 4 feet.

Also, I have the NJGIN Historic Topo for the full state ready to go, they're USGS 62k topo's, but periods are different from my map (mostly later, I think). Time period is shown as 1881 - 1924. Same resolution as the original NJGIN maps.

Also have a few small new features coming and a bug-fix. With the slow servers, this isn't gonna be done tonight and I'd also like to test the new code some more. Will plan to put all these things online sometime tomorrow. In the meantime, this image is from the revised help file that I'll also publish tomorrow. After some thought, this is how the maps are now organized in 5 regions. So, we have: Pines > NJ State > Midatlantic US > Northeast US. My plan for the year is to "promote" maps from their current level to the next level as much as possible this year. Someday, almost everything would cover the entire Northeast region.

This NJGIN stuff has been a big distraction. I was within a week of finishing the 1970's aerials of the Pines but dropped it all. Should be coming soon now. When that's complete, I'll concentrate on finishing the Midatlantic version of my road and hybrid maps - a big project that will take awhile though.

 

[Boyd]

Will plan to put all these things online sometime tomorrow.

So much for that plan! Just finished the 2019 download, it will take 20 hours or more to process. The 2015 aerials are finishing up but won't be done until late tonight. Hope I can put everything online tomorrow, but it depends on whether the 2019 files are finished.

 

[Boyd]

The update is finally online! This is quite major with many code and map changes, so see my earlier post about clearing your browser cache if you have problems. New 2015, 2019 and 2020 aerials with full New Jersey coverage are now online, joining the 2007 and 2012 aerials that were updated earlier. And the New Jersey Historic Topo (1881-1924) from NJGIN is also live.

For software, I did more work on the map menu to make it work better when you just leave it open all the time. Fixed a bug where the currently selected map could become un-highlighted, also made it update faster/more reliably as you move the map.

Another tweak involves switching between 2d and 3d modes. If you have the map menu open, when you click the compass it will immediately update to show available 3d maps (in blue). Click the compass again to switch back to 2d mode. The 3d maps will vanish from the menu if it's open.

Here's another new feature that I've wanted to include for awhile: the zoom indicator turns red when you zoom past the map's maximum resolution. This is from the updated helpfile

Links to the original NJGIN maps should be translated to the new locally-hosted maps when you click on them. With so many substitutions, I felt that the alerts where you need to click "OK" were too annoying. But I felt it was still important to indicate that the map in the link has been changed. So, now an alert will show on the screen for a few seconds, then automatically go away. It's sort of like a speed-reading exercise: you have 4 seconds for the shorter messages and 6 seconds for the long ones!

Along with the update, I deleted a bunch of remotely-hosted maps that just didn't work at all. These included most of the Pennsylvania aerials, a handful of Vermont maps, Virginia aerials, New York aerials and some West Virginia aerials.

Another minor new feature - you can access the helpfiles for both the mobile and desktop helpfiles here

https://boydsmaps.com/help/

Let me know if you have problems. I think everything is working, but have not tested extensively. Will be doing that soon... but now it's time for dinner.

 

[bobpbx]

I checked out the aerials. Looks good, I like the 2020's. Were they somehow clarified?

 

[Boyd]

No, didn't do anything special. The Southern part is the same as before, the Northern part was downloaded the same way. I am using .jpg files for all of these, the files are much smalller than .png and it's very hard to tell any difference in quality with aerials. I might add some of the other years eventually - 2006, 2010, 2013 and 2017. They are all NAIP "leaf on" aerials and I was never very impressed with the quality.

 

[bobpbx]

I some few places, the 2020's beat the 2007 in seeing outlines of what's there. Old Atsion village is an example.

 

[Boyd]

Will just post this here instead of starting a new thread. The 2006 NJ NAIP aerials are now available. This map has the same coverage and resolution as the original version from NJGIN. The imagery comes from the USDA National Agriculture Imagery Program (NAIP), photographed during the summer to show crops and foliage cover. So, you won't find any secrets hidden under the tree canopy but I like these maps anyway - mainly because they make the forests look so green and inviting. The 2019 aerials are also part of the same series and NJGIN also has versions from 2010, 2013 and 2017. Eventually, I would like to add these as well.

This might cause the usual cache issues, unfortunately. But I created a new FAQ with full instructions for how to clear your cache on your computer with Chrome, Firefox, Edge and Safari. Scroll down to the troubleshooting section at end of the helpfile here to see it:

https://boydsmaps.com/help/#desktop

 

[Boyd]

Looks like 2006 was a pretty dry year...

https://boydsmaps.com/sync/#15/39.818864/-74.544658/nj2006/0/0/nj2007/15/0/0

2006 is on the left and 2007 on the right.