What is FML? I recently spent a little time starting to put together my own USGS topo for the Northeast US, but it's a huge project and I got discouraged. If you just want New Jersey, NJGIN has that. We discussed this a few years ago, and IIRC your objections were that NJGIN's servers were slow and they don't switch to the 100k topo when you zoom out.
I think NJGIN servers work very well now. And there should be a way to combine their 100k and 24k WMS layers to accomplish what you want. In Leaflet it's very simple to create a layer group and configure the zoom levels for each map. Isn't this possible in the Google API? If not, then this open source package should be able to do it: https://mapproxy.org
If all else fails, I can put a tileset together for South Jersey that you can host locally at NJPB.
I'm not much of a fan of the arcGIS topo, there are a lot of JPEG artifacts and it isn't even a "real" USGS topo, it's the old National Geographic version with hillshading that muddies things up and adds nothing in flat areas like the pines. The NJGIN version is a little odd because they converted the maps to a 12-color palette in an attempt to standardize the colors. I don't think they're too bad though, you can compare here
Mine also goes up to level 16 while both NJGIN and arcGIS are only level 15. But it's probably a bigger tileset than you would want to host at NJPB. I could give you a level 15 version however, that would be around 1GB for South Jersey.