I've been out there a few times, but each visit was part of an organized trip for which special permission was granted by the feds. That part of the refuge is not open to the public.
In any case, there used to be a more worn path through the brush from the sandy beach to the upland. Now the easiest way is to walk the saltmarsh along the edge of the brushy corridor. You need to jump a few ditches, but it's nothing hard. The foundation of the old Mathis homestead is still visible, as well as remains of more recent structures at the other side of the island. Some the the biggest patches of prickly pear I've ever seen are on the island as well.
