You might have just missed the tiny compartment where the larvae was. The galls on oak leaves have just one small larvae in the center, while the more woody galls on the twigs, caused by a different species, produce numerous individual insects. You are right in that galls are usually not harmful...