I noticed holes in my leaves also with no bugs on them. I have no clue what was eating it and still do not. Mine r not as damaged as yours r. I dont use any kind of insecticide as i try to go organic or safe eating. Last year i had lots of holey leaves but still got a good crop of broccoli and whatever ate the leaves did not eat the broccoli.
I plan on using Neem oil if i do have a problem as it is the safest so far. Next year I will plant differnt kinds of
flowers that help keep bugs off the plants. I had a problem with the bugs that go on pickles last year and read how u plant radishes with them so this year i planted radishes right off and no bugs. %:<)