Is it pine bark, shredded wood or needles?
Depending on what you've got in those beds, it might be okay to leave it anyway. The mildew or mold that's on it is defferent than the kinds that would get on
plants, it's just a sign that the stuff is breaking down. It's composting where it sits, and that could be a good thing or bad. Once it breaks down it adds nutrients. While it's breaking down it may steal a little nitrogen from the soil. It could also change the soil ph, making it somewhat more acidic...that's good for lots of things, but if it becomes a problem, a little lime or woodash will correct the ph.