A neutral Ph is OK. It'll fall to slightly acid over time with the addition of manures and organic fertilisers, so i wouldn't be too concerned about it. Most vegies like a Ph just slightly acid but neutral will work well for what i'm about to suggest.
As suggested, your soil is low in organic matter. This means it will pack down when wet and there are no spaces for air. Plant roots need air as well as moisture. One way of getting organic matter into your soil, as well as digging thru the composts and manures already suggested, is to grow a green manure crop. Your Ph is about perfect for this. The good thing about this is you can do it in the cooler months, when your main crops have finished. So that means you can start late and still be OK.
Basically, a green manure crop is a crop of quick
growing plants which you let grow, intil they are about to
flower and then dig them back into the soil. Sounds a bit nuts but it works.
Ideally, you use legumes. These are plants like chic peas, vetches, tick beans, etc. You can buy packets of these really cheap in the supermarket. They take nitrogen from the air and with the aid of some soil critters, deposit it on their roots. So free nitrogen fertiliser is an added bonus..
Other non-legumes though, grow quickly and thickly and when dug back into the soil, create large amounts of organic matter. These are plants like rye, barley, oats and other grains. The point is you don't let them
seed or they'll start to remove nutrient from the soil, as well as the fact that the
seeds will germinate again.
So plant an area with a mixture of say barley or oats and wooly pod vetch. Whatever you can get your hands on. When the crop is just starting to
flower, smash it all down and dig it into the soil, then bury it under mulch. Next season, with the added composts and manures, your soil will be well on the way to becoming productive. I sometimes use birdseed which has become full of weevils, or other excesses of
seed like beans and peas past their use-by date, just to avoid having barren soil. Exposed, lifeless soil is dying soil.
You sow them really thickly, so if germination rates aren't great it's not a problem.
It keeps the soil alive thru the cooler months too. Maybe give it a try in a section of your garden.