Galaxies Struggle to Grow in Crowded Environments