The Arctic Baffin Bay between Canada and Greenland is sea-ice covered during the majority of the year, restricting primary production to the summer months. Sediments receive low amounts of mostly terrestrial- and less marine-derived organic matter. To study microbial communities constrained by physicochemical conditions changing with distance from land and ocean depth, we applied high-throughput 16S rRNA gene sequencing and compared sequence diversity with biogeochemical parameters in 40 different sediment samples. Samples originated from seven cores down to 470 cm below seafloor along a shelf-to-basin transect. Bacterial diversity decreased faster with depth in basin than in shelf sediments, suggesting higher organic matter content sustained diversity into greater depths. All samples were dominated by Betaproteobacteria (mostly order Burkholderiales), which were especially abundant in basin sediments with low organic carbon and high Mn and Fe pore-water concentrations. Strong statistical correlations between concentrations of reduced Mn and/or Fe and the relative abundances of Betaproteobacteria suggest that this group is involved in metal reduction in Baffin Bay sediments. Dehalococcoidia (phylum Chloroflexi) were abundant in all samples, especially in shelf sediments with high organic content. This study indicates that Mn and/or Fe play important roles structuring microbial communities in Arctic sediments poor in organic matter.