Throw your pickaxe at a group of 2 or more of the same bricks to remove that group. Remove the bricks before they reach your miner.