Under the agreement with the Authors Guild and the Association of American Publishers, Google would continue to digitise books and sell access online.
In return, the company would pay $125m (£76.9m) in royalties every year to the copyright owners of the books being scanned.
However, copyright concerns persisted, as the ownership of many of the works being scanned by Google could not be established, meaning many would be unable to claim the royalty payment.
"The [amended settlement agreement] would give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission, while releasing claims well beyond those presented in the case," said judge Denny Chin.
The US Department of Justice has approved the ruling and said it was the "right result."
It has been critical of Google's deal, saying it would give Google exclusive rights to profit from "orphan works", where the rights holders are unknown or cannot be found.
Gina Talamon, a Justice Department spokeswoman, said the agreement "created concerns regarding antitrust, class certification and copyright issues."
The agreement is also separately being investigated by the US Department of Justice on competition and copyright grounds.