My opinion is that there were trouble areas in all the books, though the first one was by far the best.
The first had the magic and as Harry evolved and slowly grown up book by book, this magic is slowly vanished and everything turned to grim and dark. While this transition is part of the human evolution and it's highly realistic, in some of the novels this change was too drastic. Order of the Phoenix is the best example (Yet, it's one of my other favorites.).
As bcomet written Harry was angry, frustrated, self-sorry, jealous, up-in-peoples' faces, misunderstood, alone, etc... The signs of a teenager and it were a realistic presentation. In the first Harry Potter, Harry was a boy. The story had a magic, because it's revealed a world what about many readers dreamt when they was a child. And Harry was the one who attached us to that world. But in Order of the Phoenix this magic was replaced with cruel realism. It was good and I also liked Order of the Phoenix, but if I compare it with the first HP novel, it wasn't balanced properly and there wasn't enough transition between the boy HP and the teenager HP.
Harry Potter #1 - Gave our dreams back, our lost childhood
Harry Potter #4 - Faced us with our own rebelling teenager nature
Both was a mirror in front of the reader, but the mirror image shown a different reflection in each novel.