Rules of writing frustration point: the ? outside of quotations

Did he say, "We should all go read Pauly's book"?

Is a correctly punctuated sentence.

And it angers me.

So is this:

"Have you read 'By the Gates of the Garden of Eden'?" he asked her.

But it doesn't anger me as much and I don't know why.