Harry wasn’t reading the journal just to feel closer to Aven, he also read it when he was about to have a panic attach, when he heard his father’s voice, when he was feeling evil and unworthy.
Aven didn’t just trash their memories, she also destroyed Harry’s way of coping.