Well-worn stone steps lead to the top walls of Richmond Castle.