Yes, the cliff is there to get down - not back up.

Those who placed a warp gem from the quest can visit!