If you mess up on the dancing part, do you need to restart the entire quest? -- No, you just have to redo the dancing sequence.

The Haunted House in West Freeport is in the West Freeport Bank

What if the ghost wont start the dance again after you fail?

Exit and re-enter the Haunted House. The dance sequence is buggy, but you do not need to delete the quest and start over. Just go back in, everything inside will be reset, so you'll have to redo everything.