Ms Norman took her sons to the pier on 13 April but the day out "turned into a nightmare" after the payment was processed this week.
Two payments of £2,104.13 have been charged to her account.
"I looked at my account this morning and nearly cried," she said. "It was very stressful and will be until I get my money back."
Anne Ackord, chief executive of the Brighton Pier Group, said the company "would like to apologise profusely to customers who have been affected by this error".
Worldpay said due to an "isolated error some customers were debited an incorrect amount on 22 June at Brighton Palace Pier".
It said it was working to refund the money as a "matter of urgency" and would cover the costs of any bank charges incurred.
"Depending on the cardholders' bank, it may take up to five days for the correction to hit their account," the company said.
"We apologise for the inconvenience this may have caused."