Electrican let me down,do I still have to pay call out fee?

This morning I booked a electrican to fix a ceiling light for me.I booked him for 4pm,I booked this time because I had to leave home at 6pm so I thought Id have plenty of time. They didnt arrive at 4pm but at 5pm they rang me to say they would be late, he said it was due to traffic and he was still around 15 miles away.I told him I have to leave for at 6pm.He said he would call tomorrow instead but I told him that wouldnt be good for me so I wished to cancel. He told me to ring their offices which I did,however I was told in no uncertain terms that I would still have to pay the £50 call out fee because I have cancelled it..I told him I wouldnt and he was very angry and quite intimidating to me,in the end I ended the call. He has now rang me 15 times but I havent answered and I feel very worried by the calls because I live on my own and am a oap. Am I best to just pay it if he comes to my door???, He didnt take any bank details when I rang him this morning.