When this happened to me I was guided by the order. It stated clearly that travel to the handover location was my responsibility. As a result I travelled to ex's house twice a week for a month and knocked. When there was no answer, I left one of the packages. Police told me to bring a witness and have a copy of the order on me. I had a reference number to the call advising me it would not be harassment if I continue attending handover location as per the order. They did warn me that police may attend if she called them.
It worked for me. My daughter still speaks of those days. She knows us not seeing eachother was not for want of effort on my part. My ex was told to follow the order at our next hearing.
Tbh, I think you need luck as well as judgment here. A situation like this has so many variables.
Your Social Worker might be overstepping the mark. But you only have this second hand from a person who does not deserve your trust. It is not the SW's responsibility to follow the order anyway. I would leave the Social Worker out of it.
I remember finding this page helpful:
This page explains the law on spending time with a child when a relationship breaks down, including the different types of contact such as supervised or indirect contact and the family court orders for contact.
childlawadvice.org.uk
Broadly, your time with the child should only be stopped in case of emergency or significant risk to the child. Other members are better at the BIFF stuff than me. I think your ex should be informed that she does not have grounds to withhold your daughter and she is obligated to comply with the court order.
Whatever about details of how to handle each bump in the road. The answer is making an application to court ASAP. Could you give yourself a firm deadline of Friday?