I think it was inadvisable to speak to her on the phone as she is just manipulating you and stringing you along. Frankly xxxx "give it time". There is a court order in place that says she spends x week with you in the holidays isn't there? She hasn't breached that, because you returned the child before the week was up. So she's manipulating both you and the child to ensure she's not in breach. If you keep that in mind, that she is keeping you powerless, it might help with keeping focused on following the court order and seeking advice - either on here or from a solicitor, if something happens when it's on your time. Presumably that's the summer holiday ordered dates done now is it? When is the next court ordered holiday period with you?
I believe, as the current order stands - there are no term time overnights ordered are there? But there is a holiday period now and then (which of course would include overnights). Is that right? The order isn't really workable if it doesn't include overnights in the wording, once they have commenced. It's parenting time, not a visit to an acquaintance.
I think hearing her talk on the phone may make you slip into behaving like a nice normal person - and it also runs the risk of being accused of being hostile or aggressive if you object to anything. As Peanut says - you need to keep everything in writing from now on. If she phones - don't answer - respond with a text. It's flogging a dead horse trying to make arrangements with your ex.