First of all ask yourself why you want to stop them. Unless there is genuinely a concern for your children's welfare then do you really want to deprive the children of a holiday? It will also inevitably cause bad feeling between you and your ex partner and this will not be in the best interests of the children. You also need to consider that there will be times when you will wish to ask them if you can take the children on holiday and you will want them to say yes.
If however you do have a serious concern then first of all you need to check whether your ex partner has a Residence Order in favour of the children. If they do then they are entitled to take the children abroad without your permission for up to one month.
If there is no Residence Order then you need to establish whether you have Parental Responsibility(PR) in relation to the children. Parental Responsibility means the same rights and duties towards the children as their mother automatically has. If you are the children's mother then you automatically have Parental Responsibility.
If you are the father then you will automatically have PR if you were married to the children's mother or if the children were born after 1st December 2003, you jointly registered the birth with the mother and your name appears on the birth certificate. All three of these conditions need to apply.
You may also have gained Parental Responsibility via a Parental Responsibility Agreement (which you will both have signed and which has been registered at the Principle Registry in London) or via a Parental Responsibility Order made by the Court.
If none of the above apply then you do not have PR and your ex partner does not need to have your permission for the children to be taken abroad. Your only option then is to apply for a Parental Responsibility Order and then a Prohibited Steps Order (as explained below).
If you do have PR then he/she does need your permission to do so.
If you refuse permission and he/she tells you that they intend to take the children anyway then your only option is to apply to the Court for a Prohibited Steps Order preventing him/her from taking the children. You may need to make this application on short notice and you will have to convince a Judge that it is not in the children's best interests for them to be taken. This is not a simple process and there will be legal costs incurred so you must be absolutely certain that this is a step you wish to take.