Mediation is voluntary. If you don’t want to speak to the other side then you do not have to and no one will make you. However, the mediation process works best when each side communicates. By not speaking you will be making it harder to achieve a settlement.
It is possible to conduct the entire mediation with each side in a separate room. The mediator will move between each room in what is called shuttle diplomacy. However, it is not the most efficient or meaningful way to communicate, as all messages will go through the mediator. Consider carefully whether this is necessary or whether with support from the mediator and ground rules you can manage to talk in a joint session.
However, there may be reasons why you do not want to speak to the other side which go beyond awkwardness or anger. If you have safety concerns then you need to share these concerns with the mediator as soon as possible.