esxi 6.0 ntpd server not responding if source port is not 123

We are running an esxi 6.0 host as an ntp server and it is working fine but it does not reply to requests where the source port is not 123.

client-ip:123 -> esxi-host-ip:123 request

esxi-host-ip:123 -> client-ip:123 response

client-ip:55239 -> esxi-host-ip:123 request

<no outgoing reply>

How can we enable replies to requests where the source port is not 123?

Is this an ntp.conf issue, or a firewall issue, or both?

In vSphere Host > Configuration > Software > Security Profile > Firewall > Properties  I can see a firewall rule that allows UDP:123 outgoing connections for NTP client. But that sounds fine to me, and it does not seem to be editable.

esxi 6.0 with ssh and ntpd enabled, otherwise very simple.

Note this was working under esxi 4.0

Thanks in advance