Using PR49-35_NCD5500 in 3.3V only system

I wish to use the PR49-35_NCD5500 Ethernet to I2C module to talk to devices at either 5V or 3.3V, with the voltage selected/provided externally before power up. I have the 5V or the 3.3V available.

Looking at the PCB Mechanical drawing, it appears that most of the PR49-35 board is for generation of 5V, with just a bit of it used to interface to the NCD5500 module.

The NCD5500 module already works at 3.3V.

Can I simply feed the 5V connector for the PR49-35 with my own 5V or 3.3V, without using the 12V-5V converter, and have it compatible with 3.3V or 5V devices?