There’s now a way to change the resolution on you Mac OS X VM running on ESXi.
If you are running Mac OS X Yosemite (or later) on ESXi you may have noticed that the screen resolution is limited to only 1024×768 even though VMware Tools has been installed. Until recently the only workaround for this issue was to add a dedicated graphics card in passthrough mode, which not always worked. Luckily there’s now a tool available that makes it easy to change the resolution to whatever you want. Before using this fix you should make sure that VMware Tools is installed and that at least 16MB video memory has been configured for your VM.
Download the vmware-resolutionSet archive and extract it on your virtual machine. Next open the terminal and navigate to the file location. Enable execution by running the command chmod +x vmware-resolutionSet and simply set the resolution by running sudo ./vmware-resolutionSet <width> <height>. The resolution should change immediately, but it’s recommended to reboot.