Install Synology DSM in a ESXi 5.x virtual machine

Back in 2011, I ordered myself a Synology Disk Station DS410 4-bay NAS in order to have a centralized and private storage solution for my files. I have to admit, the user interface and support from Synology superseded everything else at that time. Even now, four years later, I get free updates that provide many new features like running your own “Dropbox” clone. However, while the software continues to evolve, the hardware in the device sadly does not.

Introduction

A few years later I began to look at the opportunities to build a new server to be able to use virtualization without having to run this on my laptop. I jumped between using Ubuntu Server, Open Media Vault, FreeNAS and Windows Server, but was never particularly happy with the result.

After some back and forth, I found a forum that specializes in virtualizing Synology DSM, the operating system Synology uses. Some weeks later with careful testing, I was extremely pleased with the solution. At that time I used Hyper-V as my operating system, which turned out to not be as flexible as ESXi when it came to virtualizing Synology DSM. Did I mention that ESXi is free and easy to install as well? 🙂

Overview of steps:

In this guide I assume that you already have ESXi running. I’m not going to cover that part here, but let me know in the comment section if you need help with that.

While I use the video above and the written version as a foundation, I’ve made a few changes and additions on my install which I want to emphasize. Follow the steps 1-10 in order to define the basic VM configuration, but don’t create a new virtual disk and skip to step 14.

In case the guide above is changed, here is a summary of the steps you should do:

  1. Give the VM any name you want.
  2. Select where you want to store it, use SSD storage if available.
  3. Choose Virtual Machine Version 8.
  4. Choose Linux and then Other 2.6.x Linux (64-bits).
  5. Configure CPU/RAM according to what you have. I used 2 cores per virtual socket and 8GB of RAM, but 4GB should be sufficient for most tasks.
  6. Set the number of NICs to 1 and use VMXNET 3 as the adapter
  7. Choose LSI Logic Parallel as controller
  8. Choose Do not create disk
  9. Check “Edit the virtual machine…” and then Finish
  10. While not needed, I suggest to remove the CD and Floppy Drive

The reason why I do not recommend to create virtual drives or use RDM for storage is that S.M.A.R.T and temperature monitoring will not work. It’s not a requirement in any way, but I highly suggest to get yourself a dedicated SATA controller such as the LSI SAS 9201-16i I got.

1. Creating the boot image

In order to create the boot image for DSM, start WinImage. Navigate to File->Open and select the img file you downloaded earlier. Find and right click on syslinux.cfg and select “Extract”. Open syslinux.cfg with notepad and add “rmmod=ata_piix” behind every instance of sn=XXXXXXXXXX entry so it looks like this:

Save the modified syslinux.cfg file and “Inject” it back to the img file using WinImage (overwrite the file when asked).  Save the image and exit WinImage. This will ensure that all drives are loaded and accessible. If not added you’ll have the virtual IDE controller of VM taking your first few drive slots.

2. Convert the boot image to VMDK

Start Starwind V2V converter and choose the img file you want to convert and follow the on-screen instructions. Make sure to choose “VMware pre-allocated image and IDE as type. This process will create two vmdk files which you will need to upload to your ESXi data store. I suggest you upload the files in the same directory as your VM resides in.

3. Adding boot and hard drives to your VM

Now you’ll need to edit your virtual machine setting. Right click on your VM and select “Edit settings”.  Click “Add”, select “Hard Disk”->”Use existing virtual disk” and select the vmdk file you just uploaded. Be sure to add it as a IDE hard disk and don’t change any other settings. Now your boot disk is set and done! Congratulations!

If you are like me and brought a dedicated LSI (or similar) SATA controller, I highly suggest to dedicate that pci-card to the VM. In order to do so, go to Configuration->Advanced Settings and click “Edit”. Check your SATA controller, click OK and reboot your server when prompted. Once you are logged into vSphere again, go to your VM and edit it’s settings.  Now you should be able to choose “Add”, “PCI Devices” and select your SATA controller. All drives connected to it will automatically be available for Synology DSM (max 12 drives).

4. Start and install

Now it’s time to start your virtual machine for the first time. The rest of the install is no different then installing an ordinary Synology Disk Station. Just download the Synology Assistant and follow it’s instructions to download and install DSM.

5. Tweaks

I recommend to change the MAC address of your NIC to anything else but Automatic. I’ve notices that some stuff like e-mail notification might not work until you do. For example: 00:11:32:XX:XX:XX. The first numbers specify that it’s a Synology NIC. 🙂

You’ll notice that EXSI complains about VMware Tools missing. Don’t worry, you don’t need it, but if you really want to install it you can do so by manually installing this packing from the Package Installer in DSM.

Share this:

13 comments, add yours.

John

Hi, how do you get ESXi running on the server? What file do you need? I already have the client installed on my windows laptop.

Benjamin Krause

Benjamin Krause

Author

The only thing you need is the ESXi iso file from here.
When you register on their site you are provided with the ESXi ISO image and a free, personal license.

Once downloaded use something like Rufus to create a boot-able USB drive.
On my server I installed ESXi on a USB drive that’s always connected to the server. There’s no point in wasting a hard drive on ESXi itself.

The installation itself is really straight forward and you shouldn’t have any problems at all.
Hope that helps you along the way. If not just let me know and I’ll make a more detailed post about configuring ESXi. 🙂

John

Thank you Benjamin! You have been very helpful.
I appreciate all of your tutorials. 🙂

Benjamin Krause

Benjamin Krause

Author

Awesome, thank you. 🙂

Rashmi Patel

Is there any benefit to install Windows Server 2012 and then put Xpenology using ESXi? I’m just confused regarding installing this on ESXi.

Can I use Xpenology as one VM, install Unix as another VM? Also can I use my existing Synology drives, as they have all my files on them?

Thanks, great article

Benjamin Krause

Benjamin Krause

Author

I’m not sure if I understand the whole question. You can run Windows Server 2012 and Hyper-V or ESXi in order to run virtual machines. Since ESXi is free and supports a wider variety of operating systems (I run OSX, Android and a few other), I highly recommend that over Windows Server 2012. The ESXi install is pretty easy, but I’ll cover that in an upcoming post.

Yes, you can certainly setup multiple VMs. Currently I’m running XPenology as fileserver, multiple ubuntu servers for Plex, GitLab, Apache and Scripts and a OS X client for development. I have not tried to move from a physical Synology to XPenology. It should work, but since I haven’t tried that myself I can’t recommend doing that.

PeterT

Hello and thank you for your time of this great post.
I really hope you can help me out here? I am trying for a couple of days now to do a clean install of XPEnoboot with all kind of images but everytime i boot up from the usb stick or the vmware image it is asking me for a username and password and nothing works. Not even the edit boot line with resetuser=admin

Benjamin Krause

Benjamin Krause

Author

Thanks for the feedback. If you do a clean install I would recommend to follow my latest guide here: https://idmedia.no/projects/xpenology/installing-or-upgrading-to-xpenology-5-2-or-later/

You should not need any username or password. Once you’ve prepared the boot image according to my guide you just have to boot it and select “Install/Upgrade…” on the boot selection screen. Then you wait for a couple of minutes and you should be able to install Synology DSM using the Synology Assistant or find.synology.com.

I’m actually planning on making a video on how I configured my setup this weekend. Just follow me on any of the social links in the footer and you’ll be notified once it’s out. Hopefully that will help you solve this issue. 🙂

Joakim

Hi.
Great tutorial! I decided to throw my old XPenology away, dsm 5.0, and go for the 5.2 with your tutorial. the issue I now have is that every thing goes according to plan except that when synology assistant finds the booted VM and upgrades, it says it will erase disk 1 and 3, (I guess) this causes the VM to show “Operatingsystem missing” when it reboots. I use now ESXI 6 and clean install from your guide. I don´t have the PCI-card that you use so I just adds the 20Gb-drive as SCSI (have also tried IDE) and also checked the boot-order in the VM. No luck. Do you have any tip to get past the issue with formatting both “1 and 3”?

Benjamin Krause

Benjamin Krause

Author

Thanks! Sounds like you didn’t add “rmmod=ata_piix” to the boot image. If you have, I would verify that only the boot image is added as a IDE disk and all other drives are SATA/ISCSI. The parameter should remove all IDE drives from being detected by DSM so it’s not formatted during install.

If you haven’t seen it already, I recommend to watch this video tutorial:
https://www.youtube.com/watch?v=BPYadX_pjEg&list=PLAZYilcZFnWzWsT8FhnF3eQ0ThShYcdCT

AA

Hi Ben, can you share your xpenology build? I am currently a synology owner and would like to build and migrate over to a xpenology setup.

Benjamin Krause

Benjamin Krause

Author

Hi, just follow my detailed guide on how to install XPenology 5.2 and you’ll have the same setup as I have. There’s a video too if you prefer that. 🙂

Santiago Draco

Hi,

I’m having trouble getting the LSI 9201-16i (same card you have) working as passthrough for my Xpenology 5.2 VM on ESXi 6.x. Once I add the card in as a passthrough device in ESXi and reboot, then into the VM , the VM will power up and then crash before Xpenology boots. I’m missing some config steps I think. Any advice?

One Trackback

  1. […] the beginning of this year I wrote a small guide on how to “Install Synology DSM in a ESXi 5.x virtual machine“. Now that XPEnology has been updated to support version 5.2 of DSM, I though it was a good […]

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.