Emulating Systems in Qemu

I run Plan9 and NetBSD under emulation. I’m interested in the Qemu emulator because occasionally I want to bring up a system quickly to test something – usually for an OS I can’t easily use AWS for. I’ve scripted a lot of the setup and day-to-day running making it easy to setup VMs using the several operating systems’ stock installers. Please see my scripts at Github. Use at your own risk. If something breaks, you are welcome to keep the pieces. Please also see this document from Qemu.

(Note added 2/2/2021 & updated 3/3/2021: There is a bug/niggle with Qemu 5.2 on Big Sur. Under any guest OS I get short reads when ftp’ing. There is a section at the end explaining the bug. The libslirp folks have put a workaround in and so hopefully this issue will be resolved in the next Qemu release.)

./use-qemu-vm.sh [-i] [-c] [-n] [-d] [-t TargetDir] [OS [arch [ver]]]
  -i run installer ISO (i.e. setup)
  -c use -display curses
  -n use -nographic (overrides -c)  
  -d more output  
     use -t to specify an alternative target directory for files
 OS can be NetBSD, OpenBSD, FreeBSD, Plan9, Debian or Solaris
  • With -i the script will download the relevant installation ISO, make a 10G filesystem image and boot Qemu into the installer. The default OS is NetBSD/amd64.
  • If the filesystem image needs to be made, the script assumes you want to -i to install the OS.
  • In most cases, when you have finished the installer, it is best to exit to a shell and issue “shutdown -p now” or “halt” to cleanly shutdown the VM. You can exit Qemu and use use-qemu-vm.sh again to run the VM (without -i).
  • The -c and -n options ask Qemu for a text only console
  • The script needs Qemu and curl to be installed.
  • The script works with recent versions of the guest OSes:
  • NetBSD/i386, NetBSD/amd64, NetBSD/sparc, NetBSD/sparc64, NetBSD/macppc and NetBSD/hppa.
  • OpenBSD/i386, OpenBSD/amd64, OpenBSD/sparc64 and OpenBSD/hppa.
  • FreeBSD/i386, FreeBSD/amd64 and FreeBSD/sparc64.
  • Debian/amd64
  • Plan9/amd64
  • Minix v3/amd64
  • Solaris 10/i386 (you need to download the ISOs yourself)
  • Solaris 2.6, 7, 8 and 9/sparc (you need to get the ISOs)
  • There are seperate scripts to get a NetBSD/arm64 VM running

What the setup script does

We will describe what happens for NetBSD/i386. The others are very similar but with quirks to help the emulator boot. (For example macppc needs help to boot the right file from the CD.)

./use-qemu-vm.sh -i NetBSD i386      # fetch and setup i386 Qemu VM

The setup script does the following:

  • Makes a directory in your home directory to build the VM files and download the ISO
  • Fetches the NetBSD installation CD from NetBSD.org. (this needs curl installed)
  • Creates a raw image file for the emulator filesystem:
qemu-img create -f raw netbsd-disk.img 10G
  • Boots the emulator with the ISO image mounted as a CD:
qemu-system-i386 -m 256M -hda netbsd-disk.img -cdrom \
              NetBSD-9.1-i386.iso -display curses -boot d \
              -net nic -net user
  • This will boot the NetBSD installer. You can then install the VM how you see fit.
  • When you have finished installing NetBSD, it will be easier to exit the installer and issue:
shutdown -p now
  • This will power down the VM and exit the emulator.
./use-qemu-vm.sh NetBSD i386        # run the i386 VM

The run command does the following:

  • Boots the VM without the ISO mounted. On the stock Qemu built from scratch “-display curses” didn’t seem to work and you can leave this off if you want a more traditional PC display. “-nographic” seems to work more generally.
qemu-system-i386 -m 256M -hda netbsd-disk.img \
              -display curses -boot d -net nic -net user

NetBSD

NetBSD/i386, amd64, sparc64, sparc

If you forgot to setup networking in the installation, you can fix this after installation. Add the following to the /etc/rc.conf file:

dhcpcd=YES

(Note that I have not been able to get sparc or sparc64 to work on Qemu 5.1 – they boot fine on 5.2.)

NetBSD/macppc

The use-qemu-vm.sh script will get as far as booting an PowerPC emulator into the NetBSD installer. The setup script produces a file containing the name of the ISO. When booting normally, Qemu will mount the hard disc image and use the CD to boot a kernel. When you boot the VM, it won’t find the kernel itself:

  • ./use-qemu-vm.sh NetBSD macppc
  • when the Boot: prompt appears, issue: netbsd.macppc -a
  • The -a tells NetBSD to ask for the root filesystem. Choose wd0a and you will be using your image.

This will get you to a working system but with the Installation Kernel. On this kernel, the network interface does not support enough features to be able to DHCP. You can set a static IP though. This can be done in the configuration or after boot. Here are the files that need to be changed:

At install time you can take the opportunity to reserve a small slide for HFS. Later on you can format this slice with HFSTools, add the boot programme and kernel. The Open Firmware should be able to boot from that. I vaguely remember a discussion some time ago about including hfstools in the installation media. I suspect it isn’t because most people installing will be doing it on hardware and will have MacOS around to setup the disc.

The other alternative is to make a CD-ROM image containing a kernel compiled with wd0a explicitly as root and the boot loader file. The limitation here is that Open Firmware can only load from an MacOS HFS filesystem or an ISO/HFS hybrid CD. To make this convenient, I have an ISO containing a kernel hard-coded to use wd0 as the root device with kernels for 9.0 and 9.1. This ISO doesn’t boot, but it’s enough to get a working system. You can use the ISO as follows and it will reliably boot the VM to the login prompt.

qemu-system-ppc -m 1G -nographic -cdrom NetBSD-Boot-macppc-9.iso \
                -net user,ipv6=no -net nic -boot d \
                -prom-env boot-device=cd:,\ofwboot.xcf \
                -prom-env boot-file=netbsd9.wd0 netbsd-disk-macppc.img

Replace netbsd9.wd0 with netbsd91.wd0 for a 9.1 kernel.

I have trouble using reboot on this VM – to the point that I find it best to shutdown and restart the VM. I’m not sure what the problem is here but the boot loader cannot finish loading the kernel and it sits in a loop.

NetBSD/arm64

This guide explains how to start NetBSD/arm64 on the Qemu emulator. I’ve scripted this and included it in the files on Github that I mentioned above. Use the setup-arm64.sh script – it will download the various bits and pieces into a staging directory. It will resize the root filesystem image. The run-arm64.sh script will run the VM.

However here is the procedure in full:

qemu-img resize arm64.img 10g
  • Use the following to boot the Qemu emulator. Put it in a shell script – you’ll need it every time you start it:
qemu-system-aarch64 -M virt -cpu cortex-a53 -smp 4 -m 4g \       
      -drive if=none,file=arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 \
      -netdev type=user,id=net0 \ 
      -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55 \       
      -bios QEMU_EFI.fd -nographic
  • Shutdown the image from within with “shutdown -p now”. You can exit Qemu with CTRL-A X but note that this terminates the VM. Shut it down first!

OpenBSD & FreeBSD

{Open,Free}BSD/{i386, sparc64, amd64}

FreeBSD and OpenBSD can be dealt with in the same way for i386, sparc64 and amd64 as NetBSD. For example to setup OpenBSD/i386:

./setup-qemu-vm.sh OpenBSD i386

#.... install process
# at the end use shutdown -p now or halt
./run-qemu-vm.sh OpenBSD i386

On FreeBSD/sparc64 12.2 I’ve seen two issues: 1. Inability to boot to the installer without -nographic; 2. segmentation faults of sshd and sendmail during the boot. -nographic is set for default with my scripts.

OpenBSD/macppc

Unfortunately I have been unable to get OpenBSD/macppc to boot properly. Version 6.8 panics straightaway on default ppc emulator setup. Using -machine mac99 gets further but hangs during the probe of adb0. Using via=pmu gets further but panics with the USB bus is probed. Using via=pmu-adb gets further again but panics for 6.8 on the USB bus probe and hangs on earlier versions. I’ve tried 6.4, 6.6 and 6.7. No joy. I’m sure this could be fixed.

OpenBSD/macppc 6.8

FreeBSD/powerppc

More stories from the failure department I’m afraid. FreeBSD will not boot on the PowerPC emulator. The vgapci driver does not like the underlying hardware. Again, I’m sure this could be fixed with a quirk added to the FreeBSD kernel, or a quirk added to Qemu somewhere. I’ve tried FreeBSD 11.4 and 12.2.

Plan 9/amd64

The installation of Plan9 is fairly smooth if you follow it to the letter. On a normal boot you need to confirm the root filesystem and answer “glenda” to the second question. Glenda is the equivalent of Unix’s root account in Plan 9. No doubt other architectures work, but I haven’t tried them.

The network bug

After installing NetBSD/amd64 on a Qemu VM, I downloaded pkgsrc (The NetBSD packages collection). My rationale for all this emulation work is to fix some bugs in pkgsrc. The first one I looked at was a bug reported in the bind package. I could not get pkgsrc to accept the downloaded source code. Also I had trouble with downloading Perl. It was very strange. The last byte is missing from the downloaded file. I could replicate this every time.

I tried NetBSD/i386, NetBSD/sparc and NetBSD/arm64 as guest OSes. They were the same. Must be a NetBSD bug right? No. As guest OSes, OpenBSD/i386 exhibits the same behaviour, as does FreeBSD/amd64.

I used AWS to bring up a FreeBSD instance and a Debian instance. In both, I used the scripts above to bring up an OpenBSD/i386 guest Qemu instance. On FreeBSD I could only get OpenBSD into single user mode, but I was able to successfully download files. On Debian, similarly I did not see any problems in the guest OS.

I worked on this with an old friend adamc – we spent 2 hours looking at it on Macs.

There’s a FreeBSD PR raised for a similar scenario: Qemu on Mac, running FreeBSD as a Host OS. This was closed with “fixed in FreeBSD 12.1”, but I don’t think the bug is in the BSD stack. Our current thinking is the bug is either in Qemu or the Mac OS TCP/IP stack. Here is roughly what happens:

  1. Get a large file using the FTP client – it can be over FTP or HTTP. (SSH seems to be fine)
  2. The last packet in the FTP/HTTP conversation has its urgent flag set and the urgent pointer is to the last byte.
  3. This is a possible cause why the last byte is being discarded. It’s unlikely that the remote site is setting the urgent flag but the networking on Qemu is tunnelled from the host OS. So our belief is that Qemu or Mac OS is incorrectly setting the flag.

Here are some illustrations. Firstly the short read – the FTP client doesn’t complete the progress bar but fails silently.

The incorrectly downloaded bind tarball is missing the last byte (shown by hexdump as 00 for some reason):

Here is a tcpdump of the last packet – you can see that the last byte 5a is present:

And here is the top of the packet with the flags – the urgent flag is set and the urgent pointer is 640.

Here is a picture from an AWS instance running FreeBSD as the host OS and OpenBSD as the guest OS under Qemu. The downloads work fine:

See also: