Exploring QEMU and QNX on the Raspberry Pi 4: Using the Device Tree, Serial Ports and QEMU.

not a raspberry

When I discovered that qemu has added support of the raspberry pi 4 (v9.0) I was really happy, but after testing I was really disapointed. It seems like others had similar exerience as me. https://gitlab.com/qemu-project/qemu/-/issues/1208

I did some testing in this repo, but soon concluded that the simulation was not enough.

GitHub - Ebiroll/emulate-raspberry-in-qemu: Notes and scripts to start an emulated raspberry-pi in…

The Raspberry Pi 4 is a good example to learn about the kernel, device threees and QEMU. Even if you do not have access to QNX, you can still learn from this article.

Blackberry have opened QNX 8.0 for non-commercial use, https://blackberry.qnx.com/en/products/qnx-everywhere so if you want to get an image you can do so here.

I decided to merge the latest patches to my own qemu fork, here. https://github.com/Ebiroll/qemu-stm32

In this fork I also tried to include some patches that are not part of qemu yet. https://patchew.org/QEMU/20240226000259.2752893-1-sergey.kambalin@auriga.com/

The SOC on rpi4 is BCM2711, but in qemu it is called BCM2838. Something to remember when looking through the sources.

The Power of Raspberry Pi 4

The Raspberry Pi 4 Model B is an impressive upgrade from its predecessors, featuring a quad-core Cortex-A72 CPU, up to 8GB of RAM, and multiple connectivity options such as USB 3.0, Ethernet, and GPIO pins for interfacing with external devices. With its powerful hardware and widespread availability, the Raspberry Pi 4 has become a staple for developers and hobbyists alike.

• Raspberry Pi 4 Model B Specifications
• Raspberry Pi 4 Reduced Schematics
• https://datasheets.raspberrypi.com/bcm2711/bcm2711-peripherals.pdf

Unit BCM2838

Information about Stepping C0 and 8GB raspberry.

Raspberry Pi 4 model Bs arriving with newer ‘C0’ stepping | Jeff Geerling

Emulating Raspberry Pi 4 with QEMU

QEMU allows developers to emulate the Raspberry Pi 4 hardware, enabling them to run operating systems like QNX on their host machine without needing the physical device. This setup is useful for testing, development, and debugging. However, QEMU emulation presents its own challenges, particularly when it comes to emulating hardware like UARTs (Universal Asynchronous Receiver-Transmitters) and Ethernet controllers.

A typical QEMU command to run QNX on a Raspberry Pi 4 can look like this:

qemu-system-aarch64     -M raspi4b        -kernel ifs-rpi4.bin    
 -append "startup-bcm2711-rpi4 -v -D pl011-3"     
-cpu cortex-a72     -dtb bcm2711-rpi-4-b.dtb    
 -serial null -serial stdio      
  -drive file=sd.img,if=sd    
  -m 2G -smp 4      -trace "sd*"  -trace file=trace_output.log  
 -monitor telnet::45454,server,nowait 
 -display gtk

In previous version as well as main branch of qemu you can use 
-sd sd.img
This is equvalent to
-drive id=sd-card,if=none,format=raw,file=sd.img
 -device sd-card,drive=sd-card,bus=sd-bus
My patched version has a "hack" that connects the sd card to the correct bus and you only need
 -drive id=sdcard,if=none,format=raw,file=sd.img

The Role of the Device Tree

A key component of getting QNX running smoothly in an emulated Raspberry Pi 4 environment is the Device Tree. The device tree (.dts file) is a data structure that describes the hardware components of the board, including CPU, memory, peripherals, and how they are connected. When QEMU starts, it reads the device tree to configure the emulation environment.

For example, one issue we encountered was enabling the UART4 serial port on the Raspberry Pi 4. The Raspberry Pi 4 has multiple UARTs, and each one needs to be properly configured in the device tree for it to be accessible in QNX. UART4 is not part of standard qemu but I add it in my fork of qemu which also includes serveral STM32 devices. https://github.com/Ebiroll/qemu-stm32

This UART 4 are not added to standard qemu, but I wanted to add it to allow pdebug to run on a serial device.

pdebug

serial@7e201800 {
    compatible = "arm,pl011\0arm,primecell";
    reg = <0x7e201800 0x200>;
    interrupts = <0x00 0x79 0x04>;
    clocks = <0x08 0x13 0x08 0x14>;
    clock-names = "uartclk\0apb_pclk";
    arm,primecell-periphid = <0x241011>;
    status = "okay";
    phandle = <0xc2>;
};

UART0: 0x7e201000
UART2: 0x7e201400
UART3: 0x7e201600
UART4: 0x7e201800
UART5: 0x7e201a00

/*
 * UART Controllers
 */
#define BCM2711_UART0_BASE                  0xfe201000
#define BCM2711_UART0_IRQ                   (96+57)
#define BCM2711_UART1_BASE                  0xfe215000
#define BCM2711_UART1_IRQ                   (96+29)
#define BCM2711_UART2_BASE                  0xfe201400
#define BCM2711_UART2_IRQ                   (96+57)
#define BCM2711_UART3_BASE                  0xfe201600
#define BCM2711_UART3_IRQ                   (96+57)
#define BCM2711_UART4_BASE                  0xfe201800
#define BCM2711_UART4_IRQ                   (96+57)
#define BCM2711_UART5_BASE                  0xfe201a00
#define BCM2711_UART5_IRQ                   (96+57)

Additional RPi4 serial ports

As QNX is a microkernel you can add devices that are not started at boot like this. UART4

devc-serpl011 -b115200 -c48000000 -e -F -u3 0xfe201800,153

Start sdmmc for access to

devb-sdmmc-bcm2711 dos exe=all mem name=below1G sdio addr=0xfe340000,irq=158,bs=bmstr_base=0xc0000000,verbose=7 disk name=sd

Start sdmmcfor newer SOC

   devb-sdmmc-bcm2711 dos exe=all mem sdio addr=0xfe340000,irq=158 disk name=sd

QTREE

(qemu) info qtree
bus: main-system-bus
  type System
  dev: arm_gic, id ""
    gpio-out "sysbus-irq" 20
    num-cpu = 4 (0x4)
    num-irq = 224 (0xe0)
    revision = 2 (0x2)
    has-security-extensions = false
    has-virtualization-extensions = true
    num-priority-bits = 8 (0x8)
     ...
    dev: bcm2835-property, id ""
    gpio-out "sysbus-irq" 1
    board-rev = 11546901 (0xb03115)
    command-line = "startup-bcm2711-rpi4 -v -D pl011-3"
    mmio ffffffffffffffff/0000000000000010
  dev: bcm2835-fb, id ""
    gpio-out "sysbus-irq" 1
    vcram-base = 1006632960 (0x3c000000)
    vcram-size = 67108864 (0x4000000)
    xres = 640 (0x280)
    yres = 480 (0x1e0)
    bpp = 16 (0x10)
    pixo = 1 (0x1)
    alpha = 2 (0x2)
    mmio ffffffffffffffff/0000000000000010
  dev: bcm2835-mbox, id ""
    gpio-in "" 9
    gpio-out "sysbus-irq" 1
    mmio ffffffffffffffff/0000000000000400
  dev: bcm2835-aux, id ""
    gpio-out "sysbus-irq" 1
    chardev = "serial1"
    mmio ffffffffffffffff/0000000000000100
  dev: pl011, id ""
    gpio-out "sysbus-irq" 6
    clock-in "clk" freq_hz=48 MHz
    chardev = ""
    migrate-clk = true
    mmio ffffffffffffffff/0000000000001000
  dev: pl011, id ""
    gpio-out "sysbus-irq" 6
    clock-in "clk" freq_hz=48 MHz
    chardev = "serial0"
    migrate-clk = true

By looking at this we see that uart4 has not had the chardev set properly but when starting with one more -serial argument it shows up.

 dev: pl011, id ""
    gpio-out "sysbus-irq" 6
    clock-in "clk" freq_hz=48 MHz
    chardev = "serial2"
    migrate-clk = true
    mmio ffffffffffffffff/0000000000001000
  dev: pl011, id ""
    gpio-out "sysbus-irq" 6
    clock-in "clk" freq_hz=48 MHz
    chardev = "serial0"
    migrate-clk = true
    mmio ffffffffffffffff/0000000000001000

And it is now possible to output and read from the device after starting it,

# devc-serpl011 -b115200 -c48000000 -e -F -u3 0xfe201800,153
# ls /dev/ser*
/dev/ser1    /dev/ser2    /dev/ser3
# echo "Hello from serial device" > /dev/ser2

One can note here that /dev/ser3 that is started by the qnx init script is not emulated.
miniuart is attached to serial 1
# echo "hello" > /dev/ser1
hello
To get help with a specific command try
use slog2info

Memory tree from qemu,

 info mtree
address-space: bcm2835-mbox-memory
  0000000000000000-000000000000008f (prio 0, i/o): bcm2835-mbox
    0000000000000010-000000000000001f (prio 0, i/o): bcm2835-fb
    0000000000000080-000000000000008f (prio 0, i/o): bcm2835-property

address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io

address-space: bcm2838-pcie-root
  0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container

address-space: bcm2838_genet_dma
address-space: cpu-memory-0
address-space: cpu-memory-1
address-space: cpu-memory-2
address-space: cpu-memory-3
address-space: cpu-secure-memory-0
address-space: cpu-secure-memory-1
address-space: cpu-secure-memory-2
address-space: cpu-secure-memory-3
address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000000000-000000007fffffff (prio 0, ram): ram
    00000000fc000000-00000000fdffffff (prio 1, i/o): bcm2838-peripherals
      00000000fd500000-00000000fd50930f (prio 0, i/o): bcm2838_pcie_cfg_regs
      00000000fd580000-00000000fd58ffff (prio 0, i/o): bcm2838_genet_regs
    00000000fe000000-00000000ff7fffff (prio 1, i/o): bcm2835-peripherals
      00000000fe003000-00000000fe00301f (prio 0, i/o): bcm2835-sys-timer
      00000000fe004000-00000000fe004fff (prio -1000, i/o): bcm2835-txp
      00000000fe007000-00000000fe007fff (prio 0, i/o): bcm2835-dma
      00000000fe00a000-00000000fe00a023 (prio -1000, i/o): bcm2838-asb
      00000000fe00b200-00000000fe00b3ff (prio 0, i/o): alias mphi @mphi 0000000000000000-00000000000001ff
      00000000fe00b200-00000000fe00b3ff (prio 0, i/o): bcm2835-ic
      00000000fe00b400-00000000fe00b43f (prio -1000, i/o): bcm2835-sp804
      00000000fe00b800-00000000fe00bbff (prio 0, i/o): bcm2835-mbox
      00000000fe100000-00000000fe1001ff (prio 0, i/o): bcm2835-powermgt
      00000000fe101000-00000000fe102fff (prio 0, i/o): bcm2835-cprman
      00000000fe104000-00000000fe104027 (prio 0, i/o): bcm2838-rng200
      00000000fe104000-00000000fe10400f (prio 0, i/o): bcm2835-rng
      00000000fe200000-00000000fe200fff (prio 0, i/o): bcm2838_gpio
      00000000fe200000-00000000fe200fff (prio 0, i/o): bcm2835_gpio
      00000000fe201000-00000000fe201fff (prio 0, i/o): pl011
      00000000fe201800-00000000fe2027ff (prio 0, i/o): pl011
      00000000fe202000-00000000fe202fff (prio 0, i/o): bcm2835-sdhost
      00000000fe203000-00000000fe2030ff (prio -1000, i/o): bcm2835-i2s
      00000000fe204000-00000000fe204017 (prio 0, i/o): bcm2835-spi
      00000000fe205000-00000000fe20501f (prio -1000, i/o): bcm2835-i2c0
      00000000fe20f000-00000000fe20f07f (prio -1000, i/o): bcm2835-otp
      00000000fe212000-00000000fe212007 (prio 0, i/o): bcm2835-thermal
      00000000fe214000-00000000fe2140ff (prio -1000, i/o): bcm2835-spis
      00000000fe215000-00000000fe2150ff (prio 0, i/o): bcm2835-aux
      00000000fe300000-00000000fe3000ff (prio 0, i/o): sdhci
      00000000fe340000-00000000fe3400ff (prio 0, i/o): sdhci
      00000000fe600000-00000000fe6000ff (prio -1000, i/o): bcm2835-smi
      00000000fe804000-00000000fe80401f (prio -1000, i/o): bcm2835-i2c1
      00000000fe805000-00000000fe80501f (prio -1000, i/o): bcm2835-i2c2
      00000000fe900000-00000000fe907fff (prio -1000, i/o): bcm2835-dbus
      00000000fe910000-00000000fe917fff (prio -1000, i/o): bcm2835-ave0
      00000000fe980000-00000000fe990fff (prio 0, i/o): dwc2
        00000000fe980000-00000000fe980fff (prio 0, i/o): dwc2-io
        00000000fe981000-00000000fe990fff (prio 0, i/o): dwc2-fifo
      00000000fec00000-00000000fec00fff (prio -1000, i/o): bcm2835-v3d
      00000000fec11000-00000000fec110ff (prio -1000, i/o): bcm2835-clkisp
      00000000fee00000-00000000fee000ff (prio -1000, i/o): bcm2835-sdramc
      00000000fee05000-00000000fee050ff (prio 0, i/o): bcm2835-dma-chan15
    00000000ff800000-00000000ff8000ff (prio 0, i/o): bcm2836-control
    00000000ff841000-00000000ff841fff (prio 0, i/o): gic_dist
    00000000ff842000-00000000ff843fff (prio 0, i/o): gic_cpu
    00000000ff844000-00000000ff844fff (prio 0, i/o): gic_viface
    00000000ff845000-00000000ff8450ff (prio 0, i/o): gic_cpu
    00000000ff845200-00000000ff8452ff (prio 0, i/o): gic_cpu
    00000000ff845400-00000000ff8454ff (prio 0, i/o): gic_cpu
    00000000ff845600-00000000ff8456ff (prio 0, i/o): gic_cpu
    00000000ff846000-00000000ff847fff (prio 0, i/o): gic_vcpu
    0000000600000000-000000063fffffff (prio 0, i/o): alias pcie-mmio @bcm2838_pcie_mmio_window 00000000c0000000-00000000ffffffff

More cool info, https://github.com/aiminickwong/RaspberryPi-1/blob/master/docs/0045_SoC_Device_Tree.md

It turns out that the aux serial emulation and you can use pdebug if you use pdebug -

Start emulation like this,

qemu-system-aarch64     -M raspi4b       
 -kernel ifs-rpi4.bin   
 -append "startup-bcm2711-rpi4 -v -D pl011-3"   
 -serial tcp::12344,server,nowait    
-serial tcp::12345,server,nowait    
-drive file=sd.img,if=sd.img     
-trace "sd*"  -trace file=trace_output.log   
-monitor telnet::45454,server,nowait  
-nic user,model=bcmgenet,hostfwd=tcp::5555-:22  -display gtk

 nc 127.0.0.1 12345
Starting USB xHCI controller in the host mode (/dev/usb/*)...
Starting Network driver...
# pdebug -
pdebug -
~�~#
Ctrl-C


$HOME/qnx700/host/linux/x86_64/usr/bin/ntoaarch64-gdb
(gdb) target qnx 127.0.0.1:12345
Remote debugging using 127.0.0.1:12345
Remote target is little-endian
(gdb)info proc
Not supported on this target.
(gdb) info pidlist
...
bin/slogger2 - 2/1
sbin/pipe - 3/1
sbin/pipe - 3/2
sbin/pipe - 3/3
sbin/pipe - 3/4
sbin/pipe - 3/5
usr/sbin/dumper - 4/1
bin/wdtkick - 5/1
usr/sbin/random - 6/1
...
sbin/io-pkt-v6-hc - 49163/5
sbin/pci-server - 12/2
sbin/pci-server - 12/3
sbin/devc-pty - 81935/1
usr/sbin/inetd - 77840/1
usr/sbin/qconn - 81937/1
usr/sbin/qconn - 81937/2
bin/ksh - 81938/1
usr/bin/pdebug - 102420/1

This article is under constuction but maybe useful for someone. More info will be added and hopefully one day sdmmc will work. By using the debugger it should be easier to understand why devb-sdmmc-bcm2711 is failing.