[CuBox] Real-time Kernel and Tuning

*** This page only applies for old CuBox but not CuBox-i ***

You can run MuBox with real-time kernel on CuBox. Current development snapshot already runs real-time kernel as default. However, it is very tricky to tune your MuBox to run on real-time systems. Here is a general guide. This is not sliver bullet and your mileage may vary.

Install Real-Time Kernel

Follow Testing Kernel page to install real-time kernel. The most stable real-time kernel by far is 3.8.11-rt8. Reboot after install the kernel. Run the following command to verify your kernel:

# uname -rv
3.8.11-1.voyage-rt8 #5 PREEMPT RT Tue Jul 23 12:07:19 HKT 2013

If you occasionally hear random "pop" sound during playback, your MuBox is not running optimal settings for real-time kernel. From our experience, this annoying "pop" sound no longer exist after we apply the settings below:

Install MPD with real-time support option

We built a separate MPD .deb package with real-time support option from Yan's patch. To download and install it, run:

# remountrw
# cd ~/
# wget http://www.voyage.hk/dists/experimental/mpd/mpd_0.18.0git20130624-1_armhf.deb
# dpkg -i mpd_0.18.0git20130624-1_armhf.deb

After installation, enable real-time options in /etc/mpd.conf. You need to add realtime_option and change ALSA audio_output section:

# realtime option
realtime_option {
    memlock              "yes"
    stack_reserve        "1024"
    heap_reserve         "10240"

    main_priority        "OTHER:0"
    player_priority      "FIFO:32"
    decoder_priority     "FIFO:31"
    update_priority      "OTHER:0"
}
....
audio_output {
        type            "alsa"
        name            "My ALSA Device"
        device          "hw:0,0"
        priority        "FIFO:32"
...
}

In above, we set player thread to priority 32, decoder thread to 31 and ALSA output thread 32, using FIFO scheduling policy. Then, restart mpd by:

# /etc/init.d/mpd restart
Stopping Music Player Daemon: mpd.
Starting Music Player Daemon: mpd.

After all, you can verify real-time priority from ps command:

# ps -eLo pid,group,lwp,rtprio,priority,cls,time,wchan,cmd | egrep "mpd"
 2642 audio     2642      -  20  TS 00:00:15 poll_s /usr/bin/mpd /etc/mpd.conf
 2642 audio     2643      -  20  TS 00:00:00 poll_s /usr/bin/mpd /etc/mpd.conf
 2642 audio     2646     32 -33  FF 00:00:49 futex_ /usr/bin/mpd /etc/mpd.conf
 2642 audio     2647     31 -32  FF 00:33:05 futex_ /usr/bin/mpd /etc/mpd.conf
 2642 audio     2648     32 -33  FF 00:04:18 wait_f /usr/bin/mpd /etc/mpd.conf
 5539 root      5539      -  20  TS 00:00:00 pipe_w egrep mpd

Setting real-time priority for USB and NIC Interrupts

If you run MPD with real-time option, you must also set the real-time priority for USB and Ethernet interrupt handlers:

# chrt -f -p 54 `pgrep irq/24-ehci_hcd`
# chrt -f -p 54 `pgrep irq/25-ehci_hcd`
# chrt -f -p 49 `pgrep irq/29-eth0`
# chrt -f -p 48 `pgrep irq/62-sata_mv`

This time, we set USB interrupt handler to priority 54, Ethernet to 49, SATA to 48.

Please note that we intentionally set real-time priority for ethernet port, as we noticed that the "pop" sound was heard when irq/29-eth0 consuming higher CPU intermittently (as we stored music files in NFS shared drive).

To verify real-time priority:

# ps -eLo pid,group,lwp,rtprio,priority,cls,time,cmd | egrep "irq/2[459]|irq/6"
  407 root       407     48 -49  FF 00:00:00 [irq/62-sata_mv]
  441 root       441     54 -55  FF 02:41:26 [irq/24-ehci_hcd]
  444 root       444     54 -55  FF 00:00:00 [irq/25-ehci_hcd]
 1805 root      1805     49 -50  FF 00:18:48 [irq/29-eth0]
 4443 root      4443      -  20  TS 00:00:00 egrep irq/2[459]|irq/6

If you mount CIFS shared drive, you should also set the real-time priority for cifsd:

# chrt -f -p 49 `pgrep cifsd`
# ps -eLo pid,group,lwp,rtprio,priority,time,cmd | egrep "cifsd"
 1518 root      1518     49 -54 00:00:05 [cifsd]

Adjusting sysctl.conf and NIC card tuning

Append the following in /etc/sysctl.conf to increase NIC card and TCP throughput (mainly for NAS access):

net.core.rmem_max=12582912
net.core.wmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 0
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 5000
vm.overcommit_memory = 2
vm.overcommit_ratio = 50

Then run below or reboot to activate it.:

# /sbin/sysctl -p

Then, increase the transmission queue length to 10000 (default = 1000) in NIC card to increase overall throughput:

# ifconfig eth0 txqueuelen 10000
# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr d0:63:b4:00:0e:24
          inet addr:10.1.10.79  Bcast:10.1.10.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:26334878 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13905684 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10000
          RX bytes:929187122 (886.1 MiB)  TX bytes:1423838807 (1.3 GiB)
          Interrupt:29

Increase send/receive buffer for NFS and CIFS mounts

Setting rsize= and wsize= to increase send/receive buffer in network drive mounts. This again increases network file access throughput.
For NFS mount, in /etc/fstab, e.g.:

192.168.1.101:/home/public /mnt/public  nfs rw,noatime,rsize=8192          0       0

For CIFS/SAMBA mounts, e.g.:

192.168.1.101:/public /mnt/public cifs username=,password=,rsize=8192,wsize=4096,file_mode=0644,dir_mode=0755,iocharset=utf8 0 0 

Troubleshooting Real-Time Settings

We received some user feedback on above tuning steps not working for them. If you applies above but still hear "pop" sound, here are some troubleshooting steps. Please note that you mileage again may vary. You need to adopt trail-n-error approach.

Monitor CPU consumption using top

You can monitor CPU consumption by using top utility. In our case (NFS + USB-DAC) when replay, USB interrupt, Ethernet interrupt and MPD consumes most of the CPU time. Make sure you apply real-priority for them. From our experience, you should see a process suddenly consume high CPU in top after you hear the "pop" sound.
The below shows "top -H" (show threads) output:

top - 04:00:52 up 2 days, 18:11,  1 user,  load average: 0.18, 0.20, 0.12
Threads:  80 total,   1 running,  79 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.4 us,  1.4 sy,  0.0 ni, 94.2 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0 st
KiB Mem:   1032952 total,   986892 used,    46060 free,     1332 buffers
KiB Swap:        0 total,        0 used,        0 free,   864940 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
  441 root     -55   0     0    0    0 S   6.0  0.0 145:52.04 irq/24-ehci_hcd
 1806 root     -50   0     0    0    0 S   2.3  0.0  43:16.87 irq/29-eth0
 2673 mpd      -33   0 74160  69m  24m S   1.3  6.9  18:44.80 output
 1754 root      20   0  2664 1116  784 R   0.7  0.1   0:01.25 top
 2672 mpd      -32   0 74160  69m  24m S   0.7  6.9  67:02.12 thread
  300 root      20   0     0    0    0 S   0.3  0.0   7:46.15 kswapd0
 2670 mpd      -33   0 74160  69m  24m S   0.3  6.9   2:18.76 player

From above, we set real-time priority for all CPU consumed threads/processes (mpd theads, IRQs) related to music playback. Please note that your result may be different.

Our experiences told that pop/click sound was heard if there is a process suddenly consume 10-20% CPU as shown from top.

Try different real-time priority and buffer sizes

Here are some collection of feedback that works for various MuBox users.

1. Change cifsd process to lower priority 28 (but still higher than normal).

chrt -f -p 28 `pgrep cifsd`

2. Try changing MPD theads to different real-time priority:.

realtime_option {
        ....
        player_priority "FIFO:32"
        decoder_priority "FIFO:31"
...
audio_output {
        ....
        priority "FIFO:32"
...

3. Try changing different NFS receive buffer, in /etc/fstab:

192.168.1.101:/home/public /mnt/public  nfs rw,noatime,rsize=8192          0       0