multiroom synced audio using openwrt and pulseaudio (with and without rtp)

This is my first blog-post ever, but i felt the need to publish after reading lots of tutorials, each ending with “it kinda works”. Well, my solution works pretty well, so here are the details.

To get multiroom audio in sync working you need pulseaudio-sinks exposed over network. This is doable with openwrt but has a couple of drawbacks:

  • openwrt’s package of pulseaudio runs in system-mode and there is some still unsolved problems using pulseaudio on the same machine with paplay and other tools (see this post). We will solve this ūüôā
  • zeroconf / avahi is not working, so you need to add the sinks manually. This works pretty well with a script i updated from here
  • but the MAIN REASON people gave up is stuttering. There are 2 reasons for this: 1. People using rtp in a wireless-bridged network (standard openwrt way to do things) 2. People using the wrong samplerate on the host. We will also solve this!

We go step-by-step by using a simple Xubuntu Host with pulseaudio as a server and a cheap WDR3600 for receiving and playing.

Install paprefs using apt-get on your ubuntu box

sudo apt-get install paprefs

You only need this if you want multicast RTP. Multicast RTP is still the best way to get your audio to play in sync. Remember this will NOT work in a network with wireless in the same subnet. You will saturate the wifi with multicast-traffic to EVERY connected client, not broadcasting like 10mbit multicast stream to 3 wifi clients, but 3x 10mbit streams congesting your network. We will use mpd with multiple pulseaudio-sinks for synced playing (stop, start a new song or pause/unpause –> synced). If you want to test multicast streaming in your bridged network simply switch off all wireless on the routers.

That’s it for the host for now. We use Xubuntu 16.10, everything needed for pulseaudio should already be integrated in a recent ubuntu build. We still have work to do if we want stutter-free streaming from this box, but first …. :

Prepare the router. If you never used OpenWRT read the docs on how to obtain it for your wdr3600 or likewise router (Check for free space (df -h), consider using the Image Generator for the missing packages. Using the ImageBuilder will insert the packages on the jffs2-partition (compressed), so more free space ūüôā Using a router with less than 8MB of flash is possible, but VERY HARD to achieve. I recommend this way because you can see how things work without the need for extroot, etc.

ssh onto your box and check “df -h”. After installing pulseaudio-daemon and pulseaudio-tools i still had around 1MB of free space, even without baking the packages into the image:

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 CHAOS CALMER (15.05.1, r48532)
 -----------------------------------------------------
  * 1 1/2 oz Gin            Shake with a glassful
  * 1/4 oz Triple Sec       of broken ice and pour
  * 3/4 oz Lime Juice       unstrained into a goblet.
  * 1 1/2 oz Orange Juice
  * 1 tsp. Grenadine Syrup
 -----------------------------------------------------
root@wdr3600:~# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    4.5M      3.6M    908.0K  80% /
/dev/root                 2.3M      2.3M         0 100% /rom
tmpfs                    61.5M      1.3M     60.2M   2% /tmp
/dev/mtdblock3            4.5M      3.6M    908.0K  80% /overlay
overlayfs:/overlay        4.5M      3.6M    908.0K  80% /
tmpfs                   512.0K         0    512.0K   0% /dev

Install the packages using opkg (update packages first):

root@wdr3600:~# opkg update; opkg install pulseaudio-daemon pulseaudio-tools

If you want to prebake them have a look at which packages are installed now as you need to define every package in the Image Generator if it depends on another (i might be wrong on this one and you only need to define the base packages pulseaudio-deamon and pulseaudio-tools). The complete list of needed packages for pulseaudio is listed on the pulseaudio openwrt page. The list isn’t complete if you install pulseaudio-tools with it, so here’s the complete list:

libsndfile libspeexdsp libltdl librt kmod-input-core kmod-sound-core alsa-lib zlib libopenssl libwrap libcap pulseaudio-daemon pulseaudio-tools

you can try and start the pulseaudio service now using

root@wdr3600:~# /etc/init.d/pulseaudio restart

ps will show you pulseaudio running as pulse user:

 xxxx pulse    11652 S <  /usr/bin/pulseaudio --system --disallow-exit --disallow-module-loading

scp a sample wav file (thanks to helsinki.fi) onto the router in /tmp to test playback using paplay (part of pulseaudio-tools). I saved it as sample.wav

user@xubuntu:~$ scp sample.wav root@wdr3600:/tmp

Fortunately, you don’t need the openssh-sftp-server opkg (for mounting the whole router with sshfs) for this.

Then, try to play it and get a nice error:

root@wdr3600-1:/tmp# paplay /tmp/sample.wav 
Connection failure: Connection refused
pa_context_connect() failed: Connection refused

That’s sad and there are a couple things to fix, i don’t understand too much of. As far as i understood, pulseaudio drops privileges to pulse and expects everybody that wants to play audio to be in the pulse-access group. On a normal ubuntu box every user is in this group, but not on your router. We only have “root” as a user there, so we add him to the pulseaudio-access group that is still missing from /etc/group. You cannot use usermod without the extra shadow packages and i don’t know the busybox-commands to do so, so we do it by hand:

vi /etc/group should look like this:

root@wdr3600-1:/# cat /etc/group
root:x:0:
daemon:x:1:
adm:x:4:
mail:x:8:
audio:x:29:
www-data:x:33:
ftp:x:55:
users:x:100:
network:x:101:
nogroup:x:65534:
pulse:x:51:

add root to the last line, so it looks like this:

pulse:x:51:root

if you wanna do things more right, add the group of pulse-access instead and add root to it:

pulse-access:x:52:root

It’s quite nice to have the ubuntu box ready with pulseaudio, because you can look up how it’s configured there and copy it on your router. Ubuntu has both groups pulse and pulse-access. There are some distros that pack “pulse-rt” as a group to have realtime-access on pulseaudio, it’s not needed in our case.

Pulseaudio packs his configuration into /etc/pulse. There is a default.pa and a system.pa. They both(?) get loaded on Pulseaudio Startup, the system.pa is interpreted in system-mode only(?) (see more explaining here). I have no idea if default.pa is even loaded in our case, every configuration we do will be in system.pa. This blog post said the system.pa is not suitable for openwrt and should be empty (then some lines added). In case you want the default system.pa back because you want to look through, i uploaded it here and the default default.pa here

There are some interesting lines to learn about pulseaudio in there, like this one:

#load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties=”device.description=’RTP Multicast Sink'”

Which creates a new sink and gives it a nice description, we will use this later to have our sinks named more human-friendly.

if we print out the system.pa without comments we will get this:

root@wdr3600:~# grep "^[^#;]" /etc/pulse/system.pa
.ifexists module-detect.so
load-module module-detect
.endif
.ifexists module-esound-protocol-unix.so
load-module module-esound-protocol-unix
.endif
load-module module-native-protocol-unix
load-module module-stream-restore
load-module module-device-restore
load-module module-default-device-restore
load-module module-rescue-streams
load-module module-always-sink
load-module module-suspend-on-idle
load-module module-position-event-sounds

The other blog post is right, we don’t really need anything from this. Google each line to find out what the modules do, here’s an almost complete list of user modules.

The highlighted line loads the native unix protocol. This Issue on the openwrt github tells us why the playback failed and what we can do about it: Change the module-native-protocol-unix line and add a parameter to tell pulseaudio the correct access group. After this, the system.pa should look like this (only one line)

load-module module-native-protocol-unix auth-group=pulse-access

If you added root to the pulse group, instead of the pulse-access change the line accordingly. Both will work.

After those changes start pulseaudio again using “/etc/init.d/pulseaudio restart”.

Try to play the sample now and it should work. I use a cheap USB-Audio Dongle on the mr3020s and a more expensive one packing TOS-Link on one of my wdr3600s, they both work after installing the needed packages, more on the openwrt wiki page for USB Audio Support.

You can play with 2 ubuntu boxes and paprefs first to set up sound from one box to the other before we do this with the router: Start paprefs on both and activate “Make discoverable PulseAudio network sound devices available locally” on the first tab, this is equivalent to “load-module module-zeroconf-discover” in default.pa. Then activate the 3 checkboxes on the second tab. This is equivalent to “load-module module-native-protocol-tcp auth-anonymous=1” in default.pa / system.pa. The second checkbox is equivalent to “load-module module-zeroconf-publish”. You can also play around with rtp here on the third tab and experience that it won’t work on wifi bridged networks. Then, using the pulseaudio Volume Control you can reassign playback to the other machine. Using 2 ubuntu boxes will be painless because they can figure out the samplerate between them just fine. I recommend reading the network setup page on pulseaudio.

When this works it’s time to add the sink on the router. The zeroconf modules sadly will not work without recompiling the pulseaudio-daemon¬† package. But the routers will seldomly change their ip address or configuration, so adding their sinks by hand/script is just fine.

To have the routers allow access to their sinks add this to system.pa:

load-module module-native-protocol-tcp auth-anonymous=1

and restart pulseaudio using “/etc/init.d/pulseaudio restart”.

Now is a good time to start debugging ūüôā To do so, we want to see whats happening in the pulseaudio on the router, so we change the init-script and add logging. Go to /etc/init.d and vi pulseaudio so it looks like this:

#!/bin/sh /etc/rc.common
# Copyright (C) 2011 OpenWrt.org

START=65
STOP=65

USE_PROCD=1
PROG=/usr/bin/pulseaudio

start_service() {
    [ -d /var/run/pulse ] || {
        mkdir -m 0755 -p /var/run/pulse
        chmod 0750 /var/run/pulse
        chown pulse:pulse /var/run/pulse
    }
    [ -d /var/lib/pulse ] || {
        mkdir -m 0755 -p /var/lib/pulse
        chmod 0750 /var/lib/pulse
        chown pulse:pulse /var/lib/pulse
    }

    chown root:pulse /dev/snd/* /dev/mixer /dev/dsp
    chmod 664 /dev/snd/* /dev/mixer /dev/dsp

    procd_open_instance
    procd_set_param command $PROG --system --disallow-exit --disallow-module-loading --disable-shm --exit-idle-time=-1 --realtime=false -v --log-target=newfile:/tmp/pulseverbose.log --log-time=1
    procd_close_instance
}

added parameters in bold. -v for verbose output, log-target is self-explaining, log-time adds a timestamp (?)

Restart pulseaudio and watch the /tmp/pulseverbose.log log. It’s very nice for debugging stuttering and finding out how your usb-soundcard is detected. You can even see when you open pavucontrol connected to the router. Remember, a new file gets added each time you restart the service, so when things look dumb ls /tmp and choose the latest, like “pulseverbose.log.3” In my case, using a USB-Dongle with TOS-Link the log file tells us, that:

(   0.062|   0.002) I: [pulseaudio] sink.c: Created sink 0 "alsa_output.hw_0" with sample spec s16le 2ch 44100Hz and channel map front-left,front-right
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.resolution_bits = "16"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.api = "alsa"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.class = "sound"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.class = "generic"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.subclass = "generic-mix"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.name = "USB Audio"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.id = "USB Audio"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.subdevice = "0"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.subdevice_name = "subdevice #0"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.device = "0"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.card = "0"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.card_name = "USB PNP SOUND DEVICE"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.long_card_name = "C-Media Electronics Inc. USB PNP SOUND DEVICE at usb-ehci-platform-1.1, full sp"
(   0.062|   0.002) I: [pulseaudio] sink.c:     alsa.driver_name = "snd_usb_audio"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.string = "hw:0"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.buffering.buffer_size = "17632"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.buffering.fragment_size = "4408"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.access_mode = "mmap"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.description = "USB PNP SOUND DEVICE"
(   0.062|   0.002) I: [pulseaudio] sink.c:     device.icon_name = "audio-card"
(   0.063|   0.001) I: [pulseaudio] source.c: Created source 0 "alsa_output.hw_0.monitor" with sample spec s16le 2ch 44100Hz and channel map front-left,front-right
(   0.063|   0.001) I: [pulseaudio] source.c:     device.description = "Monitor of USB PNP SOUND DEVICE"
(   0.063|   0.001) I: [pulseaudio] source.c:     device.class = "monitor"
(   0.063|   0.001) I: [pulseaudio] source.c:     device.icon_name = "audio-input-microphone"
.
.
.
(1825.692|   0.000) I: [pulseaudio] protocol-native.c: Requested tlength=250.00 ms, minreq=20.00 ms
(1825.692|   0.000) I: [pulseaudio] protocol-native.c: Final latency 299.98 ms = 210.00 ms + 2*20.00 ms + 49.98 ms

Sometimes, you need to make sure every new audio source (from another computer sending to the router) is played on the correct sink, so you can set it as default. I use this line in system.pa to make sure the correct card is used for the rtp source f.e.

load-module module-native-protocol-unix auth-group=pulse-access

load-module module-alsa-sink device=hw:0
load-module module-native-protocol-tcp auth-anonymous=1

load-module module-rtp-recv sink=alsa_output.hw_0

You see on the last line that you can now make sure this sink is used for rtp if you wanna play around with that. You can also add

set-default-sink alsa_output.hw_0

To set the alsa sink as default for every source that ever starts playing on your router.

As soon as you start playing a stream through the routers, the log will show you the total latency of the stream. It should have the same values across the same routers, regardless the soundcard used, because pulseaudio uses to parameters in /etc/pulse/daemon.conf to determine the buffer sent to the card before playing:

 default-fragments = 2 
 default-fragment-size-msec = 25

This is the minimum that worked for me and results in a 300ms total latency (sending -> playing) across all clients with <0.00ms difference in between them.

You should now have a usable configuration you can play around with. On your ubuntu box, open a terminal and start a volume control ON THE ROUTER:

user@xubuntu:~$ PULSE_SERVER=10.50.0.2 pavucontrol

Change the ip to that of your router. If you’re interested, we have a blog page about our home network. Now the fun begins, as you can see which sources try to play on your router, just like your programs on the ubuntu box (mediaplayer, browser) do.

To send audio to the router, you need to tell your ubuntu box about a new network sink, as zeroconf / avahi autodiscover does not work:

user@xubuntu:~$ pacmd load-module module-tunnel-sink-new server=10.50.0.2 sink_name=wdr3600-1 sink_properties=device.description="wdr3600-1.helenenhof.org"

Other tutorials will tell you to use module-tunnel-sink which will lead to problems. Sometimes you cannot connect, need to kill the pulseaudio server on your ubuntu box and restart it, then reconnect a.s.a.s.f. I had a script do this (pulseaudio -k && sleep 10 && pacmd load-module module-tunnel-sink server=…), but this wasn’t satisfactory, i wanted a maximum one click or autostart solution and this line works EVERY time. If you studied the post closely you can see we can name the sink more friendly.

Your Volume Control now will look like this:

left side: pavucontrol on 2 routers, right side: local xubuntu box. Routers 1 & 3 play a MPD source and network connection from my box. I then play a sample and reroute it to the first router that mixes both.

You could write a script that adds the sinks and sets one as the default sink for your local ubuntu box, like this:

pacmd load-module module-tunnel-sink-new server=10.50.0.2 sink_name=wdr3600-1 sink_properties=device.description="wdr3600-1.helenenhof.org"
pacmd set-default-sink wdr3600-1

Still, pulseaudio will remember your sink of choice for every program, so if this doesn’t seem to set the default sink for a program remember this and change it manually.

You will soon realize the sound-output stutters. Watching the logs will tell you something about unexpected sample-rates. So on every box that plays audio on your router-sinks you need to edit the pulseaudio daemon.conf in /etc/pulse/. Change the file in this location and delete everything in your user .config/pulse folder, then restart pulseaudio with “pulseaudio -k” under your normal user account. If you have configuration in your .config/pulse folder it will be used instead of the /etc/pulse one.

My daemon.conf looks like this without commented lines:

user@xubuntu:~$ grep "^[^#;]" /etc/pulse/system.pa
resample-method = speex-float-5
flat-volumes = no
default-sample-format = s16le
default-sample-rate = 44100
alternate-sample-rate = 48000
default-sample-channels = 2
default-channel-map = front-left,front-right
deferred-volume-safety-margin-usec = 1

The lines in italics were not changed by me but uncommented. flat volumes is disabled, enabling can lead to the volume of the output sink raise >100% when adjusting a stream volume (through mpd / kodi f.e.)

resample-method = speex-float-5

This line chooses a resampler. -5 means middle quality, but even -1 was ok. You can choose from -1 to -9, higher means better quality.

default-sample-format = s16le
default-sample-rate = 44100
alternate-sample-rate = 48000
default-sample-channels = 2
default-channel-map = front-left,front-right

Those lines set the samplerate and format. Without those you will end up with stuttering sound and lines in your pulseverbose.log saying something about mismatched and unexpected sample rates.

For synced sound add those sinks to your mpd.conf now:

audio_output {
    type        "pulse"
    name        "wdr3000-1.helenenhof.org"
    server        "10.50.0.2"        # optional
    sink        "alsa_output.hw_0"    # optional
}

The sink line is recommended but not needed, PA will choose the first sink. Add another output for each router with pulseaudio enabled. MPD will often crash when a sink is not responding, don’t worry and kill the mpd server with “kill -9 PID”, then restart the server and start again. Disable all sinks, stop playback and enable them one-by-one, start and stop playback in between until you find the faulty sink. Another mistake i made was leaving the pavucontrol open. Over wifi i guess this creates to much traffic. Close them and try again.

On using rtp:

If you wanna play around with rtp add the following line to your routers /etc/pulse/system.pa:

load-module module-rtp-recv sink=alsa_output.hw_0

this tells pulseaudio to use your default sink for rtp receiving. Go into paprefs on your ubuntu box and enable the RTP Multicast sender on tab 3. Restart pulseaudio on the router. Play some music on the ubuntu box and reassign audio output to your newly generated RTP Multicast Sink. If you remembered to switch off wifi on your routers this will work perfectly in sync.

You can now play music in sync using mpd from your server or from any ubuntu box using rtp. There is a solution to combine multiple sinks when not using rtp to play from a ubuntu box in sync on every router, more details on this here

UPDATE

When combining the sinks pulseaudio will crash if you use module-tunnel-sink-new. The new module provides better stability for me, as i sometimes had to kill pulseaudio and readd the sinks using the old module, because playback would simply not start and pulseaudio starts to hang when reassigning a stream to the tunneled sinks. So using the normal module-tunnel-sink and module-combine-sink will offer a solution to stream from a ubuntu box on multiple speakers without rtp, not just from mpd using multiple pulseaudio audio_outputs. You can even combine a new sink that splits 4.0 into 2 2.0 sinks, more on this later.

For combining 2 sinks, use this script:

# /bin/bash
pacmd load-module module-tunnel-sink server=10.50.0.2 sink_name=wdr3600-1
pacmd load-module module-tunnel-sink server=10.50.0.4 sink_name=wdr3600-3

pacmd load-module module-combine sink_name=combined slaves="wdr3600-1, wdr3600-3"

Unfortunately, the old module provides no way (i know of) to rename the sinks in the pulseaudio volume control.

After all, it’s almost working perfectly! If you want to listen to our shared-flat-radio-station you can do so at https://helenenhof.org

And that’s it! Have fun and please correct me and comment on my post.

3 Replies to “multiroom synced audio using openwrt and pulseaudio (with and without rtp)”

  1. Hi Richard

    Thank you for your effort.
    Looking at the final results, I’ve noticed that the latency across clients never is better than 40mS not 0.00mS as you mention. Sorry, but, are you sure about it?

    Best Regards,
    Sergio

Leave a Reply

Your email address will not be published. Required fields are marked *