Python SSL sockets and Systemd activation

Systemd implements activation socket, an existing feature in inetd/xinetd. It’s awesome because systemd can start a process on demand, when needed and you don’t “manage” socket in your code, just reuse it.

From the contested Lennart Poettering

The basic idea of socket activation is not new. The inetd superserver was a standard component of most Linux and Unix systems since time began: instead of spawning all local Internet services already at boot, the superserver would listen on behalf of the services and whenever a connection would come in an instance of the respective service would be spawned. This allowed relatively weak machines with few resources to offer a big variety of services at the same time. However it quickly got a reputation for being somewhat slow: since daemons would be spawned for each incoming connection a lot of time was spent on forking and initialization of the services – once for each connection, instead of once for them all.

No troll, please ;) and instead of making very long explanations, I will make an example with a server/client, add systemd and finally SSL.

Simple server / client

Server implements SocketServer, a module that simplifies the task of writing network servers. It’s run an unix server socket and stop when Ctrl+C is triggered.

    #!/bin/python
    # -*- coding: utf-8 -*-
    import os
    import SocketServer

    socket_file_path = "/tmp/myservice.sock"

    class MyRequestHandler(SocketServer.BaseRequestHandler):
        """Handle request from client"""
        def handle(self):
            data = self.request.recv(1024).strip()
            print(str(data))

    server = SocketServer.UnixStreamServer(socket_file_path, MyRequestHandler)
    try:
        print("start server!")
        server.serve_forever()
    except KeyboardInterrupt:
        print("googbye!")
        server.shutdown()
        os.remove(socket_file_path)

And a client sending a simple “hello world!”.

    # -*- coding: utf-8 -*-
    import socket

    socket_file_path = "/tmp/myservice.sock"

    s = socket.socket(socket.AF_UNIX,socket.SOCK_STREAM)
    s.connect(socket_file_path)
    s.send("hello world!")
    s.close()

Now, run server

$ python server.py
start server!

Then client and send “hello world!”

$ python client

You will see the message in server.py output.

$ python server.py
start server!
hello world!

Press Ctrl+C

$ python server.py
start server!
hello world!
^C
googbye!

Systemd activation Socket

Systemd must be splited in some files called “unit files”. The first one is the service, get more informations here

    $ cat /lib/systemd/system/myservice.service
    [Unit]
    Description=myservice daemon
    After=network.target
    Requires=myservice.socket

    [Service]
    User=root
    Group=root
    WorkingDirectory=/opt/myservice
    ExecStart=/usr/bin/python -u /opt/myservice/server.py
    ExecReload=/bin/kill -HUP 
    ExecStop=/bin/kill -TERM
    StandardOutput=journal
    StandardError=journal

    [Install]
    WantedBy=multi-user.target

-u, StandardOutput=journal and StandardError=journal are important to enable systemd journal logging.

Then, create an unix socket service owned by user.

    $ cat /lib/systemd/system/myservice.socket
    [Unit]
    Description=myservice socket
    PartOf=myservice.service

    [Socket]
    ListenStream=/var/run/myservice.sock
    SocketMode=0660
    SocketUser=ademir
    SocketGroup=ademir
    Service=myservice.service

    [Install]
    WantedBy=sockets.target

Enable and start socket service:

$ systemctl enable myservice.socket
$ systemctl start myservice.socket

Check

$ systemctl status  myservice.socket
● myservice.socket - myservice socket
   Loaded: loaded (/lib/systemd/system/myservice.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2015-12-01 17:18:11 CET; 1min 34s ago
   Listen: /tmp/myservice.sock (Stream)
$ systemctl status  myservice.service
● myservice.service - myservice daemon
   Loaded: loaded (/lib/systemd/system/myservice.service; disabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2015-12-01 17:17:35 CET; 2min 42s ago

The socket is active and the service inactive. Check socket status with lsof

$ lsof | grep myservice
systemd       1                   root   26u     unix 0xffff88020e8fb800        0t0    1942902 /tmp/myservice.sock

Or with ss

$ ss -x -a | grep -i myservice
u_str  LISTEN     0      128    /tmp/myservice.sock 1942902               * 0      

Run the client and the socket will activate the service.

$ python client

Check the service

$ systemctl status  myservice.service
● myservice.service - myservice daemon
   Loaded: loaded (/lib/systemd/system/myservice.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2015-12-01 17:22:24 CET; 1s ago
 Main PID: 31190 (python)
   CGroup: /system.slice/myservice.service
           └─31190 /usr/bin/python -u /opt/myservice/server.py

Service is active and the server print on standard output, with systemd you can use journalctl

$ journalctl -u myservice
systemd[1]: Started myservice daemon.
systemd[1]: Starting myservice daemon...
python[31190]: Traceback (most recent call last):
python[31190]: File "/opt/myservice/server.py", line 14, in <module>
python[31190]: server = SocketServer.UnixStreamServer(socket_file_path, MyRequestHandler)
python[31190]: File "/usr/lib/python2.7/SocketServer.py", line 420, in __init__
python[31190]: self.server_bind()
python[31190]: File "/usr/lib/python2.7/SocketServer.py", line 434, in server_bind
python[31190]: self.socket.bind(self.server_address)
python[31190]: File "/usr/lib/python2.7/socket.py", line 228, in meth
python[31190]: return getattr(self._sock,name)(*args)
python[31190]: socket.error: [Errno 98] Address already in use
systemd[1]: myservice.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit myservice.service entered failed state.
systemd[1]: myservice.service failed.

Ooooops! I forget something. The socket is opened by systemd and in state LISTEN, so in code I try to create the socket, not reuse it causing an exception with server_bind.

So, how a socket works ?

socket bin listen accept schema

Before sending data between them, server and client must initiate a connection. When we run the server.py without systemd, the server create, bind, listen and accept the socket. But with systemd the server must change, it will only accept the connection and let systemd create, bind and listen the socket.

We must change the socket management in server.

Systemd documentation said

Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd’s native socket passing interface (see sd_listen_fds(3) for details) or via the traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).

OK, we will use systemd’s native socket :) Take a look at sd_listen_fds

sd_listen_fds() may be invoked by a daemon to check for file descriptors passed by the service manager as part of the socket-based activation logic. It returns the number of received file descriptors. If no file descriptors have been received, zero is returned. The first file descriptor may be found at file descriptor number 3 (i.e. SD_LISTEN_FDS_START), the remaining descriptors follow at 4, 5, 6, …, if any.

Oh, what is a “file descriptor” and why starts from 3 ?

What is a File descriptor

Wikipedia

In Unix and related computer operating systems, a file descriptor (FD, less frequently fildes) is an abstract indicator used to access a file or other input/output resource, such as a pipe or network connection. File descriptors form part of the POSIX application programming interface. A file descriptor is a non-negative integer, represented in C programming language as the type int. … On Linux, the set of file descriptors open in a process can be accessed under the path /proc/PID/fd/, where PID is the process identifier.

And we can learn that 0 is for stdin, 1 stdout, 2 stderr and they are reserved. So we understand why we should start at 3.

If each socket starts from 3, how can I identify it ?

Linux Kernel is awesome. Each process is ‘isolated’ as you can see in /proc/PID/fd

For example, take a look, at the docker service who use socket activation

$ fuser -v -n tcp 2375
                    USER        PID ACCESS COMMAND
2375/tcp:            root          1 F.... systemd
                     root      19971 F.... docker

And fd folder

$ ls -la /proc/19971/fd
0 -> /dev/null
1 -> socket:[227723125]
2 -> socket:[227723125]
3 -> socket:[227723058]
4 -> socket:[227748996]
5 -> socket:[227749933]
6 -> anon_inode:[eventpoll]
7 -> socket:[227749029]
8 -> /dev/mapper/control
9 -> socket:[227723058]
10 -> socket:[227748996]
11 -> /dev/urandom
12 -> /var/lib/docker/linkgraph.db
13 -> socket:[227724910]
14 -> /var/lib/docker/linkgraph.db
15 -> socket:[227724912]

The number in bracket is the inode

$ ss -x -a | grep -i docker 
u_str  LISTEN     0      128    /var/run/docker.sock 227723058               * 0      
* 0      

We see that the /var/run/docker.sock is on fd 3 and 9 with inode 227723058.

Reuse a socket with file descriptor

When systemd runs a service, it sends 2 environment variables:

  • LISTEN_FDS: number of file descriptor launched by systemd
  • LISTEN_PID: process ID

We just look at the LISTEN_FDS variable, if different from 0, we will use socket.fromfd and then override the UnixStreamServer.server_bind() function.

#!/bin/python
# -*- coding: utf-8 -*-
import os
import socket
import SocketServer

socket_file_path = "/tmp/myservice.sock"

LISTEN_FDS = int(os.environ.get("LISTEN_FDS", 0))
LISTEN_PID = os.environ.get("LISTEN_PID", None) or os.getpid()

class MyRequestHandler(SocketServer.BaseRequestHandler):
    """Handle request from client"""
    def handle(self):
        data = self.request.recv(1024).strip()
        print(str(data))

class UnixServer(SocketServer.UnixStreamServer):
    def server_bind(self):
        print("LISTEN_FDS: "+str(LISTEN_FDS))
        print("LISTEN_PID: "+str(LISTEN_PID))
        if LISTEN_FDS == 0:
            print("create new socket")
            SocketServer.UnixStreamServer.server_bind(self)
        else:
            print("rebind socket")
            print("address_family: "+str(self.address_family))
            print("socket_type: "+str(self.socket_type))
            self.socket = socket.fromfd(3, self.address_family, self.socket_type)

server = UnixServer(socket_file_path, MyRequestHandler)

try:
    print("start server!")
    server.serve_forever()
except KeyboardInterrupt:
    print("googbye!")
    server.shutdown()
    os.remove(socket_file_path)

Stop services and start socket service.

$ systemctl stop myservice.socket
$ systemctl stop myservice.service
$ systemctl start myservice.socket

Run client.py

$ python client.py

Look at journal

$ journalctl -u myservice
systemd[1]: Started myservice daemon.
systemd[1]: Starting myservice daemon...
python[26515]: LISTEN_FDS: 1
python[26515]: LISTEN_PID: 26515
python[26515]: rebind socket
python[26515]: address_family: 1
python[26515]: socket_type: 1
python[26515]: start server!
python[26515]: hello world!

We can see the “hello world!”.

If you wan’t to see in live data comming into the socket, you can install socat, thanks to @Valor

$ mv /tmp/myservice.sock /tmp/myservice.sock.original
$ socat -t100 -x -v UNIX-LISTEN:/tmp/myservice.sock,mode=777,reuseaddr,fork UNIX-CONNECT:/tmp/myservice.sock.original

When running client you can see

> 2015/12/01 21:14:35.553534  length=12 from=0 to=11
 68 65 6c 6c 6f 20 77 6f 72 6c 64 21              hello world!
--

Data is clear in socket, so now Let’s Encrypt!

SSL socket

Generate certificates

mkdir /opt/myservice/ssl
cd /opt/myservice/ssl
openssl genrsa -des3 -out server.orig.key 2048
openssl rsa -in server.orig.key -out server.key
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Python have ssl lib and wrap function to wrap any socket into secure socket.

And on server, the get_request() will be overrided to return the secured socket.

    #!/bin/python
    # -*- coding: utf-8 -*-
    import os
    import ssl
    import socket
    import SocketServer

    socket_file_path = "/tmp/myservice.sock"

    LISTEN_FDS = int(os.environ.get("LISTEN_FDS", 0))
    LISTEN_PID = os.environ.get("LISTEN_PID", None) or os.getpid()

    SSL_CERT_PERM = "/opt/myservice/ssl/server.crt"
    SSL_KEY_PERM = "/opt/myservice/ssl/server.key"

    class MyRequestHandler(SocketServer.BaseRequestHandler):
        """Handle request from client"""
        def handle(self):
            data = self.request.recv(1024).strip()
            print(str(data))

    class UnixServer(SocketServer.UnixStreamServer):

        def server_bind(self):
            print("LISTEN_FDS: "+str(LISTEN_FDS))
            print("LISTEN_PID: "+str(LISTEN_PID))
            if LISTEN_FDS == 0:
                print("create new socket")
                SocketServer.UnixStreamServer.server_bind(self)
            else:
                print("rebind socket")
                print("address_family: "+str(self.address_family))
                print("socket_type: "+str(self.socket_type))
                sock = socket.fromfd(3, self.address_family, self.socket_type)
                # //stackoverflow.com/a/16740536
                self.socket = socket.socket(_sock=sock)

        def get_request(self):
            newsocket, fromaddr = self.socket.accept()
            connstream = ssl.wrap_socket(newsocket,
                                         server_side=True,
                                         certfile = SSL_CERT_PERM,
                                         keyfile = SSL_KEY_PERM,
                                         ssl_version = ssl.PROTOCOL_TLSv1,
                                         do_handshake_on_connect=False)
            return connstream, fromaddr

    server = UnixServer(socket_file_path, MyRequestHandler)

    try:
        print("start server!")
        server.serve_forever()
    except KeyboardInterrupt:
        print("googbye!")
        server.shutdown()
        os.remove(socket_file_path)

And update client.py to use secure socket

    # -*- coding: utf-8 -*-
    import ssl
    import socket

    SSL_CERT_PERM = "/opt/myservice/ssl/server.crt"

    socket_file_path = "/tmp/myservice.sock"

    sock = socket.socket(socket.AF_UNIX,socket.SOCK_STREAM)
    s = ssl.wrap_socket(sock,
                               ca_certs=SSL_CERT_PERM,
                               cert_reqs=ssl.CERT_REQUIRED,
                               ssl_version=ssl.PROTOCOL_TLSv1)
    s.connect(socket_file_path)
    s.send("hello world!")
    s.close()

Stop services and start socket service.

$ systemctl stop myservice.socket
$ systemctl stop myservice.service
$ systemctl start myservice.socket

Run client.py

$ python client.py

Look at journal

$ journalctl -u myservice
...
python[29671]: hello world!

We can see the “hello world!”.

To check if the data is encrypted, use the previous socat command.

$ mv /tmp/myservice.sock /tmp/myservice.sock.original
$ socat -t100 -x -v UNIX-LISTEN:/tmp/myservice.sock,mode=777,reuseaddr,fork UNIX-CONNECT:/tmp/myservice.sock.original

When running client you can see

> 2015/12/01 21:23:11.685177  length=179 from=0 to=178
 16 03 01 00 ae 01 00 00 aa 03 01 99 67 81 fc b2  ............g...
 7e 7a b4 0c da f2 93 f4 77 02 88 f3 fd 08 2f 63  ~z......w...../c
 8d ad 6b b9 02 e7 90 c1 b7 5f 9a 00 00 38 c0 14  ..k......_...8..
 c0 0a                                            ..
 c0 0f c0 05 00 39 00 38 c0 13 c0 09 c0 0e c0 04  .....9.8........
 00 33 00 32 c0 12 c0 08 c0 0d c0 03 00 88 00 87  .3.2............
 00 16 00 13 00 45 00 44 00 35 00 2f 00 84 00 0a  .....E.D.5./....
 00 41 00 ff 01 00 00 49 00 0b 00 04 03 00 01 02  .A.....I........
 ...

Data is ecrypted \o/

Conclusion

Systemd is not so hard to understand if you take the time to understand how Linux works. …


Picture by Andreas Hopf under licence CC BY-NC 2.0


If you liked this post, share it on Twitter.