Systemd implements activation socket, an existing feature in inetd/xinetd. It’s awesome because systemd can start a process on demand, when needed and you don’t “manage” socket in your code, just reuse it.
From the contested Lennart Poettering
The basic idea of socket activation is not new. The inetd superserver was a standard component of most Linux and Unix systems since time began: instead of spawning all local Internet services already at boot, the superserver would listen on behalf of the services and whenever a connection would come in an instance of the respective service would be spawned. This allowed relatively weak machines with few resources to offer a big variety of services at the same time. However it quickly got a reputation for being somewhat slow: since daemons would be spawned for each incoming connection a lot of time was spent on forking and initialization of the services – once for each connection, instead of once for them all.
No troll, please ;) and instead of making very long explanations, I will make an example with a server/client, add systemd and finally SSL.
Simple server / client
Server implements SocketServer, a module that simplifies the task of writing network servers. It’s run an unix server socket and stop when Ctrl+C is triggered.
#!/bin/python
# -*- coding: utf-8 -*-
import os
import SocketServer
socket_file_path = "/tmp/myservice.sock"
class MyRequestHandler(SocketServer.BaseRequestHandler):
"""Handle request from client"""
def handle(self):
data = self.request.recv(1024).strip()
print(str(data))
server = SocketServer.UnixStreamServer(socket_file_path, MyRequestHandler)
try:
print("start server!")
server.serve_forever()
except KeyboardInterrupt:
print("googbye!")
server.shutdown()
os.remove(socket_file_path)
And a client sending a simple “hello world!”.
# -*- coding: utf-8 -*-
import socket
socket_file_path = "/tmp/myservice.sock"
s = socket.socket(socket.AF_UNIX,socket.SOCK_STREAM)
s.connect(socket_file_path)
s.send("hello world!")
s.close()
Now, run server
$ python server.py
start server!
Then client and send “hello world!”
$ python client
You will see the message in server.py output.
$ python server.py
start server!
hello world!
Press Ctrl+C
$ python server.py
start server!
hello world!
^C
googbye!
Systemd activation Socket
Systemd must be splited in some files called “unit files”. The first one is the service, get more informations here
$ cat /lib/systemd/system/myservice.service
[Unit]
Description=myservice daemon
After=network.target
Requires=myservice.socket
[Service]
User=root
Group=root
WorkingDirectory=/opt/myservice
ExecStart=/usr/bin/python -u /opt/myservice/server.py
ExecReload=/bin/kill -HUP
ExecStop=/bin/kill -TERM
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
-u, StandardOutput=journal and StandardError=journal are important to enable systemd journal logging.
Then, create an unix socket service owned by user.
$ cat /lib/systemd/system/myservice.socket
[Unit]
Description=myservice socket
PartOf=myservice.service
[Socket]
ListenStream=/var/run/myservice.sock
SocketMode=0660
SocketUser=ademir
SocketGroup=ademir
Service=myservice.service
[Install]
WantedBy=sockets.target
Enable and start socket service:
$ systemctl enable myservice.socket
$ systemctl start myservice.socket
Check
$ systemctl status myservice.socket
● myservice.socket - myservice socket
Loaded: loaded (/lib/systemd/system/myservice.socket; enabled; vendor preset: enabled)
Active: active (listening) since Thu 2015-12-01 17:18:11 CET; 1min 34s ago
Listen: /tmp/myservice.sock (Stream)
$ systemctl status myservice.service
● myservice.service - myservice daemon
Loaded: loaded (/lib/systemd/system/myservice.service; disabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2015-12-01 17:17:35 CET; 2min 42s ago
The socket is active and the service inactive. Check socket status with lsof
$ lsof | grep myservice
systemd 1 root 26u unix 0xffff88020e8fb800 0t0 1942902 /tmp/myservice.sock
Or with ss
$ ss -x -a | grep -i myservice
u_str LISTEN 0 128 /tmp/myservice.sock 1942902 * 0
Run the client and the socket will activate the service.
$ python client
Check the service
$ systemctl status myservice.service
● myservice.service - myservice daemon
Loaded: loaded (/lib/systemd/system/myservice.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2015-12-01 17:22:24 CET; 1s ago
Main PID: 31190 (python)
CGroup: /system.slice/myservice.service
└─31190 /usr/bin/python -u /opt/myservice/server.py
Service is active and the server print on standard output, with systemd you can use journalctl
$ journalctl -u myservice
systemd[1]: Started myservice daemon.
systemd[1]: Starting myservice daemon...
python[31190]: Traceback (most recent call last):
python[31190]: File "/opt/myservice/server.py", line 14, in <module>
python[31190]: server = SocketServer.UnixStreamServer(socket_file_path, MyRequestHandler)
python[31190]: File "/usr/lib/python2.7/SocketServer.py", line 420, in __init__
python[31190]: self.server_bind()
python[31190]: File "/usr/lib/python2.7/SocketServer.py", line 434, in server_bind
python[31190]: self.socket.bind(self.server_address)
python[31190]: File "/usr/lib/python2.7/socket.py", line 228, in meth
python[31190]: return getattr(self._sock,name)(*args)
python[31190]: socket.error: [Errno 98] Address already in use
systemd[1]: myservice.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit myservice.service entered failed state.
systemd[1]: myservice.service failed.
Ooooops! I forget something. The socket is opened by systemd and in state LISTEN, so in code I try to create the socket, not reuse it causing an exception with server_bind.
So, how a socket works ?
Before sending data between them, server and client must initiate a connection. When we run the server.py without systemd, the server create, bind, listen and accept the socket. But with systemd the server must change, it will only accept the connection and let systemd create, bind and listen the socket.
We must change the socket management in server.
Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd’s native socket passing interface (see sd_listen_fds(3) for details) or via the traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).
OK, we will use systemd’s native socket :) Take a look at sd_listen_fds
sd_listen_fds() may be invoked by a daemon to check for file descriptors passed by the service manager as part of the socket-based activation logic. It returns the number of received file descriptors. If no file descriptors have been received, zero is returned. The first file descriptor may be found at file descriptor number 3 (i.e. SD_LISTEN_FDS_START), the remaining descriptors follow at 4, 5, 6, …, if any.
Oh, what is a “file descriptor” and why starts from 3 ?
What is a File descriptor
In Unix and related computer operating systems, a file descriptor (FD, less frequently fildes) is an abstract indicator used to access a file or other input/output resource, such as a pipe or network connection. File descriptors form part of the POSIX application programming interface. A file descriptor is a non-negative integer, represented in C programming language as the type int. … On Linux, the set of file descriptors open in a process can be accessed under the path /proc/PID/fd/, where PID is the process identifier.
And we can learn that 0 is for stdin, 1 stdout, 2 stderr and they are reserved. So we understand why we should start at 3.
If each socket starts from 3, how can I identify it ?
Linux Kernel is awesome. Each process is ‘isolated’ as you can see in /proc/PID/fd
For example, take a look, at the docker service who use socket activation
$ fuser -v -n tcp 2375
USER PID ACCESS COMMAND
2375/tcp: root 1 F.... systemd
root 19971 F.... docker
And fd folder
$ ls -la /proc/19971/fd
0 -> /dev/null
1 -> socket:[227723125]
2 -> socket:[227723125]
3 -> socket:[227723058]
4 -> socket:[227748996]
5 -> socket:[227749933]
6 -> anon_inode:[eventpoll]
7 -> socket:[227749029]
8 -> /dev/mapper/control
9 -> socket:[227723058]
10 -> socket:[227748996]
11 -> /dev/urandom
12 -> /var/lib/docker/linkgraph.db
13 -> socket:[227724910]
14 -> /var/lib/docker/linkgraph.db
15 -> socket:[227724912]
The number in bracket is the inode
$ ss -x -a | grep -i docker
u_str LISTEN 0 128 /var/run/docker.sock 227723058 * 0
* 0
We see that the /var/run/docker.sock is on fd 3 and 9 with inode 227723058.
Reuse a socket with file descriptor
When systemd runs a service, it sends 2 environment variables:
- LISTEN_FDS: number of file descriptor launched by systemd
- LISTEN_PID: process ID
We just look at the LISTEN_FDS variable, if different from 0, we will use socket.fromfd and then override the UnixStreamServer.server_bind() function.
#!/bin/python
# -*- coding: utf-8 -*-
import os
import socket
import SocketServer
socket_file_path = "/tmp/myservice.sock"
LISTEN_FDS = int(os.environ.get("LISTEN_FDS", 0))
LISTEN_PID = os.environ.get("LISTEN_PID", None) or os.getpid()
class MyRequestHandler(SocketServer.BaseRequestHandler):
"""Handle request from client"""
def handle(self):
data = self.request.recv(1024).strip()
print(str(data))
class UnixServer(SocketServer.UnixStreamServer):
def server_bind(self):
print("LISTEN_FDS: "+str(LISTEN_FDS))
print("LISTEN_PID: "+str(LISTEN_PID))
if LISTEN_FDS == 0:
print("create new socket")
SocketServer.UnixStreamServer.server_bind(self)
else:
print("rebind socket")
print("address_family: "+str(self.address_family))
print("socket_type: "+str(self.socket_type))
self.socket = socket.fromfd(3, self.address_family, self.socket_type)
server = UnixServer(socket_file_path, MyRequestHandler)
try:
print("start server!")
server.serve_forever()
except KeyboardInterrupt:
print("googbye!")
server.shutdown()
os.remove(socket_file_path)
Stop services and start socket service.
$ systemctl stop myservice.socket
$ systemctl stop myservice.service
$ systemctl start myservice.socket
Run client.py
$ python client.py
Look at journal
$ journalctl -u myservice
systemd[1]: Started myservice daemon.
systemd[1]: Starting myservice daemon...
python[26515]: LISTEN_FDS: 1
python[26515]: LISTEN_PID: 26515
python[26515]: rebind socket
python[26515]: address_family: 1
python[26515]: socket_type: 1
python[26515]: start server!
python[26515]: hello world!
We can see the “hello world!”.
If you wan’t to see in live data comming into the socket, you can install socat, thanks to @Valor
$ mv /tmp/myservice.sock /tmp/myservice.sock.original
$ socat -t100 -x -v UNIX-LISTEN:/tmp/myservice.sock,mode=777,reuseaddr,fork UNIX-CONNECT:/tmp/myservice.sock.original
When running client you can see
> 2015/12/01 21:14:35.553534 length=12 from=0 to=11
68 65 6c 6c 6f 20 77 6f 72 6c 64 21 hello world!
--
Data is clear in socket, so now Let’s Encrypt!
SSL socket
Generate certificates
mkdir /opt/myservice/ssl
cd /opt/myservice/ssl
openssl genrsa -des3 -out server.orig.key 2048
openssl rsa -in server.orig.key -out server.key
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Python have ssl lib and wrap function to wrap any socket into secure socket.
And on server, the get_request() will be overrided to return the secured socket.
#!/bin/python
# -*- coding: utf-8 -*-
import os
import ssl
import socket
import SocketServer
socket_file_path = "/tmp/myservice.sock"
LISTEN_FDS = int(os.environ.get("LISTEN_FDS", 0))
LISTEN_PID = os.environ.get("LISTEN_PID", None) or os.getpid()
SSL_CERT_PERM = "/opt/myservice/ssl/server.crt"
SSL_KEY_PERM = "/opt/myservice/ssl/server.key"
class MyRequestHandler(SocketServer.BaseRequestHandler):
"""Handle request from client"""
def handle(self):
data = self.request.recv(1024).strip()
print(str(data))
class UnixServer(SocketServer.UnixStreamServer):
def server_bind(self):
print("LISTEN_FDS: "+str(LISTEN_FDS))
print("LISTEN_PID: "+str(LISTEN_PID))
if LISTEN_FDS == 0:
print("create new socket")
SocketServer.UnixStreamServer.server_bind(self)
else:
print("rebind socket")
print("address_family: "+str(self.address_family))
print("socket_type: "+str(self.socket_type))
sock = socket.fromfd(3, self.address_family, self.socket_type)
# //stackoverflow.com/a/16740536
self.socket = socket.socket(_sock=sock)
def get_request(self):
newsocket, fromaddr = self.socket.accept()
connstream = ssl.wrap_socket(newsocket,
server_side=True,
certfile = SSL_CERT_PERM,
keyfile = SSL_KEY_PERM,
ssl_version = ssl.PROTOCOL_TLSv1,
do_handshake_on_connect=False)
return connstream, fromaddr
server = UnixServer(socket_file_path, MyRequestHandler)
try:
print("start server!")
server.serve_forever()
except KeyboardInterrupt:
print("googbye!")
server.shutdown()
os.remove(socket_file_path)
And update client.py to use secure socket
# -*- coding: utf-8 -*-
import ssl
import socket
SSL_CERT_PERM = "/opt/myservice/ssl/server.crt"
socket_file_path = "/tmp/myservice.sock"
sock = socket.socket(socket.AF_UNIX,socket.SOCK_STREAM)
s = ssl.wrap_socket(sock,
ca_certs=SSL_CERT_PERM,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1)
s.connect(socket_file_path)
s.send("hello world!")
s.close()
Stop services and start socket service.
$ systemctl stop myservice.socket
$ systemctl stop myservice.service
$ systemctl start myservice.socket
Run client.py
$ python client.py
Look at journal
$ journalctl -u myservice
...
python[29671]: hello world!
We can see the “hello world!”.
To check if the data is encrypted, use the previous socat command.
$ mv /tmp/myservice.sock /tmp/myservice.sock.original
$ socat -t100 -x -v UNIX-LISTEN:/tmp/myservice.sock,mode=777,reuseaddr,fork UNIX-CONNECT:/tmp/myservice.sock.original
When running client you can see
> 2015/12/01 21:23:11.685177 length=179 from=0 to=178
16 03 01 00 ae 01 00 00 aa 03 01 99 67 81 fc b2 ............g...
7e 7a b4 0c da f2 93 f4 77 02 88 f3 fd 08 2f 63 ~z......w...../c
8d ad 6b b9 02 e7 90 c1 b7 5f 9a 00 00 38 c0 14 ..k......_...8..
c0 0a ..
c0 0f c0 05 00 39 00 38 c0 13 c0 09 c0 0e c0 04 .....9.8........
00 33 00 32 c0 12 c0 08 c0 0d c0 03 00 88 00 87 .3.2............
00 16 00 13 00 45 00 44 00 35 00 2f 00 84 00 0a .....E.D.5./....
00 41 00 ff 01 00 00 49 00 0b 00 04 03 00 01 02 .A.....I........
...
Data is ecrypted \o/
Conclusion
Systemd is not so hard to understand if you take the time to understand how Linux works. …
Picture by Andreas Hopf under licence CC BY-NC 2.0