Adding self-written TLS to a self-written web server

Run the project first

If you like to see the result before reading the explanation, run the one-command demo from the project root:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -e ".[dev]"
python3 scripts/demo.py

The plain HTTP path and the HTTP-over-self-written-TLS path should both print HTTP/1.1 200 OK.

Minimum concepts before reading

You do not need to read the previous series first, but these concepts make the walkthrough much easier.

Wrong mental model vs right mental model

TCP bytes ──► HTTP parser
              tries to handle TLS too

TCP bytes ──► TLS layer ──► HTTP parser
           encrypted       plaintext

NGINX / OpenSSL analogy

Production NGINX normally relies on a TLS library such as OpenSSL. OpenSSL handles the secure connection state; NGINX then processes plaintext HTTP. This project uses the same boundary with smaller educational pieces.

Production-ish:
TCP → OpenSSL/TLS state → plaintext HTTP → NGINX HTTP processing

This project:
TCP → ToyTLSServerConnection → plaintext HTTP → HTTPParser / route matcher

1. The one idea to keep in mind

A web server normally receives bytes from a TCP socket and gives those bytes to an HTTP parser.

After TLS is added, the bytes from the TCP socket are no longer HTTP. They are TLS records. The server must first decrypt those records. Only then can the HTTP parser run.

Plain HTTP:
client HTTP bytes ──TCP──► server HTTP parser

HTTP over self-written TLS:
client HTTP bytes
  ↓ encrypted by client TLS layer
TLS record bytes ──TCP──► server TLS layer
  ↓ decrypted by server TLS layer
plaintext HTTP bytes ───► server HTTP parser

2. First recap: what the web server already did

The final web-server project had no TLS. It was a non-blocking HTTP file server.

Plain HTTP communication

Client                                      Server
  │                                           │
  │ TCP connect                              │ accept connection
  │                                           │
  ├── GET / HTTP/1.1 ───────────────────────►│ read bytes from socket
  │   Host: localhost                        │ append bytes to DataProvider
  │                                           │ parse HTTP request
  │                                           │ choose matching location /
  │                                           │ read html/index.html
  │                                           │ build HTTP response
  │                                           │
  │◄──────────────────── HTTP/1.1 200 OK ────┤ write response bytes
  │                         body...

The important part is the buffer:

socket.recv(...) → DataProvider → HTTPParser.parse_message(...)

TCP is a stream. One read can contain half a request, exactly one request, or multiple requests. That is why the parser returns both the parsed message and the number of bytes it consumed.

3. Second recap: what the TLS project already did

The final TLS project created an authenticated encrypted channel between a client and a server. The workflow has three phases.

Phase 0 — before the connection

Before any TCP connection exists, both sides already have some long-term information.

Phase 1 — handshake

The handshake is where the client and server agree on fresh session keys and the client verifies that it is talking to the real server.

Step 1: TCP connection

Client                                      Server
  │                                           │
  │ TCP connect                              │ accept TCP connection
  │                                           │

Nothing is encrypted yet. TCP only gives both sides a byte stream.

Step 2: ClientHello

The client creates a fresh ephemeral X25519 key pair. It keeps the private key in memory and sends only the public key.

Client                                      Server
  │                                           │
  │ generate client_ephemeral_private         │
  │ derive client_ephemeral_public            │
  │                                           │
  ├── ClientHello { client_eph_pub } ───────► │ receives client's public key

Step 3: ServerHello and ServerAuth

The server receives the client's public key. Then it creates its own fresh ephemeral X25519 key pair.

The server sends two messages back:

1. ServerHello — contains the server's ephemeral public key.
2. ServerAuth — contains the server certificate chain and a signature.

Client                                      Server
  │                                           │ receives client_eph_pub
  │                                           │ generate server_ephemeral_private
  │                                           │ derive server_ephemeral_public
  │                                           │
  │ ◄──────── ServerHello { server_eph_pub } ─┤
  │ ◄──────── ServerAuth {                    │
  │              server certificate,          │
  │              intermediate certificate,    │
  │              signature                    │
  │          } ───────────────────────────────┤

The signature is the bridge between identity and this connection:

signature = sign_with_server_identity_private_key(
    client_ephemeral_public || server_ephemeral_public
)

This says: “The server that owns the certificate private key also saw these exact ephemeral keys.”

Step 4: The server computes its session keys

The server can already compute the shared secret:

shared_secret = X25519(
    server_ephemeral_private,
    client_ephemeral_public
)

Then HKDF derives two directional keys:

• client_write_key — records sent by the client, decrypted by the server.
• server_write_key — records sent by the server, decrypted by the client.

Step 5: The client verifies the server

The client does not trust the server just because it sent a certificate. It checks two things.

Step 6: The client computes the same session keys

If verification succeeds, the client computes the same shared secret:

shared_secret = X25519(
    client_ephemeral_private,
    server_ephemeral_public
)

Because of Diffie-Hellman, this produces the same bytes that the server computed.

What each side can compute

Client side state after the handshake

• server certificate chain was verified,
• CertificateVerify signature was verified,
• client_write_key is ready for encrypting client-to-server records,
• server_write_key is ready for decrypting server-to-client records,
• send and receive sequence numbers start at 0.

Server side state after the handshake

• client ephemeral public key was received,
• server ephemeral key was generated for this connection,
• ServerHello and ServerAuth were sent,
• client_write_key is ready for decrypting client-to-server records,
• server_write_key is ready for encrypting server-to-client records,
• send and receive sequence numbers start at 0.

Now HTTP can start

After the handshake, the TLS layer is ready. But HTTP has not happened yet. The next bytes sent by the client will be an HTTP request, encrypted inside an application record.

Phase 2 — encrypted HTTP

Now the client builds an ordinary HTTP request:

GET / HTTP/1.1\r\n
Host: localhost\r\n
Connection: close\r\n
\r\n

But it does not send those bytes directly over TCP. It encrypts them first.

Client                                      Server
  │                                           │
  │ plaintext HTTP request                    │
  │ encrypt with client_write_key, seq=0      │
  │                                           │
  ├── AES-GCM record { encrypted HTTP } ────► │ decrypt with client_write_key, seq=0
  │                                           │ recover plaintext HTTP request
  │                                           │ append to HTTP input buffer
  │                                           │ HTTPParser.parse_message(...)

The server then builds a normal plaintext HTTP response and encrypts it with the server-to-client key.

Client                                      Server
  │                                           │ HTTP/1.1 200 OK + body
  │                                           │ encrypt with server_write_key, seq=0
  │                                           │
  │ ◄──── AES-GCM record { encrypted HTTP } ──┤
  │ decrypt with server_write_key, seq=0       │
  │ recover plaintext HTTP response           │

4. What changes when TLS is inserted into the web server?

The HTTP parser does not change. The route matcher does not change. File serving does not change.

What changes is the connection layer. Each accepted connection gets a context object:

class ConnectionContext:
    def __init__(self, sock, server_block, tls=None):
        self.sock = sock
        self.server_block = server_block
        self.tls = tls
        self.http_input = DataProvider()
        self.output_buffer = bytearray()
        self.closing_after_write = False

The meaning of tls is simple:

5. The read path in the integrated server

Plain HTTP listener

data = sock.recv(4096)
context.http_input.append(data)
_handle_http_messages(context)

This is the same path as the original web server.

TLS listener

data = sock.recv(4096)
context.tls.feed_wire_data(data)
_queue_output(context, context.tls.pop_pending_wire_data())
plaintext = context.tls.read_plaintext()

if plaintext:
    context.http_input.append(plaintext)
    _handle_http_messages(context)

The important difference is that data is not directly given to the HTTP parser. Only plaintext is.

6. Why the TLS demo had to become a state machine

The standalone TLS series could be written as a straight-line program:

recv ClientHello
send ServerHello
send ServerAuth
recv encrypted request
send encrypted response

That is easy to read, but it blocks. A non-blocking web server cannot stop the whole event loop while one client is halfway through a handshake.

So the integrated TLS server keeps state:

WAIT_CLIENT_HELLO
  feed TCP chunks into RecordBuffer
  if a complete ClientHello exists:
      process it
      queue ServerHello
      queue ServerAuth
      derive keys
      move to ESTABLISHED

ESTABLISHED
  each incoming record is application data
  decrypt into plaintext HTTP bytes
  each outgoing HTTP response is encrypted into a record

This is why the TLS connection object has this API:

tls.feed_wire_data(data_from_socket)
bytes_to_send = tls.pop_pending_wire_data()
http_bytes = tls.read_plaintext()
record_bytes = tls.protect_application_data(http_response)

7. The full integrated message flow

Client                                      Server
  │                                           │
  │ TCP connect to TLS port                   │ accept socket
  │                                           │ create ConnectionContext(tls=ToyTLSServerConnection)
  │                                           │
  │ generate client ephemeral keypair         │
  ├── ClientHello { client_eph_pub } ───────► │ feed_wire_data(...)
  │                                           │ parse ClientHello
  │                                           │ generate server ephemeral keypair
  │                                           │ sign client_eph_pub || server_eph_pub
  │                                           │ derive session keys
  │                                           │ queue handshake responses
  │                                           │
  │ ◄──────── ServerHello { server_eph_pub } ─┤ pop_pending_wire_data()
  │ ◄──────── ServerAuth { certs, sig } ──────┤
  │                                           │
  │ verify certificate chain                  │
  │ verify signature                          │
  │ derive same session keys                  │
  │                                           │
  │ build plaintext HTTP request              │
  │ encrypt with client_write_key             │
  ├── encrypted application record ─────────► │ decrypt with client_write_key
  │                                           │ read_plaintext() gives HTTP bytes
  │                                           │ HTTPParser.parse_message(...)
  │                                           │ read html/index.html
  │                                           │ build plaintext HTTP response
  │                                           │ encrypt with server_write_key
  │ ◄──────── encrypted application record ───┤
  │ decrypt with server_write_key             │
  │ print HTTP response                       │

8. The configuration that chooses plain vs TLS

http {
    server {
        listen 8080;
        listen 8443 ssl;
        server_name localhost;

        ssl_certificate certs/server_cert.pem;
        ssl_certificate_key certs/server_key.pem;
        ssl_certificate_chain certs/intermediate_cert.pem;

        location / {
            root html;
        }
    }
}

listen 8080; creates a plain listener. listen 8443 ssl; creates a listener that attaches TLS state to accepted connections.

9. What is on the wire?

This project uses a simplified educational wire format from the TLS series. It is not real TLS 1.3.

10. Files to read in order

1. server_side.py — prepares the demo site and starts the server.
2. client_side.py — shows plain HTTP and self-written TLS client workflows.
3. tls_web_server/tls/connection.py — the explicit client/server TLS state machines.
4. tls_web_server/server.py — the event loop where plain and TLS connections split.
5. tls_web_server/http.py — HTTP parser and file response code reused by both paths.
6. tls_web_server/config.py — config parser for listen 8443 ssl.

11. How to run it

Two-terminal workflow

# Terminal 1
python3 server_side.py

# Terminal 2
python3 client_side.py

One-command workflow

python3 scripts/demo.py

Tests

python3 -m pytest -q

12. What this project proves

• TLS belongs below HTTP, not inside the HTTP parser.
• The same HTTP parser can serve plain and encrypted connections.
• A blocking TLS demo can be refactored into a buffer-oriented state machine.
• After the handshake, encrypted application records become normal HTTP bytes.
• The server can choose plain or TLS behavior from an NGINX-style config.

13. What this project does not prove

To work with a browser, the project would need real TLS 1.3 records, real ClientHello parsing, real ServerHello, transcript hashes, encrypted handshake messages, Finished messages, SNI, ALPN, alerts, and a browser-trusted certificate.

14. Final mental model

Before TLS:
TCP bytes are HTTP bytes.

After TLS:
TCP bytes are TLS records.
TLS records decrypt into HTTP bytes.
HTTP parser sees only HTTP bytes.

That is the whole integration boundary.