OpenPGP Web Key Directory

Reading time ~16 minutes

OpenPGP Web Key Directory

OpenPGP Web Key Directory

1. Introduction

If you use OpenPGP/GnuPG to secure your email communication, you know that sharing your public key with your contacts is an important step of the process. Traditionally it was done either by sending it as an attachment on the first email, or by publishing it on the so-called "keyserver network", where your contacts can find and fetch it. Both of these methods have their own problems and are not recommended.

The recommended way for publishing your public key is through Web Key Directory (WKD), where mail clients will be able to locate and fetch it automatically. This means that once you publish your key on the WKD, your contacts will be able to:

  • Verify your signed emails automatically, because their mail clients can find and fetch your public key automatically.
  • Send you immediately encrypted messages, without asking for your public key first, because their mail clients can find and fetch it automatically.

The ability to fetch automatically public keys from a WKD is already supported by almost all the mail clients (Thunderbird, KMail, Outlook, etc.) All you need to do is to publish your public key to a WKD server. Let's see how to build such a server and how to publish your key.

2. How WKD works

Before building a WKD server, let's try to understand first how it works by following the steps that a mail client would do to locate the public key corresponding to a certain email address. Let's say that the email address is <user@example.org>.

A mail client will check first whether a WKD for the domain example.org exists. It does it by checking first for the presence of:

https://openpgpkey.example.org/
        .well-known/openpgpkey/example.org/policy

If it fails, then it checks for the presence of:

https://example.org/.well-known/openpgpkey/policy

The first one is called the advanced method, and the fallback one is called the direct method. The file policy is usually just an empty file.

Then, depending on which WKD method exists for example.org, it tries to download the public key from one of these locations (lines are broken for rendering purposes):

https://openpgpkey.example.org/.well-known/openpgpkey/example.org/
                          hu/nmxk159crbcuk3imqiw13gkjmfwd8mqj?l=user

https://example.org/.well-known/openpgpkey/
                          hu/nmxk159crbcuk3imqiw13gkjmfwd8mqj?l=user

The directory hu stands for hashed-userid, and nmxk159crbcuk3imqiw13gkjmfwd8mqj is indeed a kind of hash of user, the local-part of the email address. For the time being we don't need to know how the WKD client finds or generates this hash.

3. Building a WKD

First, let's build a WKD with the direct method, since it is a bit simpler than the advanced method.

This requires that we have access to the webserver of the domain example.org, and that the server supports HTTPS.

Then we will see also the advanced method, which additionally requires access to the DNS server of the domain example.org.

3.1. Create the directory of public keys

If the DocumentRoot is /var/www/html/ we can build the web key directory like this:

mkdir -p /var/www/html/.well-known/openpgpkey/
touch /var/www/html/.well-known/openpgpkey/policy

Now we just need to export the public key, transfer it to the web server, and save it at the file:

.well-known/openpgpkey/hu/nmxk159crbcuk3imqiw13gkjmfwd8mqj

The name of the file nmxk159crbcuk3imqiw13gkjmfwd8mqj is the hashed-userid that we saw on the previous section. We can get it from a gpg command like this:

$ gpg --with-wkd-hash --fingerprint user@example.org

pub   rsa3072 2021-04-22 [SC] [expires: 2023-04-22]
      901D C530 A8E1 DEBA FED0  6C25 C802 3646 1A8D CFC2
uid           [ultimate] user@example.org
              nmxk159crbcuk3imqiw13gkjmfwd8mqj@example.org
sub   rsa3072 2021-04-22 [E]

To export the public key we can use a command like this:

gpg --no-armor --export \
    user@example.org > nmxk159crbcuk3imqiw13gkjmfwd8mqj

The option --no-armor makes sure to export it in binary format (not ASCII armored), as required by the WKD specs.

The same way we can publish the public keys for more email addresses, like user1@example.org, user2@example.org, etc. At the end, the directory should look like this:

/var/www/html/.well-known/ 
└── openpgpkey
    ├── hu
    │   ├── nmxk159crbcuk3imqiw13gkjmfwd8mqj
    │   ├── sxpkq64cy1wikgh8o8eddrx6bg8urzu8
    │   └── wgrbabzq3fs5uryhxq96e8nnwxae78fw
    └── policy

3.2. Webserver configuration

The web server of the WKD:

  • Should disable directory listing for the hu/ directory.
  • Should have the right CORS headers.
  • Should use application/octet-stream as the Content-Type for the data of public keys.

With apache2 the configuration should look like this:

<Directory "/.well-known/openpgpkey/hu">
    Options -Indexes
    ForceType application/octet-stream
    Header always set Access-Control-Allow-Origin "*"
</Directory>

This requires that the apache2 modules mime and headers are enabled:

a2enmod mime headers

With nginx the configuration should look like this:

location /.well-known/openpgpkey/hu/ {
    autoindex off;
    default_type  "application/octet-stream";
    add_header    Access-Control-Allow-Origin * always;
}

3.3. Test the WKD

We can use wget or curl to download a published key from the WKD, just for testing. Let's also use gpg-wks-client in order to find out the hash of the userid. It can be installed like this:

apt install gpg-wks-client
alias gpg-wks-client='/usr/lib/gnupg/gpg-wks-client -v'
gpg-wks-client --help

We can find the hash of user@example.org like this:

$ gpg-wks-client --print-wkd-hash user@example.org

nmxk159crbcuk3imqiw13gkjmfwd8mqj user@example.org

With this hash we can construct the URL manually and download the public key:

wget -q -O user-key.pub \
     https://example.org/.well-known/openpgpkey/hu/
                          nmxk159crbcuk3imqiw13gkjmfwd8mqj?l=user

Other ways for testing that the published key is accessible via WKD are these:

3.4. Publish the keys of an organization

We have seen already how to publish keys manually one by one, but as the number of keys to be published gets large, it becomes tedious and error-prone to manage them. However it is possible to publish them in bulk.

Let's say that we have in our GnuPG keyring the public keys of the members of an organization (for example they were sent to us by attachment, and we imported them). We can export all these keys into a WKD format like this:

mkdir wkd
gpg --list-options show-only-fpr-mbox \
    --list-keys "@example.org" \
    | gpg-wks-client --install-key --directory wkd/

The GnuPG keyring is searched for all public keys (--list-keys) matching the defined pattern (@example.org), and the output is generated as fingerprint user_id values that look like this:

901DC530A8E1DEBAFED06C25C80236461A8DCFC2 user@example.org
D67E52B28F276D2AE5250B4EFF09740FC5FD1300 user1@example.org
38186EF3FDA463907B494A4AEDF1E29CB878858F user2@example.org

With this as input, gpg-wks-client creates a WKD directory structure that looks like this:

$ tree wkd/

wkd/
└── example.org
    ├── hu
    │   ├── nmxk159crbcuk3imqiw13gkjmfwd8mqj
    │   ├── sxpkq64cy1wikgh8o8eddrx6bg8urzu8
    │   └── wgrbabzq3fs5uryhxq96e8nnwxae78fw
    └── policy

Now we can just use rsync to synchronize the directory hu/ with the one on the webserver, for example:

rsync -a --delete \
    ./wkd/example.org/hu/ \
    webserver:/var/www/html/.well-known/openpgpkey/hu/

3.5. The advanced method of WKD

With the advanced method, the key is published in a location like: https://openpgpkey.example.org/.well-known/openpgpkey/example.org/hu/, instead of https://example.org/.well-known/openpgpkey/hu/, which is the location for the direct method. Everything else is the same.

Notice that this requires the subdomain openpgpkey.example.org to be resolvable. On the other hand, if we want to use the direct method, we should make sure that this subdomain is not resolvable, since WKD clients will first try the advanced method, and only if the openpgpkey subdomain is not resolvable will fall back to the direct method.

While the direct method is simpler because it does not need any DNS modifications, the advanced method is more flexible because:

  • It allows us to use a WKD server that is different from the webserver.
  • It allows us to support more than one email domain in the same WKD server.

4. WKD server with docker-scripts

We can install a WKD server with Docker and docker-scripts. This container actually supports a WKS (Web Key Service) server as well, but we will see this later.

This container supports the advanced method of WKD, so we need to define a DNS record for the subdomain openpgpkey.example.org, which resolves to the IP of the host where this container is installed.

4.1. Install dependencies

This container depends on docker, docker-scripts, and wsproxy, which is a container that acts as a reverse proxy for the HTTPS requests.

  1. Install docker:

    curl -fsSL https://get.docker.com -o get-docker.sh
    sudo sh get-docker.sh
    
  2. Install docker-scripts:

    apt install m4 make
    git clone https://gitlab.com/docker-scripts/ds \
              /opt/docker-scripts/ds
    cd /opt/docker-scripts/ds/
    make install
    
  3. Install WebServer Proxy:

    ds pull wsproxy
    ds init wsproxy @wsproxy
    cd /var/ds/wsproxy/
    vim settings.sh
    ds make
    

4.2. Install the WebKey container

  1. Add records like these on the DNS configuration of the domain example.org:

    webkey.example.org.      IN  A      10.11.12.13
    openpgpkey.example.org.  IN  CNAME  webkey.example.org.
    

    The subdomain webkey can be named to anything, but the subdomain openpgpkey is not up to us because it is part of the WKD specification.

  2. Get the scripts:

    ds pull webkey
    
  3. Create a directory for the container:

    ds init webkey @webkey.example.org
    
  4. Fix the settings:

    cd /var/ds/webkey.example.org/
    vim settings.sh
    

    We will see later that this container can be used both as a WKD and a WKS (Web Key Service) server. For the time being we are using it only as a WKD server, so we can comment out the settings PORTS and ALLOWED_NETWORKS (although it doesn't harm if we don't comment them).

    As HOSTNAME set webkey.example.org and as WEBKEY_DOMAINS set example.org (without the prefix openpgpkey, it will be added automatically by the configuration scripts).

  5. Make the container:

    ds make
    

4.3. Checking and testing

After installation, let's check some configurations:

  • On /var/ds/webkey.example.org/ there should be the subdirectory wkd/. (There is also the subdirectory wks/ but we can safely ignore it for now).
  • On wsproxy, the configuration file /var/ds/wsproxy/sites-enabled/webkey.example.org.conf should contain ServerName and ServerAlias like this:

    ServerName webkey.example.org
    ServerAlias openpgpkey.example.org
    
  • The configuration of apache2 inside the container should look like this:

    $ ds exec cat /etc/apache2/conf-enabled/wkd.conf
    
    Alias /.well-known/openpgpkey /host/wkd
    <Directory /host/wkd>
        Require all granted
        Options -Indexes
    </Directory>
    <Directory ~ "/host/wkd/.*/hu">
        ForceType application/octet-stream
        Header always set Access-Control-Allow-Origin "*"
    </Directory>
    

If we add keys to the directory wkd/example.org/hu/ they will be available immediately for being accessed through WKD. Initially only the public key for the email address keys@example.org is published (which is used for the WKS, as we will see later). We can use this published key to test whether WKD works correctly:

  1. Install gpg-wks-client:

    apt install gpg-wks-client
    alias gpg-wks-client='/usr/lib/gnupg/gpg-wks-client -v'
    gpg-wks-client --help
    
  2. Check that the public key of keys@example.org is published on WKD:

    $ gpg-wks-client -v --check keys@example.org
    
    gpg-wks-client: public key for 'keys@example.org' found via WKD
    gpg-wks-client: gpg: Total number processed: 1
    gpg-wks-client: fingerprint: B3883F4D3D926E19DAD558D968BA810F826B7DF7
    gpg-wks-client:     user-id: keys@example.org
    gpg-wks-client:     created: Tue 27 Apr 2021 07:03:45 PM CEST
    gpg-wks-client:   addr-spec: keys@example.org
    

    Alternatively, check it using a WKD validator like this one: https://metacode.biz/openpgp/web-key-directory

  3. Get the key and import it on the GnuPG keyring:

    $ gpg-wks-client --print-wkd-url keys@example.org
    
    https://openpgpkey.example.org/.well-known/openpgpkey/
    example.org/hu/mfnkhjkbtuim3hwf51o7qepzxnrpi1r5?l=keys
    
    $ wget -qO- $(gpg-wks-client --print-wkd-url keys@example.org) \
        | gpg --import
    
    gpg: key 68BA810F826B7DF7: public key "keys@example" imported
    gpg: Total number processed: 1
    gpg:               imported: 1
    

    Alternatively, we can get it with gpg --locate-keys:

    $ gpg --delete-keys keys@example.org
    $ gpg --list-keys keys@example.org
    
    $ gpg -v --locate-keys keys@example.org
    
    gpg: using pgp trust model
    gpg: error retrieving 'keys@example.org' via Local: No public key
    gpg: pub  rsa3072/68BA810F826B7DF7 2021-04-27  keys@example.org
    gpg: key 68BA810F826B7DF7: public key "keys@example.org" imported
    gpg: Total number processed: 1
    gpg:               imported: 1
    gpg: auto-key-locate found fingerprint B3883F4D3D926E19DAD558D968BA810F826B7DF7
    gpg: automatically retrieved 'keys@example.org' via WKD
    pub   rsa3072 2021-04-27 [SC] [expires: 2023-04-27]
          B3883F4D3D926E19DAD558D968BA810F826B7DF7
    uid           [ unknown] keys@example.org
    sub   rsa3072 2021-04-27 [E]
    

    Notice the message: automatically retrieved 'keys@example.org' via WKD.

5. Web Key Service (WKS)

What is a WKS? So far we have seen how to publish keys manually (one by one or in bulk). For a large organization it is more suitable if each member can publish and update his key himself. One way to implement this is with a web interface, where the members can upload their public key, after authenticating themselves somehow. Another way is to send the public key to the WKD by email.

WKS allows users to publish their public key by email, through a specific email protocol.

5.1. How WKS works

To understand how WKS works, let's see how a key is published through it.

First of all, the user (or his mail client) needs to know the email address where he can send the key. By the WKS specification, this email address can be found on the file submission-address on this URL (the line is broken for rendering purposes):

https://openpgpkey.example.org/.well-known/openpgpkey/example.org/
                                                  submission-address

This file contains a single line with an email address like this:

keys@example.org

Then, to publish the key, these steps are followed:

  1. The user (or his mail client) sends by attachment to the submission address the public key corresponding to his email address. This is the key publication request.
  2. The WKS replies with an encrypted message that contains a nonce and the fingerprint of the key. The message is encrypted with the public key that is received by attachment (that is to be published on the WKD). This is the request verification step.
  3. The user decrypts the confirmation message with his private key and sends back the nonce. This reply message should be encrypted with the public key of the submission address, which can be retrieved from the WKD. This is the request confirmation step.
  4. Upon receiving the nonce and checking that it is correct, the WKS publishes the key of the user to the WKD and sends a notification message to the user, informing him that the key has been published. This is the key publication step.

The steps (2) and (3) above (request verification and request confirmation) ensure that:

  1. The key submission request is not fake (it is coming from the user that owns that email address).
  2. The public key that is submitted for publication is genuine (since the user is able to decrypt a message encrypted with it).

A WKS is usually integrated with the mail server of the domain example.org. But it can also be installed on a separate server and work as an extension to the mail server.

5.2. Install the WKS container

The docker container that we installed on the section (4.) can serve both for WKD and WKS. Installation is the same, just make sure to uncomment PORTS and ALLOWED_NETWORKS on settings.sh, and then rebuild with ds make.

Make sure to add to the ALLOWED_NETWORKS the IP addresses (or networks) of the mail servers that are going to be served by this WKS container. For security reasons, it accepts SMTP connections only from the allowed networks. You will understand why this is needed on the next section, which explains how it works.

After installation, we can make a quick sanity check of the installed container like this:

cd /var/ds/webkey.example.org/
ds play tests/test1.sh

5.3. How the WKS container works

This WKS container works as an external extension to one or more existing mail servers.

  1. When the mail server receives an email for keys@example.org (which is the submission-address for the mail domain @example.org) it forwards it to keys@webkey.example.org and sends it by SMTP on the port 10025 of the WKS server (with hostname webkey.example.org). For security reasons, the WKS server will accept this SMTP connection only if the IP of the mail server is listed on its ALLOWED_NETWORKS variable.
  2. The WKS server has an alias that forwards this email to the local account webkey (which belongs to the local user webkey):

    $ ds exec postconf mydestination
    
    mydestination = webkey.example.org, localhost
    
    $ ds exec cat /etc/aliases
    
    keys webkey
    
  3. The WKS server also has enabled maildrop processing:

    $ ds exec postconf mailbox_command
    
    mailbox_command = /usr/bin/maildrop -d ${USER}
    
  4. The webkey account has a mailfilter like this:

    $ cat wks/.mailfilter
    
    logfile maildrop.log
    #cc archive/    # debug
    to "| gpg-wks-server --directory /host/wkd \
                         --receive --send &>>maildrop.log"
    

    It pipes the arriving emails to the command gpg-wks-server, which processes them and sends any replies through the mailserver.

  5. For the messages and replies from gpg-wks-server to get through, the mail server has to add the IP of the WKS server on its trusted hosts. It also has to rewrite the sender address from webkey@webkey.example.org to keys@example.org, in order to avoid any SPF failures. For more details about this check the next section.

5.4. Configuration on the mailserver

On the mail server we need to make these configurations in order to integrate it with the WKD+WKS server:

  1. Add a virtual alias from keys@example.org to keys@webkey.example.org. It might be done like this:

    postconf 'virtual_alias_maps = hash:/host/config/virtual_alias_maps'
    
    cat << EOF >> /host/config/virtual_alias_maps
    keys@example.org  keys@webkey.example.org
    EOF
    
    postmap /host/config/virtual_alias_maps
    
    postfix reload
    

    This ensures that key-submission emails and submission-confirmation emails are forwarded to the WKS server.

  2. Tell the mailserver to send (transport) the mails for the domain webkey.example.org to smtp:[webkey.example.org]:10025. It may be done like this:

    postconf 'transport_maps = hash:/host/config/transport_maps'
    
    cat << EOF > /host/config/transport_maps
    webkey.example.org smtp:[webkey.example.org]:10025
    EOF
    
    postmap /host/config/transport_maps
    
    postfix reload
    
  3. Allow the WKS server to send replies (emails) through the mailserver, without authentication, by adding its IP to the list of mynetworks. It might be done like this:

    postconf 'mynetworks = /host/config/trusted_hosts'
    
    cat << EOF >> /host/config/trusted_hosts
    12.34.56.78
    EOF
    
    postfix reload
    
  4. The envelope of the emails (replies) sent from the WKS server will have a sender address like webkey@webkey.example.org. This may confuse the SPF check of mail providers (in case the emails of the clients are forwarded to external addresses). As a result these emails may be considered less trusty and may end up as spam.

    To prevent this we should configure the mailserver to rewrite this address to keys@example.org. It might be done like this:

    postconf 'smtp_generic_maps = hash:/host/config/smtp_generic_maps'
    
    cat << EOF > /host/config/smtp_generic_maps
    webkey@webkey.example.org keys@example.org
    EOF
    
    postmap /host/config/smtp_generic_maps
    
    postfix reload
    

5.5. Publishing a key through WKS

Let's assume that we have an authenticated mail account on the mailserver, with username test1@example.org and password pass1. We want to publish the GnuPG public key for the email address test1@example.org.

We will use the command gpg-wks-client to generate the necessary emails and replies that should be sent to the WKS. But we also need a tool for sending authenticated emails from the command line, and msmtp is a good one. Let's install these dependencies first:

apt install gpg-wks-client msmtp
alias gpg-wks-client='/usr/lib/gnupg/gpg-wks-client'

Before sending the key, let's make sure that WKS is supported by our mailserver:

$ gpg-wks-client -v --supported test1@example.org

gpg-wks-client: provider for 'test1@example.org' supports WKS

Now we can follow these steps to publish the key:

  1. Send a key publishing request like this:

    gpg --list-keys test1@example.org
    
    gpg-wks-client --create \
            AB97233AD0EB0180882D1227799020EF6FF16876 \
            test1@example.org \
        | msmtp \
            --read-envelope-from --read-recipients \
            --tls=on --auth=on \
            --host=smtp.example.org --port=587 \
            --user=test1@example.org --passwordeval="echo pass1"
    

    The command gpg-wks-client --create creates a request email in the format that is required by the WKS, and msmtp sends it through the mailserver, with authentication.

  2. Soon WKS will send as followup an email with subject Confirm your key publication. Save it as a text file: Confirm-your-key-publication.eml
  3. Send the confirmation email with the command gpg-wks-client --receive, like this:

    cat Confirm-your-key-publication.eml \
        | gpg-wks-client --receive \
        | msmtp  \
            --read-envelope-from --read-recipients \
            --tls=on --auth=on \
            --host=smtp.example.org --port=587 \
            --user=test1@example.org --passwordeval="echo pass1"
    

We should receive a notification email that the key has been published. For a quick check we can use the command:

gpg-wks-client -v --check test1@example.org

Note: Key publishing process was automated by Thunderbird+Enigmail. However Thunderbird 78 dropped Enigmail. Its functionality was supposed to be merged to Thunderbird, but not all of it is already merged. In particular the features that facilitate and automate the interaction with a WKS server are missing. Let’s hope that they will be added back soon.

Date: 2021-04-22

Author: Dashamir Hoxha

Created: 2021-05-07 Fri 15:49

Validate

SMTP Server with LDAP Authentication

SMTP Server with LDAP AuthenticationSMTP Server with LDAP AuthenticationTable of Contents1. Introduction2. Send email only from trusted h...… Continue reading

Using WireGuard VPN

Published on November 09, 2020

Distributed Computer Lab

Published on November 08, 2020