\documentclass[11pt]{article} \usepackage{times} \usepackage{pl} \usepackage{html} \sloppy \makeindex \onefile \htmloutput{\Sdot} % Output directory \htmlmainfile{ssl} % Main document file \bodycolor{white} % Page colour \begin{document} \title{SWI-Prolog SSL Interface} \author{\href{https://www.metalevel.at}{Markus Triska}, Jan van der Steen, Matt Lilley and Jan Wielemaker \\[5pt] E-mail: \email{jan@swi-prolog.org} } \maketitle \begin{abstract} The SWI-Prolog SSL (Secure Socket Layer) library implements a pair of \jargon{filtered streams} that realises an SSL encrypted connection on top of a pair of Prolog \jargon{wire} streams, typically a network socket. SSL provides public key based encryption and digitally signed identity information of the \jargon{peer}. The SSL library is well integrated with SWI-Prolog's HTTP library for both implementing HTTPS servers and communicating with HTTPS servers. It is also used by the \href{http://www.swi-prolog.org/pack/list?p=smtp}{smtp pack} for accessing secure mail agents. Plain SSL can be used to realise secure connections between e.g., Prolog agents. \end{abstract} \pagebreak \tableofcontents \pagebreak \section{Introduction} \label{sec:ssl-intro} Raw TCP/IP networking is dangerous for two reasons: \begin{enumerate} \item It is hard to tell whether the party you think you are talking to is indeed the right one. \item Anyone with access to a subnet through which your data flows can `tap' the wire and listen for sensitive information such as passwords, credit card numbers, etc. \end{enumerate} Transport Layer Security~(TLS) and its predecessor Secure Socket Layer~(SSL), which are both often collectively called~SSL, solve both problems. SSL~uses: \begin{itemize} \item certificates to establish the \textit{identity} of the peer \item \textit{encryption} to make it useless to tap into the wire. \end{itemize} SSL allows agents to talk in private and create secure web services. The SWI-Prolog \pllib{ssl} library provides an API to turn a pair of arbitrary Prolog \jargon{wire} streams into SSL powered encrypted streams. Note that secure protocols such as secure HTTP simply run the plain protocol over (SSL) encrypted streams. The \pllib{crypto} library provides additional predicates related to cryptography and authentication, secure hashes and elliptic~curves. Cryptography is a difficult topic. If you just want to download documents from an HTTPS server without worrying much about security, \index{http_open/3}\predref{http_open}{3} will do the job for you. As soon as you have higher security demands we strongly recommend you to read enough background material to understand what you are doing. See \secref{ssl-security} for some remarks regarding this implementation. This \href{http://www.tldp.org/HOWTO/SSL-Certificates-HOWTO/index.html}{The Linux Documentation Project page} provides some additional background and tips for managing certificates and keys. \input{ssllib.tex} \input{crypto.tex} \section{XML cryptographic libraries} \label{sec:ssl-xml-libs} The SSL package provides several libraries dealing with cryptographic operations of XML documents. These libraries depend on the \const{sgml} package. These libraries are part of this package because the \const{sgml} package has no external dependencies and will thus be available in any SWI-Prolog installation while configuring and building this \const{ssl} package is much more involved. \input{saml.tex} \input{xmlenc.tex} \input{xmldsig.tex} \section{SSL Security} \label{sec:ssl-security} Using SSL (in this particular case based on the OpenSSL implementation) to connect to SSL services (e.g., an \verb$https://$ address) easily gives a false sense of security. This section explains some of the pitfalls.\footnote{We do not claim to be complete, just to start warning you if security is important to you. Please make sure you understand (Open)SSL before relying on it.}. As stated in the introduction, SSL aims at solving two issues: tapping information from the wire by means of encryption and make sure that you are talking to the right address. Encryption is generally well arranged as long as you ensure that the underlying SSL library has all known security patches installed and you use an encryption that is not known to be weak. The Windows version and MacOS binaries of SWI-Prolog ships with its own binary of the OpenSSL library. Ensure this is up-to-date. On systems that ship with the OpenSSL library SWI-Prolog uses the system version. This applies notably for all Linux packages. Check the origin and version of the OpenSSL libraries and verify there are no more recent security patches regularly if security is important to you. The OpenSSL library version as reported by \cfuncref{SSLeay_version}{} is available in the Prolog flag \const{ssl_library_version} as illustrated below on Ubuntu 14.04. \begin{code} ?- [library(ssl)]. ?- current_prolog_flag(ssl_library_version, X). X = 'OpenSSL 1.0.1f 6 Jan 2014'. \end{code} \noindent Whether you are talking to the right address is a complicated issue. The core of the validation is that the server provides a \jargon{certificate} that identifies the server. This certificate is digitally \jargon{signed} by another certificate, and ultimately by a \jargon{root certificate}. (There may be additional links in this chain as well, or there may just be one certificate signed by itself) Verifying the peer implies: \begin{enumerate} \item Verifying the chain or digital signatures until a trusted root certificate is found, taking care that the chain does not contain any invalid certificates, such as certificates which have expired, are not yet valid, have altered or forged signatures, are valid for the purposes of SSL (and in the case of an issuer, issuing child certificates) \item Verifying that the signer of a certificate did not \jargon{revoke} the signed certificate. \item Verifying that the host we connected to is indeed the host claimed in the certificate. \end{enumerate} The default https client plugin (\pllib{http/http_ssl_plugin}) registers the system trusted root certificate with OpenSSL. This is achieved using the option \term{cacerts}{[system(root_certificates)]} of \index{ssl_context/3}\predref{ssl_context}{3}. The verification is left to OpenSSL. To the best of our knowledge, the current (1.0) version of OpenSSL \textbf{only} implements step (1) of the verification process outlined above. This implies that an attacker that can control DNS mapping (host name to IP) or routing (IP to physical machine) can fake to be a secure host as long as they manage to obtain a certificate that is signed from a recognised authority. Version 1.0.2 supports hostname checking, and will not validate a certificate chain if the leaf certificate does not match the hostname. 'Match' here is not a simple string comparison; certificates are allowed (subject to many rules) to have wildcards in their SubjectAltName field. Care must also be taken to ensure that the name we are checking against does not contain embedded NULLs. If SWI-Prolog is compiled against a version of OpenSSL that does NOT have hostname checking (ie 1.0.0 or earlier), it will attempt to do the validation itself. This is not guaranteed to be perfect, and it only supports a small subset of allowed wildcards. If security is important, use OpenSSL 1.0.2 or higher. After validation, the predicate \index{ssl_peer_certificate/2}\predref{ssl_peer_certificate}{2} can be used to obtain the peer certificate and inspect its properties. \section{CRLs and Revocation} \label{sec:crl} Certificates must sometimes be revoked. Unfortunately this means that the elegant chain-of-trust model breaks down, since the information you need to determine whether a certificate is trustworthy no longer depends on just the certificate and whether the issuer is trustworthy, but now on a third piece of data - whether the certificate has been revoked. These are managed in two ways in OpenSSL: CRLs and OCSP. SWI-Prolog supports CRLs only. (Typically OCSP responders are configured in such a way as to just consult CRLs anyway. This gives the illusion of up-to-the-minute revocation information because OCSP is an interactive, online, real-time protocol. However the information provided can still be several \emph{weeks} out of date!) To do CRL checking, pass require_crl(true) as an option to the \index{ssl_context/3}\predref{ssl_context}{3} (or \index{http_open/3}\predref{http_open}{3}) option list. If you do this, a certificate will not be validated unless it can be \emph{checked} for on a revocation list. There are two options for this: First, you can pass a list of filenames in as the option \index{crl/1}\predref{crl}{1}. If the CRL corresponds to an issuer in the chain, and the issued certificate is not on the CRL, then it is assumed to not be revoked. Note that this does NOT prove the certificate is actually trustworthy - the CRL you pass may be out of date! This is quite awkward to get right, since you do not necessarily know in advance what the chain of certificates the other party will present are, so you cannot reasonably be expected to know which CRLs to pass in. Secondly, you can handle the CRL checking in the cert_verify_hook when the Error is bound to unknown_crl. At this point you can obtain the issuer certificate (also given in the hook), find the CRL distribution point on it (the \index{crl/1}\predref{crl}{1} argument), try downloading the CRL (the URL can have literally any protocol, most commonly HTTP and LDAP, but theoretically anything else, too, including the possibility that the certificate has no CRL distribution point given, and you are expected to obtain the CRL by email, fax, or telegraph. Therefore how to actually obtain a CRL is out of scope of this document), load it using \index{load_crl/2}\predref{load_crl}{2}, then check to see whether the certificate currently under scrutiny appears in the list of revocations. It is up to the application to determine what to do if the CRL cannot be obtained - either because the protocol to obtain it is not supported or because the place you are obtaining it from is not responding. Just because the CRL server is not responding does not mean that your certificate is safe, of course - it has been suggested that an ideal way to extend the life of a stolen certificate key would be to force a denial of service of the CRL server. \subsubsection{Disabling certificate checking} \label{sec:disable-certificate} In some cases clients are not really interested in host validation of the peer and whether or not the certificate can be trusted. In these cases the client can pass \term{cert_verify_hook}{cert_accept_any}, calling \index{cert_accept_any/5}\predref{cert_accept_any}{5} which accepts any certificate. Note that this will accept literally ANY certificate presented - including ones which have expired, have been revoked, and have forged signatures. This is probably not a good idea! \subsubsection{Establishing a safe connection} \label{sec:ssl-safe-connection} Applications that exchange sensitive data with e.g., a backend server typically need to ensure they have a secure connection to their peer. To do this, first obtain a non-secure connection to the peer (eg via a TCP socket connection). Then create an SSL context via \index{ssl_context/3}\predref{ssl_context}{3}. For the client initiating the connection, the role is 'client', and you should pass options \index{host/1}\predref{host}{1} and \index{cacerts/1}\predref{cacerts}{1} at the very least. If you expect the peer to have a certificate which would be accepted by your host system, you can pass \term{cacerts}{[system(root_certificates)]}, otherwise you will need a copy of the CA certificate which was used to sign the peer's certificate. Alternatively, you can pass \index{cert_verify_hook/1}\predref{cert_verify_hook}{1} to write your own custom validation for the peer's certificate. Depending on the requirements, you may also have to provide your /own/ certificate if the peer demands mutual authentication. This is done via the \index{certificate_file/1}\predref{certificate_file}{1}, \index{key_file/1}\predref{key_file}{1} and either \index{password/1}\predref{password}{1} or \index{pem_password_hook/1}\predref{pem_password_hook}{1}. Once you have the SSL context and the non-secure stream, you can call \index{ssl_negotiate/5}\predref{ssl_negotiate}{5} to obtain a secure stream. \index{ssl_negotiate/5}\predref{ssl_negotiate}{5} will raise an exception if there were any certificate errors that could not be resolved. The peer behaves in a symmetric fashion: First, a non-secure connection is obtained, and a context is created using \index{ssl_context/3}\predref{ssl_context}{3} with the role set to server. In the server case, you must provide \index{certificate_file/1}\predref{certificate_file}{1} and \index{key_file/1}\predref{key_file}{1}, and then either \index{password/1}\predref{password}{1} or \index{pem_password_hook/1}\predref{pem_password_hook}{1}. If you require the other party to present a certificate as well, then peer_cert(true) should be provided. If the peer does not present a certificate, or the certificate cannot be validated as trusted, the connection will be rejected. By default, revocation is not checked. To enable certificate revocation checking, pass require_crl(true) when creating the SSL context. See \secref{crl} for more information about revocations. \section{Example code} \label{sec:ssl-examples} Examples of a simple server and client (\file{server.pl} and \file{client.pl} as well as a simple HTTPS server (\file{https.pl}) can be found in the example directory which is located in \file{doc/packages/examples/ssl} relative to the SWI-Prolog installation directory. The \file{etc} directory contains example certificate files as well as a \file{README} on the creation of certificates using OpenSSL tools. \subsection{Accessing an HTTPS server} \label{sec:ssl-https-client} Accessing an \verb$https://$ server can be achieved using the code skeleton below. The line \verb$:- use_module(library(http/http_ssl_plugin)).$ can actually be omitted because the plugin is dynamically loaded by \index{http_open/3}\predref{http_open}{3} if the \const{https}~scheme is detected. See \secref{ssl-security} for more information about security aspects. \begin{code} :- use_module(library(http/http_open)). :- use_module(library(http/http_ssl_plugin)). ..., http_open(HTTPS_url, In, []), ... \end{code} \noindent \subsection{Creating an HTTPS server} \label{sec:ssl-https-server} The SWI-Prolog infrastructure provides two main ways to launch an HTTPS~server: \begin{itemize} \item Using \const{library(http/thread_httpd)}, the server is started in HTTPS~mode by adding an option~\const{\index{ssl/1}\predref{ssl}{1}} to \index{http_server/2}\predref{http_server}{2}. The argument of \const{\index{ssl/1}\predref{ssl}{1}} is an option list that is passed as the third argument to \index{ssl_context/3}\predref{ssl_context}{3}. \item Using \const{library(http/http_unix_daemon)}, an HTTPS~server is started by using the command line argument~\const{--https}. \end{itemize} Two items are typically specified as, respectively, options or additional command~line arguments: \begin{itemize} \item \textbf{server certificate}. This identifies the server and acts as a \jargon{public key} for the encryption. \item \textbf{private key} of the server, which must be kept secret. The key \textit{may} be protected by a password. If this is the case, the server must provide the password by means of the \const{password} option, the \const{pem_password_hook} callback or, in case of the Unix daemon, via the \const{--pwfile} or \const{--password} command line options. \end{itemize} Here is an example that uses the self-signed demo certificates distributed with the SSL package. As is typical for publicly accessible HTTPS~servers, this version does \textit{not} require a certificate from the~client: \begin{code} :- use_module(library(http/thread_httpd)). :- use_module(library(http/http_ssl_plugin)). https_server(Port, Options) :- http_server(reply, [ port(Port), ssl([ certificate_file('etc/server/server-cert.pem'), key_file('etc/server/server-key.pem'), password("apenoot1") ]) | Options ]). \end{code} \noindent There are two \jargon{hooks} that let you extend HTTPS servers with custom definitions: \begin{itemize} \item \texttt{http:ssl_server_create_hook(+SSL0, -SSL, +Options)}: This extensible predicate is called exactly \textit{once}, after creating an HTTPS~server with Options. If this predicate succeeds, \texttt{SSL} is the~context that is used for negotiating all new connections. Otherwise, \texttt{SSL0} is used, which is the context that was created with the given options. \item \texttt{http:ssl_server_open_client_hook(+SSL0, -SSL, +Options)}: This predicate is called before \textit{each} connection that the server negotiates with a client. If this predicate succeeds, \texttt{SSL} is the context that is used for the new~connection. Otherwise, \texttt{SSL0} is~used, which is the context that was created when launching the~server. \end{itemize} Important use cases of these hooks are running dual-stack RSA/ECDSA servers, updating certificates while the server keeps running, and tweaking SSL~parameters for connections. Use \index{ssl_set_options/3}\predref{ssl_set_options}{3} to create and configure copies of existing contexts in these hooks. The example file \file{https.pl} also provides a server that \textit{does} require the client to show its certificate. This provides an additional level of security, often used to allow a selected set of clients to perform sensitive tasks. Note that a single Prolog program can call \index{http_server/2}\predref{http_server}{2} with different parameters to provide services at several security levels as described below. These servers can either use their own dispatching or commonly use \index{http_dispatch/1}\predref{http_dispatch}{1} and check the \const{port} property of the request to verify they are called with the desired security level. If a service is approached at a too low level of security, the handler can deny access or use HTTP redirect to send the client to to appropriate interface. \begin{itemize} \item A plain HTTP server at port 80. This can either be used for non-sensitive information or for \jargon{redirecting} to a more secure service. \item An HTTPS server at port 443 for sensitive services to the general public. \item An HTTPS server that demands for a client key on a selected port for administrative tasks or sensitive machine-to-machine communication. \end{itemize} \subsection{HTTPS behind a proxy} \label{sec:https-proxy} The above expects Prolog to be accessible directly from the internet. This is becoming more popular now that services are often deployed using \jargon{virtualization}. If the Prolog services are placed behind a reverse proxy, HTTPS implementation is the task of the proxy server (e.g., Apache or Nginx). The communication from the proxy server to the Prolog server can use either plain HTTP or HTTPS. As plain HTTP is easier to setup and faster, this is typically preferred if the network between the proxy server and Prolog server can be trusted. Note that the proxy server \emph{must} decrypt the HTTPS traffic because it must decide on the destination based on the encrypted HTTP header. \jargon{Port forwarding} provides another option to make a server running on a machine that is not directly connected to the internet visible. It is not needed to decrypt the traffic using port forwarding, but it is also not possible to realise \jargon{virtual hosts} or \jargon{path-based} proxy rules. Virtual hosts for HTTPS are available via \jargon{Server Name Indication}~(SNI). This is a TLS extension that allows servers to host different domains from the same IP address. See the \index{sni_hook/1}\predref{sni_hook}{1} option of \index{ssl_context/3}\predref{ssl_context}{3} for more information. \section{Compatibility of the API} \label{sec:compatibility} Previous versions of the library used plain Prolog terms to represent the certificate objects as lists of fields. Newer versions of the library preserve the raw underlying structures as opaque handles to allow for more complicated operations to be performed on them. Any old code which obtains fields from the certificate using \index{memberchk/2}\predref{memberchk}{2} should be modified to use \index{certificate_field/2}\predref{certificate_field}{2} instead. For example, \begin{code} memberchk(subject(Subject), Certificate) \end{code} \noindent will instead need to be \begin{code} cerficate_field(Certificate, subject(Subject)) \end{code} \noindent Note that some of the fields do not match up exactly with their previous counterparts (key is now public_key, for example). \section{Acknowledgments} \label{sec:ssl-acknowledgments} The development of the SWI-Prolog SSL interface has been sponsored by \href{http://www.sss.co.nz}{Scientific Software and Systems Limited}. The current version contains contributions from many people. Besides the mentioned authors, \href{https://www.metalevel.at}{Markus Triska} has submitted several patches, and improved and documented the integration of this package with the HTTP infrastructure. %\bibliographystyle{plain} %\bibliography{ssl} \printindex \end{document}