% This LaTeX document was generated using the LaTeX backend of PlDoc,
% The SWI-Prolog documentation system



\section{library(protobufs): Google's Protocol Buffers ("protobufs")}

\label{sec:protobufs}

\begin{tags}
\mtag{author}- Jeffrey Rosenwald (JeffRose@acm.org) \\- Peter Ludemann (peter.ludemann@gmail.org)\mtag{See also}- \url{https://developers.google.com/protocol-buffers} \\- \url{https://developers.google.com/protocol-buffers/docs/encoding}
    \tag{Compatibility}
SWI-Prolog
\end{tags}

Protocol buffers are Google's language-neutral, platform-neutral,
extensible mechanism for serializing structured data -- think XML, but
smaller, faster, and simpler. You define how you want your data to be
structured once. This takes the form of a template that describes the
data structure. You use this template to encode and decode your data
structure into wire-streams that may be sent-to or read-from your peers.
The underlying wire stream is platform independent, lossless, and may be
used to interwork with a variety of languages and systems regardless of
word size or endianness. Techniques exist to safely extend your data
structure without breaking deployed programs that are compiled against
the "old" format.

The idea behind Google's Protocol Buffers is that you define your
structured messages using a domain-specific language and tool
set. Further documentation on this is at
\href{https://developers.google.com/protocol-buffers}{https:\Sidiv{}developers.google.com/protocol-buffers}.

There are two ways you can use protobufs in Prolog:

\begin{itemize}
    \item with a compiled \verb$.proto$ file: \predref{protobuf_parse_from_codes}{3} and
\predref{protobuf_serialize_to_codes}{3}.
    \item with a lower-level interface \predref{protobuf_message}{2}, which allows you
to define your own domain-specific language for parsing and
serializing protobufs.
\end{itemize}

The \predref{protobuf_parse_from_codes}{3} and \predref{protobuf_serialize_to_codes}{3}
interface translates between a "wire stream" and a Prolog term. This
interface takes advantage of SWI-Prolog's
\href{/pldoc/man?section=bidicts}{dict}.
There is a \const{protoc} plugin (\verb$protoc-gen-swipl$) that generates a
Prolog file of meta-information that captures the \verb$.proto$ file's
definition in the \const{protobufs} module:

\begin{shortlist}
    \item \verb$proto_meta_normalize(Unnormalized, Normalized)$
    \item \verb$proto_meta_package(Package, FileName, Options)$
    \item \verb$proto_meta_message_type( Fqn, Package, Name)$
    \item \verb$proto_meta_message_type_map_entry( Fqn)$
    \item \verb$proto_meta_field_name( Fqn, FieldNumber, FieldName, FqnName)$
    \item \verb$proto_meta_field_json_name( FqnName, JsonName)$
    \item \verb$proto_meta_field_label( FqnName, LabelRepeatOptional) % 'LABEL_OPTIONAL', 'LABEL_REQUIRED', 'LABEL_REPEATED'$
    \item \verb$proto_meta_field_type( FqnName, Type) % 'TYPE_INT32', 'TYPE_MESSAGE', etc$
    \item \verb$proto_meta_field_type_name( FqnName, TypeName)$
    \item \verb$proto_meta_field_default_value( FqnName, DefaultValue)$
    \item \verb$proto_meta_field_option_packed( FqnName)$
    \item \verb$proto_meta_enum_type( FqnName, Fqn, Name)$
    \item \verb$proto_meta_enum_value( FqnName, Name, Number)$
    \item \verb$proto_meta_field_oneof_index( FqnName, Index)$
    \item \verb$proto_meta_oneof( FqnName, Index, Name)$
\end{shortlist}

The \predref{protobuf_message}{2} interface allows you to define your message
template as a list of predefined
Prolog terms that correspond to production rules in the Definite Clause
Grammar (DCG) that realizes the interpreter. Each production rule has an
equivalent rule in the protobuf grammar. The process is not unlike
specifiying the format of a regular expression. To encode a template to
a wire-stream, you pass a grounded template, \verb$X$, and variable, \verb$Y$, to
\predref{protobuf_message}{2}. To decode a wire-stream, \verb$Y$, you pass an ungrounded
template, \verb$X$, along with a grounded wire-stream, \verb$Y$, to
\predref{protobuf_message}{2}. The interpreter will unify the unbound variables in
the template with values decoded from the wire-stream.

For an overview and tutorial with examples, see
\verb$library(protobufs)$: Google's Protocol Buffers (\secref{protobufs-main})
Examples of usage may also be found by inspecting
\href{https://github.com/SWI-Prolog/contrib-protobufs/blob/master/test_protobufs.pl}{test_protobufs.pl}
and the
\href{https://github.com/SWI-Prolog/contrib-protobufs/tree/master/demo}{demo}
directory, or by looking at the "addressbook" example that is typically
installed at
/usr/lib/swi-\file{prolog/doc/packages/examples/protobufs/interop/addressbook.pl}\vspace{0.7cm}

\begin{description}
    \predicate[semidet]{protobuf_parse_from_codes}{3}{+WireCodes:list(int), +MessageType:atom, -Term}
Process bytes (list of int) that is the serialized form of a message (designated
by \verb$MessageType$), creating a Prolog term.

\verb$Protoc$ must have been run (with the \verb$--swipl_out=$ option and the resulting
top-level _\file{pb.pl} file loaded. For more details, see the "protoc" section of the
overview documentation.

Fails if the message can't be parsed or if the appropriate meta-data from \const{protoc}
hasn't been loaded.

All fields that are omitted from the \verb$WireCodes$ are set to their
default values (typically the empty string or 0, depending on the
type; or \verb$[]$ for repeated groups). There is no way of testing
whether a value was specified in \verb$WireCodes$ or given its default
value (that is, there is no equivalent of the Python
implementation's =HasField`). Optional embedded messages and groups
do not have any default value -- you must check their existence by
using \predref{get_dict}{3} or similar. If a field is part of a "oneof" set,
then none of the other fields is set. You can determine which field
had a value by using \predref{get_dict}{3}.

\begin{arguments}
\arg{WireCodes} & Wire format of the message from e.g., \predref{read_stream_to_codes}{2}.
(The stream should have options \verb$encoding(octet)$ and \verb$type(binary)$,
either as options to \predref{read_file_to_codes}{3} or by calling \predref{set_stream}{2}
on the stream to \predref{read_stream_to_codes}{2}.) \\
\arg{MessageType} & Fully qualified message name (from the \verb$.proto$ file's \const{package} and \const{message}).
For example, if the \const{package} is \verb$google.protobuf$ and the
message is \verb$FileDescriptorSet$, then you would use
\verb$'.google.protobuf.FileDescriptorSet'$ or \verb$'google.protobuf.FileDescriptorSet'$.
If there's no package name, use e.g.: \verb$'MyMessage$ or \verb$'.MyMessage'$.
You can see the packages by looking at
\verb$protobufs:proto_meta_package(Pkg,File,_)$
and the message names and fields by
\verb$protobufs:proto_meta_field_name('.google.protobuf.FileDescriptorSet', FieldNumber, FieldName, FqnName)$ (the initial '.' is not optional for these facts,
only for the top-level name given to \predref{protobuf_serialize_to_codes}{3}). \\
\arg{Term} & The generated term, as nested \href{/pldoc/man?section=bidicts}{dict}s. \\
\end{arguments}

\begin{tags}
    \tag{Errors}
\verb$version_error(Module-Version)$ you need to recompile the \verb$Module$
with a newer version of \verb$protoc$.
    \tag{See also}
\verb$library(protobufs)$: Google's Protocol Buffers (\secref{protobufs-serialize-to-codes})\mtag{bug}- Ignores \verb$.proto$ \href{https://developers.google.com/protocol-buffers/docs/proto\#extensions}{extensions}. \\- \const{map} fields don't get special treatment (but see \predref{protobuf_map_pairs}{3}). \\- Generates fields in a different order from the C++, Python,
Java implementations, which use the field number to determine
field order whereas currently this implementation uses field
name. (This isn't stricly speaking a bug, because it's allowed
by the specification; but it might cause some surprise.)\mtag{To be done}- document the generated terms (see \file{library(http/json)} and \predref{json_read_dict}{3}) \\- add options such as \const{true} and \verb$value_string_as$ (similar to \predref{json_read_dict}{3}) \\- add option for form of the \href{/pldoc/man?section=bidicts}{dict} tags (fully qualified or not) \\- add option for outputting fields in the C++/Python/Java order
(by field number rather than by field name).
\end{tags}

    \predicate[det]{protobuf_serialize_to_codes}{3}{+Term:dict, -MessageType:atom, -WireCodes:list(int)}
Process a Prolog term into bytes (list of int) that is the serialized form of a
message (designated by \verb$MessageType$).

\verb$Protoc$ must have been run (with the \verb$--swipl_out=$ option and the resulting
top-level _\file{pb.pl} file loaded. For more details, see the "protoc" section of the
overview documentation.

Fails if the term isn't of an appropriate form or if the appropriate
meta-data from \const{protoc} hasn't been loaded, or if a field name is incorrect
(and therefore nothing in the meta-data matches it).

\begin{arguments}
\arg{Term} & The Prolog form of the data, as nested \href{/pldoc/man?section=bidicts}{dict}s. \\
\arg{MessageType} & Fully qualified message name (from the \verb$.proto$ file's \const{package} and \const{message}).
For example, if the \const{package} is \verb$google.protobuf$ and the
message is \verb$FileDescriptorSet$, then you would use
\verb$'.google.protobuf.FileDescriptorSet'$ or \verb$'google.protobuf.FileDescriptorSet'$.
If there's no package name, use e.g.: \verb$'MyMessage$ or \verb$'.MyMessage'$.
You can see the packages by looking at
\verb$protobufs:proto_meta_package(Pkg,File,_)$
and the message names and fields by
\verb$protobufs:proto_meta_field_name('.google.protobuf.FileDescriptorSet', FieldNumber, FieldName, FqnName)$ (the initial '.' is not optional for these facts,
only for the top-level name given to \predref{protobuf_serialize_to_codes}{3}). \\
\arg{WireCodes} & Wire format of the message, which can be output using
\verb$format('~s', [WireCodes])$. \\
\end{arguments}

\begin{tags}
\mtag{Errors}- \verb$version_error(Module-Version)$ you need to recompile the \verb$Module$
with a newer version of \verb$protoc$. \\- existence_error if a field can't be found in the meta-data
    \tag{See also}
\verb$library(protobufs)$: Google's Protocol Buffers (\secref{protobufs-serialize-to-codes})\mtag{bug}- \const{map} fields don't get special treatment (but see \predref{protobuf_map_pairs}{3}). \\- \const{oneof} is not checked for validity.
\end{tags}

    \predicate[semidet]{protobuf_message}{2}{?Template, ?WireStream}
\nodescription
    \predicate[nondet]{protobuf_message}{3}{?Template, ?WireStream, ?Rest}
Marshals and unmarshals byte streams encoded using Google's
Protobuf grammars. \predref{protobuf_message}{2} provides a bi-directional
parser that marshals a Prolog structure to \arg{WireStream}, according
to rules specified by \arg{Template}. It can also unmarshal \arg{WireStream}
into a Prolog structure according to the same grammar.
\predref{protobuf_message}{3} provides a difference list version.

\begin{arguments}
\arg{Template} & is a protobuf grammar specification. On decode,
unbound variables in the \arg{Template} are unified with their respective
values in the \arg{WireStream}. On encode, \arg{Template} must be ground. \\
\arg{WireStream} & is a code list that was generated by a protobuf
encoder using an equivalent template. \\
\end{arguments}

\begin{tags}
    \tag{bug}
The protobuf specification states that the wire-stream can have
the fields in any order and that unknown fields are to be ignored.
This implementation assumes that the fields are in the exact order
of the definition and match exactly. If you use
\predref{protobuf_parse_from_codes}{3}, you can avoid this problem.o
\end{tags}

    \predicate[semidet]{protobuf_field_is_map}{2}{+MessageType, +FieldName}
Succeeds if \verb$MessageType$'s \verb$FieldName$ is defined as a map$<$...$>$ in
the .proto file.

    \predicate[det]{protobuf_map_pairs}{3}{+ProtobufTermList:list, ?DictTag:atom, ?Pairs}
Convert between a list of protobuf map entries (in the form
\verb$DictTag{key:Key, value:Value}$ and a key-value list as described
in \file{library(pairs)}. At least one of \verb$ProtobufTermList$ and \verb$Pairs$
must be instantiated; \verb$DictTag$ can be uninstantiated. If
\verb$ProtobufTermList$ is from a term created by
\predref{protobuf_parse_from_codes}{3}, the ordering of the items is undefined;
you can order them by using \predref{keysort}{2} (or by a predicate such as
\predref{dict_pairs}{3}, \predref{list_to_assoc}{2}, or \predref{list_to_rbtree}{2}.
\end{description}