Implement serializing the state of packet traversal in "continuations".

One purpose of OpenFlow packet-in messages is to allow a controller to
interpose on the path of a packet through the flow tables.  If, for
example, the controller needs to modify a packet in some way that the
switch doesn't directly support, the controller should be able to
program the switch to send it the packet, then modify the packet and
send it back to the switch to continue through the flow table.

That's the theory.  In practice, this doesn't work with any but the
simplest flow tables.  Packet-in messages simply don't include enough
context to allow the flow table traversal to continue.  For example:

    * Via "resubmit" actions, an Open vSwitch packet can have an
      effective "call stack", but a packet-in can't describe it, and
      so it would be lost.

    * A packet-in can't preserve the stack used by NXAST_PUSH and
      NXAST_POP actions.

    * A packet-in can't preserve the OpenFlow 1.1+ action set.

    * A packet-in can't preserve the state of Open vSwitch mirroring
      or connection tracking.

This commit introduces a solution called "continuations".  A continuation
is the state of a packet's traversal through OpenFlow flow tables.  A
"controller" action with the "pause" flag, which is newly implemented in
this commit, generates a continuation and sends it to the OpenFlow
controller in a packet-in asynchronous message (only NXT_PACKET_IN2
supports continuations, so the controller must configure them with
NXT_SET_PACKET_IN_FORMAT).  The controller processes the packet-in,
possibly modifying some of its data, and sends it back to the switch with
an NXT_RESUME request, which causes flow table traversal to continue.  In
principle, a single packet can be paused and resumed multiple times.

Another way to look at it is:

    - "pause" is an extension of the existing OFPAT_CONTROLLER
      action.  It sends the packet to the controller, with full
      pipeline context (some of which is switch implementation
      dependent, and may thus vary from switch to switch).

    - A continuation is an extension of OFPT_PACKET_IN, allowing for
      implementation dependent metadata.

    - NXT_RESUME is an extension of OFPT_PACKET_OUT, with the
      semantics that the pipeline processing is continued with the
      original translation context from where it was left at the time
      it was paused.

Signed-off-by: Ben Pfaff <blp@ovn.org>
Acked-by: Jarno Rajahalme <jarno@ovn.org>

lib/learning-switch.c

@@ -511,7 +511,6 @@ static void
 process_packet_in(struct lswitch *sw, const struct ofp_header *oh)
 {
     struct ofputil_packet_in pi;
-    size_t total_len;
     uint32_t buffer_id;
     uint32_t queue_id;
     ofp_port_t out_port;
@@ -525,7 +524,7 @@ process_packet_in(struct lswitch *sw, const struct ofp_header *oh)
     struct dp_packet pkt;
     struct flow flow;
 
-    error = ofputil_decode_packet_in(oh, &pi, &total_len, &buffer_id);
+    error = ofputil_decode_packet_in(oh, true, &pi, NULL, &buffer_id, NULL);
     if (error) {
         VLOG_WARN_RL(&rl, "failed to decode packet-in: %s",
                      ofperr_to_string(error));