The number of pending events returned by epoll_wait is overridden by the
first call to an event handler. Using an additional local variable resolves
this issue.
travis-ci: success for lazy-pages: add non-#PF events handling
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Currently we are waiting for lazy-pages daemon to finish as a part of
.restore method, which may cause filling test process memory before the
test process resumed it's execution after call to test_waitsig(). In such
case, no page faults occur, but rather all the memory is copied in
handle_remaining_pages method in uffd.c.
Let's move wait(<lazy-pages-pid>,..) after call to test.stop().
travis-ci: success for lazy-pages: add non-#PF events handling
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for revert zero pagemaps
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for revert zero pagemaps
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Note, that since zero pages stats never been into master we can make
incompatible changes to stats image.
travis-ci: success for revert zero pagemaps
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
The pagemap entries for pages mapped to zero pfn proved to be not useful...
travis-ci: success for revert zero pagemaps
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
A page that explicitly mapped to zero pfn or a page that is not present
should be treated in the same way, therefore the zero pagemaps are not
required and will be removed by the following commits.
travis-ci: success for revert zero pagemaps
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
If a page was not marked "present" at the dump time it will not be covered
by the pagemap and it will remain unmapped in the restored process. We
should uffdio_zero such pages and let kernel mm to take over.
travis-ci: success for revert zero pagemaps
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
CID 173076, issues/259
travis-ci: success for pagemap: verify the number of pages returned by receive_remote_pages_info
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
This function does weird things, so better have it at least somehow
documented.
travis-ci: success for lazy-pages: add comments to update_lazy_iovecs
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Instead of checking for availability of userfaultfd late during the restore
process, make the detection of supported userfaultfd functionality part of
kerndat. As a bonus, I've extended criu check with ability to verify
presence of userfaultfd.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Use latest version from usefaultfd tree [1]. Judging by comments about the
last re-spin of userfaultfd updates, the API will go in "as is", so we can
pretty much rely on the current API definitions for proper detection of
supported userfaultfd features.
[1] https://git.kernel.org/cgit/linux/kernel/git/andrea/aa.git/
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Add pre-dump and remote-lazy-pages passes to criu-lazy-pages.sh
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Currently we poll userfaultfd for page faults and if there were no page
faults during 5 seconds we stop monitoring the userfaultfd and start
copying remaining pages chunk by chunk.
If a page fault occurs during the copy, the faulting process will be stuck
until the page it accessed would be copied to its address space.
This patch limits the initial "page fault only" stage to 1 second instead
of 5, and interleaves non-blocking poll of userfaultfd with copying of the
remaining memory afterwards.
travis-ci: success for lazy-pages: interleave #PF handling with transfers of remaining pages
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for lazy-pages: spelling: s/pagefalt/#PF
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
After commit a97d6d3f1609 (pagemap: replace seek_page with seek_pagemap
method), uffd only searches the pagemap containing the faulting page, but
it not for the page itself. For local restore it causes wrong data to be
read from pages*img and subsequent crash of the restored process.
Adding a call to pr->skip_pages fixes the problem.
travis-ci: success for lazy-pages: fix searching for the page at #PF time
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Lazy migration requires both dumped and restored processes to coexist at
the same time. This breaks some basic assumptions in the zdtm design.
Simulation of lazy migration with the page server allows testing most of
the involved code paths without major intervention into zdtm
infrastructure.
travis-ci: success for lazy-pages: improve testability (rev2)
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
The kernel support for lazy pages (userfaultfd) lacks many important
features which effectively prevents success in certain tests.
Allow skipping such test with somewhat informative message
travis-ci: success for lazy-pages: improve testability (rev2)
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Currently, standalone page-server can only receive pages from the remote
dump. Extend it with the ability to serve local memory dump to a remote
lazy-pages daemon.
travis-ci: success for lazy-pages: improve testability (rev2)
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Instead of crashing dump/page-server when a problem detected after the
page-pipe was split, print nice error messages and return error.
travis-ci: success for page-xfer: page_server_get_pages: replace BUG_ONs with 'return -1' (rev2)
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Splitting of the trailing part of page-pipe buffer worked by coincidence
for single page requests. Request longer than a single page were not
handled correctly.
The proper point for splitting the trailing part of the page-pipe buffer is
the IOV following the IOV containing the desired page(s).
travis-ci: success for page-pipe: (yet another) fix for split page-pipe buffers (rev2)
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for Some more cleanups over uffd.c (rev3)
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
With epoll helpers in util we can stop exposing the
page-server socket to the oter world.
travis-ci: success for Some more cleanups over uffd.c (rev3)
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
v2: Move epoll_prepare() too
travis-ci: success for Some more cleanups over uffd.c (rev3)
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
The uffd code only needs the pstree items themselves, not
any IDs and relations they might have.
travis-ci: success for Some more cleanups over uffd.c (rev3)
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
This run away from previous set :) Two routines are now
identical, only page-read flags differ.
v2: Keep the uffd_hanle_pages() name
travis-ci: success for Some more cleanups over uffd.c (rev3)
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Extend the RPC feature check functionality to also test for lazy-pages
support. This does not check for certain UFFD features (yet). Right now
it only checks if kerndat_uffd() returns non-zero.
The RPC response is now transmitted from the forked process instead of
encoding all the results into the return code. The parent RPC process
now only sends an RPC message in the case of a failure.
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Currently, lazy dump starts page server regardless of errors that might
have been encountered at earlier stages. Fix it.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Le sigh.
travis-ci: success for more pr_perror() usage fixes
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Add a queue of async-read jobs into page-xfer. When the
page_server_sk gets a read event from epoll it reads as
many bytes into page_server_iov + page buffer as recv
allows and returns.
Once the full iov+data is ready the requestor is notified
and the next async read is started.
This patch removes calls to recv(...MSG_WAITALL) from all
remote async paths.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Finally, page_fault_local and page_fault_remote are
absolutely identical, so we can just merge them.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
This one is called by PR once IO is complete (right now
for sync cases only, more work is required here) and
lets us unify local and remote PF code in uffd.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
The _copy and _update_lazy_iovecs are both called by hands
once the data is ready.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
This flag means, that the PR_ASYNC is valid, but the IO
should be started ASAP. This is how remote reader works,
so this flag is mostly for the local reader. It will let
us unify page-fault handlers for local and remote cases.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
We already have routines that do send-req, recv-info
and recv-page, so no need in yet another one.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
All the "lower" page-read-s should have already arrived with
pre-dump. This fixes the combined scheme.
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
The page transfer protocol is completely synchronous on the dump side,
therefore we can presume that when we get POLLIN event on the page server
socket it is either page info response for the last sent page request or
the page data following the last page info.
In the first case we set ev_data associated with page server socket events
to values received in receive_remote_page_info and in the second case we
reset ev_data to zero. This allows us to distinguish what was the reason
page_server_event have been called.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
The synchronous remote page transfer prevents reception of uffd events
during the communications with the page server on the dump side. Adding
socket file descriptor to epoll_wait allows processing of incoming uffd
events after non-blocking request for remote page is issued and before the
dump side page server replies.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
The asynchronous version of remote page_read will send the request to the
dump side and return happily.
The response will be handled by the uffd.c because it's epoll loop is the
only place where we can handle events.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
This part of code is responsible for reseting pagemap to proper locatation,
and mapping requested address to zero pfn if needed. The upcoming addtions
to uffd.c will reuse this code.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
For asynchrounous page transfers in post-copy migration we need to be able
to request a remote pages, receive back information about the data is going
to arrive and receive the page data itself.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
It will used by lazy-pages daemon to enable polling for reception of page
data from remote dump
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
In early days of uffd.c return value from uffd_copy was used to count
transferred pages. Since this is not the case anymore we can use 0 as
success.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Currently lazy-pages daemon uses either pr->read_pages or get_remote_pages
to get actual page data from local images or remote server. From now on,
page_read will be completely responsible for getting the page data.
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
travis-ci: success for uffd: A new set of improvements
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
