The lazy-thp may fail with older kernels if khugepaded will be fast enough
to merge pages in the partially populated areas.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
so that .gcda (coverage data) files are created with read-write
permissions for all, otherwise uns tests will not be able to
modify these files and code coverage data will not be collected
for them.
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Running all the tests with --lazy-pages and --remote-lazy-pages takes too
long. Let's run only several representing tests for those cases.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Otherwise we will see warnings like these ones:
Warn (criu/kerndat.c:805): Can't load /run/criu.kdat
Warn (criu/net.c:2726): Unable to get a socket network namespace
Warn (criu/kerndat.c:844): Can't keep kdat cache on non-tempfs
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
We found a good plugin for Jenkins to analize test results,
but it can handle only the junit format.
v2: work without the junit python module
v3: install junit-xml, because zdtm.py is executed
with the --report option from jenkins scripts.
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
The x86_64 job is the one we run inside Docker (as opposed to 'local',
which runs on the host). Let's update this to use Ubuntu Zesty, and
since it has clang-4 (unlike the 'local' one) it makes sense to add
a clang build.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Also, use more official / better maintained Docker image repo.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Numerous improvement to extract-deb-pkg, in no particular order:
* Use bash
* More strict error checking (set -u -o pipefail)
* Print error message if target dir exists
* Check number of arguments, provide usage info
* Separate mirror to a variable for easier changing
* Use mirrors.kernel.org
* Use https and check cert with curl
* Make curl silent
* Use zgrep instead of gunzip | grep
* Error out with a message in case more than 1 package was found
* Do not create a target directory before we need it
* Fix shellcheck warnings (way too many to mention)
* ...
* PROFIT!!!111
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
This comes with newer libc headers so we can catch errors earlier.
Also, clang 4.0.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Same as commit e4000c59a7 ("Dockerfiles: don't ignore make failures").
Seems that Dockerfile.fedora-asan copy/pasted an old version.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
* drop --keep-going etc from --lazy-pages pass
* add --remote-lazy-pages pass
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Most of zdtm test should pass with --lazy-pages with kernels newer than
4.11.
Some test excluded for older kernels surprisingly pass even now, mainly
becuase they do not actually stress userfaultfd, which will be fixed in the
upcoming commits :)
The cmdlinenv00 fails even with kernel 4.11 because of a race between uffd
and gup in the case external process reads /proc/<pid>/cmdline before
memory containing the command line is populated.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Add linux/userfaultfd.h to criu sources. This header is a part
of the kernel API and I see nothing wrong to have in the repo.
Why we want to do this:
* to check that criu works correctly if a kernel doesn't
support userfaultfd.
* to check compilation of the userfaultfd part in travis-ci.
v2: remove UFFD from FEATURES_LIST
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Acked-by: Adrian Reber <areber@redhat.com>
Cc: Mike Rapoport <rppt@linux.vnet.ibm.com>
Cc: Adrian Reber <areber@redhat.com>
Signed-off-by: Andrew Vagin <avagin@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
This is a first try to include userfaultfd with criu. Right now it
still requires a "normal" checkpoint. After checkpointing the
application it can be restored with the help of userfaultfd.
All restored pages with MAP_ANONYMOUS and MAP_PRIVATE set are marked as
being handled by userfaultfd.
As soon as the process is restored it blocks on the first memory access
and waits for pages being transferred by userfaultfd.
To handle the required pages a new criu command has been added. For a
userfaultfd supported restore the first step is to start the
'lazy-pages' server:
criu lazy-pages -v4 -D /tmp/3/ --address /tmp/userfault.socket
This is part 1 of the userfaultfd integration which provides the
'lazy-pages' server implementation.
v2:
* provide option '--lazy-pages' to enable uffd style restore
* use send_fd()/recv_fd() provided by criu (instead of own
implementation)
* do not install the uffd as service_fd
* use named constants for MAP_ANONYMOUS
* do not restore memory pages and then later mark them as uffd
handled
* remove function find_pages() to search in pages-<id>.img;
now using criu functions to find the necessary pages;
for each new page search the pages-<id>.img file is opened
* only check the UFFDIO_API once
* trying to protect uffd code by CONFIG_UFFD;
use make UFFD=1 to compile criu with this patch
v3:
* renamed the server mode from 'uffd' -> 'lazy-pages'
* switched client and server roles transferring the UFFD FD
* the criu part running in lazy-pages server mode is now
waiting for connections
* the criu restore process connects to the lazy-pages server
to pass the UFFD FD
* before UFFD copying anything else the VDSO pages are copied
as it fails to copy unused VDSO pages once the process is running.
this was necessary to be able to copy all pages.
* if there are no more UFFD messages for 5 seconds the lazy-pages
server switches in copy mode to copy all remaining pages, which
have not been requested yet, into the restored process
* check the UFFDIO_API at the correct place
* close UFFD FD in the restorer to remove open UFFD FD in the
restored process
v4:
* removed unnecessary madvise() calls ; it seemed necessary when
first running tests with uffd; it actually is not necessary
* auto-detect if build-system provides linux/userfaultfd.h
header
* simplify unix domain socket setup and communication.
* use --address to specify the location of the used
unix domain socket
v5:
* split the userfaultfd patch in multiple smaller patches
* introduced vma_can_be_lazy() function to check if a page
can be handled by uffd
* moved uffd related code from cr-restore.c to uffd.c
* handle failure to register a memory page of the restored process
with userfaultfd
v6:
* get PID of to be restored process from the 'criu restore' process;
first the PID is transferred and then the UFFD
* code has been re-ordered to be better prepared for lazy-restore
from remote host
* compile test for UFFD availability only once
Signed-off-by: Adrian Reber <areber@redhat.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
The ppc64le docker image has broken /etc/apt/sources.list. A small fixup to
it allows running ppc64le tests.
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
We have to use "debian.jessie" instead of "ubuntu.xenial" because
qemu-s390x-static can only run z900 instructions and Ubuntu is
built with a more recent architecture level set.
We first enable only gcc because there are still some issues with clang.
Signed-off-by: Michael Holzheu <holzheu@linux.vnet.ibm.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
I was adapting CRIU with ia32 support for building with Koji,
and found that Koji can't build x86_64 packages and have
i686 libs installed.
While at it, I found that i686 libraries requirement is
no longer valid since I've deleted the second parasite.
Drop feature test for i686 libs and put test for gcc.
That will effectively test if gcc can compile 32-bit code
and bug with debian's gcc (#315).
Signed-off-by: Dmitry Safonov <dsafonov@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
I need to add feature test written in assembly to check
if the feature can be compiled.
Add a make function for this purpose.
Signed-off-by: Dmitry Safonov <dsafonov@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
We only needed it for kernel 3.19. Apparently, Ubuntu 14.04.5 comes
with a kernel from 16.04 (i.e. 4.4), so we can disable this workaround!
Anyway, just in case, let's do it conditionally.
While at it, slightly improve the comment.
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
asciidoc pulls in a lot of dependencies, most of those are not
needed as we just use it to convert txt to a man page. Adding
--no-install-recommended option to apt-get makes it skip those
additional dependencies. The only needed package is xmlto, so
let's add it explicitly.
This results is some 50 packages being skipped (mostly TeX/LaTeX and
some extra SGML tools), wow!
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
v2: Use the devicemapper storage driver for docker containers to
run more tests. AUFS doesn't support file handles, so a few tests
(e.g. inotify00 can't be run).
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Commit 299e4b4 ("travis: enable ccache for docker/qemu builds") combined
multiple RUN statements in Dockerfiles into a single one, which is good
for performance (as there is an intermediate cache created after each
statement). Unfortunately, it partially did so by combining statements
with use of semicolons, meaning we are ignoring non-zero exit codes.
As a result, failure from make are ignored and Travis builds are all
green.
To fix, replace all occurences of semicolon with &&.
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Commit d9486bd720 ("arm/pie/build: do not produce relocatable parasite
object") removed -r from LDFLAGS used to compile criu pie. This
functionality somehow never made it to criu-dev, and was also lost
in master then compel was ported to it.
Make it work with compel.
Unfortunately it was not as simple as I initially thought, as -r flag
to ld was built into nmk. This patch removes it, and adds it to all
places that need intermediate linking.
Cc: Dmitry Safonov <dsafonov@virtuozzo.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Acked-by: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
When performing static linking, the order of objects is important
(when the linker finds an undefined reference, it looks forward,
never back). Therefore, sorting objects breaks things.
Required for the next patch.
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Acked-by: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
"make uninstall" is supposed to remove all the files that
"make install" (with the same arguments) have created.
This is a test to check that.
PS ideally, "make uninstall" should also remove any empty directories,
but let's not care about it for now.
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
As we compile-test non-x86_64 architectures under qemu emulation,
it works pretty slow.
Dmitry Safonov suggested, and Andrey Vagin initially implemented
supporting ccache for such builds. This patch is based heavily
on Andrey's work -- all the bugs added are purely mine though.
Performance results: in an ideal environment (two builds of the same
code, one with cold (empty) ccache, another with the hot one)
I saw compile time improvements of 4x to 5x, and total test run time
improvement up to 2x to 2.5x. In layman terms, the complete test run
that was taking more than 50 minutes now takes about 25!
Notes on handling .ccache directory:
1. Before running docker build, .ccache directory (saved in between
runs by Travis) is moved to criu source code root, from where it
is copied by docker together with criu sources.
2. In Dockerfile, .ccache gets moved to /tmp, and CCACHE_DIR
is set accordingly.
3. After running docker build, .ccache is copied out from docker
container back to the host (from where it is saved by Travis).
Ccache envorinment notes:
1. CCACHE_NOCOMPRESS is needed because otherwise tons of time is spent
on compression/decompression (big performance hit under qemu).
2. CCACHE_CPP2 is required with clang, see detailed explanation at
http://petereisentraut.blogspot.com/2011/09/ccache-and-clang-part-2.html
The logic of setting CCACHE_CPP2 in Dockerfile is somewhat fancy;
unfortunately I was not able to come up with a simpler approach.
Misc:
1. Travis runs "ccache -s" after the build is completed. A call to
"ccache -s" is called to Dockerfile before make, so one can
compare before/after numbers.
2. make invocations are surrounded by "date" calls so one can get the
compilation times.
Changes in v2:
- consolidate Dockerfile statements (ENV, RUN)
- single object make test is no longer commented out
- simplify "make mrproper" test
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Reviewed-by: Dmitry Safonov <dsafonov@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
For now, it's done for local builds only (i.e. no per-arch Docker
builds yet). The reason is, it's easier to play with ccache when
the compiles are (relatively) fast.
Performance: there is 2x to 3x improvement in build speeds for
ideal cases (same code, hot cache), but the absolute savings
are negligible (example: 7 seconds instead of 23).
Note that ccache is not compatible with gcov, so we only enable it
for non-gcov build (which happens to be the one with clang).
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Unify alpine and non-alpine builds. The only difference is foreign
arch builds need some preparation -- separate that to a dependency.
Unfortunately we can't use wildcard targets ("%: ") as non-wildcard
ones are prevaling. Therefore, a somewhat ugly hack to generate
$arch: Dockerfile.$arch dependency is added.
While at it:
- rename DB_ARGS to DB_CC
- mark clean as phony
- make "all" really run all the builds we can
Nice side effect: autocompletion ("make -C scripts/build <TAB>")
now works!
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
As pointed out by Andrey, arch clang tests are using gcc regardless
of CLANG=1 set in travis environment.
Frankly, I do not understand how it worked before (while being
pretty sure it worked!), but here is a way to fix it.
Reported-by: Andrey Vagin <avagin@virtuozzo.com>
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Apparently @true was used as a rule for any target which should have
a non-empty rule doing nothing. For this, there's a "<target>: ;"
syntax, let's use this and eliminate an unnecessary fork/exec :)
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
This adds the reverse of CFLAGS_obj.o with CFLAGS_REMOVE_obj.o.
This allows to prevent certain CFLAGS from being used to
compile files.
travis-ci: success for series starting with [1/6] compel/infect: fix out-of-bounds parasite memcpy()
Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Dmitry Safonov <dsafonov@virtuozzo.com>
Reviewed-by: Dmitry Safonov <dsafonov@virtuozzo.com>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
I am not quite sure how that happened, but compel hgen was not used for
ARM/ARM64, instead there's a simple version of it, called
gen-offsets.sh. The main difference is, shell script doesn't handle ELF
relocations, which apparently is not (currently?) needed for ARM.
It's bad to maintain two tools for the same functionality, so this
patch kills gen-offsets.sh and related stuff, making compel hgen
working on ARM. ELF relocations are still not handled, this code
is #ifdef-ed out for now and can be fixed to work on ARM later.
This patch also kills some macros and defines that seem obsoleted
now. For example, compel_relocs_apply() is now called unconditionally,
as it handles the trivial case of 0 relocs just fine.
Now, I checked that the blob headers generated by compel hgen and
gen-offsets.h are similar (i.e. generated blob code and values defined
are the same), but haven't done much above that.
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
Currently, some compel internals are exposed to user API
(both C and CLI), making its usage more complicated than
it can be.
In particular, compel user have to specify a number of parameters
(names for various data) on the command line, and when in C code
assign a struc piegen_opt_t fields using the same names, without
using those identifiers anywhere else in the code.
It makes sense to hide this complexity from a user, which is what
this commit does.
First, remove the ability to specify individual names for data,
instead introducing a prefix that is prepended to all the names.
Second, generate a function %PREFIX%_setup_c_header() which does
all the needed assignments.
Third, convert users (criu/pie and compel test) to the new API.
NOTE that this patch breaks ARM, as compel hgen is not used for ARM.
This is to be fixed by a later patch in the series.
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
