This patch series adds the VM based selftest framework for Fedora kexec-tools. Currently there are 3 test cases, more could be added later. And the usage and details are included in the document for selftest.
This help do quick sanity check of new patches, also help debug issues when developing kexec-tools.
The test output looks like this:
==== Starting all tests: ==== local-kdump nfs-kdump ssh-kdump
======== Running Test Case local-kdump ======== ---- Building image for: 0-local.sh ---- -------- Output image is: /home/kasong/fedpkg/kexec-tools/tests/output/local-kdump/0-local.img -------- Building log is: /home/kasong/fedpkg/kexec-tools/tests/output/local-kdump/0-local.img.log ---- Starting test VM: 0-local.sh ---- -------- Qemu cmdline: /home/kasong/fedpkg/kexec-tools/tests/output/local-kdump/0-local.qemu_cmd -------- Console log: /home/kasong/fedpkg/kexec-tools/tests/output/local-kdump/0-local.console -------- Test log: /home/kasong/fedpkg/kexec-tools/tests/output/local-kdump/0-local.output --------Test finished: local-kdump TEST PASSED ======== Running Test Case nfs-kdump ======== ---- Building image for: 1-client.sh ---- -------- Output image is: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/1-client.img -------- Building log is: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/1-client.img.log ---- Building image for: 0-server.sh ---- -------- Output image is: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/0-server.img -------- Building log is: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/0-server.img.log ---- Starting VM: 0-server.sh ---- -------- Qemu cmdline: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/0-server.qemu_cmd -------- Console log: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/0-server.console -------- Test log: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/0-server.output ---- Starting test VM: 1-client.sh ---- -------- Qemu cmdline: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/1-client.qemu_cmd -------- Console log: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/1-client.console -------- Test log: /home/kasong/fedpkg/kexec-tools/tests/output/nfs-kdump/1-client.output /home/kasong/fedpkg/kexec-tools/tests/scripts/run-test.sh: line 6: kill: (1611239) - No such process --------Test finished: nfs-kdump TEST PASSED ======== Running Test Case ssh-kdump ======== ---- Building image for: 1-client.sh ---- -------- Output image is: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/1-client.img -------- Building log is: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/1-client.img.log ---- Building image for: 0-server.sh ---- -------- Output image is: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/0-server.img -------- Building log is: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/0-server.img.log ---- Starting VM: 0-server.sh ---- -------- Qemu cmdline: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/0-server.qemu_cmd -------- Console log: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/0-server.console -------- Test log: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/0-server.output ---- Starting test VM: 1-client.sh ---- -------- Qemu cmdline: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/1-client.qemu_cmd -------- Console log: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/1-client.console -------- Test log: /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/1-client.output /home/kasong/fedpkg/kexec-tools/tests/scripts/run-test.sh: line 6: kill: (1613373) - No such process --------Test finished: ssh-kdump TEST PASSED ======== Test results ======== ---------------- nfs-kdump: TEST PASSED ---------------- ssh-kdump: TEST PASSED ---------------- local-kdump: TEST PASSED
Kairui Song (3): selftest: Add basic infrastructure to build test image selftest: Add basic test framework selftest: Add document for selftests
tests/Makefile | 87 +++++++ tests/README | 65 +++++ tests/scripts/build-image.sh | 57 +++++ tests/scripts/build-scripts/base-image.sh | 11 + .../scripts/build-scripts/test-base-image.sh | 20 ++ tests/scripts/build-scripts/test-image.sh | 21 ++ tests/scripts/image-init-lib.sh | 241 ++++++++++++++++++ tests/scripts/kexec-kdump-test/init.sh | 102 ++++++++ .../kexec-kdump-test/kexec-kdump-test.service | 9 + tests/scripts/kexec-kdump-test/test.sh | 15 ++ tests/scripts/run-test.sh | 131 ++++++++++ tests/scripts/spawn-image-shell.sh | 16 ++ tests/scripts/test-lib.sh | 177 +++++++++++++ .../scripts/testcases/local-kdump/0-local.sh | 32 +++ tests/scripts/testcases/nfs-kdump/0-server.sh | 38 +++ tests/scripts/testcases/nfs-kdump/1-client.sh | 30 +++ tests/scripts/testcases/ssh-kdump/0-server.sh | 34 +++ tests/scripts/testcases/ssh-kdump/1-client.sh | 40 +++ 18 files changed, 1126 insertions(+) create mode 100644 tests/Makefile create mode 100644 tests/README create mode 100755 tests/scripts/build-image.sh create mode 100755 tests/scripts/build-scripts/base-image.sh create mode 100755 tests/scripts/build-scripts/test-base-image.sh create mode 100755 tests/scripts/build-scripts/test-image.sh create mode 100644 tests/scripts/image-init-lib.sh create mode 100755 tests/scripts/kexec-kdump-test/init.sh create mode 100644 tests/scripts/kexec-kdump-test/kexec-kdump-test.service create mode 100755 tests/scripts/kexec-kdump-test/test.sh create mode 100755 tests/scripts/run-test.sh create mode 100755 tests/scripts/spawn-image-shell.sh create mode 100644 tests/scripts/test-lib.sh create mode 100755 tests/scripts/testcases/local-kdump/0-local.sh create mode 100755 tests/scripts/testcases/nfs-kdump/0-server.sh create mode 100755 tests/scripts/testcases/nfs-kdump/1-client.sh create mode 100755 tests/scripts/testcases/ssh-kdump/0-server.sh create mode 100755 tests/scripts/testcases/ssh-kdump/1-client.sh
The Makefile In tests/ could help build a VM image using Fedora cloud image as base image, or, user can specify a base image using BASE_IMAGE=<path/to/file>. The current repo will be packeged and installed in the image, so the image could be used as a test image to test kexec-tools.
The image building is splited into two steps: The first step, it either convert the base image to qcow2 or create a snapshot on it, and install basic packages (dracut, grubby, ...) and do basic setups (setup crashkernel=, disable selinux, ...). See tests/scripts/build-scripts/base-image.sh for detail.
The second step, it creates a snapshot on top of the image produced by the previous step, and install the packaged kexec-tools of current repo. See tests/scripts/build-scripts/test-base-image.sh for detail.
In this way, if repo's content is changes, `make` will detect it and only rebuild the second snapshot which speed up the rebuild by a lot.
The image will be located as tests/output/test-base-image, and in qcow2 format. And default user/password is set to root/fedora.
Signed-off-by: Kairui Song kasong@redhat.com --- tests/Makefile | 79 ++++++ tests/scripts/build-image.sh | 57 +++++ tests/scripts/build-scripts/base-image.sh | 11 + .../scripts/build-scripts/test-base-image.sh | 15 ++ tests/scripts/image-init-lib.sh | 241 ++++++++++++++++++ 5 files changed, 403 insertions(+) create mode 100644 tests/Makefile create mode 100755 tests/scripts/build-image.sh create mode 100755 tests/scripts/build-scripts/base-image.sh create mode 100755 tests/scripts/build-scripts/test-base-image.sh create mode 100644 tests/scripts/image-init-lib.sh
diff --git a/tests/Makefile b/tests/Makefile new file mode 100644 index 0000000..ed3e21a --- /dev/null +++ b/tests/Makefile @@ -0,0 +1,79 @@ +BASE_IMAGE ?= + +TEST_ROOT := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST)))) +BUILD_ROOT := $(TEST_ROOT)/build +REPO = $(shell realpath $(TEST_ROOT)/../) +ARCH ?= $(shell arch) +SPEC = kexec-tools.spec + +DIST ?= fedora +DIST_ABR ?= f +DIST_ABRL ?= fc +DIST_UNSET ?= rhel +RELEASE ?= 32 + +DEFAULT_BASE_IMAGE_VER ?= 1.6 +DEFAULT_BASE_IMAGE ?= Fedora-Cloud-Base-$(RELEASE)-$(DEFAULT_BASE_IMAGE_VER).$(ARCH).raw.xz +DEFAULT_BASE_IMAGE_URL ?= https://dl.fedoraproject.org/pub/fedora/linux/releases/$(RELEASE)/Cloud/$(AR...) + +BUILD_ROOT = $(TEST_ROOT)/build +RPMDEFINE = --define '_sourcedir $(REPO)'\ + --define '_specdir $(REPO)'\ + --define '_builddir $(BUILD_ROOT)'\ + --define '_srcrpmdir $(BUILD_ROOT)'\ + --define '_rpmdir $(BUILD_ROOT)'\ + --define 'dist %{?distprefix}.$(DIST_ABRL)$(RELEASE)'\ + --define '$(DIST) $(RELEASE)'\ + --eval '%undefine $(DIST_UNSET)'\ + --define '$(DIST_ABRL)$(RELEASE) 1'\ + +KEXEC_TOOLS_SRC = $(filter-out $(REPO)/tests,$(wildcard $(REPO)/*)) +KEXEC_TOOLS_NVR = $(shell rpm $(RPMDEFINE) -q --specfile $(REPO)/$(SPEC) 2>/dev/null | grep -m 1 .) +KEXEC_TOOLS_RPM = $(BUILD_ROOT)/$(ARCH)/$(KEXEC_TOOLS_NVR).rpm + +all: $(TEST_ROOT)/output/test-base-image + +# Use either: +# fedpkg --release $(DIST_ABR)$(RELEASE) --path ../../ local +# or +# rpmbuild $(RPMDEFINE) -ba $(REPO)/$(SPEC) +# to rebuild the rpm, currently use rpmbuild to have better control over the rpm building process +# +$(KEXEC_TOOLS_RPM): $(KEXEC_TOOLS_SRC) + sh -c "cd .. && fedpkg sources" + @echo Rebuilding RPM due to modification of sources: $? + rpmbuild $(RPMDEFINE) -ba $(REPO)/$(SPEC) + +$(BUILD_ROOT)/base-image: + mkdir -p $(BUILD_ROOT) +ifeq ($(strip $(BASE_IMAGE)),) + wget $(DEFAULT_BASE_IMAGE_URL) -O $(BUILD_ROOT)/$(DEFAULT_BASE_IMAGE) + $(TEST_ROOT)/scripts/build-image.sh \ + $(BUILD_ROOT)/$(DEFAULT_BASE_IMAGE)\ + $(BUILD_ROOT)/base-image +else + $(TEST_ROOT)/scripts/build-image.sh \ + $(BASE_IMAGE)\ + $(BUILD_ROOT)/base-image +endif + +$(BUILD_ROOT)/inst-base-image: $(BUILD_ROOT)/base-image + @echo "Building installation base image" + echo $(KEXEC_TOOLS_NVR) + $(TEST_ROOT)/scripts/build-image.sh \ + $(BUILD_ROOT)/base-image \ + $(BUILD_ROOT)/inst-base-image \ + $(TEST_ROOT)/scripts/build-scripts/base-image.sh + +$(TEST_ROOT)/output/test-base-image: $(BUILD_ROOT)/inst-base-image $(KEXEC_TOOLS_RPM) + @echo "Building test base image" + mkdir -p $(TEST_ROOT)/output + $(TEST_ROOT)/scripts/build-image.sh \ + $(BUILD_ROOT)/inst-base-image \ + $(TEST_ROOT)/output/test-base-image \ + $(TEST_ROOT)/scripts/build-scripts/test-base-image.sh \ + $(KEXEC_TOOLS_RPM) + +clean: + rm -rf $(TEST_ROOT)/build + rm -rf $(TEST_ROOT)/output diff --git a/tests/scripts/build-image.sh b/tests/scripts/build-image.sh new file mode 100755 index 0000000..c196bfb --- /dev/null +++ b/tests/scripts/build-image.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash + +if [ $# -lt 2 ]; then + echo "Usage: $(basename $0) <base-image> <output-image> <build-script> [<build-script-args>] + Build a new <output-image> on top of <base-image>, and install + contents defined in <build-script>. <args> are directly passed + to <build-script>. + + If <base-image> is raw, will copy it and create <output-image> + in qcow2 format. + + If <base-image> is qcow2, will create <output-image> as a snapshot + on top of <base-image>" + exit 1 +fi + +BASEDIR=$(realpath $(dirname "$0")) +. $BASEDIR/image-init-lib.sh + +# Base image to build from +BASE_IMAGE=$1 && shift +if [[ ! -e $BASE_IMAGE ]]; then + perror_exit "Base image '$BASE_IMAGE' not found" +else + BASE_IMAGE=$(realpath "$BASE_IMAGE") +fi + +OUTPUT_IMAGE=$1 && shift +if [[ ! -d $(dirname $OUTPUT_IMAGE) ]]; then + perror_exit "Path '$(dirname $OUTPUT_IMAGE)' doesn't exists" +fi + +INST_SCRIPT=$1 && shift + +create_image_from_base_image $BASE_IMAGE $OUTPUT_IMAGE.building + +mount_image $OUTPUT_IMAGE.building + +img_inst() { + inst_in_image $OUTPUT_IMAGE.building $@ +} + +img_inst_pkg() { + inst_pkg_in_image $OUTPUT_IMAGE.building $@ +} + +img_run_cmd() { + run_in_image $OUTPUT_IMAGE.building "$@" +} + +img_add_qemu_cmd() { + QEMU_CMD+="$@" +} + +[ -e "$INST_SCRIPT" ] && source $INST_SCRIPT + +mv $OUTPUT_IMAGE.building $OUTPUT_IMAGE diff --git a/tests/scripts/build-scripts/base-image.sh b/tests/scripts/build-scripts/base-image.sh new file mode 100755 index 0000000..59f4574 --- /dev/null +++ b/tests/scripts/build-scripts/base-image.sh @@ -0,0 +1,11 @@ +#!/bin/sh + +img_inst_pkg grubby\ + dnsmasq\ + openssh openssh-server\ + dracut-network dracut-squash squashfs-tools ethtool snappy + +img_run_cmd "grubby --args systemd.journald.forward_to_console=1 systemd.log_target=console --update-kernel ALL" +img_run_cmd "grubby --args selinux=0 --update-kernel ALL" +img_run_cmd "grubby --args crashkernel=224M --update-kernel ALL" +img_run_cmd "mkdir -p /kexec-kdump-test" diff --git a/tests/scripts/build-scripts/test-base-image.sh b/tests/scripts/build-scripts/test-base-image.sh new file mode 100755 index 0000000..a649306 --- /dev/null +++ b/tests/scripts/build-scripts/test-base-image.sh @@ -0,0 +1,15 @@ +#!/bin/sh + +# Test RPMs to be installed +TEST_RPMS= +for _rpm in $@; do + if [[ ! -e $_rpm ]]; then + perror_exit "'$_rpm' not found" + else + TEST_RPMS=$(realpath "$_rpm") + fi +done + +img_inst_pkg $TEST_RPMS +# Test script should start kdump manually to save time +img_run_cmd "systemctl disable kdump.service" diff --git a/tests/scripts/image-init-lib.sh b/tests/scripts/image-init-lib.sh new file mode 100644 index 0000000..a4a453d --- /dev/null +++ b/tests/scripts/image-init-lib.sh @@ -0,0 +1,241 @@ +#!/usr/bin/env bash +[ -z "$TESTDIR" ] && TESTDIR=$(realpath $(dirname "$0")/../) + +SUDO="sudo" + +declare -A MNTS=() +declare -A DEVS=() + +perror() { + echo $@>&2 +} + +perror_exit() { + echo $@>&2 + exit 1 +} + +is_mounted() +{ + findmnt -k -n $1 &>/dev/null +} + +clean_up() +{ + for _mnt in ${MNTS[@]}; do + is_mounted $_mnt && $SUDO umount -f $_mnt + done + + for _dev in ${DEVS[@]}; do + [ ! -e "$_dev" ] && continue + [[ "$_dev" == "/dev/loop"* ]] && $SUDO losetup -d "$_dev" + [[ "$_dev" == "/dev/nbd"* ]] && $SUDO qemu-nbd --disconnect "$_dev" + done + + [ -d "$TMPDIR" ] && $SUDO rm --one-file-system -rf -- "$TMPDIR"; + + sync +} + +trap ' +ret=$?; +clean_up +exit $ret; +' EXIT + +# clean up after ourselves no matter how we die. +trap 'exit 1;' SIGINT + +readonly TMPDIR="$(mktemp -d -t kexec-kdump-test.XXXXXX)" +[ -d "$TMPDIR" ] || perror_exit "mktemp failed." + +get_image_fmt() { + local image=$1 fmt + + [ ! -e "$image" ] && perror "image: $image doesn't exist" && return 1 + + fmt=$(qemu-img info $image | sed -n "s/file format:\s*(.*)/\1/p") + + [ $? -eq 0 ] && echo $fmt && return 0 + + return 1 +} + +# If it's partitioned, return the mountable partition, else return the dev +get_mountable_dev() { + local dev=$1 parts + + $SUDO partprobe $dev && sync + parts="$(ls ${dev}p*)" + if [[ -n "$parts" ]]; then + if [[ $(echo "$parts" | wc -l) -gt 1 ]]; then + perror "can't handle base image with multiple partition" + fi + echo "$parts" + else + echo "$dev" + fi +} + +prepare_loop() { + [ -n "$(lsmod | grep "^loop")" ] && return + + $SUDO modprobe loop + + [ ! -e "/dev/loop-control" ] && perror_exit "failed to load loop driver" +} + +prepare_nbd() { + [ -n "$(lsmod | grep "^nbd")" ] && return + + $SUDO modprobe nbd max_part=4 + + [ ! -e "/dev/nbd0" ] && perror_exit "failed to load nbd driver" +} + +mount_nbd() { + local image=$1 size dev + for _dev in /sys/class/block/nbd* ; do + size=$(cat $_dev/size) + if [ "$size" -eq 0 ] ; then + dev=/dev/${_dev##*/} + $SUDO qemu-nbd --connect=$dev $image 1>&2 + [ $? -eq 0 ] && echo $dev && break + fi + done + + return 1 +} + +image_lock() +{ + local image=$1 timeout=5 fd + + eval "exec {fd}>$image.lock" + if [ $? -ne 0 ]; then + perror_exit "failed acquiring image lock" + exit 1 + fi + + flock -n $fd + rc=$? + while [ $rc -ne 0 ]; do + echo "Another instance is holding the image lock ..." + flock -w $timeout $fd + rc=$? + done +} + +# Mount a device, will umount it automatially when shell exits +mount_image() { + local image=$1 fmt + local dev mnt mnt_dev + + # Lock the image just in case user run this script in parrel + image_lock $image + + fmt=$(get_image_fmt $image) + [ $? -ne 0 ] || [ -z "$fmt" ] && perror_exit "failed to detect image format" + + if [ "$fmt" == "raw" ]; then + prepare_loop + + dev="$($SUDO losetup --show -f $image)" + [ $? -ne 0 ] || [ -z "$dev" ] && perror_exit "failed to setup loop device" + + elif [ "$fmt" == "qcow2" ]; then + prepare_nbd + + dev=$(mount_nbd $image) + [ $? -ne 0 ] || [ -z "$dev" ] perror_exit "failed to connect qemu to nbd device '$dev'" + else + perror_exit "Unrecognized image format '$fmt'" + fi + DEVS[$image]="$dev" + + mnt="$(mktemp -d -p $TMPDIR -t mount.XXXXXX)" + [ $? -ne 0 ] || [ -z "$mnt" ] && perror_exit "failed to create tmp mount dir" + MNTS[$image]="$mnt" + + mnt_dev=$(get_mountable_dev "$dev") + [ $? -ne 0 ] || [ -z "$mnt_dev" ] && perror_exit "failed to setup loop device" + + $SUDO mount $mnt_dev $mnt + [ $? -ne 0 ] && perror_exit "failed to mount device '$mnt_dev'" +} + +shell_in_image() { + local image=$1 && shift + local root=${MNTS[$image]} + + $SUDO chroot $root /bin/bash +} + +inst_pkg_in_image() { + local image=$1 && shift + local root=${MNTS[$image]} + + # LSB not available + # release_info=$($SUDO chroot $root /bin/bash -c "lsb_release -a") + # release=$(echo "$release_info" | sed -n "s/Release:\s*(.*)/\1/p") + # distro=$(echo "$release_info" | sed -n "s/Distributor ID:\s*(.*)/\1/p") + # if [ "$distro" != "Fedora" ]; then + # perror_exit "only Fedora image is supported" + # fi + release=$(cat $root/etc/fedora-release | sed -n "s/.*[Rr]elease\s*([0-9]*).*/\1/p") + [ $? -ne 0 ] || [ -z "$release" ] && perror_exit "only Fedora image is supported" + + $SUDO dnf --releasever=$release --installroot=$root install -y $@ +} + +run_in_image() { + local image=$1 && shift + local root=${MNTS[$image]} + + echo $SUDO chroot $root /bin/bash -c $@ > /dev/stderr + $SUDO chroot $root /bin/bash -c "$@" +} + +inst_in_image() { + local image=$1 src=$2 dst=$3 + local root=${MNTS[$image]} + + $SUDO cp $src $root/$dst +} + +# If source image is qcow2, create a snapshot +# If source image is raw, convert to raw +# If source image is xz, decompress then repeat the above logic +# +# Won't touch source image +create_image_from_base_image() { + local image=$1 + local output=$2 + local decompressed_image + + local ext="${image##*.}" + if [[ "$ext" == 'xz' ]]; then + echo "Decompressing base image..." + xz -d -k $image + decompressed_image=${image%.xz} + image=$decompressed_image + fi + + local image_fmt=$(qemu-img info $image | sed -n "s/file format:\s*(.*)/\1/p") + if [ "$image_fmt" != "raw" ]; then + if [ "$image_fmt" == "qcow2" ]; then + echo "Source image is qcow2, using snapshot..." + qemu-img create -f qcow2 -b $image $output + else + perror_exit "Unrecognized base image format $image_mnt" + fi + else + echo "Source image is raw, converting to qcow2..." + qemu-img convert -f raw -O qcow2 $image $output + fi + + # Clean up decompress temp image + if [ -n "$decompressed_image" ]; then + rm $decompressed_image + fi +}
Now, by execute `make test-run` in tests/, kexec-tools sanity tests will be performed using VMs. There are currently 3 test cases, for local kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer, do setup as required by the test case, this ensure each test runs in a clean VM.
This framework will install a custom systemd service that starts when system have finished booting, and the service will do basic routine (fetch and set boot counter, etc..), then call the test case which is installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage, one for the test communication and log. The test script will watch the second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song kasong@redhat.com --- tests/Makefile | 8 + .../scripts/build-scripts/test-base-image.sh | 5 + tests/scripts/build-scripts/test-image.sh | 21 +++ tests/scripts/kexec-kdump-test/init.sh | 102 ++++++++++ .../kexec-kdump-test/kexec-kdump-test.service | 9 + tests/scripts/kexec-kdump-test/test.sh | 15 ++ tests/scripts/run-test.sh | 131 +++++++++++++ tests/scripts/spawn-image-shell.sh | 16 ++ tests/scripts/test-lib.sh | 177 ++++++++++++++++++ .../scripts/testcases/local-kdump/0-local.sh | 32 ++++ tests/scripts/testcases/nfs-kdump/0-server.sh | 38 ++++ tests/scripts/testcases/nfs-kdump/1-client.sh | 30 +++ tests/scripts/testcases/ssh-kdump/0-server.sh | 34 ++++ tests/scripts/testcases/ssh-kdump/1-client.sh | 40 ++++ 14 files changed, 658 insertions(+) create mode 100755 tests/scripts/build-scripts/test-image.sh create mode 100755 tests/scripts/kexec-kdump-test/init.sh create mode 100644 tests/scripts/kexec-kdump-test/kexec-kdump-test.service create mode 100755 tests/scripts/kexec-kdump-test/test.sh create mode 100755 tests/scripts/run-test.sh create mode 100755 tests/scripts/spawn-image-shell.sh create mode 100644 tests/scripts/test-lib.sh create mode 100755 tests/scripts/testcases/local-kdump/0-local.sh create mode 100755 tests/scripts/testcases/nfs-kdump/0-server.sh create mode 100755 tests/scripts/testcases/nfs-kdump/1-client.sh create mode 100755 tests/scripts/testcases/ssh-kdump/0-server.sh create mode 100755 tests/scripts/testcases/ssh-kdump/1-client.sh
diff --git a/tests/Makefile b/tests/Makefile index ed3e21a..9f8e939 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -1,3 +1,4 @@ +TEST_CASE ?= BASE_IMAGE ?=
TEST_ROOT := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST)))) @@ -74,6 +75,13 @@ $(TEST_ROOT)/output/test-base-image: $(BUILD_ROOT)/inst-base-image $(KEXEC_TOOLS $(TEST_ROOT)/scripts/build-scripts/test-base-image.sh \ $(KEXEC_TOOLS_RPM)
+test-run: $(TEST_ROOT)/output/test-base-image +ifeq ($(strip $(TEST_CASE)),) + $(TEST_ROOT)/scripts/run-test.sh +else + $(TEST_ROOT)/scripts/run-test.sh --console $(TEST_CASE) +endif + clean: rm -rf $(TEST_ROOT)/build rm -rf $(TEST_ROOT)/output diff --git a/tests/scripts/build-scripts/test-base-image.sh b/tests/scripts/build-scripts/test-base-image.sh index a649306..4154b73 100755 --- a/tests/scripts/build-scripts/test-base-image.sh +++ b/tests/scripts/build-scripts/test-base-image.sh @@ -10,6 +10,11 @@ for _rpm in $@; do fi done
+img_inst $TESTDIR/scripts/kexec-kdump-test/init.sh /kexec-kdump-test/init.sh +img_inst $TESTDIR/scripts/kexec-kdump-test/test.sh /kexec-kdump-test/test.sh +img_inst $TESTDIR/scripts/kexec-kdump-test/kexec-kdump-test.service /etc/systemd/system/kexec-kdump-test.service +img_run_cmd "systemctl enable kexec-kdump-test.service" + img_inst_pkg $TEST_RPMS # Test script should start kdump manually to save time img_run_cmd "systemctl disable kdump.service" diff --git a/tests/scripts/build-scripts/test-image.sh b/tests/scripts/build-scripts/test-image.sh new file mode 100755 index 0000000..d8e907e --- /dev/null +++ b/tests/scripts/build-scripts/test-image.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +. $TESTDIR/scripts/test-lib.sh +TEST_SCRIPT=$1 + +QEMU_CMD="$DEFAULT_QEMU_CMD \ +-serial stdio \ +-serial file:$(get_test_output_file $TEST_SCRIPT) \ +-monitor none \ +-hda $OUTPUT_IMAGE" + +img_add_qemu_cmd() { + QEMU_CMD+=" $@" +} + +source $TEST_SCRIPT + +on_build + +img_inst $TEST_SCRIPT /kexec-kdump-test/test.sh + +echo $QEMU_CMD > $(get_test_qemu_cmd_file $TEST_SCRIPT) diff --git a/tests/scripts/kexec-kdump-test/init.sh b/tests/scripts/kexec-kdump-test/init.sh new file mode 100755 index 0000000..9c1a269 --- /dev/null +++ b/tests/scripts/kexec-kdump-test/init.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env sh +BOOT_ARG="test_boot_count" +_YELLOW='\033[1;33m' +_GREEN='\033[0;32m' +_RED='\033[0;31m' +_NC='\033[0m' # No Color + +if [ -n "$(cat /proc/cmdline | grep "\bno_test\b")" ]; then + exit 0 +fi + +get_test_boot_count() { + local boot_count=$(cat /proc/cmdline | sed -n "s/.*$BOOT_ARG=([0-9]*).*/\1/p") + + if [ -z "$boot_count" ]; then + boot_count=1 + fi + + echo $boot_count +} + +test_output() { + echo $@ > /dev/ttyS1 + echo $@ > /dev/ttyS0 +} + +test_passed() { + echo -e "${_GREEN}TEST PASSED${_NC}" > /dev/ttyS1 + echo -e "${_GREEN}kexec-kdump-test: TEST PASSED${_NC}" > /dev/ttyS0 + + echo $@ > /dev/ttyS1 + echo $@ > /dev/ttyS0 + + shutdown -h 0 + + exit 0 +} + +test_failed() { + echo -e "${_RED}TEST FAILED${_NC}" > /dev/ttyS1 + echo -e "${_RED}kexec-kdump-test: TEST FAILED${_NC}" > /dev/ttyS0 + + echo $@ > /dev/ttyS1 + echo $@ > /dev/ttyS0 + + shutdown -h 0 + + exit 1 +} + +test_abort() { + echo -e "${_YELLOW}TEST ABORTED${_NC}" > /dev/ttyS1 + echo -e "${_YELLOW}kexec-kdump-test: TEST ABORTED${_NC}" > /dev/ttyS0 + + echo $@ > /dev/ttyS1 + echo $@ > /dev/ttyS0 + + shutdown -h 0 + + exit 2 +} + +has_valid_vmcore_dir() { + local path=$1 + local vmcore_dir=$path/$(ls -1 $path | tail -n 1) + + if [ -e $vmcore_dir/vmcore ]; then + makedumpfile --dump-dmesg $vmcore_dir/vmcore $vmcore_dir/vmcore-dmesg.txt.2 + elif [ -e $vmcore_dir/vmcore.flat ]; then + makedumpfile --dump-dmesg $vmcore_dir/vmcore.flat $vmcore_dir/vmcore-dmesg.txt.2 + fi + + [ $? -ne 0 ] && return 1 + + [ ! -e $vmcore_dir/vmcore-dmesg.txt ] && return 1 + + # Checking with `crash` is slow and consume a lot of memory/disk, + # just do a sanity check by check if log are available. + + if diff $vmcore_dir/vmcore-dmesg.txt.2 $vmcore_dir/vmcore-dmesg.txt; then + return 1 + fi + + test_output "Found a valid vmcore in "$vmcore_dir"" + + return 0 +} + +BOOT_COUNT=$(get_test_boot_count) +test_output "Kexec-Kdump-Test Boot #$BOOT_COUNT" + +echo 'fedora' | passwd --stdin root + +test_output "Updating kernel cmdline" +grubby --update-kernel ALL --args $BOOT_ARG=$(expr $BOOT_COUNT + 1) && sync + +test_output "Executing test hook" +source /kexec-kdump-test/test.sh + +on_test; + +test_output "Test exited, system hang for inspect" diff --git a/tests/scripts/kexec-kdump-test/kexec-kdump-test.service b/tests/scripts/kexec-kdump-test/kexec-kdump-test.service new file mode 100644 index 0000000..ba7b11e --- /dev/null +++ b/tests/scripts/kexec-kdump-test/kexec-kdump-test.service @@ -0,0 +1,9 @@ +[Unit] +Description=Kexec Kdump Test Service + +[Service] +ExecStart=/kexec-kdump-test/init.sh +Type=idle + +[Install] +WantedBy=multi-user.target diff --git a/tests/scripts/kexec-kdump-test/test.sh b/tests/scripts/kexec-kdump-test/test.sh new file mode 100755 index 0000000..66ac15d --- /dev/null +++ b/tests/scripts/kexec-kdump-test/test.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env sh +# A test example that do nothing + +# Executed before VM starts +on_build() { + : +} + +# Executed when VM boots +on_test() { + : + # call get_test_boot_count to get boot cound + # call test_passed if test passed + # call test_failed if test passed +} diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh new file mode 100755 index 0000000..cdba7f4 --- /dev/null +++ b/tests/scripts/run-test.sh @@ -0,0 +1,131 @@ +#!/bin/bash + +_kill_all_jobs() { + local _jobs=$(jobs -r -p) + + [ -n "$_jobs" ] && kill $_jobs +} + +trap ' +ret=$?; +_kill_all_jobs +exit $ret; +' EXIT + +trap 'exit 1;' SIGINT + +BASEDIR=$(realpath $(dirname "$0")) +. $BASEDIR/test-lib.sh +TESTCASEDIR="$BASEDIR/testcases" + +console=0 +testcases="" + +while [ $# -gt 0 ]; do + case $1 in + '') + break + ;; + --console ) + console=1 + ;; + -*) + echo "Invalid option $1" + ;; + *) + testcases+=" $1" + ;; + esac + shift; +done + +if [ -z "$testcases" ]; then + echo "==== Starting all tests: ====" + testcases=$(ls -1 $TESTCASEDIR) +else + echo "==== Starting specified tests: ====" +fi +echo ${testcases##*/} +echo + +declare -A results +ret=0 + +for test_case in $testcases; do + echo "======== Running Test Case $test_case ========" + results[$test_case]="<Test Skipped>" + + testdir=$TESTCASEDIR/$test_case + script_num=$(ls -1 $testdir | wc -l) + scripts=$(ls -r -1 $testdir | tr '\n' ' ') + test_outputs="" + read main_script aux_script <<< "$scripts" + + if [ -z "$main_script" ]; then + echo "ERROR: Empty testcase dir $testdir" + continue + fi + + for script in $scripts; do + echo "---- Building image for: $script ----" + echo "-------- Output image is: $(get_test_image $testdir/$script)" + echo "-------- Building log is: $(get_test_image $testdir/$script).log" + + mkdir -p $(dirname $(get_test_image $testdir/$script)) + build_test_image $testdir/$script &> $(get_test_image $testdir/$script).log + + if [ $? -ne 0 ]; then + echo "Failing building image!" + continue 2 + fi + done + + for script in $aux_script; do + echo "---- Starting VM: $script ----" + + script="$testdir/$script" + echo "-------- Qemu cmdline: $(get_test_qemu_cmd_file $script)" + echo "-------- Console log: $(get_test_console_file $script)" + echo "-------- Test log: $(get_test_output_file $script)" + test_outputs+="$(get_test_output_file $script) " + rm -f $(get_test_console_file $script) + rm -f $(get_test_output_file $script) + + $(run_test_sync $script > $(get_test_console_file $script)) & + + sleep 3 + done + + script="$main_script" + echo "---- Starting test VM: $(basename $script) ----" + script="$testdir/$script" + + echo "-------- Qemu cmdline: $(get_test_qemu_cmd_file $script)" + echo "-------- Console log: $(get_test_console_file $script)" + echo "-------- Test log: $(get_test_output_file $script)" + test_outputs+="$(get_test_output_file $script) " + rm -f $(get_test_console_file $script) + rm -f $(get_test_output_file $script) + + if [ $console -eq 1 ]; then + run_test_sync $script | tee $(get_test_console_file $script) + [ -n "$(jobs -p)" ] && kill $(jobs -p) + else + $(run_test_sync $script > $(get_test_console_file $script)) & + watch_test_outputs $test_outputs + fi + + res="$(gather_test_result $test_outputs)" + [ $? -ne 0 ] && ret=$(expr $ret + 1) + results[$test_case]="$res" + + echo -e "-------- Test finished: $test_case $res --------" +done + +echo "======== Test results ========" +for i in ${!results[@]}; do + echo "----------------" + echo -e "$i:\t\t${results[$i]}" +done + +exit $ret diff --git a/tests/scripts/spawn-image-shell.sh b/tests/scripts/spawn-image-shell.sh new file mode 100755 index 0000000..ccfb655 --- /dev/null +++ b/tests/scripts/spawn-image-shell.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash + +BASEDIR=$(realpath $(dirname "$0")) +. $BASEDIR/image-init-lib.sh + +# Base image to build from +BOOT_IMAGE=$1 +if [[ ! -e $BOOT_IMAGE ]]; then + perror_exit "Image '$BOOT_IMAGE' not found" +else + BOOT_IMAGE=$(realpath "$BOOT_IMAGE") +fi + +mount_image $BOOT_IMAGE + +shell_in_image $BOOT_IMAGE diff --git a/tests/scripts/test-lib.sh b/tests/scripts/test-lib.sh new file mode 100644 index 0000000..aff5cd3 --- /dev/null +++ b/tests/scripts/test-lib.sh @@ -0,0 +1,177 @@ +#!/usr/bin/env sh +[ -z "$BASEDIR" ] && BASEDIR=$(realpath $(dirname "$0")) +[ -z "$TESTDIR" ] && TESTDIR=$(realpath $BASEDIR/../) +[ -z "$TEST_BASE_IMAGE" ] && TEST_BASE_IMAGE=$TESTDIR/output/test-base-image + +[[ ! -e $TEST_BASE_IMAGE ]] && echo "Test base image not found." && exit 1 + +DEFAULT_QEMU_CMD="-nodefaults \ +-nographic \ +-smp 2 \ +-m 768M \ +-monitor none" + +_YELLOW='\033[1;33m' +_GREEN='\033[0;32m' +_RED='\033[0;31m' +_NC='\033[0m' # No Color + +get_test_path() { + local script=$1 + local testname=$(basename $(dirname $script)) + local output=$TESTDIR/output/$testname + + echo $output +} + +get_test_entry_name() { + echo $(basename ${1%.*}) +} + +get_test_image() { + local script=$1 + local testout=$(get_test_path $script) + local entry=$(get_test_entry_name $script) + + echo $testout/$entry.img +} + +get_test_qemu_cmd_file() { + local script=$1 + local testout=$(get_test_path $script) + local entry=$(get_test_entry_name $script) + + echo $testout/$entry.qemu_cmd +} + +get_test_qemu_cmd() { + cat $(get_test_qemu_cmd_file $1) +} + +get_test_output_file() { + local script=$1 + local testout=$(get_test_path $script) + local entry=$(get_test_entry_name $script) + + echo $testout/$entry.output +} + +get_test_console_file() { + local script=$1 + local testout=$(get_test_path $script) + local entry=$(get_test_entry_name $script) + + echo $testout/$entry.console +} + +get_test_output() { + local output=$(get_test_output_file $1) + if [ -e "$output" ]; then + cat $(get_test_output_file $1) + else + echo "<No Output>" + fi +} + +build_test_image() { + local script=$1 + local test_image=$(get_test_image $script) + mkdir -p $(dirname $test_image) + + $BASEDIR/build-image.sh \ + $TEST_BASE_IMAGE \ + $test_image \ + $BASEDIR/build-scripts/test-image.sh \ + $script +} + +run_test_sync() { + local qemu_cmd=$(get_test_qemu_cmd $1) + + if [ -n "$qemu_cmd" ]; then + timeout --foreground 10m qemu-kvm $(get_test_qemu_cmd $1) + else + echo "error: test qemu command line is not configured" > /dev/stderr + return 1 + fi +} + +_check_test_result() { + grep "TEST PASSED" $1 2>/dev/null + [ $? -eq 0 ] && return 0 + + grep "TEST FAILED" $1 2>/dev/null + [ $? -eq 0 ] && return 1 + + grep "TEST ABORTED" $1 2>/dev/null + [ $? -eq 0 ] && return 2 + + return 255 +} + +# Print test result and return below value: +# 0: Test passed +# 1: Test failed +# 2: Test aborted, test scripts errored out +# 3: Test exited unexpectely, VM got killed early, or time out +gather_test_result() { + local ret=255 + local res="" + + for i in $@; do + res=$(_check_test_result $i) + ret=$? + + if [ $ret -ne 255 ]; then + echo $res + return $ret + fi + done + + echo "${_RED}TEST RESULT NOT FOUND!${_NC}" + return 3 +} + +# Wait and watch for test result +watch_test_outputs() { + local ret=255 + local res="" + # If VMs are still running, check for test result, if + # test finished, kill remaining VMs + while true; do + if [ -n "$(jobs -r)" ]; then + # VMs still running + for i in $@; do + res=$(_check_test_result $i) + ret=$? + + if [ $ret -ne 255 ]; then + # Test finished, kill VMs + kill $(jobs -p) + break 2 + fi + done + else + # VMs exited + ret=255 + + for i in $@; do + res=$(_check_test_result $i) + ret=$? + + if [ $ret -ne 255 ]; then + break 2 + fi + done + + if [ $ret -eq 255 ]; then + ret=3 + break + fi + fi + + sleep 1 + done + + return $ret +} diff --git a/tests/scripts/testcases/local-kdump/0-local.sh b/tests/scripts/testcases/local-kdump/0-local.sh new file mode 100755 index 0000000..d09c6f0 --- /dev/null +++ b/tests/scripts/testcases/local-kdump/0-local.sh @@ -0,0 +1,32 @@ +on_build() { + : +} + +on_test() { + local boot_count=$(get_test_boot_count) + + if [ $boot_count -eq 1 ]; then + cat << EOF > /etc/kdump.conf +path /var/crash +core_collector makedumpfile -l --message-level 1 -d 31 +EOF + kdumpctl start || test_failed "Failed to start kdump" + + sync + + echo 1 > /proc/sys/kernel/sysrq + echo c > /proc/sysrq-trigger + + elif [ $boot_count -eq 2 ]; then + + if has_valid_vmcore_dir /var/crash; then + test_passed + else + test_failed + fi + + shutdown -h 0 + else + test_failed "Unexpected reboot" + fi +} diff --git a/tests/scripts/testcases/nfs-kdump/0-server.sh b/tests/scripts/testcases/nfs-kdump/0-server.sh new file mode 100755 index 0000000..04f102e --- /dev/null +++ b/tests/scripts/testcases/nfs-kdump/0-server.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env sh + +# Executed before VM starts +on_build() { + img_inst_pkg "nfs-utils dnsmasq" + + img_run_cmd "mkdir -p /srv/nfs/var/crash" + img_run_cmd "echo /srv/nfs 192.168.77.1/24(rw,async,insecure,no_root_squash) > /etc/exports" + img_run_cmd "systemctl enable nfs-server" + + img_run_cmd "echo interface=eth0 > /etc/dnsmasq.conf" + img_run_cmd "echo dhcp-authoritative >> /etc/dnsmasq.conf" + img_run_cmd "echo dhcp-range=192.168.77.50,192.168.77.100,255.255.255.0,12h >> /etc/dnsmasq.conf" + img_run_cmd "systemctl enable dnsmasq" + + img_run_cmd 'echo DEVICE="eth0" > /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo BOOTPROTO="none >> /etc/sysconfig/network-scripts/ifcfg-eth0"' + img_run_cmd 'echo ONBOOT="yes" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo PREFIX="24" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo IPADDR="192.168.77.1" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo TYPE="Ethernet" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + + img_add_qemu_cmd "-nic socket,listen=:8010,mac=52:54:00:12:34:56" +} + +# Executed when VM boots +on_test() { + while true; do + if has_valid_vmcore_dir /srv/nfs/var/crash; then + # Wait a few seconds so client finish it's work to generate a full log + sleep 5 + + test_passed + fi + + sleep 1 + done +} diff --git a/tests/scripts/testcases/nfs-kdump/1-client.sh b/tests/scripts/testcases/nfs-kdump/1-client.sh new file mode 100755 index 0000000..df62463 --- /dev/null +++ b/tests/scripts/testcases/nfs-kdump/1-client.sh @@ -0,0 +1,30 @@ +# Executed before VM starts +on_build() { + img_inst_pkg "nfs-utils" + img_add_qemu_cmd "-nic socket,connect=127.0.0.1:8010,mac=52:54:00:12:34:57" +} + +on_test() { + local boot_count=$(get_test_boot_count) + local nfs_server=192.168.77.1 + + if [ "$boot_count" -eq 1 ]; then + cat << EOF > /etc/kdump.conf +nfs $nfs_server:/srv/nfs +core_collector makedumpfile -l --message-level 1 -d 31 +EOF + + while ! ping -c 1 $nfs_server -W 1; do + : + done + + kdumpctl start || test_failed "Failed to start kdump" + + sync + + echo 1 > /proc/sys/kernel/sysrq + echo c > /proc/sysrq-trigger + else + shutdown -h 0 + fi +} diff --git a/tests/scripts/testcases/ssh-kdump/0-server.sh b/tests/scripts/testcases/ssh-kdump/0-server.sh new file mode 100755 index 0000000..f1e5073 --- /dev/null +++ b/tests/scripts/testcases/ssh-kdump/0-server.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env sh + +# Executed before VM starts +on_build() { + img_add_qemu_cmd "-nic socket,listen=:8010,mac=52:54:00:12:34:56" + + img_run_cmd "echo root:fedora | chpasswd" + img_run_cmd 'sed -i "s/^.*PasswordAuthentication .*$/PasswordAuthentication yes/" /etc/ssh/sshd_config' + img_run_cmd 'sed -i "s/^.*PermitRootLogin .*$/PermitRootLogin yes/" /etc/ssh/sshd_config' + img_run_cmd "systemctl enable sshd" + + img_run_cmd "echo interface=eth0 > /etc/dnsmasq.conf" + img_run_cmd "echo dhcp-authoritative >> /etc/dnsmasq.conf" + img_run_cmd "echo dhcp-range=192.168.77.50,192.168.77.100,255.255.255.0,12h >> /etc/dnsmasq.conf" + img_run_cmd "systemctl enable dnsmasq" + + img_run_cmd 'echo DEVICE="eth0" > /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo BOOTPROTO="none >> /etc/sysconfig/network-scripts/ifcfg-eth0"' + img_run_cmd 'echo ONBOOT="yes" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo PREFIX="24" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo IPADDR="192.168.77.1" >> /etc/sysconfig/network-scripts/ifcfg-eth0' + img_run_cmd 'echo TYPE="Ethernet" >> /etc/sysconfig/network-scripts/ifcfg-eth0' +} + +# Executed when VM boots +on_test() { + while true; do + if has_valid_vmcore_dir /var/crash; then + test_passed + fi + + sleep 1 + done +} diff --git a/tests/scripts/testcases/ssh-kdump/1-client.sh b/tests/scripts/testcases/ssh-kdump/1-client.sh new file mode 100755 index 0000000..d79f00f --- /dev/null +++ b/tests/scripts/testcases/ssh-kdump/1-client.sh @@ -0,0 +1,40 @@ +# Executed before VM starts +on_build() { + img_inst_pkg "sshpass" + img_add_qemu_cmd "-nic socket,connect=127.0.0.1:8010,mac=52:54:00:12:34:57" +} + +on_test() { + local boot_count=$(get_test_boot_count) + local ssh_server=192.168.77.1 + + if [ "$boot_count" -eq 1 ]; then +cat << EOF > /etc/kdump.conf +ssh root@192.168.77.1 +core_collector makedumpfile -l --message-level 1 -d 31 -F +EOF + + ssh-keygen -q -t rsa -N '' -f /root/.ssh/id_rsa <<< y &>/dev/ttyS1 + + while ! ping -c 1 $ssh_server -W 1; do + sleep 1 + done + + while [ -z "$(cat /root/.ssh/known_hosts)" ]; do + ssh-keyscan -H 192.168.77.1 > /root/.ssh/known_hosts + done + + sshpass -p fedora ssh-copy-id root@$ssh_server -f &>/dev/ttyS1 + + sshpass -p fedora kdumpctl propagate &>/dev/ttyS1 + + kdumpctl start || test_failed "Failed to start kdump" + + sync + + echo 1 > /proc/sys/kernel/sysrq + echo c > /proc/sysrq-trigger + else + shutdown -h 0 + fi +}
Hi Kairui,
+has_valid_vmcore_dir() {
- local path=$1
- local vmcore_dir=$path/$(ls -1 $path | tail -n 1)
- if [ -e $vmcore_dir/vmcore ]; then
makedumpfile --dump-dmesg $vmcore_dir/vmcore $vmcore_dir/vmcore-dmesg.txt.2
- elif [ -e $vmcore_dir/vmcore.flat ]; then
makedumpfile --dump-dmesg $vmcore_dir/vmcore.flat $vmcore_dir/vmcore-dmesg.txt.2
- fi
Assume the $vmcore_dir/vmcore* is dumped by makedumpfile, but according to man page of makedumpfile, --dump-dmsg is use with the elf format /proc/vmcore, if it can also work with compressed vmcore then probably the man page just does not describe the case?
- [ $? -ne 0 ] && return 1
- [ ! -e $vmcore_dir/vmcore-dmesg.txt ] && return 1
- # Checking with `crash` is slow and consume a lot of memory/disk,
- # just do a sanity check by check if log are available.
- if diff $vmcore_dir/vmcore-dmesg.txt.2 $vmcore_dir/vmcore-dmesg.txt; then
return 1
- fi
Is it doable to print the vmcore saved path, kernel version etc, so that people can manually test the crash utility?
Thanks Dave
On Wed, Jul 22, 2020 at 11:37 AM Dave Young dyoung@redhat.com wrote:
Hi Kairui,
+has_valid_vmcore_dir() {
local path=$1
local vmcore_dir=$path/$(ls -1 $path | tail -n 1)
if [ -e $vmcore_dir/vmcore ]; then
makedumpfile --dump-dmesg $vmcore_dir/vmcore $vmcore_dir/vmcore-dmesg.txt.2
elif [ -e $vmcore_dir/vmcore.flat ]; then
makedumpfile --dump-dmesg $vmcore_dir/vmcore.flat $vmcore_dir/vmcore-dmesg.txt.2
fi
Assume the $vmcore_dir/vmcore* is dumped by makedumpfile, but according to man page of makedumpfile, --dump-dmsg is use with the elf format /proc/vmcore, if it can also work with compressed vmcore then probably the man page just does not describe the case?
It works well with the compressed format and flat and elf format, maybe makedumpfile man page need a update. I'll check the code and send an update.
[ $? -ne 0 ] && return 1
[ ! -e $vmcore_dir/vmcore-dmesg.txt ] && return 1
# Checking with `crash` is slow and consume a lot of memory/disk,
# just do a sanity check by check if log are available.
if diff $vmcore_dir/vmcore-dmesg.txt.2 $vmcore_dir/vmcore-dmesg.txt; then
return 1
fi
Is it doable to print the vmcore saved path, kernel version etc, so that people can manually test the crash utility?
Yes, that's very doable. I didn't add that in the test routine because debuginfo is too large and take a lot of time to install it and all the dependencies, and will need to resize the VM image to contain that, and also need to increase the VM memory to start the crash utility. But give user a change to review it later is a good idea.
Thanks Dave
On 07/22/20 at 01:02pm, Kairui Song wrote:
On Wed, Jul 22, 2020 at 11:37 AM Dave Young dyoung@redhat.com wrote:
Hi Kairui,
+has_valid_vmcore_dir() {
local path=$1
local vmcore_dir=$path/$(ls -1 $path | tail -n 1)
if [ -e $vmcore_dir/vmcore ]; then
makedumpfile --dump-dmesg $vmcore_dir/vmcore $vmcore_dir/vmcore-dmesg.txt.2
elif [ -e $vmcore_dir/vmcore.flat ]; then
makedumpfile --dump-dmesg $vmcore_dir/vmcore.flat $vmcore_dir/vmcore-dmesg.txt.2
fi
Assume the $vmcore_dir/vmcore* is dumped by makedumpfile, but according to man page of makedumpfile, --dump-dmsg is use with the elf format /proc/vmcore, if it can also work with compressed vmcore then probably the man page just does not describe the case?
It works well with the compressed format and flat and elf format, maybe makedumpfile man page need a update. I'll check the code and send an update.
[ $? -ne 0 ] && return 1
[ ! -e $vmcore_dir/vmcore-dmesg.txt ] && return 1
# Checking with `crash` is slow and consume a lot of memory/disk,
# just do a sanity check by check if log are available.
if diff $vmcore_dir/vmcore-dmesg.txt.2 $vmcore_dir/vmcore-dmesg.txt; then
return 1
fi
Is it doable to print the vmcore saved path, kernel version etc, so that people can manually test the crash utility?
Yes, that's very doable. I didn't add that in the test routine because debuginfo is too large and take a lot of time to install it and all the dependencies, and will need to resize the VM image to contain that, and also need to increase the VM memory to start the crash utility. But give user a change to review it later is a good idea.
Ok, thanks. I think crash test no need to go into the vm, it can be done in host side since it only needs vmcore and debuginfo. We can start with tell people how to find the vmcore captured and then do some crash tool test manually.
In the future, maybe we can try to hook some host side crash tool test with those vmcores.
--- tests/README | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 tests/README
diff --git a/tests/README b/tests/README new file mode 100644 index 0000000..6ab63a8 --- /dev/null +++ b/tests/README @@ -0,0 +1,65 @@ +===================== +Kexec Kdump Self-test +===================== + + +Introduction +============ +The self-tests here are useful for quick sanity tests for new patches, and also helpful for debugging issues. + + +How it works +============ +All tests are run within VMs using qemu. By default, VM images are based on Fedora Cloud image, and the image for each test run is a layered qcow2 snapshot on top of the base image. +Test images are managed by Makefile, so if there are any code change in the kexec-tools repository, `make` command will detect that and only rebuild the top image layer. This makes the test runs boot fast and each test run is clean. + + +Basic usage +=========== +Before you start, you can make the self-tests use your own base image by running following command: + +`make clean && make BASE_IMAGE=<path/to/your/image>` + +This is helpful if you have a slow network, else self-test will try to download the cloud image from Fedora's official website using `wget`. + +- Use the following command to run all tests: + $ make test-run + + All available tests will be executed. + + Test artifacts are stored in output/<testcase> + +- For easier debugging, you can run only on test with the following command: + $ make TEST_CASE=<testcase> test-run + + This way, VM's console is directly connected to stdin/out so debugging will be easier. + If there are multiple VMs used in a test case, the VM performing actual kdump/kexec operation will be connected to stdin/out. + +Test Cases +========== +Each test case is a folder under scripts/testcases/, a test case folder will contain at least one executable shell script, and each script should contain two functions: "on_build" and "on_test". + +"on_build" is called when building the test image, which can instruct the self-test framework to install packages or create files, etc. +"on_test" is called when VM finished booting, which can get the boot count by calling "get_test_boot_count" and determine what to do. It should call "test_passed" on success, and call "test_failed" on failure. "test_aborted" is called when unexpected behavior occurs. + +When there are multiple scripts in a single test case folder, they will spawn VMs in lexical order, and the last VM is considered the VM performing the actual test. Other VMs could be hosting test required service. This is useful for the network dump test. However, "test_passed" or "test_failed" or "test_aborted" could be called in any of these VMs, so during network kdump test, the dump target can also terminate the test and mark it passed when a valid vmcore is detected. + + +Debugging +========= + +- When the test VM boots, you can append "no_test" to kernel args in grub, which tells the test services to quit early. + +- You can launch the VMs manually or inspect the image after ran a test. + + Test images are located as: + + output/<testcase>/<vm-name>.img + + Test images' corresponding qemu command are located as: + + output/<testcase>/<vm-name>.qemu_cmd + + To repeat/debug a test manually, you should launch all VMs in output/<testcase> menu in lexical order, and append 'no_test' in the last VM's grub cmdline, then VM will hang on login prompt, login with root/fedora. Test script is located as /kexec-kdump-test/test.sh + +- If you just want to inspect the images file content, you can also use scripts/spawn-image-shell.sh <test-image> to spawn a shell in the image quickly.