解决 libguestfs: error: /usr/libexec/qemu-kvm exited with error status 1 的问题

我在使用 guestfish 命令的时候遇到了这个错误。

# guestfish add /dev/vm_storage_pool_vg/vm01_data : run : mount /dev/datavg/home / : find /
libguestfs: error: /usr/libexec/qemu-kvm exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

但是很奇怪的是,敲两次 run 就好了:

# guestfish -a /dev/vm_storage_pool_vg/vm01_data

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: ‘help’ for help on commands
‘man’ to read the manual
‘quit’ to quit the shell

><fs> run
libguestfs: error: /usr/libexec/qemu-kvm exited with error status 1.
To see full error messages you may need to enable debugging.
See http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
><fs> run
><fs> mount /dev/datavg/home /

debug 一下:

# LIBGUESTFS_DEBUG=1 time guestfish -a /dev/null run
libguestfs: create: flags = 0, handle = 0x17f64a0
libguestfs: launch: attach-method=appliance
libguestfs: launch: tmpdir=/tmp/libguestfs1ssRuO
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: command: run: febootstrap-supermin-helper
libguestfs: command: run: \ –verbose
libguestfs: command: run: \ -f checksum
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ x86_64
supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null)
supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d
checking modpath /lib/modules/2.6.32-279.el6.x86_64 is a directory
picked vmlinuz-2.6.32-279.el6.x86_64 because modpath /lib/modules/2.6.32-279.el6.x86_64 exists
supermin helper [00000ms] finished creating kernel
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/base.img
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/daemon.img
supermin helper [00000ms] visiting /usr/lib64/guestfs/supermin.d/hostfiles
supermin helper [00016ms] visiting /usr/lib64/guestfs/supermin.d/init.img
supermin helper [00016ms] visiting /usr/lib64/guestfs/supermin.d/udev-rules.img
supermin helper [00016ms] adding kernel modules
supermin helper [00041ms] finished creating appliance
libguestfs: checksum of existing appliance: 60ae12cd12306a513e0caec33c626dd574c942d10774918ace4779651ba62663
libguestfs: [00048ms] begin testing qemu features
libguestfs: command: run: /usr/libexec/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/libexec/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 0.12
libguestfs: command: run: /usr/libexec/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: error: /usr/libexec/qemu-kvm exited with error status 1, see debug messages above
libguestfs: closing guestfs handle 0x17f64a0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfs1ssRuO
0.03user 0.06system 0:00.09elapsed 101%CPU (0avgtext+0avgdata 23072maxresident)k
0inputs+0outputs (0major+7951minor)pagefaults 0swaps

手动敲一下报错的命令:

# /usr/libexec/qemu-kvm -nographic -machine accel=kvm:tcg -device \?
qemu-kvm: -machine: invalid option

看一下 qemu-kvm 的版本:

# /usr/libexec/qemu-kvm -version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard

可能是版本低的问题,先升级 qemu-kvm:

yum -y install qemu-kvm

再看一下版本:

# /usr/libexec/qemu-kvm -version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2-2.479.el6), Copyright (c) 2003-2008 Fabrice Bellard

此时 /usr/libexec/qemu-kvm -nographic -machine accel=kvm:tcg -device \? 已经可以正常运行了,问题解决。

 

从镜像中 exclude 文件或目录, 缩减镜像大小

我的目标是从在基于镜像创建虚拟机的时候 exclude 部分文件或目录,使得创建虚机更快。本来我想如果 qemu-image convert 的时候支持 exclude,那就简单了,但是不支持,查了更多资料以后,我发现 guestfish 的 tar-out 支持 exclude,但是出来的是 tar 包,而不是 image 了(virt-tar-out 这个命令和 tar-out 类似,但是不支持 exclude)。


下面举个栗子,记录一下过程。

我的 domain 是 vm1, 系统盘名称叫 vm1,数据盘(/home)叫 vm1_data。

首先,对于系统盘的 vm1,我打算不使用 tar-out 打包出来,而是还是用 qemu-image convert 命令转换成 qcow2 格式,即使 vm1 已经是 qcow2,转换之后的 image 大小也可能减小,而且我使用 -c 选项启用压缩,转换之后的 image 大概 900M,大小我能接受。

下面是完整的转换命令:

qemu-img convert -c -O qcow2 vm1 /mfs/vm1

这种方法有个问题,如果系统盘不止 vm1,还有其他盘,会有问题,但是这种情况几乎没有,暂不考虑。


对于数据盘 vm1_data,不用 image ,使用 tar-out,而且我准备 exclude 出名字为 log 或者 logs 的目录,所以我先要获取 log 或者 logs 的目录路径。

在 vm1 关机情况下,对整个 domain 查找 /home 下 log 或 logs 的目录:

guestfish -d vm1 -i sh \
"find /home -type d -name log -o -name logs "

结果是:

/home/op/assets_agent_v2/logs
/home/ossec/ossec/logs

然后,用 tar-out 打包出来(启用压缩),格式如下:

guestfish -d vm1 \
run : mount /dev/datavg/home / : \
tar-out / home.tgz compress:gzip \
"excludes:op/assets_agent_v2/logs/* ossec/ossec/logs/*"

home.tgz 即是数据盘中去除日志的目录,当然在系统的实现中,我们支持自定义 exclude 目录。


在开机状态下,不能使用 -d vm1,只能用 add vm1_data 这种格式,此次 find 不支持 -type d 类似的参数,只能使用简单的 find 目录:

guestfish add vm1_data \
: run : mount /dev/datavg/home / : find /

这条命令会显示所有的文件和目录,如果想去重 log 或者 logs 目录,可以做聚合。

然后 tar-out 命令类似下面:

guestfish add vm1_data \
: run : mount /dev/datavg/home / : \
tar-out / home.tgz compress:gzip \
"excludes:op/assets_agent_v2/logs/* ossec/ossec/logs/*"

由于在开机状态下,我们指定了 /home 的 image,如果 /home 有多个 image,那么这种情况就需要关机了,系统中应该判断,如果有多个数据盘,需提示关机。


那么,对于 tar-out 出来的包,怎么「塞进」新的数据盘呢。

先创建 vm2_data:

virsh vol-create-as –pool vm_storage_pool \
–name vm2_data –capacity 10G –allocation 1G \
–format qcow2

然后创建 lvm 和文件系统,和 vm1_data 保持一致:

virt-format -a vm2_data –lvm=/dev/datavg/home \
–filesystem=ext4

查看 lvm group:

# virt-filesystems -a vm2_data –volume-groups
/dev/datavg

查看 physical volumes:

# virt-filesystems -a vm2_data –physical-volumes
/dev/sda1

查看文件系统:

# virt-filesystems -a vm2_data -l

Name Type VFS Label Size Parent
/dev/datavg/home filesystem ext4 – 10733223936 –

准备工作完成了,可以 tar-in 数据了:

guestfish add vm2_data \
: run : mount /dev/datavg/home / : \
tar-in home.tgz / compress:gzip

如果 home.tgz 通过 http 获取,可以这样:

curl http://nosa.me/vm2_home.tgz | \
guestfish add vm2_data : run : \
mount /dev/datavg/home / : \
tar-in – / compress:gzip

最后,借贵地记录一下 allocation 和 capacity 的说明。

allocation
Providing the total storage allocation for the volume. This may be smaller than the logical capacity if the volume is sparsely allocated. It may also be larger than the logical capacity if the volume has substantial metadata overhead. This value is in bytes. If omitted when creating a volume, the volume will be fully allocated at time of creation. If set to a value smaller than the capacity, the pool has the option of deciding to sparsely allocate a volume. It does not have to honour requests for sparse allocation though. Different types of pools may treat sparse volumes differently. For example, the logical pool will not automatically expand volume’s allocation when it gets full; the user is responsible for doing that or configuring dmeventd to do so automatically.By default this is specified in bytes, but an optional attribute unit can be specified to adjust the passed value. Values can be: ‘B’ or ‘bytes’ for bytes, ‘KB’ (kilobytes, 103 or 1000 bytes), ‘K’ or ‘KiB’ (kibibytes, 210 or 1024 bytes), ‘MB’ (megabytes, 106 or 1,000,000 bytes), ‘M’ or ‘MiB’ (mebibytes, 220 or 1,048,576 bytes), ‘GB’ (gigabytes, 109 or 1,000,000,000 bytes), ‘G’ or ‘GiB’ (gibibytes, 230 or 1,073,741,824 bytes), ‘TB’ (terabytes, 1012 or 1,000,000,000,000 bytes), ‘T’ or ‘TiB’ (tebibytes, 240 or 1,099,511,627,776 bytes), ‘PB’ (petabytes, 1015 or 1,000,000,000,000,000 bytes), ‘P’ or ‘PiB’ (pebibytes, 250 or 1,125,899,906,842,624 bytes), ‘EB’ (exabytes, 1018 or 1,000,000,000,000,000,000 bytes), or ‘E’ or ‘EiB’ (exbibytes, 260 or 1,152,921,504,606,846,976 bytes). Since 0.4.1, multi-character unit since 0.9.11
capacity
Providing the logical capacity for the volume. This value is in bytes by default, but a unit attribute can be specified with the same semantics as for allocationThis is compulsory when creating a volume. Since 0.4.1

用 guestfish 修改虚拟机镜像里的文件

接上文,在新宿主机上生成的新虚拟机为 vm2 ,镜像文件分别为 /vm_sp/vm2.img (/ 分区)和 /vm_sp/vm2_data.img (/home分区,做数据盘),格式都是 qcow2,/vm_sp/ 是基于 目录的存储池。

 

首先先把 vm2 关机,否则无法修改 虚拟机内容。

 

安装 guestfish 命令
# yum -y install guestfish

 

我们先来看看单条命令,比如你只想修改主机名,可以这样:

[shell]
guestfish -d vm2 -i sh \
"sed -i ‘s/HOSTNAME.*/HOSTNAME=test0.hlg01/g’ /etc/sysconfig/network"
[/shell]

-d 表示 指定 虚拟机 域,-i 会自动 挂载 域 的磁盘,当然也可以不用 -d 而用 -a ,表示直接指定一个 img 文件。

另外,修改文件也可以用 virt-edt 命令,但是没上条命令那么通用,上条命令所有的 shell 都能执行。
# virt-edit -d $name /etc/sysconfig/network -e “s/HOSTNAME=.*/HOSTNAME=test0.hlg01/g

 

再来看看交互情况下的用法。
# guestfish -d vm2

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: ‘help’ for help on commands
‘man’ to read the manual
‘quit’ to quit the shell

><fs> run
><fs> list-filesystems
/dev/sda1: ext3
/dev/sda2: swap
/dev/datavg/home: ext4
/dev/domovg/root: ext4
><fs> mount /dev/domovg/root /
><fs> mount /dev/datavg/home /home/
><fs> cat /etc/fstab

/dev/mapper/domovg-root / ext4 defaults,data=ordered 1 1
UUID=7415a5d3-bd20-4a08-9288-66c49ab20b8d /boot ext3 defaults,data=ordered,nodev,nosuid,noexec 1 2
UUID=df3f6ed1-40e7-4da4-b481-465c7ac5fc17 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/datavg/home /home ext4 defaults,data=ordered,nodev,nosuid 1 2

><fs> cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.
# PCI device 0x1af4:0x1000 (virtio-pci)
SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?*”, ATTR{address}==”52:54:00:b0:4a:a1″, ATTR{type}==”1″, KERNEL==”eth*”, NAME=”em2″

# PCI device 0x1af4:0x1000 (virtio-pci)
SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?*”, ATTR{address}==”52:54:00:6a:e8:21″, ATTR{type}==”1″, KERNEL==”eth*”, NAME=”em1″

><fs> cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=test0.hlg01

><fs> cat /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=”em2″
BOOTPROTO=”static”
DNS1=”10.0.12.234″
GATEWAY=”10.19.26.1″
IPADDR=”10.19.26.101″
NETMASK=”255.255.255.0″
ONBOOT=”yes”
TYPE=”Ethernet”

><fs> ls /home/
lost+found

 

上面的交互不易于自动化,这里我们可以编写脚本,如下:

[shell]

#!/bin/bash

name=$1
ip=$2
hostname=$3
hwaddr_em2=$4
hwaddr_em1=$5

netmask=255.255.255.0

x=`echo $ip | awk -F. ‘{print $1"."$2"."$3}’`
gateway="$x.1"

# 定义虚拟机, 否则无法根据域修改镜像.
virsh define /etc/libvirt/qemu/$name.xml || exit 1

# 修改系统镜像, 包括:
# 1. 修改主机名;
# 2. 修改网卡设置(增加HWADDR的原因是系统可以把eth变成em);
# 3. 修改路由信息;
# 4. 删除/etc/udev/rules.d/70-persistent-net.rules;
# 5. 删除/var/lib/puppet;
# 6. 删除/etc/sn;
# 然后启动.
guestfish –rw -d $name <<_EOF_
run
mount /dev/domovg/root /
mount /dev/datavg/home /home/
command "sed -i ‘s/HOSTNAME=.*/HOSTNAME=${hostname}/g’ /etc/sysconfig/network"

write /etc/sysconfig/network-scripts/ifcfg-em2 "DEVICE=em2\nHWADDR=${hwaddr_em2}\nBOOTPROTO=static\nIPADDR=$ip\nNETMASK=$netmask\nONBOOT=yes\nTYPE=Ethernet"

write /etc/sysconfig/network-scripts/ifcfg-em1 "DEVICE=em1\nHWADDR=${hwaddr_em1}"

write /etc/sysconfig/network-scripts/route-em2 "192.168.0.0/16 via 10.19.28.1\n10.0.0.0/8 via 10.19.28.1\n100.64.0.0/16 via 10.19.28.1\n0.0.0.0/0 via 10.19.28.1"

command "/bin/rm -rf /etc/udev/rules.d/70-persistent-net.rules"
command "/bin/rm -rf /var/lib/puppet/"
command "/bin/rm -rf /etc/sn"
_EOF_

[/shell]

说明一下,command 命令里 echo xxx > file 是无效的。