vendredi 23 juin 2017

Shell looping vs xargs

For a very long time I was using the following construct to execute a command on foo, bar, and baz (all typed into an interactive shell prompt)

for F in foo bar baz; do command $F ; done

Now I finally got the -n option of xargs (or --max-args if you're using GNU) which limits to number of parameters to pass to the next command. So i can replace the previous command with:

echo foo bar baz | xargs -n 1 command

Here the -n option tells xargs to give at max one parameter to proceed, and create a new command with the next parameter.

As a bonus, instead of constructing an array of numbers with

seq 1 10

don't forget bash and zsh brace expansion

{1..10}

vendredi 16 juin 2017

Tomcat7 not starting after Jessie->Stretch ugrade

I was wondering why the tomcat7 debian package would not start after the upgrade.

systemctl status tomcat7 output would look quite OK

systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
   Loaded: loaded (/etc/init.d/tomcat7; generated; vendor preset: enabled)
   Active: active (exited) since Fri 2017-06-16 10:10:34 CEST; 29min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 536 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/tomcat7.service

Jun 16 10:10:34 new-ada systemd[1]: Starting LSB: Start Tomcat....
Jun 16 10:10:34 new-ada systemd[1]: Started LSB: Start Tomcat..
Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-dae…tory)
Jun 16 10:10:34 new-ada tomcat7[536]:  (already running).
Hint: Some lines were ellipsized, use -l to show in full.

But not java process was created

pgrep -lf java
echo $?

so something was obviously wrong.

But the devil is in the details, can you see this truncated line ?

Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-dae…tory)

If you run it in full with systemctl status -l tomcat7 it expends to

Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-daemon: unable to stat /usr/lib/jvm/java-7-openjdk-amd64//bin/java (No such file or directory)

Now I get it, the upgrade removed the openjdk-7-jre-headless package.Time to switch to openjdk8 ! The package openjdk-8-jre-headless was already installed, so I justed needed to set JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ in /etc/default/tomcat7

Note for the future:

  • Why was this package removed ?
  • Should I configure the systemctl status to show ellipsized lined by default ?

jeudi 15 juin 2017

Render HTML from standard input with w3m

From a very long time, I was using the links browser as my default browser. It allows quick testing of a web site with a simple

links -dump debian.org

Alas links does not support rendering html from standard input, so I finally switched to w3m, handles that well. So I could do:

cat bla.html | w3m -T text/html -dump

Using curl and any member of the Unix Toolkit gang, you see which possibilities it opens. Oh and UTF-8 works well too !

links -dump https://www.debian.org/index.ru.html | grep свободная
Debian — это свободная операционная система (ОС) для вашего компьютера.

свободная that’s Russian for Free.

Testing upgrades on a backup system

Like many Debian users, I am planning very soon to upgrade to my own personal server ada to Debian Stretch.
Since I do a full rsync backup of the server to a different location, I was wondering if it was possible to use this backup root fs in a container to test the upgrade. Turns is out it works very well with systemd-nspawn

Copy the backup to a separate dir in /var/lib/machines
# /var/lib/machines is where machinectl look for images
# see https://superuser.com/a/307542 for rsync flags detial
rsync -axHAX --info=progress2 --numeric-ids --whole-file /home/manu/Projects/backups/ada_rootfs/ /var/lib/machines/ada/
For networking this container, I installed libvirtd, which comes with a DHCP server serving private IPs on the virbr0 bridge.
systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-06-08 23:15:33 CEST; 6 days ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2609 (libvirtd)
    Tasks: 18 (limit: 4915)
   Memory: 53.0M
      CPU: 2.448s
   CGroup: /system.slice/libvirtd.service
           ├─2609 /usr/sbin/libvirtd
           ├─4868 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile
           └─4869 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile

juin 09 11:34:34 django dnsmasq-dhcp[4868]: not giving name ada to the DHCP lease of 192.168.122.
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPREQUEST(virbr0) 192.168.122.17 06:91:4f:92:3b:a8
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPACK(virbr0) 192.168.122.17 06:91:4f:92:3b:a8 ada
Start the container, giving a a Virtual Ethernet Link connected to virbr0
systemd-nspawn --directory /var/lib/machines/ada --machine ada --network-bridge=virbr0 --boot --network-veth
(observe a nice booting sequence, up to a login prompt)
Login inside the container and configure the network
# snippet for /etc/network/interfaces
# the Virtual Ethernet Link appears as host0 inside the container
auto host0
iface host0 inet dhcp
# restart the network
systemctl restart networking
Find out from the host side which IP was assigned
machinectl list
MACHINE CLASS     SERVICE        OS     VERSION ADDRESSES
ada     container systemd-nspawn debian 8       192.168.122.17...

1 machines listed.

Inspect from the host system if the Apache virtual hosts are running
curl --silent --header 'Host: meinoesterreich.ist-ur.org' http://192.168.122.17/titelblatt | w3m -T text/html -dump
Happy Upgrades !

vendredi 6 janvier 2017

Hard disk performance measured with fio

Since some times now, I collect the performance of various storages devices, using fio, the flexible Input/Ouput tester.

With FIO you can collect sequential and IOPS statistics. Sequential thoughput is what you would need if for instance extracting a large tar archive. or copying a disk image. IOPS, where I measure a mix of 70% reads, 30% write, would be what you need for an OLTP workload.

Sequential writes:

Let's try from the slowest to the fastest, using the command
fio  --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
The ouput from this test should match the column wMB/s of the command iostat -mx 2, executed during the test. 

1) SD Card Sandisk Extreme Pro : 10MB/s
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

64kwrite: (groupid=0, jobs=1): err= 0: pid=12583: Sat Jan  7 15:24:55 2017
  write: io=512000KB, bw=11702KB/s, iops=182, runt= 43754msec

 ...
    bw (KB  /s): min= 1280, max=13184, per=99.97%, avg=11697.93, stdev=2798.43


2) A 5400rpm disk, mounted over a USB 3.0 port: 35 MB/s
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

...
64kwrite: (groupid=0, jobs=1): err= 0: pid=13635: Fri Jan  6 14:55:11 2017
  write: io=2056.1MB, bw=35035KB/s, iops=547, runt= 60120msec
...
    bw (KB  /s): min= 7168, max=61696, per=100.00%, avg=35101.89, stdev=10726.73
...


3) Samusung SSD 840 over SATA port: 360MB/s

 64kwrite: (groupid=0, jobs=1): err= 0: pid=16095: Fri Jan  6 15:17:56 2017
 write: io=9216.0MB, bw=365400KB/s, iops=5709, runt= 25827msec
...
    bw (KB  /s): min=345472, max=396160, per=99.87%, avg=364934.33, stdev=10505.38

Random Readwrites:


Using the command:
 fio  --size=2G --bs=4k --rw=randrw --rwmixread=70 --direct=1 --runtime=60 --numjobs=4 --name=4k7030test --group_reporting
The ouput from this test should match the column r/s and w/s of the command iostat -mx 2, executed during the test.

1) SD Card Sandisk Extreme Pro: 364 iops reading, 154 writing
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

4k7030test: (groupid=0, jobs=4): err= 0: pid=12724: Sat Jan  7 15:29:45 2017
  read : io=87488KB, bw=1457.8KB/s, iops=364, runt= 60017msec
  write: io=37108KB, bw=633130B/s, iops=154, runt= 60017msec



 2) A 5400rpm disk, over a USB 3.0 port: 37 iops reading, 15 writing
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

4k7030test: (groupid=0, jobs=4): err= 0: pid=12756: Sat Jan  7 15:31:02 2017
  read : io=9044.0KB, bw=154135B/s, iops=37, runt= 60084msec

  write: io=3796.0KB, bw=64694B/s, iops=15, runt= 60084msec


3) Samusung SSD 840 over SATA port: 22484 iops reading, 9648 writing

4k7030test: (groupid=0, jobs=4): err= 0: pid=12265: Sat Jan  7 15:15:54 2017
  read : io=5269.9MB, bw=89937KB/s, iops=22484, runt= 60001msec

  write: io=2261.4MB, bw=38594KB/s, iops=9648, runt= 60001msec


A very good summ up of fio benchmarks is also available on the Thomas Krenn wiki (in german)

samedi 24 décembre 2016

Quick debugging of a Linux printer via cups command line tools

Step by step cups debugging ( here with a network printer)

Which printer queue do I have configured ?
lpstat -p
printer epson is idle.  enabled since Sat Dec 24 13:18:09 2017
#here I have a printer called 'epson", doing nothing, that the cups daemon considers as enabled

Which connection am I using to get to this printer ?
lpstat -v
device for epson: lpd://epson34dea0.local:515/PASSTHRU
# here the locally configured 'epson' printer queue is backed by a network device at the adress epson34dea0.local, to which I am sending my print jobs via the lpd protocol

Is my printer ready ?
lpq
epson is ready
no entries

# here my local print queue 'epson' is accepting print jobs ( which does not say anything about the physical device, it might be offline

If here you local print queue 'epson' is not ready, you can try to reenable it in the cups system with:

sudo cupsenable epson

If you notice that the printer is disabled all the time, because for instance of a flaky network, you can edit /etc/cups/printers.conf and change the ErrorPolicy for each printer from stop-printer to retry-job.
It should be also possible to set this parameter in cupsd.conf

mercredi 23 novembre 2016

Small Summary of the Debian cloud sprint in Seattle

Last week I was In Seattle for the first Debian cloud team sprint The aim of it was to streamline the producing of official Debian ready to use images for various cloud providers.
The three biggest cloud providers were there (Google, Amazon, Microsoft Azure), and my humble self was here because of the work I have been doing producing Debian base boxes for Vagrant.
 During the three days the spring took we did the following:
* Reviewed the current state of cloud images and their build tools
After a demonstration of the vmdebootstrap and fai-diskimage creation tools, agreed we should try fai-diskimage as a default tool. Since a consensus here seemed to be reached I refrained myself from advocating packer or virt-install too much 
 * Reviewed the current state of cloud-init in Debian and 
considering its non working state in stable, thought about an upgrade in Debian Stable
Agreed about having unattended super axes by default inside the team,and proposed it to debian-devel
 * Reviewed all the bugs of the debian-cloud virtual package.
As with probably every so sprint, getting to bring all the people in one room brings a lot of synergies which couldn't happen on a mailing list. Also we noticed a lot of interest for Debian from the respective cloud providers which is interesting considering Debian is "only" a community based distribution.
Seattle is also a nice town, with lots of pine trees and bicycle lanes, looked to me like Canada, a troll statue(?) though I didn't have the time to see that much of it.