lundi 26 juin 2017

Java DOM from XML is not namespace aware by default

Something which costed me a couple of headaches using getElementsByTagNameNS with a DocumentBuilderFactory from the java.xml.parsers package.

You find in the class properties:

    private boolean namespaceAware = false;

and I thought XML namespaces where the bread and butter of XML ...

So when building a DOM from an XML document you need to change the defaults of the factory like:

    DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
    factory.setNamespaceAware(true);
    DocumentBuilder docBuilder = factory.newDocumentBuilder();

vendredi 23 juin 2017

Shell looping vs xargs

For a very long time I was using the following construct to execute a command on foo, bar, and baz (all typed into an interactive shell prompt)

for F in foo bar baz; do command $F ; done

Now I finally got the -n option of xargs (or --max-args if you're using GNU) which limits to number of parameters to pass to the next command. So i can replace the previous command with:

echo foo bar baz | xargs -n 1 command

Here the -n option tells xargs to give at max one parameter to proceed, and create a new command with the next parameter.

As a bonus, instead of constructing an array of numbers with

seq 1 10

don't forget bash and zsh brace expansion

{1..10}

vendredi 16 juin 2017

Tomcat7 not starting after Jessie->Stretch ugrade

I was wondering why the tomcat7 debian package would not start after the upgrade.

systemctl status tomcat7 output would look quite OK

systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
   Loaded: loaded (/etc/init.d/tomcat7; generated; vendor preset: enabled)
   Active: active (exited) since Fri 2017-06-16 10:10:34 CEST; 29min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 536 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/tomcat7.service

Jun 16 10:10:34 new-ada systemd[1]: Starting LSB: Start Tomcat....
Jun 16 10:10:34 new-ada systemd[1]: Started LSB: Start Tomcat..
Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-dae…tory)
Jun 16 10:10:34 new-ada tomcat7[536]:  (already running).
Hint: Some lines were ellipsized, use -l to show in full.

But not java process was created

pgrep -lf java
echo $?

so something was obviously wrong.

But the devil is in the details, can you see this truncated line ?

Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-dae…tory)

If you run it in full with systemctl status -l tomcat7 it expends to

Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-daemon: unable to stat /usr/lib/jvm/java-7-openjdk-amd64//bin/java (No such file or directory)

Now I get it, the upgrade removed the openjdk-7-jre-headless package.Time to switch to openjdk8 ! The package openjdk-8-jre-headless was already installed, so I justed needed to set JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ in /etc/default/tomcat7

Note for the future:

  • Why was this package removed ?
  • Should I configure the systemctl status to show ellipsized lined by default ?

jeudi 15 juin 2017

Render HTML from standard input with w3m

From a very long time, I was using the links browser as my default browser. It allows quick testing of a web site with a simple

links -dump debian.org

Alas links does not support rendering html from standard input, so I finally switched to w3m, handles that well. So I could do:

cat bla.html | w3m -T text/html -dump

Using curl and any member of the Unix Toolkit gang, you see which possibilities it opens. Oh and UTF-8 works well too !

links -dump https://www.debian.org/index.ru.html | grep свободная
Debian — это свободная операционная система (ОС) для вашего компьютера.

свободная that’s Russian for Free.

Testing upgrades on a backup system

Like many Debian users, I am planning very soon to upgrade to my own personal server ada to Debian Stretch.
Since I do a full rsync backup of the server to a different location, I was wondering if it was possible to use this backup root fs in a container to test the upgrade. Turns is out it works very well with systemd-nspawn

Copy the backup to a separate dir in /var/lib/machines
# /var/lib/machines is where machinectl look for images
# see https://superuser.com/a/307542 for rsync flags detial
rsync -axHAX --info=progress2 --numeric-ids --whole-file /home/manu/Projects/backups/ada_rootfs/ /var/lib/machines/ada/
For networking this container, I installed libvirtd, which comes with a DHCP server serving private IPs on the virbr0 bridge.
systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-06-08 23:15:33 CEST; 6 days ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2609 (libvirtd)
    Tasks: 18 (limit: 4915)
   Memory: 53.0M
      CPU: 2.448s
   CGroup: /system.slice/libvirtd.service
           ├─2609 /usr/sbin/libvirtd
           ├─4868 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile
           └─4869 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile

juin 09 11:34:34 django dnsmasq-dhcp[4868]: not giving name ada to the DHCP lease of 192.168.122.
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPREQUEST(virbr0) 192.168.122.17 06:91:4f:92:3b:a8
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPACK(virbr0) 192.168.122.17 06:91:4f:92:3b:a8 ada
Start the container, giving a a Virtual Ethernet Link connected to virbr0
systemd-nspawn --directory /var/lib/machines/ada --machine ada --network-bridge=virbr0 --boot --network-veth
(observe a nice booting sequence, up to a login prompt)
Login inside the container and configure the network
# snippet for /etc/network/interfaces
# the Virtual Ethernet Link appears as host0 inside the container
auto host0
iface host0 inet dhcp
# restart the network
systemctl restart networking
Find out from the host side which IP was assigned
machinectl list
MACHINE CLASS     SERVICE        OS     VERSION ADDRESSES
ada     container systemd-nspawn debian 8       192.168.122.17...

1 machines listed.

Inspect from the host system if the Apache virtual hosts are running
curl --silent --header 'Host: meinoesterreich.ist-ur.org' http://192.168.122.17/titelblatt | w3m -T text/html -dump
Happy Upgrades !

vendredi 6 janvier 2017

Hard disk performance measured with fio

Since some times now, I collect the performance of various storages devices, using fio, the flexible Input/Ouput tester.

With FIO you can collect sequential and IOPS statistics. Sequential thoughput is what you would need if for instance extracting a large tar archive. or copying a disk image. IOPS, where I measure a mix of 70% reads, 30% write, would be what you need for an OLTP workload.

Sequential writes:

Let's try from the slowest to the fastest, using the command
fio  --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
The ouput from this test should match the column wMB/s of the command iostat -mx 2, executed during the test. 

1) SD Card Sandisk Extreme Pro : 10MB/s
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

64kwrite: (groupid=0, jobs=1): err= 0: pid=12583: Sat Jan  7 15:24:55 2017
  write: io=512000KB, bw=11702KB/s, iops=182, runt= 43754msec

 ...
    bw (KB  /s): min= 1280, max=13184, per=99.97%, avg=11697.93, stdev=2798.43


2) A 5400rpm disk, mounted over a USB 3.0 port: 35 MB/s
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

...
64kwrite: (groupid=0, jobs=1): err= 0: pid=13635: Fri Jan  6 14:55:11 2017
  write: io=2056.1MB, bw=35035KB/s, iops=547, runt= 60120msec
...
    bw (KB  /s): min= 7168, max=61696, per=100.00%, avg=35101.89, stdev=10726.73
...


3) Samusung SSD 840 over SATA port: 360MB/s

 64kwrite: (groupid=0, jobs=1): err= 0: pid=16095: Fri Jan  6 15:17:56 2017
 write: io=9216.0MB, bw=365400KB/s, iops=5709, runt= 25827msec
...
    bw (KB  /s): min=345472, max=396160, per=99.87%, avg=364934.33, stdev=10505.38

Random Readwrites:


Using the command:
 fio  --size=2G --bs=4k --rw=randrw --rwmixread=70 --direct=1 --runtime=60 --numjobs=4 --name=4k7030test --group_reporting
The ouput from this test should match the column r/s and w/s of the command iostat -mx 2, executed during the test.

1) SD Card Sandisk Extreme Pro: 364 iops reading, 154 writing
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

4k7030test: (groupid=0, jobs=4): err= 0: pid=12724: Sat Jan  7 15:29:45 2017
  read : io=87488KB, bw=1457.8KB/s, iops=364, runt= 60017msec
  write: io=37108KB, bw=633130B/s, iops=154, runt= 60017msec



 2) A 5400rpm disk, over a USB 3.0 port: 37 iops reading, 15 writing
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

4k7030test: (groupid=0, jobs=4): err= 0: pid=12756: Sat Jan  7 15:31:02 2017
  read : io=9044.0KB, bw=154135B/s, iops=37, runt= 60084msec

  write: io=3796.0KB, bw=64694B/s, iops=15, runt= 60084msec


3) Samusung SSD 840 over SATA port: 22484 iops reading, 9648 writing

4k7030test: (groupid=0, jobs=4): err= 0: pid=12265: Sat Jan  7 15:15:54 2017
  read : io=5269.9MB, bw=89937KB/s, iops=22484, runt= 60001msec

  write: io=2261.4MB, bw=38594KB/s, iops=9648, runt= 60001msec


A very good summ up of fio benchmarks is also available on the Thomas Krenn wiki (in german)

samedi 24 décembre 2016

Quick debugging of a Linux printer via cups command line tools

Step by step cups debugging ( here with a network printer)

Which printer queue do I have configured ?
lpstat -p
printer epson is idle.  enabled since Sat Dec 24 13:18:09 2017
#here I have a printer called 'epson", doing nothing, that the cups daemon considers as enabled

Which connection am I using to get to this printer ?
lpstat -v
device for epson: lpd://epson34dea0.local:515/PASSTHRU
# here the locally configured 'epson' printer queue is backed by a network device at the adress epson34dea0.local, to which I am sending my print jobs via the lpd protocol

Is my printer ready ?
lpq
epson is ready
no entries

# here my local print queue 'epson' is accepting print jobs ( which does not say anything about the physical device, it might be offline

If here you local print queue 'epson' is not ready, you can try to reenable it in the cups system with:

sudo cupsenable epson

If you notice that the printer is disabled all the time, because for instance of a flaky network, you can edit /etc/cups/printers.conf and change the ErrorPolicy for each printer from stop-printer to retry-job.
It should be also possible to set this parameter in cupsd.conf