Generating a visual history of a git repo

If you haven't met it yet the gource programm really rocks: it generates a OpenGL rendered graph of all the commits of a git project.

You can see the video on real time just by calling the command gource in your git repo, or generate a video of it by piping to ffmpeg

This is the command line I used to generate my own video:

gource --hide filenames --hide files --title "the history of pmc" -s 0.1 --highlight-users -1280x720 -o - | ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -preset ultrafast -pix_fmt yuv420p -crf 1 -threads 0 -bf 0 gource.mp4

Java DOM from XML is not namespace aware by default

Something which costed me a couple of headaches using getElementsByTagNameNS with a DocumentBuilderFactory from the java.xml.parsers package.

You find in the class properties:

    private boolean namespaceAware = false;

and I thought XML namespaces where the bread and butter of XML ...

So when building a DOM from an XML document you need to change the defaults of the factory like:

    DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
    factory.setNamespaceAware(true);
    DocumentBuilder docBuilder = factory.newDocumentBuilder();

Shell looping vs xargs

For a very long time I was using the following construct to execute a command on foo, bar, and baz (all typed into an interactive shell prompt)

for F in foo bar baz; do command $F ; done

Now I finally got the -n option of xargs (or --max-args if you're using GNU) which limits to number of parameters to pass to the next command. So i can replace the previous command with:

echo foo bar baz | xargs -n 1 command

Here the -n option tells xargs to give at max one parameter to proceed, and create a new command with the next parameter.

As a bonus, instead of constructing an array of numbers with

seq 1 10

don't forget bash and zsh brace expansion

{1..10}

Tomcat7 not starting after Jessie->Stretch ugrade

I was wondering why the tomcat7 debian package would not start after the upgrade.

systemctl status tomcat7 output would look quite OK

systemctl status tomcat7
● tomcat7.service - LSB: Start Tomcat.
   Loaded: loaded (/etc/init.d/tomcat7; generated; vendor preset: enabled)
   Active: active (exited) since Fri 2017-06-16 10:10:34 CEST; 29min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 536 ExecStart=/etc/init.d/tomcat7 start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/tomcat7.service

Jun 16 10:10:34 new-ada systemd[1]: Starting LSB: Start Tomcat....
Jun 16 10:10:34 new-ada systemd[1]: Started LSB: Start Tomcat..
Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-dae…tory)
Jun 16 10:10:34 new-ada tomcat7[536]:  (already running).
Hint: Some lines were ellipsized, use -l to show in full.

But not java process was created

pgrep -lf java
echo $?

so something was obviously wrong.

But the devil is in the details, can you see this truncated line ?

Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-dae…tory)

If you run it in full with systemctl status -l tomcat7 it expends to

Jun 16 10:10:34 new-ada tomcat7[536]: Starting Tomcat servlet engine: tomcat7start-stop-daemon: unable to stat /usr/lib/jvm/java-7-openjdk-amd64//bin/java (No such file or directory)

Now I get it, the upgrade removed the openjdk-7-jre-headless package.Time to switch to openjdk8 ! The package openjdk-8-jre-headless was already installed, so I justed needed to set JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ in /etc/default/tomcat7

Note for the future:

• Why was this package removed ?
• Should I configure the systemctl status to show ellipsized lined by default ?

Render HTML from standard input with w3m

From a very long time, I was using the links browser as my default browser. It allows quick testing of a web site with a simple

links -dump debian.org

Alas links does not support rendering html from standard input, so I finally switched to w3m, handles that well. So I could do:

cat bla.html | w3m -T text/html -dump

Using curl and any member of the Unix Toolkit gang, you see which possibilities it opens. Oh and UTF-8 works well too !

w3m -T text/html -dump https://www.debian.org/index.ru.html | grep свободная
Debian — это свободная операционная система (ОС) для вашего компьютера.

свободная that’s Russian for Free.

Testing upgrades on a backup system

Like many Debian users, I am planning very soon to upgrade to my own personal server ada to Debian Stretch.
Since I do a full rsync backup of the server to a different location, I was wondering if it was possible to use this backup root fs in a container to test the upgrade. Turns is out it works very well with systemd-nspawn

Copy the backup to a separate dir in /var/lib/machines

# /var/lib/machines is where machinectl look for images
# see https://superuser.com/a/307542 for rsync flags detial
rsync -axHAX --info=progress2 --numeric-ids --whole-file /home/manu/Projects/backups/ada_rootfs/ /var/lib/machines/ada/

For networking this container, I installed libvirtd, which comes with a DHCP server serving private IPs on the virbr0 bridge.

systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-06-08 23:15:33 CEST; 6 days ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2609 (libvirtd)
    Tasks: 18 (limit: 4915)
   Memory: 53.0M
      CPU: 2.448s
   CGroup: /system.slice/libvirtd.service
           ├─2609 /usr/sbin/libvirtd
           ├─4868 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile
           └─4869 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile

juin 09 11:34:34 django dnsmasq-dhcp[4868]: not giving name ada to the DHCP lease of 192.168.122.
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPREQUEST(virbr0) 192.168.122.17 06:91:4f:92:3b:a8
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPACK(virbr0) 192.168.122.17 06:91:4f:92:3b:a8 ada

Start the container, giving a a Virtual Ethernet Link connected to virbr0

systemd-nspawn --directory /var/lib/machines/ada --machine ada --network-bridge=virbr0 --boot --network-veth

(observe a nice booting sequence, up to a login prompt)
Login inside the container and configure the network

# snippet for /etc/network/interfaces
# the Virtual Ethernet Link appears as host0 inside the container
auto host0
iface host0 inet dhcp

# restart the network
systemctl restart networking

Find out from the host side which IP was assigned

machinectl list
MACHINE CLASS     SERVICE        OS     VERSION ADDRESSES
ada     container systemd-nspawn debian 8       192.168.122.17...

1 machines listed.

Inspect from the host system if the Apache virtual hosts are running

curl --silent --header 'Host: meinoesterreich.ist-ur.org' http://192.168.122.17/titelblatt | w3m -T text/html -dump

Happy Upgrades !

Hard disk performance measured with fio

Since some times now, I collect the performance of various storages devices, using fio, the flexible Input/Ouput tester.

With FIO you can collect sequential and IOPS statistics. Sequential thoughput is what you would need if for instance extracting a large tar archive. or copying a disk image. IOPS, where I measure a mix of 70% reads, 30% write, would be what you need for an OLTP workload.

Sequential writes:

Let's try from the slowest to the fastest, using the command
fio  --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
The ouput from this test should match the column wMB/s of the command iostat -mx 2, executed during the test. 

 

0) Mist FPGA exposed as USB 1.1 device: 244 KB/s  (well that is slow !)
lsusb
Bus 001 Device 070: ID 1c40:0537 EZPrototypes MIST Board

fio  --size=50M --bs=4k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
64kwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1

   bw (  KiB/s): min=   16, max=  504, per=100.00%, avg=309.30, stdev=152.13, samples=93
   iops        : min=    4, max=  126, avg=77.24, stdev=38.00, samples=93

  WRITE: bw=239KiB/s (244kB/s), 239KiB/s-239KiB/s (244kB/s-244kB/s), io=14.1MiB (14.7MB), run=60340-60340msec


1) SD Card Sandisk Extreme Pro : 10MB/s
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

64kwrite: (groupid=0, jobs=1): err= 0: pid=12583: Sat Jan  7 15:24:55 2017
  write: io=512000KB, bw=11702KB/s, iops=182, runt= 43754msec
 ...
    bw (KB  /s): min= 1280, max=13184, per=99.97%, avg=11697.93, stdev=2798.43

2) A 5400rpm disk, mounted over a USB 2.0 port: 35 MB/s
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

...
64kwrite: (groupid=0, jobs=1): err= 0: pid=13635: Fri Jan  6 14:55:11 2017
  write: io=2056.1MB, bw=35035KB/s, iops=547, runt= 60120msec
...
    bw (KB  /s): min= 7168, max=61696, per=100.00%, avg=35101.89, stdev=10726.73
...

3) Same 5400rpm disk, mounted over a USB 3.0 port:  112 MB/s

64kwrite: (groupid=0, jobs=1): err= 0: pid=7877: Fri Jan 19 17:01:47 2018
  write: IOPS=1703, BW=106MiB/s (112MB/s)(6388MiB/60001msec)

4) USB 3.2 flash drive via USB 3.2 (Superspeed +) port: 235 MB/s
...
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 006: Dev 002, If 0, Class=Mass Storage, Driver=uas, 5000M


64kwrite: (groupid=0, jobs=1): err= 0: pid=155664: Wed Nov 13 11:26:51 2024
  write: IOPS=3579, BW=224MiB/s (235MB/s)(9216MiB/41192msec); 0 zone resets

 5) Samusung SSD 840 over SATA port: 360MB/s

 64kwrite: (groupid=0, jobs=1): err= 0: pid=16095: Fri Jan  6 15:17:56 2017
 write: io=9216.0MB, bw=365400KB/s, iops=5709, runt= 25827msec
...
    bw (KB  /s): min=345472, max=396160, per=99.87%, avg=364934.33, stdev=10505.38

Random Readwrites:

Using the command:
 fio  --size=2G --bs=4k --rw=randrw --rwmixread=70 --direct=1 --runtime=60 --numjobs=4 --name=4k7030test --group_reporting
The ouput from this test should match the column r/s and w/s of the command iostat -mx 2, executed during the test.

1) SD Card Sandisk Extreme Pro: 364 iops reading, 154 writing
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

4k7030test: (groupid=0, jobs=4): err= 0: pid=12724: Sat Jan  7 15:29:45 2017
  read : io=87488KB, bw=1457.8KB/s, iops=364, runt= 60017msec
  write: io=37108KB, bw=633130B/s, iops=154, runt= 60017msec


 2) A 5400rpm disk, over a USB 3.0 port: 37 iops reading, 15 writing
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

4k7030test: (groupid=0, jobs=4): err= 0: pid=12756: Sat Jan  7 15:31:02 2017
  read : io=9044.0KB, bw=154135B/s, iops=37, runt= 60084msec
  write: io=3796.0KB, bw=64694B/s, iops=15, runt= 60084msec

3) USB 3.2 flash drive via USB 3.2 (Superspeed +) port: 9816 iops reading,  4214 writing
...
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 006: Dev 002, If 0, Class=Mass Storage, Driver=uas, 5000M

4k7030test: (groupid=0, jobs=4): err= 0: pid=155797: Wed Nov 13 11:29:49 2024
  read: IOPS=9816, BW=38.3MiB/s (40.2MB/s)(2301MiB/60001msec)
  write: IOPS=4214, BW=16.5MiB/s (17.3MB/s)(988MiB/60001msec);

4) Samusung SSD 840 over SATA port: 22484 iops reading, 9648 writing

4k7030test: (groupid=0, jobs=4): err= 0: pid=12265: Sat Jan  7 15:15:54 2017
  read : io=5269.9MB, bw=89937KB/s, iops=22484, runt= 60001msec
  write: io=2261.4MB, bw=38594KB/s, iops=9648, runt= 60001msec


A very good summ up of fio benchmarks is also available on the Thomas Krenn wiki (in german)