Thursday, June 15, 2017

Render HTML from standard input with w3m

From a very long time, I was using the links browser as my default browser. It allows quick testing of a web site with a simple
links -dump debian.org
Alas links does not support rendering html from standard input, so I finally switched to w3m, handles that well. So I could do:
cat bla.html | w3m -T text/html -dump
Using curl and any member of the Unix Toolkit gang, you see which possibilities it opens. Oh and UTF-8 works well too !
w3m -T text/html -dump https://www.debian.org/index.ru.html | grep свободная
Debian — это свободная операционная система (ОС) для вашего компьютера.
свободная that’s Russian for Free.

Testing upgrades on a backup system

Like many Debian users, I am planning very soon to upgrade to my own personal server ada to Debian Stretch.
Since I do a full rsync backup of the server to a different location, I was wondering if it was possible to use this backup root fs in a container to test the upgrade. Turns is out it works very well with systemd-nspawn

Copy the backup to a separate dir in /var/lib/machines
# /var/lib/machines is where machinectl look for images
# see https://superuser.com/a/307542 for rsync flags detial
rsync -axHAX --info=progress2 --numeric-ids --whole-file /home/manu/Projects/backups/ada_rootfs/ /var/lib/machines/ada/
For networking this container, I installed libvirtd, which comes with a DHCP server serving private IPs on the virbr0 bridge.
systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-06-08 23:15:33 CEST; 6 days ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2609 (libvirtd)
    Tasks: 18 (limit: 4915)
   Memory: 53.0M
      CPU: 2.448s
   CGroup: /system.slice/libvirtd.service
           ├─2609 /usr/sbin/libvirtd
           ├─4868 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile
           └─4869 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile

juin 09 11:34:34 django dnsmasq-dhcp[4868]: not giving name ada to the DHCP lease of 192.168.122.
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPREQUEST(virbr0) 192.168.122.17 06:91:4f:92:3b:a8
juin 09 12:04:24 django dnsmasq-dhcp[4868]: DHCPACK(virbr0) 192.168.122.17 06:91:4f:92:3b:a8 ada
Start the container, giving a a Virtual Ethernet Link connected to virbr0
systemd-nspawn --directory /var/lib/machines/ada --machine ada --network-bridge=virbr0 --boot --network-veth
(observe a nice booting sequence, up to a login prompt)
Login inside the container and configure the network
# snippet for /etc/network/interfaces
# the Virtual Ethernet Link appears as host0 inside the container
auto host0
iface host0 inet dhcp
# restart the network
systemctl restart networking
Find out from the host side which IP was assigned
machinectl list
MACHINE CLASS     SERVICE        OS     VERSION ADDRESSES
ada     container systemd-nspawn debian 8       192.168.122.17...

1 machines listed.

Inspect from the host system if the Apache virtual hosts are running
curl --silent --header 'Host: meinoesterreich.ist-ur.org' http://192.168.122.17/titelblatt | w3m -T text/html -dump
Happy Upgrades !

Friday, January 6, 2017

Hard disk performance measured with fio

Since some times now, I collect the performance of various storages devices, using fio, the flexible Input/Ouput tester.

With FIO you can collect sequential and IOPS statistics. Sequential thoughput is what you would need if for instance extracting a large tar archive. or copying a disk image. IOPS, where I measure a mix of 70% reads, 30% write, would be what you need for an OLTP workload.

Sequential writes:

Let's try from the slowest to the fastest, using the command
fio  --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
The ouput from this test should match the column wMB/s of the command iostat -mx 2, executed during the test. 

 

0) Mist FPGA exposed as USB 1.1 device: 244 KB/s  (well that is slow !)
lsusb
Bus 001 Device 070: ID 1c40:0537 EZPrototypes MIST Board

fio  --size=50M --bs=4k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting
64kwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1

   bw (  KiB/s): min=   16, max=  504, per=100.00%, avg=309.30, stdev=152.13, samples=93
   iops        : min=    4, max=  126, avg=77.24, stdev=38.00, samples=93

  WRITE: bw=239KiB/s (244kB/s), 239KiB/s-239KiB/s (244kB/s-244kB/s), io=14.1MiB (14.7MB), run=60340-60340msec



1) SD Card Sandisk Extreme Pro : 10MB/s
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

64kwrite: (groupid=0, jobs=1): err= 0: pid=12583: Sat Jan  7 15:24:55 2017
  write: io=512000KB, bw=11702KB/s, iops=182, runt= 43754msec

 ...
    bw (KB  /s): min= 1280, max=13184, per=99.97%, avg=11697.93, stdev=2798.43


2) A 5400rpm disk, mounted over a USB 2.0 port: 35 MB/s
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

...
64kwrite: (groupid=0, jobs=1): err= 0: pid=13635: Fri Jan  6 14:55:11 2017
  write: io=2056.1MB, bw=35035KB/s, iops=547, runt= 60120msec
...
    bw (KB  /s): min= 7168, max=61696, per=100.00%, avg=35101.89, stdev=10726.73
...

3) Same 5400rpm disk, mounted over a USB 3.0 port:  112 MB/s

64kwrite: (groupid=0, jobs=1): err= 0: pid=7877: Fri Jan 19 17:01:47 2018
  write: IOPS=1703, BW=106MiB/s (112MB/s)(6388MiB/60001msec)

4) USB 3.2 flash drive via USB 3.2 (Superspeed +) port: 235 MB/s
...
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 006: Dev 002, If 0, Class=Mass Storage, Driver=uas, 5000M


64kwrite: (groupid=0, jobs=1): err= 0: pid=155664: Wed Nov 13 11:26:51 2024
  write: IOPS=3579, BW=224MiB/s (235MB/s)(9216MiB/41192msec); 0 zone resets


 5) Samusung SSD 840 over SATA port: 360MB/s

 64kwrite: (groupid=0, jobs=1): err= 0: pid=16095: Fri Jan  6 15:17:56 2017
 write: io=9216.0MB, bw=365400KB/s, iops=5709, runt= 25827msec
...
    bw (KB  /s): min=345472, max=396160, per=99.87%, avg=364934.33, stdev=10505.38

Random Readwrites:


Using the command:
 fio  --size=2G --bs=4k --rw=randrw --rwmixread=70 --direct=1 --runtime=60 --numjobs=4 --name=4k7030test --group_reporting
The ouput from this test should match the column r/s and w/s of the command iostat -mx 2, executed during the test.

1) SD Card Sandisk Extreme Pro: 364 iops reading, 154 writing
dmesg | grep mmc0
[    3.019323] mmc0: new ultra high speed SDR50 SDHC card at address e624
(SDR50 indicates a 50MB/s bus speed)

4k7030test: (groupid=0, jobs=4): err= 0: pid=12724: Sat Jan  7 15:29:45 2017
  read : io=87488KB, bw=1457.8KB/s, iops=364, runt= 60017msec
  write: io=37108KB, bw=633130B/s, iops=154, runt= 60017msec



 2) A 5400rpm disk, over a USB 3.0 port: 37 iops reading, 15 writing
[6:0:0:0]    disk    WDC WD20 NPVZ-00WFZT0     01.0  /dev/sdb

4k7030test: (groupid=0, jobs=4): err= 0: pid=12756: Sat Jan  7 15:31:02 2017
  read : io=9044.0KB, bw=154135B/s, iops=37, runt= 60084msec

  write: io=3796.0KB, bw=64694B/s, iops=15, runt= 60084msec

3) USB 3.2 flash drive via USB 3.2 (Superspeed +) port: 9816 iops reading,  4214 writing
...
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 006: Dev 002, If 0, Class=Mass Storage, Driver=uas, 5000M

4k7030test: (groupid=0, jobs=4): err= 0: pid=155797: Wed Nov 13 11:29:49 2024
  read: IOPS=9816, BW=38.3MiB/s (40.2MB/s)(2301MiB/60001msec)
  write: IOPS=4214, BW=16.5MiB/s (17.3MB/s)(988MiB/60001msec);


4) Samusung SSD 840 over SATA port: 22484 iops reading, 9648 writing

4k7030test: (groupid=0, jobs=4): err= 0: pid=12265: Sat Jan  7 15:15:54 2017
  read : io=5269.9MB, bw=89937KB/s, iops=22484, runt= 60001msec

  write: io=2261.4MB, bw=38594KB/s, iops=9648, runt= 60001msec


A very good summ up of fio benchmarks is also available on the Thomas Krenn wiki (in german)

Saturday, December 24, 2016

Quick debugging of a Linux printer via cups command line tools

Step by step cups debugging ( here with a network printer)

Which printer queue do I have configured ?
lpstat -p
printer epson is idle.  enabled since Sat Dec 24 13:18:09 2017
#here I have a printer called 'epson", doing nothing, that the cups daemon considers as enabled

Which connection am I using to get to this printer ?
lpstat -v
device for epson: lpd://epson34dea0.local:515/PASSTHRU
# here the locally configured 'epson' printer queue is backed by a network device at the adress epson34dea0.local, to which I am sending my print jobs via the lpd protocol

Is my printer ready ?
lpstat -a
epson is ready
no entries

# here my local print queue 'epson' is accepting print jobs ( which does not say anything about the physical device, it might be offline

If here you local print queue 'epson' is not ready, you can try to reenable it in the cups system with:

sudo cupsenable epson

If you notice that the printer is disabled all the time, because for instance of a flaky network, you can edit /etc/cups/printers.conf and change the ErrorPolicy for each printer from stop-printer to retry-job.
It should be also possible to set this parameter in cupsd.conf

Finally you can print a test page with
lpr /usr/share/cups/data/testprint

Wednesday, November 23, 2016

Small Summary of the Debian cloud sprint in Seattle

Last week I was In Seattle for the first Debian cloud team sprint The aim of it was to streamline the producing of official Debian ready to use images for various cloud providers.
The three biggest cloud providers were there (Google, Amazon, Microsoft Azure), and my humble self was here because of the work I have been doing producing Debian base boxes for Vagrant.
 During the three days the spring took we did the following:
* Reviewed the current state of cloud images and their build tools
After a demonstration of the vmdebootstrap and fai-diskimage creation tools, agreed we should try fai-diskimage as a default tool. Since a consensus here seemed to be reached I refrained myself from advocating packer or virt-install too much 
 * Reviewed the current state of cloud-init in Debian and 
considering its non working state in stable, thought about an upgrade in Debian Stable
Agreed about having unattended super axes by default inside the team,and proposed it to debian-devel
 * Reviewed all the bugs of the debian-cloud virtual package.
As with probably every so sprint, getting to bring all the people in one room brings a lot of synergies which couldn't happen on a mailing list. Also we noticed a lot of interest for Debian from the respective cloud providers which is interesting considering Debian is "only" a community based distribution.
Seattle is also a nice town, with lots of pine trees and bicycle lanes, looked to me like Canada, a troll statue(?) though I didn't have the time to see that much of it.

Sunday, November 20, 2016

Créer un clip vidéo (waveform) pour n'importe quel mp3 avec ffmpeg/avconv

J'avais envie de créer pour une collection de fichiers audio une vidéo  affichant une courbe audio, afin d'avoir quelque chose de mieux à afficher sur youtube qu'un bête logo.
( le resultat final est visible ici: https://www.youtube.com/watch?v=oI32z19iu_o )

En utilisant ffmpeg et son filtre "showvaves", on arrive a un résultat assez convaincant.

ffmpeg -i mon_fichier.wav -filter_complex "[0:a]showwaves=s=720x576:mode=cline:rate=12:scale=log:colors=Black|LightBlue,format=yuv420p[v]" -map "[v]" -map 0:a -c:v libx264 -c:a aac -b:a 192k ma_video.mp4

Cette commande va créer a partir d'un fichier wav un video au format mp4, en utilisant les bons codecs que recommande youtube (x264 pour la video, aac pour l'audio)
La vidéo sera en résolution DVD ce qui est largement suffisant a mon avis pour le web, mais vous pouvez remplacer 720x576
par 1920×1080 si vous voulez absolument du full hd.

Maintenant que l'on a une belle courbe audio, pourquoi ne pas ajouter une image en fond ?
Cette fois ci nous allons utiliser le filtre "blend" avec un fichier mon_logo.jpg, qui devra être à la même résolution que le fichier audio (ici 720x576 donc) L'image sera ici en arrière plan, car mon_logo.jpg est specifié en deuxième paramètre sur la ligne de commande.

ffmpeg -i ma_video.mp4 -i mon_logo.jpg -filter_complex "blend=all_mode=overlay:all_opacity=0.7" ma_video_logo.mp4

Sunday, January 3, 2016

Pinky syndrom ou syndrome du petit doigt sur clavier

La semaine dernière je me suis rendu compte que mon auriculaire de la main gauche était victime du"pinky syndrome" : la partie médiane de l’auriculaire étant particulièrement douloureuse alors que le reste de la main en pleine forme. J'ai rapidement relié ceci a un abus de l'utilisation de l'auriculaire sur mon clavier, j'utilise en effet l'auriculaire de la main gauche a la fois pour acceder à la touche 'shift', à la touche 'Control' et a la touche 'esc'
Quand on travaille avec  Eclipse, Vim et la console Unix, Dieu sait si ces touches sont souvent utilisées !

Comme solution à ce problème j'ai configuré deux options pour mon clavier:
* la touche 'Control' de gauche est échangée avec la touche 'Alt' de gauche, ce qui me force à utiliser le pouce, plus musclé, pour entrer 'Control'.
* la touche 'Caps lock' est replacée par une touche 'Control'.
* j'essaye d'utiliser 'Control-C' pour passer an mode commande dans vim, ce qui évite d'aller chercher une touche bien loin de la position départ de la main.

Les deux premières options se configurent avec
XKBOPTIONS="ctrl:swap_lalt_lctl,caps:ctrl_modifier,terminate:ctrl_alt_bksp"

dans  /etc/default/keyboard

ou avec setxkbmap -model pc105 -layout us -option ctrl:swap_lalt_lctl,caps:ctrl_modifier,terminate:ctrl_alt_bksp en ligne de commande.

Enfin si vous utilisez Gnome 3, il est possible que Gnome écrase ces règlages, il faudra les réintroduire pour Gnome avec:
 dconf write "/org/gnome/desktop/input-sources/xkb-options" "['ctrl:swap_lalt_lctl','caps:ctrl_modifier','terminate:ctrl_alt_bksp']"

NB : on obtient aussi le même résultat en insérant dans le fichier .Xmodmap:

clear control
clear mod1
keycode 37 = Alt_L Meta_L
keycode 64 = Control_L
add control = Control_L Control_R
add mod1 = Alt_L


La liste complète des options d'agencement se trouve dans /usr/share/X11/xkb/rules/base.xml, ces options sont accessibles graphiquement via gnome-tweak-tool.
Le détail du fonctionnement du clavier sous Linux / X11 est expliqué en détail ici.