tag:blogger.com,1999:blog-71434013282775222442024-03-13T22:55:56.084+01:00::meta synthax::Sociologie des systèmes libres ...Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.comBlogger36125tag:blogger.com,1999:blog-7143401328277522244.post-41527517564279560562022-05-25T22:58:00.000+02:002022-05-25T22:58:19.746+02:00One of the strangest bug I have ever seen on Linux<p>Networking starts when you login as root, stops when you log off !</p>
<a href="https://asciinema.org/a/497220" target="_blank"><img src="https://asciinema.org/a/497220.svg" /></a>
<p>SeLinux messages can be ignored I guess, but we see clearly the devices being activated (it's a Linux bridge)</p>
<p>If you have any explanations I am curious.</p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com1tag:blogger.com,1999:blog-7143401328277522244.post-62601397806085673272021-10-25T09:39:00.002+02:002021-10-25T09:39:18.583+02:00Booting the plan9 installer in libvirt<p>After dealing with Unix and Linux systems for so many years, I wanted to have to have a look
at Plan9, the post-unix operating system. I am using the 9front variant, which is
the most active Plan9 variant.
Booting an iso in libvirt is as simple as every Linux or BSD distribution as Plan9
supports the <code>virtio-net</code> and <code>virtio-scsi</code> virtual devices.</p>
<pre><code>virt-install \
--connect qemu:///session \
--name 9front \
--ram 512 \
--vcpus 2 \
--disk path=$PWD/9front.qcow2,size=4,bus=scsi,format=qcow2 \
--controller type=scsi,model=virtio-scsi \
--cdrom=9front.iso \
--virt-type kvm \
--os-variant generic \
--boot uefi
</code></pre>
<p>Once the CD boots, just press enter to accept the detected defaults, which should work fine, expect
for the screen detection: you have to choose <code>vga</code> or <code>lcd</code> otherwise the installer hangs
trying to detect vesa modes.</p>
<p>On the iso is booted you land up in a live cd environment where you can <a href="http://fqa.9front.org/fqa4.html">install the OS</a>. The environment looks
at the same time similar and very different from Unix, which is where the challenge is !</p>
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-35059784848221921042021-05-26T14:38:00.001+02:002021-05-26T14:38:54.913+02:00Testing a CIFS / Samba share is browsable from the command line<p>Since I always forget what is the right synthax:</p><p><span style="font-family: courier;">smbclient --user <username> --list <servername><br /></span></p><p><b><span style="font-family: courier;">smbclient</span></b> is available in the appropriately named <b>smbclient</b> package.<br /></p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-61913878707071440542021-05-07T14:55:00.003+02:002021-05-20T18:49:41.515+02:00Opensource Operating Systems for 16/32 bits wonders <p><strong>Debian</strong> does not provide official releases for the Atari/m68k since the 2000s, but there is still an ongoing porting effort to make Debian run in the <a href="https://lists.debian.org/debian-68k/">debian-m68k</a> mailing list (and even port the Rust compiler to m68k, hey John Paul:)</p>
<p>The <strong><a href="https://emutos.sourceforge.io/">EmuTOS project</a></strong> has released version 1.0 of its Atari TOS GPL clone, providing better hard disk disk support, and allowing thanks to binary compatibility to play the myriad of games released on that platform during the 80s and 90s.</p>
<p>Finally there is <strong><a href="https://subsole.org/st_mint">FreeMiNT</a></strong>, an Atari specific Unix kernel and OS, also under GPL, bringing true multitasking and memory protection to the cost of lower software compatibility. Currently at release 1.18, and still slowly developed.</p>
<p>As the hardware itself is getting old and overpriced, my next Atari machine will be a FPGA, the <a href="http://harbaum.org/till/mist/index.shtml">MiST</a>. Basically a FGPA is a re-programmable hardware platform. Instead of having transistors and logical gates of a chipset burned to the silicon, the circuit description is loaded on power-on, and thus reconfigurable. MiST can also reproduce the hardware of an Amiga and most of the 16 bits heroes of the late 80s.</p>
<p>Having a MiST available will allow me to reuse my joysticks and Midi gear, have more RAM, and a GPL OS that I can update without having to burn EEPROMS. Retrocomputing meets opensource, a good match. Note for self: those Atari related projects have a disposition for complicated mixed-case names.</p>
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-66616783664132315692021-03-30T17:58:00.002+02:002021-04-07T13:41:55.809+02:00Manually install a single node Kubernetes cluster on Debian<p>Debian has work-in-progress packages for Kubernetes, which work well enough enough for a testing and learning environement. Bootstraping a cluster with the <code>kubeadm</code> deployer with these packages is not that hard, and is similar to the <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/">upstream kubeadm documentation</a></p>
<h2 id="install-necessary-packages-in-a-vm">Install necessary packages in a VM</h2>
<p>Install a throwaway VM with Vagrant.</p>
<pre><code>apt install vagrant vagrant-libvirt
vagrant init debian/testing64</code></pre>
<p>Bump the RAM and CPU of the VM, Kubernetes needs at least 2 gigs and 2 cores.</p>
<pre><code>awk -i inplace '1;/^Vagrant.configure\("2"\) do \|config/ {print " config.vm.provider :libvirt do |vm| vm.memory=2048 end"}' Vagrantfile
awk -i inplace '1;/^Vagrant.configure\("2"\) do \|config/ {print " config.vm.provider :libvirt do |vm| vm.cpus=2 end"}' Vagrantfile</code></pre>
<p>Start the VM, login, update the package index.</p>
<pre><code>vagrant up
vagrant ssh
sudo apt update</code></pre>
<p>Install a container engine, here we use docker.io, we could also use containerd (both are packaged in Debian) or cri-o.</p>
<pre><code>sudo apt install --yes --no-install-recommends docker.io curl</code></pre>
<p>Install kubernetes binaries. This will install <code>kubelet</code>, the system service which will manage the containers, and <code>kubectl</code> the user/admin tool to manage the cluster.</p>
<pre><code>sudo apt install --yes kubernetes-{node,client} containernetworking-plugins</code></pre>
<p>Although it is not technically mandatory, we will use <code>kubeadm</code>, the most popular installer to create a Kubernetes cluster. Kubeadm is not packaged in Debian, we have to download an upstream binary.</p>
<pre><code>wget https://dl.k8s.io/v1.20.5/kubernetes-server-linux-amd64.tar.gz
sha512sum kubernetes-server-linux-amd64.tar.gz
28529733bf34f5d5b72eabe30a81df98cc7f8e529590f807745cd67986a2c5c3eb86cebc7ecbcfc3df3c50416306e5d150948f2483933ea46c2aebaeb871ea8f kubernetes-server-linux-arm64.tar.gz
sudo tar --directory=/usr/local/sbin --strip-components 3 -xaf kubernetes-server-linux-amd64.tar.gz kubernetes/server/bin/kubeadm
sudo chmod +x /usr/local/sbin/kubeadm
sudo kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:08:27Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}</code></pre>
<p>Add a <code>kubelet</code> systemd unit:</p>
<pre><code>RELEASE_VERSION="v0.4.0"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sudo tee /etc/systemd/system/kubelet.service
sudo systemctl enable kubelet</code></pre>
<p>and a default config file for <code>kubeadm</code></p>
<pre><code>RELEASE_VERSION="v0.4.0"
sudo mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf</code></pre>
<p>finally we need to help <code>kubelet</code> find the components needed for container networking</p>
<pre><code>echo 'KUBELET_EXTRA_ARGS="--cni-bin-dir=/usr/lib/cni"' | sudo tee /etc/default/kubelet</code></pre>
<h2 id="create-a-cluster">Create a cluster</h2>
<p>Initialize a cluster with <code>kubeadm</code>: this will download container images for the Kubernetes <a href="https://kubernetes.io/docs/concepts/overview/components/"><strong>control plane</strong></a> (= the brain of the cluster), and start the containers via the kubelet service. Yes a good part of Kubernetes itself run in containers.</p>
<pre><code>sudo kubeadm init --pod-network-cidr=10.244.0.0/16
...
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config</code></pre>
<p>Follow the instructions from the kubeadm output, and verify you have a single node cluster, with the status <code>NotReady</code>.</p>
<pre><code>kubectl get nodes
NAME STATUS ROLES AGE VERSION
testing NotReady control-plane,master 9m9s v1.20.5</code></pre>
<p>At that point you should also have a bunch of containers running on the node:</p>
<pre><code>sudo docker ps --format '{{.Names}}'
k8s_kube-apiserver_kube-apiserver-testing_kube-system_2711c230d39ccda1e74d1d6386a05cee_0
k8s_POD_kube-apiserver-testing_kube-system_2711c230d39ccda1e74d1d6386a05cee_0
k8s_etcd_etcd-testing_kube-system_4749b1bca3b1a73fd09c8e299d7030fe_0
k8s_POD_etcd-testing_kube-system_4749b1bca3b1a73fd09c8e299d7030fe_0
...</code></pre>
<p>The <code>kubelet</code> service also needs an external network plugin to get the cluster in Ready state.</p>
<pre><code>sudo systemctl status kubelet
...
Mar 28 09:28:43 testing kubelet[9405]: E0328 09:28:43.958059 9405 kubelet.go:2188] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized</code></pre>
<p>Let’s add that network plugin. Download the flannel network plugin definition, and schedule flannel to run on all nodes of your cluster:</p>
<pre><code>wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply --filename=kube-flannel.yml</code></pre>
<p>After a dozen of seconds your node should be in ready status.</p>
<pre><code>kubectl get nodes
NAME STATUS ROLES AGE VERSION
testing Ready control-plane,master 16m v1.20.5</code></pre>
<h2 id="deploy-a-test-application">Deploy a test application</h2>
<p>Our node is now in Ready status, but we cannot run application on it, since we only have a <em>master</em> node, an administrative node which by default cannot run user applications.</p>
<pre><code>kubectl describe node testing | grep ^Taints
Taints: node-role.kubernetes.io/master:NoSchedule</code></pre>
<p>Let’s allow node <code>testing</code> to run user applications:</p>
<pre><code>kubectl taint node testing node-role.kubernetes.io/master-</code></pre>
<p>Deploy a nginx container:</p>
<pre><code>kubectl run my-nginx-pod --image=docker.io/library/nginx --port=80 --labels="app=http-content"
</code></pre>
<p>Create a Kubernetes service to access this pod externally:</p>
<pre><code>cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-k8s-service
spec:
type: NodePort
ports:
- port: 80
nodePort: 30000
selector:
app: http-content
kubectl create --filename service.yaml
</code></pre>
<p>Access the service via IP adress:</p>
<pre><code>curl 192.168.121.63:30000
...
Thank you for using nginx.
</code></pre>
<h4 id="notes">Notes</h4>
<p>I will try to get this blog post in a Debian Wiki article, or maybe in the <code>kubernetes-node</code> documentation. Blog posts deprecate and disappear, wiki and project docs live longer.</p>
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-87842940624220010942021-03-30T17:54:00.001+02:002021-03-30T17:54:32.318+02:00Playing with cri-o, a container runtime built for Kubernetes<p>Kubernetes is <a href="https://kubernetes.io/blog/2020/12/02/dockershim-faq/">moving aways from docker</a> to alternative container engines presenting a smaller core having just the functionality needed. The two most populars alternatives are:</p>
<ul>
<li>containerd, a subset of docker, used for instance in Google Kubernetes Engine</li>
<li>cri-o, a new implementation of a container engine, used for instance in Red Hat's Kubernetes offering (OpenShift)</li>
</ul>
<p>These alternatives are meant to be used programatically via a unix domain socket, and therefore have a limited command line interface.</p>
<p>Let's play around in a VM.</p>
<p>Install a throwaway VM with Vagrant</p>
<pre><code>apt install vagrant vagrant-libvirt
vagrant init debian/testing64
</code></pre>
<p>Start the VM, install dependencies</p>
<pre><code>vagrant up
vagrant ssh
</code></pre>
<pre><code>sudo apt update
sudo apt install --yes curl gnupg jq
</code></pre>
<p>Install cri-o the container engine</p>
<pre><code>sudo bash
export OS=Debian_Testing VERSION=1.20
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/libcontainers.list
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/cri-o:$VERSION.list
curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -
apt install cri-o cri-o-runc containernetworking-plugins conntrack
</code></pre>
<p>Verify it is running properly</p>
<pre><code>systemctl restart cri-o
systemctl status cri-o
...
Started Container Runtime Interface for OCI (CRI-O).
</code></pre>
<p>Say hello to cri-o via its unix domain socket</p>
<pre><code>curl --silent --unix-socket /var/run/crio/crio.sock http://localhost/info | jq
{
"storage_driver": "overlay",
"storage_root": "/var/lib/containers/storage",
"cgroup_driver": "systemd",
"default_id_mappings": {
"uids": [
{
"container_id": 0,
"host_id": 0,
"size": 4294967295
}
],
"gids": [
{
"container_id": 0,
"host_id": 0,
"size": 4294967295
}
]
}
}
</code></pre>
<p>Install crictl, a Kubernetes debugging tool for containers</p>
<pre><code>wget --directory-prefix=/tmp https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.20.0/crictl-v1.20.0-linux-amd64.tar.gz
tar -xaf /tmp/crictl-v1.20.0-linux-amd64.tar.gz -C /usr/local/sbin/
chmod +x /usr/local/sbin/crictl
crictl info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
}
}
</code></pre>
<p>From there on you can create a container following the examples in <a href="https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md" class="uri">https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md</a></p>
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-43817192702187437592021-03-28T10:16:00.002+02:002021-03-28T10:16:42.760+02:00Switching to FAI (Fully Automatic Installer) for creating Vagrant Boxes<p>Have you heard of <a href="https://www.vagrantup.com/">Vagrant</a> ? It is a command line tool to get ready to use, disposable Virtual Machines (VM) from an online catalog. Vagrant works on Linux, FreeBSD, Windows and Mac and you only need three commands to get a shell prompt in a VM (<a href="https://wiki.debian.org/Teams/Cloud/VagrantQuickStart">see the Debian wiki</a>).<br />The online catalog has images for the majority of the OSes you can think of.<br /><br />We've been building the Debian disk images for Vagrant (available on <a href="https://app.vagrantup.com/debian/">https://app.vagrantup.com/debian/)</a> with a number of tools over the years:</p><ul style="text-align: left;"><li>first the ad-hoc <a href="https://github.com/EmmanuelKasper/vagrant-debian-wheezy-64">https://github.com/EmmanuelKasper/vagrant-debian-wheezy-64</a> which was hacking the Debian installer CD with a builtin preseed file, and passed that CD to a VirtualBox VM to automate the installer</li></ul><ul style="text-align: left;"><li>then <a href="https://liw.fi/vmdebootstrap/">vmdebootstrap</a>, which is one of <a href="https://manpages.debian.org/buster/openstack-debian-images/build-openstack-debian-image.1.en.html">the</a> <a href="https://github.com/go-debos/debos">many</a> <a href="https://github.com/systemd/mkosi">different</a> <a href="https://liw.fi/vmdb2/">wrappers</a> over debootstrap, which creates a debian install on a loopback mounted filesystem.<br /></li></ul><ul style="text-align: left;"><li>then <a href="https://www.packer.io/">packer</a>, which is wrapping qemu and the Debian installer CD with automated bootparams and preseed file.<br /></li></ul><ul style="text-align: left;"><li>and then <a href="https://fai-project.org/doc/man/fai-diskimage.html">fai-diskimage</a>, again a wrapper over debootstrap using loopback mounts</li></ul><p>Basically there are two category of tools for building a disk image:</p><p>- those using an emulator and the OS installer in a automated way</p><p>- those using debootstrap/pacstrap/rpmstrap on a loopback mounted filesystem</p><p>Personally I prefer the first approach, as you can run the build process as non root, and you benefit from all the quality work of the official installer.<br />However this requires virtualization, and nested virtualization if your build process run insides a VM. Unfortunately nested virtualization is not that common, for instance my <a href="https://www.hetzner.com/">cloud provider</a>, and the VMs used for <a href="http://ci.debian.net">Debian Continuous Integration</a>, are not supporting nested virtualization.<br />As the maintainer of fai-diskimage is a Debian Developer (hey MrFAI ! :) and as the debian-cloud folks are it using for Amazon, Azure and Google Cloud Debian images, it made sense to switch to fai-diskimage for now. The fai-diskimage learning curve is a bit steep as you have to learn many internal concepts before using it, but once you get the bits connected it works quite well.<br /></p>
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-13283919643055798752021-03-09T16:44:00.001+01:002021-03-09T16:44:54.111+01:00Displaying CSV files in a readable way on the terminal<p> Until this week I did not know about the <b><span style="font-family: courier;">column</span></b><span style="font-family: courier;"><span style="font-family: inherit;"> <span style="font-family: inherit;"></span></span></span>command.<br /><br /></p><p><span style="font-family: courier;"><span style="font-family: inherit;">$ head -5 zillow.csv <br />"Index", "Living Space (sq ft)", "Beds", "Baths", "Zip", "Year", "List Price ($)"<br /> 1, 2222, 3, 3.5, 32312, 1981, 250000<br /> 2, 1628, 3, 2, 32308, 2009, 185000<br /> 3, 3824, 5, 4, 32312, 1954, 399000<br /> 4, 1137, 3, 2, 32309, 1993, 150000<br /><br /></span></span></p><p>Turned out this file is much more readable with a good pipe (and a large screen)<br /></p><p><span style="font-family: courier;"><span style="font-family: inherit;">$ head -5 zillow.csv | column --table --separator ,<br />"Index" "Living Space (sq ft)" "Beds" "Baths" "Zip" "Year" "List Price ($)"<br /> 1 2222 3 3.5 32312 1981 250000<br /> 2 1628 3 2 32308 2009 185000<br /> 3 3824 5 4 32312 1954 399000<br /> 4 1137 3 2 32309 1993 150000<br /><br /></span></span><b><span style="font-family: courier;">column</span></b> is part of util-linux and is thus available in all distributions.<br />Example file taken from this <a href="https://people.sc.fsu.edu/~jburkardt/data/csv/csv.html">example list</a>.<br /></p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-20417293871447718782021-01-30T20:10:00.000+01:002021-01-30T20:10:10.198+01:00Playing Tetris over serial console<p>Today I played Tetris over a serial console connection, on a Vax 4000 running OpenBSD. I haven't felt that <i>1337 </i>since a long time.<br />I am going to get rid of that Vax system though. If that's your stuff, contact me privately.<br /><br /><a href="https://asciinema.org/">asciinema</a> in its greatness:<br /><br /></p>
<a href="https://asciinema.org/a/388067" target="_blank"><img src="https://asciinema.org/a/388067.svg" /></a>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-17624969670282831482021-01-03T13:25:00.002+01:002021-01-03T13:58:24.485+01:00 How to move a single VM between cloud providers<p>I am running since a decade a small Debian VM, that I use for basic web and mail hosting. Since most of the VM setup is done manually and not following the <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">Infrastructure As Code</a> pattern, it is faster to simply copy the filesystem when switching providers instead of reconfiguring everything.<br />The steps involved are:</p><p>1. create a backup of the filesystem using <span style="font-family: courier;">tar</span> of <span style="font-family: courier;">rsync</span>, excluding dynamic content<br /><span style="font-family: courier;">rsync --archive \<br /> --one-file-system --numeric-ids \<br /> --rsh "ssh -i private_key root@server:/ /local_dir</span><br />or<br /><span style="font-family: courier;">tar -cvpzf backup.tar.gz \<br />--numeric-owner \<br />--exclude=/backup.tar.gz \<br />--one-file-system /</span> <br /><br />Notice here the <span style="font-family: courier;">--one-file-system</span> switch which avoids back'ing up the content of mount points like <span style="font-family: courier;">/proc</span>, <span style="font-family: courier;">/dev</span>.<br />If you have extra partitions with a mounted filesystem, like <span style="font-family: courier;">/boot</span> or <span style="font-family: courier;">home</span> you need do add a separate backup for those.<br /><br />2. create a new VM on the new cloud provider, verify you have a working console access, and power it off.<br />3. boot on the new cloud provider a rescue image<br />4. partition the disk image on the new provider.<br />5. mount the new root partition, and untar your backup on it. You could for instance push the local backup via rsync, or download the tar archive using https.<br />6. update network configuration and <span style="font-family: courier;">/etc/fstab</span><br />7. chroot into the target system, and <a href="https://wiki.debian.org/GrubEFIReinstall">reinstall grub</a><br /><br />This works surprisingly well, and you if made your backup locally, you can test the whole procedure <a href="https://blog.entek.org.uk/technology/2020/06/06/building-debian-vms-with-debootstrap.html">by building a test VM with your backup</a>. Just replace the <span style="font-family: courier;">deboostrap</span> step with a command like <span style="font-family: courier;">tar -xvpzf /path/to/backup.tar.gz -C /mount_point --numeric-owner</span><br /><br />Using this procedure, I moved from <a href="http://meta.libera.cc/2011/09/serveur-prive-vps-chez-hetznerde-pour.html">Hetzner</a> (link in French language) to Digital Ocean, from Digital Ocean to Vultr, and now back at Hetzner.<br /></p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-22703508826100075792020-12-28T18:39:00.000+01:002020-12-28T18:39:13.168+01:00Quick NetBSD serial console install on libvirt<p>I wanted to set up a small VM with <a href="netbsd.org">NetBSD</a> to test a couple of <b><span style="font-family: courier;">virt-install</span> </b>option. It turns out it you can get to the installer prompt quite fast.</p><p>
get the NetBSD installer for serial console: <br /><span style="font-family: courier;"><b></b></span></p><pre><span style="font-family: courier;"><b>wget https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/i386/installation/cdrom/boot-com.iso </b></span></pre><p></p><p>start the install <span style="font-family: courier;"><b> </b></span><br /></p><pre><span style="font-family: courier;"><b>$ virt-install \
--connect qemu:///session \
--name netbsd \
--ram 64 \
--vcpus 2 \
--disk path=$HOME/netbsd.qcow2,size=4,bus=scsi,format=qcow2 \
--controller type=scsi,model=virtio-scsi \
--cdrom=boot-com.iso \
--virt-type kvm \
--os-variant netbsd8.0 \
--graphics none \
--arch i686 \
--console pty,target_type=serial</b></span> </pre><pre> </pre>
<p>
This will start a VM in usermode networking, so no need to be root, but
the VM won’t be reachable from the outside world, except if you add <a href="https://serverfault.com/a/898171/69764">qemu usermode port forwarding.</a>
</p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-58820218726928074052020-09-14T17:18:00.002+02:002020-09-14T17:18:53.861+02:00Using Debian and RHEL troubleshootings containers on Kubernetes & OpenShift<p>You can connect to a running pod with <b><span style="font-family: courier;">oc/kubectl rsh pod_name</span></b>, or start a copy of a running pod with <b><span style="font-family: courier;">oc debug pod_name</span></b>, but as best practises recommend unprivileged, slim container images, where do you get sosreport, kdump, dig and nmap for troubleshooting ? </p><p>Fortunately you can start either a transient Debian troubleshooting container with:</p><p><b><span style="font-family: courier;"></span></b></p><p><b><span style="font-family: courier;">oc run troubleshooting-pod --stdin --tty --rm --image=docker.io/library/debian:buster</span></b></p><p>or a Red Hat Entreprise Linux:<br /></p><p><b><span style="font-family: courier;">oc run troubleshooting-pod --stdin --tty --rm --image=registry.access.redhat.com/rhel7/rhel-tools</span></b></p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-34175250851171750442020-06-30T13:29:00.000+02:002020-06-30T13:29:01.649+02:00Learning openshift: a good moment to revisit awk too
<p>I can’t believe I spent all these years using only grep.</p>
<p>Most of us know how to use <strong><code>awk</code></strong> to print the nth column of a file:</p>
<pre><code>$ awk '{print $1}' /etc/hosts
</code></pre>
<p>will print all IP addresses from <code>/etc/hosts</code></p>
<p>But you can also do filtering before printing the chosen column:</p>
<pre><code>$ awk '$5 >= 2 {print $2}' /path/to/file
</code></pre>
<p>will print the second column of all lines, where the 5th column is greater than 2.
That would have been hard with grep.</p>
<p>Now I can use that to find out all deployments on my openshift cluster, where the number of current replicas is greater than 2.</p>
<pre><code>$ oc get deployments --all-namespaces | awk '$5 >= 2 {print $2}'
NAME
oauth-openshift
console
downloads
router-default
etcd-quorum-guard
prometheus-adapter
thanos-querier
packageserver
</code></pre>
<p>I know that openshift/kubernetes both have a powerful query selector syntax, but <em>for the moment</em> <strong><code>awk</code></strong> will do.</p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com1tag:blogger.com,1999:blog-7143401328277522244.post-53976591106617847362020-06-16T19:13:00.005+02:002020-07-02T09:30:46.924+02:00Test a webcam from the command line on Linux with VLCSince this info was too well hidden on the internet, here is the information:
<strong><pre>cvlc v4l2:///dev/video0</pre></strong>
and there you go.<br>
If you have multiple cameras connected, you can try /dev/video0 up to /dev/video5
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com1tag:blogger.com,1999:blog-7143401328277522244.post-62676405602338604532020-04-14T16:11:00.001+02:002020-04-14T21:08:26.040+02:00Recommended keyboard settings for Productivity and Usability, for European ProgrammersTLDR:
<code>
setxkbmap -layout us -variant altgr-intl
</code>
and become a happier programmer.<br />
<h2>
The case for QWERTY for European Programmers</h2>
If you’re working on Unix / Linux, or C based programming languages, it can make sense to switch to the qwerty(us) keyboard layout. Why ?<br />
Unix, C, Perl, Java, and most of programming languages have been conceived on QWERTY keyboards.<br />
So when the designers choose special characters to use for the language synthax, they simply choose what was easy to access on their own keyboard.
This has been <a href="http://www.catonmat.net/blog/why-vim-uses-hjkl-as-arrow-keys/">historically documented for the vi editor.</a><br />
<br />
To give an example, using an Unix shell you have to type the dot . and slash / symbols quite often to navigate the filesystem. The two keys producing these symbols, are nicely aligned on a QWERTY layout and do not require a key combination to be entered. So you can quickly enter something like ‘../..’ using a single hand.<br />
Now using a QWERTZ layout, like in Germany / Austria, you have the ‘.’ symbol easily accessible, but you need to combine two keys ( Shift + 7 ) to get a ‘/’.<br />
And if you are a poor soul using an AZERTY layout, to get the ‘.’ and ‘/’ symbol you need each time a key combo.<br />
The need of key combos is bad not only for speed (multiple keys to lookup) but also for usability, as you have to stretch your fingers to reach the key if using a single hand, provoking repetitive strain injury. You might be smiling but this is commonly known amongst Emacs Users, due to the prominent use of commands using Ctrl and Alt combos, and led to the creation of an <a href="https://www.blogger.com/">Emacs Ergonomic wiki.</a><br />
<br />
This goes as well for many symbol commonly used in programming languages, think for instance about the semicolon ‘;’ for terminating a statement, the ‘[’, and ‘]’ symbols for defining an array, and the backslash ‘\’ for escaping.<br />
All these keys are accessible via a single keypress on qwerty and require key combos on qwertz and azerty. No wonder Linux, Minix and BSD were invented on non-azerty layouts: in France we were still busy typing the path to the source code, when in other parts of the world people already had the file open in their editor.<br />
You don’t need to throw away your existing keyboards when learning the qwerty (us) layout: for a couple of euros, you can find on ebay alternate keys stickers to put on your laptop.<br />
<h2>
Accessing keys with diacritics with the AltGr International variant</h2>
Now if you want to switch to a qwerty keyboard layout, and you’re French or German, you might wonder how to access the characters with diacritics, the é and è of French and ü and ö of German.<br />
Fortunately there is a very clever keyboard <i>variant</i> for the us <i>layout</i> who uses the AltGr key, to make all these keys accessible, just hiding them behind the AltGr key.<br />
Need é ? that’s just AltGr + e away. Needs ç ? That’s just AltGr + ,
Needs ö ? AltGr +p<br />
See the coolness of that ? You can type all international diacritics in an easy way, and there are <a href="https://www.ebay.at/itm/Stickers-Autocollants-de-clavier-QWERTY-US-INTERNATIONAL-keyboard-layout-keys/352638169883">even keyboard stickers for that too</a>.<br />
<br />
Now how to access to this layout of wonder ?<br />
You can configure the layout and variant in Xorg, in debian/ubuntu that would be entering in <i><code>/etc/default/keyboard</code></i><br />
<pre><code>XKBLAYOUT="us"
XKBVARIANT="altgr-intl"
</code></pre>
After restarting the X server, you can check that the settings have been applied with<br />
<pre><code>setxkbmap -print -verbose 10</code></pre>
If using Gnome, you can also set the keyboard layout and variant by changing the schema<i> org.gnome.desktop-inputsources</i>, which will override the desktop-agnostic settings of /etc/default/keyboard.<br />
For this you can either call<br />
<pre><code>dconf write /org/gnome/desktop/input-sources/sources "[('xkb', 'us+altgr-intl')]"
</code></pre>
or navigate with the gui tool <b><span style="font-family: "courier new" , "courier" , monospace;">dconf-settings</span></b> to <i>org.gnome.desktop-inputsources</i> and set the value there.<br />
<br />
If you want to further improve your keyboard layout, you can also have a look at <a href="http://meta.libera.cc/2016/01/pinky-syndrom-ou-syndrome-du-petit.html">swapping Ctrl and Alt</a>, as described here (French language article) Happy hacking !Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-69998606652647329862020-04-12T13:25:00.000+02:002020-04-12T13:26:34.432+02:00Putting a Red Hat on<p>I am switching jobs, and after a two year stint as a full stack developer at BPMasters, I am joining Red Hat as a <a href="https://www.redhat.com/en/blog/being-technical-account-manager-tam-day-life">Technical Account Manager</a> (Platform and Openshift). A good friend of a good friend already worked at Red Hat, and advised me to apply, which I did, and I am starting in May in Vienna, the place where I live.<br/>
I have heard that there is already a number of Debian Developers working for Red Hat, so if you happen to be one of them, I hope we’ll get in touch !</p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-47205672191942870782020-03-23T18:37:00.000+01:002020-03-23T18:45:26.377+01:00Two Factor Authentification on gitlab with YubikeyI wanted to have a working <b>Two Factor Authentification</b> (2FA) setup to login on salsa.debian.org, Debians’s gitlab instance.<br />
You might already know Two Factor Authentification via a One Time Password (OTP) generating app on your smartphone, like FreeOTP or Google Authenticator.
But it is possible to use a physical device, and a keypress on the device is enough to authenticate (speed up things !). Here I am using a Yubikey 4, a popular USB device for Two Factor Authentification which is officially supported by gitlab, and whose tooling is well packaged in Debian.<br />
<h2>
Get to know the device</h2>
Install the needed packages to work with the yubikey<br />
<pre><code># apt install yubikey-manager libu2f-host0
</code></pre>
List connected devices on your usb bus:<br />
<pre><code>$ lsusb
Bus 002 Device 109: ID 1050:0407 Yubico.com Yubikey 4 OTP+U2F+CCID
</code></pre>
Get info about the device capability<br />
<pre><code>$ ykman info
Device type: YubiKey 4
Serial number: 1234567
Firmware version: 4.3.7
Enabled USB interfaces: OTP+FIDO+CCID
Applications
OTP Enabled
FIDO U2F Enabled
OpenPGP Enabled
PIV Enabled
OATH Enabled
FIDO2 Not available
</code></pre>
The capability which interests us here is <i>FIDO U2F</i>. The Yubikey 4 supports Two Factor Authentification via the <b>U2F</b> standard, and this standard is maintained by the <b>FIDO</b> Industry Association, hence the name.
As I plan to only use the FIDO U2F capability of the key, I set ‘FIDO’ to be the single mode of the key.<br />
<pre><code>ykman mode FIDO
</code></pre>
<h2>
Testing web browser interaction with Yubico demo system</h2>
Now we need to have to have a browser with support for the U2F standard. Firefox has builtin support since Version 67. Debian 10 “Buster”
has <b><code>firefox-esr</code></b> Version 68, so that will work. For testing yubikeys, the manufacturer has a demo website, where you can test U2F.
Go to <a href="https://demo.yubico.com/">https://demo.yubico.com</a> and follow the “Explore the Yubikey” link.<br />
Once there you will be asked to register an account on yubicom’s demo systems, to which you will add the Yubikey as an Authenticating Device.
After that you can add your security key. First step will be to register the device, which will require a light touch on the Yubikey button, and acceptance of this Firefox warning Window, as the demo website wants to know the model of the device.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheor9WOJHZlEMK2KVFY-p3-WFe3Rap2xpusqLk3UcaHurk5tnRpME5AkeYGEiUSYCzfU5by5dMipGzHcmhXD-tzYn8bKtG0Sbf8Q6-0hTwSLlr3d0PMLEQmdNpj9CH8R72wPyqjGm8MUnl/s1600/firefox-warning.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="408" data-original-width="1117" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheor9WOJHZlEMK2KVFY-p3-WFe3Rap2xpusqLk3UcaHurk5tnRpME5AkeYGEiUSYCzfU5by5dMipGzHcmhXD-tzYn8bKtG0Sbf8Q6-0hTwSLlr3d0PMLEQmdNpj9CH8R72wPyqjGm8MUnl/s1600/firefox-warning.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Firefox message on the yubikey demo site. A normal site with U2F would not require the extended information, and have a simpler popup message.</td></tr>
</tbody></table>
As soon as the device is registered, you can login and logout and you
will be prompted again to lightly touch the Yubikey button to
authenticate, in addition to the classical login / password.<br />
<h2>
Using U2F on gitlab</h2>
When you want to register your yubikey for logging on salsa, you need first to register a One Time Password device in Settings -> Account -> Manage two-factor authentication, and Register Universal Two-Factor (U2F) Device. After the usual Firefox Popup, and the light touch on the key button, that'it you have a fast, and reliable Two Factor Authentification !<br />
<h2>
Conclusion</h2>
Each time I have to look on anything close to cryptography / authentification, it is a terminology avalanche. Here we had already 2FA, OTP, U2F, FIDO. And now there is FIDO2 too. It is the next version of the U2F standard, but this time it was named after the standardizing organization, FIDO. The web browser part of FIDO2 is called Webauthn. Also sometimes the whole FIDO2 is called Webauthn too. Easy to get, isn’t it ?Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-6352400609823245002020-03-22T17:51:00.002+01:002020-03-22T17:52:37.872+01:00Big Iron UNIX emulated on ARM<p>I have somewhere in the basement a DEC Vax workstation, but in the end it was a bigger fun to run an emulated Vax 11/780 (size of two refrigerators) in Beagle Bone Black (size of a big matchbox).
For this I used the Dockerfiles available in <a href="https://github.com/jguillaumes/dockersimh">this git repo</a> using the <strong><code>simh</code></strong> emulator, and tweaked a bit for ARM.
<br>
I recorded the boot sequence with the very nice <strong><code>asciinema</code></strong>, also available in the Debian archive, so here is <a href="https://asciinema.org/a/312498">4.3 BSD, in all its 1986 glory.</a></p>Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-28632720467947784752020-01-27T17:09:00.001+01:002020-01-27T18:13:06.961+01:00Mark a Screenshot on LinuxMore that than often to explain things quickly, I like to take a screenshot of the (web) application I am talking about, and then circle the corresponding area so that everything is clear. Possibly with a rounded rectangle, as I find it the cutest variant.<br />
<br />
This is how I do it on Linux:<br />
Install necessary tools:<br />
<pre><code><b>apt install gimp scrot </b>
</code></pre>
Take the screenshot: <br />
<pre><code># Interactively select a window or rectangle with the mouse
<b>scrot --selection screenshot.png </b>
</code></pre>
Open the screenshot and annotate it with gimp:<br />
<pre><code><b>gimp screenshot.png </b>
</code></pre>
Then in gimp:<br />
<ul>
<li>Tools -> Selection Tools -> Rectangle Select, and mark the area</li>
<li>Select -> Rounded Rectangle, and keep the default</li>
<li>Change the color to a nice blue shade in the toolbox</li>
<li>Edit -> Stroke selection</li>
</ul>
Maybe gimp is a bit overkill for that. But instead of learning a limited tool, I prefer to learn an advanced one like gimp step by step.Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com2tag:blogger.com,1999:blog-7143401328277522244.post-75239678920701996012019-12-21T21:34:00.002+01:002019-12-21T21:34:23.130+01:00Opensource Retrocomputing: FreeMiNT on Atari STIf you have an Atari ST sleeping in the attic, and have an interest for
open source Unix, you might be interested to try out <a href="https://subsole.org/st_mint">ST Mint</a>, a
distribution of the <a href="https://freemint.github.io/">FreeMiNT</a> kernel tailored for this platform.<br />
<br />
When preparing the last ST Mint release, I noticed that bash needed too much memory for the ST, so I included the <a href="https://packages.debian.org/fr/buster/sash">sash shell</a>, <a href="http://vincent.riviere.free.fr/soft/m68k-atari-mint/">cross-compiled</a> from the Debian source package.<br />Funnily enough, although the Atari hardware is physically large and heavy, working on ST Mint feels like doing embedded hardware development: you cross-compile, link with the smallest possible libc, copy stuff on a SD Card, and try to fit everything in 4MB of RAM.Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-78574446579005236432019-08-04T17:23:00.000+02:002019-08-04T17:23:13.315+02:00Debian 9 -> 10 Ugrade reportI upgraded my laptop and VPS to Debian 10, as usual in Debian everything worked out of the box, the necessary daemons restarted without problems.<br />I followed my usual upgrade approach, which involves <a href="http://meta.libera.cc/2017/06/testing-upgrades-on-backup-system.html">upgrading a backup of the root FS of the server in a container</a>, to test the upgrade path, followed by <a href="http://meta.libera.cc/2009/02/gestion-des-fichiers-de-configuration.html">a config file merge.</a><br /><br />I had one major problem, though, connecting to my php based Dokuwiki <a href="http://subsole.org/">subsole.org</a> website, which displayed a rather unwelcoming screen after the upgrade:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0sTDTndUER4nS3Oy8HC8YJCBCyrVgRDBR983iGCZxiK7_5a_OKbYNiTzE8T3LYwdTdOyBicIaeytJNfiH4z9zNRGcQkwNeqcAU7Dm8c0dgMjSkNcPpMkEpmJtlf4gKRWhdKISlHmwRo_4/s1600/2019-08-04-163827_777x333_scrot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="333" data-original-width="777" height="273" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0sTDTndUER4nS3Oy8HC8YJCBCyrVgRDBR983iGCZxiK7_5a_OKbYNiTzE8T3LYwdTdOyBicIaeytJNfiH4z9zNRGcQkwNeqcAU7Dm8c0dgMjSkNcPpMkEpmJtlf4gKRWhdKISlHmwRo_4/s640/2019-08-04-163827_777x333_scrot.png" width="640" /></a></div>
<br />
<br />I was a bit unsure at first, as I thought I would need to fight my way through the nine different config files of the dokuwiki debian package in /etc/dokuwiki<br /><br />However the issue was not so complicated: as the apache2 php module was disabled, apache2 was outputting the source code of dokuwiki instead of executing it. As you see, I don't php that often.<br /><br />A simple <br /><b><span style="font-family: "Courier New", Courier, monospace;">a2enmod php7.3<br />systemctl restart apache2</span></b><br /><br />fixed the issue.<br /><br />I understood the problem after noticing that <a href="http://docs.gurock.com/testrail-admin/howto-running-phpinfo">a simple phpinfo() would not get executed by the server.</a><br /><br />I would have expected the upgrade to automatically enable the new php7.3 module, since the oldstable php7.0 apache module was removed as part of the upgrade, but I am not sure what the Debian policy would recommend here, or if I am missing something else.<br />
If I can reproduce the issue in a upgrade scenario, I'll probably submit a bug to the php package maintainers.Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com1tag:blogger.com,1999:blog-7143401328277522244.post-34535265475159640092019-06-08T13:14:00.001+02:002019-06-08T13:18:48.501+02:00PowerShell on DebianI heard some time ago that Microsoft released their interactive and<br />
scripting language PowerShell under an opensource license (MIT) but I completely missed that they were providing a repository and ready to use packages <a href="https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-6#debian-9">for your favorite distribution.</a><br />
<br />
Anyway an apt-get away and that's it:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyCe5kwNMe9TRK_YBdJhgnawqkc6l7LQ3e4EPntJnOYmUsIvKoiA3ShuIFGv-GlGfgNpE_fN_tSoVZhi4mh70XkCzTTjAxsnhn70nq478xAG-2KJOYYaevkiEyUjxIWTIQ2gvVw3k38x0d/s1600/powershell_on_linux.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="337" data-original-width="960" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyCe5kwNMe9TRK_YBdJhgnawqkc6l7LQ3e4EPntJnOYmUsIvKoiA3ShuIFGv-GlGfgNpE_fN_tSoVZhi4mh70XkCzTTjAxsnhn70nq478xAG-2KJOYYaevkiEyUjxIWTIQ2gvVw3k38x0d/s1600/powershell_on_linux.png" /></a><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><b>New-Object net.sockets.tcpclient("libera.cc", 80)</b></span> opens a TCP connection to a target host, a quick way to test if a port is open ( look for Connected: True for a successful socket creation)Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-59914319502218530902019-03-17T19:28:00.001+01:002019-03-17T19:50:01.722+01:00Splitting a large mp3 / flac / ogg by detecting silence gapsIf you have a large audio file coming for instance from a whole music album, the excellent <b><span style="font-family: "courier new" , "courier" , monospace;">mp3splt</span></b> can do this for you:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><b>mp3splt -o @n-@f -s my_long_file.mp3</b></span><br />
<br />
will autodetect the silences, and create a list of tracks based on the large file.<br />
<br />
<span style="font-family: "Courier New", Courier, monospace;"><b>mp3splt</b></span> is available in the <a href="https://packages.debian.org/unstable/sound/mp3splt">Debian / Ubuntu archive</a>.Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-87331423013363430582018-11-20T18:46:00.002+01:002019-06-17T16:30:10.838+02:00Normalize a bunch of audio files to the same loudnessI had a bunch of audio files in a directory, each recorded live with different devices, and it proved very ear-painful to hear the audio files in a playlist because of the difference of loudness.<br />
To normalize audio filesm you can find a number of tool working with ID3 tags, but after testing with vlc, mplayer, and the pogo mp3 player none of them did produce a measurable change.
So I converted everything to wav, normalized the wav files, then converted back to mp3.<br />
<br />
delete funny chars and spaces in file names<br />
<code>detox music_dir
</code>
<br />
converting files to wav
is just a matter of <br />
<code># this uses zsh recursive globbing<br />for file in **/*.mp3 ; do ffmpeg -i $file "$(basename $file .mp3).wav"; done</code>
<br />
normalizing files
with the <b><code>normalize-audio</code></b> program, from the debian package of the same name.<br />
<code># this uses zsh recursive globbing</code> <br />
<code>normalize-audio **/*.wav
</code>
<br />
converting back to mp3<br />
<code></code><code>for file in **/*.wav ; do ffmpeg -b:a 192k -acodec libmp3lame -i $file "$(basename $file .wav).mp3"; done</code>
Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0tag:blogger.com,1999:blog-7143401328277522244.post-76197789773360580752018-09-02T11:59:00.001+02:002018-09-02T11:59:40.887+02:00Test if your microphone input is working from the command line on Linux<span style="font-family: "Courier New",Courier,monospace;">arecord -vv /dev/null</span><br />
<br />
That's it. Now clap your hand and observe the ascii vumeter.Emmanuel Kasperhttp://www.blogger.com/profile/09406228937831964688noreply@blogger.com0