xfs vs ext4 squential operation benchmark (both on HDD and nvme)

Methodology

sequentially writing a 5GB file with cache --
cat /dev/urandom | tee /dev/stdout | tee /dev/stdout| tee /dev/stdout| tee /dev/stdout| tee /dev/stdout| tee /dev/stdout | tee /dev/stdout| tee /dev/stdout| tee /dev/stdout | dd of=random iflag=fullblock bs=1M count=5120

Next write without cache --
rm random
sync; echo 3 > /proc/sys/vm/drop_caches
cat /dev/urandom | tee /dev/stdout | tee /dev/stdout| tee /dev/stdout| tee /dev/stdout| tee /dev/stdout| tee /dev/stdout | tee /dev/stdout| tee /dev/stdout| tee /dev/stdout | dd of=random iflag=fullblock bs=1M count=5120 oflag=direct

Read without cache --
sync; echo 3 > /proc/sys/vm/drop_caches
dd if=random of=/dev/null iflag=direct bs=1M

Read with cache --
dd if=random of=/dev/null bs=1M

 

Repeated read with cache --
sync; echo 3 > /proc/sys/vm/drop_caches
for i in {1..10}; do dd if=random of=/dev/null bs=1M; sleep 1; done

 

FS format parameters and mount options -- 

2 benchmarks will be done for XFS. one with rmapbt=0 and the other with rmapbt=1

These are the xfs parameters -- 

mkfs.xfs -f -m rmapbt=0,reflink=0

mkfs.xfs -f -m rmapbt=1,reflink=0

ext4 format options are either for large file or for both large and small files.

ext4 format options optimized for large files --
mkfs.ext4 -m 1 -O none,dir_index,extent,^flex_bg,^bigalloc,has_journal,large_file,sparse_super2,^uninit_bg 

 

ext4 format options optimized for both large and small files --
mkfs.ext4 -g 256 -G 4 -J size=100 -m 1 -C 2097152 -O none,bigalloc,extent,flex_bg,has_journal,large_file,sparse_super2,^uninit_bg,dir_index,dir_nlink,^sparse_super,^sparse_super2

xfs mount options -- 

mount -o logbufs=8,logbsize=256k,noquota,noatime

ext4 mount options (when formatted for large file optimization) -- 

mount -o noquota,noatime,data=writeback,journal_async_commit,inode_readahead_blks=32768,max_batch_time=10000000

Benchmark results

xfs rmapbt on vs off in nvme -- 

Without rmapbt
    sequentially writing a 5GB file with cache --
    2.3 GB/s
    
    Next write without cache --
    1.7 GB/s
    
    Read without cache --
    2.2 GB/s
    
    Read with cache --
    2.9 GB/s
    
    Repeated read with cache --
    2.8
    17.3
    16.2
    16.2
    16.2
    16.2
    16.3
    16.3
    16.1
    16.3
With rmapbt
    sequentially writing a 5GB file with cache --
    2.4 GB/s
    
    Next write without cache --
    1.7 GB/s
    
    Read without cache --
    2.2 GB/s
    
    Read with cache --
    2.8 GB/s
    
    Repeated read with cache --
    2.8 GB/s
    16.4 GB/s
    16.5 GB/s
    16.5 GB/s
    16.5 GB/s
    16.4 GB/s
    16.5 GB/s
    16.4 GB/s
    16.5 GB/s
    16.5 GB/s

Sequential read/write operations with rmapbt on/off in XFS

XFS (rmapbt=0) --
sequentially writing a 1GB file with cache --
116 MB/s,112 MB/s
Next write without cache --
105 MB/s
Read without cache --
104 MB/s
Read with cache --
104 MB/s
Read with cache again --
13.8 GB/s
Repeated read with cache --
This was done after formatting + sequentially writing a 1GB file with cache
105,17.4,17.3,17.3,17.3,17.3,17.4,17.4,17.4,17.4
Avg: 17.35555555555555555555

XFS format options with rmapbt=1 --
sequentially writing a 1GB file with cache --
115 MB/s
Repeated read with cache --
This was done after formatting + sequentially writing a 1GB file with cache
106, 13.9,13.8,13.4,13.9,13.7,14.1,13.9,13.8,14.0
Avg: 13.833333

Sequential read/write operations on ext4

ext4 (optimized for large files) --
sequentially writing a 1GB file with cache --
112 MB/s,112 MB/s
Next write without cache --
104 MB/s
Read without cache --
105 MB/s
Read with cache --
105 MB/s
Read with cache again --
11.2 GB/s
Repeated read with cache --
This was done after formatting + sequentially writing a 1GB file with cache
108,11.4,11.3,11.2,13.7,12.5,12.3,12.4,12.2,12.3
Avg:12.14444444444444444444

ext4 mount options optimized for both small and large files -
sequentially writing a 1GB file with cache --
115 MB/s
Repeated read with cache --
This was done after formatting + sequentially writing a 1GB file with cache
103,11.8,12.0,12.0,11.9,12.8,12.8,12.8,12.9,12.9
Avg: 12.433333

Conclusion -- 

For nvme/ssd, xfs with rmapbt on is the way for sequential operations on large file. This is also better than ext4 even for small file operations (benchmark published later).

For HDD Storage, xfs without rmapbt (or rmapbt=0) will perform the best.

Getting HP K209a-z to work on Gentoo (hpaio SANE backend)

This printer requires the hpaio SANE backend which will be made available if you install net-print/hplip[-minimal] along with sane-backends. You can have no USE enabled in SANE_BACKENDS and it'll still work.

k8sCertsFlatify: Script to dump TLS certificates in Kubernetes to flat files.

A script which extracts 1 or multiple tls/ssl certificates from a kubernetes cluster to PWD.
NOTE: Will not check the TLS certificate of the connecting kubernetes cluster as of the current time.
Switch --kubeconfig/-c <kubernetes kubeconfig file>. If not present default to ~/.kube/config
switch --namespaces/-n -- Namespace to dump certificates in. If not present, will dump certificates of all namespaces.
--context/-k -- The context to use in the kubeconfig file.
--dumpdir/-d -- Dump certificates to this directory instead of PWD
Will dump certificates in PWD in a directory with the name as the DNS to which the certificate belongs.

Install the Ruby gem (gem install k8sCertsFlatify) or use the docker image detechno/k8scertsflatify

Canon Pixma E470 on Linux.

This printer works out of the box on Debian 10 (which is deprecated as of now). It was added to CUPS automatically. I'm not sure which packages provided the filters or ppds...

The scanner also works out of the box with sane.

Apache + PHP: too many connections in CLOSE-WAIT state.

 This is happening because your PHP scripts are not exiting. They hang up for whatsoever reason (like waiting form some network i/o etc...). The client has requested a closure of the connection but apache will not close the connection (or consider it closed) until the script terminates, and therefore Apache will not close the TCP socket by doing the right system calls.

In case you where wondering, max_execution_time does not count the time taken by external commands which are executed or other external I/O activity like connection to databases etc...

Clang "error: unknown target triple 'unknown'" on Gentoo.

 If you've set 'LLVM_TARGETS=' remove it and rebuild llvm + clang (in order). I couldn't reproduce the bug however. If you face it, ensure to file a bug citing this blog or post because now there are multiple users facing it.

agetty autologin prompting for password

You've added --autologin to agetty in whatever ways (like modifying or creating a new systemd unit). In that vtty, the user DOES login but not without a password prompt.

Check for --login-options/-o and try removing it. It interferes with -a, --autologin

Prevent systemd from killing a child (sub-process) process on a service stop.

Because of cgroups, systemd knows which process belongs to which systemd unit even if it daemonizes (even to systemd/init). Therefore if you stop a systemd service unit (like a display manager), it’ll kill all processes which was spawned directly or indirectly by the display manager. To prevent this from happening, you need to play around with cgroup. 

Find the target PID which you want to avoid being killed. Then move it to another cgroup which systemd did not make (this is all about echoing it’s Pid to the new cgroup) --

 echo <pid> >> /sys/fs/cgroup/cgroup.procs

Now systemd has lost track of this process and you can stop the systemd service unit without systemd killing the process.

Washermod vs contact frame.

I recently got a contact frame and replaced it with washermod -- benchmarked it and found no difference.

Secure openwrt WOL with no open ports (firewall/nat etc...)

The objective of this article is to achieve WOL in a setup where Internet access is behind a NAT or has a firewall which allows no open connections. We'll also cover the security aspect using purely iptables (instead of openwrt's built in firewall) -- this's particularly important since the openwrt installed on the router is outdated and it's discontinued (so it won't receive any security updates).

To achieve WOL, we'll be using a simple shell script which will periodically download a text file and check it's contents; for a certain value within the text file, it'll trigger a WOL for a certain hardware address. Here is the script -- 

#! /bin/ash
while [[ j != k ]]
do
    if test '<wol string>' = "$(wget -q -O - -U 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Edg/91.0.864.37' --no-check-certificate '<URL of text file>')"
    then
        /usr/bin/etherwake -D -i '<interface>' <hardware address of your system>
        sleep 30
    fi
done

For this you need to install the etherwake package.

<wol string> is the string written in the text file. For this string the WOL signal will be emitted. Therefore to disable WOL, you need to modify the text file to anything else other than this string.

<URL of text file> is a HTTP link. This may point to an s3 object which is a good candidate or any online office document (something hosted by google drive). Regardless, you must be directly able to download a text file using the link using wget.

<interface> is the interface via which your to-be-wol system is accessible.

Make a file /usr/bin/wol.sh, write the script there and -- 

chmod 755 /usr/bin/wol.sh

Add /usr/bin/wol.sh to the local startup script (found in luci in the startup page) as -- 

/usr/bin/wol.sh &

 And you're done!

Now for the firewall part. I've disabled the buitin firewall of openwrt because it was not working as expected -- 

service firewall disable

Reboot router.

Add the firewall rules -- 

iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -o <router interface> -p icmp -s <router IP> -d <your system IP>,<default gateway IP>,255.255.255.255,<broadcast IP of your subnet> -j ACCEPT
iptables -A INPUT -i <router interface> -p icmp -s <your system IP>,<default gateway IP> -d <router IP>,255.255.255.255,<broadcast IP of your subnet>  -j ACCEPT
iptables -A INPUT -i <router interface> -p tcp -m conntrack --ctstate NEW,RELATED,ESTABLISHED --dport <ssh port of your router> -s <your system IP> -d <router IP> -j ACCEPT
iptables -A OUTPUT -o <router interface> -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -d <your system IP> -s <router IP> --sport <ssh port of your router> -j ACCEPT
iptables -A OUTPUT -o <router interface> -p udp -m conntrack --ctstate NEW,RELATED,ESTABLISHED --dport 53 -d <DNs server IP> -j ACCEPT
iptables -A INPUT -i <router interface> -p udp -m conntrack --ctstate RELATED,ESTABLISHED --sport 53 -s <DNs server IP> -j ACCEPT
iptables -A OUTPUT -o <router interface> -p udp -m conntrack --ctstate NEW,RELATED,ESTABLISHED --dport 123 -d <NTP server IP> -j ACCEPT
iptables -A INPUT -i <router interface> -p udp -m conntrack --ctstate RELATED,ESTABLISHED --sport 123 -s <NTP server IP> -j ACCEPT
iptables -A OUTPUT -o <router interface> -p tcp -m conntrack --ctstate NEW,RELATED,ESTABLISHED -d <list of public IPs> -s <router IP> -m multiport --dports 80,443 -j ACCEPT
iptables -A INPUT -i <router interface> -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -m multiport --sports 80,443 -s <list of public IPs> -d <router IP> -j ACCEPT
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

<your system IP> is the system using which you're accessing the router over SSH.

This system of rules assume you access the luci GUI over ssh tunneling which is recommended.

You need to change your ntp servers to something fixed -- otherwise most NTP server DNS has so many IPs behind it... Good luck finding such a service.

<list of public IPs> is the list of public IPs of the service provider hosting your text file which the WOL script will monitor. Best of luck finding that.

After ensuring you're not cut off ssh access (otherwise reboot and then reattempt to fix the firewall rules) -- 

iptables-save > /etc/custom-iptables

Then add to the local startup stript (via luci GUI) -- 

iptables-restore < /etc/custom-iptables

Test all desired functionality.

Washer mod results on an i3.

I noticed that the temps on my i3 (Alder lake) was pretty high for an i3. So I did a washer mod and calculated an approx 12 degree drop in temps. FYI.

Using the script command to record all your shell output and commands transparently.

 In your bashrc file (either /etc/bash.bashrc, or /etc/bashrc or /etc/bash/bashrc etc...) add the following lines by the very end -- 

if test -z "$script_running"; then export script_running=1; script -a <destination directory>`date +%s`.txt; exit; fi

AFTER creating <destination directory> -- this is the place where all your recordings will be placed.

Improving cooling of laminar cooler (on steroids, faster fans, mod/hack) by replacing it's stock fan.

The great thing about Intel's laminar coolers is that you can take the fan off by removing 4 screws -- 

Here I have it attached on the motherboard after removing the fan.

Now you can attach a much more powerful fan on it by using hot glue on the plastic clips (4 in no.; the thing that fixes the heat sink to the motherboard). If you wish to attach a smaller fan, you can stick it directly to the copper heat sink.

Hot glue sticks are good enough for the purpose and is easy to take off when the need arises. Here is the result -- 

Here I have a 92mm PWM server fan attached on the heat sink.

This resulted in 5 degree lower temps.

Promql query to get the average/max/min CPU utlization, network rate and memory

CPU utilization in the last 24 hours -- 

highest –
100 - min_over_time((avg without(cpu)(((node_cpu_seconds_total{mode=`idle`} - (node_cpu_seconds_total{mode=`idle`} offset 1m))/60*100)))[24h:1m])

lowest -- 

100 - max_over_time((avg without(cpu)(((node_cpu_seconds_total{mode=`idle`} - (node_cpu_seconds_total{mode=`idle`} offset 1m))/60*100)))[24h:1m])

 Average -- 

100 - ((avg without(cpu) (max_over_time(node_cpu_seconds_total{mode="idle"}[24h])) - avg without(cpu) (min_over_time(node_cpu_seconds_total{mode="idle"}[24h])))/86400*100)

Network upload/download rate (MBPS) for an interface in the last 24 hours -- 

Average -- 

((max_over_time(node_network_receive_bytes_total{device="team0"}[24h]) – min_over_time(node_network_receive_bytes_total{device="team0"}[24h]))/86400)/1024/1024

((max_over_time(node_network_transmit_bytes_total{device="team0"}[24h]) – min_over_time(node_network_transmit_bytes_total{device="team0"}[24h]))/86400)/1024/1024

Lowest -- 

min_over_time(((delta(node_network_receive_bytes_total{device="team0"}[1m])/60))[24h:1m])/1024/1024

min_over_time(((delta(node_network_transmit_bytes_total{device="team0"}[1m])/60))[24h:1m])/1024/1024

Highest -- 

max_over_time(((delta(node_network_receive_bytes_total{device="team0"}[1m])/60))[24h:1m])/1024/1024

max_over_time(((delta(node_network_transmit_bytes_total{device="team0"}[1m])/60))[24h:1m])/1024/1024

Memory utilization (in %) in the last 24 hours -- 

Average -- 

avg_over_time((((node_memory_MemTotal_bytes)-(node_memory_MemAvailable_bytes))/node_memory_MemTotal_bytes*100)[24h:1m])

minimum -- 

min_over_time((((node_memory_MemTotal_bytes)-(node_memory_MemAvailable_bytes))/node_memory_MemTotal_bytes*100)[24h:1m])

maximum -- 

max_over_time((((node_memory_MemTotal_bytes)-(node_memory_MemAvailable_bytes))/node_memory_MemTotal_bytes*100)[24h:1m])

Memory utilization (in GB) in the last 24 hours --

Average -- 

avg_over_time(((node_memory_MemTotal_bytes/1024/1024/1024)-(node_memory_MemAvailable_bytes/1024/1024/1024))[24h:1m])

Minimum -- 

min_over_time(((node_memory_MemTotal_bytes/1024/1024/1024)-(node_memory_MemAvailable_bytes/1024/1024/1024))[24h:1m])

Maximum -- 

max_over_time(((node_memory_MemTotal_bytes/1024/1024/1024)-(node_memory_MemAvailable_bytes/1024/1024/1024))[24h:1m])

Ignoring XXX because its extensions are not built. Try: gem pristine…

 After trying out whatever tips and tricks that others have suggested, and this issue still doesn't resolve, this maybe a permission issue; that's why things might running as root.

And no -- it's not less permissions, it maybe related to MORE permissions -- for certain files, the group or others executable permission bits might have been set. To fix this -- 

find <gem paths> -type f -perm -u=x -exec chmod g+x,o+x {} +

find <gem paths> -type f -perm -u=rx -exec chmod g+rx,o+rx {} +

Of course if you're planning to use the gems system wide, all files and directories must be readable -- 

find <gem path> -type f -exec chmod o+r,g+r {} +; find <gem path> -type d -exec chmod o+rx,g+rx {} +

Backporting gtk-gnutella on Debian buster.

 It seems Debian 10 does not have this package in the repository, but Debian unstable has. So we'll try building a deb for Debian buster -- 

aptitude install libdbus-1-dev libglib2.0-dev libgnutls28-dev=3.6.7-4+deb10u6 libgtk2.0-dev libxml2-dev zlib1g-dev fakeroot

apt-get source --compile gtk-gnutella

This'll result in the deb being generated. Install it -- 

dpkg -i gtk-gnutella_1.1.15-1_amd64.deb

Alternatively, you may download the deb directly -- 

https://drive.google.com/file/d/1YAMfQpgwWGWotwG7NZtRO-WNZMBHobCF/view?usp=sharing

Cleanup -- 

aptitude markauto libdbus-1-dev libglib2.0-dev libgnutls28-dev libgtk2.0-dev libxml2-dev zlib1g-dev fakeroot
apt-get autoremove

Debian buster -- Working VAAPI (hardware video decoding) for newer intel hardware (like ice lake/gen 11 intel GPU (UHD)).

In case you cannot get hardware video acceleration to work on your new Intel processor, apart from trying to install the backported kernel, you may also need a newer intel-media-va-driver (as of the current time 21.1.1 is the latest from testing).

In this article, it'll be shown how to backport these yourself (since no backports are available) from testing. Alternatively, you can find prebuild backports from here -- 

 https://drive.google.com/file/d/10rcxvetlJbe4wMUijficd-263S_QYhIj/view?usp=sharing

Extract and install all the debs (dpkg -i *.deb)

To test -- 

LIBVA_DRIVER_PATHS=/usr/lib/x86_64-linux-gnu/dri/ LIBVA_DRIVER_NAME=iHD vainfo

In case you want to build this yourself, take the following instructions -- 

Add the following to /etc/apt/sources.list -- 

deb http://mirror.csclub.uwaterloo.ca/debian-multimedia/ stable main
deb-src http://mirror.csclub.uwaterloo.ca/debian-multimedia/ stable main
#bullseye
deb http://mirror.csclub.uwaterloo.ca/debian/ bullseye main contrib non-free
deb-src http://mirror.csclub.uwaterloo.ca/debian/ bullseye main contrib non-free
deb http://security.debian.org/debian-security bullseye/updates main contrib non-free
deb-src http://security.debian.org/debian-security bullseye/updates main contrib non-free
deb http://mirror.csclub.uwaterloo.ca/debian/ bullseye-updates main contrib non-free
deb-src http://mirror.csclub.uwaterloo.ca/debian/ bullseye-updates main contrib non-free

#sid
deb http://mirror.csclub.uwaterloo.ca/debian/ sid main contrib non-free
deb-src http://mirror.csclub.uwaterloo.ca/debian/ sid main contrib non-free

Next install packages --

aptitude install debhelper=13.3.3~bpo10+1 dwz=0.13-5~bpo10+1 libdrm-dev libgl1-mesa-dev libwayland-dev libx11-dev libxext-dev libxfixes-dev pkg-config build-essential libset-scalar-perl

Generate debs to be installed -- 

apt-get source --compile libva=2.10.0-1

Install all the resulting debs -- 

dpkg -i libva-dev_2.10.0-1_amd64.deb libva-drm2_2.10.0-1_amd64.deb libva-glx2_2.10.0-1_amd64.deb libva-wayland2_2.10.0-1_amd64.deb libva-x11-2_2.10.0-1_amd64.deb libva2_2.10.0-1_amd64.deb

Install build-depends of intel-media-driver -- 

aptitude install debhelper=13.3.3~bpo10+1 dh-sequence-libva cmake libigdgmm-dev=20.4.1+ds1-1 libx11-dev pkg-config

Generate the debs -- 

apt-get source --compile intel-media-driver=21.1.1+dfsg1-1

And install the generated debs.

Cleanup -- 

aptitude markauto debhelper dwz libdrm-dev libgl1-mesa-dev libwayland-dev libx11-dev libxext-dev libxfixes-dev pkg-config build-essential libset-scalar-perl libva-dev libva-drm2 libva-glx2 libva-wayland2 libva-x11-2 libva2 dh-sequence-libva cmake libigdgmm-dev libx11-dev pkg-config

apt-get autoremove

Error: Server asked us to run CSD hostscan.

Anyconnect has provisions of a ‘CSD script’… via which basically a remote program which’ll be downloaded from the VPN server and will be executed on the host machine to gather information about it and to be sent to the server.

If a VPN server mandates running such a scan the following errors will come up –

"Error: Server asked us to run CSD hostscan."

For openconnect, you’ve to download external CSD scripts. There are 2 CSD scripts – which communicate to the VPN server either via post or by some other means.

https://gist.githubusercontent.com/l0ki000/56845c00fd2a0e76d688/raw/61fc41ac8aec53ae0f9f0dfbfa858c1740307de4/csd-wrapper.sh

The above is a script sends the collected info via non-POST means. Another official, openconnect CSD script sends it via POST. It’s called csd-post.sh. If you’ve used the wrong script, the following errors will occur –

"Refreshing +CSCOE+/sdesktop/wait.html after 1 second"

Repetitively.

In the above csd-wrapper.sh script, you’ve edit it and fill in your VPN host’s DNS name in an environment variable.

Switches to openconnect –

--csd-wrapper <path to CSD wrapper script>

--csd-user <user name> – Run the CSD script as this user.

Restricting access based on IP on NFS v4 with fsid=0

There’s a scenario when you want to restrict people from mounting things under a directory, for e.g. /home/test/ based on their IP address; but as you know the /etc/exports entry for /home/test/ which has fsid=0 must allow for Ips which is a superset of all other host entries in /etc/exports (and under /home/test); otherwise access will be denied for the other entries. Here you can use nocrossmnt. With nocrossmnt for the /etc/exports entry if you’ve mount –bind inside a directory X inside /home/test, the NFS server will not allow the client to descent into X unless you’ve another entry for X in /etc/exports and if it explicitly allows the client’s IP to mount it.

Running older systems (which need cgroupv1) on systems running over cgroupv2 (systemd.unified_cgroup_hierarchy)

Run the command -- mount | grep cgroup on your host system, and if you see the all the mount entries as cgroup2 fs (instead of cgroup), then you wont be able to run run older OSs as containers on this host. If you try to force cgroup2 over cgroupv1, the following errors will occur -- 

Cannot determine cgroup we are running in: No such file or directory

Failed to allocate manager object: No such file or director

An e.g. of what happens in centos 7 on lxc.

For older systems which don't support cgroupv2, you’ll need cgroupv1 mounted in /sys/fs/cgroup/systemd on the host. There doesn't seems to be way to do this using lxc.mount.auto = ; so you’ve to use scripts (lxc.hook.mount). For this script to mount a cgroup (named X) in the guest, a cgroup named X must also be mounted on the host; this same cgroup will be made available to to the guest. Alternatively, you may mount –bind in this script from the host’s cgroupv1 mounted directory to the guest’s directory; this’s a better approach since this allows you to create cgroups inside X exclusively for the container, so the guest may not play around with other processes's cgroups.
As an e.g. –
#! /bin/bash
mount -t tmpfs -o size=1M tmpfs $LXC_ROOTFS_MOUNT/sys/fs/cgroup/
mkdir -p $LXC_ROOTFS_MOUNT/sys/fs/cgroup/systemd
#mount -t cgroup -o none,name=cgroupv1 cgroupv1 $LXC_ROOTFS_MOUNT/sys/fs/cgroup/systemd &>> /tmp/script_out.log
mount --bind /tmp/cgroup1/lxc_containers $LXC_ROOTFS_MOUNT/sys/fs/cgroup/systemd
exit 0

Can't get cgroupv1 mounted no your host? Getting "already mounted or mount point busy." -- in this case ensure the cgroup that you're mounting is not being attached to any subsystem/controllers, which is the default behavior. This's the right approach -- 

mount -t cgroup -o none,name=lxc_compat systemd /tmp/cgroup1

Asus P1440FA-3410Z linux compatibility.

 This laptop in reality comes with Linux pre-installed (mine did); so is 100% linux compatible including the wifi.

Moto 3G (2015) (osprey) -- no audio from speaker or wifi.

I think this's a hardware issue.

To try and resolve the issue, make a call on mobile network and turn the speaker on. The issue must resolve.

[spreadsheet][ods]Unsprung/rotating mass (wheel/sprocket/tyre) power loss calculator for cars bikes and motorcycles

 In case you're wondering how much power will you get when you replace you wheel or sprocket or tyres to lighter ones, this spreadsheet is for you.

https://drive.google.com/file/d/1bM1nyAbg6gJ8RFpCKRujqXe6EF4voAlF/view?usp=sharing

Open in either libreoffice or google docs. 

Realize that the power loss is not only dependent on unsprung mass, but also on other factors such as wind resistance (your vehicle's aerodynamics), mechanical losses etc... unsprung mass is only one of the losses. These other losses changes over the speed in which you're at, so while calculating, apart from dimensions, you've to also enter the speed and the time required to reach that speed in order to determine the power lost because of the wheel/sprocket/tyre. Another reason why you need to enter the speed and time it takes to reach that speed is that power is a function of energy. So if your vehicle takes less time to reach a certain speed, the mass will take less time to attain that RPM, but ultimately will result in having the same energy. Thus, same energy attained in less time means more power taken by up the rotating mass while accelerating.

Only fill the required values in column B against the non-colored cells. The colored cells are calculated values.

D-Link DWM-222 4G on Linux.

Will work on any new Linux distribution out of the box. No need to install the 'driver's.

In case yours is an old Linux distribution, just eject the detected corresponding cdrom device (/dev/sr0 or /dev/sr1, sr2 etc...) and a modem will be spawned which can be used just as a standard modem using your networkmanager or using wvdial.

In networkmanager or wvdial, just do not set the APN (or INIT3 string), the device will pick it up automatically. Older versions of networkmanager do not allow this, so you may face issues on it. In this caseu use wvdial with a high BAUD rate.

Mystery high feaver (ranging from 99 to 103) comes and goes with extreme chills (sometimes)

One of my relatives (old) had this kind of mysterious fever. It used to go away in 3 days, and then used to come back within around 5 days. The first day, fever was high (like 103), then it used to reduce over the next 2 days. The fever was high promenantly at night.

'Modern' medicine and 'specialists' got stuck with lung infection and various tests which gave no results. The blood test results were erratic and inconsistent pointing to a mix of all diseases. This had been going on with 6 months.

Then he though of taking a remedy of alternative medicine based on Indian origin (something related to Yoga). The practitioner said this's a result of food allergy. Apart from giving medications, he a black and whitelist of foods to avoid and prefer.

And that was it ... fever was gone.

Matching encoded URLs using regexp/regular expressions (optionally in fail2ban).

Your regular expression can fail against attackers doing attacks by encoding their URLs; fail2ban will not detect those, neither your regular expression; But you can modify your regexpes to match these encoded URLs also even in mixed form (partly encoded, and partly not); create regular expressions to replace each character with something like -- 

(c|%63|%43)

Here I replace c with the above; this will match c, and it's capital and small form in encoded URLs. In fail2ban you need to replace the % with a %% -- 

(c|%%63|%%43)

So I write .php as -- 

(\.|%%2E)(p|%%70|%%50)(h|%%68|%%48)(p|%%70|%%50)

You may begin the regular expression with (?i) in fail2ban or define it as (?i:<your regexp>) elsewhere to ignore case of the character (so C and c are alike and %2e and %2E is also alike.

To convert URLs to their encoded form I've created a simple script -- 

#! /usr/bin/ruby
# Converts the input string to a regular expression which will match the string either in the URL encoded form or mixed or unencoded form and case insensitively
# First argument is the string.
input = ARGV[0].dup
input.gsub!(/a/,'(a|%61|%41)')
input.gsub!(/b/,'(b|%62|%42)')
input.gsub!(/c/,'(c|%63|%43)')
input.gsub!(/d/,'(d|%64|%44)')
input.gsub!(/e/,'(e|%65|%45)')
input.gsub!(/f/,'(f|%66|%46)')
input.gsub!(/g/,'(g|%67|%47)')
input.gsub!(/h/,'(h|%68|%48)')
input.gsub!(/i/,'(i|%69|%49)')
input.gsub!(/j/,'(j|%6A|%4A)')
input.gsub!(/k/,'(k|%6B|%4B)')
input.gsub!(/l/,'(l|%6C|%4C)')
input.gsub!(/m/,'(m|%6D|%4D)')
input.gsub!(/n/,'(n|%6E|%4E)')
input.gsub!(/o/,'(o|%6F|%4F)')
input.gsub!(/p/,'(p|%70|%50)')
input.gsub!(/q/,'(q|%71|%51)')
input.gsub!(/r/,'(r|%72|%52)')
input.gsub!(/s/,'(s|%73|%53)')
input.gsub!(/t/,'(t|%74|%54)')
input.gsub!(/u/,'(u|%75|%55)')
input.gsub!(/v/,'(v|%76|%56)')
input.gsub!(/w/,'(w|%77|%57)')
input.gsub!(/x/,'(x|%78|%58)')
input.gsub!(/y/,'(y|%79|%59)')
input.gsub!(/z/,'(z|%7A|%5A)')
input.gsub!(/\./,'(\.|%2E)')
input.gsub!(/-/,'(-|%2D)')
puts input

The first argument to this script will be your text input.

Nikon A900 review and issues/problems/drawbacks.

Everything about the camera is expected; for the size, it's the best that you can get at night photography (which is still deficient) as of 2019.

Before you buy, these are a few drawbacks --

1) The slowest shutter speed is 8 seconds in reality. No it's not 25; 25 seconds is given by some 'mode' which useless actually.
2) Black round bands are seen on the edges of the pictures sometimes. I think this is is because of the image stabilizer. Solution is to zoom in and then zoom out and soon it'll fix itself. This issue calls in for a warranty claim! And yes -- warranty has been claimed. The highly abused lens has now been replaced.
3) Autofocus is terrible! Even while shooting videos. And there's no manual focus to make matters worst. To provide a global e.g. just try to shoot moon so it's craters can be seen. This's not possible without fully zooming into the moon.
4) The camera hangs sometimes.
5) Wi-Fi picture transfer feature is non-standard. It requires a Windows 'driver'; so it wont work on Linux/BSD. I use P2P over USB instead.
6) Transferring pictures to phone over wifi is broken. So is remote photography (actually the phone never connects to the 'smart device' over wifi).
7) Battery charging is extremely slow.
8) Battery display has only 2 levels -- full, and low (that 50% mark is not, medium, it's low actually, and you're hardly going to get any backup beyond that).

On the very plus size, the IS is very good! Audio recording is great too!

Lineage/resurrection remix/Android: Audio stops when earphones are plugged in.

Sometimes, when you plug in your earphones, the audio stop coming (from the earphones), but the notification sounds continue to come. You've to restart your device multiple times to resolve the issue.

The corresponding logs in logcat --

08-11 22:41:36.375   315  8872 E qcvirt  :      [vendor/qcom/proprietary/mm-audio-noship/audio-effects/safx/android-adapter/qcvirt/qcvirt.c:477] Assertion fail: status == PPSUCCESS

Solution -- disable the equalizer in audioFX.

Linux technologies (kernel, bash etc...) support for Windows -- for a better monopoly.

And when you start to think in modern times when Microsoft loves Linux and opensource; the question arises, is it really the truth? Does Microsoft really love opensource?

No, in fact Microsoft is still trying to enforce it's monopoly and support for opensource technologies makes it's monopoly stronger.

Reviewing from a few pages of history realize why Microsoft is a monopoly -- 
1) It keeps all protocols hidden
2) All technologies will be patented in the US (Microsoft tax)
3) It tires to hide the formats of files that their programs use and when they open up the format, the specs are not complete (to ensure only their programs are able to open their files) and ladened with patent warnings.

None of this has changed; but now Linux programs can run on Windows officially. So to the unsuspecting consumer -- Windows has the power to run both their propitiatory, cryptic and hidden Windows program and open Windows files along with Linux capabilities; so the obvious question is, why will it switch to Linux? So let the monopoly commence and be better.

Fixing kernel: "unregister_netdevice: waiting for to become free. Usage count = "

This's a kernel bug which'll cause docker to hang and is triggered by you stopping a container (which possibly does not stop gracefully, i.e. does not respond to SIGTERM). The only solution is a reboot. It's speculated that this's a network namespace related problem and reproducible on all lxc/docker/rkt etc....

The thing that worked for me to reduce the probability of this bug is removing limits from the docker systemd service. Newer systemd has a default limit even if you didnt set it. Set LimitNOFILE=1048576, LimitNPROC=infinity, LimitCORE=infinity, TasksMax=infinity in docker systemd unit and this may just fix the issue; this also reduced the load average (CPU based).

Bash history sanitize/cleaner.

instead of cleaning your bash history, this script will remove problematic history entries, thus sanitize it.

#! /bin/bash
# Without argument will print what it'll delete. If 1st argument is y, then it'll clean the history of the user.
# The regular expressions catch the good commands which are to be retained.
echo 'Would delete commands -- '
grep -vP --regexp='^[a-zA-Z0-9/./.#~>]' ~/.bash_history
grep -vP --regexp='^.{0,1000}$' ~/.bash_history
if test "$1" == 'y'
then
 grep -P --regexp='^[a-zA-Z0-9/.#~>]' ~/.bash_history | grep -P --regexp='^.{0,1000}$' > /tmp/bash_history_cleaned || exit
 mv /tmp/bash_history_cleaned ~/.bash_history
fi

Read the comments for how to get this to work.

The mysterious case of engine oil thinning (AKA oil sheering)

If you're someone who rides at high RPM and have a vehicle which's capable of going to high RPMs (6000+) your engine oil might be subject to a phenomenon called oil sheering which thins down your engine oil and makes it's grade lower. Bad quality engine oil means more sheering.

So it's better to check your engine oil for quality. Now question is what to check? Feel the viscosity of the engine oil on your fingers, and if it does not feel oily (and feels more watery), the engine oil is subjected to sheering and has thinned down.

For other aspects, the engine oil might be ok -- it wont smell burnt, will not leave a soot when you rub it and of course will not be excessively thick; but regardless, if it has thinned this much, it's time for a change, and next time switch to fully synthetic engine oil since engine oils must not thin like this at all.

Understanding inner workings of crossdev.

The crossdev executable is going to install a toolchain in your host machine for a foreign architecture as regular ebuilds. The root of this foreign architecture (RF) will be placed in /usr/ where is in the same syntax as the -t switch to crossdev.

Now question is, where do you get the ebuild of the toolchain for the foreign architecture? crossdev creates a separate overlay (the directory of the overlay must be added to PORTDIR_OVERLAY in make.conf of the host machine) which contains packages within a newly created category (in that overlay) named ; the packages (belonging to the toolchain) in this overlay are basically symlinks to certain directories (belonging to the toolchain) on the host system. These packages within category only contains the essential components of the toolchain and are merged just like any other package. The result of the installation is that the toolchain for the foreign architecture is installed with the prefix (-); emerge is also installed in the same way.

As said before these executables (including emerge) operate assuming RF as the root dir. All configuration in RF will be respected by these commands which includes make.conf, package.use, package.keyword etc... this includes the overlays, but it appears that the gentoo portage tree of the host is always searched. Don't understand why, or the mechanism.

Executing this crossdev with the switches will start building the toolchain using the emerge command itself (unless you've passed some switches which present it from doing so).

These packages will populate a few things in /usr/.

After these have been build, you can either build @system thus create the installation from scratch without a stage3; however I dont think this'll work, so I would say you extract a stage 3 in /usr/ without overwriting the make.conf in it since it's a special make.conf which works with the toolchain command as installed on the host; however this make.conf is underoptimized, so I would say you merge both the make.conf with the crossdev specific parts commented out when actually running the system and when using crossdev, you toggle the crossdev parts.

crossdev will also make a make.profile but unfortunately it's symlinked to the wrong profile, overwrite it with the one in the stage3 tarballs and change it to your preferred profile.

Whatever change you do, remember that running crossdev again will overwrite those, so 1) back them up and 2) never run crossdev again. Upgrade these packages using your host's PM.

As of using eix in RF, use EIX_PREFIX= for that.

systemd-logind -- "Failed to start Login Service." on Debian 8

This issue comes on LXC when you enable dbus so you can communicate to systemd; don't know why this triggers.

You need to add the sys_resource capability top fix this.

Using curb with custom verb/methods, headers and body.

The Easy method of Curb does not allow arbitrary request methods to be made.

Instead of using Easy, you can use the Curl module. In curb's rubydoc, you'll see hardly any description of these methods, so I explain them here.

There are methods named after REST verbs in the Curl module and there's http method which allows you to send custom verbs. These methods also take a code block in which case it'll pass a newly created object (just like Easy.new) as the first argument to the body. All instance methods which applies to the object of Easy.new also apply to this object. Before existing the code block, the request will be made.

Script to group files/directories to reach the closest desired/target size.

Program takes a list of files and directories as a set of last arguments and outputs the paths of each of the files/directories which has been selected to reach the target size (which is the first argument). Will ignore files which do not exist.

First argument -- target size in bytes.

Off the selected files, will print each of them in a new line with their size, separated with a tab in the same line. Specify units as KB, MB, GB, TB in the environment variable UNIT.

script -- 
https://paste.ubuntu.com/25206842/

Before you run, install run 'gem install ClosestSum' on your system as root. This'll install the
ClosestSum gem which's a library implementing the algorithm.

lxc-console prints duplicate characters (crippled/corrupt console).

This happen on Debian. After an upgrade to lxc 2.0.x, lxc-console does not work well. Produces duplicate characters, cannot log in because of the same reason.


In lxc-1.x you must have made getty@tty*.service somewhere in /etc/systemd/system/. They're no longer needed. Remove them to fix the issue.

-flto-partition=balanced and -flto-partition=1to1 benchmark

I compiled grep and libpcre, once with balanced and once with 1to1 (and of course with -lto=4 in both the cases), and the result was that grep was 10% faster with balanced which makes sense too.

Persistent/resilient ssh sessions for unstable internet connections.

Long back SSH had introduced an experimental 'roaming' feature where changes to the ssh client IP resulted in resuming the session on the server regardless of the changed IP. This feature was never implemented on the server, rending it useless on the client but causing a vulnerability.

Instead of using roaming, a much better approach is using screen with shell scripting. This has serious advantages like resuming the session over a different client machine, the program running in foreground won't slow down even if the terminal (or Internet connection) is slow etc...

Just install screen on the server and run the following commands for a presistant session --

while [[ j != k ]]; do ssh -tt screen -r / -p 0; done

This'll reconnect on disconnecting. You can use tabs in screen and take multiple sessions over the same screen instance. Open the other tabs using --

while [[ j != k ]]; do ssh -tt screen -r / -p 1; done

while [[ j != k ]]; do ssh -tt screen -r / -p 2; done

while [[ j != k ]]; do ssh -tt screen -r / -p 3; done

For tabs numbered 1, 2, 3 etc...

I use Gentoo's default config for the screen on the server, it works great!

Incremental backup system of your Android app settings and your data.

Phone is an unreliable device which can always be stolen and get bricked regardless of how expensive the phone is or how must reliable the manufacture claims to be. This's primarily because of the fact that the storage cannot be detached from the phone and the storage's content is highly integrated to the phone's hardware.

So I've created a system to regularly backup your app data  in an incremental way -- so the old data gets retained and snapshot of the latest backups is also taken all using less space. You can restore all this data to a new phone or revert an older version of the data to your existing phone (maybe to get it unbricked without loosing all your settings).

Of course I know about Google's cloud backup, but in my experience it's unreliable, requires a lot of bandwidth and works only on select (Google only) apps. This works on all apps. I also know about adb backup and restore feature, but that also does not work on all apps.

This system requires sshelper app and it must run in the background all the time. You must configure key based login as specified in this (Public-key (passwordless) logins) tutorial. After configuring that, you can disable password based login and disable the 'keep device awake' checkbox to improve on the battery and security.

Other things that is requires is root access.

sshelper installs a busybox. You need to use the tar command cron command of that. The scripts I've deployed use exactly that --

Place this script in /system/bin/custom_data_backup.sh -- 

# backups data only if the latest one is less than 12 hours old
#! /system/bin/sh
SECONDS=$((12*60*60))
SD_CARD="Your sdcard mount point"
mkdir $SD_CARD/custom_backup
latest=`ls -tr $SD_CARD/custom_backup/ | tail -1`
if test \( -z "$latest" \) -o \( `date +%s` -gt $(($latest + $SECONDS)) \)
then
 cd /data/data && /data/data/com.arachnoid.sshelper/bin/tar -cpf $SD_CARD/custom_backup/`date +%s` *
fi

This can be done by the command (as root) --

 vim /system/bin/custom_data_backup.sh

And then pressing 'i' to got to edit mode. Then paste, make changes, then press ESC a few times and type ':x' (without the single quotes).

Modify SD_CARD variable to point to the mount point where your sdcard is mounted. Use the mount command to see the various mount points. One of these must be your SDcard. cd to that place and verify by looking at it's contents if it is indeed the place.

It happens that Android has a bug or a problem etc... the system call which these basic utilities use to seep for a certain period of time is inaccurate. This systemcall never returns when sleep is done for a long period of time. This system works around this problem.

Next you need to setup cron.

Run these commands root -- 

mkdir /data/data/com.arachnoid.sshelper/spool
vim /data/data/com.arachnoid.sshelper/spool/root

Now press i, then copy paste the following text -- 

*/30 * * * * custom_data_backup.sh

Then press ESC a few times and type ':x' (without the single quotes).

Then run -- 

vim /etc/init.d/99backup.sh

Now press i, then copy paste the following text -- 

#!/system/bin/sh
mount -o remount,rw /
ln -s /system/bin /bin
mount -o remount,ro /
/data/data/com.arachnoid.sshelper/bin/crond -c /data/data/com.arachnoid.sshelper/spool

Then press ESC a few times and type ':x' (without the single quotes).

Then run -- 

chmod 755 /etc/init.d/99backup.sh.

Install universal init.d and enable init.d scripts support. If you've a rom which has inbuilt support of init.d, you will not require this.

After this you must see backups created in directory /custom_backup. The file name is the timestamp of the date at which the backup was taken. These files will be created ~ every 12 hours.

Next configure your desktop system to take incremental backups. This's the script which runs a daemon and keeps running until the backups are complete. -- 

https://paste.ubuntu.com/24521288/

Modify the variables ($*) as per your needs and as per your phone.

$sdcard is the directory where backups are placed (that custom_backup directory) on your phone. It must be a full path.

$sshkey is the ssh key which you generated for key-based login to your phone.

$ip is the IP of your phone on the wi-fi network. You must configure your phone for static IP.

$backupDest is the path of your desktop system where the backups will be placed.

$sdcardInternal is the full path of your internal SDcard in your phone. $sdcardExternal is similar but for your external SDcard.

Your SDcards will be incrementally backed up too.

This script is resilient to failure -- so even if your phone is gone, it'll wait until your backup is complete after you've come back and your phone's presence is detected. Setup a cron job once a day which triggers this script. If you wanna know -- the script is released under GPL v3 license.