Category: Computer

Simple way to plot multi-color line in matplotlib

There are multiple ways of plotting a multi-color line in matplotlib. A special property of the plot command allowing plotting multiple datasets in columns of 2D arrays makes it possible to do this with a single plot command using the cycler for the colors. Here are a sample code segment for this.

Generate sample data

import numpy as np
import matplotlib.pyplot as plt
from cycler import cycler

x = np.linspace(-np.pi,np.pi,9)
y = np.sin(x)
rng = np.random.default_rng(123)
c = rng.uniform(size=(len(x),3))

plt.scatter(x,y,c=c)
plt.show()

Quick plotting with line colors

plt.gca().set_prop_cycle(cycler('color',c[:-1]))
plt.plot(np.c_[x[:-1],x[1:]].T,np.c_[y[:-1],y[1:]].T)
plt.show()

More careful plotting with point colors

y_ = (y[:-1]+y[1:])/2
x_ = (x[:-1]+x[1:])/2

plt.gca().set_prop_cycle(cycler('color',c[:-1]))
plt.plot(np.c_[x[:-1],x_].T,np.c_[y[:-1],y_].T)
plt.gca().set_prop_cycle(cycler('color',c[1:]))
plt.plot(np.c_[x_,x[1:]].T,np.c_[y_,y[1:]].T)
plt.show()

Day of the week calculation

For given date in year-month-day, the day of the week can be deduced for the rule of leap years [1] and the fact that January 1, 1 is a Monday [2].

Since a year is a leap year if the year number is a multiple of 4 that is not a multiple of 100 that is not a multiple of 400. The number of leap year up to year y is given by:

def num_leap(y):
    return y//4-y//100+y//400

For a common year, the numbers of days for the twelve months are:

[31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]

The number of days before each month in a common year is calculated with:

p = 0
days_before_month = [
    [p, p := p + m][0] for m in
    [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
]
print('[',', '.join(map(str,days_before_month)),']',sep='')

The result is:

[0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334]

Since the extra day of a leap year is added to February, the leap status of current year is only relevant when month number m<3. We can calculate the number of days from January 1, 1 as:

def day_number(y,m,d):
    return (y-1)*365+num_leap(y-(m<3))+days_before_month[m-1]+d

Or, pack everything in the function:

def day_number(y,m,d):
    return (y-1)*365+(lambda y:y//4-y//100+y//400)(y-(m<3))+[0,31,59,90,120,151,181,212,243,273,304,334][m-1]+d

[1] https://en.wikipedia.org/wiki/Leap_year
[2] https://en.wikipedia.org/wiki/AD_1

Reverse SSH tunnel from a firewalled host

You have an SSH server A behind a firewall that you wish to connect to. You have no control over the firewall but has an account on an open SSH server B on the internet outside the firewall. You can setup a reverse tunnel from A to B so that you can connect to A through B from anywhere.

In the SSH configuration file A:~/.ssh/config, add the following section

Host b-tun
	HostName B
	ServerAliveInterval 30
	ExitOnForwardFailure yes
	RemoteForward 8822 localhost:22

Afterward, you can create the reverse tunnel on A by running ssh -Nf b-tun. and login with your credential on B or setup key authentication for automatic login. If the tunnel has been created successfully, you can now connect to the port 8822 of localhost on B to login to A, e.g., ssh -p 8822 localhost. If you have access to the nc command on B, you can add the section below to the ~/.ssh/config on a machine outside the firewall:

Host A
	ProxyCommand ssh -qax B 'nc -w 600 localhost 8822'
	ServerAliveInterval 30

You will then be able to connect to A with a simple ssh A on that machine.

You can also make the creation of the reverse tunnel automatic by creating a systemd unit, say, ~/.config/systemd/user/b-tun.service with the content

[Unit]
Description=Create SSH tunnel through B
After=network-online.target

[Service]
Type=idle
ExecStart=/usr/bin/ssh -N b-tun
RestartSec=39
Restart=always

[Install]
WantedBy=default.target

and start the process with systemctl --user start b-tun.service. You may need to enable unattended job for the user on A with loginctl enable-linger user as the root on A. To make the service start on boot automatically, you need to enable it with systemctl --user enable b-tun.service.

L1 or L2 in Support Vector Machine of scikit-learn

The SVC in the sklearn.svm sub-module claims to use “a squared l2 penalty”. However, the mathematical formulation in the user guide describes an L1 penalty term. Unless I have misunderstood the writing, there must be an error among the two.

Since the implementation claims to use the libsvm, which has similar documentation with linear slack terms, I am assuming the SVC is actually L1 instead of L2 as suggested by the API documentation.

Walrus operator can inline a block of code into one-liner

Walrus operator, or assignment expression, was introduced into Python at version 3.8. This saves some additional line of assignment in storing-and-checking scenario following evaluations.

However, it also brings the containers data types, such as a list, a step closer to the Turing completeness. Following is a piece of real code:

[d:=get_scans(fn),b:=d[0],c:=d[1],c[np.isin(b,bs)][np.argsort(b)]][-1]

Or, in a more structured format:

[
    d:=get_scans(fn),
    b:=d[0],
    c:=d[1],
    c[np.isin(b,bs)][np.argsort(b)]
][-1]

This performs some additional processing from the result returned by get_scans(fn) and turn the whole thing into a value by extracting the last element of the list. One application of this kind of composite expressions is the possibility of replacing traditional loops with list comprehension. For example, without assignment expressions, we need a loop to collect the data:

r = []
for fn in fns:
    [b,c] = get_scans(fn)[:2]
    r.append(c[np.isin(b,bs)][np.argsort(b)])
r = np.array(r)

Using the walrus operator, we can do it in “one-line” (which is broken down to several lines for clarity):

r = np.array([[
    d:=get_scans(fn),b:=d[0],c:=d[1],
    c[np.isin(b,bs)][np.argsort(b)]
][-1] for fn in fns])

Some may consider this is an abuse of the container types of Python. But, as long as it is used with care, it can help to make elegant code without hindering readability.

Opening Zotero record with a URL

When writing notes for research, it is desirable to have link that can open up citation management software with selected references so that we can easily access our personal entries on the citations.

This can be achieved by using the Zutilo add-on. It provides a menu item “Copy selected item links”. The link is in the form of zotero://select/library/items/XXXXXXXX. It is possible to configure the “mime type” for this protocol so it can be opened by xdg-open or a web browser.

Default values of GL_TEXTURE_MIN_FILTER and GL_TEXTURE_MAG_FILTER

From the documentation available at https://registry.khronos.org/OpenGL-Refpages/gl4/html/glTexParameter.xhtml, the default or initial values are:

GL_TEXTURE_MIN_FILTER = GL_NEAREST_MIPMAP_LINEAR
GL_TEXTURE_MAG_FILTER = GL_LINEAR

This makes glGenerateMipmap necessary after glTexImage2D. Otherwise, the texture rendering will not work properly.

Saving from pickle.dumps to h5py

Saving a chunk of pickled data to a h5py dataset with following code result in “ValueError: VLEN strings do not support embedded NULLs”. 

a = [1,2,3,4,5]
with h5py.File('test.h5','a') as f:
	f['a'] = pickle.dumps(a)

This can be worked around with, e.g., “np.string_”:

a = [1,2,3,4,5]
with h5py.File('test.h5','a') as f:
	f['a'] = np.string_(pickle.dumps(a))

To load it back:

with h5py.File('test.h5','r') as f:
	a = pickle.loads(f['a'][()])

I am not aware if there is any problem with this approach.

Dovecot 2.3.18 failed to load certificate with OpenSSL 3.0

This problem prevents user login with SSL to the Dovecot IMAP server. The errors in the journal look like this:

... imap-login: Error: Failed to initialize SSL server context: Can't load SSL certificate (ssl_cert setting): error:25066067:DSO support routines:dlfcn_load:could not load the shared library: filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory, error:25070067:DSO support routines:DSO_load:could not load the shared library, error:0E07506E:configuration file routines:module_load_dso:error loading dso: module=providers, path=providers, error:0E076071:configuration file routines:module_run:unknown module name: module=providers: ...

A fix/workaround was found at Bug#1011051: libssl3: upgrade to libssl3 broke my dovecot setup. By commenting out the line “providers = provider_sect” in the configuration file “/etc/ssl/openssl.cnf”, the service is restored. Hope this will be resolved properly in the near future.