How to add TAP adapter interface in Windows

This article was useful to add interface into Windows OS quickly. Also this differentiation between TAP and TUN.

Acronyms explanation:

TAP = Terminating Access Point

TUN = Terminating Unbroadcasting Node

VPN = Virtual Private Network

How to insert non-breaking hyphen

Here is non-breaking hyphen: ‑. You can copy & paste it freely where you want (it is open-source).

It was forked from this one (credit them for the original work).

OpenConnect as Cisco AnyConnect VPN client better alternative

Cisco AnyConnect VPN client is sometimes too restrictive (e.g. disconnecting internet on client side, when split tunnelling is not configured on server side, or running under privileged user - root or SYSTEM), and there comes OpenConnect giving more freedom to the user.

OpenConnect is truly multi-platform and partially also OpenConnect-GUI bundling OpenConnect for Windows and macOS. This GUI wrapper gives to using OpenConnect very similar way and feeling, as GUI of Cisco AnyConnect VPN client.

In Windows, after VPN connection established, if "split tunnelling" is not allowed on (Cisco) VPN server (concentrator), this can be bypassed on client side by increasing metric of TAP interface (to be higher than metric of physical network interface), and configuration of routes (slices) to be routed to VPN, but not anything other. This can be done e.g. via administrative shell (cmd.exe):

route add <IPsubnet> mask <mask> <IPdefaultGW-VPN> IF <IFnumber>
netsh interface ip set interface <IFnumber> metric=5

Routes, interfaces, default gateways, etc., can be displayed this way:

route print

Above commands may be chained (e.g. within the BAT file) this way (see https://stackoverflow.com/questions/8055371/how-do-i-run-two-commands-in-one-line-in-windows-cmd for details):

cmd /k "route print & route-add-command & netsh-interface-command & route print"

For macOS, this GUI wrapper is experimental, so OpenConnect can be installed also directly, not as part of GUI bundle (with sample usage with vpn-slice, to do client-side "split tunnelling" - allowing internet usage bypassing VPN):

(sudo) brew install openconnect
(sudo) brew install vpn-slice
(sudo) openconnect https://<address> -s 'vpn-slice <IPsubnet>/<mask>'

Similar setup should be possible also in Linux.

More info:

• routing table modification in Windows
• vpn-slice (currently lacking Windows OS support)
• using netstat and route on Oracle Solaris

SCP via hop nodes

 There are various ways, how to SCP through intermediary nodes, e.g. ProxyCommand (using netcat = nc), ProxyJump (scp -J), pipes, tunnels, ...

There is a new project hopscp, doing it without any of them, but prospectively may incorporate them in the future too, to make its possiblities wider.

By default, in its initial version, it is implementing this multi-hop SCP by sequential secure copying through whole chain, from source to target. Current maximum is:

[source_end] -> [source_hop] -> [local_host] -> [target_hop] -> [target_end]

For more information, download hopscp, read its heading synopsis, or run it to see its output in command-line.

Software license: The Unlicense.

HTTP(s) checks with httpstat.us & curl

Useful site for HTTP(S) testing: https://httpstat.us/

Example 1:

curl --connect-timeout 5 -m 10 https://httpstat.us/200?sleep=7000 > test.curl

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100     6    0     6    0     0      0      0 --:--:--  0:00:07 --:--:--     1

echo $?

0

Example 2:

curl --connect-timeout 5 -m 10 https://httpstat.us/200?sleep=10000 > test.curl

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0

curl: (28) Operation timed out after 10001 milliseconds with 0 out of -1 bytes received

echo $?

28

Notes about timeouts in curl:

https://unix.stackexchange.com/questions/94604/does-curl-have-a-timeout/94612

Managing log rotation in Linux

 Add new file to rotation:

• copy one of /etc/logrotate.d/* configuration files to working (e.g. home) directory, name it e.g. new-file
• vim new-file
• change path to point to new (existing) file to be rotated
• change other parameters, if needed
• sudo mv new-file /etc/logrotate.d/
• sudo chown root:root /etc/logrotate.d/new-file

Dry run:

logrotate -d /etc/logrotate.d/new-file

Forced rotation:

sudo logrotate -vf /etc/logrotate.d/new-file

More info:

man logrotate
https://www.tecmint.com/install-logrotate-to-manage-log-rotation-in-linux/

Bash tips I

Inputs to functions:

https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html

Outputs from functions:

https://www.linuxjournal.com/content/return-values-bash-functions

Check for empty variable:

https://www.cyberciti.biz/faq/unix-linux-bash-script-check-if-variable-is-empty/

Keyboard shortcuts:

http://teohm.com/blog/shortcuts-to-move-faster-in-bash-command-line/

4 ways to search and replace with regular expressions

 See here.

Bonus: grep for literal strings

How to analyze active cron jobs in CentOS 7

 sudo cat /var/log/cron | cut -d ' ' -f 6- | sort | uniq -c | sort -nr

Statistical analysis of MySQL/MariaDB slow queries

One-liner:

sudo grep -e '^select' -e '^insert' -e '^update' -e '^delete' /srv/data/mysql/mysql_slow.log | cut -d ' ' -f 1-3 | sort | uniq -c | sort -nr | head

How to extract raw value from JSON and cut it into pieces by size

Sample data:

{
  "var_name1": {},
  "var_name2": "string",
  "var_name3": "long string",
  "var_name4": 15
}

Fill Bash variable with extracted long string raw value from JSON:

VAR_NAME=`cat file.json | jq --raw-output '.var_name3'`

Then reference "chunks" of data using Bash variable index modifiers, e.g. by 255 characters:

• ${VAR_NAME:0:255}
• ${VAR_NAME:255:255}
• ${VAR_NAME:510:255}
• ${VAR_NAME:765:255}
• ${VAR_NAME:1020:255}

One-liner alternative extracting and displaying output in single command line:

VAR_NAME=`cat file.json  | jq --raw-output '.var_name3'` ; echo $VAR_NAME ; echo ${VAR_NAME:0:255} ; echo ${VAR_NAME:255:255} ; echo ${VAR_NAME:510:255} ; echo ${VAR_NAME:765:255} ; echo ${VAR_NAME:1020:255}

How to check website availability for various users

https://www.isitdownrightnow.com/

https://downforeveryoneorjustme.com/

How to configure service principal for automated / scripted logins to Azure CLI

Steps:

• (check permissions)
• Azure Active Directory (AD) (AAD) level
• Azure Subscription level
  
• app registration
• AAD > App registrations > New registration 
• get tenant (directory) ID and app (client) ID
• configure authentication ( (select app) > Certificates & secrets )
  
• certificate-based (> Certificates > Upload certificate)
  
• password-based (> Client secrets > New client secret)
  
• app role assignment
• Subscriptions > (select) > Access control (IAM) > Add role assignment
• (configure access policies on resources)
  

Using Azure portal:

https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

Using Azure CLI:

https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli

Login via Azure CLI (password-based):

az login --service-principal --tenant TENANT_ID  --username APP_ID --password PASSWORD

 Login via Azure CLI (certificate-based):

az login --service-principal --tenant TENANT_ID --username APP_ID --password /path/to/cert

Other sign-in options:

https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli

How to commit & push local changes to separate (new) Git branch

git checkout -b new-branch

# more files can be added in this step too, wildcards supported
git add /path/to/locally/changed/file

git commit -m "commit message"

git show-branch

git push origin new-branch

Inspired by https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging.

How to setup NextCloud (NC) client on Ubuntu

This howto inspired by this article was tested on Ubuntu 16.04 LTS, but should also work on 18.04 LTS and other non-LTS versions.

Installation and startup of NC desktop sync client

sudo add-apt-repository ppa:nextcloud-devs/client
sudo apt update
sudo apt install nextcloud-client
mkdir ~/nextcloud.user@nextcloud.service # sync folder creation
nextcloud # setup authentication and choose created sync folder

Change sync folder icon

It can be useful for user to know, that NC sync folder is "special". We can do it by setting custom folder icon via folder properties in Nautilus. E.g. this icon can be used: /usr/share/icons/Humanity/places/48/folder-remote.svg.

Migrate Ubuntu/Unity/GNOME known folders to NC

mv ~/{Desktop,Documents,Downloads,Music,Pictures,Public,Templates,Videos} ~/nextcloud.user@nextcloud.service
ln -s ~/nextcloud.user@nextcloud.service/{Desktop,Documents,Downloads,Music,Pictures,Public,Templates,Videos} ~

After migration, symlinks to known folders in home directory preserve their icons, but not in NC sync folder, so they can be also changed as above, to correspond with their original locations.

How to upgrade Google Chrome on Ubuntu

Tested on Ubuntu 16.04 LTS (and should work also on 18.04 LTS and other non-LTS versions):

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
# on 32-bit system remove [arch=amd64] from:
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get update
sudo apt-get upgrade google-chrome-stable

Inspired by this article.

WSL home directory migration to MS OneDrive

motivation

There were problems with using WSL on multiple computers (separate home directories) and accessing Google Drive from them (for details, see older posts in this blog, there are some problems and uncomfortable workarounds, when using Google Drive in read-write mode from WSL). Therefore created proof of concept how to have just single "centralized" home directory on OneDrive, accessed from multiple WSLs via C: mounted in WSL as /mnt/c.

setup

It is practical (but not mandatory) to have the same path to OneDrive folder on each Windows computer (unify user and home folder names in Windows, if feeling it that way), e.g.:

C:\Users\Richard\OneDrive\

what in WSL means:

/mnt/c/Users/Richard/OneDrive/

and it is also practical having all "known folders" (as "Desktop", "Documents", "Pictures", etc.) also migrated to OneDrive, to be more sure, that all your files are safely backed by cloud (another story).

Let's assume user and home folder names "Richard" in Windows and "richard" in WSL.

I have not tested following commands as written, just summarized what I did, and it may not be complete or precise enough, because there was a lot of tuning, so be careful and think before doing anything.

From WSL on all computers determined to having /home/richard/ centralized via OneDrive do this:

sudo ln -s /mnt/c/Users/Richard/OneDrive/ /home/richard2

sudo chown richard:richard /home/richard2

sudo mv /home/richard /home/richard_backup

sudo mv /home/richard2 /home/richard

Now you can start new WSL session and see, if your (or richard's) home directory is already placed into OneDrive. From now, you have your home folder accessible from any WSL, where you have your OneDrive and this "symlinking mount" in place.

(Note: it is also possible to change user name in Windows and there is more than one way to do it.)

migration

Migration of files and folders from your WSL (or other Linux/Unix) home directory can be very specific and differ case by case. Simplistic example:

cp /home/richard_backup/* /home/richard_backup/.* /home/richard

Maybe you will want to migrate only subset of all files and folders, and maybe you will want to do more sorting what to place to which OneDrive subfolder, because in this step you are integrating (merging) your your WSL home directory with your OneDrive, and you want to have an order, not chaos in your files, after that. Also be careful about risk of unwanted file replacements, resolve collisions before it's too late.

file permissions

One caveat is, that file permissions are not set correctly in WSL, and this mask hack in ~/.profile can be useful:

if [[ "$(umask)" = "0000" ]]; then

  umask 0022 # or umask 0027 or umask 0077 for enhanced confidentiality, further reading

fi

but it was not enough in this case and files were switched from originally non-executables to executables, without knowing exactly why. The consequence of executableness were also missing colors in terminal, because ~/.profile need not to be executable for Bash to execute it (counter-intuitive, but safer).

Mask applies to future permisions changes, but past permissions changes can be fixed e.g. this way:

# all permissions removal from all unauthorized:

chmod -R o-rwx /home/richard/

# executable permission removal only from files:

# (potentially dangerous, depending on the specific contents of OneDrive)

sudo find /home/richard/ -type f -exec chmod a-x {} +

# fix executability selectively:

chmod 750 /home/richard/Workspace/*/.git/hooks/{pre-commit,post-commit}

In order to avoid doing this every time changing WSL instance with potentially different UID, it is also practical to use the same UID according to /etc/passwd in every WSL instance. 

perl

When there is directory ~/.cpan in migrated folder, you may decide not to transfer it, but doing this instead of it:

perl -MCPAN -e shell

install Bundle::CPAN

reload index

reload cpan

exit

other hacks

https://stackoverflow.com/questions/38779801/move-wsl-bash-on-windows-root-filesystem-to-another-hard-drive

https://www.makeuseof.com/tag/how-to-prevent-auto-added-files-folders-in-quick-access/
https://www.howtogeek.com/396117/how-to-enable-or-disable-a-windows-10-user-account/

Hooking git to update blob ID ($Id$) on every commit automatically

Git does not update $Id$ placeholder in working copies automatically out-of-box. One way to achieved that, is described in this blog post. Lets assume, that your git-versioned project folder is your current working directory.

Set .gitattributes

First, ensure, that files, wich needs to have $Id$ populated with their current blob ID, have set ident attribute in .gitattributes file, e.g.:

*       ident whitespace export-subs

More information about .gitattributes.

Create post-commit hook

Create .git/hooks/post-commit file with this content:

#!/bin/bash

# change all files temporarily, to checkout them back, but with updated $Id$ (only that, which were changed before commit)

echo | tee --append *

git checkout *

Make this file executable, e.g.:

chmod 755 .git/hooks/post-commit

After this, each $Id$ occurence should be replaced with $Id: <blob_ID> $ in each file, whose file name matched pattern in .gitattributes with ident.

Maybe post-commit file exists, when applying this, and is serving some purpose already. In such case, you'll need to somehow integrate the script above into existing post-commit script.

Checking all $Id$ occurences in versioned files

grep '\$Id' *

More information about git post-commit hook. This "hack" was inspired by this post.

Improving visibility of multiple hard-links of the same file

Command below color-differentiate files (in current working directory) having 2 to 9 hard-links (in whole filesystem) and prefixing all files (in current working directory) with command for finding all directory entries for particular inode.

Tested on WSL / Ubuntu 18.04:

ls -dali --time-style +"%Y-%m-%d %H:%M" * .* | sed 's/^/sudo find \/ 2> \/dev\/null -inum /g' | grep -e "[-xtTsS] [234567890] " -e ""

alias lnshow='ls -dali --time-style +"%Y-%m-%d %H:%M" * .* | sed "s/^/sudo find \/ 2> \/dev\/null -inum /g" | grep -e "[-xtTsS] [234567890] " -e ""'

echo "alias lnshow='ls -dali --time-style +\"%Y-%m-%d %H:%M\" * .* | sed \"s/^/sudo find \/ 2> \/dev\/null -inum /g\" | grep -e \"[-xtTsS] [234567890] \" -e \"\"'" >> ~/.profile

lnshow

OS command-line interfaces basics (slides from workshop)

https://docs.google.com/presentation/d/1tV_sAoAvWvQDR6enrTyWo9N66uhOm7ifMPGPTzfLGi4