Zobrazujú sa príspevky s označením Bash. Zobraziť všetky príspevky
Zobrazujú sa príspevky s označením Bash. Zobraziť všetky príspevky

pondelok 26. októbra 2020

Bash tips I

Inputs to functions:

https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html

Outputs from functions:

https://www.linuxjournal.com/content/return-values-bash-functions

Check for empty variable:

https://www.cyberciti.biz/faq/unix-linux-bash-script-check-if-variable-is-empty/

Keyboard shortcuts:

http://teohm.com/blog/shortcuts-to-move-faster-in-bash-command-line/

utorok 13. októbra 2020

How to extract raw value from JSON and cut it into pieces by size

Sample data:

{
  "var_name1": {},
  "var_name2": "string",
  "var_name3": "long string",
  "var_name4": 15
}

Fill Bash variable with extracted long string raw value from JSON:

VAR_NAME=`cat file.json | jq --raw-output '.var_name3'`

Then reference "chunks" of data using Bash variable index modifiers, e.g. by 255 characters:

  • ${VAR_NAME:0:255}
  • ${VAR_NAME:255:255}
  • ${VAR_NAME:510:255}
  • ${VAR_NAME:765:255}
  • ${VAR_NAME:1020:255}
One-liner alternative extracting and displaying output in single command line:

VAR_NAME=`cat file.json  | jq --raw-output '.var_name3'` ; echo $VAR_NAME ; echo ${VAR_NAME:0:255} ; echo ${VAR_NAME:255:255} ; echo ${VAR_NAME:510:255} ; echo ${VAR_NAME:765:255} ; echo ${VAR_NAME:1020:255}

streda 18. decembra 2019

WSL home directory migration to MS OneDrive

motivation

There were problems with using WSL on multiple computers (separate home directories) and accessing Google Drive from them (for details, see older posts in this blog, there are some problems and uncomfortable workarounds, when using Google Drive in read-write mode from WSL). Therefore created proof of concept how to have just single "centralized" home directory on OneDrive, accessed from multiple WSLs via C: mounted in WSL as /mnt/c.

setup

It is practical (but not mandatory) to have the same path to OneDrive folder on each Windows computer (unify user and home folder names in Windows, if feeling it that way), e.g.:

C:\Users\Richard\OneDrive\
what in WSL means:
/mnt/c/Users/Richard/OneDrive/

and it is also practical having all "known folders" (as "Desktop", "Documents", "Pictures", etc.) also migrated to OneDrive, to be more sure, that all your files are safely backed by cloud (another story).

Let's assume user and home folder names "Richard" in Windows and "richard" in WSL.

I have not tested following commands as written, just summarized what I did, and it may not be complete or precise enough, because there was a lot of tuning, so be careful and think before doing anything.

From WSL on all computers determined to having /home/richard/ centralized via OneDrive do this:

sudo ln -s /mnt/c/Users/Richard/OneDrive/ /home/richard2
sudo chown richard:richard /home/richard2
sudo mv /home/richard /home/richard_backup
sudo mv /home/richard2 /home/richard

Now you can start new WSL session and see, if your (or richard's) home directory is already placed into OneDrive. From now, you have your home folder accessible from any WSL, where you have your OneDrive and this "symlinking mount" in place.

(Note: it is also possible to change user name in Windows and there is more than one way to do it.)

migration

Migration of files and folders from your WSL (or other Linux/Unix) home directory can be very specific and differ case by case. Simplistic example:

cp /home/richard_backup/* /home/richard_backup/.* /home/richard

Maybe you will want to migrate only subset of all files and folders, and maybe you will want to do more sorting what to place to which OneDrive subfolder, because in this step you are integrating (merging) your your WSL home directory with your OneDrive, and you want to have an order, not chaos in your files, after that. Also be careful about risk of unwanted file replacements, resolve collisions before it's too late.

file permissions

One caveat is, that file permissions are not set correctly in WSL, and this mask hack in ~/.profile can be useful:

if [[ "$(umask)" = "0000" ]]; then
  umask 0022 # or umask 0027 or umask 0077 for enhanced confidentiality, further reading
fi

but it was not enough in this case and files were switched from originally non-executables to executables, without knowing exactly why. The consequence of executableness were also missing colors in terminal, because ~/.profile need not to be executable for Bash to execute it (counter-intuitive, but safer).

Mask applies to future permisions changes, but past permissions changes can be fixed e.g. this way:

# all permissions removal from all unauthorized:
chmod -R o-rwx /home/richard/

# (potentially dangerous, depending on the specific contents of OneDrive)
sudo find /home/richard/ -type f -exec chmod a-x {} +

# fix executability selectively:
chmod 750 /home/richard/Workspace/*/.git/hooks/{pre-commit,post-commit}

In order to avoid doing this every time changing WSL instance with potentially different UID, it is also practical to use the same UID according to /etc/passwd in every WSL instance. 

perl

When there is directory ~/.cpan in migrated folder, you may decide not to transfer it, but doing this instead of it:

perl -MCPAN -e shell
install Bundle::CPAN
reload index
reload cpan
exit

other hacks


streda 6. novembra 2019

Speeding up filesystem search indexing in WSL by keeping unnecessary drives unmounted

The Problem

Indexing search for locate was very slow, sometimes maybe infinetely. Idexing was starting up by sudo updatedb, but not finishing.

The Hypothesis

As seen via mount command in WSL, there were mounted several drives slowing down indexing search for locate:
  • C: (default system drive of Windows host) - needed sometimes
  • G: (drive from Google Drive File Stream) - however, not working correctly
  • multiple (?) N: drives (resulted from these last month experiments) - not needed already
I wanted to unmount and keep them unmounted.

The Solution

The /etc/wsl.conf was not existing initially in WSL (ls -al /etc/wsl.conf), so created it this way

sudo bash -c "echo [automount] >> /etc/wsl.conf"
sudo bash -c "echo enabled=false >> /etc/wsl.conf"

Restart WSL instance via Windows command-prompt (cmd.exe):

wsl --list --running
wsl --terminate "Ubuntu-18.04"ubuntu1804

Now we see in WSL via mount, that those Windows drives in WSL are not mounted, and sudo updatedb is remarkably faster.

Drives can be mounted and unmounted on demand - example:

sudo mount -t drvfs C: /mnt/c
sudo umount /mnt/c

More information about wsl.conf: https://devblogs.microsoft.com/commandline/automatically-configuring-wsl/.

utorok 8. októbra 2019

Windows Subsystem for Linux (WSL) + Google Drive mount

Motivation

There was a need to access files mounted to Windows 10 machine via Google Drive File Stream (GDFS, as G: drive) from WSL, to be able to work with them on the same machine with Linux tools like vim, bash, etc., not to be dependent on separate Linux machine, nor inefficiently installing those tools in Windows (some could work, some not without problems or at all), nor copy them after modifications from local to GDFS locations other way (manually).

Components

Solution consists of:
  • Windows 10 OS
  • drive G: mounted by Google Drive File Stream
  • OpenSSH server for Windows (running as Windows 10 service)
  • SFTP Net Drive
  • Windows Subsystem for Linux (WSL)
Tools used during implementation:
  • Windows PowerShell
  • WSL Bash

Hacks honorable to be mentioned

Steps

  1. install Ubuntu 18.04 LTS based WSL in Windows
  2. install Google Drive File Stream and connect to the service (mount as e.g. G:)
  3. install OpenSSH Server for Windows - optional feature (Settings > Apps > Manage optional features > Add a feature > OpenSSH Server > Install
  4. via Services management enable, set to start automatically & start OpenSSH Authentication Client and then via administrative PowerShell:
    • Start-Service ssh-agent
    • Start-Service sshd
    • Install-Module -Force OpenSSHUtils
  5. comment AuthorizedKeysFile in C:\ProgramData\ssh\sshd_config
  6. non-administrative PowerShell:
  7. (troubleshoot if needed) and restart OpenSSH Server
  8. download SFTP Net Drive and install it (+ register to start on OS startup), connect with authorized username to localhost (mount as e.g. N:), there are these alternatives:

  9. set installed program to start automatically after log-on (click on program in start menu, open file location, Windows+R: shell:startup, create shortcut of the program there)
  10. make directory symlink (if using SFTP Net Drive Free) or directory junction (if using SFTP Net Drive V2 or Full) on Windows from mounted user profile to G: - now G: can be accessed also from WSL as /mnt/n, but after mount in WSL:
  11. from cmd.exe: ubuntu
  12. mount N: from WSL: sudo mkdir /mnt/n; sudo mount -t drvfs N: /mnt/n

Further hacks

There were problems with perpetually changing inodes of files trying to write modifications by vim, therefore this workaround is currently in place:

cat ~/.vimrc

set nobackup
set backupcopy=yes
set noswapfile
set noundofile
set nowritebackup

Automatic mount of N: in WSL with owner's privileges:

tail -n 10 ~/.profile

if [ ! -d "/mnt/n/" ]; then
        sudo mkdir /mnt/n
fi

sudo mount -t drvfs -o uid=1001,gid=1001 N: /mnt/n

# mount some folder
if [ ! -d "SYMLINKED_FOLDER_NAME" ]; then
        rm -f SYMLINKED_FOLDER_NAME
        ln -s /mnt/n/GDrive/... SYMLINKED_FOLDER_NAME
fi

Requires user to be in admin or sudo group in /etc/group and this settings in /etc/sudoers (via visudo in WSL) - not very secure setup, but WSL is not "mission critical server" :) :

sudo cat /etc/sudoers

# Members of the admin group may gain root privileges
%admin ALL=(ALL) NOPASSWD: ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) NOPASSWD: ALL

Conclusion

The purpose of this article is me to be able re-run these steps on another computers or user profiles and can be continually improved in the future, to be more exact.