Thu, Oct 21, 21, cmder bash commandline tools hotkeys and how to create sh file to automate actions

from bash pipeline.

checing serial port connection by debugging

dmesg | egrep -i --color 'serail|ttyS'

setserial -g /dev/ttyS[0123]

screen /dev/device baud-rate
screen /dev/ttyS- 19200
screen /dev/ttyUSB0 19200,cs8

sudo screen /dev/ttyUSB0 115200

nmcli device

ip l show

ip a show eth0

route -n

ip r
ip r | grep default
ip l show | grep -A1 eth0
ip l show | egrep -A1 'eth0|lo'

sudo ifconfig
sudo /sbin/ifconfig

nmcli device
nmcli connection

nmcli radio
nmcli wifi off
nmcli wifi on
nmcli device wifi list

nmcli device wifi connect nixcraft_guest passowrd {my password here}
nmcli wwan off
nmcli wwan on

to scp from a remote with a port number

scp -r -P [portnumber] user@ip:/pathtofile /local/pathtoafile/

'rsync -a ./dist/spa/* -e 'ssh -p 2222' --progress --delete tonylee@eggs.or.kr:/controller'

changing default branch of github

hahwul

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

git diff for files yet to be pushed

git diff --stat --cahced origin/master

git diff --numstat origin/master

remove files from a commit

git reset --soft HEAD^ or HEAD~1
git reset HEAD file

"to completely delet the file
git rm --cached <file>

git ccommit --amend

pushing a series of commits on separate push actions

git push origin commit-sha:branch_name
git push origin +HEAD^:branch_name
git push origin +HEAD~1:branch_name

+HEAD^ denotes a previous commit to the lastes and +HEAD~1 refers to the rest of commits except the current or the latest commit.

git log origin/branch_name..HEAD

to display the size of a commit before push

git log -1 --stat

copying a same file over to multiple location

f=text.txt; tee ~/folder{1..2}/$f < ~/$f > /dev/null
# sample implementation on learningjavascript
touch .gitkeep

f=repos/portfolio/documentation/webdev-node/.gitkeep; tee chapter{1..10}/$f < ~/$f > /dev/null

Tree /F

│   .gitkeep
│
├───chapter1
│       .gitkeep
│       index.html
│       index.html~
│
├───chapter10
│       .gitkeep
│
├───chapter2
│       .gitkeep
│
├───chapter3
│       .gitkeep
│

mkdir -p

mkdir sa{1..50}
mkdir -p sa{1..50}/sax{1..50}
mkdir {a-z}12345 
mkdir {1,2,3}
mkdir test{01..10}
mkdir -p `date '+%y%m%d'`/{1,2,3} 
mkdir -p $USER/{1,2,3}

download using request api

 wget --no-parent -r -l inf --wait 5 --random-wait 'http://WEBSITE.com/DIRECTORY' 

curl -LO http://example.com/someFile.type

add suffix to each file in a folder

for i in *.png; do mv "$i" "${i%.*}_3.6.14.png"; done  # It replaces .png in all the .png files with _3.6.14.png.
mv $i ${i%.*}_3.6.14.png Rename original .png files with the filename+_3.6.14.png. # ${i%.*} Anything after last dot would be cutdown. So .png part would be cutoff from the filename.

cloning usb stick to an image

  1. download etcher from download and install linux x64 version
  2. you can do this when you have either img file or bootable diskinternals
  3. run this cli as described in link
dmesg | grep -i usb
dmesg | grep -i 'attached'

lsblk 

sudo fdisk -l /dev/sdX

sudo dd if=/dev/sdX of=/path/to/file.img bs=SIZE bs=4 status=progrss

sudo chown vivek:USERS xxx.img

sudo apt install gddrescue

ddrescue /dev/INPUT /dev/OUTPUT

lspci
lspci | grep -i eth

ifconfig eth0

ip link
ip -c link

ifconfig -s -a
netstat -i

ethtool eth0

sudo lshw -short -c network
sudo lshw -c network

to check ip / port availability on remote server

sudo apt install nmap

sudo apt install telnetd

sudo nmap -A ip -p PORT

telent javaxxxxx 14550
sudo nmap -F IP

git clone branches

git config --global alias.clone-branches '! git branch -a | sed -n "/\/HEAD /d; /\/master$/d; /remotes/p;" | xargs -L1 git checkout -t'

to create multiple branches, run the following cli;

for branch in alpha{1..3}; do git checkout -b $branch; done; git push origin --all

git colour output to a file

You can pass -c color.ui=always to any git command and it will keep coloring on redirection. For example: git -c color.ui=always status > file

example

git create repo via api

curl -i -u 'aiegoo:$(ghp-token)' https://api.github.com/user/repos -d '{"name": "udemy-qt5", "private": true, "description": "pf3.36io.co/udemy"}'

Bash pipeline

appending_headoffile

Bash automation is simple and delivers any job scripted to perform. I love it because it touches on the process/pid of a task.

github repo

automation

curious programmer

I want to create a symlink for all hidden files excluding the hidden directories. I want to put my configuration files in my home directory into version control.

I need to exclude the hidden directories because they contain binaries and what not.

Using this one liner, I can create a symbolic link for every hidden file in my home directory to my working (or target) directory.

for f in .*; do if [[ -f $f ]]; then ln -s /home/me/$f /home/me/working/directory/$f; fi; done

I execute this from my source directory.

I can verify that my symlinks exist in my target directory

cd /home/me/working/directory
ls -lah

I should see a few lines that look like this:

lrwxrwxrwx. 1 me me   26 Jan 27 10:49 .zsh_history -> /home/me/.zsh_history
lrwxrwxrwx. 1 me me   20 Jan 27 10:49 .zshrc -> /home/me/.zshrc

This post forms part of a sequence of command line references that I will be writing where I forget the command or its syntax or find it interesting enough to document it.

Although it is easily Google-able, there are usually a chain of commands that I want kept together for ease of use.

The commands I use should be universal but just in case, I am running Fedora release 28 (Twenty Eight) and Zsh.

As far as I understand, bash±tar files typically have the tar or tar.gz extensions. bash±tar is the archive that can preserve permissions and directory structures and tar.gz is the compressed bash±gzip-ed bash±tar file.

Listing the contents of an archive

You can list the contents of an archive bash±tar -tf archive.tar.gz without having to extract the data. When you want to see more information bash±tar -tvf archive.tar.gz prints out details about the file including the permissions, owner, group, file size, modified date and time and the filename.

Your can also run these commands as bash±tar --file archive.tar.gz --list and bash±tar --list --file archive.tar.gz --verbose

What are all these switches?

  • **t – list**: tells tar to list everything in the archive
  • **v – verbose**: verbose (optional), prints out as much information as it can for you
  • f | – file: specifies the file of the archive you want to list. (This must always be the last flag as it precedes the filename in the command unless you use –file)

Creating a new archive

Archive

Create a new archive for the awesome directory bash±tar -cf archive.tar awesome or bash±tar --create --file awesome. You can create an archive from any working directory bash±tar -cf archive.tar /home/user/workspace/awesome

bash±tar -cvf archive.tar awesome or bash±tar --create --verbose --file awesome will print out the files names that have been added to the archive.

Compressed archive

The same switches apply as above, however we include the -z or --gzip switches to indicate that the archive must be compressed.

Create a compressed gzip tar archive bash±tar -cvzf archive.tar.gz awesome.json or bash±tar --create --verbose --gzip --file archive.tar.gz awesome.json for the awesome.json file.

Create a compressed gzip tar archive bash±tar -cvzf archive.tar.gz /path/to/awesome for the /path/to/awesome directory.

If you want to archive and compress multiple members, chain the files and directories bash±tar -cvzf archive.tar.gz /path/to/something another/awesome somewhere/sauce.json

Exclusions

In cases where you need to omit specific files or directories from an archive bash±tar -czvf archive.tar.gz --exclude=\*.gz /path/to/awesome will exclude any files with the .gz extension when creating the archive on path /path/to/awesome.

To exclude multiple files and directions, exclusions can be chained bash±tar -czvf archive.tar.gz --exclude=something.tar --exclude=somewhere /path/to/awesome The archive on path /path/to/awesome is created which excludes the something.tar file and somewhere directory.

What are all these switches?

  • **c –create**: tells tar to create a new archive
  • **v –verbose**: verbose (optional), prints out as much information as it can for you
  • **z –gzip**: compresses the archive using gzip _(you can also use
    -j –bzip2 for bzip2 compression)_
  • f | –file: specifies the file of the archive name that you want to create. (This must always be the last flag as it precedes the file name in the command unless you use the –file switch)
  • –exclude: exclude files given as a pattern

Clean up

Sometimes I need to clean up files in a directory once I have created the archive. An example is one of my logs directories. Once I have archived all the logs, I want to delete them but keep my archive and any existing archives. Using Zsh I run

# In Zsh, you can use ^ to negate
# a pattern with extendedglob enabled
setopt extendedglob
rm -- ^*.tar.gz

Updating an archive

bash±tar cannot update compressed archives so you need to create an uncompressed archive. The -u or --update switch will append files newer than the copy in the archive.

# Create a new archive without compression
tar -cvf archive.tar /path/to/archive
# Update or add files in /path/to/archive
# tar will update and changed or added files
tar -uvf archive.tar /path/to/awesome
# You can then compress your archive
gzip archive.tar.gz

Extracting data

Extract data bash±tar -xf archive.tar or bash±tar --extract --file archive.tar to its local directory.

Extract a compressed archive bash±tar -xvzf archive.tar.gz to its local directory. This extracts the files and decompresses them using gzip.

Extract a compressed archive to another directory bash±tar -xvzC /path/to/awesome -f sauce.tar The files in sauce.tar.gz are extracted to /path/to/awesome

What are all these switches?

  • **x extract**: tells tar to extract the files from the archive
  • **v –verbose**: verbose (optional), prints out as much information as it can for you
  • **z –gzip**: tells tar to decompress the archive using gzip - you can use **j –bzip2** for bzip2
  • **C –directory**: change to directory
  • f | –file: specifies the file of the archive you want to extract. (This must always be the last flag as it precedes the filename in the command)

Extracting specific files

You don’t have to extract the entire file. If you have a tar file but are only interested in a few files then grab them out of the tar file:

  1. List the contents of the tar file bash±tar -ztvf archive.tar.gz | grep filename and look for the file name(s) you are interested in.

  2. Extract the file bash±tar -zxvf archive.tar.gz ./awesome.clj or directory bash±tar --extract --file=archive.tar.gz src

Extracting wildcards

If you can’t use a rigid file name or directory then you can use wildcards (globbing patterns) bash±tar -xf archive.tar --wildcards --no-anchored "*.clj"

--wildcards tells tar to accept globbing patterns while --no-anchored tells it that the pattern applies to the member names after any / delimiter.

Comparing against the file system

-d, --diff, --compare finds the differences between the archive and the file system. The arguments are optional and you specify archive members to compare. If not given, the current working directory is assumed.

This is handy when you want to see whether there are files in the bash±tar file that are not yet in the local filesystem or visa versa. It also reports differences in attributes including the file size, mode, owner, modification date and contents.

The following example compares the archive members ‘rock’, ‘blues’ and ‘funk’ in the archive ‘bluesrock.tar’ with files of the same name in the file system. (Note that there is no file, `funk’; tar will report an error message.)

# this section and example was found at gnu.org
# http://www.gnu.org/software/tar/manual/html_node/compare.html
tar --compare --file=bluesrock.tar rock blues funk
rock
blues
tar: funk not found in archive

Using zip

This is outside the scope of this post but I have the base commands available for in case I need them.

To list the contents of the archive without unzipping it bash±less file.zip or bash±unzip -l file.zip.

To create a new archive bash±zip file.zip file1 file2 file3 or for a directory bash±zip -r file.zip /path/to/archive.

To exclude files from an archive being created bash±zip -9 -r --exclude='*.zip' file.zip /path/to/archive.

To unzip the archive bash±unzip file.zip.

Cheat sheet

List

tar -tf archive.tar.gz
tar -tvf archive.tar.gz

Create

Archive

tar -cvf archive.tar file1 file2 file3

Compressed archive

tar -cvzf archive.tar.gz file1 file2 file3

Exclusions

tar -czvf archive.tar.gz --exclude=\*.gz file1 file2 file3
tar -czvf archive.tar.gz --exclude=\*.gz --exclude=file\* file1 file2 file3

Exclusion files except some

setopt extendedglob
rm -- ^*.tar.gz

Update

tar -cvf archive.tar /path/to/archive
tar -uvf archive.tar member/director
gzip archive.tar

Extract

Archive

tar -xvf archive.tar

Compressed

tar -xvzf archive.tar.gz
tar -xvC directory -f archive.tar

Compare / Diff

tar --compare --file=archive.tar file1 file2 file3

Zip

less archive.zip
unzip -l archive.zip
zip archive.zip file1 file2 file3
zip -r archive.zip /path/to/archive
zip -9 -r --exclude='*.zip' archive.zip /path/to/archive
unzip archive.zip

References

Thi

Bash commands are mainly supported in MacOS, Linux but also support in Windows. You can use integrated tools for using bash on these platforms.

👉 Note: Terminals. 👉 Note: Bash screen.

Tools

Hotkeys

  • Ctrl + C : interrupt current tasks.
  • Ctrl + L : clear the screen.
  • Tab : autocomplete the commands / directories / file names / ….
  • Ctrl + Shift + V : paste from clipboard.
  • For a long list: Enter to continue to read, q to quit.
  • Ctrl + A : move cursor to the beginning of the line.

Multiple commands

# run at once
command_1 && command_2

.sh file

# using script: file.sh
#!/bin/sh
echo 'some info'
command_1
command_2

# and then sh file.sh

# with arguments
$file1 = $1
wc $file1 # word count

# multiple input args
for FILE1 in "$@"; do
	wc $FILE1
done

NAME="defaut" # default value! DON'T HAVE SPACE!!!
# with flags
while getopts n:f: option; do
	case "${option}"
		in
			n) NAME=${OPTARG};;
			f) FILE=${OPTARG};;
	esac
done

echo $NAME
wc $FILE

# how to use?
sh test.sh -n "ThiD" -f test.md

Search / grep / sed

# all files / folders containing 'abc'
ls | grep -i abc

# find command lines containing 'abc'
dpkg -l | grep -i abc

# search and extract a part of result
pip show numpy
# Location: /usr/lib/python3/dist-packages
pip show numpy | sed -n 's/Location: //p'
# /usr/lib/python3/dist-packages

Check info

System

# Change password
passwd

# DISK SPACE
df -h

# like monitor
top

# MEM USAGE
free -m

# ALL ENV
printenv

# add new
export ABC=/xyz/thi/

# NVIDIA
nvidia-smi
lspci -nn | grep '\[03' # another way

# list of devices
lsusb

# list display screen
xrandr --listactivemonitors

# CPU
cat /proc/cpuinfo | grep 'model name' | uniq # model
cat /proc/cpuinfo | grep 'vendor' | uniq # vendor
cat /proc/cpuinfo | grep processor | wc -l # number of processes

Folders / Files

prepending-file

find . -name "*.md" -print | while read fn; do echo fixing $fn; cat .test.txt "$fn" > $fn.modified; mv $fn.modified $f done

I have copied a test.txt file with a frontmatter into the first line of a file.

sed -n '/aa/,/fff/p' file1 > tmp
sed '1 r tmp' file2

# CHANGE ACTIVE DIR
cd <dir>
cd # to the startup dir
cd / # to root
cd .. # to father dir
cd - # back to previous dir

# CREATE NEW FOLDER
mkdir <dir>

# LIST
ls
ls -a # including hidden
ls | grep 'ubuntu' # files containing 'ubuntu' in name

# CURRENT PATH
pwd

# FOLDER/FILE SIZE
du -hs <directory / file>
# `h` : human readable (6.7G)
# `s` : display size

# all folders/files of current folder
du -hs * | sort -rh

# only folders
du -sh ./*/

# only first 5 retrieves
du -h /home/thi/ | sort -rh | head -5

# REMOVING
rm <file>
rm -f <file> # force to remove
rm -rf <dir> # remove folder
rmdir <empty-dir> # remove empty

# COMPRESS
zip file.zip file/folder
unzip file.zip # decompress

# PRINT TREE folder structure
tree
tree -d # only folders
tree -d -I 'abc' # except folder "abc"
tree -I 'abc|xyz' # except folder "abc" and "xyz"
tree -I 'test_*|__pycache__|__init__.py' # use wildcat
tree -L 2 # level 2
tree -P 'test_' # list only files starting with "test_"

Permission

👉 Chmod code calculator.

# list of groups
groups

# which groups a user belongs to
group <user_name>
id -nG # or

# check info of a current user
id <user_name>

# list all members of a group
grep <group_name> /etc/group

# CHECK PERMISSION
ls -l
ls -l <file>

# ADD existing USER to existing GROUP
sudo usermod -a -G groupName userName

# CHANGE PERMISSION
chown <user>:<group> file
chown -R thi:root folder # folder & children

Network

# CHECK IP
ifconfig
ipconfig # windows

# DOWNLOAD A FILE
wget https://website.com/filename.ext

# open ports
sudo apt install nmap
nmap localhost

# very simple server
python3 -m http.server # localhost:8000
python3 -m http.server 1337 # localhost:1337

# current running servers
sudo apt install net-tools
netstat -lepunt

# kill a process, e.g. 29231/ssh
kill <pid> # eg. kill 29231

sudo kill -9 $(sudo lsof -t -i:8000)

# mb data used
sudo apt install vnstat
vnstat -d

# INTERNET SPEED (need python)
curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

# (1st test on mac)
# Look for which ports are running
# Find their PID
# Kill them
brew install nmap
nmap localhost
lsof -i:8080
kill <PID>

# in case above command cannot solve the problem
sudo lsof -i -P | grep "8080"

Text file

# QUICK LOOK CONTENT
more file.txt
cat file.txt

# JUST CREATE
touch file.txt

# CREATE + MODIFY
nano file.txt # Ctrl+X to quit
vim file.txt # ESC, :q to quit

# SEARCH STRING
grep "string" file.txt

# ADD A LINE TO A FILE WITHOUT OPENNING IT
echo "hello 'thi' world" >> my_file.txt

Images

# open an image
eog image_file.jpg

ln -s original_folder sym_folder

# remove
rm sym_folder

Alias

Create your own “alias” command for short,

# CREATE
alias yourAlias='cd /usr/'
alias yourAlias=cd /usr/ # windows

# CALL
yourAlias

# LIST OF ALIASES
alias
alias abc # "abs" stands for what?

# remove an alias
unalias abc

# group of commands
my_alias() {
    screen -S dat -dm bash -c "cd /dinhanhthi.com; iserve; exec sh"
}

# list of commands
my_alias(){
    cd /home/user/git/abc/
    git add .
    git commit -m "abc"
    git push
}
  • Linux / MacOS: Add your alias to .bash_aliases (in home dir, printenv HOME) if you wanna store your alias permanently.
  • Windows: Using cmder (its setting file), add more aliases to <cmder-install>/config/user_aliases.cmd. You can also add (automatically) on the cmder UI, it adds them for you to the .cmd file.

Create / Copy / Cut / Paste

# Create a new folder
mkdir <folder>
mkdir -p <folder> # already exist accepted

# MOVE
mv <old-dir> <new-dir>
move <old-dir> <new-dir> # windows

# RENAME a file/folder
mv olname.txt newname.txt

# COPY
cp file file
cp -r file&dir file&dir

Display

# only display 3 last directory names
PROMPT_DIRTRIM=3

# display only user:current_folder#
PS1='\u:\W\$ '

References

ssh

ssh how it works!

How it works?

  1. Local creates public_key (id_rsa.pub) & private_key (id_rsa).
  2. Only private_key can understand public_key.
  3. Remote sends messages encrypted based on public_key.
  4. Local has to use private_key to understand (decrypt) remote’s messages.

Generate a public key

  • Windows: Using below command, if it asks for a location, indicate C:\Users\dinha\.ssh\

  • Linux: /home/thi/.ssh/

     ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
 # without email
 ssh-keygen -t rsa -f ~/.ssh/id_rsa.home

Multiple ssh keys

  1. Create key with different names, e.g. id_rsa.home, id_rsa.work.
  2. Add to ~/.ssh/config,
  Host home
  Hostname home.example.com
  IdentityFile ~/.ssh/id_rsa.home
  User <your home acct>
  #
  Host work
  Hostname work.example.com
  IdentityFile ~/.ssh/id_rsa.work
  User <your work acct>

  1. Add to ssh-agent (don’t need to retype password again)

    eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa.home
ssh-add ~/.ssh/id_rsa.work

  2. Don’t forget to clone you repo with git instead of https.

Add public key to remote

Suppose that we wanna connect to a remote host username@remote.com from a local machine.

  1. On local machine, copy public key at C:/Users/hsyyulim/.ssh (Windows) and ~/.ssh (Linux) (something like id_rsa.pub) (copy its content).
  2. On remote server (Linux), go to ~/.ssh, open file authorized_keys by vim authorized_keys
    1. Be carefull, you can modify the current keys!
    2. Go to the end of this file (by W)
    3. Press I to enter to the editing mode, press Enter for a new line.
    4. Using mouse to copy/paste the key in the 1st step (on your local machine).
    5. Note that, each key stays in a separated line.
    6. ESC and then type :wq to quick and save.
    7. Try to connect again!
  3. using ssh-copy-id -i ~/.ssh/id_rsa.pub user@hostname -p [port] you need to provide login password for the first time trying out.

    Connecting

ssh remote_username@remote_host
ssh remote_username@remote_host -p remote_port

# CHECK VERSION
ssh -V

# DISCONNECT
exit

# COPY FILE: LOCAL -> REMOTE
scp local_file user@remote-host:/var/tmp/

# multiple files, using wildcat "\*"

# REMOTE -> LOCAL
scp user@remote:/usr/local/bin/add.sh .

# multiple files, using wildcat "\*"

# Pass inside the command
sudo apt-get install sshpass
sshpass -p your_password ssh user@hostname

# copy with sudo on remote
# 1. copy to a place you have permissions
scp * thi@remote:/home/thi/tmp/
# 2. move to the place you want
ssh thi@remote sudo mv /home/thi/tmp/\* /place/we/want

Command line parameters

# FOR EXAMPLE
ssh -C # use data compression

Usage: Access jupyter notebooks from remote server on local machine.

Below are some popular commandslink:

# check the full list
man ssh

# exit background running
sudo apt install net-tools
netstat -lepunt

# kill a process, e.g. 29231/ssh
kill <pid> # eg. kill 29231



- `-C`: use data compression.
- `-f`: Requests ssh to go to background just before command execution
- `-L`: local port forwarding[link](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding).
- `-N`: Do not execute a remote command.  This is useful for just forwarding ports
- `-p <port>`: port to connect.
- `-q`: quiet mode.
- `-v`: verbose mode.
- `-X`: running GUI remote app locally.

Errors

- REMOTE HOST IDENTIFICATION HAS CHANGED
@@ Offending ECDSA key in /home/thi/.ssh/known_hosts:21

+ SOLUTION:
! Open /home/thi/.ssh/known_host and delete line 21

The following wiki, pages and posts are tagged with

TitleTypeExcerpt
download to install and customize post Wed, Oct 20, 21, like windows reinstall, like macos install, a to z steps
bash command line summary post Thu, Oct 21, 21, cmder bash commandline tools hotkeys and how to create sh file to automate actions
references for using and customizing Linux post Sat, Oct 23, 21, run macos apps, android apps, shell cli, screenshot, drive and app process
connecting raspi on ubuntun20.04 without a monitor post Tue, Jan 25, 22, change network/ssh and user configuration to use rasp without a monitor