Рубрики
k8s Linux Performance tests storage

fio (IOPS test)

Original topic can be accessed by https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd


Test IOPS in Linux system.

tar -xzf fio.tgz
yum localinstall *.rpm
cd /data0/etcd
mkdir test-data;fio —rw=write —ioengine=sync —fdatasync=1 —directory=test-data —size=22m —bs=2300 —name=mytest

#!/bin/bash
now=$(date "+%Y%d%m-%H%M");
while true;
do
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/data0/etcd/test-data --size=22m --bs=2300 --name=${now}_mytest >> /data0/etcd/results.log;
sleep 600;
done
Рубрики
Bash Linux

Linux repositories

To install Centos 7 EPEL repository execute:
#yum install -y epel-release

or install it from rpm:
#wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
#yum install -y epel-release-latest-7.noarch.rpm


list repos:
yum repolist

Search package in the repository:
yum search package_name

Get information about the package:
yum info package_name

Install package:
yum install package_name

List all available packages in a specific repository called repo_name (will also disable all other repositories):
yum --disablerepo="*" --enablerepo="repo_name" list available

Find a specific package in a specific repository:
yum --disablerepo="*" --enablerepo="repo_name" list available | grep 'package_name'

List output of packages in a specific repository:
yum --disablerepo="*" --enablerepo="repo_name" list available | less

Рубрики
Bash Linux

find

Find files in current catalog with text «sometext»:
find -type f -exec grep -q sometext {} \; -print

Find all files in all subdirectories and execute script/command:
find . -type d -print0 | xargs -0 -L1 sh -c 'cd "$0" && pwd && ./script.sh unpack'

Find all files in the current folder that are less than 1 kilobyte and delete them:
find . -type 'f' -size -1k -delete

Рубрики
Bash Linux wget

wget

Simple download:
wget http://website.com/file.txt

Download with reconnecting:
wget -c http://website.com/file.txt

Download with 20 retries for errors and reconnections:
wget -t 20 —retry-connrefused http://website.com/file.txt

Download the whole website up to 4 layers in-depth (-l 4):
wget -r -l 4 -p -E -k http://website.com
-r — recursively
-l 4 — 4 layers in depth
-p — download css and images
-E — download php and asp scripts
-k — converts links in files to point to local copies

Download website:
wget -p -k -E -H http://website.com/index.html
-H — checks external links to be accessible.

Рубрики
Bash Linux

process priority in Linux

execute the program with different priority:
nice -n [19 to -20] program
-n 0 — standard priority
-n 19 — lowest priority
-n -20 — highest priority

change already running program priority:
renice -n [19 to -20] -p PID
renice -n [19 to -20] -u user

change IO priority:
ionice -c [1 to 3] -n [19 to -20] command or program
ionice -c 2 -n 7 cat /var/logs/some.log
or with process:
ionice -c2 -n7 -p PROCESS(PID)

-c means class of IO priority:
-c 1: Real-time. The highest IO priority can interrupt the operating system’s normal behavior.
-c 2: Best effort. The default class of priority, nice and renice will do the work with the second class of IO priority.
-c 3: Idle. The process will get IO only when no other process is using the drives.
more info: man-page ionice(1)

Рубрики
k8s Linux screen

screen

Start command in a detached screen with logging to file, example captures pod logs in the background:

screen -dmS screen_name bash -c 'kubectl logs -f pod_name | tee /home/user/log.txt';

Close all detached screens:
screen -ls | grep ‘(Detached)’ | awk ‘sys {screen -S $1 -X quit}’

Рубрики
Bash Linux

Bash

Shortcuts:
Ctrl+A: to the beginning of the line (like Home)
Ctrl+C: cancel the program
Ctrl+E: to the end of the line (like End)
Ctrl+K: Delete a string from a where cursor located now to the end of the string.
Ctrl+Y: paste just deleted text
Ctrl+Z: stop a program (stop PID). can be restored with fg PID or bg PID

Alt+B: move the cursor one word backward
Alt+F: move the cursor one word forward
Alt+D: delete the word, the cursor must be before the word
Alt+T: switch two previous words
Crtl+T: switch two previous strings

####################################################

Vi:
i — insert mode
i i — replace mode
Esc — exit from edit mode

in view mode:
H — move the cursor to the right
L — move the cursor to the left
J — move the cursor one string down
K — move the cursor one string up
X — Delete a symbol
D D — delete a string
P — paste deleted string
U — undo all
: — switch to command mode

####################################################

list file in the directory:
ls
-t : by time
-S : by size
-X : by extension
-r : reverse
-l : more info, like owner etc.
-h : human-readable size
-R : recursively with sub-catalogs
-F : show file type
/ — catalog
* — executable
@ — symlink
— — symbol device
+ — block device
= — Pipe, FIFO

combine ls and less to list though long output:
ls -lR | less

####################################################

While true loops:

This script makes log rotate to execute every 20 minutes and writes status to file.
while true;
do sleep 1200;
logrotate -s /var/log/logstatus /etc/logrotate.conf;
done

jq — makes pretty print for json. («.» after is mandatory for filtering, without «.« the file will not be written)
while true;
do sleep 1;
curl -v -u login:passwd http://ip:port/api/nodes | jq . >> /home/user/curl.20200512.log;
done

To delete all commented lines from file use next script:
./stripcomments.sh file.name
#!/bin/sh
grep -v ^[[:space:]]*\# $1 | grep -v ^[[:space:]]*\; | grep -v ^$

Copy all files in catalog with «.bak» extensions to «.bak~»
for i in *.bak;
do cp $i $i~;
done

Save logs for all pods without sidecars in k8s default namespace to separate files:
now=$(date "+%Y%d%m-%H%M"); for item in $(kubectl get pods | awk '{print $1}'); do kubectl logs ${item} > ${item}.$now.logs; done;

More advanced script for logs from k8s:

#!/bin/bash now=$(date "+%Y%d%m-%H%M");
mkdir "${now}_logs";
for item in $(kubectl get pods | awk '{print $1}')
do if [[ ${item} = NAME ]] || [[ ${item} = *"cpro"* ]] || [[ ${item} = *"snmp-webhook"* ]] || [[ ${item} = *"reloader"* ]] || [[ ${item} = *"coap-adapter"* ]]
then
echo "Pod ${item} has multiply containers or there is some error or it may be filtered!" else
kubectl logs "${item}" > "${now}_logs/${item}.${now}.logs" fi
done
tar -zcvf "logs-${now}.tar.gz" "${now}_logs" echo echo echo "Done!"

Copy SSH identity without password prompt and fingerprint confirmation:
sshpass -f password.txt ssh-copy-id 10.241.45.73 -o StrictHostKeyChecking=no;

Change user password in one string:
echo -e "userpassword\nuserpassword" | passwd username

Create a file in each directory and all subdirectories recursively:
find . -type d -exec touch .gitkeep {} \;

Execute command in each subdirectory:
find . -type d -print0 | xargs -0 -L1 sh -c 'cd "$0" && pwd && ./packer.sh unpack'

xargs flags:
-0 — input is terminated (split) by end of the line and not with white space.
-L1 — max 1 not blank line per argument

Generate ssh key in one string to the default file:
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa <<<y 2>&1 >/dev/null

Turn off swap and comment it in fstab:
swapoff -a
sed -i 's/^\/swap/#\/swap/' /etc/fstab

Рубрики
Docker

Docker

Save an image to a tar.gz file using gzip:
docker save myimage:latest | gzip > myimage_latest.tar.gz

Check mounted volumes for a specific docker container:
docker inspect -f '{{ .Mounts }}' containerid