bash

for((i=1;i<=10;i+=2)); do echo "Welcome $i times"; done

vim

# 只留下包含pattern的行
:v/pattern/d

# 移除包含pattern的行
:g/pattern/d

Probe point 'bch_journal' not found. Error: Failed to add events.

[root@kevin ~]# perf probe bch_journal
Probe point 'bch_journal' not found.
  Error: Failed to add events.

[root@kevin ~]# perf probe -m /lib/modules/3.10.0-514.16.1.el7.4.x86_64/kernel/drivers/md/bcache/bcache.ko -a bch_journal

Added new event:

  probe:bch_journal    (on bch_journal in bcache)

You can now use it in all perf tools, such as:

        perf record -e probe:bch_journal -aR sleep 1

ref: http://oliveryang.net/2016/07/linux-perf-tools-tips/#311-perf-cannot-find-external-module-symbols

ftrace / trace-cmd usage, perf

trace-cmd record -p function -l generic_make_request --func-stack
trace-cmd report |less

trace-cmd record -p function -l submit_bio

trace-cmd record -p function_graph -g submit_bio

perf probe -a 

bcache

Q:

1. 大block_size writethrough?
2. fio IOPS with writeback going -> 40k ~ 50k, initialized cache 50k, used cache 40k
3. after trigger_gc, cache_available don't get larger. --> pin: 0, mark: 1(dirty)

code flow:

• writeback:
  
  • request_write -> bch_writeback_add -> bch_writeback_queue -> refill_dirty -> read_dirty -> read_dirty_submit -> write_dirty -> write_dirty_finish
• write_dirty_finish:
  
  • bch_btree_insert
    
    • bch_btree_insert_recurse
      
      • __bkey_put(b->c, insert); // set pin to 0

pr_debug

enable pr_debug in bcache

1. mount -t debugfs /mnt/debugfs/ if debugfs is not mounted in /sys/kernel/debug/
2. echo 'file btree.c line 2349 +p' > /mnt/debugfs/dynamic_debug/control #show pr_debug in btree.c line 2349
3. echo 'module bcache +p' > /mnt/debugfs/dynamic_debug/control #show all pr_debug in bcache
4. "grep bch_btree_refill_keybuf /var/log/debug" or "dmesg|grep bch_btree_refill_keybuf"

screen to tmux

.tmux.conf:
unbind C-b
set -g prefix C-g
bind C-g last-window

set-option -g allow-rename off

-- hotkey of tmux --
ctrl-g l: last-windows
tmux attach: attach to existing session

RHEL 7.2 - dracut unable to assemble(boot) from a degraded raid1 array

問題：

Reproduce:
1. 組一個/dev/md0 is a RAID1 made of /dev/sda /dev/sdb,
mdid：5068eccc:e5b86ea8:be8fe496:61ee1fd8
blkid：21df6a4c-bce3-4fe4-8739-8e804f4893dd

2. 把/dev/dm0放在kernel boot parameter，
/boot/grub2/grub.conf ：linux16 kernel .. rd.md.uuid=5068eccc:e5b86ea8:be8fe496:61ee1fd8

3. shutdown, unplug /dev/sdb
4. power on, cat /proc/mdstat --> /dev/md0 is inactive. If you configure /dev/md0 as /boot, or / partition in /etc/fstab, you will enter "user intervention" mode, because /dev/md0 can't be mounted.
5. you need to run "mdadm --run /dev/md0" manually to assemble /dev/md0 and reboot successfully.

Conclusion:
Entering "user intervention" mode is correct behavior when we have a degraded raid.

Root cause:
dracut don't assemble the degraded raid unless you configure "hostonly_cmdline" in /usr/lib/dracut/dracut.conf.d/01-dist.conf and re-generate initramfs by executing "dracut -f".
1. when you configure "hostonly_cmdline=yes", we can find 90mdraid.conf under the re-generated $initramfs/etc/cmdline.d
2. $dracut/modules.d/99base/dracut-lib.sh
 135     for _i in /etc/cmdline.d/*.conf; do
 136         [ -e "$_i" ] || continue
 137         while read -r _line || [ -n "$_line" ]; do
 138             CMDLINE_ETC_D="$CMDLINE_ETC_D $_line";
 139         done <"$_i";
 140     done

hostonly_cmdline的意思是，把kernel parameter存到initramfs/etc/cmdline.d/底下，這樣dracut的機制會去retry在initramfs/etc/cmdline.d/* 裡面的device。可以打開/boot/grub2/grub.conf "rd.shell rd.debug"去追dracut怎麼跑

easystack IT

https://wiki.easystack.cn/index.php/ESCore
kc.chuang@easystack.cn / d..9

fio

== param sync and buffer ==

man fio:

       sync=bool
              Use synchronous I/O for buffered writes.  For the majority of I/O engines, this means using O_SYNC.  Default: false.

       direct=bool

              If true, use non-buffered I/O (usually O_DIRECT).  Default: false.

==

結論: --sync=1必須要搭配--buffer=1使用, 不能搭配--direct=1使用, --sync=1就強制了把io寫到了buffer

解釋: fio "sync=1" 代表 io寫到buffer後, 同時把buffer裡面的資料寫到block device才返回, 通常此選項會搭配"--buffered=1". 也就是說此選項是告訴fio 當寫一筆io到block device時, 要不要同時把buffer的資料寫到block device, 然後才會返回.
在http://172.16.182.40/redmine/issues/2052的測試中 我取出兩個有疑問的fio指令做解釋:
4k随机写， 440K IOPS：
/config/fio --numjobs=8 --iodepth=1024 --runtime=6000 --ioengine=libaio --direct=1 --sync=0 --name task_sdg --rw=randwrite --bs=4k --filename=/dev/md144
4k随机写： 12K IOPS （sync =1 ）
/config/fio --numjobs=8 --iodepth=1024 --runtime=6000 --ioengine=libaio --direct=1 --sync=1 --name task_sdg --rw=randwrite --bs=4k --filename=/dev/md144

第一個fio指令: --direct=1 --sync=0, 意思很明顯io直接寫到/dev/md144就馬上返回
第二個fio指令: --direct=1 --sync=1, 我在猜因為"--sync=1", 造成了"--buffer=1", 所以造成"--direct=1"失效了, 因此這意思就變成io先寫到buffer, 然後再寫到/dev/md144, 然後才返回(等同於--buffer=1 --sync=1). 如果我的猜測是正確的,這個指令"--buffered=1 --sync=1"應該會得到跟"--direct=1 --sync=1"一樣的IOPS 12k. 但是我目前手上沒有fs3000 with SSD, 因此我用210的機器做以下實驗, 驗證我的推測試正確的.
Environment: md1331 raid0 made of 2 SAS disks, drbd3988 normal LV created on md1331.
1. 4k写， 264K IOPS (--direct=1 sync=0)：
/config/fio --numjobs=8 --runtime=30 -ioengine=libaio --direct=1 --sync=0 --name task_sdg --rw=write --bs=4k --filename=/dev/drbd3988

2. 4k写， 140K IOPS (--direct=1 sync=1)：
/config/fio --numjobs=8 --runtime=30 -ioengine=libaio --direct=1 --sync=1 --name task_sdg --rw=write --bs=4k --filename=/dev/drbd3988

3. 4k写， 143K IOPS (--buffer=1 sync=1)：
/config/fio --numjobs=8 --runtime=30 -ioengine=libaio --buffer=1 --sync=1 --name task_sdg --rw=write --bs=4k --filename=/dev/drbd3988

== 測試cache hit IOPS ==

fio --name task --rw=randwrite --iodepth=64 --size=500M --runtime=6000 --time_based --loops=1 --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --ioengine=libaio --group_reporting --filename=/dev/bcache0

== verify ==
--do_verify=1 --verify=md5 --verify_fatal=1 --verify_dump=1 


--verify_backlog=100, do verify every 100 blocks