DSA(Direct Attacted Storage 直接附加存储) 本地磁盘接口:IDE SATA SAS SCSI
NAS(NetWork Attacted Storage 网络附加存储) 网络服务共享: 共享的是文件夹,如 samba NFS
SAN(Storage Attacted Network 存储附加网络) 网络服务共享: 共享的是设备,如iscsi
ISCSI(internet SCSI)是一个协议,通过TCP/IP网络的方式将存储设备共享到客户端,仿真为本地设备,让本地计算机像管理直连存储一样,对共享的设备存储进行分区,格式化,挂载使用等.ISCSI之间通信时一种C/S架构
客户端C:为发起者
服务端S:为目标,包含多个LUN(Logical Unit Number逻辑单元),真正提供服务的是LUN,一个LUN为一个设备,可用同时共享多个设备给客户端
ISCSI原理
·SCSI层根据客户端发出的请求建立SCSI CDB(命令描述块）,并且传给ISCSI层。同时接收来自ISCSI层的CDB,并向应用返回数据。
·ISCSI对SCSICDB进行封装，以便能够在基于TCP/IP协议的网络上进行传输，完成SCSI到TCP/IP的协议映射。这一层是ISCS协议的核心层。
·TCP/IP层对IP报文进行路由和转发，并且提供端到端的透明可靠传输。
SAN/IPSAN
IPSAN是在SAN后产生的，
SAN默认指FCSAN,以光纤通道构建存储网络，
IPSAN则以IP网络构建存储网络，较FCSAN,具有更经济、自由扩展等特点。
# 安装ISCSI服务
yum install -y targetcli
systemctl enable target
systemctl start target
准备工作:准备共享硬盘设备或磁盘文件
[root@centos2 ~]# lsblk # 查看设备
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 10G 0 disk
└─sdb1 8:17 0 5G 0 part
sdc 8:32 0 10G 0 disk
sdd 8:48 0 20G 0 disk
sde 8:64 0 10G 0 disk
sdf 8:80 0 10G 0 disk
sr0 11:0 1 1G 0 rom
# 可用dd创建一个5G的文件
dd if=/dev/zero of=/opt/sdb2 bs=1M count=5000
工作流:将设备添加到backstores存储库--->设置全球唯一标识名IQN ---> 设置TPG组,定义谁可用重哪个IP即端口访问IQN标识对应的哪些设备
[root@centos2 ~]# targetcli # targetcli 进行交互操作
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type ‘help‘.
/> help
/> ls # 如下4类存储可用 块设备,文件,物理磁盘,划分的磁盘
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
/>
/> cd backstores #进入存储库
/backstores> block/ create block1 /dev/sde # 创建一个块设备,名称block1,使用系统上/dev/sde设备
Created block storage object block1 using /dev/sde.
/backstores> ls
o- backstores ............................................................ [...]
o- block ................................................ [Storage Objects: 1]
| o- block1 ...................... [/dev/sde (10.0GiB) write-thru deactivated] #刚创建的设备
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- fileio ............................................... [Storage Objects: 0]
o- pscsi ................................................ [Storage Objects: 0]
o- ramdisk .............................................. [Storage Objects: 0]
/backstores>
/backstores> block/ delete block1 #删除设备
Deleted storage object block1.
/backstores>
/backstores> cd block #进入块设备
/backstores/block> create block002 /dev/sdb1 # 创建块设备block002
Created block storage object block002 using /dev/sdb1
/backstores/block>
/backstores/block> create block001 /dev/sde # 重新创建设备block001
Created block storage object block001 using /dev/sde.
/backstores/block>
/> cd backstores/fileio
/backstores/fileio> create block003 /opt/sdb2 # 进入文件设备库,并创建文件设备
Created fileio block003 with size 5242880000
/backstores/fileio> ls # 查看创建的设备
o- fileio ................................................. [Storage Objects: 1]
o- block003 ...................... [/opt/sdb2 (4.9GiB) write-back deactivated]
o- alua ................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ....................... [ALUA state: Active/optimized]
/backstores/fileio> cd ..
/backstores> ls
o- backstores ............................................................ [...]
o- block ................................................ [Storage Objects: 2]
| o- block001 .................... [/dev/sde (10.0GiB) write-thru deactivated]
| | o- alua ................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
| o- block002 .................... [/dev/sdb1 (5.0GiB) write-thru deactivated]
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- fileio ............................................... [Storage Objects: 1]
| o- block003 .................... [/opt/sdb2 (4.9GiB) write-back deactivated]
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- pscsi ................................................ [Storage Objects: 0]
o- ramdisk .............................................. [Storage Objects: 0]
/backstores>
# IQN格式: iqn.年-月.二级域名倒写:共享名
/backstores> cd /iscsi
/iscsi> create iqn.2021-05.com.iscsitest:storage # 设置iqn
Created target iqn.2021-05.com.iscsitest:storage.
Created TPG 1. # 自动设置TPG组为 1
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260. #默认监听的IP和端口号
/iscsi> ls
o- iscsi .......................................................... [Targets: 1]
o- iqn.2021-05.com.iscsitest:storage ............................... [TPGs: 1]
o- tpg1 ............................................. [no-gen-acls, no-auth] # TPG1组
o- acls ........................................................ [ACLs: 0] # 谁可访问
o- luns ........................................................ [LUNs: 0] # 可访问哪些设备
o- portals .................................................. [Portals: 1]
o- 0.0.0.0:3260 ................................................... [OK] # 或者配置服务端自己的IP
/iscsi>
/iscsi> cd iqn.2021-05.com.iscsitest:storage/tpg1/acls/ # 进入访问权限设置
/iscsi/iqn.20.../tpg1/portals> delete 192.168.198.129 3260
Deleted network portal 192.168.198.132:3260
/iscsi/iqn.20.../tpg1/portals> create 192.168.198.129 3260
# 设置用户名 iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> create iqn.2021-05.com.iscsitest:cliente001
Created Node ACL for iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> ../luns/ create /backstores/block/block001 # 将block001设备共享出来
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> cd ../luns/
/iscsi/iqn.20...age/tpg1/luns> create /backstores/block/block002 # 将block002设备共享出来
Created LUN 1.
Created LUN 1->1 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/luns> create /backstores/fileio/block003 # 将block003文件设备共享出来
Created LUN 2.
Created LUN 2->2 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/luns> cd ..
/iscsi/iqn.20...:storage/tpg1> ls
o- tpg1 ................................................. [no-gen-acls, no-auth]
o- acls ............................................................ [ACLs: 1]
| o- iqn.2021-05.com.iscsitest:cliente001 ................... [Mapped LUNs: 3]
| o- mapped_lun0 ................................ [lun0 block/block001 (rw)]
| o- mapped_lun1 ................................ [lun1 block/block002 (rw)]
| o- mapped_lun2 ............................... [lun2 fileio/block003 (rw)]
o- luns ............................................................ [LUNs: 3]
| o- lun0 ..................... [block/block001 (/dev/sde) (default_tg_pt_gp)]
| o- lun1 .................... [block/block002 (/dev/sdb1) (default_tg_pt_gp)]
| o- lun2 ................... [fileio/block003 (/opt/sdb2) (default_tg_pt_gp)]
o- portals ...................................................... [Portals: 1]
o- 0.0.0.0:3260 ....................................................... [OK]
/iscsi/iqn.20...:storage/tpg1>
/iscsi/iqn.20...:storage/tpg1> portals/ delete 0.0.0.0 3260 # 也可以对访问的IP和端口进行删除和创建
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20...:storage/tpg1> portals/ create 0.0.0.0 3260
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3260.
/iscsi/iqn.20...:storage/tpg1> exit # 保存退出
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json
[root@centos2 ~]#
[root@centos2 ~]# vi /etc/target/saveconfig.json # 刚才设置保存的文件
[root@centos2 ~]# systemctl restart target # 设置完后重启服务
工作流程:安装客户端 ---> 设置客户端iqn设备名称 ---> 安装客户端服务并启动 ---> 访问共享 ---> 分区,格式化,自动挂载
1. 安装客户端
[root@centos ~]# yum install -y iscsi-initiator-utils
2. 设置iscsi名称
[root@centos ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2021-05.com.iscsitest:cliente001
3. 启动iscsi客户端服务
[root@centos ~]# systemctl start iscsi
[root@centos ~]# systemctl enable iscsi
[root@centos ~]# man iscsiadm
#iscsid 为iscsi服务的守护进程,会随iscsi自动起来,配置文件/etc/iscsi/iscsid.conf
4. 发现共享设备
[root@centos ~]# iscsiadm --mode discoverydb --type sendtargets --portal 192.168.198.129 --discover
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
[root@centos ~]# ll /var/lib/iscsi/nodes #查看
total 0
drw------- 3 root root 36 May 23 02:36 iqn.2021-05.com.iscsitest:storage
5. 连接共享设备
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] successful.
[root@centos ~]# lsblk # 查看挂载成功
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk # sdc/sdd/sde为新添加
sdd 8:48 0 5G 0 disk
sde 8:64 0 4.9G 0 disk
sr0 11:0 1 1G 0 rom
6. 进行分区格式化挂载
[root@centos ~]# fdisk /dev/sdc
[root@centos ~]# vi /etc/fstab
/dev/sdc1 /mnt/block1 ext4 _netdev 0 0 # _netdev 网络文件,不要用默认
[root@centos ~]# mkfs.ext4 /dev/sdc1
[root@centos ~]# mount -a
[root@centos ~]#
[root@centos ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 899M 0 899M 0% /dev
tmpfs 910M 0 910M 0% /dev/shm
tmpfs 910M 9.5M 901M 2% /run
tmpfs 910M 0 910M 0% /sys/fs/cgroup
/dev/mapper/centos-root 17G 4.7G 13G 28% /
/dev/sda1 1014M 181M 834M 18% /boot
tmpfs 182M 0 182M 0% /run/user/0
192.168.198.130:/test 17G 4.7G 13G 28% /mnt/nfs
/dev/sdc1 9.7G 37M 9.2G 1% /mnt/block1 # 查看已经挂载成功
客户端停止ISCSI 服务
# 查看服务名
[root@centos ~]# iscsiadm --mode node
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
# 客户端停止使用ISCSI共享
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129:3260 --logout
# 彻底删除,如果login登录不了,需要删除设备,重新发现并登录
[root@centos ~]# rm -rf /var/lib/iscsi/nodes/iqn.2021-05.com.iscsitest:storage/
[root@centos ~]# rm -rf /var/lib/iscsi/send_targets/192.168.198.129,3260/
# 以上相关命令可用简写
解决单点故障
服务端和客户端都配备两块网卡进行连接
添加网卡,配置IP地址
# 服务端,添加新网卡
[root@centos2 ~]# ip a
192.168.198.129/24
192.168.198.132/24
# 客户端也添加新网卡
[root@centos ~]# ip a
192.168.198.133/24
192.168.198.130/24
# nmcli配置IP地址
nmcli con add con-name eth1 ifname ens36 type 802-3-ethernet ipv4.method manual ipv4.add 192.168.198.136/24
# 检查以上网卡是否相互ping通
[root@centos2 ~]# ping 192.168.198.133
[root@centos2 ~]# ping 192.168.198.130
[root@centos ~]#192.168.198.132
[root@centos ~]#192.168.198.129
参考:ISCSI服务端部署,配置成如下配置
[root@centos2 ~]# targetcli
/> ls
o- / ...................................................................... [...]
o- backstores ........................................................... [...]
| o- block ............................................... [Storage Objects: 1]
| | o- block001 ..................... [/dev/sdc (10.0GiB) write-thru activated]
| | o- alua ................................................ [ALUA Groups: 1]
| | o- default_tg_pt_gp .................... [ALUA state: Active/optimized]
| o- fileio .............................................. [Storage Objects: 0]
| o- pscsi ............................................... [Storage Objects: 0]
| o- ramdisk ............................................. [Storage Objects: 0]
o- iscsi ......................................................... [Targets: 1]
| o- iqn.2021-05.com.iscsitest:storage1 ............................. [TPGs: 1]
| o- tpg1 ............................................ [no-gen-acls, no-auth]
| o- acls ....................................................... [ACLs: 1]
| | o- iqn.2021-05.com.iscsitest:clientemultipath ........ [Mapped LUNs: 1]
| | o- mapped_lun0 ........................... [lun0 block/block001 (rw)]
| o- luns ....................................................... [LUNs: 1]
| | o- lun0 ................ [block/block001 (/dev/sdc) (default_tg_pt_gp)] # LUN 把设备共享出来
| o- portals ................................................. [Portals: 2]
| o- 192.168.198.129:3260 .......................................... [OK] # 配置服务端的IP进行共享
| o- 192.168.198.132:3260 .......................................... [OK]
o- loopback ...................................................... [Targets: 0]
工作流程: 安装客户端并启动 ---> 连接共享设备从两条线路 ---> 分区格式化 ---> 安装多路径软件 ---> 实现多路径负载均衡
# 客户端发现两个设备,并进行挂载
[root@centos ~]# iscsiadm --mode discoverydb --type sendtargets --portal 192.168.198.129 --discover
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
192.168.198.132:3260,1 iqn.2021-05.com.iscsitest:storage
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] successful.
[root@centos ~]#
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.132 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.132,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.132,3260] successful.
(多路径不能格式化为xfs格式)
[root@centos ~]# fdisk /dev/sdc
[root@centos ~]# mkfs.ext4 /dev/sdc1
[root@centos ~]# partprobe #更新分区标信息
[root@centos ~]#
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─sdc1 8:33 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─sdd1 8:49 0 10G 0 part
sr0 11:0 1 1G 0 rom
# 安装多路径软件
yum install -y device-mapper-multipath
# 拷贝配置文件
cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/
# 启动服务
systemctl enable multipathd; systemctl start multipathd
# 查看
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─mpatha 253:2 0 10G 0 mpath
└─mpatha1 253:3 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─mpatha 253:2 0 10G 0 mpath
└─mpatha1 253:3 0 10G 0 part
sr0 11:0 1 1G 0 rom
[root@centos ~]# multipath -ll # 多路径block001设备id 360014051f73df963d734b45aa3f450e9
mpatha (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features=‘0‘ hwhandler=‘0‘ wp=rw
|-+- policy=‘service-time 0‘ prio=1 status=active # 默认是提供AB模式,可修改配置文件,变成负载均衡模式
| `- 21:0:0:0 sdc 8:32 active ready running
`-+- policy=‘service-time 0‘ prio=1 status=enabled
`- 22:0:0:0 sdd 8:48 active ready running
[root@centos ~]# vi /etc/multipath.conf
#defaults {
# user_friendly_names yes
# find_multipaths yes
#}
multipaths {
multipath {
wwid 360014051f73df963d734b45aa3f450e9 # 设备编号
alias multipathtest # 别名
path_grouping_policy multibus # 多路径组策略
path_selector "round-robin 0" # 负载均衡模式
failback manual
rr_weight priorities # 按优先级轮询
no_path_retry 5 # 重试时间
}
multipath {
wwid 1DEC_____321816758474
alias red
}
}
[root@centos ~]# systemctl start multipathd
[root@centos ~]# systemctl start iscsi
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features=‘1 queue_if_no_path‘ hwhandler=‘0‘ wp=rw
`-+- policy=‘round-robin 0‘ prio=1 status=active
|- 21:0:0:0 sdc 8:32 active ready running # 再次查看,多路径已经变成rr负载均衡模式
`- 22:0:0:0 sdd 8:48 active ready running
# multipath -F #删除现有路径,新的路径就会被删除
# multipath -v2 #格式化路径,格式化后又出现
[root@centos ~]# ll /dev/mapper
total 0
lrwxrwxrwx 1 root root 7 May 23 16:24 centos-root -> ../dm-0
lrwxrwxrwx 1 root root 7 May 23 16:24 centos-swap -> ../dm-1
crw------- 1 root root 10, 236 May 23 14:28 control
lrwxrwxrwx 1 root root 7 May 23 16:56 multipathtest -> ../dm-2
lrwxrwxrwx 1 root root 7 May 23 16:56 multipathtest1 -> ../dm-3
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─multipathtest 253:2 0 10G 0 mpath # 这里是刚刚配置文件中设置的别名,也可以不设置别名
└─multipathtest1 253:3 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─multipathtest 253:2 0 10G 0 mpath
└─multipathtest1 253:3 0 10G 0 part
sr0 11:0 1 1G 0 rom
# 挂载使用(也可以umount后重新格式化)
[root@centos ~]# mount /dev/mapper/multipathtest1 /mnt/multi/
#在客户端可进行读写操作
[root@centos ~]# touch /mnt/multi/file1
# 将服务端网卡down掉
[root@centos2 ~]# ifdown ens37
# 在客户端查看,掉线一条链路,另一条正常运行
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features=‘1 queue_if_no_path‘ hwhandler=‘0‘ wp=rw
`-+- policy=‘round-robin 0‘ prio=1 status=active
|- 21:0:0:0 sdc 8:32 active faulty running
`- 22:0:0:0 sdd 8:48 active ready running
# 将服务端网卡启动
[root@centos2 ~]# ifup ens37
# 在客户端查看,两条链路都正常运行
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features=‘1 queue_if_no_path‘ hwhandler=‘0‘ wp=rw
`-+- policy=‘round-robin 0‘ prio=1 status=active
|- 21:0:0:0 sdc 8:32 active ready running
`- 22:0:0:0 sdd 8:48 active ready running
原文：https://www.cnblogs.com/lixianjin/p/14802685.html