背景
为了更多的满足TOB场景下的需求，360虚拟化团队一直不断丰富完善openstack侧的功能，近期对接过信创存储腾凌存储等商业存储，所以梳理一下整个流程，下面进入正文从架构以及源码了解openstack如何支持SAN存储。
一、相关概念
提到 IP SAN 必然会想到磁盘阵列，磁盘阵列有三种架构分别为：DAS,NAS,SAN。而SAN里面主要又分为IP SAN和FC SAN。
FC-SAN（Fibre Channel Storage Area Network）是一种基于光纤通道技术的存储网络，它将存储设备和服务器连接在一起，形成一个高速、高性能的存储区域网络。FC-SAN的核心是光纤通道交换机，它实现了光纤通道协议，使得存储设备和服务器之间的数据传输更加可靠和高效。
IP-SAN（Internet Protocol Storage Area Network）是一种基于IP协议的存储网络，它将存储设备、连接设备和接口集成在高速网络中。IP-SAN使用IP网络将存储设备连接在一起，实现数据的可靠传输和共享。由于IP网络具有广泛的普及性和互操作性，IP-SAN具有较好的扩展性和灵活性。
DAS（Direct Attached Storage）是一种直接附加存储技术，它将存储设备通过电缆直接连接到服务器上。DAS的优点是简单、成本低，适用于小型网络和单机环境。但是，DAS的缺点也很明显，如存储容量受限、扩展性差、数据共享困难等。
NAS（Network Attached Storage）是一种网络附加存储技术，它将存储设备连接到现有的网络上，提供数据和文件服务。NAS实际上是一个专门优化了的文件服务器，具有独立的操作系统和文件系统。NAS的优点是易于部署和管理，可以实现数据的集中存储和共享。但是，NAS的缺点是性能受限于网络带宽和稳定性，不适合大规模数据存储和高性能计算。
二、OpenStack实现
2.1 cinder侧实现
我们知道在openstack中cinder负责存储volume 的生命周期管理，而cinder中cinder-volume负责转发控制面请求从而对存储执行 action，对于每一种存储介质，cinder-volume需要调用对应的driver才可以，这里以最近对接过的信创腾凌ip san 存储为例讲解实现。
我们首先需要配置一个cinder 的volume backend tldriver
ini
[tldriver]volume_backend_name = tldrivervolume_driver = cinder.volume.drivers.tengling.tengling_driver.TenglingISCSIDrivertengling_sanip = {{ tengling_sanip }}tengling_username = {{ tengling_username }}tengling_password = {{ tengling_password }}tengling_storagepool = {{ tengling_storagepool }}tengling_max_clone_depth = {{ tengling_max_clone_depth }}tengling_flatten_volume_from_snapshot=True我们以创建一个volume从源码刨析整个的流程，前面将新建的volume type指定capability 为backend tldriver，这样scheduler就能调度到我们的新存储上了。cinder scheduler 通过rpc 请求volume 调用 volumeManager 的create_volume函数
python
@objects.Volume.set_workers def create_volume(self, context, volume, request_spec=None, filter_properties=None, allow_reschedule=True): ..........try: # NOTE(flaper87): Driver initialization is # verified by the task itself. flow_engine = create_volume.get_flow( context_elevated, self, self.db, self.driver, self.scheduler_rpcapi, self.host, volume, allow_reschedule, context, request_spec, filter_properties, image_volume_cache=self.image_volume_cache, ) except Exception: msg = _("Create manager volume flow failed.") LOG.exception(msg, resource={'type': 'volume', 'id': volume.id}) raise exception.CinderException(msg)cinder volume创建volume时通过task flow执行了核心任务 CreateVolumeFromSpecTask，这里用户创建了一个系统盘，指定了image，所以执行了_create_from_image ，最终调用了_create_from_image_cache_or_download 方法
python
class CreateVolumeFromSpecTask(flow_utils.CinderTask): .......... def execute(self, context, volume, volume_spec): .......... elif create_type == 'image': model_update = self._create_from_image(context, volume, **volume_spec) .......... def _create_from_image(self, context, volume, image_location, image_id, image_meta, image_service, **kwargs): .......... if not cloned: model_update = self._create_from_image_cache_or_download( context, volume, image_location, image_id, image_meta, image_service) def _create_from_image_cache_or_download(self, context, volume, image_location, image_id, image_meta, image_service, update_cache=False): .......... try: if not cloned: try: with image_utils.TemporaryImages.fetch( image_service, context, image_id, backend_name) as tmp_image: if CONF.verify_glance_signatures != 'disabled': # Verify image signature via reading content from # temp image, and store the verification flag if # required. verified = \ image_utils.verify_glance_image_signature( context, image_service, image_id, tmp_image) self.db.volume_glance_metadata_bulk_create( context, volume.id, {'signature_verified': verified}) # Try to create the volume as the minimal size, # then we can extend once the image has been # downloaded. data = image_utils.qemu_img_info(tmp_image)
virtual_size = image_utils.check_virtual_size( data.virtual_size, volume.size, image_id)
if should_create_cache_entry: if virtual_size and virtual_size != original_size: volume.size = virtual_size volume.save() model_update = self._create_from_image_download( context, volume, image_location, image_meta, image_service ) finally: # If we created the volume as the minimal size, extend it back to # what was originally requested. If an exception has occurred or # extending it back failed, we still need to put this back before # letting it be raised further up the stack. if volume.size != original_size: try: self.driver.extend_volume(volume, original_size) finally: volume.size = original_size volume.save() ..........在 _create_from_image_cache_or_download 中会将镜像下载到本地临时文件，再通过 qemu 获取info信息，最终调用了 _create_from_image_download。在 _create_from_image_download中调用的本backend的driver执行create_volume 操作，并调用 copy_image_to_volume将镜像数据写入到volume
python
def _create_from_image_download(self, context, volume, image_location, image_meta, image_service): .......... model_update = self.driver.create_volume(volume) or {} self._cleanup_cg_in_volume(volume) model_update['status'] = 'downloading' try: volume.update(model_update) volume.save() except exception.CinderException: LOG.exception("Failed updating volume %(volume_id)s with " "%(updates)s", {'volume_id': volume.id, 'updates': model_update}) try: volume_utils.copy_image_to_volume(self.driver, context, volume, image_meta, image_location, image_service) except exception.ImageTooBig: with excutils.save_and_reraise_exception(): LOG.exception("Failed to copy image to volume " "%(volume_id)s due to insufficient space", {'volume_id': volume.id}) return model_update def copy_image_to_volume(driver, context, volume, image_meta, image_location, image_service): .......... try: image_encryption_key = image_meta.get('cinder_encryption_key_id')
if volume.encryption_key_id and image_encryption_key: # If the image provided an encryption key, we have # already cloned it to the volume's key in # _get_encryption_key_id, so we can do a direct copy. driver.copy_image_to_volume( context, volume, image_service, image_id) elif volume.encryption_key_id: # Creating an encrypted volume from a normal, unencrypted, # image. driver.copy_image_to_encrypted_volume( context, volume, image_service, image_id) else: driver.copy_image_to_volume( context, volume, image_service, image_id)这里调用了腾凌存储的 cinder.volume.drivers.tengling.tengling_driver.TenglingISCSIDriver.create_volume 创建云盘，并调用腾凌driver执行了copy_image_to_volume。这里TenglingISCSIDriver继承自了driver.ISCSIDriver ，所以调用了原生ISCSIDriver的
python
def create_volume(self, volume): """Create a volume.""" volume_type = self._get_volume_type(volume) opts = self._get_volume_params(volume_type) if (opts.get('hypermetro') == 'true' and opts.get('replication_enabled') == 'true'): err_msg = _("Hypermetro and Replication can not be " "used in the same volume_type.") LOG.error(err_msg) raise exception.VolumeBackendAPIException(data=err_msg)
lun_params, lun_info, model_update = ( self._create_base_type_volume(opts, volume, volume_type))
model_update = self._add_extend_type_to_volume(opts, lun_params, lun_info, model_update) return model_update原生ISCSIDriver 会通过os_brick模块远程iscsi挂载磁盘到本地
python
class ISCSIDriver(VolumeDriver): ......... def copy_image_to_volume(self, context, volume, image_service, image_id): """Fetch image from image_service and write to unencrypted volume.
This does not attach an encryptor layer when connecting to the volume. """ self._copy_image_data_to_volume( context, volume, image_service, image_id, encrypted=False) ......... def _copy_image_data_to_volume(self, context, volume, image_service, image_id, encrypted=False): """Fetch the image from image_service and write it to the volume.""" LOG.debug('copy_image_to_volume %s.', volume['name'])
use_multipath = self.configuration.use_multipath_for_image_xfer enforce_multipath = self.configuration.enforce_multipath_for_image_xfer properties = utils.brick_get_connector_properties(use_multipath, enforce_multipath) attach_info, volume = self._attach_volume(context, volume, properties) # 这里会挂载远程disk try: if encrypted: encryption = self.db.volume_encryption_metadata_get(context, volume.id) utils.brick_attach_volume_encryptor(context, attach_info, encryption) try: image_utils.fetch_to_raw( context, image_service, image_id, attach_info['device']['path'], self.configuration.volume_dd_blocksize, size=volume['size']) #这里写入镜像数据 except exception.ImageTooBig: with excutils.save_and_reraise_exception(): LOG.exception("Copying image %(image_id)s " "to volume failed due to " "insufficient available space.", {'image_id': image_id})
finally: if encrypted: utils.brick_detach_volume_encryptor(attach_info, encryption) finally: self._detach_volume(context, attach_info, volume, properties, force=True)fetch_to_volume_format 中会fetch镜像文件到本地临时目录，最后通过执行qemu-img convert 命令将镜像临时文件数据写入到本地iscsi远程磁盘中，至此虚机的系统盘数据写入完成。
python
def fetch_to_volume_format(context, image_service, image_id, dest, volume_format, blocksize, volume_subformat=None, user_id=None, project_id=None, size=None, run_as_root=True): ......... convert_image(tmp, dest, volume_format, out_subformat=volume_subformat, src_format=disk_format, run_as_root=run_as_root)
def _convert_image(prefix, source, dest, out_format, out_subformat=None, src_format=None, run_as_root=True, cipher_spec=None, passphrase_file=None): cmd = _get_qemu_convert_cmd(source, dest, out_format=out_format, src_format=src_format, out_subformat=out_subformat, cache_mode=cache_mode, prefix=prefix, cipher_spec=cipher_spec, passphrase_file=passphrase_file) #拼接 qemu-img convert 命令最后cinder侧更新数据库，至此cinder侧工作完成了，概括性的流程如下:
2.2 nova侧实现
nova侧在给虚机挂载磁盘时，nova-compute收到请求后attach volume的请求后调用nova.compute.manager.ComputeManager.attach_volume
python
def _attach_volume(self, context, instance, bdm): context = context.elevated() LOG.info('Attaching volume %(volume_id)s to %(mountpoint)s', {'volume_id': bdm.volume_id, 'mountpoint': bdm['mount_device']}, instance=instance) compute_utils.notify_about_volume_attach_detach( context, instance, self.host, action=fields.NotificationAction.VOLUME_ATTACH, phase=fields.NotificationPhase.START, volume_id=bdm.volume_id) try: bdm.attach(context, instance, self.volume_api, self.driver, do_driver_attach=True) ................nova.virt.block_device.DriverVolumeBlockDevice.attach中会 调用 _legacy_volume_attach 进行挂载磁盘
python
@update_db def attach(self, context, instance, volume_api, virt_driver, do_driver_attach=False, **kwargs): .................. # Check to see if we need to lock based on the shared_targets value. # Default to False if the volume does not expose that value to maintain # legacy behavior. if volume.get('shared_targets', False): # Lock the attach call using the provided service_uuid. @utils.synchronized(volume['service_uuid']) def _do_locked_attach(*args, **_kwargs): self._do_attach(*args, **_kwargs)
_do_locked_attach(context, instance, volume, volume_api, virt_driver, do_driver_attach) else: # We don't need to (or don't know if we need to) lock. self._do_attach(context, instance, volume, volume_api, virt_driver, do_driver_attach)
def _do_attach(self, context, instance, volume, volume_api, virt_driver, do_driver_attach): """Private method that actually does the attach.
This is separate from the attach() method so the caller can optionally lock this call. """ context = context.elevated() connector = virt_driver.get_volume_connector(instance) if not self['attachment_id']: self._legacy_volume_attach(context, volume, connector, instance, volume_api, virt_driver, do_driver_attach) else: self._volume_attach(context, volume, connector, instance, volume_api, virt_driver, self['attachment_id'], do_driver_attach)legacyvolume_attach 会 调用cinder 的 initialize_connection 获取volume挂载的connection info，这里因为是iscsi 云盘，cinder会返回iSCSI的 connection info。
python
def _legacy_volume_attach(self, context, volume, connector, instance, volume_api, virt_driver, do_driver_attach=False): volume_id = volume['id']
connection_info = volume_api.initialize_connection(context, volume_id, connector) ..................
# If do_driver_attach is False, we will attach a volume to an instance # at boot time. So actual attach is done by instance creation code. if do_driver_attach: encryption = encryptors.get_encryption_metadata( context, volume_api, volume_id, connection_info)
try: virt_driver.attach_volume( context, connection_info, instance, self['mount_device'], disk_bus=self['disk_bus'], device_type=self['device_type'], encryption=encryption) except Exception: with excutils.save_and_reraise_exception(): LOG.exception("Driver failed to attach volume " "%(volume_id)s at %(mountpoint)s", {'volume_id': volume_id, 'mountpoint': self['mount_device']}, instance=instance) volume_api.terminate_connection(context, volume_id, connector) .................. if volume['attach_status'] == "detached": # NOTE(mriedem): save our current state so connection_info is in # the database before the volume status goes to 'in-use' because # after that we can detach and connection_info is required for # detach. self.save() try: volume_api.attach(context, volume_id, instance.uuid, self['mount_device'], mode=mode)virt_driver.attach_volume 会调用libvirt 挂载磁盘 ，self._connect_volume 会依据connection info判断为iSCSI driver 会Calling os-brick to attach iSCSI Volume ，最终libvirt 在线添加了disk到虚机里面，至此虚机侧挂载完成！
python
def attach_volume(self, context, connection_info, instance, mountpoint, disk_bus=None, device_type=None, encryption=None): guest = self._host.get_guest(instance)
disk_dev = mountpoint.rpartition("/")[2] bdm = { 'device_name': disk_dev, 'disk_bus': disk_bus, 'device_type': device_type}
..................
self._connect_volume(context, connection_info, instance, encryption=encryption)
.................. try: state = guest.get_power_state(self._host) live = state in (power_state.RUNNING, power_state.PAUSED)
guest.attach_device(conf, persistent=True, live=live)三、总结
IP SAN 由于基于传统IP网络，所以优势是成本低、易维护且比较灵活，但同时也限制了他只适合应用于对于性能要求并不是太高的场景，性能上比较不如FS SAN以及nvmf等，因此IP SAN 适合于中小型应用场景，所以需要针对不同的使用场景下进行选择不同的存储协议。
本文借以兼容腾凌存储梳理了openstack侧虚机挂载使用商业 IP SAN存储的整个流程，对于理解包括支持其他类型的SAN/NVMF等存储有一定的参考意义。
更多技术干货，
请关注"360智汇云开发者"👇
360智汇云官网：https://zyun.360.cn（复制在浏览器中打开）
更多云产品欢迎试用体验~