bluefs _flush_range allocated: osd用空间但是显示ceph_bluefs_db_used_bytes is 100%

1、查看日志

bash 复制代码

tail -f  /var/log/ceph/ceph-osd.2.log

cpp 复制代码

   -6> 2024-05-23 09:29:20.734328 7f837d847d00  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1716452960734313, "job": 1, "event": "recovery_started", "log_files": [1064066, 1064069]}
    -5> 2024-05-23 09:29:20.734340 7f837d847d00  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:482] Recovering log #1064066 mode 0
    -4> 2024-05-23 09:29:52.503818 7f837d847d00  5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:815] [default] [WriteLevel0TableForRecovery] Level-0 table #1064067: started
    -3> 2024-05-23 09:30:08.239889 7f837d847d00  1 bluefs _allocate failed to allocate 0x47d00000 on bdev 1, free 0x1b300000; fallback to bdev 2
    -2> 2024-05-23 09:30:08.239910 7f837d847d00 -1 bluefs _allocate failed to allocate 0x47d00000 on bdev 2, dne
    -1> 2024-05-23 09:30:08.239912 7f837d847d00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x47c54820
     0> 2024-05-23 09:30:08.246076 7f837d847d00 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f837d847d00 time 2024-05-23 09:30:08.239921
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")

 ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f837e2f6850]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x7f837e27501b]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x7f837e275588]
 4: (BlueRocksWritableFile::Flush()+0x3c) [0x7f837e29113c]
 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x7f837e662a6f]
 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x7f837e662c83]
 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int)+0x1004) [0x7f837e679d54]
 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x90c) [0x7f837e5c621c]
 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x1430) [0x7f837e5c7e90]
 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x7e6) [0x7f837e5c92e6]
 11: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xed3) [0x7f837e5ca5a3]
 12: (rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**)+0x186) [0x7f837e5cb7e6]
 13: (RocksDBStore::do_open(std::ostream&, bool)+0x8e8) [0x7f837e212368]
 14: (BlueStore::_open_db(bool)+0xdb3) [0x7f837e19d313]
 15: (BlueStore::_mount(bool)+0x40e) [0x7f837e1cf06e]
 16: (OSD::init()+0x3bd) [0x7f837dd8434d]
 17: (main()+0x2d07) [0x7f837dc89947]
 18: (__libc_start_main()+0xf5) [0x7f837a0c6b35]
 19: (()+0x4b9003) [0x7f837dd28003]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this

使用ceph-bluestore-tool

ceph-bluestore-tool 是一个对 BlueStore 实例执行低级管理操作的实用工具。

使用前，关闭对应 OSD 服务

bash 复制代码

[root@node-1 ceph-objectstore-tool-test]# systemctl stop ceph-osd@0

2. 示例

元数据、数据校验（--deep true，开启数据校验）

这实际是调用bluestore中的fsck()函数，有deep、repair等选项，支持不同程度的检查工作。具体工作有：更新kvdb中super前缀信息（ondisk_format、min_alloc_size等，具体可查阅BlueStore::_upgrade_super()函数）、重放deferred_transaction、删除失效blob、删除错误pextent、更新bluefs文件系统信息等。

所以fsck主要针对bluestore中的元数据信息的修复和检查。

bash 复制代码

ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ fsck [--deep true]


[root@localhost build]# ./bin/ceph-bluestore-tool --log-level 10 --log-file /root/bluestore-tool.log --path /var/lib/ceph/osd/ceph-2/ fsck --deep true
fsck success

修复

调用bluestore中的fsck()函数实现功能。

bash 复制代码

ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ [repair|quick-fix] [--deep true]

[root@localhost build]# ./bin/ceph-bluestore-tool --log-level 10 --log-file /root/bluestore-tool.log --path /var/lib/ceph/osd/ceph-2/ repair --deep true
repair success