Apache BookKeeper 4.14 releases: low latency storage service

by · Published · Updated

Apache BookKeeper is a scalable, fault tolerant and low latency storage service optimized for append-only workloads.

It is suitable for being used in following scenarios:

  • WAL (Write-Ahead-Logging), e.g. HDFS NameNode.
  • Message Store, e.g. Apache Pulsar.
  • Offset/Cursor Store, e.g. Apache Pulsar.
  • Object/Blob Store, e.g. storing state machine snapshots.

Changelog v4.14

adds FIPS compliance, improves compaction logic and the Stream Storage, improves data reliability in the recovery scenarios, fixes multiple bugs and brings critical dependencies up-to-date.

Highlights

Bookkeeper is FIPS compliant by default now

  • Make Bookkeeper FIPS compliant by defaultFIPS is ‘Federal Information Processing Standard’. It’s a set of guidelines for security functions such as encryption/decryption/RNG etc. Applications running in FIPS mode are said to be more secure as they adhere to more stringent standards.

Data reliability

  • Add fencing to recovery reads to avoid data loss issue

Table Service (stream storage) reliability improvements

  • Improved handling of RocksDB tombstones
  • Checksum validation for SST files
  • Better handling of corrupted checkpoints
  • Adjusted default rocksDbBlockCache size to 10%/numberOfLedgers of direct memory
  • RocksDB log path is configurable now

Compaction logic improvements

  • forceAllowCompaction to run only when force is set or configured interval
  • Allow a customer to set a limit on the duration of the major and minor compaction runs
  • Fix: The compaction status report is off by 1
  • Allow force compact entry log when entry log compaction is disabled
  • Allow DBLedgerStorage to force GC by disk listener

Dependency updates

  • SECURITY: Upgraded Netty to 4.1.63.Final
  • SECURITY: Removed jackson-mapper-asl dependency to resolve multiple CVEs
  • Upgraded Lombok to 1.18.20 (required for Java 16 support)
  • Upgraded rocksdb to 6.16.4

Other improvements and fixes

  • Fix: always select the same region set bug for RegionAwareEnsemblePlacementPolicy
  • Allow to attach labels to metrics
  • Allow to bypass journal for writes
  • Imposed a memory limit on the bookie journal
  • Bookkeeper client throttling logic is based upon entryId instead of ledgerId
  • Performance: unnecessary copy to heap from CompositeByteBuf
  • Ensure that only entries of the current ensemble are included in the ledger recovery process
  • Auto-throttle read operations
  • Limit read-ahead bytes to the size of the read cache
  • Fixed NetworkTopologyImpl#getLeaves returning set with null value in case of non existing scope

Other

Documentation, build, CI, tests improvements

Download

Tags: