Thinking & Writing

0%

AEROSPIKE notes

Posted on Edited on
Symbols count in article: 3.9k Reading time ≈ 4 mins.

AEROSPIKE is a really fast NoSQL database.

And we will adapt it and do DB change in our product. Here are some learning notes.

Using 20-byte digest stored as primary key in database

By default, Aerospike does not store the actual primary key in the database. It stores the 20-byte digest (hash of the key) by default. This will be a huge saving for the large keys.

The hash function that Aerospike used is RIPEMD-160, that is known to be secure against collision based attacks. Here is some research on whether RIPEMD-160 is collision free. If you see messages on key collisions which in Aerospike log, please report this. And, you may be able to write a research paper based on the two keys that caused the collision!

If the user key needs to persist on the server, use one of the following methods:

  • Explicitly store and retrieve the user key in an application defined bin.
  • Ask the database to explicitly store key and detect collisions (e.g., in Java, set WritePolicy.sendKey to true).

Record Deletion

Aerospike separates the data into two parts: index and value. The index is always stored in DRAM, the value can be stored in either SSD/HDD or DRAM (with or without disk for persistence). Delete operations only do an in-memory index deletion for the records. The records on the disk are asynchronously removed by a separate defragmentation process. With defragmentation, the database reclaims storage that is no longer needed. In Aerospike there is no need to explicitly mark records for deletion, thereby reducing writes. This deletion approach is extremely effective in case of flash devices, which may have limited write cycles.

However, with the above performance gains that Aerospike achieves with in-memory index deletion, there is a small chance that the deleted data may not actually be deleted. Consider a scenario where a delete is issued. Before the asynchronous process deletes the record from the disk, the node itself is rebooted. If there is a cold start reboot, the deleted data will be re-indexed and appeared again.

There are some best practices to avoid this scenario. Starting with release Enterprise Edition 2.1.2, with a Fast Restart, Aerospike does not rebuild the whole index from disk on reboot. This will make sure that the deleted data will not get restored to the index. Additionally, instead of a delete, we recommend that you set a TTL for all the records that go into the database. Records whose TTL have expired will never be re-indexed after a reboot. Setting a low TTL value will ensure the expected deletion behavior.

Fast Restart (aka Warm Restart), makes use of Linux system shared memory to enable nodes to start up and re-join their cluster much faster. Database index on a server node, along with various other critical data, is allocated in system shared memory which persists even when the server process is stopped. On restarting, the process re-attaches to the persisted memory containing the index and other data, quickly rebuilds some internal state and re-joins its cluster. It does not need to read all the record data from storage drives to rebuild the index. In this way, a node with over 1 billion records will restart in about 10 seconds (as opposed to 40+ minutes without this feature). This allows cluster upgrades and various other operations to go much faster.

  • Not using fast-restart feature and cold-start-empty for namespace is set to false (this is the default value). In this case when the node starts, it will read the data from disk and rebuild the index. Because the data has not been removed from disk, the node will think it is still active and build a new index entry for it. So the deleted object will return.
  • Those who doesn’t want to pay money for this, please watch on this github project. tiancaiamao give his implementation for this.
  • ‘cold-start-empty’ configuration which prevents data to be loaded from disk, but only by migration from other nodes in the cluster. However, this is a static configure item and cannot be dynamic changed using AMC (Aerospike-management-Console). Which means if you need this configuration in working, you need add the line in configuration file always. But if we include this in the cluster startup, no record will be got from persistent devices.
  • A better way is making a monitor process with 2 copy of configuration files, one without ‘cold-start-empty’ configuration and the other one include it. During the very beginning startup, use the one without ‘cold-start-empty’ and we can load data from persistent devices. After that, the monitor process can monitor Aerospike instances. Use the configuration file with ‘cold-start-empty’ when Aerospike instance got down and need to restart.