Hazelcast introduces IMDG 3.10 with data structures, flake ID generator and CRDT, enhanced split-brain protection

Hazelcast, an open source in-memory data management company announced on Tuesday Hazelcast IMDG 3.10 which is generally available as a production ready build. Major themes, which have been driven by open source community input, include the release of a conflict-free replicated data type (CRDT), a Flake ID Generator and several split-brain protection enhancements.

IMDG 3.10 is the first version of Hazelcast IMDG to include a CRDT implementation – CRDT PN-Counter (positive-negative counter). CRDT is a specially-designed data structure used to achieve strong eventual consistency. With this new data structure, updates are local on a Hazelcast member.

The CRDT PN-Counter allows counting (increment/decrement/query) operations and guarantees that, once all members are connected, the value on all counters is the same. Since operations are local, the CRDT PN-Counter provides consistent low latency and high throughput enabling applications to achieve high performance regardless of load (such as counting likes, page views or connected user count).  Additional CRDT data structures will follow in future releases.

Headquartered in Silicon Valley’s Palo Alto, Hazelcast addresses the growing demand for enhanced application performance, data processing speed and scalability. It has two products: Hazelcast IMDG, an operational, in-memory computing platform which enables enterprises to manage their data and distribute processing using in-memory storage and parallel execution to achieve breakthrough application speed and scale; and Hazelcast Jet, the ultra fast, application embeddable, stream and batch processing engine.

IMDG 3.10 also introduces the Hazelcast Flake ID Generator, a tool used to generate cluster-wide unique identifiers. In single-node systems, unique ID generation is a relatively straightforward process. However, problems can occur in distributed implementations, for example preserving uniqueness, eliminating contention, and maintaining safety during failures.

The Hazelcast Flake ID Generator replaces the existing ID Generator. It generates IDs without coordination between Hazelcast members, meaning it can generate unique IDs even during split-brain and network partitioning failures.

In IMDG 3.9 split-brain protection was available on IMap, ICache, IQueue and ILock. This has now been extended to other structures including ISet, IList, ISemaphore, ICountDownLatch, IReplicatedMap, IAtomicLong, IExecutorService, IDurableExecutorService, IScheduledExecutorServicem, IMultiMap, IRingBuffer, CardinalityEstimator (HyperLogLog), IAtomicReference and CRDT PN-Counter.

Generally, split-brain protection is more flexible in IMDG 3.10. Users can now apply different levels of severity to split-brain protection. Previously protection acted on information regarding the number of members currently observed.

To improve this and provide better protection, split-brain protection now acts on additional information such as member heartbeats, membership changes, lost ICMP pings as well as successful pings following a lost ping. IMDG 3.10 includes numerous split-brain enhancements which provide better protection to structures during split-brain, consequently minimizing inconsistent reads or writes.

For enterprise customers, IMDG 3.10 includes improved split-brain merging for HD structures and optimizations for merging large data sets, reducing the time HD structures take to merge after the cluster has reconciled. IMDG 3.10 also includes ICMP Failure Detector, a detector which works on ISO OSI Layer 3 and allows for the faster detection of dead members. Finally, IMDG 3.10 is the first IMDG on the market to be compliant with JCache 1.1.

In response to community feedback about split-brain protection and overall split-brain functionality, Hazelcast has introduced several new split-brain enhancements in its latest release. Previously only Map, ICache and ReplicatedMap supported merging after a network partition had occurred. In IMDG 3.10, split-brain merging policies are available for a host of new structures: ISet, IList, IQueue, Ringbuffer, MultiMap, IAtomicLong, IAtomicReference, IScheduledExecutorService and CardinalityEstimator. In addition to providing out-of-the box policies and split-brain merging functionality for these new structures, split-brain merge configuration and SPI has been enhanced to ensure configuration and implementation is straightforward.

“A major reason why we are adopted by open source projects and developers is because we listen to community feedback and act upon it. This new release is a testament to the strength of our community following and the quality advice we receive from it,” said Greg Luck, CEO of Hazelcast. “This release brings the fastest split-brain detection, protection against mutation across almost all data structures during split brain and comprehensive healing across almost all data structures. To cap all that off, we have added some conflict-free data structures such as Flake IDs for ID generation, and CRDT Counters, as a conflict free alternative to AtomicLong.”


IoT Innovator Newsletter

Get the latest updates and industry news in your inbox! Enter your email address and name below to be the first to know.