Glass’ evolution, some suggestions

Short link for PRESENTATION

I am a huge fan of Google Glass’ concept. The idea of having information pop up that only you can see to add context to whatever is in your line of sight is a wildly interesting technology.

I bought a Google Glass about 4 months ago and was totally OK w/ paying $1500. Since purchasing the Glass I have swayed between being euphoric about some features (e.g. POV camera) and disappointed about the product’s many shortcomings.

Recently I started cataloging all the changes I would make to Glass and wrote up a (not perfectly organized) power point presentation. More recently Google announced and released Android wear, which is very much in line with some of my suggestions, so I felt validated that my ideas were relevant and thought I should share them.

Hopefully some of my other ideas are in line w/ what Google is thinking, because a technology like a Glass is a certainty to be in your household, but not in its current form.

Without further ado … CLICK FOR PRESENTATION

 

* On re-reading my presentation, it is pretty poorly organized. The interesting ideas start around page 20 and the even better ones start at page 30, which is to say the first 20 are boring 🙂

AlchemyDB is all grown up

I started the AlchemyDB project in early 2010. After two years, the project had graduated to become a hybrid Document-store/GraphDB/RDBMS/redis-store. AlchemyDB was then acquired by Aerospike in early 2012. The next 2 years were spent integrating AlchemyDB’s codebase/architecture/philosophy into Aerospike, eventually yielding Aerospike 3. Four years years after it’s inception, AlchemyDB has provided the right ammunition to catapult Aerospike into the visionary enterprise NOSQL quadrant.

Integrating AlchemyDB into Aerospike was, of course, much harder than expected. The bulk of this integration work was accomplished not by merging code, but by redoing AlchemyDB’s innovative features in Aerospike’s architecture. What was interesting is that the merging of the two platforms didn’t just result in features being dropped (e.g. GraphDB support), it also took an unchartered course and some new features were born of the process (e.g. LargeDataTypes).

All in all, I am genuinely pleased with the architecture that emerged from the integration. Aerospike 3 is a document-store, w/ robust secondary indexing, a high performance User-Defined-Function (UDF) implementation, ultra-low-latency aggregations (e.g. <1ms possible), SSD-optimized LargeDataTypes, and support for a subset of SQL. What’s more, ALL of these new features have the same good-ole Aerospike guarantees of performance, elasticity, fault-tolerance, etc… Complexity was added w/o significant compromise on the original product’s strengths.

The rest of this post will dive into the details of Aerospike 3’s LargeDataType feature. Aerospike’s LargeDataTypes are standard data-types (e.g. Stack, List, Map) but each instance is stored as many different physical pages tied-together logically via a directory. LargeDataTypes deal w/ the inherent flaw in the document model that over time documents themselves tend towards becoming BigData: the hotter the document, the larger it becomes, the more the DB bottlenecks on I/O & de/serialization.

Aerospike’s LargeDataTypes greatly benefit from piggybacking on Aerospike’s SSD optimized data-layer. For example: the LargeStack data-type has near optimal I/O when the use-case is predominantly pushing and popping a small number of items. This pattern of usage results in only the directory and the top page being accessed: a near optimal I/O which is independent of the size of the LargeStack, as the cold-parts of the LargeStack simply sit idle on SSD.

Diving even deeper: Aerospike’s LargeDataTypes are native database types at the API level, but under the covers they are user-space server-side code making calls into Aerospike’s linked-pages API (still private as of Mar 2014). The linked-pages API enables UDFs to access raw data pages on the server. It also insures that these raw data pages are logically linked & bound to the data-structure’s primary key record such that they are guaranteed to always be co-located on the same server as the primary-key’s record. This guarantee covers before, during, & after cluster re-orgs (both elastic and fault tolerant).

Implementing LargeDataTypes in server-side UDF space has one priceless advantage: extensibility. One customer benefited tremendously from byte-packing & compressing data in their LargeStacks (which held the bulk of their data). This extensibility was easily accomplished by cloning the LargeStack code (we named the clone: CompressedLargeStack) and then inserting a few lines of compression algorithms into the clone’s code. The customer’s client code switched to using CompressedLargeStacks and WHAMMY: 10X space reduction.

This customer story is an example of how such extensibility allows for the creation of new LargeDataTypes in UDF space. Aerospike users can now roll their own customized LargeDataTypes, as Aerospike’s engineers did for this customer’s use-case and before that for the LargeDataType implementations themselves. The linked-pages API that guarantees co-location of many pages bound to a single primary key, can be used to create a wide variety of different LargeDataTypes (e.g. Graphs, ScoreBoards, Tries, Sparse-bitmaps) in server-side user-space code. With proper data structure architecting, these user-created LargeDataTypes can grow arbitrarily large and still perform according to strict SSD based SLAs. This makes the linked-pages API a fantastic tool for certain high velocity BigData use-cases. 

The power of being able to define a customized data-type that:

1. is a single logical entity, that is made up of many physical pages
2. has strict transactional guarantees
3. has a guarantee that all of the physical pages are located on a single server even during cluster re-orgs

enables customization down to the very data-type definition & functionality, to better fit a customer’s unique data.

My personal goal with Aerospike 3 was to deliver on NOSQL’s promise of storing & querying data in it’s most natural form. Enabling developers to easily roll their own data-types to better fit their unique data and to tweak those data-types to improve performance is an ace in the hole for those that need to continually push their system that much further.