MaverickCloudStorageSpec

Differences between revisions 3 and 5 (spanning 2 versions)
⇤ ← Revision 3 as of 2010-05-26 19:05:26 →
Size: 5442
Editor: 76-216-240-245
Comment: Moving old summary to rationale, writing new summary. Moving other databases to notes section. Adding Digg.com link for Thrift
← Revision 5 as of 2010-06-15 23:02:21 → ⇥
Size: 5682
Editor: 76-216-240-245
Comment:
Deletions are marked like this. Additions are marked like this.
Line 42: Line 42:
[[http://cassandra.apache.org/|Cassandra]] needs packaging for universe. There are a number of concerning dependencies, including Hadoop and Thrift. [[http://cassandra.apache.org/|Cassandra]] is a very popular, fast moving database engine that needs to be made available to users of Ubuntu. It is not quite ready for Universe inclusion, as its dependencies are in flux, and releases are deprecated at a very rapid rate.
Line 44: Line 44:
==== Hadoop ==== ==== Recommended PPA or, "RPA" ====
Line 46: Line 46:
[[http://hadoop.apache.org|Hadoop]] is in Debian testing and should auto sync soon. It may also be possible to drop this as a dependency as it is not necessary for normal operation, but just for data analysis. The Cassandra project puts out binary packages that are not hard to modify to build under a PPA on Launchpad. These packages with the required changes to build will be uploaded there.
Line 48: Line 48:
==== Thrift ==== ==== Discoverability ====
Line 50: Line 50:
[[http://incubator.apache.org/thrift/|Thrift]] will need to be packaged for universe. Digg.com has already done packaging work for thrift, to be found here: [[http://about.digg.com/opensource/ops]] ===== Upstream Invitation for Involvement =====
Line 52: Line 52:
Users will still need to generate their own language bindings using [[http://wiki.apache.org/cassandra/ThriftExamples|the thrift tools and documentation on Cassandra's website]]. We will encourage the Cassandra Development team to contribute here and upload their packages directly the RPA above as well, to help users stay up to date as Cassandra releases are made. Membership in the restricted cassandra-ubuntu team on Launchpad will allow committing and uploading, so upstreams will be free to update the packages if they are granted membership.

===== Server Team Endorsement =====

We will endorse the PPA's on our blog and mailing list.

===== RPA's List =====

Once a list of "recommended package archives" is available Cassandra's will be listed in it. It should materialize at [[ServerTeam/RPA]].
Line 62: Line 70:
=== Cassandra Thrift Language Bindings ===

Cassandra will still be difficult to use without language specific bindings already generated. It may be useful to build debian packages for these.		 N/A

Summary

This spec details the steps we will take in Maverick to improve support for popular distributed/cloud based storage technologies such as CouchDB, MongoDB, and Cassandra.

Release Note

Maverick includes CouchDB, a distributed data store, in main, and a number of other cloud friendly data storage packages in Universe/Multiverse, including MongoDB, Cassandra, and Drizzle

Rationale

In order to make Ubuntu Server the platform of choice on cloud environments, we need to support the workloads that users typically need on such environments. This include cloud-oriented databases or datastores.

User stories

As a web developer building modern, distributed, highly scalable applications, I want to deploy applications using distributed data storage with a minimum of customization.

As a web developer, I want to be able to try out popular cloud/distributed data storage solutions and remove them cleanly if they do not fit my needs.

As an ops engineer supporting distributed applications in and out of the cloud, I want to deploy critical infrastructure pieces such as data storage from known distributions without having to create custom builds of complicated software.

Assumptions

Design

CouchDB

With desktop couch already in main, CouchDB is a logical choice for promotion from universe to main.

MongoDB

MongoDB is already packaged in Debian. Ensure that merge/sync with debian has the most up to date version possible for Maverick.

Cassandra

Cassandra is a very popular, fast moving database engine that needs to be made available to users of Ubuntu. It is not quite ready for Universe inclusion, as its dependencies are in flux, and releases are deprecated at a very rapid rate.

The Cassandra project puts out binary packages that are not hard to modify to build under a PPA on Launchpad. These packages with the required changes to build will be uploaded there.

Discoverability

Upstream Invitation for Involvement

We will encourage the Cassandra Development team to contribute here and upload their packages directly the RPA above as well, to help users stay up to date as Cassandra releases are made. Membership in the restricted cassandra-ubuntu team on Launchpad will allow committing and uploading, so upstreams will be free to update the packages if they are granted membership.

Server Team Endorsement

We will endorse the PPA's on our blog and mailing list.

RPA's List

Once a list of "recommended package archives" is available Cassandra's will be listed in it. It should materialize at ServerTeam/RPA.

Implementation

See blueprint whiteboard server-maverick-cloud-datastores

Test/Demo Plan

Unresolved issues

N/A

UDS session agenda and discussion

Discussion notes:

Top candidates:

  • Couchdb
  • Mongo
  • Cassandra
  • Drizzle

Overview

  • Document store
    • CouchDB (in main for karmic+)
    • MongoDB (in universe for lucid+)
      • debian uptodate, we are not
  • Eventually‐consistent key‐value store (Dynamo implementation):
    1. Cassandra (not in Ubuntu)

    2. Project_Voldemort (LinkedIn) (not in ubuntu)

  • Tabular
    • Hbase (build on top of hadoop, see server-maverick-hadoop-pig): available in maverick
    • hypertable
  • Key/value store on disk
    • redis (in universe karmic+)
    • tokyo cabinet (in universe hardy+) - 1.4.37 (from debian) vs 1.4.44 (upstream)
      • maybe bug in watchfile
    • tokyotyrant
      • - in debian unstable, up to date 1.1.40 - network enabled tokyocabinet storage - native, memcache, http REST access - async replication
    • memcachedb (in universe jaunty+) - current but 'stable' for a long time
      • there are better way to do things now
      • Oracle is interested in it because of berkley db
  • Key/value store in RAM : see server-m-web20-workloads

  • Other NOSQL databases (from http://en.wikipedia.org/wiki/Nosql)

    • Neo4j (graph db)
    • Keyspace (graph db)
    • ndb -- uptodate - part of mysql cluster
    • RIAK
      • how fast does it move? is it worth packaging?
      • seeing adoption
  • SQL cloud-oriented databases
    • Drizzle
      • in debian, but moving fast
      • should be synced to maverick, keep following
      • candidate for nightly vcs

Actions

  • CouchDB: move couchdb server binary pkg into main => YES

  • MongoDB: merge/sync with Debian => YES

  • Cassandra: package for universe

Notes

Other Databases

The following would be interesting to revisit for later cycles:

  • tokyotyrant
  • RIAK
  • Project_Voldemort

CategorySpec

MaverickCloudStorageSpec (last edited 2010-06-15 23:02:21 by 76-216-240-245)