Jay Taylor's notes

back to listing index

sharding - Google Groups

[web search]
Original source (groups.google.com)
Tags: groups.google.com
Clipped on: 2013-02-01

sharding
7 posts by 2 authors in Squeryl
Image (Asset 1/1) alt=

Here's a scheme that would be more strongly typed :

trait ShardingFunction[K] {
  def selectShard(k: K): Session
}

class ShardedTable[A,K](table: Table[A], invokeShardedFieldGetter: A => AnyRef, shardedFieldMetaData: FieldMetaData, f: ShardingFunction[K]) {

  def shardedLookup(k: K)(implicit dsl: QueryDsl) = {

    import dsl._

    using(f.selectShard(k)) {

      val q = from(table)(a => dsl.where {
        FieldReferenceLinker.createEqualityExpressionWithLastAccessedFieldReferenceAndConstant(invokeShardedFieldGetter(a), k)
      } select(a))

      q
    }
  }
}

class ShardedTablePrecursor[A](table: Table[A]) {


   def shardedOn[K,T](f:  A => T)(implicit dsl: QueryDsl, ev: T => TypedExpressionNode[K], sf: ShardingFunction[K]) = {

     import dsl._
     
     val q = from(table)(a=> select(a)) : Query[A]

     val n = 
       Utils.mapSampleObject(q, (a0:A) => ev(f(a0)))
     
     val invoker = f.asInstanceOf[A=>AnyRef]
     
     new ShardedTable[A,K](table, invoker, n._fieldMetaData, sf)
   }
}


object YourSchema extends Schema {

  implicit object yourHashFunction extends ShardingFunction[Long] {
    def selectShard(k: Long): Session = sys.error("implemente me !")
  }

Then you define a sharded table : 

  val shardedTable = table[YourObject].shardedOn(_.theShardingKey)
  
  shardedTable.shardedLookup(k) // here it is enforced at compile time that k is the same type as _.theShardingKey ...

Now the insert and updates are left as an exercise ! ;-)

It probably would be wise to disable the partial update function, i.e. only update with def update(a: A),
or maybe provide a partial update that takes a shard key in a first argument list

Cheers !

2012/3/14 Artie Pesh-Imam <ar...@cx.com>
- show quoted text -



--
Seuls les poissons morts nagent avec le courant
Max
Join group to reply
3/14/12

The ShardedTable class could also have this method :

  def shardedTransaction[U](shardKey: K)(block: Table[A] => U)(implicit dsl: QueryDsl) = 
    dsl.transaction(f.selectShard(shardKey)) {
      block(table)
    }


shardedTable.shardedTransaction(aValueToShardOn) { table => // the 'raw' table is available here

   // squeryl code here will run against the chosen shard,

   table.insert(...)  // inserts, updates and deletes will magically be ran against the correct shard...
}


2012/3/14 Maxime Lévesque <maxime....@gmail.com>
- show quoted text -
- show quoted text -
3/15/12
Artie Pesh-Imam Max, Thanks for the suggestion! I do like it. Quick (possibly dumb) question… when defining the table, you're calling sharedOn which is defined in ShardedTablePrecursor. Is there an implicit conversion going on that Im missing. Im not sure how that m
Max
Join group to reply
3/15/12


Yes... there needs to be this implicit conversion in the the Schema trait :

  protected implicit def tableToSharded[A](t: Table[A]) = new ShardedTablePrecursor(t)


2012/3/15 Artie Pesh-Imam <ar...@cx.com>
- show quoted text -
- show quoted text -
3/19/12
Artie Pesh-Imam Max, Thanks I've got this working. A few things: 
Max
Join group to reply
3/19/12

The more self contained and clean, the more I'll want to pull it ;-)
I will change to an sbt multiproject setup pretty soon, sharding could 
go in it's own sub project. Contact me privately, we can discuss it.

BTW, did you read this :  http://www.codefutures.com/database-sharding/ 
?