What is sharding?

Sharding is the process of splitting large datasets into smaller, manageable parts called shards. Each shard is an independent database or table containing a portion of the data. This horizontal scaling allows distributing load and storage across multiple servers or database instances, enhancing system performance, availability, and capacity.

Key aspects:

• Sharding Key: An attribute of the data (e.g., user ID, geolocation) used to determine which shard to place a specific record in. Choosing the right key is critical for even data distribution.
• Sharding Strategies: Different methods of determining how data is distributed across shards:
  • Range Sharding: Data is distributed based on ranges of the sharding key values.
  • Hash Sharding: The sharding key is hashed, and the result determines the shard.
  • Directory-Based Sharding: A lookup table is used to map the sharding key to a shard.
• Routing: The mechanism to determine which shard contains the data needed to fulfill a query.
• Rebalancing: The process of redistributing data among shards, for example, when adding or removing shards.

Advantages:

• Scalability: Handles growing data volumes and loads.
• Performance: Reduces load on individual servers, speeding up query execution.
• Availability: Failure of one shard does not necessarily bring down the entire system.
• Manageability: Simplifies management of large datasets.

Disadvantages:

• Complexity: Implementation and management of a sharded system are more complex.
• Cross-shard queries: Queries requiring data from multiple shards can be slower and more complex.
• Schema changes: Modifying the data schema may require coordination across all shards.

Example of simple routing implementation in Golang using hashing:

package main

import (
	"fmt"
	"hash/fnv"
)

// determineShard determines the shard for a key
func determineShard(key string, numShards int) uint32 {
	h := fnv.New32a() // using FNV hash function
	h.Write([]byte(key))
	return h.Sum32() % uint32(numShards) // modulo operation to get shard number
}

func main() {
	numShards := 4
	keys := []string{"user1", "user2", "user3", "user4", "user5", "user6"}

	for _, key := range keys {
		shard := determineShard(key, numShards)
		fmt.Printf("Key '%s' goes to shard %d\n", key, shard)
	}
}