How did you write data to the database — one record at a time or in batches? Could one Kafka message contain 500 records?

In most cases, to improve performance and reduce load on the database, data is written in batches rather than one record at a time. If 500 records arrive in a single Kafka message, it makes sense to process them as one batch and perform a single transaction or batch insert.

Example in Go using a database and batch insert:

records := getRecordsFromKafkaMessage(msg) // []Record

// Prepare batch
var values []interface{}
query := "INSERT INTO table_name (col1, col2) VALUES "
placeholders := []string{}

for i, r := range records {
    placeholders = append(placeholders, fmt.Sprintf("($%d, $%d)", i*2+1, i*2+2))
    values = append(values, r.Field1, r.Field2)
}
query += strings.Join(placeholders, ",")

// Execute query
_, err := db.Exec(query, values...)
if err != nil {
    log.Fatal(err)
}

This approach reduces the number of database queries and increases the overall system throughput.