How did you collect and store data for anti-fraud (where did the data come from, how was it done)?

Anti-fraud data is usually collected from multiple sources: transaction logs, user behavior events, data from external services (such as geolocation, IP addresses, devices), as well as from internal databases with transaction history.

Streaming via Kafka or RabbitMQ is often used to obtain data, allowing real-time event processing. REST APIs may also be used for requests to external systems.

Data storage is implemented in several layers: hot data for quick access — in Redis or in-memory cache, historical data — in relational databases (PostgreSQL) or big data warehouses (ClickHouse, Hadoop). This enables fast analysis and model building for fraud detection.

An example of simplified data retrieval from Kafka in Go:

import (
    "github.com/segmentio/kafka-go"
    "context"
    "log"
)

func consumeAntiFraudData() {
    r := kafka.NewReader(kafka.ReaderConfig{
        Brokers: []string{"localhost:9092"},
        Topic:   "transactions",
        GroupID: "antifraud-service",
    })

    for {
        m, err := r.ReadMessage(context.Background())
        if err != nil {
            log.Println("error reading message:", err)
            continue
        }
        // Processing message for anti-fraud
        processTransaction(m.Value)
    }
}