Task: what is more efficient in terms of speed — submitting 2 inputs of 512 tokens each to the transformer separately (2x512) or combining them into 1x1024?

In transformers, the computational complexity of attention is proportional to the square of the input sequence length (O(n²)).

If two inputs of 512 tokens each are fed separately (2×512), each attention operation will have a complexity of approximately 512² = 262144, and in total — 2 × 262144 = 524288.

If they are combined into a single input of 1024 tokens (1×1024), the complexity will be 1024² = 1,048,576, which is twice as much.

Therefore, in terms of speed and computational resources, it is more efficient to process two inputs of 512 tokens separately rather than combining them into one long input.

However, it is important to consider the model architecture and the task objectives: combining may be useful for modeling interactions between two sequences, but from a performance perspective, it is more costly.