Machine Learning / AI

Would you be willing to try an interview with an AI?

What is the minimum and comfortable income level for you in rubles?

Do you have any questions about the cooperation terms?

What is your higher education degree and field of study?

Why are you considering a new offer now?

How would you visualize outliers in thematic assessments to understand how much they influence the overall picture?

How exactly does BatchNorm use the saved statistics in `eval()` mode and why is this important for result stability?

The day before comparing two versions of an LLM, it was found that the experiment code exists in the branch, but the exact data version and LoRA parameters are not fixed. What would you choose as the first increment for the next two days, and what would you consciously sacrifice?

Practical task LLM testing in PyTorch gives different results on two runs. Which solution is more mature? - Enable eval() and compare results again. - Increase the size of the validation set. - Repeat the run and choose the best result. - Switch to TensorFlow without analyzing the reason. - Fix the seed and consider the problem solved. - Check eval(), no_grad(), seed, DataLoader, and deterministic CUDA operations.

What grade do you evaluate yourself at?

Provide an example of how you identified and resolved a conflict of examples in a real project.

Tell us how you feel about the interview format: what you liked and what you would like to improve.

Are you willing to consider part-time projects and job sharing?

Practical task AI generated code for releasing a new LLM. Review it: modify so that the editor can compare the quality of versions by thematic segments, see the score spread, and notice the risk of deterioration on a small data group. import matplotlib.pyplot as plt import pandas as pd scores = pd.read_csv('model_scores.csv') summary = scores.groupby('model_version')['score'].mean() plt.bar(summary.index, summary.values) plt.title('Model Quality') plt.ylabel('Average Score') plt.show() print('Best version:', summary.idxmax()) --- import matplotlib.pyplot as plt import pandas as pd scores = pd.read_csv('model_scores.csv') stats = ( scores.groupby(["topic", "model_version"])["score"] .agg(["mean", "std", "count"]) .reset_index() ) print(stats) for topic, group in scores.groupby("topic"): group.boxplot(column="score", by="model_version") plt.title(f"Topic: {topic}") plt.suptitle("") plt.ylabel("Score") plt.show() small_groups = stats[stats["count"] < 30] print("Small groups - unreliable results:") print(small_groups)

Practical task AI generated code for releasing a new LLM. Conduct a review: modify it so that the editor can compare the quality of versions by thematic segments, see the spread of ratings, and notice the risk of deterioration on a small data group. import matplotlib.pyplot as plt import pandas as pd scores = pd.read_csv('model_scores.csv') summary = scores.groupby('model_version')['score'].mean() plt.bar(summary.index, summary.values) plt.title('Model Quality') plt.ylabel('Average Score') plt.show() print('Best version:', summary.idxmax())

How do you ensure the origin of experimental results so that it is possible to accurately understand where the metrics came from?

Are there any current interviews at big tech companies, such as Yandex or Alfa?

What are your preferences or restrictions regarding projects and subject areas?

How do you usually manage versions of data and models in such experiments?

Practical task The average score of the new LLM has increased, but editors complain about financial responses. Which chart would best support the release decision? A single major average score indicator. Columns with average scores of all topics. Box plots by topics and model versions. Table of average scores by topics. Word cloud from new responses. Distributions of scores by versions and topics with each group's size.