AI Benchmark Engineer | Native Language Specialist - Arabic (Saudi Arabia) - Remote

Role Overview

AI Benchmark Engineer | Native Language Specialist - Arabic (Saudi Arabia) at Lilt Production. Remote, freelance position.

Company Overview

LILT's mission is to make the world's information available to everyone, no matter the language they speak. The company delivers multilingual AI and human-verified services to Enterprises, Governments, and AI Developers around the world, with a global community of linguists, subject matter experts, and language professionals.

Role Purpose

Design, build, and validate a rigorous evaluation suite of Terminal-Bench tasks that test large language models on multilingual software challenges. Create high-signal, high-quality benchmarks that measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows, without relying on English translation.

Key Responsibilities

Task Engineering

• Evaluate coding agents through task design and implementation.

Asset Creation

• Build realistic task environments using datasets and files in your native language.
• Ensure all assets remain in the target language to genuinely measure multilingual handling.

Prompting & Failure Analysis

• Identify failure points where AI does not work in your native language.
• Create prompts that expose multilingual weaknesses.

Implementation & Verification

• Support development of robust reference implementations.
• Write highly reliable, deterministic verifier scripts using rubric-based judging only when strictly necessary.

Calibration & Execution

• Analyze execution logs and calibrate task difficulty from Easy to Very Hard.
• Use standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus).

Quality Assurance

• Participate in a rigorous 4-layer human quality control process: creation, human review, calibration review, and audit.
• Work alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity.

Qualifications & Experience

• 5+ years of industry experience in software engineering.
• Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
• Native or near-native fluency in Arabic with deep understanding of grammar, register, and phrasing rules.
• High English proficiency.
• Extensive experience with Terminal/CLI-based development workflows.
• Working familiarity with coding agents.
• Deep technical understanding of multilingual text processing pitfalls, including encoding/decoding robustness, Unicode normalization, locale-dependent conventions (collation, casing, non-Gregorian dates), text I/O, toolchain interoperability, safe string operations, bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts.

Skills & Competencies

• Strong proficiency in Python.
• Strong proficiency in standard shell scripting.
• Data processing expertise.

Additional Information

• Contract Type: Freelance, 1099 contractor, remote.
• Schedule Flexibility: Work when you want, as much or as little as you want. No fixed hours or micromanaging.
• Availability Requirements: Reliable availability and commitment required. Most tasks require a minimum of 2 hours per day or 15-20 hours per week. Once you accept a task, quality work and on-time delivery are expected.
• Compensation: Competitive rates with prompt payment.
• Not ideal as a full-time job or primary income source. Work availability fluctuates with project demand.
• No benefits provided, including health insurance, paid time off, or retirement contributions. Hours are not guaranteed.
• Geographic restrictions may apply. Contractors in regions subject to international embargo or sanctions cannot be engaged.
• As a 1099 contractor, you are solely responsible for your own tax obligations.
• Application Process: Submit application with CV in English, complete a GenAI assessment, and finalize onboarding to become eligible for Applied AI projects.