AI Benchmark Engineer | Native Language Specialist - Arabic (UAE) - Remote

Role Overview

AI Benchmark Engineer | Native Language Specialist - Arabic (UAE) with Lilt Production. Remote, freelance position.

Company Overview

LILT's mission is to make the world's information available to everyone, no matter the language they speak. The company delivers multilingual AI and human-verified services to Enterprises, Governments, and AI Developers around the world.

Role Purpose

Design, build, and validate a rigorous, verifiable evaluation suite of Terminal-Bench tasks that test the limits of large language models on multilingual software challenges. Measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows without relying on English translation crutches.

Key Responsibilities

Task Engineering

• Evaluate coding agents and design high-signal, high-quality benchmark tasks.

Asset Creation

• Build realistic task environments using datasets and files in your native language.
• Ensure all assets remain in the target language to genuinely measure multilingual handling.

Prompting & Validation

• Identify failure points where AI does not work in your native language.

Implementation & Verification

• Support the development of robust reference implementations.
• Write highly reliable, deterministic verifier scripts using rubric-based judging only when strictly necessary.

Calibration & Execution

• Analyze execution logs and calibrate task difficulty from Easy to Very Hard.
• Run tasks against various model tiers (Haiku, Sonnet, Opus) using standard Terminal-Bench configurations.

Quality Assurance

• Participate in a rigorous 4-layer human quality control process: creation, human review, calibration review, and audit.
• Ensure fairness, grammatical accuracy, and benchmark integrity alongside automated LLM-based checks.

Qualifications & Experience

• 5+ years of industry experience in software engineering.
• Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
• Native or near-native fluency in Arabic with deep understanding of its grammar, register, and phrasing rules.
• High English proficiency.

Skills & Competencies

• Strong proficiency in Python and standard shell scripting.
• Data processing expertise.
• Extensive experience with Terminal/CLI-based development workflows.
• Working familiarity with coding agents.
• Deep technical understanding of multilingual text processing pitfalls, including:
• Encoding/decoding robustness and Unicode normalization.
• Locale-dependent conventions such as collation, casing, and non-Gregorian dates.
• Text I/O, toolchain interoperability, and safe string operations.
• Bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts.

Additional Information

• Contract Type: Remote freelance position (1099 contractor).
• Schedule: Flexible. Work when you want, as much or as little as you want. No fixed hours, no check-ins, no micromanaging.
• Commitment: Once you accept a task, quality work and on-time delivery are expected. Most tasks require a minimum of 2 hours per day or 15-20 hours per week.
• Not ideal as a full-time job or primary income source. Work availability fluctuates with project demand.
• No benefits provided such as health insurance, paid time off, or retirement contributions. Hours are not guaranteed.
• As a 1099 contractor, you are solely responsible for your own tax obligations.
• Geographic restrictions apply. Cannot engage contractors in regions subject to international embargo or sanctions.
• Application Process: Submit your CV in English, complete a GenAI assessment, and finalize onboarding and profile setup in the system to become eligible for Applied AI projects.