Job Title: AI Engineer / Generative AI Engineer
Location: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Extension Possible)
Experience: 15+ Years
Job Summary
We are seeking a hands-on AI Engineer with expertise in Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation to build, deploy, monitor, and continuously improve production-grade AI agents.
Key Responsibilities
• Develop AI agents using Python and Claude Agent SDK.
• Deploy agents using AWS Bedrock AgentCore Runtime.
• Design golden datasets and implement LLM-as-a-Judge evaluations.
• Integrate automated evaluations into CI/CD pipelines as quality gates.
• Implement Langfuse/OpenTelemetry for AI observability and tracing.
• Evaluate agent accuracy, task completion, tool execution, and performance.
• Support RAG pipelines and AWS data processing workflows.
Required Skills
• Python and Agentic AI development.
• AWS Bedrock AgentCore Evaluations / Bedrock evaluation.
• Golden-dataset design and curation.
• LLM-as-a-Judge design and calibration against human labels.
• Automated evaluation and CI gate integration.
• Langfuse and/or OpenTelemetry.
Good to Have
• RAGAS or equivalent evaluation frameworks.
• Trajectory evaluation and agentic red-teaming.
• Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.
Qualifications
• 6–8+ years of software engineering, cloud engineering, or AI/ML experience.
• Hands-on experience with production AI agents and continuous evaluation.
• Strong problem-solving and troubleshooting skills.
All your information will be kept confidential according to EEO guidelines.