Text Situated Alignment Specialist (Contract)

Posted yesterday

careerflowDenver (CO)

Social Scientists and Related Workers, All Other Other Scientific and Technical Consulting Services

SENIORITY

Senior

Apply

About the role

Subject Matter Expert (SME) for AI Conversation Evaluation Research Careerflow Human Data Labs partners with AI companies to bring real-world professional expertise into their products.

About the role: We are seeking Subject Matter Experts (SMEs) with expertise in psychology, social work, counseling, qualitative research, HCI, communication, conflict resolution, coaching, or related advisory fields to support AI conversation evaluation research. SMEs will review conversations between participants and AI models, assess whether sufficient context was gathered before recommendations were made, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality.

Job Responsibilities:

Review AI conversations involving real-world interpersonal and everyday decision-making scenarios.
Identify missing context and recommend relevant follow-up questions.
Evaluate whether additional information would have changed the AI's recommendations.
Distinguish between critical ("load-bearing") and non-essential information.
Conduct qualitative reviews and document findings using structured evaluation guidelines.
Collaborate with research teams to ensure consistent, high-quality assessments.
Required Qualification:
Candidates should have at least 3–5 years of direct practice relevant qualifications and professional experience in at least one of the following areas:
Licensed Social WorkMSW or equivalent graduate-level qualification in Social Work.
LCSW, LICSW, LMSW, or equivalent applicable social work license/credential.
Experience with client assessment, case review, interviewing, counseling, or related social-work practice.
Counseling, Therapy, or PsychologyMA/MS with LMFT, LPC, LCPC, or equivalent counseling/therapy license or credential; orPhD/PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology.
Experience with counseling, therapy, psychological assessment, interviewing, case review, or other relevant client-facing work.
Qualitative Research, HCI, or CommunicationMA/PhD or equivalent advanced expertise in Human-Computer Interaction (HCI), Communication, Anthropology, Sociology, Qualitative Research, or a closely related discipline.
Experience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, qualitative coding, or similar research activities.
Relevant backgrounds may include Qualitative Research, HCI Research, UX/User Research, Communication Research, Anthropology, Sociology, or related disciplines.
Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields
Relevant professional qualification, credential, or demonstrated subject-matter expertise.
Significant direct experience in advisory, client-facing, coaching, mediation, student advising/services, organizational consulting, or comparable work.
Experience gathering context, understanding individual or organizational needs, and providing informed recommendations or guidance. Must be based in the USA, Canada or Western Europe

Preferred experience:

Experience eliciting relevant context when clients initially provide incomplete information
Experience working across relationships, work, education, family, planning, wellbeing, and resource constraints, considering biological, psychological, social, spiritual/existential, financial, and practical factors
Qualitative coding or structured case review
Ability to distinguish between context that is load-bearing for the recommendation and that is merely descriptive/non-informative
Experience reviewing interviews, consultations, coaching sessions, or client interactions to identify missed opportunities for clarification or information gathering
Project Details:
Project duration - 3 weeks (short term project)
Project type - Pay-per-task (PPT)AHT - 75 minutes (approx.)Engagement - Full time
Estimated project start date - 1st October (estimated)
Sample70 total professionals35 participants from High-Cost Geographies35 participants from Mid-Cost Geographies
Experience Requirement

Candidates must have at least 3–5 years of direct practice experience in their relevant professional field. Eligible Qualifications Social Work: MSW with LCSW or equivalent social-work license Counseling/Therapy: MA/MS with LMFT, LPC, LCPC, or equivalent counseling license Psychology: PhD/PsyD in clinical, counseling, social, or industrial-organizational psychology Research and Related Fields: MA/PhD in conflict resolution, communication, HCI, anthropology, sociology, or qualitative research Advisory/Client-Facing Work: Relevant professional credential plus at least five years of direct advisory or client-facing experience 3+ years of professional experience Total number of positions: 320 Engagement Length: Full-time or Part time. 40h or 20h per week with 4 hours PST overlap.

Location: United States and Saudi ArabiaCanada (NOT Quebec), United Kingdom, Ireland, Belgium, Netherlands, Luxembourg, Switzerland, Austria, Denmark, Sweden, Norway, Finland, Iceland, Italy, Spain, Portugal, Australia, New Zealand6-week project, with a preferred commitment of up to 40 hours per week. Fully remote, with payment in USD.Start as soon as you successfully pass the assessment.

Before you apply

Applying takes about a minute. These four things decide how fast it moves after that.

Your profile is current

It's what we read first. Occupations, seniority and locations matter more than a long history.

Two examples you can talk through

Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.

A number in mind

What you're on now and what would make you move. We negotiate better when we know both.

Your notice period

Employers plan around it, and it's the question that stalls offers most often.

Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.