One member on r/PromptWizards just applied for a job as a Prompt Engineer in a company, and they tasked him to craft a prompt system that generates high-quality SAT-style multiple-choice questions. Quite a quest, right? Well, stick around, and we'll take a deep dive into the prompt engineering we used to help him.
The mission was precise: Write a prompt to yield an SAT-style multiple-choice question that rigorously tests a student's understanding of "Algebraically solving systems of 2 linear equations in 2 variables". The challenge didn't end there; the question produced had to meet the hard/difficult mark set by real SAT questions.
Using the OpenAI Playground, we conducted incisive iterations, testing each prompt separately to mitigate any bias from prior outputs.
Our approach was:
- Build an initial system prompt devoid of question examples and guidelines.
- Iterate on this by introducing examples of potential questions and guidelines.
- Craft a user prompt instructed to randomize elements or detailed instructions.
- Refine the user prompt, including elements of randomization and instructions for the output format.
Here is my answer to the member:
“In order to prevent bias, I suggest you only use the Playground of OpenAI, if this recruiter is knowledgeable enough he'll recognize that this is the way to properly do it (as then it can be programmed to be API calls with system/user/assistant slots
"You can test variants of your prompt separately to prevent bias from prior outputs. Use a new chat instance for each iteration."
---
Then I create for you something to start with that is consistent in the output, only outputs the questions, with ranking in difficulties... Hope this helps, if you don't know how to use the playground let me know I can guide you otherwise here are the prompts.
--
Little suggestion to make your documentation:
Make a first prompt (system): That doesn't have examples of questions/or/doesn't have guidelines
Make a second prompt (system): That does have these last elements
Make a first prompt (user): without "Randomize the generation of the questions while following the system guidelines. Consider the examples and mix it with the guidelines to generate real-life specific examples.3
Make a second prompt (user): With these last things.
Hope this helps! I'm really curious to know more about the context, how you got this offer, what company hires about this role etc... If you can share in the DMs I'm pretty curious.
System prompt:
We are trying to generate high-quality, SAT-style multiple-choice questions that meet the specified standards for the "Heart of Algebra" section of the SAT test. This process involves creating questions that assess a student's understanding of linear equations, systems of linear equations, and functions in detailed, engaging, and challenging ways. The questions should cover a range of skills, from creating and solving equations and inequalities to recognizing relationships between algebraic and graphical representations.
Guidelines: Creating SAT-style multiple-choice questions for the "Heart of Algebra" section involves:
Making questions based on the specified algebraic standards like solving linear equations, and interpreting relationships between different algebraic forms, framing scenario-based questions to test practical application of algebraic knowledge, ensuring a variety of questions covering all aspects of the "Heart of Algebra" segment for comprehensive assessment, providing clear, quality distractors that reflect common student misunderstandings, having conceptually-challenging questions, not just those based on rote memorization, reviewing and testing questions for suitability, difficulty, and clarity, adjusting questions based on feedback received after review and testing to ensure alignment with the "Heart of Algebra" standards.
Heart of Algebra includes the following types of questions:
Create, solve, or interpret a linear expression or equation in 1 variable.
Create, solve, or interpret linear inequalities in 1 variable.
Build a linear function that models a linear relationship between 2 quantities.
Create, solve, and interpret systems of linear inequalities in 2 variables.
Create, solve, and interpret systems of 2 linear equations in 2 variables.
Algebraically solve linear equations (or inequalities) in 1 variable.
Algebraically solve systems of 2 linear equations in 2 variables.
Interpret the variables and constants in expressions for linear functions.
Understand connections between algebraic and graphical representations."
User prompt:
Your main goal is to generate high-quality, SAT-style multiple-choice questions that meet the specified standards for the "Heart of Algebra" section of the SAT test.
Randomize the generation of the questions while following the system guidelines. Consider the examples and mix it with the guidelines to generate real-life specific examples.
Your output is only the questions (ranked from easiest to hardest) (5 maximum), nothing more.
“
Anyone else has got an offer or applied to jobs as a Prompt Engineer here? Would be curious if they also gave you such a challenge, or something else!
Keep the wizardry alive!
[link] [comments]