Track, share, code and integrate prompts easily.
Starting your first LLM project can feel like diving into a sea of endless possibilities! It’s like learning to juggle a whole bunch of things at once. First, you’re caught up in deciding which models to use, then you’re deep into setting up the model APIs, and organizing heaps of data.
But the adventure doesn’t stop there! Next, we step into the shoes of a “Prompt Engineer”. This is where the real fun begins as we start trying out different prompts for the same task and tweaking various settings like temperature and top_p values. It’s like experimenting with different ingredients to create the most delicious recipe!
However, there’s a catch. Sometimes, we end up creating so many different versions of prompts for one task that it gets a bit overwhelming. Managing, tracking, and integrating them with our own code to find the best prompts turns into quite a task. It becomes a crucial step in our LLM project journey, one that can be as challenging as solving a tedious task!
To make the tedious tasks into easy, I developed/ designed an experimental open-source attempt to make life easy for prompt engineers & developers. This can be easily used in the development lifecycle of LLM-based projects. I call it “Prompt Organizer”.
Understanding the Prompt Organizer: A Simple Guide
The Prompt Organizer is a helpful tool designed to make life easier for developers and prompt engineers. It helps in managing and organizing different versions of prompts in a structured way. This tool is open-source, meaning anyone can use and modify the code as per their requirements!
The Prompt Organizer allows users to manage prompts under different tasks like Summarization, Topic Discovery, and Intent Identification. Within each task, users can create and manage multiple versions of prompts, each with its unique set of parameters. This helps in having a clear and categorized view of each prompt.
Key Features:
- Task-Based Organization: Organize prompts under different tasks for a clear view.
- Prompt Versions Management: Manage multiple versions of prompts within each task.
- Difference Visualization: Easily visualize and compare differences between various prompt versions.
- Prompt Parameters Configuration: Configure parameters like temperature, top_p, max tokens, and threshold for each prompt version.
- Status Tracking: Set and track the status of each prompt.
- Commenting Feature: Annotate important notes or information related to the prompt.
- System Prompt Management: Manage system prompts along with user prompts.
- Save and Download: Save progress and download the organized prompts in YAML format.
- YAML Integration for Developer Pipelines: This application seamlessly facilitates developers by allowing the direct incorporation of YAML files into their development pipelines, making the development process more intuitive and less error-prone.
- Single user app: This application is currently designed for individual use, with plans for future upgrades to support multiple users.
How to run Prompt Organizer?
Go to my Github repo and download the project. Go to the Prompt Organizer folder and run below command to run the app. The Prompt Organizer is hosted as a flask application.
python app.pyNow, say you have used the app and created your prompts & set the best parameters for the final prompts and you want to use it in your code.
You have to just click on the “Download” button and Prompt Organizer will give you a YAML file which will have all the Task information which you need to use it.
Using YAML file in your code:
Download the YAML file and load it using above method.
import yaml
def read_template():
directory_path = "data.yaml"
with open(directory_path, "r") as f:
try:
yaml_content = yaml.safe_load(f)
except yaml.YAMLError as e:
print(f"Error parsing {directory_path}: {e}")
return yaml_contentGetting Prompts and Parameters
def get_prompt(task, version):
yaml_content = read_template()
version = "version" + "_" + str(version)
return yaml_content[task]['prompts']['version_1']["prompt"]
def get_parameters(task, version):
yaml_content = read_template()
version = "version" + "_" + str(version)
temp = yaml_content[task]['prompts'][version]['temperature']
top_p = yaml_content[task]['prompts'][version]['top_p']
max_tokens = yaml_content[task]['prompts'][version]['max_tokens']
threshold = yaml_content[task]['prompts'][version]['threshold']
return {"temperature": temp, "top_p": top_p, "max_tokens": max_tokens, "threshold": threshold}Handling Dynamic Input Value in Prompt:
For this step I have a very basic trick, where I know I have to place dynamic inputs from other sources there I put my placeholders like ##placeholder_1## inside the prompts and change them with replace method. You can choose your own placeholder tags.
passage_content = "Your passage data"
prompt_passage = get_prompt('Intent',1)
prompt_passage = prompt_passage.replace("##placeholder_1##", passage_content)I will share an example here how to use it with OpenAI API but do not worry you can use the same pattern with any other API or models.
import openai
openai.api_key = 'your-api-key-here'
prompt_passage = get_prompt("Intent", 1)
prompt_param = get_parameters("Intent", 1)
response = openai.ChatCompletion.create(
model = "gpt-3.5-turbo",
temperature = prompt_param["temperature"],
top_p = prompt_param["top_p"],
max_tokens = prompt_param["max_tokens"],
messages=[
{"role": "system", "content": prompt_passage},
{"role": "user", "content": system_prompt},
]
)
answer = response['choices'][0]['message']['content']
print(answer)For now, I run this app from my project to avoid going back and forth with YAML file. All my modification stays in my project level. Idea is in recent future to make this integration seamless.
By adhering to the outlined steps, you can adopt a structured and systematic approach. Prompt Organizer empowers you to design, meticulously examine, and implement your prompts with assurance. This leads to the development of advanced and reliable AI applications.
Future Ideas:
- Auto-save
- Setup git integration.
- Design for multi-user use.
- Integration of data and OpenAI/ Custom LLM APIs for auto evaluation.
- more…
I hope you will find this helpful and make your life little bit easier. I developed the Prompt Organizer with an intent to aid in the development process, making it user-friendly and well-organized and it’s an experimental open-source attempt to bring a structured approach to help prompt engineers & developers track, share, and handle prompts easily with their team. Please do visit my repo, use it & share your feedbacks.
👏 Happy Prompting 👏
Prompt Organizer
