# Prompt‑Lab Tracker

## Overview
The Prompt‑Lab Tracker workspace lets prompt engineers, data scientists, and product managers **log every prompt experiment**, **run it against multiple LLMs**, **auto‑calculate token usage & cost**, and **roll‑up metrics for quick insight**. The workspace is built around three core relational databases (Prompts, Experiments, Models) and a set of role‑specific checklists that enforce best‑practice governance. Exportable audit reports let you share results with stakeholders or archive them for compliance.

## Databases

### 1️⃣ Prompts
| Prompt ID | Title | Prompt Text | Owner | Tags | Created |
|-----------|-------|-------------|-------|------|---------|
| PR‑001 | Summarize Article | *Summarize the following article in 3 sentences:* | Alice | summary, research | 2024‑03‑12 |
| PR‑002 | Generate JSON | *Return a JSON object with keys `name`, `age`, `city` based on the description:* | Bob | json, data‑prep | 2024‑04‑05 |
| PR‑003 | Tone Shift | *Rewrite the paragraph in a friendly, conversational tone:* | Carol | style, rewrite | 2024‑05‑01 |

> 💡 Tip: Keep the **Prompt Text** concise but include placeholders (e.g., `{{input}}`) that you’ll replace in the Experiments table.

### 2️⃣ Models
| Model ID | Provider | Model Name | Context Length (tokens) | Pricing (USD / 1k tokens) |
|----------|----------|------------|--------------------------|---------------------------|
| MD‑001 | OpenAI | gpt‑4‑turbo | 128k | 0.03 |
| MD‑002 | Anthropic | claude‑3‑sonnet | 200k | 0.018 |
| MD‑003 | Cohere | command‑r | 128k | 0.025 |

> 💡 Tip: Add a **Formula** property `Cost per Token = round(Pricing / 1000, 6)` for quick reference in the Experiments roll‑up.

### 3️⃣ Experiments (relational hub)
| Experiment ID | Prompt (rel) | Model (rel) | Input Sample | Output Sample | Tokens Used | Cost (USD) | Success Metric | Run Date | Status |
|---------------|--------------|------------|--------------|---------------|-------------|------------|----------------|----------|--------|
| EX‑101 | PR‑001 | MD‑001 | *Article about climate change* | *Three‑sentence summary* | 842 | 0.025 | ROUGE‑L = 0.78 | 2024‑06‑10 | ✅ Completed |
| EX‑102 | PR‑002 | MD‑002 | *John, 29, lives in Berlin* | `{"name":"John","age":29,"city":"Berlin"}` | 124 | 0.002 | JSON Valid = True | 2024‑06‑12 | ✅ Completed |
| EX‑103 | PR‑003 | MD‑003 | *“Our product offers unparalleled performance.”* | *“Hey there! Our product totally rocks with amazing performance.”* | 58 | 0.001 | Human Rating = 4/5 | 2024‑06‑13 | ⏳ In‑Progress |

> 💡 Tip: Use a **Rollup** on Experiments → `Tokens Used` → `Sum` in the Prompt page to see total token consumption per prompt.

## Views & Dashboards
- **Prompt Dashboard** (Board view grouped by `Owner`): visualizes which team member owns which prompts and their experiment status.
- **Experiment Log** (Table view with filters): shows only experiments from the last 30 days, sortable by `Cost (USD)`.
- **Model Cost Tracker** (Gallery view): each model card displays a roll‑up of total cost across all experiments, helping you budget per provider.
- **Metric Summary** (Table view): roll‑up of `Success Metric` averages per Prompt, with a conditional color formula (green ≥ 0.8, orange ≥ 0.5, red < 0.5).

## Checklists & Role‑Specific Workflows
### Prompt Engineer Checklist
1. Draft prompt in **Prompts** database.  
2. Tag with `#review‑needed`.  
3. Assign **Owner** and set **Created** date.  
4. Link to relevant **Models** via relation.  

### Data Scientist Checklist
1. Pull prompt from **Prompts** → create **Experiment** entry.  
2. Populate **Input Sample** and run the LLM via API integration (see Automation).  
3. Verify **Tokens Used** auto‑calculates; confirm **Cost (USD)**.  
4. Record **Success Metric** (e.g., BLEU, human rating).  

### Product Manager Checklist
1. Review **Metric Summary** dashboard weekly.  
2. Approve prompts that meet the success threshold (`Success Metric ≥ 0.75`).  
3. Export audit report (see Reporting) for stakeholder review.

> 💡 Tip: Use Notion’s **Template Buttons** on the Experiments database to pre‑fill the checklist steps for each new run.

## Automation & Calculations
1. **Token‑Cost Formula (Experiments)**  
   ```
   Cost (USD) = round((Tokens Used / 1000) * prop("Model").Pricing, 4)
   ```
2. **Success Roll‑up (Prompts)** – average of linked Experiments’ `Success Metric`.  
3. **Status Automation** – a **Select** property `Status` auto‑updates to “✅ Completed” when `Cost (USD)` is not empty and `Success Metric` is filled (via Notion’s built‑in formula).  
4. **API Integration** – embed a **Notion‑to‑Zapier** webhook that triggers a Python script to call the selected model’s endpoint, returns the output, token count, and writes back to the Experiment row.

## Reporting & Export
- **Audit Report Template** (Page linked from each Prompt) contains:
  - Prompt details (text, owner, tags).  
  - Table of all Experiments (filtered by Prompt relation).  
  - Roll‑up of total tokens, total cost, average success metric.  
  - Export button (via Notion’s “Export → PDF/HTML”) for compliance archives.  

> 💡 Tip: Set the page’s **Properties** to “Show as PDF” and schedule a weekly Zapier automation that emails the latest audit PDFs to the compliance mailbox.

## How to import into Notion
1. **Create a new page** in your Notion workspace where you want the template.  
2. Click **Import** → choose **Markdown & CSV** and upload this `.md` file.  
3. Notion will render the headings and tables as plain tables.  
4. For each table (Prompts, Models, Experiments):  
   - Click the three‑dot menu on the table → **Convert to Database** → select **Table – Full Page** (or **Inline** if you prefer).  
   - Rename the database to match the heading (e.g., “Prompts”).  
5. Re‑create the **Relation** properties:  
   - In **Experiments**, add a Relation to **Prompts** (link by `Prompt ID`).  
   - Add another Relation to **Models** (link by `Model ID`).  
6. Add the **Formula**, **Rollup**, and **Select** properties as described in the Automation section.  
7. Duplicate the provided **Template Buttons** and **Checklists** into the respective databases.  
8. Set up any external automation (Zapier/Webhooks) using the URLs and formulas noted above.  

Your Prompt‑Lab Tracker workspace is now ready for production use—log, run, and analyze every LLM prompt experiment with full cost transparency and auditability. Happy prompting!