Your biggest obstacle is proving fine-tuning is more effective than prompting, workflow design, RAG, etc during the initial pass. Most of my customers are still getting big improvements by picking the low-hanging fruit with those approaches. A much smaller fraction is at a place where they're ready to start fine-tuning. Obviously, this will change as AI programs mature.
Exactly! Finetuning needs at least 10 examples to even work. That’s why Promptrepo begins with prompting and schema-based generation when teams have little or no data. As they gather more examples, it gradually shifts to fine-tuning. It’s the classic cold start problem and we’ve simplified it for product teams who want to launch quickly but improve accuracy over time.
I love integrating into spreadsheets. Super easy to use. Reminds me a bit of mailmerge.
I don't have much experience with modern finetuning, but isn't it highly technical. How many layers do you want to change? What is the learning rate? Does that need to be visible to the user? How many examples are needed in practice
Incredibly crowded space, but this is a great insight and UI... engineers probably have to integrate the model but we should empower non-technical customer-facing people to give feedback to the model in a way that improves it.
The blocker for me (and likely other cost-conscious early stage groups)? I have free credit and existing integrations with more mainstream platforms (OpenAI, anthropic, together). Trying this out will cost both eng time and money, so I won't be an early adopter. I wonder if there's a way to pass the cost through / use my API keys with credits. Maybe it's more for enterprise teams or cases where you're already confident about the fine-tuning approach.
Thanks! Yes, that's a popular request from other developers as well. That's why our basic plan is self managed plan (Bring your own OpenAI account) - https://promptrepo.com/finetune/pricing.html
We have one month free trial as well. Free free to ping me if you need more time.
What’s the thinking of spreadsheet first? Just making it super accessible for people who already have data?
I’m building a UI for fine tuning (and evals, and synthetic data gen) - https://github.com/Kiln-AI/Kiln - and went the custom UI route. From chatting with folks - most people don’t have datasets, and need help building them.
It depends on the use case. For many business workflows, where structured data is key - spreadsheets are already the source of truth. But for chat-based or unstructured tasks, a custom UI might make more sense.
this is great, and right up my alley. i built a customer support chatbot using a google sheet for our CS folks to input & structure their question/answer pairs. that is auto xformed into markdown and fed into the bot for context. it works fairly well considering how simple it was to do. i'm really intrigued as to what promptrepo can add to that. will definitely give it some R&D time!
Yeah, Google Sheets is a surprisingly powerful interface for business teams, especially when they’re the ones curating the data. Curious to see how Promptrepo fits into your workflow. Happy to help if you explore finetuning on top of your existing setup.
I don't like the pricing. $38 a month headline much smaller billed annually with the price more than doubling. This is the trick your customers strategy and hope for no chargebacks.
I understand your concern. We’ve had cases where short-term users abused our product for phishing, which led us to remove the monthly plan initially. To address that without locking out genuine users, we’ve added an option to extend the trial for up to 3 months before choosing a monthly or yearly plan. Feel free to message me if you’d like more time.
When I see a product — especially an early-stage product — trying to funnel me into a yearly subscription, the signal I get is that you think I’ll quit in less than a year which, in turn, implies that your product isn’t very good. It just makes me trust you less.
There's some truth to that. Early stage products often start out rough and part of the journey is finding early users who believe in the potential enough to stick around while we improve.
That said, in our case, the switch away from monthly plans wasn't just about churn. We actually got shut down by GoDaddy due to phishing abuse [1], which forced us to rethink our approach. We've since added a flexible trial extension to avoid punishing genuine users, but I’m open to feedback and willing to change if enough people feel strongly.
Neat! Wonder though if you should be even offering the BYO option as a separate lite package. As a dev I would not buy this and as a non-tech person I would be confused by your pricing page.
But I do see the value! Think sales or marketing folks looking to get a bit more hands on. These will likely be your first visitor and be okay with your 50 dollar price. Then, their IT department will say “we want to hook up our own API key for that,” to which you can confidently say “sure, we can do that too.”
honestly i like seeing tools get simpler like this, makes me wonder though how much of the real value comes from the tech itself vs just making everything less scary for folks who aren't engineers. you think easy interfaces actually help people get better results or just get more folks trying stuff?
Depends on who’s using the tool. Developers might be fine coding a form into their app, but an HR person needs a form builder. Similarly, the data for training models usually lives with domain experts, but they don’t have a tool to actually do the training. That’s why, in this use case, a simple interface makes sense, IMO.
We don’t have automated evals, latency, or cost comparisons yet. But, Promptrepo does offer versioning and lets you deploy the same model across providers for comparison. Automating these comparisons is definitely on our roadmap.
In OpenAI Pro ($20/mo) one can start a project with a set of files. Various chats can be had about this project topic with the files providing additional information. I've discovered the projects are isolated. They'll use memory configurable in settings but they don't use chat history outside the project. This can give ChatGPT chats in projects a different tone.
My question is this: Is this fine tuning with those project documents or RAG and what's the difference?
Thanks for the clarification. OpenAI projects in ChatGPT are meant for end users to get personalized help using their own documents, inside the ChatGPT UI.
Promptrepo is for developers and product teams to build new AI-powered features in their product. It’s about creating custom models that run behind the scenes in apps, not just improving a personal chat experience.
So while OpenAI projects use RAG for better chats, Promptrepo helps teams build and deploy fine-tuned APIs that serve structured outputs like JSON, labels, or extracted fields to build your own AI powered product.
Don’t want to sound like a shill, but Google doesn’t use data from your Google Sheets to train its models or for advertising. By default, your data stays private and is protected under Google Workspace’s privacy policies.
Agree, not super familiar with this space but that you are using GS as the UI makes this super accessible to so many more people given familiarity and low intimidation factor. Best of luck.
Yeah, just like Dropbox was a passthrough for aws s3.
Edit: Sorry about the snide comment. But if this ends up as a simple utility for finetuning, I would be happy with that too. Just want to share a tool that's been very useful in building ai features in our products.
That's the biggest problem with showcasing to developers. They are just not representative of the user base at large. For example, any Desktop application showcased will getting annihilated for being Electron. You very much take a risk putting it out to developers first because they will leave commentary about your product in this way (forever, this is the internet, it's not going anywhere). We eat out our own basically.
I am not saying there isn't value for this, just saying what it does. I might actually use this if they don't tie the fine tuned models to only be used within their platform.
Your biggest obstacle is proving fine-tuning is more effective than prompting, workflow design, RAG, etc during the initial pass. Most of my customers are still getting big improvements by picking the low-hanging fruit with those approaches. A much smaller fraction is at a place where they're ready to start fine-tuning. Obviously, this will change as AI programs mature.
Exactly! Finetuning needs at least 10 examples to even work. That’s why Promptrepo begins with prompting and schema-based generation when teams have little or no data. As they gather more examples, it gradually shifts to fine-tuning. It’s the classic cold start problem and we’ve simplified it for product teams who want to launch quickly but improve accuracy over time.
Can you share an example of such real world win where fine tuning was less effective ? I’m curious about sample business cases.
I love integrating into spreadsheets. Super easy to use. Reminds me a bit of mailmerge.
I don't have much experience with modern finetuning, but isn't it highly technical. How many layers do you want to change? What is the learning rate? Does that need to be visible to the user? How many examples are needed in practice
Incredibly crowded space, but this is a great insight and UI... engineers probably have to integrate the model but we should empower non-technical customer-facing people to give feedback to the model in a way that improves it.
The blocker for me (and likely other cost-conscious early stage groups)? I have free credit and existing integrations with more mainstream platforms (OpenAI, anthropic, together). Trying this out will cost both eng time and money, so I won't be an early adopter. I wonder if there's a way to pass the cost through / use my API keys with credits. Maybe it's more for enterprise teams or cases where you're already confident about the fine-tuning approach.
Anyway, congrats on the launch!
Thanks! Yes, that's a popular request from other developers as well. That's why our basic plan is self managed plan (Bring your own OpenAI account) - https://promptrepo.com/finetune/pricing.html
We have one month free trial as well. Free free to ping me if you need more time.
What’s the thinking of spreadsheet first? Just making it super accessible for people who already have data?
I’m building a UI for fine tuning (and evals, and synthetic data gen) - https://github.com/Kiln-AI/Kiln - and went the custom UI route. From chatting with folks - most people don’t have datasets, and need help building them.
It depends on the use case. For many business workflows, where structured data is key - spreadsheets are already the source of truth. But for chat-based or unstructured tasks, a custom UI might make more sense.
this is great, and right up my alley. i built a customer support chatbot using a google sheet for our CS folks to input & structure their question/answer pairs. that is auto xformed into markdown and fed into the bot for context. it works fairly well considering how simple it was to do. i'm really intrigued as to what promptrepo can add to that. will definitely give it some R&D time!
Yeah, Google Sheets is a surprisingly powerful interface for business teams, especially when they’re the ones curating the data. Curious to see how Promptrepo fits into your workflow. Happy to help if you explore finetuning on top of your existing setup.
I don't like the pricing. $38 a month headline much smaller billed annually with the price more than doubling. This is the trick your customers strategy and hope for no chargebacks.
I understand your concern. We’ve had cases where short-term users abused our product for phishing, which led us to remove the monthly plan initially. To address that without locking out genuine users, we’ve added an option to extend the trial for up to 3 months before choosing a monthly or yearly plan. Feel free to message me if you’d like more time.
When I see a product — especially an early-stage product — trying to funnel me into a yearly subscription, the signal I get is that you think I’ll quit in less than a year which, in turn, implies that your product isn’t very good. It just makes me trust you less.
There's some truth to that. Early stage products often start out rough and part of the journey is finding early users who believe in the potential enough to stick around while we improve.
That said, in our case, the switch away from monthly plans wasn't just about churn. We actually got shut down by GoDaddy due to phishing abuse [1], which forced us to rethink our approach. We've since added a flexible trial extension to avoid punishing genuine users, but I’m open to feedback and willing to change if enough people feel strongly.
[1] https://news.ycombinator.com/item?id=36934109
Neat! Wonder though if you should be even offering the BYO option as a separate lite package. As a dev I would not buy this and as a non-tech person I would be confused by your pricing page.
But I do see the value! Think sales or marketing folks looking to get a bit more hands on. These will likely be your first visitor and be okay with your 50 dollar price. Then, their IT department will say “we want to hook up our own API key for that,” to which you can confidently say “sure, we can do that too.”
N=1, just my two cents. Good luck!
Heads up I get "ERR_CONNECTION_RESET" when navigating to https://promptrepo.com/finetune
Thanks for the heads up. Looks like a traffic spike from HN. Can you please check now?
Still down for me but I bet it's the traffic spike still. I'll check back in later!
honestly i like seeing tools get simpler like this, makes me wonder though how much of the real value comes from the tech itself vs just making everything less scary for folks who aren't engineers. you think easy interfaces actually help people get better results or just get more folks trying stuff?
Depends on who’s using the tool. Developers might be fine coding a form into their app, but an HR person needs a form builder. Similarly, the data for training models usually lives with domain experts, but they don’t have a tool to actually do the training. That’s why, in this use case, a simple interface makes sense, IMO.
FineTuning Ops platform requirement - compare evals, latency, cost across models.
We don’t have automated evals, latency, or cost comparisons yet. But, Promptrepo does offer versioning and lets you deploy the same model across providers for comparison. Automating these comparisons is definitely on our roadmap.
How is this different than OpenAI projects?
Can you please clarify what you mean by "OpenAI projects"? Are you referring to the playground or the API for prompting or fine-tuning?
In OpenAI Pro ($20/mo) one can start a project with a set of files. Various chats can be had about this project topic with the files providing additional information. I've discovered the projects are isolated. They'll use memory configurable in settings but they don't use chat history outside the project. This can give ChatGPT chats in projects a different tone.
My question is this: Is this fine tuning with those project documents or RAG and what's the difference?
Thanks for the clarification. OpenAI projects in ChatGPT are meant for end users to get personalized help using their own documents, inside the ChatGPT UI.
Promptrepo is for developers and product teams to build new AI-powered features in their product. It’s about creating custom models that run behind the scenes in apps, not just improving a personal chat experience.
So while OpenAI projects use RAG for better chats, Promptrepo helps teams build and deploy fine-tuned APIs that serve structured outputs like JSON, labels, or extracted fields to build your own AI powered product.
Why Google Sheet though? Why would you want your customers to give their training data to Google?!
Don’t want to sound like a shill, but Google doesn’t use data from your Google Sheets to train its models or for advertising. By default, your data stays private and is protected under Google Workspace’s privacy policies.
This is awesome, I don't have a fine-tuning use case yet, but I can't imagine something being easier than a spreadsheet.
Agree, not super familiar with this space but that you are using GS as the UI makes this super accessible to so many more people given familiarity and low intimidation factor. Best of luck.
Essentially this is a frontend to automate the process of converting a csv file into jsonl and pass through a fine-tuning service.
Yeah, just like Dropbox was a passthrough for aws s3.
Edit: Sorry about the snide comment. But if this ends up as a simple utility for finetuning, I would be happy with that too. Just want to share a tool that's been very useful in building ai features in our products.
That's the biggest problem with showcasing to developers. They are just not representative of the user base at large. For example, any Desktop application showcased will getting annihilated for being Electron. You very much take a risk putting it out to developers first because they will leave commentary about your product in this way (forever, this is the internet, it's not going anywhere). We eat out our own basically.
I am not saying there isn't value for this, just saying what it does. I might actually use this if they don't tie the fine tuned models to only be used within their platform.