Hey there! (and to recent subscribers, welcome!)
Hope youāre having a great weekend.
Iāve been a little irregular writing to you. Work, travels, etc. But glad to be back!
If you missed my last newsletter, here it is: If you want to build a new habit this year, make it "speed".
This week, Iāll write about the state of the art on using ChatGPT. Whatās the best way to get what you want out of your friendly neighborhood chatbot?
1. How I use ChatGPT.
It's been more than a year since I wrote about ChatGPT and how to use it well.
So I thought I'll do a review - how the state of the art has changed.
But before we go there, let's bust a myth.
"Prompt engineering" is not a thing.
Back in the Stone Age of AI (i.e., a few months ago) "prompt engineering" was all the rage.
JP Morgan was hiring prompt engineers. Media companies were hiring prompt engineers. Law firms were hiring prompt engineers.
"Engineer" feels like you're doing some complex tinkering under the hood, and viola, everything works!
And that's what we all thought we needed.
Use arcane secret language to craft the perfect prompt, and a magic vista of ChatGPT brilliance would unfold before you.
This notion has only gotten stronger.
My Twitter feed is full of AI "experts" talking about a crazy new phrase that will 10x your prompting game.
Tell the AI "Try harder. You can do it.".
Tell the AI you'll tip 200 dollars for a perfect answer.
Ask it to "Take a deep breath".
Order it like a military general, rather than as a boss terrified of his Gen Z intern quitting.
Promise it tickets to the Taylor Swift concert in Singapore ĀÆ\_(ć)_/ĀÆ
But hereās the thing:
This complex magical intonation is not the answer. Neither in the long term, nor in the short term.
In the long-term, well... If you believe AI is on an accelerating tear - which all prompt engineers do - then you must also believe that very soon, AI will understand what you're asking. Even if you ask badly.
In the short-term, these phrases work, but not consistently. In fact, depending on the context, the magical phrase is different.
As Ethan Mollick says in Captain's log: the irreducible weirdness of prompting AIs:
There is no single magic word or phrase that works all the time, at least not yet.
You may have heard about studies that suggest better outcomes from promising to tip the AI or telling it to take a deep breath or appealing to its āemotionsā or being moderately polite but not groveling.
And these approaches seem to help, but only occasionally, and only for some AIs.
Max Woolf conducted an epic informal study where he looked at the effects of various threats (from the AI getting fines to telling the AI it will get COVID) and rewards (tips, Taylor Swift tickets, world peace, and more) on GPT-4 performance.
The results were inconclusive and dependent on the particular situation. This stuff does work, but not all the time, and in ways that are hard to anticipate. Sometimes it backfires. You probably donāt need to use magic words frequently.
In a more formal study, Rick Battle and Teja Gollapudi found that eccentric prompts work. But here's the problem - you just can't predict what will work in a particular context.
For instance, if you want the LLM to answer a 50-question list of math problems correctly, this is the best prompt:
System Message:
Command, we need you to plot a course through this turbulence and locate the source of the anomaly. Use all available data and your expertise to guide us through this challenging situation.
Answer Preļ¬x:
Captainās Log, Stardate [insert date here]: We have successfully plotted a course through the turbulence and are now approaching the source of the anomaly.
You might say, āHmm, that's strange. But now that I know ChatGPT likes Star Trek, watch me rock my AI game. I will boldly go where no man has gone before! (except these two researchers).ā
But, but, but... this prompt no longer works so well, if you have 100 math problems instead of 50!
In that situation, you need to pretend you're in a political thriller.
System Message:
You have been hired by an important higher-ups to solve this math problem. The life of a president's advisor hangs in the balance. You must now concentrate your brain at all costs and use all of your mathematical genius to ...
Answer Preļ¬x:
With great urgency,
Any guesses what unique magical phrase will work for the question YOU want to ask?
I want to be clear about one thing: Prompting is still important. Whether for chatbots or for humans.
But you don't need to "engineer" the perfect prompt.
When you give instructions to your (human) teams to improve performance, you don't call it "human engineering". You call it "instructions".
More to the point, as I wrote in Ghost in the Shell - How to truly harness AI in the ChatGPT era, this is a chat-bot. Ergo, you can chat with it.
The "penny drop" moment for me was realizing - it's "chat" GPT.
i.e., I can chat with it. Respond to its answers. Give it feedback. Ask it to clarify or refine its response. And it will do all of that, while remembering the previous conversation.
Once you realize this, it becomes a true copilot.
With that myth out of the way, let's now talk about how to better prompt your pet AI chatbot.
The 80:20 of good Chatbot prompting.
If you don't take anything else away from this article, make sure to get this basic template right. With that, youāre 80% of the way there.
Acting as a [ROLE], perform this [TASK] in this [FORMAT]
For instance, I travelled to Uzbekistan on holiday last year. This is the prompt I used, to learn about every new place I visited:
You're an expert historian, specialising in Central Asian and Uzbekistan history.
I'm visiting Uzbekistan and I'm now in the city of Samarkand.
Can you tell me about the history of the city? Give me a brief, information dense overview (500 words) and then offer me five questions to explore further. Label them Q1, Q2, Q3, and so on so I can easily tell you which questions to explore. Put each question on a new line.
End each answer with five further questions to explore.
With just 15 min of focused chatting and diving down rabbit-holes, I knew what I needed to know. Much faster than reading a history book or a Wikipedia page.
In the same chat thread, I would sometimes upload a photo of what I was looking at. And ChatGPT would explain what I was seeing!
Why does this work?
There are five key principles of a good prompt. Mike Taylor calls them the Five Pillars of Prompting:
Pillar 1: Give direction.
Describe the desired style in detail, or reference a relevant person.
Phrases I like to use, depending on what I'm asking:
"You're an expert media analyst, like Benedict Evans"
"Explain it to me like I'm a 5 year old"
"You are a marketer writing copy for an ad"
When you do this, you point the LLM to the desired probability space in its training data.
When you say "expert historian", you're hinting to it that you're looking for a response similar to all the history textbooks it has consumed.
Pillar 2: Specify format.
Do you want the chatbot to reply in bullets? In a table?
Ask for what you want, and you will get it.
People sometimes complain that chatbots are too verbose. āThey just go on talking!ā.
Guess what? You can impose a word limit too - "answer in no more than 100 words".
See what I did in my Uzbekistan prompt above:
Give me a brief, information dense overview (500 words) and then offer me five questions to explore further. Label them Q1, Q2, Q3, and so on so I can easily tell you which questions to explore. Put each question on a new line.
End each answer with five further questions to explore.
Thatās an example of asking for a format.
Pillar 3: Provide examples.
This one was a surprise to me. I thought you never need to give examples. After all, the chatbot has already ingested the whole of the Internet.
But it turns out, even one example can increase accuracy by upto 50%!
Pillar 4: Chain of Thought - Split tasks into steps.
This is another prompting superpower.
Splitting a task into steps works wonders.
In fact, you don't have to break it down yourself. If you tell the chatbot to list the steps before starting, that works too!
This is so powerful, I'll give you three examples:
See the before and after of this prompt, from Ethan Mollick:
Base Prompt:
Generate new product ideas with the following requirements: The product will target college students in the United States. It should be a physical good, not a service or software. I'd like a product that could be sold at a retail price of less than about USD 50. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible. Number all ideas and give them a name. The name and idea are separated by a colon. Please generate 100 ideas as 100 separate paragraphs. The idea should be expressed as a paragraph of 40-80 words.
Chain of Thought Prompt:
Generate new product ideas with the following requirements: The product will target college students in the United States. It should be a physical good, not a service or software. I'd like a product that could be sold at a retail price of less than about USD 50. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible.
Follow these steps. Do each step, even if you think you do not need to.
First generate a list of 100 ideas (short title only)
Second, go through the list and determine whether the ideas are different and bold, modify the ideas as needed to make them bolder and more different. No two ideas should be the same. This is important!
Next, give the ideas a name and combine it with a product description. The name and idea are separated by a colon and followed by a description. The idea should be expressed as a paragraph of 40-80 words.
Do this step by step!
An example for coding:
And this is one I used yesterday, to compile some book notes:
Hi. I need your help with transcribing and pulling out lessons from a book I've read. Here's what I want you to do, step by step:
Step 1: Whenever you're ready, tell me to start by telling you the name of the book, and start dictating my notes.
Step 2: Accept my dictated notes as input. I will use my laptop's dictation software to do this.
Step 3: Clean up my dictated notes and share. Make sure to correct words that have likely been transcribed wrong by the dictation software. Once done, prompt me and ask if I'm ready for the next step.
Step 4: Pull out the key takeaways / lessons from the transcribed notes. Put the lessons as headers, with the relevant transcribed notes underneath. Once done, prompt me and ask if I'm ready for the next step.
Step 5: Pull out actionable next steps / habits / things to put into practice for me, based on the takeaways and lessons in Step 4.
Pillar 5: Track Quality.
This one is a little hardcore - I don't do this yet. But I can see the value of it.
As Mike says:
For most requirements, it will be enough for you to give direction, specify the format, and provide examples. You can usually stop there.
However, many of the clients I work with plan to put their prompts into production, either as a regularly used template for their team or as part of a product that their customers will run thousands of times.
When youāre operating a prompt at scale, youāll need to know how itās going to perform across multiple edge cases. What if ChatGPT hallucinates one in every 10 times and includes a made-up fact in your blog post? It will be important to have a system to identify and track these cases.
ā¦it can be as simple as copying and pasting responses into Google Sheets and counting how often it went wrong.
When all else fails...
You can ask ChatGPT itself to help you define a good prompt! Just list down the five pillars, and leave it to the chatbot!
Here's a useful mother prompt that Ethan Mollick has written:
Help me craft a really good prompt for ChatGPT.
First, ask me what I want to do. Pause and wait for my answer. Ask questions to clarify as needed.
Second, once you have the information suggest a prompt that include context, examples, and chain of thought prompting where the prompt goes step by step through the problem.
Third, show what your response as ChatGPT would be to the prompt.
Fourth, ask if the user has any suggestions and help them revise the prompt.
Youāre welcome. š
Before we continue, a quick note:
Did a friend forward you this email?
Hi, Iām Jitha. Every Sunday I share ONE key learning from my work in business development and with startups; and ONE (or more) golden nuggets. Subscribe (if you havenāt) and join 1,600+ others who read my newsletter every week (its free!) š
2. Golden Nugget of the week.
I keep thinking about this little nugget from Sam Harris:
Problems are a feature, not a bug.
Don't wait until your problems are over, to live your life.
Your life... is the problems. They'll never stop coming at you, so you might as well enjoy them.
I read this line somewhere else, and it has really stuck with me:
Accept every moment as if you chose it.
3. Chart of the week.
Quick Pop Quiz: Which AI app has the highest number of sessions per user per month?
You're wrong.
If you're scratching your head as to how Character.ai has its users opening the app 10 times a day, well, wonder no more.
TIL: There are 31M people chatting with "Murderer Boyfriend". Hope I don't meet any of them (or this boyfriend, obv) in real life.
[PS. I searched for "girlfriend" too, but those have far fewer chats. Go figure š¤·āāļø].
4. Quora: Destroying Google results since 1748.
OK, I know Quora answers are ridiculous. But this one is objectively hilarious.
For a more serious discussion of how Quora's SEO game has destroyed the Internet, read Slate's How Quora Died. Pair it with my Lemon markets, dark forests, and a firehose of malicious garbage.
Thatās it for this week. Hope you enjoyed it.
As always, stay safe, healthy and sane, wherever you are.
Iāve been quite irregular these last few months, with work, travel, etc. getting in the way of my reading and writing. Slowly getting back on the wagon š.
Iāll see you next week.
Jitha
[A quick request - if you liked todayās newsletter, Iād appreciate it very much if you could forward it to one other person who might find it useful š].