chatgpt max_tokens的进一步展开说明

Breaking the Token Limit: How to Work with Large Amounts of Text in ChatGPT

Have you ever been hindered by the maximum token limit when using ChatGPT to handle a large amount of text? In this article, we will explore how to overcome this limitation using the OpenAI API and Python.

The Challenge of Token Limitation

When I encountered a project that required me to analyze a video transcript with over 50,000 words, I realized that ChatGPT could not process such a high word count. With ChatGPT’s token limit of around 4000 tokens (equivalent to approximately 8,000 words), I needed to find a workaround.

The Solution: Batch Processing

To surpass this limitation, I adopted a technique called “batch processing.” By breaking down the script into smaller chunks of text, I could use the OpenAI API to process each batch separately. To ensure sufficient context, I set the batch size to 250 words. The AI would receive 500 words of context, with 250 words before and after the target text. To prevent sentence cutoffs, which I encountered during initial testing, I set the max_tokens to 1000.

Here is an example of the code snippet we implemented:

“`python
import openai

openai.api_key = “your api key here”

# Your large text body here
script = “paste your text here”

# Setting batch size and context size
batch_size = 250

# Tokenize the script
script_tokens = script.split(” “)

for i in range(0, len(script_tokens), batch_size):
if i < batch_size: before_context = "" else: before_context = " ".join(script_tokens[i-batch_size:i]) text_to_edit = " ".join(script_tokens[i:i+batch_size]) if i+batch_size*2 >= len(script_tokens):
after_context = “”
else:
after_context = ” “.join(script_tokens[i+batch_size:i+batch_size*2])

prompt = f”Please proofread, rewrite, and improve the following text inside the brackets (in the context that it is a youtube script for a narrated video), considering the context given before and after it: before:”{before_context}” text to edit:{text_to_edit} after:”{after_context}” []”

response = openai.Completion.create(
model=”text-davinci-003″,
prompt=prompt,
temperature=0.9,
max_tokens=1000,
top_p=1,
frequency_penalty=0.25,
presence_penalty=0
)

# Print the response from the GPT⑶ API
print(response[“choices”][0][“text”])
“`

Limitations and Considerations

It is important to acknowledge that this method comes with limitations. Since we only provide small chunks of context before and after the target text, GPT⑶ lacks the understanding of the entire story.

I also want to highlight the approximate cost of running this script. In my case, when processing a 50,000-word video transcript, it amounted to around 9 dollars. While I used a paid account, it’s worth noting that OpenAI offers an $18 credit when signing up.

By breaking down the text and utilizing batch processing, you can leverage the power of ChatGPT to handle larger amounts of text within the token limitations.

chatgpt max_tokens的常见问答Q&A