By default we serve all models that are submitted to Guanaco with the Pygmalion formatting. Should you wish for your model to be served with custom formatting, you can create a new PromptFormatter sub-class and pass this as a parameter in your model submission. Below we demonstrate this with the PygmalionFormatter.

from chai_guanaco.formatters import PromptFormatter

class PygmalionFormatter(PromptFormatter):
    memory_template = "{bot_name}'s Persona: {memory}\\n####\\n"
    prompt_template = "{prompt}\\n<START>\\n"
    bot_template = "{bot_name}: {message}\\n"
    user_template = "{user_name}: {message}\\n"
    response_template = "{bot_name}:"

Each field in the PromptFormatter coresponds to an individual section in the overall model input:

• memory_template
  • This field is where the bot's memory is interpolated, and is always included in the model input (up to necessary truncation to fit the model's context window).
  • Any of {user_name, bot_name, memory} can be interpolated into this template.
• prompt_template
  • This field is where the bot's prompt is interpolated. The prompt is used to help initiate the conversation, and is truncated to fit the model's context window when the chat history is large.
  • Any of {user_name, bot_name, memory} can be interpolated into this template.
• bot_template
  • This string is used to template the section corresponding to a bot's message in the chat history.
• user_template
  • This string is used to template the section corresponding to a user's message in the chat history.
• response_template
  • This string templates the final section of the model input after the chat history.

Once the above strings are templated, they are combined in the following order:

memory_template + prompt_template + chat_history + response_template

where chat_history is obtained by templating the chat history according to bot_template and user_template, and then combining the resulting strings chronologically.

In order not to affect the user experience too much, we do not support applying custom post-processing to the values {user_name, bot_name, memory, prompt}

To submit your model with a custom formatter, you can pass the model_submission parameter as follows:

submission_parameters = {
	"model_repo": "EleutherAI/gpt-j-6b",
	"generation_params": {
		"temperature": 0.99,
    "top_p": 1.0,
    "top_k": 50,
    "repetition_penalty": 1.0,
    "stopping_words": ["\\n"]
	},
  "formatter": PygmalionFormatter()
}

Here the optional value stopping_words corresponds to the end-of-string words that are used to tell your model to stop outputting tokens. You should ensure that it matches the token used in your custom formatter (if any).