🌱 New Techniques🟦 Narrative-of-Thought (NoT)

🟦 Narrative-of-Thought (NoT)

🟦 This article is rated medium

Reading Time: 5 minutes

Last updated on October 17, 2024

What is Narrative-of-Thought (NoT)?

Narrative-of-Thought (NoT) is a new prompting technique designed to improve the ability of large language models (LLMs) to handle temporal reasoning. Temporal reasoning involves understanding the relationships and sequences of events over time, which is often difficult for AI models to perform accurately due to its complexity.

LLMs often struggle with organizing events correctly in time, but NoT seeks to improve this by having models recount events as a story before generating a temporal graph—a visual representation of the sequence of events.

In essence, NoT helps LLMs organize unordered events into a clear, temporally grounded narrative by:

Representing the events as a Python class.
Prompting the LLM to generate a story (narrative) that organizes these events in the correct order.
Using this narrative to guide the final generation of a temporal graph.

This approach leverages the storytelling capabilities of LLMs and taps into the models’ pre-trained knowledge of event structures.

Benefits and Applications:

Improves performance of smaller models: NoT enables even smaller LLMs (less than 10 billion parameters) to catch up to larger models like GPT-3.5/4.
Useful in tasks requiring temporal understanding: It can be applied in scenarios like business process modeling, legal event sequences, or historical timelines.
Cost-effective: NoT works without model fine-tuning, making it budget-friendly for organizations using smaller models.

How Narrative-of-Thought differs from existing techniques

NoT builds upon techniques like Chain-of-Thought (CoT), which break down complex tasks into smaller steps. However, CoT struggles with temporal reasoning because it doesn’t generate a coherent story to bind events together in time. NoT goes beyond by creating a narrative-aware reasoning process, which results in more accurate event ordering.

For example, while CoT might output simple steps like "A -> B", NoT would generate a full story: "First, the government approves the deal, then the companies merge."

How to Use Narrative-of-Thought

Step 1. Event Setup

Gather a set of unordered events related to the goal or scenario.

For example, let's take a scenario with a walk into a store.

Step 2. Python Class Creation

Convert these events into a Python class format.

Here's an example:

class WalkIntoStore:
  title = "walk into store"
  steps = 9
  def stepE(self):
    return "stop for red lights and stop signs"
  def stepC(self):
    return "shut car door and press lock button"
  def stepH(self):
    return "get in car and go to store"
  def stepG(self):
    return "pull into store driveway"
  def stepA(self):
    return "park the car"
  def stepB(self):
    return "take the key out of the ignition"
  def stepD(self):
    return "get out of the car"
  def stepI(self):
    return "walk into store"
  def stepF(self):
    return "push gas pedal to move vehicle"

Step 3. Narrative Prompt

Use NoT to generate a narrative by asking the model to organize the events in a logical, temporal order.

Narrative Prompt:

class WalkIntoStore:
  title = "walk into store"
  steps = 9
  def stepE(self):
    return "stop for red lights and stop signs"
  def stepC(self):
    return "shut car door and press lock button"
  def stepH(self):
    return "get in car and go to store"
  def stepG(self):
    return "pull into store driveway"
  def stepA(self):
    return "park the car"
  def stepB(self):
    return "take the key out of the ignition"
  def stepD(self):
    return "get out of the car"
  def stepI(self):
    return "walk into store"
  def stepF(self):
    return "push gas pedal to move vehicle"

# Let’s think of a narrative to link aforementioned events in the correct temporal order.
def get_narrative(self):
# TODO

Output:

def get_narrative(self):
    return "This is a report about walking into a store. First, someone gets in the car and starts to go to the store. While driving, they push the gas pedal to move the vehicle but stop for red lights and stop signs along the way. After safely navigating the roads, they pull into the store's driveway and then park the car. Once the car is parked, the key is taken out of the ignition. Next, the person gets out of the car and shuts the car door, pressing the lock button to ensure the car is locked. Finally, they walk into the store. By adhering to the provided temporal information, the desired goal is achieved."

Here's a table of prompt templates you can use in the Narrative Prompt:

Instruction Type	Detailed Instruction
News Report	You are provided with a set of unordered event descriptions. You are also provided with a set of event relations which instructs you how to temporally link a pair of events. They are displayed as functions defined within a Python class. Your goal is to write a news report based on the provided event descriptions and event relations set. The generated news report should adhere to the non-fiction genre. Meanwhile, the generated news report should honor the provided temporal information.
Simple English	You are provided with a set of unordered event descriptions. You are also provided with a set of event relations which instructs you how to temporally link a pair of events. They are displayed as functions defined within a Python class. Your goal is to write a simple and concise story based on the provided event descriptions and event relations set. The generated story should be simple such that it can be understood by a 10-year-old child, and it should be concise such that it can be written within a short paragraph. Meanwhile, the generated story should honor the provided temporal information.
Role Play	You are provided with a set of unordered event descriptions. You are also provided with a set of event relations which instructs you how to temporally link a pair of events. They are displayed as functions defined within a Python class. Your goal is to write a simple and concise story based on the provided event descriptions and event relations set. The generated story should honor the provided temporal information. Now, imagine you are a character in the story. Let’s write a story that clearly depicts how you, as a character, experience the events, and how you react to them.
Simple Report	You are provided with a set of unordered event descriptions. You are also provided with a set of event relations which instructs you how to temporally link a pair of events. They are displayed as functions defined within a Python class. Your goal is to write a simple and concise report based on the provided event descriptions and event relations set. The generated report should be simple such that it can be understood by a 10-year-old child, and it should be concise such that it can be written within a short paragraph. Meanwhile, the generated report should honor the provided temporal information.

Step 4. Temporal Graph Generation

The model then uses the narrative to create a temporal graph showing the event sequence.

Temporal Graph Generation Prompt:

class WalkIntoStore:
  title = "walk into store"
  steps = 9
  def stepE(self):
    return "stop for red lights and stop signs"
  def stepC(self):
    return "shut car door and press lock button"
  def stepH(self):
    return "get in car and go to store"
  def stepG(self):
    return "pull into store driveway"
  def stepA(self):
    return "park the car"
  def stepB(self):
    return "take the key out of the ignition"
  def stepD(self):
    return "get out of the car"
  def stepI(self):
    return "walk into store"
  def stepF(self):
    return "push gas pedal to move vehicle"

def get_narrative(self):
    return "This is a report about walking into a store. First, someone gets in the car and starts to go to the store. While driving, they push the gas pedal to move the vehicle but stop for red lights and stop signs along the way. After safely navigating the roads, they pull into the store's driveway and then park the car. Once the car is parked, the key is taken out of the ignition. Next, the person gets out of the car and shuts the car door, pressing the lock button to ensure the car is locked. Finally, they walk into the store. By adhering to the provided temporal information, the desired goal is achieved."

def get_relations(self):
# TODO
# END

Output:

def get_relations(self):
    return [
      "stepF -> stepE",
      "stepE -> stepG",
      "stepG -> stepA",
      "stepB -> stepD",
      "stepA -> stepB",
      "stepD -> stepC",
      "stepC -> stepI",
      "stepH -> stepF",
    ]

How to Use NoT with One-Shot Demonstration?

Here's an example of a prompt template for NoT with one-shot demonstration including a high-quality reference narrative. The first past is a demonstration of a Python class WalkIntoStore and the corresponding correct narrative get_narrative and temporal graph get_relations. The second one, class BusinessChange could be replaced by your task represented by a Python class.

One-shot NoT prompt:

# *** Complete the class “WalkIntoStore" by implementing "get_relations()" function marked by #TODO. You should *ONLY* implement the function "get_relations()" and not generate anything else. Don't generate the entire class "BusinessChange". Don't generate comments. Your response must end in "# END".

# *** You are first given a set of demonstrations of how to implement the "get_relations()" function for different classes.

class WalkIntoStore:
  title = "walk into store"
  steps = 9
  def stepE(self):
    return "stop for red lights and stop signs"
  def stepC(self):
    return "shut car door and press lock button"
  def stepH(self):
    return "get in car and go to store"
  def stepG(self):
    return "pull into store driveway"
  def stepA(self):
    return "park the car"
  def stepB(self):
    return "take the key out of the ignition"
  def stepD(self):
    return "get out of the car"
  def stepI(self):
    return "walk into store"
  def stepF(self):
    return "push gas pedal to move vehicle"

#Let's think about a narrative to link aforementioned events in the correct temporal order.

  def get_narrative(self):
    return "This is a report about walking into a store. First, someone gets in the car and starts to go to the store. While driving, they push the gas pedal to move the vehicle but stop for red lights and stop signs along the way. After safely navigating the roads, they pull into the store's driveway and then park the car. Once the car is parked, the key is taken out of the ignition. Next, the person gets out of the car and shuts the car door, pressing the lock button to ensure the car is locked. Finally, they walk into the store. By adhering to the provided temporal information, the desired goal is achieved."
  def get_relations(self):
    return [
      "stepF -> stepE",
      "stepE -> stepG",
      "stepG -> stepA",
      "stepB -> stepD",
      "stepA -> stepB",
      "stepD -> stepC",
      "stepC -> stepI",
      "stepH -> stepF",
    ]
# END

# *** Complete the class "BusinessChange" by implementing "get_narrative()" and "get_relations()" functions marked by #TODO. "get_narrative()" serves as an auxiliary function facilitating the temporal cohesion of events. Essentially, it helps ensure the temporal accuracy of the predicted temporal graph produced in "get_relations()", by explicitly constructing a coherent, temporally correct story involving all provided events.

# You should *ONLY* implement the function "get_narrative()" and "get_relations()", but not generate anything else. Don't generate the entire class "BusinessChange". Don't generate comments. Your response must end in "# END".

class BusinessChange:
  title = "business change"
  steps = 6
  def stepC(self):
    return "offer acquisition deal"
  def stepF(self):
    return "companies reach a deal"
  def stepE(self):
    return "companies merge"
  def stepD(self):
    return "companies negotiate"
  def stepA(self):
    return "government approve the deal"
  def stepB(self):
    return "company plans on acquisition"

#Let's think of a narrative to link aforementioned events in the correct temporal order.
  def get_narrative(self):
    #TODO
  def get_relations(self):
    #TODO
# END

Results of Narrative-of-Thought

The authors tested NoT across three benchmarks (ProScript, Schema-11, WikiHow) and compared it against models like GPT-3.5 and GPT-4. The results showed significant improvements, especially for smaller models.

Key Findings:

Better performance for smaller models: NoT helped smaller models achieve results similar to GPT-3.5, improving their F1 scores by 16% to 71%.
Most effective for temporal reasoning: NoT outperformed Chain-of-Thought (CoT) in tasks requiring temporal understanding, particularly for small LLMs.
Top structural similarity: NoT produced the most accurate temporal graphs, even surpassing GPT-4 in some structural metrics.

Conclusion

NoT significantly narrows the gap between small LLMs and larger models like GPT-4, providing a cost-effective method to enhance temporal reasoning in AI systems.

Valeriia Kuka

Valeriia Kuka, Head of Content at Learn Prompting, is passionate about making AI and ML accessible. Valeriia previously grew a 60K+ follower AI-focused social media account, earning reposts from Stanford NLP, Amazon Research, Hugging Face, and AI researchers. She has also worked with AI/ML newsletters and global communities with 100K+ members and authored clear and concise explainers and historical articles.

Footnotes

Zhang, X. F., Beauchamp, N., & Wang, L. (2024). Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives. https://arxiv.org/abs/2410.05558 ↩