Prompt refinement: From Company Descriptions to Impact Reporting August 12, 2024 | Philipp Gross

Given a company, how can we estimate its impact? In an ongoing research project we ask this seemingly simple question over and over again.

Since there is a lot of ambiguity involved it helps to carefully analyze the problem: Company related data is mapped to an impact estimation, so input and output needs to be defined rigoursly, as well the function that performs the mapping. This approach follows the mathematical method and is often used in reseach papers, but it rarely follows the timeline of how the solution was found. Nevertheless we zone out a bit to be aware of the terminology before we dive into an iterative approach.

Definitions

Input: What is a company?

Organizations generate observable data as a result of their activities, and to some extent we are able to model how it operates. For example, companies are legal entities created by individuals, and hence inherently involve human interaction at some level. We won't push boundaries like DAO here, so we can assume that there are people in charge of the companies' operations. They plan the products or services and are in charge of executing the operations.

Those activities leave a trail of datapoints:

• Publicly Accessible: Data that can be accessed by anyone without restrictions, usually intended for marketing, compliance or informational purposes. For example, company websites, social media accounts, press releases, regulatory fillings and general marketing material.
• Restricted Access: Data that requires specific permissions to access, typically for operational, customer service, or internal management purposes. For example, customer databases, internal reports, employee data, internal communications, project management tools, or intranet of documents.
• Private: Highly confidential data that is strictly controlled and only accessible to a limited group of authorized personnel within the company. For example, proprietary research and development, strategic plans, financial records, legal documents, or sensitive customer data.

There is also a ton of unknown or "dark data", information the company generates but often doesn't use or even realize exists:

• Web traffic and SEO data
• Job postings and employee activity:
• Social media and public forums
• Import/Export records
• Vendor and partner activity
• Credit card transactions
• Stock market activity
• Regulatory filings and patent applications
• Satellite imagery and geolocation data
• Dark web monitoring
• Domain and IP tracking
• Participation in industry events and other networking interactions

Output: What is impact?

Over time the word "impact" evolved from its literal Latin origins ("impingere" - "to push/strike against") and broadened to include the figurative sense of having a strong effect or influence on something or someone. In the context of companies, "impact" refers to the various effects that a company has on its stakeholders, the environment and society as a whole. This concept encompasses a wide range of dimensions, including:

• Economic impact: Job creation, taxes, investment and economic activity.
• Social impact: Community engagement, employee well-being, diversity, inclusion, and ethical practices.
• Environmental impact: Resource consumption, pollution and conservation efforts.
• Customer impact: Quality and safety of products and services, customer satisfaction and ethical marketing.
• Governance impact: Corporate governance, ethical business conduct and compliance, and accountability.
• Reputational impact: Public perception, media coverage and crisis management.

Companies often measure and manage their impact through frameworks such as Corporate Social Responsibility (CSR), Environmental, Social, and Governance (ESG) criteria, and sustainable reporting. By understanding and improving their impact, companies can contribute to sustainable development, enhance their competitiveness, and build long-term value for their stakeholders.

Function: Mapping organizations to impact

Once we have a basic understanding of what goes in and what should come out, we need to address the actual system that performs the mapping. Here, we focus on large language models (LLMs) as building blocks because to some large extend the information provided as input – but also desired as output – is human readable. So developing the function boils down to prompt writing and chaining various LLM calls.

Iterative development

It is good practice to begin the development of data science models with a discussion of the metrics used to evaluate the performance of the model. Only then start with implementing a baseline, evaluate it, update it, and repeat. In reality, there is no obvious metric, let alone a dataset, so it has to be developed in parallel. Here, we are either using our own understanding of impact, and you should at some point directly talk to impact analysts to check the quality of the results.

We proceed now by a step by step approach and attack all definition problems as we go along. As our building block we are using language models since it comes very natural to the problem.

Step 1 - The basic

To get the ball rolling, let's start with a minimal prompt template example that basically rephrases our original research question (setting aside any security issues for now):

"""
{context}

What is the impact of the organization mentioned above?
"""

The prompt offers a clear and straightforward approach to evaluating the impact of an organization by focusing on its effects based on the provided context. This enables a well-rounded assessment of both positive and negative consequences, making it flexible for various aspects like social, economic, or environmental impacts. Since it relies on the context given, it avoids the need for additional external knowledge and directs attention to the specific details provided, ensuring that responses are focused and relevant.

However, the prompt also has limitations, including a lack of specificity which may result in vague or overly general responses if the context or type of impact isn't well-defined. Its effectiveness depends on the completeness of the provided context, and without it, responses might lack depth or relevance. There’s also a risk of potential bias and information overload if the context is too extensive. Additionally, without explicit criteria for assessing significance, responses can vary widely in focus and depth, leading to potential inconsistencies in interpretation.

Step 2 - First enhancement

"""
{context}

Based on the text provided, what is the social and environmental impact of the organization?
"""

While we intially set out to estimate the impact of an organisation, we added the seemingly small constraint now that our analysis is based on a given text, which could contain ridiculous claims. While this is not what we where looking for, it is a pivot of the original question to be able to to address the question as best as possible. It also teaches us an implicit ground truth that is always relevant: You can only make estimations based on data. If you have no data, there are no estimations, only prior knowledge.

The prompt also assumes that the context clearly states a certain organization. Moreover, it doesn't specify any aspects of social or environmental knowledge. And since the respondent is allowed to answer freely, there is no easy way to verify the results since it isn't enforced to substantiate the claims based on the input context.

Step 3 - Justifying analysis

Impact prompt:

"""
{context}

Based on the provided text about {organization_name}, please analyze its social and environmental impact. List the pros and cons of these impacts and provide specific examples from the text to support your analysis.
"""

It seems simple enough, but nevertheless, if we don't exactly have the name of the organization (let's say we have just the company website URL), we can offload it to a subtask:

Organization name identification prompt:

"""
{context}

What is the name of the organization presented in the text above?
”""

While the latter auxiliary prompt is easy enough (let's not go into the ambiguities of names), our core prompt is far from perfect. It doesn't provide any contextual information about the organization, which might make it difficult to tailor the analysis appropriately. Also, we haven't attacked the impact definition yet.

Step N - Climbing the ladder of complexity

As we proceed with our refinement the prompt gets larger and gives more control and better quality of the output.

Impact prompt:

"""
{context}

Key statements:
{statements}

Ojective: Based on the provided text about {organization_name}, analyze its social and environmental impact using the Theory of Change framework and IRIS+ metrics.

Instructions:
1. **Theory of Change Analysis:** Outline how the organization’s activities lead to its intended social and environmental outcomes. Identify inputs, activities, outputs, outcomes, and long-term impacts.
2. **IRIS+ Metrics:** Apply IRIS+ metrics to assess the organization's performance in terms of social and environmental impact. Consider metrics related to efficiency, effectiveness, and sustainability.
3. **Pros and Cons:** Evaluate the strengths and weaknesses of the organization’s approach based on the theory of change and IRIS metrics.

Please ensure that the analysis is detailed, incorporates relevant data from the provided statements, and offers actionable insights into the organization’s impact.

"""

Statement prompt:

"""
{context}

Extract explicit statements (mission, vision, ethics & values, taglines) from the organization {organization_name} in the text above.
"""

The impact prompt is focused now on explicit statements which are extracted in a separated step and instead of just ambiguous terms like "impact" it is asked to generate impact analysis based on established principles like Theory of Change and IRIS Metrics.

By this we are approaching the limits of the method though. While it might be tangible to generate the right list of metrics of mentrics, the calculation of number estimates requires access to proprietary information. But as an intermediate solution we can generate code to calculate IRIS metrics depending on input variables which is tailored to a company. Those code fragments will be just explanatory examples of how one could attack impact estimations if one had access to data.

Conclusion

There is no end in improving the prompt, but the system you end up with is flexible enough to change easily. If the results feed into a reporting tool, it's useful to generate structured JSON data which allows other UI components other than free text.