calendar 3. Oktober 2024
by user James Kelly  

Over drinks at a Eurofinance event, we discussed use cases and how treasurers could create simple automations and so we discussed how to create a form filling app using AI which would save time and tedious work. I proposed using ChatGPT to design and prototype something and overall the majority of what would be needed to build an application is there.

The technology is good enough that if you dream it, it's probably possible!
Here was the prompt and the result:

Prompt: I would like to create a prototype of an application which reads a pdf (using Python) uses gen ai to translate the field names to the database hierarchy and performs any data transformations needed and then uses Python to complete the final form. Please provide the code and descriptions for an example of how this could work in practice

Response: We can break down the solution into multiple steps. Below is a high-level description of each part of the process, followed by example code snippets:

Steps:

  1. Read and extract data from the PDF using a library like PyPDF2 or pdfplumber.
  2. Use a generative AI model (e.g., OpenAI's GPT or any other AI model you prefer) to translate the field names from the PDF into a structured database schema.
  3. Perform necessary data transformations to ensure the data aligns with your target database structure.
  4. Complete a form or generate an output using the transformed data.

Example Workflow:

  1. Extract data from PDF: We can use a Python library like pdfplumber to read and extract the text from the PDF.
Use python to extract the fields from the pdf 
import pdfplumber

def extract_pdf_data(pdf_path):
    # Open the PDF file
    with pdfplumber.open(pdf_path) as pdf:
        text = ''
        # Extract text from each page
        for page in pdf.pages:
            text += page.extract_text()
    return text

# Example usage
pdf_data = extract_pdf_data('example.pdf')
print(pdf_data)

2. Use Generative AI for field name translation: You can integrate a generative AI model (e.g., OpenAI GPT) to translate the raw text into field names and map them to a database hierarchy. For the sake of simplicity, I'll simulate the AI response here.

Access GPT to perform the next step 
import openai

# Example AI prompt function for translating field names
def translate_field_names(raw_text):
    # Simulating the AI translation (this could be an actual API call to GPT)
    prompt = f"Translate the following text fields to database hierarchy:\n{raw_text}\n"
    # Assuming the AI model provides a structured response
    # In practice, this would be an API call to OpenAI or another service
    response = {
        "Name": "Customer_Name",
        "Date of Birth": "DOB",
        "Address": "Customer_Address",
        # More mappings as per your structure
    }
    return response

# Example usage
mapped_fields = translate_field_names(pdf_data)
print(mapped_fields)

3. Perform Data Transformations: Based on the AI response, you will likely need to clean and transform the data to fit your database. This could involve converting dates, normalising text fields, etc.

Use GPT to convert field types and other data clean up 
from datetime import datetime

def transform_data(raw_data, field_mappings):
    # Simulate raw data extraction from PDF (in practice, use regex or similar)
    extracted_data = {
        "Name": "John Doe",
        "Date of Birth": "01/15/1985",
        "Address": "123 Main St, City, Country"
    }

    # Applying transformations (e.g., converting dates, cleaning addresses)
    transformed_data = {}
    for key, db_field in field_mappings.items():
        if key in extracted_data:
            if "Date of Birth" in key:
                # Convert date format to YYYY-MM-DD for database
                transformed_data[db_field] = datetime.strptime(
                    extracted_data[key], "%m/%d/%Y").strftime("%Y-%m-%d")
            else:
                # Add other transformations as necessary
                transformed_data[db_field] = extracted_data[key]
    return transformed_data

# Example usage
transformed_data = transform_data(pdf_data, mapped_fields)
print(transformed_data)

4. Generate Final Form or Database Entry: After transforming the data, you can use it to either populate a form, save it in a database, or generate a report.

Use python to complete the form 
import json

# Example of generating a final JSON output (simulating form completion)
def generate_final_form(transformed_data):
    # You could save this data into a database or use it to fill a form
    form_output = json.dumps(transformed_data, indent=4)
    return form_output

# Example usage
final_output = generate_final_form(transformed_data)
print(final_output)

Example Output:

This would output a JSON object or any other form output, representing the final transformed data:

Example Javascript output for web use 
{
   "Customer_Name": "John Doe",
   "DOB": "1985-01-15",
   "Customer_Address": "123 Main St, City, Country"
}

Integration Points:

  • Generative AI: In practice, you can replace the simulated AI call with an actual API call to a service like OpenAI GPT. You would send the extracted PDF text as input and ask the model to output mappings of the fields.
    Example of calling OpenAI API:
Example api code to perform GPT queries 
def translate_field_names_via_api(raw_text):
    openai.api_key = 'your-api-key'
    response = openai.Completion.create(
        engine="text-davinci-003",
        prompt=f"Translate the following fields into a database schema:\n{raw_text}\n",
        max_tokens=1000
    )
    return response.choices[0].text

Database/Forms: Depending on your use case, the final form output can be directly inserted into a database, used in an API, or saved in a document.

This workflow gives you a starting point to create a robust system for PDF processing, field translation using AI, and data transformation. You can modify each step to suit the specific requirements of your project.