One of the most difficult yet vital aspects of being a software engineer, or working in technology is the requirement for never-ending constant learning in order to be an expert in your field. Every day, a new article, framework/library documentation, or research paper is published, adding to the growing pile of unread knowledge. For me, and certainly for others, having a tool that might operate as a “cliff note” creator would relieve a huge mental burden.
Thanks to recent breakthroughs in Large Language Models like GPT-3.5/4, LLaMA, and PaLM, we can now do this. I created two scripts, the PDF Summary Generator and the PDF Uploader with Timing and Response Saving, by leveraging the strength and flexibility of Python and Node.js. They will work together as a command line interface (CLI) tool for anyone who is familiar with using a terminal to automate the process of uploading and summarizing PDF files. These programs also allow users to manage vast numbers of PDFs with varying page lengths.
First, the Python based PDF Summary Generator uses OpenAI’s GPT-3.5 language model to extract summaries, notes, and additional content from PDF documents automatically. It provides a RESTful API endpoint, accepting PDF files as input and returning a JSON object with the generated content. This application relies on the pdfplumber library for text extraction from PDFs and uses the openai library to interact with the GPT-3.5 API. Flask, a popular web framework, forms the backbone of this application, making it easily deployable and user-friendly.
Second, the Node.js based PDF Uploader with Timing and Response CLI script uploads all PDF files from a specified directory to the PDF Summary Generator’s endpoint, measures the processing time, and saves the response data as separate files within a directory titled after the original PDF’s file name. This script automates uploading, summarizing, and saving the results of multiple PDF files from the specified directory.
Before utilizing these scripts, users need to have the necessary software installed. For the Python-based PDF Summary Generator, Python 3.6 or higher is required, alongside an OpenAI API key and several Python libraries such as flask, pdfplumber, openai, glob2, and textwrap. The Node.js script, on the other hand, requires Node.js (version 14 or higher).
The installation process for both scripts involves cloning the respective repositories and installing the required dependencies. For the Python script:
|
1 2 3 4 5 |
git clone https://github.com/antoine1anthony/flask-pdf-summary-api.git cd flask-pdf-summary-api pip install -r requirements.txt |
The PDF Summary Generator needs an additional step of creating a file named openaiapikey.txt in the project directory and pasting the OpenAI API key into it. Customization of GPT-3.5 prompts and chatbot responses is optional.
Using the applications is straightforward. For the PDF Summary Generator, users need to run the command python main.py, which starts the Flask development server, making the application accessible at http://localhost:8000/pdfsummary.
For example, to use the API endpoint, a POST request with the PDF files as multipart file attachments should be sent:
|
1 |
curl -X POST -F "pdfs=@example.pdf" http://localhost:8000/pdfsummary |
As for the Node.js script, users should run:
|
1 2 |
npm install // Can also use yarn install node <script_filename> |
Replace <script_filename> with the name of the script file. The script looks for PDF files in a directory named ‘PDFs’ by default. However, users can change the directory by modifying the value of the ‘pdfDirectory’ variable in the script:
|
1 2 3 |
// Directory containing the PDF files const pdfDirectory = 'my_pdfs'; |
Both scripts have been designed to provide clear output and error messages. The Node.js script measures the total time taken by the script. It displays a spinner while the PDF files are being uploaded and saves the response data as separate .txt files. The PDF Summary Generator returns a JSON object containing the generated summaries, notes, and additional content.
Contributions to the development of these scripts are always welcome. Developers can fork the repository, make changes, and submit a pull request. If you stumble upon any issues or have suggestions for improvements, please open an issue on the GitHub repository.
Both the PDF Summary Generator and the PDF Uploader with Timing and Response Saving are released under the MIT License, providing a fair amount of flexibility for further development and usage.
While these scripts aim to provide an automated and efficient solution for managing and processing PDF files, users are reminded that they use the generated content at their own risk. The quality of the output may vary, and it is advisable to review the output before using it for any critical purposes.
In conclusion, these Python and Node.js applications offer a comprehensive solution for processing PDF files, extracting valuable information from them, and managing the responses effectively. From extracting text to generating notes and summaries, and from uploading multiple PDF files to timing the response, these scripts have got it all covered!
Antoine Anthony
Senior Software Engineer