GridDB<\/h3>\n
GridDB is an open-source, in-memory NoSQL database optimized for IoT and big data applications. It is a highly scalable database that can handle large volumes of data with high throughput and low latency. GridDB is also ACID compliant and supports SQL-like queries. For more information about GridDB, please visit the GridDB website<\/a>.<\/p>\n Node.js is an open-source, cross-platform JavaScript runtime environment that allows developers to build scalable network applications. It is built on top of Google’s V8 JavaScript engine. It uses an event-driven, non-blocking I\/O model that is lightweight and efficient. For more information about Node.js, please visit the Node.js website<\/a>.<\/p>\n React.js is an open-source JavaScript library for building user interfaces. It is maintained by Facebook and a community of individual developers and companies. React.js allows developers to create reusable UI components that can be used across different applications. For more information about React.js, please visit the React.js website<\/a>.<\/p>\n This project is tested on Ubuntu 20.04.2 LTS WSL 2. To run this project, you need an OpenAI API key, GridDB, Node.js, and React.js. You can install them by following the instructions below.<\/p>\n To install Node.js v20 LTS on Ubuntu using the package manager, you need to add an additional The GridDB documentation for installation in Ubuntu on WSL can be found here<\/a>. To check if GridDB is installed correctly, you can run the following command:<\/p>\n If GridDB is not started, you can start it using this command:<\/p>\n You can get the OpenAI API key by signing up for an account at OpenAI<\/a>. To keep your API key safe, you can store it in a To get started, you must clone this repository<\/a> and install the dependencies. You can do this by running the following commands:<\/p>\n Change the directory to the project root Create a And add Run the server:<\/p>\n In the browser, go to the default project URL: GPT-4 Vision will process the image and extract the tabular data from it then the tabular data will be displayed on the web page.<\/p>\n \n Please read this note<\/a> about the limitation of this project.\n<\/p><\/blockquote>\n This architecture diagram outlines a data processing and visualization workflow that incorporates GPT-4 Vision, Node.js, and GridDB. Here’s the breakdown:<\/p>\n The display data component at the bottom of the diagram indicates that the application shows the data. It’s a web interface built using React.js<\/strong>.<\/p>\n<\/li>\n<\/ul>\n The project at hand is a web application that is designed to be simple yet powerful. The application is enriched with the advanced capabilities of GPT-4 Vision model, which elevates its functionality to a whole new level.<\/p>\n The GPT4 Vision model is best at answering general questions about what is present in the images. For example, to describe an image, we can use a simple prompt:<\/p>\n And then use the prompt in GPT4-Vision API with some image from the internet:<\/p>\n The GPT4 Vision model will process the image based on the prompt and will return the result for further processing.<\/p>\n For our project, we can design a prompt to only recognize the tabular data from the uploaded image. For example, we can use this prompt to extract tabular data from the image:<\/p>\n Then, in the code, we can use the prompt to only extract the tabular data from the image:<\/p>\n The advantage of using GPT4 Vision from OpenAI is that we don’t need to train the model to recognize the tabular data from the image. We only need to design a prompt that will be used to process the image and the model is well integrated with other OpenAI models, such as GPT4, to further process the result.<\/p>\n We can use Node.js and Express.js to build a simple web server and then create an image processing route that calls GPT4-Vision API after the image is finished and uploaded.<\/p>\n For example, the The In the code, we use the encoded For a better result, we will feed the result from the Essentially, the After image processing and data cleanup, we can store the result in GridDB. We will use the The GridDB container only has two columns: The The result will be directly displayed on the web page. The reason not to retrieve the data from GridDB is to keep the project simple. You can find the code for the However, if you want to retrieve the data from GridDB, you can use the If we run the route in the browser, the server will respond with all the data saved in the GridDB database.<\/p>\n The source code for the client or user interface is in the The upload user interface is pretty simple. We use the React library for easier user interface development.<\/p>\n Here is the main entry code for the app:<\/p>\n The main application entry consists of two components: The Luckily, there is a React component in the npm package readily used for rendering markdown purposes. The react-markdown<\/a> component is easy to use and integrate with any React application.<\/p>\n Also, we need to add This project is a simple demonstration of how to use GPT4 Vision to process image data and store the result in GridDB. There are many ways to improve this project.<\/p>\n Currently, the project only supports a clear and clean image containing only a table in it. With the current state of the GPT4 Vision model, which is still in preview, it is not possible to process a noisy image, such as an image containing table data mixed with other text or images. The best way to improve this project is to crop the image containing the table data and then process it using GPT4 Vision.<\/p>\n This blog will explore how to use GPT-4 Vision to process image data, specifically tabular data from images, and store the results in GridDB. We will build a simple web application using React.js and Node.js that allows users to upload pictures and view the results of GPT-4V’s image processing. Meet the Stacks We will use […]<\/p>\n","protected":false},"author":41,"featured_media":29932,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[121],"tags":[],"class_list":["post-46789","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"\nNode.js<\/h3>\n
React.js<\/h3>\n
Prerequisites<\/h2>\n
Install Node.js<\/h3>\n
deb<\/code> repository. Please read and follow this documentation<\/a>. After that, test if the Node.js is installed correctly by typing this command in the terminal:<\/p>\nnode -v<\/code><\/pre>\n<\/div>\nInstall GridDB<\/h3>\n
sudo systemctl status gridstore<\/code><\/pre>\n<\/div>\nsudo systemctl start gridstore<\/code><\/pre>\n<\/div>\nOpenAI Account and API Key<\/h3>\n
.env<\/code> file in the project’s root directory (more on this later).<\/p>\nRunning the Project<\/h2>\n
git clone https:\/\/github.com\/griddbnet\/Blogs.git -- branch image-processing<\/code><\/pre>\n<\/div>\nserver<\/code> directory and install the dependencies:<\/p>\ncd extract-image-table-gpt4-vision\/server\nnpm install<\/code><\/pre>\n<\/div>\n.env<\/code> file in the root directory of the project and write the OpenAI API key. The .env<\/code> file should look like this:<\/p>\nOPENAI_API_KEY=[your-api-key]<\/code><\/pre>\n<\/div>\n.env<\/code> to the .gitignore<\/code> file to prevent it from being pushed to the repository.<\/p>\nnpm start<\/code><\/pre>\n<\/div>\nhttp:\/\/localhost:5115<\/code> and try to upload an image with tabular data within.<\/p>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2024\/02\/app-screenshot.png\")
<\/a><\/p>\nProject Architecture<\/h2>\n
\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2024\/02\/project-data-gpt4.png\")
<\/a><\/p>\nUnderstanding GPT4-Vision<\/h2>\n
\"What's in this image?\"<\/code><\/pre>\n<\/div>\nimport OpenAI from \"openai\";\n\nconst openai = new OpenAI();\n\nasync function getImageDescription() {\n const response = await openai.chat.completions.create({\n model: \"gpt-4-vision-preview\",\n messages: [\n {\n role: \"user\",\n content: [\n { type: \"text\", text: \"What's in this image?\" },\n {\n type: \"image_url\",\n image_url: {\n \"url\": \"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/d\/dd\/Gfp-wisconsin-madison-the-nature-boardwalk.jpg\/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\",\n },\n },\n ],\n },\n ],\n });\n console.log(response.choices[0]);\n}<\/code><\/pre>\n<\/div>\nRecreate the table in the image.<\/code><\/pre>\n<\/div>\nmessages: [\n {\n role: \"user\",\n content: [\n { type: \"text\", text: \"Recreate table in the image.\" },\n { type: \"image_url\", \n image_url: {\n \"url\": sampleImageAddress,\n },\n },\n ],\n },\n],<\/code><\/pre>\n<\/div>\nIntegrating GPT4-Vision with Node.js<\/h2>\n
process-image<\/code> route will accept the uploaded image and then will call the processImageRequest<\/code> function to process the image using OpenAI API:<\/p>\napp.post('\/process-image', upload.single('image'), async (req, res) => {\n try {\n const result = await processImageRequest(req.file.path);\n \/\/ save the result into the database\n res.json(result);\n } catch (error) {\n res.status(500).send('Error processing image request');\n }\n});<\/code><\/pre>\n<\/div>\nprocessImageRequest<\/code> function is essentially a GPT4 Vision API wrapper function. The code will call the API and then process the image based on the designed prompt earlier:<\/p>\nasync function processImageRequest(filePath) {\n const imageBuffer = await fs.readFile(filePath);\n const base64Image = imageBuffer.toString('base64');\n const encodedImage = `data:image\/jpeg;base64,{${base64Image}}`;\n\n const response = await openai.chat.completions.create({\n model: \"gpt-4-vision-preview\",\n messages: [\n {\n role: \"user\",\n content: [\n { type: \"text\", text: tablePrompt },\n { type: \"image_url\", image_url: { \"url\": encodedImage } },\n ],\n },\n ],\n max_tokens: 1024,\n });\n return response;\n}<\/code><\/pre>\n<\/div>\nbase64<\/code> format for the image input in GPT4 Vision API. This is because it is easier to reference the image stored in the server’s local storage than create a public URL for the image.<\/p>\nprocessImageRequest<\/code> function into a more general OpenAI model called GPT4 to extract the tabular data only:<\/p>\nconst cleanupPrompt = `I need you to extract the table data from this message and provide the answer in markdown format. Answer only the markdown table data, nothing else. Do not use code blocks. nn`;\n\nasync function cleanupData(data) {\n const response = await openai.chat.completions.create({\n model: \"gpt-4-1106-preview\",\n messages: [\n {\n \"role\": \"system\",\n \"content\": \"you are a smart table data extractor\"\n },\n {\n \"role\": \"user\",\n \"content\": `${cleanupPrompt} nn${data}`\n }\n ],\n temperature: 1,\n max_tokens: 2000,\n top_p: 1,\n frequency_penalty: 0,\n presence_penalty: 0,\n });\n\n return response;\n}<\/code><\/pre>\n<\/div>\ncleanupData<\/code> function will clean up the result from the processImageRequest<\/code> function by the help of specific prompts. The result will be a markdown table of data that can be displayed on the web page.<\/p>\nStoring Processed Data in GridDB<\/h2>\n
saveData<\/code> function, a wrapper function for the put<\/code> operation in GridDB. This function is in the server\/griddbservices.js<\/code> file:<\/p>\nexport async function saveData({ tableData }) {\n const id = generateRandomID();\n const data = String(tableData);\n const packetInfo = [parseInt(id), data];\n const saveStatus = await GridDB.insert(packetInfo, collectionDb);\n return saveStatus;\n}<\/code><\/pre>\n<\/div>\nid<\/code> and data<\/code>. The id<\/code> column is used to identify the data, and the data<\/code> column is used to store the markdown table data from GPT4. This snippet code is in the server\/libs\/griddb.cjs<\/code> file.<\/p>\nfunction initContainer() {\n const conInfo = new griddb.ContainerInfo({\n name: containerName,\n columnInfoList: [\n ['id', griddb.Type.INTEGER],\n ['data', griddb.Type.STRING],\n ],\n type: griddb.ContainerType.COLLECTION,\n rowKey: true,\n });\n\n return conInfo;\n}<\/code><\/pre>\n<\/div>\nsaveData<\/code> function can be called in the process-image<\/code> route. The code in the route will check if the result from GPT4 is finished. If the result is finished, the code will clean up the result and then save it to the GridDB database:<\/p>\napp.post('\/process-image', upload.single('image'), async (req, res) => {\n log.info('Processing image request')\n try {\n const result = await processImageRequest(req.file.path);\n log.info(`Result: ${JSON.stringify(result)}`);\n\n if (result.choices[0].finish_reason === 'stop') {\n const cleanedData = await cleanupData(result.choices[0].message.content);\n const saveStatus = await saveData({ tableData: JSON.stringify(cleanedData) });\n\n if (saveStatus.status === 0) {\n log.error(`Save data to GridDB: ERROR`);\n } else {\n log.info(`Save data to GridDB: OK`);\n }\n res.json({ result: cleanedData, status: true });\n } else {\n res.json({ result, status: false });\n }\n } catch (error) {\n log.error(error);\n res.status(500).send('Error processing image request');\n }\n});<\/code><\/pre>\n<\/div>\nprocess-image<\/code> route in the server\/server.js<\/code> file.<\/p>\nget-all-data<\/code> route. This route will retrieve all the data from GridDB.<\/p>\napp.get('\/get-all-data', async (req, res) => {\n log.info('Getting all data from GridDB');\n try {\n const result = await getAllData();\n res.json(result);\n } catch (error) {\n res.status(500).send('Error getting all data');\n }\n});<\/code><\/pre>\n<\/div>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2024\/02\/get-all-data.png\")
<\/a><\/p>\nBuilding an End-to-End Application<\/h2>\n
client<\/code> directory. The client is a React.js application.<\/p>\nMain Application Entry<\/h3>\n
![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2024\/02\/upload-ui.png\")
<\/a><\/p>\nimport React, { useState } from 'react';\nimport ImageUploader from '.\/ImageUploader';\nimport ReactMarkdown from 'react-markdown';\nimport remarkGfm from 'remark-gfm';\nimport rehypeRaw from 'rehype-raw';\n\nconst App = () => {\n const [markdown, setMarkdown] = useState('');\n\n const handleMarkdownFetch = (markdownData) => {\n setMarkdown(markdownData.result.choices[0].message.content);\n };\n\n return (\n Extract Table From Image Using GPT4<\/h1>\n
<ImageUploader><\/code> and <ReactMarkdown><\/code>. The <ImageUploader><\/code> component is responsible for uploading the image to the server, and it returns a result that will be handled by any function that is attached to the onMarkdownFetch<\/code> prop and then the markdown data will be rendered in the <ReactMarkdown><\/code> component.<\/p>\nImage Uploader<\/h3>\n
<ImageUploader><\/code> component code resides in the client\/src\/ImageUploader.jsx<\/code> file. The handler for the image upload is in the handleUpload<\/code> function. This function will post the image to the \/process-image<\/code> route in the server, and the result from the server will be handled by the onMarkdownFetch<\/code> prop.<\/p>\nconst handleUpload = async () => {\n if (selectedFile && !uploading) { \n setUploading(true); \n\n const formData = new FormData();\n formData.append('image', selectedFile);\n\n try {\n const response = await fetch('\/process-image', {\n method: 'POST',\n body: formData,\n });\n if (!response.ok) {\n throw new Error('Network response was not ok');\n }\n const markdownData = await response.json();\n props.onMarkdownFetch(markdownData);\n } catch (error) {\n console.error('Error posting image:', error);\n } finally {\n setUploading(false);\n }\n } else {\n alert('Please select an image first.');\n }\n };<\/code><\/pre>\n<\/div>\nMarkdown Renderer<\/h3>\n
import ReactMarkdown from 'react-markdown';\nimport remarkGfm from 'remark-gfm';\nimport rehypeRaw from 'rehype-raw';\n\n...\nremark-gfm<\/code> and rehype-raw<\/code> plugins. The remark-gfm<\/code> plugin is used to render the markdown table, and the rehype-raw<\/code> plugin is used to render the markdown table as a raw HTML element so we can add style to it later.<\/p>\nLimitations<\/h2>\n
References<\/h2>\n
\n