This project solution makes AI image generation more accessible through voice interaction.<\/p>\n
Architecture & Tech Stack<\/h2>\n
This diagram shows a pipeline for converting speech into images and storing the result in the GridDB database:<\/p>\n In this project, we will use Next.js<\/a> as the frontend framework.<\/p>\n There is no specific backend code because all services used are from APIs. There are three main APIs:<\/p>\n This project utilizes the ElevenLabs API for speech-to-text transcription. The API is athttps:\/\/api.elevenlabs.io\/v1\/text-to-speech\/<\/a>. ElevenLabs also provides JavaScript SDK for easier API integration. You can see the SDK documentation<\/a> for more details.<\/p>\n This project uses Imagen 4 API from fal<\/a>. The API is hosted on https:\/\/fal.ai\/models\/fal-ai\/imagen4\/preview<\/a>. Fal provides JavaScript SDK for easier API integration. You can see the SDK documentation<\/a> for more details.<\/p>\n We will use the GridDB Cloud version in this project. So there is no need for local installation. Please read the next section on how to set up GridDB Cloud.<\/p>\n This project is built using Next.js, which requires Node.js version 16 or higher. You can download and install Node.js from https:\/\/nodejs.org\/en<\/a>.<\/p>\n If you would like to sign up for a GridDB Cloud Free instance, you can do so at the following link: https:\/\/form.ict-toshiba.jp\/download_form_griddb_cloud_freeplan_e<\/a>.<\/p>\n After successfully signing up, you will receive a free instance along with the necessary details to access the GridDB Cloud Management GUI, including the GridDB Cloud Portal URL<\/strong>, Contract ID<\/strong>, Login<\/strong>, and Password<\/strong>.<\/p>\n Go to the GridDB Cloud Portal and copy the WebAPI URL from the Clusters<\/strong> section. It should look like this:<\/p>\n Go to the GridDB Users<\/strong> section of the GridDB Cloud portal and create or copy the username for For more details, to get started with GridDB Cloud, please follow this quick start guide<\/a>.<\/p>\n When running this project, please ensure that the IP address where the project is running is whitelisted. Failure to do so will result in a 403 status code or forbidden access.<\/p>\n You can use a website like What Is My IP Address<\/a> to find your public IP address.<\/p>\n To whitelist the IP, go to the GridDB Cloud Admin and navigate to the Network Access<\/strong> menu.<\/p>\n You need an ElevenLabs account and API key to use this project. You can sign up for an account at https:\/\/elevenlabs.io\/app\/sign-up<\/a>.<\/p>\n After signing up, go to the Account<\/strong> section, and create and copy your API key.<\/p>\n You need an Imagen 4 API key to use this project. You can sign up for an account at https:\/\/fal.ai<\/a>.<\/p>\n After signing up, go to the Account<\/strong> section, and create and copy your API key.<\/p>\n Clone the repository from https:\/\/github.com\/junwatu\/speech-image-gen<\/a> to your local machine.<\/p>\n Install all project dependencies using npm.<\/p>\n Copy file Please look the section on Prerequisites<\/a> before running the project.<\/p>\n Run the project using the following command:<\/p>\n Open the application in your browser at http:\/\/localhost:3000<\/a>. You also need to allow the browser to access your microphone.<\/p>\n The user will speak into the microphone and the audio will be recorded. The audio will be sent to ElevenLabs API for speech-to-text transcription. Please, remember that the language supported is English.<\/p>\n The code to save the recording file is in the main The audio processing flow is as follows:<\/p>\n The audio data will be saved in the The full code for the The transcribed text will be sent to ElevenLabs API for text-to-speech synthesis. The code to send the audio to ElevenLabs API is in the The The route will return the following JSON object:<\/p>\n The transcribed text serves as the input prompt for image generation using Imagen 4 from fal.ai, which creates high-quality images based on the provided text description.<\/p>\n![\"\"](\"https:\/\/griddb.net\/wp-content\/uploads\/2025\/11\/system-arch-scaled.png\")
<\/a><\/p>\n\n
\n
Frontend Stack<\/h3>\n
Backend Stack<\/h3>\n
1. Speech-to-Text API<\/h4>\n
2. Image Generation API<\/h4>\n
3. Database API<\/h4>\n
Prerequisites<\/h2>\n
Node.js<\/h3>\n
GridDB<\/h3>\n
Sign Up for GridDB Cloud Free Plan<\/h4>\n
GridDB WebAPI URL<\/h4>\n
<\/a><\/p>\nGridDB Username and Password<\/h4>\n
GRIDDB_USERNAME<\/code>. The password is set when the user is created for the first time, use this as the GRIDDB_PASSWORD<\/code>.<\/p>\n<\/a><\/p>\nIP Whitelist<\/h4>\n
<\/a><\/p>\nElevenLabs<\/h3>\n
<\/a><\/p>\nImagen 4 API<\/h3>\n
<\/a><\/p>\nHow to Run<\/h2>\n
1. Clone the repository<\/h3>\n
git clone https:\/\/github.com\/junwatu\/speech-image-gen.git\ncd speech-image-gen\ncd apps<\/code><\/pre>\n<\/div>\n2. Install dependencies<\/h3>\n
npm install<\/code><\/pre>\n<\/div>\n3. Set up environment variables<\/h3>\n
.env.example<\/code> to .env<\/code> and fill in the values:<\/p>\n# Copy this file to .env.local and add your actual API keys\n# Never commit .env.local to version control\n\n# Fal.ai API Key for Imagen 4\n# Get your key from: https:\/\/fal.ai\/dashboard\nFAL_KEY=\n\n# ElevenLabs API Key for Speech-to-Text\n# Get your key from: https:\/\/elevenlabs.io\/app\/speech-synthesis\nELEVENLABS_API_KEY=\n\nGRIDDB_WEBAPI_URL=\nGRIDDB_PASSWORD=\nGRIDDB_USERNAME=<\/code><\/pre>\n<\/div>\n4. Run the project<\/h3>\n
npm run dev<\/code><\/pre>\n<\/div>\n5. Open the application<\/h3>\n
Implementation Details<\/h2>\n
Speech Recording<\/h3>\n
page.tsx<\/code>. It uses a native media recorder HTML 5 API to record the audio. Below is the snippet code:<\/p>\nconst startRecording = useCallback(async () => {\n try {\n setError(null);\n const stream = await navigator.mediaDevices.getUserMedia({ \n audio: {\n echoCancellation: true,\n noiseSuppression: true,\n sampleRate: 44100\n } \n });\n \n \/\/ Try different MIME types based on browser support\n let mimeType = 'audio\/webm;codecs=opus';\n if (!MediaRecorder.isTypeSupported(mimeType)) {\n mimeType = 'audio\/webm';\n if (!MediaRecorder.isTypeSupported(mimeType)) {\n mimeType = 'audio\/mp4';\n if (!MediaRecorder.isTypeSupported(mimeType)) {\n mimeType = ''; \/\/ Let browser choose\n }\n }\n }\n \n const mediaRecorder = new MediaRecorder(stream, {\n ...(mimeType && { mimeType })\n });\n \n mediaRecorderRef.current = mediaRecorder;\n audioChunksRef.current = [];\n recordingStartTimeRef.current = Date.now();\n\n mediaRecorder.ondataavailable = (event) => {\n if (event.data.size > 0) {\n audioChunksRef.current.push(event.data);\n }\n };\n\n mediaRecorder.onstop = async () => {\n const duration = Date.now() - recordingStartTimeRef.current;\n const audioBlob = new Blob(audioChunksRef.current, { \n type: mimeType || 'audio\/webm' \n });\n const audioUrl = URL.createObjectURL(audioBlob);\n \n const recording: AudioRecording = {\n blob: audioBlob,\n url: audioUrl,\n duration,\n timestamp: new Date()\n };\n \n setCurrentRecording(recording);\n await transcribeAudio(recording);\n stream.getTracks().forEach(track => track.stop());\n };\n\n mediaRecorder.start(1000); \/\/ Collect data every second\n setIsRecording(true);\n } catch (error) {\n setError('Failed to access microphone. Please check your permissions and try again.');\n }\n}, []);<\/code><\/pre>\n<\/div>\n\n
startRecording()<\/code> is called.<\/li>\ngetUserMedia()<\/code>.<\/li>\nMediaRecorder<\/code> with optimal settings.<\/li>\npublic\/uploads\/audio<\/code> folder. Below is the snippet code to save the audio file:<\/p>\nexport async function saveAudioToFile(audioBlob: Blob, extension: string = 'webm'): PromisesaveAudioToFile()<\/code> function is in the app\/lib\/audio-storage.ts<\/code> file.<\/p>\nSpeech to Text Transcription<\/h3>\n
transcribeAudio()<\/code> function. The full code is in the lib\/elevenlabs-client.ts<\/code> file.<\/p>\n\/\/ Main transcription function\nexport async function transcribeAudio(\n client: ElevenLabsClient, \n audioBuffer: Buffer, \n modelId: ElevenLabsModel = ELEVENLABS_MODELS.SCRIBE_V1\n) {\n try {\n const result = await client.speechToText.convert({\n audio: audioBuffer,\n model_id: modelId,\n }) as TranscriptionResponse;\n\n return {\n success: true,\n text: result.text,\n language_code: result.language_code,\n language_probability: result.language_probability,\n words: result.words || [],\n additional_formats: result.additional_formats || []\n };\n } catch (error) {\n console.error('ElevenLabs transcription error:', error);\n return {\n success: false,\n error: error instanceof Error ? error.message : 'Unknown error'\n };\n }\n}<\/code><\/pre>\n<\/div>\nTranscription Route<\/h3>\n
transcribeAudio()<\/code> function is called when accessing the \/api\/transcribe<\/code> route. This route only accepts the POST<\/code> method and processes the audio file sent in the request body. The ELEVENLABS_API_KEY<\/code> environment variable in the .env<\/code> is used in the route to initialize the ElevenLabs client.<\/p>\nexport async function POST(request: NextRequest) {\n \/\/ Get audio file from form data\n const formData = await request.formData();\n const audioFile = formData.get('audio') as File;\n \n \/\/ Convert to buffer\n const arrayBuffer = await audioFile.arrayBuffer();\n const audioBuffer = Buffer.from(arrayBuffer);\n \n \/\/ Initialize ElevenLabs client\n const elevenlabs = new ElevenLabsClient({ apiKey: apiKey });\n \n \/\/ Convert audio to text\n const result = await elevenlabs.speechToText.convert({\n file: audioBlob,\n modelId: \"scribe_v1\",\n languageCode: \"en\", \n tagAudioEvents: true,\n diarize: false,\n });\n \n return NextResponse.json({ \n transcription: result.text,\n language_code: result.languageCode,\n language_probability: result.languageProbability,\n words: result.words\n });\n}<\/code><\/pre>\n<\/div>\n{\n \"transcription\": \"Transcribed text from the audio\",\n \"language_code\": \"en\",\n \"language_probability\": 0.99,\n \"words\": [\n {\n \"start\": 0.0,\n \"end\": 1.0,\n \"word\": \"Transcribed\",\n \"probability\": 0.99\n },\n \/\/ ... more words\n ]\n}<\/code><\/pre>\n<\/div>\nImage Generation<\/h3>\n