In vertex AI conversation , BigQuery in datastore is not giving any result

[amitagarg22]

Hi , I have structured data in BigQuery. I created a datastore with BigQuery data and used it in DialogFlow CX. 
I am trying to ask questions from the database like , customer bank balance etc. But nothing is happening . Everytime default negative response is coming.
To give more context, When I am using some manual document in cloud storage as datastore then it works fine. Everytime correct response comes. 
Not sure what I am doing wrong using bigquery as data store. 

[xavidop]

Hi @amitagarg22 ,

That is weird, make sure that the Data Store is being hit. Also, can you please confirm that data store is fully indexed?

Thanks,

Xavi

[amitagarg22]

Thank you for replying @xavidop .
I found some more thread on the same topic in this community and many people are facing the same issue. They are solving it by converting structured data in BigQuery to unstructured in pdf.
I am not sure if that is the correct solution. 
To answer your question I am not sure how can i check whether datastore is hit or not and no documentation seems to help. 
Regarding index I suppose you mean search index, there were none. When I am trying to create it goes on disable mode since dataset size is less than 10GB.
I am new to GCP platform so excuse if my questions are a bit naive.

[xavidop]

Hi @amitagarg22 ,

You can check if the data store is being hit by clicking this button in the simulator:

You will see the logs and there you could see the data store response if any:

On the other hand, when I talk about the data store being indexed is when you go to the data store and checks that it is indexed (check the green check):

Of course exporting your data from BQ as pdf and create a data store from those PDFs is another option, but the data store from BQ should work.

Best,

Xavi

[amitagarg22]

Hey Xavi , 

My datastore is indexed.

I also checked in logs ... Seems structured datastore is not getting hit. All questions are hitting another datastore with unstructured data. manuals_1707201760846 is my unstructured datastore. 

Not sure what step I am missing. Mentioning again that I am getting response from unstructured datastore which i created from cloud storage but not from structured datastore created from BigQuery.

[xavidop]

Can you please show me the page where the data store is configured and the data store handler configuration?

[YogeshKumar03]

Hi, 

    We're facing a similar challenge with our Dialogflow CX chatbot Agent which uses Vertex AI Search to answer user queries directly within Google Chat. Currently, our data source is hosted as a PDF in Cloud Storage, leading to inconsistencies in the chatbot's responses. Sometimes it provides inaccurate information, and in other cases, it omits relevant data. We then migrated the data to BigQuery for improved accuracy and structure. However, it seems Vertex AI Search currently doesn't directly recognize BigQuery tables. Therefore, we're interested in knowing: Is it feasible for Vertex AI Search to be expanded to search and answer questions based on BigQuery tables with 50 columns and 5,000 rows?

[xavidop]

what I would suggest is to tune the summarization prompt: https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent#data_store_model_selection_...

if the result is not the expected one, I would suggest to build a cutom webhook using Vertex AI Search and Conversation + langchain so there you will be able to build more customization.

[amitagarg22]

In Dialogflow CX , I have not done any additional settings. 

I hope I am providing the details you asked for. 

[xavidop]

that is the issue! you have to link the data store. I saw that you only have linked the manuals data store and not the BQ one. I would recommend to only link one or another.

[amitagarg22]

Ok. Now i can see datastore being hit when I am asking the questions but still getting negative response.
One question , My bigQuery data is not FAQ types . Its more like customer details like his name , address etc etc. Its giving negative response in this case.
I also tried qith another structures data where its more like two columns with question n answer types. Giving negative response for this as well.

Only success is that now I can see datastore is getting hit. 
Thank you for your patience and support.

[xavidop]

Hi @amitagarg22 can you elaborate more what do you mean when you say negative response?

Best,

Xavi

[amitagarg22]

For every question asked I am getting below response . Customer with code 731150 exist in databse. 

Below is the Log 

[xavidop]

Okay so here are a couple of things:

1. the no match event is the event that is being triggered. this means that Dialogflow CX agent does not undertand what you are requesting and the information in the data store is not enough to answer that request

2. the use case that you are using for the data store is not a good one. what I would suggest is to get that information using a webhook and accessing BQ or any db from it. I think you wrongly understood what a data store is. A data store can be used as a backend-retrieveal information. I would suggest to take a look at this video to understand the usage of it: https://youtu.be/w8D-p0rqQKI

Best,

Xavi

[amitagarg22]

@xavidop Thank you Xavi . I will go through the video. Yes I have implemented webhook to get BigQuery data.
But I was Hoping to find out what kind of data in bigQuery can be used for this datastore feature. I just want to create something where i can use  Bigquery as datastore in chat and search app.

[amitagarg22]

@xavidop One more thing, I have organization access and have only editor role on the project.

Could it some access issue ? Though I am not seeing any error related to access. 

[xavidop]

that should be good!

[xavidop]

Hi @amitagarg22 data stores are just to retrieve information from a knowledge base. I think does not fit your use case

[amitagarg22]

Jus to update : My issue is fixed and now results from bigquery data is coming. May be it was temp issue as it was preview mode. 

[xavidop]

pretty weird yeah