Question

SQL Syntax Error on a PDT Create


I am playing around with PDTs and running into an error on the table create. This is a memSQL database…


derived_table: {
create_process: {

sql_step: CREATE TABLE ${SQL_TABLE_NAME}
(start_date DATETIME NOT NULL,
end_date DATETIME NOT NULL,
stream_id INT UNSIGNED NOT NULL,
program_identifier SMALLINT UNSIGNED,
device_identifier INT UNSIGNED,
input_information VARCHAR(255),
input_type VARCHAR(255),
encoder_decoder VARCHAR(255),
encoder_decoder_profile VARCHAR(255),
chroma_subsampling VARCHAR(255),
expected_frame_rate DECIMAL(10,3) UNSIGNED,
frame_latency_millisec DECIMAL(10,6),
horizontal_resolution_pixels SMALLINT UNSIGNED,
vertical_resolution_pixels SMALLINT UNSIGNED,

KEY pk_stream_id_start_date_end_date (
stream_id,
start_date,
end_date
)

) ;;
sql_step: INSERT INTO ${SQL_TABLE_NAME} (

SELECT
meta_data.effective_time AS start_date,
CASE WHEN (lead(meta_data.effective_time) over (partition by meta_data.stream_id) IS NULL) THEN CONVERT("9999-12-31 23:59:59",DATETIME)
ELSE lead(meta_data.effective_time) over (partition by meta_data.stream_id) END AS end_date,
meta_data.stream_id,
meta_data.program_identifier,
meta_data.device_identifier,
meta_data.input_information,
meta_data.input_type,
meta_data.encoder_decoder,
meta_data.encoder_decoder_profile,
meta_data.chroma_subsampling,
meta_data.expected_frame_rate,
meta_data.frame_latency_millisec,
meta_data.horizontal_resolution_pixels,
meta_data.vertical_resolution_pixels
FROM
(SELECT
effective_time,
stream_id,
program_identifier,
device_identifier,
input_information,
input_type,
encoder_decoder,
encoder_decoder_profile,
chroma_subsampling,
expected_frame_rate,
frame_latency_millisec,
horizontal_resolution_pixels,
vertical_resolution_pixels
FROM
live_raw_meta_video_stream AS e
ORDER BY stream_id,effective_time
) AS meta_data)
;;

sql_step: ANALYZE ${SQL_TABLE_NAME} ;;
}

If I manually run the exact SQL generated by the explore the syntax work fine in MEMSQL studio…Is there some validation using MySQL syntax ?


10 replies

The syntax error is on the create table first step… The MemSQL database encountered an error while running this query.


derived_table pdt_live_raw_meta_video_stream creation failed: SQL Error in CREATE TABLE: java.sql.SQLSyntaxErrorException: (conn=542726) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘insights.LC$KPV6V1560777912807_pdt_live_raw_meta_video_stream’ at line 1

Userlevel 7
Badge +1

Hmm.


Could you share the generated SQL for that step that you say works directly in Memsql but is failing in Looker? That might hold the key to what’s causing the hiccup.

I put all the create_process sql_steps in the post…it is the first step that it is complaining about.


Let me know if something is missing.


Regards,

Userlevel 7
Badge +1

Hmmmm.


I’m just taking stabs here. It’s saying the error is near the table name. Does MemSQL have some requirements to put the table name in quotes or `` blocks? One thing you could try is taking the actual generated SQL that you’re saying runs fine against your MemSQL db directly, making sure that runs fine in SQL runner also, and then trying to put that literally in the sql_step (ie: the fully generated SQL that doesn’t have ${SQL_TABLE_NAME} but actually has the underlying table reference inserted).


That would help narrow it down to that table reference/confirm or reject my hunch.

So a behavior update…the sql runs in SQL runner for the table create…I think the error message is wrong when I run the explore the create table error is displayed but when I look in the database the table has been created…I made sure the table was not there prior to running the explore…moving on the the next sql steps to see if the problem is really some other step.

Noticed another thing…the join to the PDT is using a different table same than the sql steps used to create the PDT ???


-- generate derived table pdt_live_raw_meta_video_stream
-- Building live_monitor::pdt_live_raw_meta_video_stream in dev mode on instance 18a313debffdaa6ef246d2b1ae4aa1e0
CREATE TABLE insights.LC$KPV6V1561123494236_pdt_live_raw_meta_video_stream
(start_date DATETIME NOT NULL,
end_date DATETIME NOT NULL,
stream_id INT UNSIGNED NOT NULL,
program_identifier SMALLINT UNSIGNED,
device_identifier INT UNSIGNED,
input_information VARCHAR(255),
input_type VARCHAR(255),
encoder_decoder VARCHAR(255),
encoder_decoder_profile VARCHAR(255),
chroma_subsampling VARCHAR(255),
expected_frame_rate DECIMAL(10,3) UNSIGNED,
frame_latency_millisec DECIMAL(10,6),
horizontal_resolution_pixels SMALLINT UNSIGNED,
vertical_resolution_pixels SMALLINT UNSIGNED,

KEY pk_stream_id_start_date_end_date (
stream_id,
start_date,
end_date
)

)
-- Building live_monitor::pdt_live_raw_meta_video_stream in dev mode on instance 18a313debffdaa6ef246d2b1ae4aa1e0
INSERT INTO insights.LC$KPV6V1561123494236_pdt_live_raw_meta_video_stream (

SELECT
meta_data.effective_time AS start_date,
CASE WHEN (lead(meta_data.effective_time) over (partition by meta_data.stream_id) IS NULL) THEN CONVERT("9999-12-31 23:59:59",DATETIME)
ELSE lead(meta_data.effective_time) over (partition by meta_data.stream_id) END AS end_date,
meta_data.stream_id,
meta_data.program_identifier,
meta_data.device_identifier,
meta_data.input_information,
meta_data.input_type,
meta_data.encoder_decoder,
meta_data.encoder_decoder_profile,
meta_data.chroma_subsampling,
meta_data.expected_frame_rate,
meta_data.frame_latency_millisec,
meta_data.horizontal_resolution_pixels,
meta_data.vertical_resolution_pixels
FROM
(SELECT
effective_time,
stream_id,
program_identifier,
device_identifier,
input_information,
input_type,
encoder_decoder,
encoder_decoder_profile,
chroma_subsampling,
expected_frame_rate,
frame_latency_millisec,
horizontal_resolution_pixels,
vertical_resolution_pixels
FROM
live_raw_meta_video_stream AS e
ORDER BY stream_id,effective_time
) AS meta_data)

-- Building live_monitor::pdt_live_raw_meta_video_stream in dev mode on instance 18a313debffdaa6ef246d2b1ae4aa1e0
ANALYZE insights.LC$KPV6V1561123494236_pdt_live_raw_meta_video_stream
-- finished pdt_live_raw_meta_video_stream => insights.LR$KPV6VJRQ61JA4AXU9YUM_pdt_live_raw_meta_video_stream
SELECT
DATE(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York')) AS `live_agg_video_stream.interval_start_date`,
organization.organization_name AS `live_agg_video_stream.organization`,
pdt_live_raw_meta_video_stream.program_identifier AS `pdt_live_raw_meta_video_stream.program_identifier`,
if(ISNULL(((COALESCE(COALESCE((0E0 + ( SUM(DISTINCT (CAST(FLOOR(COALESCE(live_agg_video_stream.sum_absolute_quality_score ,0)*(1000000*1.0)) AS DECIMAL(65,0))) + (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65))) ) - SUM(DISTINCT (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65)))) ) ) / (0E0 + (1000000*1.0)), 0), 0)) / nullif((COALESCE(COALESCE((0E0 + ( SUM(DISTINCT (CAST(FLOOR(COALESCE(live_agg_video_stream.frame_count_absolute_quality_score ,0)*(1000000*1.0)) AS DECIMAL(65,0))) + (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65))) ) - SUM(DISTINCT (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65)))) ) ) / (0E0 + (1000000*1.0)), 0), 0)),0))),((COALESCE(COALESCE((0E0 + ( SUM(DISTINCT (CAST(FLOOR(COALESCE(live_agg_video_stream.sum_no_reference_score ,0)*(1000000*1.0)) AS DECIMAL(65,0))) + (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65))) ) - SUM(DISTINCT (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65)))) ) ) / (0E0 + (1000000*1.0)), 0), 0)) / nullif((COALESCE(COALESCE((0E0 + ( SUM(DISTINCT (CAST(FLOOR(COALESCE(live_agg_video_stream.frame_count_no_reference_score ,0)*(1000000*1.0)) AS DECIMAL(65,0))) + (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65))) ) - SUM(DISTINCT (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65)))) ) ) / (0E0 + (1000000*1.0)), 0), 0)),0)),((COALESCE(COALESCE((0E0 + ( SUM(DISTINCT (CAST(FLOOR(COALESCE(live_agg_video_stream.sum_absolute_quality_score ,0)*(1000000*1.0)) AS DECIMAL(65,0))) + (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65))) ) - SUM(DISTINCT (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65)))) ) ) / (0E0 + (1000000*1.0)), 0), 0)) / nullif((COALESCE(COALESCE((0E0 + ( SUM(DISTINCT (CAST(FLOOR(COALESCE(live_agg_video_stream.frame_count_absolute_quality_score ,0)*(1000000*1.0)) AS DECIMAL(65,0))) + (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65))) ) - SUM(DISTINCT (CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ),1,16),16,10) AS DECIMAL(65)) *18446744073709551616 + CAST(CONV(SUBSTR(MD5(concat((CAST(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York') AS CHAR)),(CAST(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York') AS CHAR)),live_agg_video_stream.stream_id) ), 17, 16), 16, 10) AS DECIMAL(65)))) ) ) / (0E0 + (1000000*1.0)), 0), 0)),0))) AS `live_agg_video_stream.avg_quality_of_experience_score`
FROM insights.live_agg_video_stream AS live_agg_video_stream
INNER JOIN insights.dim_stream AS stream ON live_agg_video_stream.stream_id = stream.stream_id
INNER JOIN insights.dim_service AS service ON stream.service_id = service.service_id
INNER JOIN insights.dim_site AS site ON service.site_id = site.site_id
INNER JOIN insights.dim_organization AS organization ON site.organization_id = organization.organization_id
LEFT JOIN insights.LR$KPV6VJRQ61JA4AXU9YUM_pdt_live_raw_meta_video_stream AS pdt_live_raw_meta_video_stream ON live_agg_video_stream.stream_id = pdt_live_raw_meta_video_stream.stream_id
AND ((DATE_FORMAT(CONVERT_TZ(pdt_live_raw_meta_video_stream.start_date ,'UTC','America/New_York'),'%Y-%m-%d %H:%i')) <= (DATE_FORMAT(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York'),'%Y-%m-%d %H:%i')) AND
(DATE_FORMAT(CONVERT_TZ(pdt_live_raw_meta_video_stream.end_date ,'UTC','America/New_York'),'%Y-%m-%d %H:%i')) >= (DATE_FORMAT(CONVERT_TZ(live_agg_video_stream.interval_end ,'UTC','America/New_York'),'%Y-%m-%d %H:%i')))

GROUP BY 1,2,3
ORDER BY DATE(CONVERT_TZ(live_agg_video_stream.interval_start ,'UTC','America/New_York')) DESC
LIMIT 500

So the SQL error was really on the last sql_step not the first one as indicated in the error message. I suspect it is incorrectly relating the error to the first step in a multi step create process.

It also seems the SQL you see in the explore is not really the exact SQL that is going to be run. The generated table name is different in the sql steps and is actually correct in the final join to the PDT…the final join refers to the actual table name that is created. Very confusing.

Userlevel 7
Badge +1

That would be something we’d definitely want to fix up. I’ll test it out and see if I can reproduce the behavior.



I believe this happens due to the way PDTs are built— We build a PDT with one name so as not to disturb the other already-existing PDT while it’s building. Then, once the new one is complete, Looker drops+renames, so the final step will reference a differently named table than the initial ones.

Ok so I understand the PDT build and why it is done that way…I guess the only outstanding thing is the error message referring to the wrong step. Thk for your responses.

Reply