Is there a way to do this? Thank you in advance. 1 Answer Sorted by: 0 this regex will select the table or schema.table from stlquery.querytxt select slc.query as queryid, trim (slc.filename) as file, slc.curtime as updated, slc.linesscanned as rows, sq.querytxt as querytxt, REGEXPREPLACE (LOWER (sq. The dynamic file name shall change to August2021_Batch01 & August2021_Batch02 next month and so forth.
The parquet files are created using pandas as part of a python ETL script. Batch number ranges 1-6.Ĭurrently, here is what I have which is not efficient: COPY tbl_name ( column_name1, column_name2, column_name3 )įROM 'S3://bucket_name/folder_name/Static_File_Label_July2021_Batch01.CSV'ĬREDENTIALS 'aws_access_key_id = xxx aws_secret_access_key = xxxxx'ĬOPY tbl_name ( column_name1, column_name2, column_name3 )įROM 'S3://bucket_name/folder_name/Static_File_Label_July2021_Batch02.CSV' I am copying multiple parquet files from s3 to redshift in parallel using the copy command.
#REDSHIFT COPY COMMAND TRUNCATION CODE#
I'd like to modify the below COPY command to change the file naming in the S3 directory dynamically so I won't have to hard code the Month Name and YYYY and batch number.
File names follow a standard naming convention as " file_label_MonthNameYYYY_Batch01.CSV". This section presents the required COPY command parameters and groups the optional parameters by function. I have job in Redshift that is responsible for pulling 6 files every month from S3. Amazon Redshift extends the functionality of the COPY command to enable you to load data in several data formats from multiple data sources, control access to load data, manage data transformations, and manage the load operation.
0 kommentar(er)
