Notes
Notes - notes.io |
415,95,36,2020-03-10,sddsd
261,98,14,2020-04-10,dfgfsdkk;
301,56,10,2020-06-21,fdgfdsf
413,95,36,2020-04-11,sddsd
262,97,15,2020-04-13,dfgfgg
300,50,16,2021-06-24,fdgfdsf
304,65,33,2021-07-15,fdssf
305,34,23,2020-02-11,fddassf
314,15,13,2021-01-26,fjhisa
309,64,19,2020-09-25,ioihjn
and the schema for that would be in json..
below it is:
{
"BigQuery Schema": [{
"name": "order_no",
"type": "INTEGER"
},
{
"name": "item_no",
"type": "INTEGER"
},
{
"name": "purchased_quantity",
"type": "INTEGER"
},
{
"name": "purchased_date",
"type": "DATE"
}
]
}
so basically I do not want that last column which has garbage value and want only first 4 columns with the respective heading.... So what I did in GCP I used this csv as input and used a pre built template for dataflow to write to BigQuery and the transformation or the UDF function which i applied was this
function transform(line) {
var values = line.split(',');
var obj = new Object();
obj.order_no = values[0];
obj.item_no = values[1];
obj.purchased_quantity = values[2];
obj.purchased_date = values[3];
var jsonString = JSON.stringify(obj);
return jsonString;
}
So now what the requirement is that I have to use dataflow and write a python script to do the same and want to upload it as a template in GCS and then I would use that template to run the batch job in dataflow... So what are the steps required give me everything in detail and also the script and the necessary steps
|
|
Notes.io is a web-based application for taking notes. You can take your notes and share with others people. If you like taking long notes, notes.io is designed for you. To date, over 8,000,000,000 notes created and continuing...
With notes.io;
- * You can take a note from anywhere and any device with internet connection.
- * You can share the notes in social platforms (YouTube, Facebook, Twitter, instagram etc.).
- * You can quickly share your contents without website, blog and e-mail.
- * You don't need to create any Account to share a note. As you wish you can use quick, easy and best shortened notes with sms, websites, e-mail, or messaging services (WhatsApp, iMessage, Telegram, Signal).
- * Notes.io has fabulous infrastructure design for a short link and allows you to share the note as an easy and understandable link.
Fast: Notes.io is built for speed and performance. You can take a notes quickly and browse your archive.
Easy: Notes.io doesn’t require installation. Just write and share note!
Short: Notes.io’s url just 8 character. You’ll get shorten link of your note when you want to share. (Ex: notes.io/q )
Free: Notes.io works for 12 years and has been free since the day it was started.
You immediately create your first note and start sharing with the ones you wish. If you want to contact us, you can use the following communication channels;
Email: [email protected]
Twitter: http://twitter.com/notesio
Instagram: http://instagram.com/notes.io
Facebook: http://facebook.com/notesio
Regards;
Notes.io Team
