Now, I deleted everything, created a new topic, created some new topic, when what ever I do, it all works, just the preview don't tell the truth sometime. while after calling some select * from [table] order by stamp desc, then it's also OK.
Gos knows if it will fail down again after days.......................?????????????
(actually everything's good, just use select count(*) will be able to trigger the batch and see the change.)
let's from beginning again:
new topic aaa + new db (with only 1 column) = works! added more columns, still works.
if you create a bigquery table which have already existed before, but was deleted already, then don't be surprised if you found there unknow records in the new created table!!!!
existing topic: pixel-raw-events, + new subscription for it: pixel_raw_events_sub, + new table pixel_raw_events -------> send event, no data appear in bq, -----------> goto subscription + do pull -------------->no message appear, but the data in bq suddenly appear! -------------> send event again, works!
is do pull the key which will make the data appear?-------no, it just need time, the case I met was around 3 minute. you can try if pull can make it happen faster? thought I think shouldn't.
if I swith idsync to send message to other topic, then switch back, should still work, right?----yes.
can I have multi subscriptions?---------yes, but need to prove when each have a job to consume, they will not interrupt each other.
I have two subscription: aaa_sub3 aaa_sub4.
make a job for aaa_sub3. and stream data to table aaa. to see if it can save message into it, should be no. (NO!, the new message is consumed after a while, but no record in aaa, seems the aaa table is not good anymore?)
make a job for aaa_sub4. and stream data to new table fff, this should be OK, then check if the other tables will be wake up for refresh?---------------before it was all ok, but this time the record didn't appear in fff, an no other table waked up :(, don't know why....I noticed that the message appear in subscription for a while, then it disppeared....while not as aspected appear in fff table, so where did it go?
where
is a new data base table necessary?
if there's alreay messages under a subscription, now create a new table for it, will not work?
wait for hours, then found that the pipeline do not works anymore, send event, nothing appears in BQ table, (also nothing displayed in subscription, that should mean the message is cosumed?
could it because that the pipeline went to sleep if there's no coming data?------No, it might actually works, just it do not refresh quickly.....because as I remember, the records number in pixel.pixel_raw_events was 10 this afternoon, I tried to send a new message to pixel_raw_events topic, didn't work at that time............!!!!! While when I check again at 23:30, it become 12 records!!!!!!(approved again at 2:03 4/18/2020)
beside the record count in pixel.raw.events changed, seems the other job all stopped working, so I'll write down current records number to compare after hours. to see if it's really because bigquery doesn't refresh quick if there's no operation for a long time.
aaa 0 rows
bbb 2 rows
ccc 20 rows
ddd 36 rows
eee: 5 rows
pixel_raw_events 12 rows
test 0 rows
I send at least 10 message to ddd, eee and pixel_raw_events!----wait for hours to see if it changes..------(YES, changed after hours, but pixel_raw_event is still 12, don't know why...then after more hours, the pixel_raw_event changed finally to 22.)
I created new subscription under topic aaa, then, found the releven tables(new created) no response, why???? --------（I'll leave these tables there to check it again tomorrow.....)
As a truth that the bigquery stop response quickly soon after the job created or event as soon as the job created? then how can we do test for an existing running job? (we send an request, then no response in the bigquery?????)
the second pipeline and saving to the same table? do NOT work.
same topic to an other new same structured table? -------do NOT work.
delete the first two job. could the last job work?-----------still NOT work, Maybe I should delete the last two jobs first to see if the first coudl start to work agian?...................or.......maybe the job will die if it idle too long time?...no...nuts!
delete all the job and subject, then created a new subscriptioin, work, it can receive message.
delete the message, and create a dataflow, use test table, not work.
Fuck up!
let's do it again, createa new topic bbb and a new table b, and create a new dataflow from bbbb to bbb- works, but take pretty long time.
created a second subscription bbb-s, it also got messages. now I delete this subscription to see if the other one will be affected
fffffuuuucccccckkkkk!!!!!, it's all of a sudden, the test table start to display 4 records!!!!!!4 old records!, because the new records should display data of good format!.....................while I tried to create a new pipline to writ info into it, no display.......how can I know if it will display some time in futer?
I created a new table ccc, and created a new pipline bbb-ccc. it works
delete the bbb-s, to see if bbb-ccc still works. it works
Fuck up!!!!!!!
it doesn't work again!!!!!!! I created a new topic aaa, a new table aaa, then createa default dataflow, it stop working.
I first find out with old topic, it works with a subscription, bug do not persist into new created raw_events table neither do test table....