google cloud platform – Fastest way to ingest data from BigQuery to PubSub

We also have a need to get data from BigQuery into PubSub and we do so using Dataflow. I’ve just looked at one of the jobs we ran today and we loaded 3.4million rows in about 5 minutes (so ~11000 rows per second). Our Dataflow jobs are written in java but you could write them … Read more

Snowflake taps Python to take on Teradata, Google BigQuery, and Amazon Redshift

Cloud-based data warehouse company Snowflake on Tuesday at its annual Snowflake Summit introduced a new set of tools and integrations to take on rival firms such as Teradata, and services such as Google BigQuery, and Amazon Redshift. The new capabilities, which include data access tools and support for Python on the company’s Snowpark application development … Read more

Snowflake taps Python to take on Teradata, Google BigQuery, and AWS Redshift

Cloud-based data warehouse company Snowflake on Tuesday at its annual Snowflake Summit introduced a new set of tools and integrations to take on rival firms such as Teradata, and services such as Google BigQuery, and Amazon Redshift. The new capabilities, which include data access tools and support for Python on the company’s Snowpark application development … Read more

google bigquery – Get a rolling order count into session data

A couple of ways.Check which runs faster if processing time is an issue.Both use 2 window functions but #2 uses one on a much smaller table, so it should run faster. #1 ### create sample table (helps to introduce these in your questions) WITH base AS ( SELECT ‘A’ AS client_id, 1 AS session_id, ‘pv’ … Read more

best way to append +30 tables using sql bigquery

I have 30+ tables, all with different #’s of columns. I’m hoping to append all the tables together into 1 single table that would share the same column names i’m having trouble with the query below. I’ve essentially selected the column names i want (but not all appear in each table) SELECT vertical,opportunity_name, project_name_, campaign_id, … Read more