Data Lakes Daily

Different partitioning strategies within BigQuery have various pros and cons. A typical strategy for partitioning is by DAY and partitioned on the field _PARTITIONTIME. For context, _PARTITIONTIME is a pseudo column that contains the UTC day that the partitioned data was loaded. In many cases, this is a good default. However, there may be cases where you want to optimize a table beyond Google defaults. For example, if you have a high volume of loads for a common data source, using a different strategy can lower query costs. Be aware that some partitioning strategies, while improving costs, may increase the complexity of your queries. There is no universal approach for all use cases. In this post, we detail how to migrate from one type of partitioned table to another. Before proceeding, please review the documentation from Google on the topic: https://cloud.google.com/bigquery/docs/partitioned-tables Convert Partitioning On An Existing BigQuery Table In our example, we will show y

Timezones are a confusing topic when it comes to a global marketplace like Amazon. For example, a customer in New York places an order on March 11, 2021, at 1 AM (EST), but it is March 10, 2021, at 10 PM (PST) in Seattle. What when did the order occur? March 11th or March 10th? Another wrinkle is that Amazon defaults to using a Coordinated Universal Time (UTC) standard in its systems. In our example above, the timestamp of the order would be March 11th, 2021, at 8 AM! Seller Central User Interface So, where does a chunk of the confusion arise? Amazon Seller Central. The Seller Central interface displays information in the Pacific timezone (PST). However, Amazon stores information in its systems using UTC. Per Amazon: “Amazon Marketplace Web Service (Amazon MWS) formats date- and time-related data similar to the RFC 3339 standard, which defines a subset profile of ISO 8601 for use in Internet protocols and standards. This section will clarify just how you should format and use d