You are responsible for writing your company's ETL pipelines to run on an Apache Hadoop cluster. The pipeline will require some checkpointing and splitting pipelines. Which method should you use to write the pipelines?
Correct Answer: A
Question 147
You decided to use Cloud Datastore to ingest vehicle telemetry data in real time. You want to build a storage system that will account for the long-term data growth, while keeping the costs low. You also want to create snapshots of the data periodically, so that you can make a point-in-time (PIT) recovery, or clone a copy of the data for Cloud Datastore in a different environment. You want to archive these snapshots for a long time. Which two methods can accomplish this? Choose 2 answers.
Correct Answer: B,D
Question 148
You need to compose visualization for operations teams with the following requirements: Telemetry must include data from all 50,000 installations for the most recent 6 weeks (sampling once every minute) The report must not be more than 3 hours delayed from live data. The actionable report should only show suboptimal links. Most suboptimal links should be sorted to the top. Suboptimal links can be grouped and filtered by regional geography. User response time to load the report must be <5 seconds. You create a data source to store the last 6 weeks of data, and create visualizations that allow viewers to see multiple date ranges, distinct geographic regions, and unique installation types. You always show the latest data without any changes to your visualizations. You want to avoid creating and updating new visualizations each month. What should you do?
Correct Answer: B
Question 149
Your organization uses a multi-cloud data storage strategy, storing data in Cloud Storage, and data in Amazon Web Services' (AWS) S3 storage buckets. All data resides in US regions. You want to query up-to-date data by using BigQuery. regardless of which cloud the data is stored in. You need to allow users to query the tables from BigQuery without giving direct access to the data in the storage buckets What should you do?
Correct Answer: B
BigQuery Omni enables you to run BigQuery analytics directly on data stored in AWS S3 buckets without having to move or copy the data. This provides several benefits: Reduced Data Movement Costs: Eliminates the need to egress data from AWS, potentially saving significant costs. Real-Time Analytics: Allows you to query data in AWS S3 in real-time, providing up-to-date insights. Simplified Architecture: Reduces the complexity of managing data pipelines and ETL processes. Here's a breakdown of the steps involved in using BigQuery Omni: Set up a BigQuery Omni connection: This involves configuring the connection between your Google Cloud project and your AWS S3 bucket. This connection establishes the secure link for BigQuery to access the data in AWS S3. Create external tables: BigQuery external tables are a way to query data residing in external storage systems, such as AWS S3, without having to import the data into BigQuery. This enables you to directly query the data in its original location. Query the data using BigQuery: Once the external tables are created, you can use standard SQL queries to analyze the data stored in both Cloud Storage and AWS S3, just as if it were native BigQuery data. Why other options are not suitable: Option A: BigLake tables are designed for storing large volumes of structured data within BigQuery itself, not for directly querying data in external storage systems. Option C and D: While the Storage Transfer Service is useful for moving data between cloud providers, it introduces unnecessary data movement and latency if the goal is to simply query the data in its original location. Key Points: BigQuery Omni extends BigQuery's capabilities to analyze data stored in other cloud providers, such as AWS. External tables provide a way to query data in external storage systems without having to import it into BigQuery. By leveraging BigQuery Omni and external tables, you can efficiently and cost-effectively query data stored in multiple cloud environments using a single tool, BigQuery.
Question 150
You have a network of 1000 sensors. The sensors generate time series data: one metric per sensor per second, along with a timestamp. You already have 1 TB of data, and expect the data to grow by 1 GB every day You need to access this data in two ways. The first access pattern requires retrieving the metric from one specific sensor stored at a specific timestamp, with a median single-digit millisecond latency. The second access pattern requires running complex analytic queries on the data, including joins, once a day. How should you store this data?
Correct Answer: A
To store your data in a way that meets both access patterns, you should: A). Store your data in Bigtable Concatenate the sensor ID and timestamp and use it as the row key Perform an export to BigQuery every day. This option allows you to leverage the high performance and scalability of Bigtable for low-latency point queries on sensor data, as well as the powerful analytics capabilities of BigQuery for complex queries on large datasets. By using the sensor ID and timestamp as the row key, you can ensure that your data is sorted and distributed evenly across Bigtable nodes, and that you can easily retrieve the metric for a specific sensor and time. By performing an export to BigQuery every day, you can transfer your data to a columnar storage format that is optimized for analytical queries, and take advantage of BigQuery's features such as partitioning, clustering, and caching. B). Store your data in BigQuery Concatenate the sensor ID and timestamp. and use it as the primary key. This option is not optimal because BigQuery is not designed for low-latency point queries, and using a concatenated primary key may result in poor performance and high costs. BigQuery does not support primary keys natively, and you would have to use a unique constraint or a hash function to enforce uniqueness. Moreover, BigQuery charges by the amount of data scanned, so using a long and complex primary key may increase the query cost and complexity. C). Store your data in Bigtable Concatenate the sensor ID and metric, and use it as the row key Perform an export to BigQuery every day. This option is not optimal because using the sensor ID and metric as the row key may result in data skew and hotspots in Bigtable, as some sensors may generate more metrics than others, or some metrics may be more common than others. This may affect the performance and availability of Bigtable, as well as the efficiency of the export to BigQuery. D). Store your data in BigQuery. Use the metric as a primary key. This option is not optimal because using the metric as a primary key may result in data duplication and inconsistency in BigQuery, as multiple sensors may generate the same metric at different times, or the same sensor may generate different metrics at the same time. This may affect the accuracy and reliability of your analytical queries, as well as the query cost and complexity.
