Google Cloud Platform入门 Getting started with Google Cloud Platform

We’re using the built-in temperature sensor from ESP32 to sent data every 5 seconds to a pub/sub service in GCP. This data will be processed by a Go service and sent to Prometheus, and we’re using Grafana to visualize this data.

我们使用ESP32内置的温度传感器每5秒将数据发送到GCP中的发布/订阅服务。 这些数据将由Go服务处理并发送到Prometheus ，我们正在使用Grafana可视化此数据。

Grafana is a visualization tool. Prometheus is a data source for Grafana that collects the data in time series and displays in a way that Grafana understands. Since Prometheus can’t collect data directly to pub/sub, we need a third service to send it to Prometheus.

Grafana是一种可视化工具。 Prometheus是Grafana的数据源，它按时间序列收集数据并以Grafana理解的方式显示。 由于Prometheus无法直接将数据收集到pub / sub，因此我们需要第三项服务将其发送给Prometheus。

I made a Github repository with all the codes used and instructions in this article, fell free to check it out.

我创建了一个Github存储库，其中包含本文中使用的所有代码和说明，可以随意查看。

So let’s get our hands dirty!

因此，让我们动手吧！

Google Cloud Platform入门 Getting started with Google Cloud Platform

On Google Cloud, we’ll be using Core IoT to manage our devices, pub/sub as messaging system and Google Run to host our containers.

在Google Cloud上，我们将使用Core IoT管理设备，将发布/订阅作为消息传递系统，并在Google Run中托管容器。

First, let’s set up our project. You’ll need a Google account and a credit card, but don’t worry you won’t be charged for anything (if you don’t do some heavy work), your free trial lasts for 3 months and you have US$300 to spend in any Google Cloud service. But you can always keep an eye on your billing board to not have any surprises on your credit card.

首先，让我们建立项目。 您需要一个Google帐户和一张信用卡，但是不用担心，您无需支付任何费用(如果您不做繁重的工作)，免费试用期为3个月，您有300美元在任何Google Cloud服务中花费。 但是，您始终可以随时注意帐单上的内容，以确保信用卡不会出现任何意外。

I’ll use the Google Cloud CLI to set up our environment but feel free to use the web console. Install it for your operational system.

我将使用Google Cloud CLI设置环境，但可以随时使用Web控制台。 为您的操作系统安装它。

To make things easier, you can export this environment variables and just paste the commands from this tutorial (choose your own names):

为了简化操作，您可以导出此环境变量，然后仅粘贴本教程中的命令(选择您自己的名称)：

export PROJECT_ID=export REGION=export TOPIC_ID=export SUBSCRIPTION=export REGISTRY=export DEVICE_ID=export USER_NAME=export IMAGE_NAME=export SERVICE_NAME=

export PROJECT_ID=temperature-grafanaexport PROJECT_ID=temperature-grafana

To start, log in with your Google account on CLI, create a project, and select the project created. Open a terminal and type the following commands:

首先，在CLI上使用您的Google帐户登录，创建一个项目，然后选择创建的项目。 打开终端并输入以下命令：

$ gcloud auth login
$ gcloud projects create $PROJECT_ID
$ gcloud config set project $PROJECT_ID

You can check your project dashboard, and if everything goes well, you should see your project there.

您可以检查项目仪表板，如果一切顺利，则应该在此处看到您的项目。

Now let's enable pub/sub and IoT Core services in our project. But before that, you'll need to enable the billing into your project. To do that, run the following command and continue to the browser and link a profile:

现在，让我们在项目中启用发布/订阅和IoT核心服务。 但是在此之前，您需要在您的项目中启用计费功能。 为此，请运行以下命令并继续浏览器并链接配置文件：

$ open "https://console.cloud.google.com/billing/linkedaccount?project=$PROJECT_ID"
$ gcloud services enable cloudiot.googleapis.com pubsub.googleapis.com

We also need to permit IoT Core to publish into pub/sub service. Since IoT Core is responsible for our devices, and they don’t need to subscribe to any topic, we’re giving them just the publishing role.

我们还需要允许IoT Core发布到发布/订阅服务中。 由于IoT核心负责我们的设备，并且它们不需要订阅任何主题，因此我们只赋予他们发布角色。

$ gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:cloud-iot@system.gserviceaccount.com \
--role=roles/pubsub.publisher

Choose a region here. I’m using us-central1, but pick the one that better suits you. We also need a pub/sub topic, a subscription, and a registry.

在这里选择一个地区。 我正在使用us-central1 ，但请选择一个更适合您的。 我们还需要一个pub / sub主题，一个subscription和一个注册表。

On MQTT, pub/sub works like an Instagram/Twitter hashtag, where you can publish a post using a hashtag and who is following (or subscribed) to that hashtag, we'll see your post. The same works for MQTT, but the hashtag is the topic, the photo is the message and the people following that topic is the subscription.

在MQTT上，发布/订阅的工作方式类似于Instagram / Twitter的标签，您可以在其中使用标签来发布帖子，并且关注(或订阅)该标签的人，我们将看到您的帖子。 MQTT的工作原理相同，但主题标签是主题标签，照片是消息，主题之后的人员是订阅。

A registry is like a bucket for our IoT devices. It allows us to group devices and set properties that they all share, such as connection protocol, data storage location, and Cloud pub/sub topics.

注册表就像我们的物联网设备的存储桶。 它使我们可以对设备进行分组并设置它们都共享的属性，例如连接协议，数据存储位置和Cloud pub / sub主题。

Follow these commands:

请遵循以下命令：

$ gcloud pubsub topics create $TOPIC_ID
$ gcloud pubsub subscriptions create --topic $TOPIC_ID $SUBSCRIPTION
$ gcloud iot registries create $REGISTRY \
--region=$REGION \
--event-notification-config=topic=temperature-topic \
--enable-mqtt-config --enable-http-config

You can check your registries, and if everything goes well, you should see your registry with your topic and subscription there.

您可以检查注册表，如果一切顺利，则应该在其中看到带有主题和订阅的注册表。

设置ESP32 Setting up ESP32

We’ll be using the Espressif micro-controller ESP32 for its WiFi and a built-in temperature sensor. Also, I’m using the Arduino IDE, so make sure you have it installed and set up for ESP32 usage, if you need some help you can follow this tutorial:

我们将使用Espressif微控制器ESP32的WiFi和内置温度传感器。 另外，我使用的是Arduino IDE，因此请确保已安装并设置为ESP32使用，如果需要帮助，可以按照本教程进行操作：

We need to generate an Elliptic Curve (EC) ES256 private/public key pair for our device authentication. Make sure to generate them into a “safe place”:

我们需要为设备认证生成一个椭圆曲线(EC)ES256私钥/公钥对。 确保将它们生成到“安全的地方”：

$ openssl ecparam -genkey -name prime256v1 -noout -out ec_private.pem$ openssl ec -in ec_private.pem -pubout -out ec_public.pem

Now we have to register our device into Core IoT, so run the following commands:

现在我们必须将设备注册到Core IoT中，因此运行以下命令：

$ gcloud iot devices create $DEVICE_ID \
  --region=$REGION \
  --registry=$REGISTRY \
  --public-key="path=./ec_public.pem,type=es256"

Install “Google Cloud IoT Core JWT” and lwmMQTT from Joel Garhwller libraries on your Arduino IDE. They’re responsible for connecting, authenticating, and sending messages to GCP.

从Arduino IDE上的Joel Garhwller库安装“ Google Cloud IoT Core JWT”和lwmMQTT。 他们负责连接，验证消息并将消息发送到GCP。

Now let's use the library code example for ESP32-lwmqtt:

现在使用ESP32-lwmqtt的库代码示例：

On ciotc_config.h, set your WiFi network and credentials:

在ciotc_config.h上，设置您的WiFi网络和凭据：

// Wifi network details
const char *ssid = "SSID";
const char *password = "PASSWORD";// Cloud iot details
const char *project_id = "project-id";
const char *location = "us-central1";
const char *registry_id = "my-registry";
const char *device_id = "my-esp32-device";

To get your private_key_str, run the following command at the same directory where you saved your public/private keys and paste the "priv" it into the code:

要获取您的private_key_str ，请在保存公钥/私钥的同一目录中运行以下命令，并将“ priv ”粘贴到代码中：

$ openssl ec -in ec_private.pem -noout -text

PS: The key length should be 32 pairs of hex digits. If your private key is bigger, remove the “00:” and if its smaller add “00:”. It should look like this:

PS：密钥长度应为32对十六进制数字。 如果私钥较大，则删除“ 00：”，如果较小，则添加“ 00：”。 它看起来应该像这样：

You’ll need to set up your root_cert as well. Do the same steps as previously:

您还需要设置root_cert 。 执行与之前相同的步骤：

$ openssl s_client -showcerts -connect mqtt.googleapis.com:8883

It should look something like this:

它看起来应该像这样：

On Esp32-lwmqtt.ino file, let's do some changes to get the ESP32 temperature. This is how our code looks like:

在Esp32-lwmqtt.ino文件上，进行一些更改以获取ESP32温度。 我们的代码如下所示：

#include "esp32-mqtt.h"


#ifdef __cplusplus
extern "C" {
#endif
  uint8_t temprature_sens_read();
#ifdef __cplusplus
}
#endif


uint8_t temprature_sens_read();


void setup() {
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);
  setupCloudIoT();
}


unsigned long lastMillis = 0;
void loop() {
  mqtt->loop();
  delay(10);


  if (!mqttClient->connected()) {
    connect();
  }


  
  if (millis() - lastMillis > 5000) {
    lastMillis = millis();


    const float temperature = (temprature_sens_read() - 32) / 1.8;


    String payload =
      String("{\"temperature\":") + String(temperature) + String("}");
    publishTelemetry(payload);
    
  }
}

Now plug in your ESP32 into the USB and upload the sketch. Open the console, set it to 115200 baud rate and the logs should look like this:

现在将ESP32插入USB并上传草图。 打开控制台，将其设置为115200波特率，日志应如下所示：

Now if you check your pub/sub topics, you may see something like this:

现在，如果您查看pub / sub主题，您可能会看到类似以下内容：

Golang指标转换器 Golang Metrics Converter

We need someone subscribed to the topic where our device is publishing and send those messages to Prometheus. So let’s create a service with Golang and put it into a container. First, we need a service-account and give it the pub/sub subscribe role:

我们需要有人订阅我们设备发布的主题，然后将这些消息发送给Prometheus。 因此，让我们使用Golang创建服务并将其放入容器中。 首先，我们需要一个服务帐户并将其授予pub / sub订阅角色：

$ gcloud iam service-accounts create $USER_NAME$ gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:$USER_NAME@$PROJECT_ID.iam.gserviceaccount.com \
--role=roles/pubsub.subscriber

There are several ways to our service authenticate into GCP, so let’s use the JSON file method. So let’s generate a file with credentials for the service account we just created and export its path to an environment variable called GOOGLE_APPLICATION_CREDENTIALS:

我们的服务可以通过多种方式向GCP进行身份验证，因此让我们使用JSON文件方法。 因此，让我们为刚刚创建的服务帐户生成一个带有凭证的文件，并将其路径导出到名为GOOGLE_APPLICATION_CREDENTIALS的环境变量中：

$ gcloud iam service-accounts keys create resources/service-account-key.json --iam-account $USER_NAME@$PROJECT_ID.iam.gserviceaccount.com

$ export GOOGLE_APPLICATION_CREDENTIALS=$CREDENTIAL_PATH

export GOOGLE_APPLICATION_CREDENTIALS=/home/leo/resources/service-account-key.jsonexport GOOGLE_APPLICATION_CREDENTIALS=/home/leo/resources/service-account-key.json

Now let's create our Golang service:

现在让我们创建我们的Golang服务：

package main


import (
	"context"
	"encoding/json"
	"log"
	"net/http"
	"strings"
	"sync"


	"cloud.google.com/go/pubsub"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)


// Payload receveid from pub/sub
type Payload struct {
	Temperature float64 `json:"temperature"`
}


// Gauge metric temperature for Prometheus
var (
	temperatureGauge = promauto.NewGauge(prometheus.GaugeOpts{
		Name: "temperature_gauge",
		Help: "Temperature in °C",
	})
)


// Pulling messages from pub/sub
func pullMessages() {
	log.Println("STARTED PULLING MESSAGES")


	ctx := context.Background()


	// Set your $PROJECT_ID
	client, err := pubsub.NewClient(ctx, "temperature-grafana")
	if err != nil {
		log.Fatal(err)
	}


	// Set your $SUBSCRIPTION
	subID := "temperature-subscription"
	var mu sync.Mutex


	sub := client.Subscription(subID)
	cctx, cancel := context.WithCancel(ctx)
	err = sub.Receive(cctx, func(ctx context.Context, msg *pubsub.Message) {
		mu.Lock()
		defer mu.Unlock()


		log.Print("Got message: " + string(msg.Data))


		if !containsInitPayload(string(msg.Data)) {
			var t Payload


			err := json.Unmarshal(msg.Data, &t)
			if err != nil {
				panic(err)
			}


			temperatureGauge.Set(t.Temperature)
		}


		msg.Ack()
	})
	if err != nil {
		cancel()
		log.Fatal(err)
	}
	cancel()
}


func containsInitPayload(payload string) bool {
	if strings.Contains(payload, "esp32-connected") {
		return true
	}
	return false
}


func main() {
	go pullMessages()


	log.Println("STARTED PROMETHEUS")


	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":2112", nil)
}

And a Dockerfile for it:

还有一个Dockerfile：

FROM golang


COPY . /app
WORKDIR /app


ENV GOOGLE_APPLICATION_CREDENTIALS=/app/resources/service-account-key.json


RUN go mod download


CMD ["go", "run", "pubsub.go"]

Google Cloud RunGoogle Cloud Run

Cloud Run it's an awesome fully managed compute platform for deploying and scaling containerized applications quickly and securely. If you’re new to it, check this video

Cloud Run是一个很棒的完全托管的计算平台，可快速安全地部署和扩展容器化的应用程序。 如果您是新手，请观看此视频

First, we need to enable Cloud Build, Cloud Run, and Container Registry in our project:

首先，我们需要在我们的项目中启用Cloud Build，Cloud Run和Container Registry：

$ gcloud services enable cloudbuild.googleapis.com run.googleapis.com containerregistry.googleapis.com

Now let’s build and push our Golang service Docker image to the Cloud Build:

现在，让我们构建并将Golang服务Docker映像推送到Cloud Build：

$ gcloud builds submit --tag gcr.io/$PROJECT_ID/$IMAGE_NAME

To deploy it, run this command:

要部署它，请运行以下命令：

$ gcloud run deploy $SERVICE_NAME --image gcr.io/$PROJECT_ID/ $IMAGE_NAME --region us-central1 --plataform managed --allow-authenticated --port 2112

GCP will generate an URL for your container, copy it. You can get it on your terminal or accessing your project on your project Cloud Run page.

GCP将为您的容器生成一个URL，然后将其复制。 您可以在终端上获取它，也可以在项目“云运行”页面上访问项目。

Now paste it on your prometheus.yml file, on your targets (remove the https://):

现在将其粘贴到您的目标的prometheus.yml文件中(删除https：//)：

global:
  scrape_interval:     10s
  evaluation_interval: 10s
  external_labels:
    monitor: 'codelab-monitor'


scrape_configs:
  - job_name: 'temperature'
    scrape_interval: 5s
    static_configs:
    - targets:
      - 'temperature-grafana-utsma6q3sq-uc.a.run.app' # Your project URL

Since we can't deploy existing images from Docker Hub to Cloud Run, we need to make a custom Docker image for Prometheus and Grafana, then deploy them. Let's deploy a Prometheus container:

由于我们无法将现有映像从Docker Hub部署到Cloud Run，因此我们需要为Prometheus和Grafana创建自定义Docker映像，然后进行部署。 让我们部署一个Prometheus容器：

FROM prom/prometheus
ADD ./prometheus.yml /etc/prometheus/prometheus.yml
EXPOSE 9090

Building and submitting to production:

建立并提交生产：

$ gcloud builds submit --tag gcr.io/$PROJECT_ID/prometheus .
$ gcloud run deploy prometheus --image gcr.io/$PROJECT_ID/prometheus --region $REGION --platform managed --allow-unauthenticated --port 9090

Make sure to save the URL generated. Same thing for Grafana:

确保保存生成的URL。 Grafana也是这样：

FROM grafana/grafana
EXPOSE 3000
ENTRYPOINT [ "/run.sh" ]

Building and submitting to production:

建立并提交生产：

$ gcloud builds submit --tag gcr.io/$PROJECT_ID/grafana .$ gcloud run deploy grafana --image gcr.io/$PROJECT_ID/grafana --region $REGION --platform managed --allow-unauthenticated --port 3000

Now you can access your Grafana board through the generated URL. You can log in with the admin login (default user: admin, pass: admin, make sure to change that).

现在，您可以通过生成的URL访问Grafana板。 您可以使用admin登录名登录(默认用户：admin，pass：admin ，请确保进行更改)。

Now we have to set up Grafana to listen to our Prometheus. After logging in, go to "Data Source" on the right menu bar, click on "Add data source" and select Prometheus.

现在，我们必须设置Grafana来收听我们的Prometheus。 在登录后，进入“数据源”右侧的菜单栏上，单击“添加数据源”，然后选择普罗米修斯。

On the Prometheus data source page, paste the URL to your Prometheus instance on the HTTP > URL and hit "save & test".

在Prometheus数据源页面上，将该URL粘贴到HTTP> URL上的Prometheus实例中，然后单击“保存并测试”。

On my Github repository, there’s a JSON file that will import a Grafana Dashboard. Feel free to use it or create your own. If you looking into creating your dashboards, Grafana uses PromQL for querying metrics data, take a look into its documentation for more information.

在我的Github存储库中，有一个JSON文件将导入Grafana仪表板。 随意使用它或创建自己的。 如果您想创建仪表板，则Grafana使用PromQL查询指标数据，请查看其文档以获取更多信息。

To import my dashboard, go to the right menu bar, then to Create and Import. Paste the JSON content into the text box and hit load.

要导入我的仪表板，请转到右侧的菜单栏，然后​​转到“创建并导入”。 将JSON内容粘贴到文本框中，然后点击load 。

Select Prometheus as your data source and boom, you should a dashboard like this one:

选择Prometheus作为您的数据源和繁荣，您应该像这样一个仪表板：