45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例

一瓢一瓢的饮 alanchan2023-11-30 18:10

1、Flink 部署、概念介绍、source、transformation、sink使用示例、四大基石介绍和示例等系列综合文章链接

13、Flink 的table api与sql的基本概念、通用api介绍及入门示例
14、Flink 的table api与sql之数据类型: 内置数据类型以及它们的属性
15、Flink 的table api与sql之流式概念-详解的介绍了动态表、时间属性配置（如何处理更新结果）、时态表、流上的join、流上的确定性以及查询配置
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及FileSystem示例（1）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Elasticsearch示例（2）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Apache Kafka示例（3）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及JDBC示例（4）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Apache Hive示例（6）
17、Flink 之Table API: Table API 支持的操作（1）
17、Flink 之Table API: Table API 支持的操作（2）
18、Flink的SQL 支持的操作和语法
19、Flink 的Table API 和 SQL 中的内置函数及示例（1）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（2）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（3）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（4）
20、Flink SQL之SQL Client: 不用编写代码就可以尝试 Flink SQL，可以直接提交 SQL 任务到集群上
21、Flink 的table API与DataStream API 集成（1）- 介绍及入门示例、集成说明
21、Flink 的table API与DataStream API 集成（2）- 批处理模式和inser-only流处理
21、Flink 的table API与DataStream API 集成（3）- changelog流处理、管道示例、类型转换和老版本转换示例
21、Flink 的table API与DataStream API 集成（完整版）
22、Flink 的table api与sql之创建表的DDL
24、Flink 的table api与sql之Catalogs（介绍、类型、java api和sql实现ddl、java api和sql操作catalog）-1
24、Flink 的table api与sql之Catalogs（java api操作数据库、表）-2
24、Flink 的table api与sql之Catalogs（java api操作视图）-3
24、Flink 的table api与sql之Catalogs（java api操作分区与函数）-4
25、Flink 的table api与sql之函数(自定义函数示例)
26、Flink 的SQL之概览与入门示例
27、Flink 的SQL之SELECT (select、where、distinct、order by、limit、集合操作和去重)介绍及详细示例（1）
27、Flink 的SQL之SELECT (SQL Hints 和 Joins)介绍及详细示例（2）
27、Flink 的SQL之SELECT (窗口函数)介绍及详细示例（3）
27、Flink 的SQL之SELECT (窗口聚合)介绍及详细示例（4）
27、Flink 的SQL之SELECT (Group Aggregation分组聚合、Over Aggregation Over聚合 和 Window Join 窗口关联)介绍及详细示例（5）
27、Flink 的SQL之SELECT (Top-N、Window Top-N 窗口 Top-N 和 Window Deduplication 窗口去重)介绍及详细示例（6）
27、Flink 的SQL之SELECT (Pattern Recognition 模式检测)介绍及详细示例（7）
28、Flink 的SQL之DROP 、ALTER 、INSERT 、ANALYZE 语句
29、Flink SQL之DESCRIBE、EXPLAIN、USE、SHOW、LOAD、UNLOAD、SET、RESET、JAR、JOB Statements、UPDATE、DELETE（1）
29、Flink SQL之DESCRIBE、EXPLAIN、USE、SHOW、LOAD、UNLOAD、SET、RESET、JAR、JOB Statements、UPDATE、DELETE（2）
30、Flink SQL之SQL 客户端（通过kafka和filesystem的例子介绍了配置文件使用-表、视图等）
31、Flink的SQL Gateway介绍及示例
32、Flink table api和SQL 之用户自定义 Sources & Sinks实现及详细示例
33、Flink 的Table API 和 SQL 中的时区
35、Flink 的 Formats 之CSV 和 JSON Format
36、Flink 的 Formats 之Parquet 和 Orc Format
41、Flink之Hive 方言介绍及详细示例
40、Flink 的Apache Kafka connector（kafka source的介绍及使用示例）-1
40、Flink 的Apache Kafka connector（kafka sink的介绍及使用示例）-2
40、Flink 的Apache Kafka connector（kafka source 和sink 说明及使用示例） 完整版
42、Flink 的table api与sql之Hive Catalog
43、Flink之Hive 读写及详细验证示例
44、Flink之module模块介绍及使用示例和Flink SQL使用hive内置函数及自定义函数详细示例--网上有些说法好像是错误的
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版
46、Flink 的table api与sql之配项列表及示例

文章目录

本文简单的介绍了Flink 的指标体系的第一部分，即指标类型以及四种类型的代码实现示例。

本专题分为三部分，即：
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版

本文依赖nc能正常使用。

本文分为5个部分，即指标分类、计数器、gauge、histogram和meter四个指标的代码实现。

本文的示例是在Flink 1.17版本中运行。

Flink暴露了一个度量系统，允许收集度量并将其公开给外部系统。

本文涉及的maven依赖

xml 复制代码 
	<properties>
		<encoding>UTF-8</encoding>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		<java.version>1.8</java.version>
		<scala.version>2.12</scala.version>
		<flink.version>1.17.0</flink.version>
	</properties>

	<dependencies>
		<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-clients</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-streaming-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-csv</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-json</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>

		<!-- flink连接器 -->
		<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-connector-kafka</artifactId>
			<version>${flink.version}</version>
		</dependency>
	</dependencies>

1、Registering metrics 注册指标

通过调用getRuntimeContext().getMetricGroup()，您可以从任何扩展RichFunction的用户函数访问度量系统。此方法返回一个MetricGroup对象，您可以在该对象上创建和注册新度量。

1）、指标类型

Flink支持计数器、仪表盘、柱状图和计量表。Counters, Gauges, Histograms and Meters.

2）、计数器

计数器是用来统计数量的。当前值可以是in-或使用 inc()/inc(long n)或dec()/dec(long n)增减。您可以通过调用MetricGroup上的 counter(String name)来创建和注册计数器。

本示例提供了多种实现方式，供参考。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsDemo {

//	public class LineMapper extends RichMapFunction<String, String> {
//		private transient Counter counter;
//
//		@Override
//		public void open(Configuration config) {
//			this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter");
//		}
//
//		@Override
//		public String map(String value) throws Exception {
//			this.counter.inc();
//			return value;
//		}
//	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		DataStream<Tuple2<String, Integer>> result = lines.flatMap(new FlatMapFunction<String, String>() {
			@Override
			public void flatMap(String value, Collector<String> out) throws Exception {
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(word);
				}
			}
		}).map(new MapFunction<String, Tuple2<String, Integer>>() {

			@Override
			public Tuple2<String, Integer> map(String value) throws Exception {
				return Tuple2.of(value, 1);
			}
		}).keyBy(t -> t.f0).sum(1);

//		SingleOutputStreamOperator<Tuple2<Integer, Integer>> result1 = lines.map(new RichMapFunction<String, Tuple2<Integer, Integer>>() {
//
//			@Override
//			public Tuple2<Integer, Integer> map(String value) throws Exception {
//				int subTaskId = getRuntimeContext().getIndexOfThisSubtask();// 子任务id/分区编号
//				return new Tuple2(subTaskId, 1);
//			}
//			// 按照子任务id/分区编号分组,并统计每个子任务/分区中有几个元素
//		}).keyBy(t -> t.f0).sum(1);

		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, Long, Integer> 输入的字符串，行数，统计单词的总数
		DataStream<Tuple3<String, Long, Integer>> result2 = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, Long>>() {
//			private transient Counter counter;
			private long result2LineCounter = 0;

			@Override
			public void open(Configuration config) {
//				this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter:");
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("result2LineCounter:").getCount();
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, Long>> out) throws Exception {
//				this.counter.inc();
				result2LineCounter++;
				System.out.println("计数器行数：" + result2LineCounter);
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, result2LineCounter));
				}
			}

		}).map(new MapFunction<Tuple2<String, Long>, Tuple3<String, Long, Integer>>() {

			@Override
			public Tuple3<String, Long, Integer> map(Tuple2<String, Long> value) throws Exception {
//				Tuple3<String, Long, Integer> t = Tuple3.of(value.f0, value.f1, 1);
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");
		result2.print("result2:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();
//		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//		env.setParallelism(1);
//		DataStream<String> input = env.fromElements("a", "b", "c", "a", "b", "c");
//
//		input.keyBy(value -> value).map(new RichMapFunction<String, String>() {
//			private long count = 0;
//
//			@Override
//			public void open(Configuration parameters) throws Exception {
                super.open(parameters);
//				count = getRuntimeContext().getMetricGroup().counter("myCounter").getCount();
//			}
//
//			@Override
//			public String map(String value) throws Exception {
//				count++;
//				return value + ": " + count;
//			}
//		}).print();
//
//		env.execute("Flink Count Counter Example");
	}

}
///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1
result:> (hello,1)
result2:> (hello,1,1)
result:> (123,1)
result2:> (123,1,1)
计数器行数：2
result2:> (alan,2,1)
result:> (alan,1)
result2:> (flink,2,1)
result:> (flink,1)
result2:> (good,2,1)
result:> (good,1)
计数器行数：3
result:> (alan_chan,1)
result2:> (alan_chan,3,1)
result:> (hi,1)
result2:> (hi,3,1)
result:> (flink,2)
result2:> (flink,2,2)

或者，您也可以使用自己的Counter实现：

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsDemo {

	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);

		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// Tuple3<String, Long, Integer> 输入的字符串，行数，统计单词的总数
		DataStream<Tuple3<String, Long, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, Long>>() {
			private transient Counter counter;
			@Override
			public void open(Configuration config) {
						this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter", new AlanCustomCounter());

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, Long>> out) throws Exception {
						this.counter.inc();
//				result2LineCounter++;
				System.out.println("计数器行数：" + this.counter.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, this.counter.getCount()));
				}
			}

		}).map(new MapFunction<Tuple2<String, Long>, Tuple3<String, Long, Integer>>() {

			@Override
			public Tuple3<String, Long, Integer> map(Tuple2<String, Long> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static class AlanCustomCounter implements Counter {
		private long count;

		@Override
		public void inc() {
			count += 2;
		}

		@Override
		public void inc(long n) {
			count += n;
		}

		@Override
		public void dec() {
			count -= 2;
		}

		@Override
		public void dec(long n) {
			count -= n;
		}

		@Override
		public long getCount() {
			return count;
		}

	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：2
result:> (hello,2,1)
result:> (123,2,1)
计数器行数：4
result:> (alan,4,1)
result:> (flink,4,1)
result:> (good,4,1)
计数器行数：6
result:> (alan_chan,6,1)
result:> (hi,6,1)
result:> (flink,4,2)

3）、Gauge

仪表可根据需要提供任何类型的值。为了使用Gauge，您必须首先创建一个实现org.apache.flink.metrics.Guge接口的类。返回值的类型没有限制。您可以通过调用MetricGroup上的gauge(String name, Gauge gauge) 来注册gauge。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsGaugeDemo {
//	public class MyMapper extends RichMapFunction<String, String> {
//		private transient int valueToExpose = 0;
//
//		@Override
//		public void open(Configuration config) {
//			getRuntimeContext().getMetricGroup().gauge("MyGauge", new Gauge<Integer>() {
//				@Override
//				public Integer getValue() {
//					return valueToExpose;
//				}
//			});
//		}
//
//		@Override
//		public String map(String value) throws Exception {
//			valueToExpose++;
//			return value;
//		}
//	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				System.out.println("计数器行数：" + result2LineCounter);
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();

	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

报告器会将暴露的对象转换为String，这意味着需要一个有意义的toString()实现。

4）、Histogram

直方图测量长值的分布。您可以通过调用MetricGroup上的histogram(String name, Histogram histogram) 来注册一个对象。

下面的示例是自己实现的Histogram接口，仅仅用于演示实现过程。

java 复制代码 
import java.io.Serializable;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Gauge;
//import com.codahale.metrics.Histogram;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.HistogramStatistics;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsHistogramDemo {

//	public class MyMapper extends RichMapFunction<Long, Long> {
//		private transient Histogram histogram;
//
//		@Override
//		public void open(Configuration config) {
//			this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());
//		}
//
//		@Override
//		public Long map(Long value) throws Exception {
//			this.histogram.update(value);
//			return value;
//		}
//	}

	public static class AlanHistogram implements Histogram {
		private CircularDoubleArray descriptiveStatistics = new CircularDoubleArray(10);;

		public AlanHistogram() {
		}

		public AlanHistogram(int windowSize) {
			this.descriptiveStatistics = new CircularDoubleArray(windowSize);
		}

		@Override
		public void update(long value) {
			this.descriptiveStatistics.addValue(value);
		}

		@Override
		public long getCount() {
			return this.descriptiveStatistics.getElementsSeen();
		}

		@Override
		public HistogramStatistics getStatistics() {
//			return new DescriptiveStatisticsHistogramStatistics(this.descriptiveStatistics);
			return null;
		}

		class CircularDoubleArray implements Serializable {
			private static final long serialVersionUID = 1L;
			private final double[] backingArray;
			private int nextPos = 0;
			private boolean fullSize = false;
			private long elementsSeen = 0;

			CircularDoubleArray(int windowSize) {
				this.backingArray = new double[windowSize];
			}

			synchronized void addValue(double value) {
				backingArray[nextPos] = value;
				++elementsSeen;
				++nextPos;
				if (nextPos == backingArray.length) {
					nextPos = 0;
					fullSize = true;
				}
			}

			synchronized double[] toUnsortedArray() {
				final int size = getSize();
				double[] result = new double[size];
				System.arraycopy(backingArray, 0, result, 0, result.length);
				return result;
			}

			private synchronized int getSize() {
				return fullSize ? backingArray.length : nextPos;
			}

			private synchronized long getElementsSeen() {
				return elementsSeen;
			}
		}

	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});

				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;
				this.histogram.update(result2LineCounter * 3);

				// 此处仅仅示例this.histogram.getCount()的值，没有实际的意义
				System.out.println("计数器行数：" + result2LineCounter + "  histogram:" + this.histogram.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();

	}

}


///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1  histogram:1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2  histogram:2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3  histogram:3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

Flink没有提供直方图的默认实现，但提供了一个允许使用Codahale/DropWizard直方图的包装器。要使用此包装器，

在pom.xml中添加以下依赖项：

xml 复制代码 
<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-metrics-dropwizard</artifactId>
      <version>1.17.1</version>
</dependency>

下面的示例是使用 Codahale/DropWizard直方图，如下所示：

java 复制代码 
import java.io.Serializable;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.metrics.Gauge;
//import com.codahale.metrics.Histogram;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.HistogramStatistics;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsHistogramDemo {

	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
//				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;
				
				this.histogram.update(result2LineCounter * 3);

				// 此处仅仅示例this.histogram.getCount()的值，没有实际的意义
				System.out.println("计数器行数：" + result2LineCounter + "  histogram:" + this.histogram.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：

//控制台输出：
计数器行数：1  histogram:1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2  histogram:2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3  histogram:3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

5）、Meter

仪表测量平均吞吐量。可以使用markEvent()方法注册事件的发生。可以使用markEvent(long n)方法注册同时发生多个事件。您可以通过在MetricGroup上调用meter(String name, Meter meter)来注册meter。

下面的示例展示了自定义的Meter实现，可能很不严谨，实际上应用更多的是本部分的第二个示例。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.Meter;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

//import com.codahale.metrics.Meter;
import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsMeterDemo {

	public class MyMapper extends RichMapFunction<Long, Long> {
		private transient Meter meter;

		@Override
		public void open(Configuration config) {
			this.meter = getRuntimeContext().getMetricGroup().meter("myMeter", new AlanMeter());
		}

		@Override
		public Long map(Long value) throws Exception {
			this.meter.markEvent();
			return value;
		}
	}

	public static class AlanMeter implements Meter {
		/** The underlying counter maintaining the count. */
		private final Counter counter = new SimpleCounter();;
		/** The time-span over which the average is calculated. */
		private final int timeSpanInSeconds = 0;
		/** Circular array containing the history of values. */
		private final long[] values = null;;
		/** The index in the array for the current time. */
		private int time = 0;
		/** The last rate we computed. */
		private double currentRate = 0;

		@Override
		public void markEvent() {
			this.counter.inc();
		}

		@Override
		public void markEvent(long n) {
			this.counter.inc(n);
		}

		@Override
		public long getCount() {
			return counter.getCount();
		}

		@Override
		public double getRate() {
			return currentRate;
		}

		public void update() {
			time = (time + 1) % values.length;
			values[time] = counter.getCount();
			currentRate = ((double) (values[time] - values[(time + 1) % values.length]) / timeSpanInSeconds);
		}

	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;
			private Meter meter;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new AlanMeter());
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				this.histogram.update(result2LineCounter * 3);
				this.meter.markEvent();
				// 此处仅仅示例this.histogram.getCount()、this.meter.getRate()的值，没有实际的意义，具体使用以实际使用场景为准
				System.out.println("计数器行数：" + result2LineCounter + ",  histogram:" + this.histogram.getCount() + ",   meter.getRate:" + this.meter.getRate());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1,  histogram:1,   meter.getRate:0.0
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2,  histogram:2,   meter.getRate:0.0
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3,  histogram:3,   meter.getRate:0.0
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

Flink提供了一个允许使用Codahale/DropWizard仪表的包装器。要使用此包装器，

在pom.xml中添加以下依赖项：

xml 复制代码 
<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-metrics-dropwizard</artifactId>
      <version>1.17.1</version>
</dependency>

下面使用Codahale/DropWizard注册的示例，如下所示：

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.Meter;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

//import com.codahale.metrics.Meter;
import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsMeterDemo {
	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;
			private Meter meter;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

//				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new AlanMeter());
				com.codahale.metrics.Meter dropwizardMeter = new com.codahale.metrics.Meter();
				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new DropwizardMeterWrapper(dropwizardMeter));
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				this.histogram.update(result2LineCounter * 3);
				this.meter.markEvent();
				// 此处仅仅示例this.histogram.getCount()、this.meter.getRate()的值，没有实际的意义，具体使用以实际使用场景为准
				System.out.println("计数器行数：" + result2LineCounter + ",  histogram:" + this.histogram.getCount() + ",   meter.getRate:" + this.meter.getRate());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}


//控制台输出：
计数器行数：1,  histogram:1,   meter.getRate:0.0
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2,  histogram:2,   meter.getRate:0.0
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3,  histogram:3,   meter.getRate:0.0
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

以上，本文简单的介绍了Flink 的指标体系的第一部分，即指标类型以及四种类型的代码实现示例。

本专题分为三部分，即：
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版

相关推荐
Data跳动5 小时前
Spark内存都消耗在哪里了？
大数据·分布式·spark
woshiabc1116 小时前
windows安装Elasticsearch及增删改查操作
大数据·elasticsearch·搜索引擎
lucky_syq6 小时前
Saprk和Flink的区别
大数据·flink
lucky_syq6 小时前
流式处理，为什么Flink比Spark Streaming好？
大数据·flink·spark
袋鼠云数栈6 小时前
深入浅出Flink CEP丨如何通过Flink SQL作业动态更新Flink CEP作业
大数据
小白学大数据8 小时前
如何使用Selenium处理JavaScript动态加载的内容？
大数据·javascript·爬虫·selenium·测试工具
15年网络推广青哥8 小时前
国际抖音TikTok矩阵运营的关键要素有哪些？
大数据·人工智能·矩阵
节点。csn9 小时前
Hadoop yarn安装
大数据·hadoop·分布式
arnold669 小时前
探索 ElasticSearch：性能优化之道
大数据·elasticsearch·性能优化
NiNg_1_23410 小时前
基于Hadoop的数据清洗
大数据·hadoop·分布式
热门推荐
01〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇02ARM学习（31）编译器对overlay方式的支持03玄机平台应急响应—webshell查杀04DisplayPort AltMode On Type-c协议_Ver2.0(学习笔记)05校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站06Docker 夺命连环 15 问076个主流的工业3D管道设计软件08基于大语言模型（LLM）的合成数据生成、策展和评估的综述09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10树莓派 5 AI 套件（Hailo-8L）使用教程