45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例

一瓢一瓢的饮 alanchan2023-11-30 18:10

1、Flink 部署、概念介绍、source、transformation、sink使用示例、四大基石介绍和示例等系列综合文章链接

13、Flink 的table api与sql的基本概念、通用api介绍及入门示例
14、Flink 的table api与sql之数据类型: 内置数据类型以及它们的属性
15、Flink 的table api与sql之流式概念-详解的介绍了动态表、时间属性配置（如何处理更新结果）、时态表、流上的join、流上的确定性以及查询配置
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及FileSystem示例（1）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Elasticsearch示例（2）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Apache Kafka示例（3）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及JDBC示例（4）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Apache Hive示例（6）
17、Flink 之Table API: Table API 支持的操作（1）
17、Flink 之Table API: Table API 支持的操作（2）
18、Flink的SQL 支持的操作和语法
19、Flink 的Table API 和 SQL 中的内置函数及示例（1）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（2）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（3）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（4）
20、Flink SQL之SQL Client: 不用编写代码就可以尝试 Flink SQL，可以直接提交 SQL 任务到集群上
21、Flink 的table API与DataStream API 集成（1）- 介绍及入门示例、集成说明
21、Flink 的table API与DataStream API 集成（2）- 批处理模式和inser-only流处理
21、Flink 的table API与DataStream API 集成（3）- changelog流处理、管道示例、类型转换和老版本转换示例
21、Flink 的table API与DataStream API 集成（完整版）
22、Flink 的table api与sql之创建表的DDL
24、Flink 的table api与sql之Catalogs（介绍、类型、java api和sql实现ddl、java api和sql操作catalog）-1
24、Flink 的table api与sql之Catalogs（java api操作数据库、表）-2
24、Flink 的table api与sql之Catalogs（java api操作视图）-3
24、Flink 的table api与sql之Catalogs（java api操作分区与函数）-4
25、Flink 的table api与sql之函数(自定义函数示例)
26、Flink 的SQL之概览与入门示例
27、Flink 的SQL之SELECT (select、where、distinct、order by、limit、集合操作和去重)介绍及详细示例（1）
27、Flink 的SQL之SELECT (SQL Hints 和 Joins)介绍及详细示例（2）
27、Flink 的SQL之SELECT (窗口函数)介绍及详细示例（3）
27、Flink 的SQL之SELECT (窗口聚合)介绍及详细示例（4）
27、Flink 的SQL之SELECT (Group Aggregation分组聚合、Over Aggregation Over聚合 和 Window Join 窗口关联)介绍及详细示例（5）
27、Flink 的SQL之SELECT (Top-N、Window Top-N 窗口 Top-N 和 Window Deduplication 窗口去重)介绍及详细示例（6）
27、Flink 的SQL之SELECT (Pattern Recognition 模式检测)介绍及详细示例（7）
28、Flink 的SQL之DROP 、ALTER 、INSERT 、ANALYZE 语句
29、Flink SQL之DESCRIBE、EXPLAIN、USE、SHOW、LOAD、UNLOAD、SET、RESET、JAR、JOB Statements、UPDATE、DELETE（1）
29、Flink SQL之DESCRIBE、EXPLAIN、USE、SHOW、LOAD、UNLOAD、SET、RESET、JAR、JOB Statements、UPDATE、DELETE（2）
30、Flink SQL之SQL 客户端（通过kafka和filesystem的例子介绍了配置文件使用-表、视图等）
31、Flink的SQL Gateway介绍及示例
32、Flink table api和SQL 之用户自定义 Sources & Sinks实现及详细示例
33、Flink 的Table API 和 SQL 中的时区
35、Flink 的 Formats 之CSV 和 JSON Format
36、Flink 的 Formats 之Parquet 和 Orc Format
41、Flink之Hive 方言介绍及详细示例
40、Flink 的Apache Kafka connector（kafka source的介绍及使用示例）-1
40、Flink 的Apache Kafka connector（kafka sink的介绍及使用示例）-2
40、Flink 的Apache Kafka connector（kafka source 和sink 说明及使用示例） 完整版
42、Flink 的table api与sql之Hive Catalog
43、Flink之Hive 读写及详细验证示例
44、Flink之module模块介绍及使用示例和Flink SQL使用hive内置函数及自定义函数详细示例--网上有些说法好像是错误的
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版
46、Flink 的table api与sql之配项列表及示例

文章目录

本文简单的介绍了Flink 的指标体系的第一部分，即指标类型以及四种类型的代码实现示例。

本专题分为三部分，即：
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版

本文依赖nc能正常使用。

本文分为5个部分，即指标分类、计数器、gauge、histogram和meter四个指标的代码实现。

本文的示例是在Flink 1.17版本中运行。

Flink暴露了一个度量系统，允许收集度量并将其公开给外部系统。

本文涉及的maven依赖

xml 复制代码 
	<properties>
		<encoding>UTF-8</encoding>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		<java.version>1.8</java.version>
		<scala.version>2.12</scala.version>
		<flink.version>1.17.0</flink.version>
	</properties>

	<dependencies>
		<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-clients</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-streaming-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-csv</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-json</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>

		<!-- flink连接器 -->
		<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-connector-kafka</artifactId>
			<version>${flink.version}</version>
		</dependency>
	</dependencies>

1、Registering metrics 注册指标

通过调用getRuntimeContext().getMetricGroup()，您可以从任何扩展RichFunction的用户函数访问度量系统。此方法返回一个MetricGroup对象，您可以在该对象上创建和注册新度量。

1）、指标类型

Flink支持计数器、仪表盘、柱状图和计量表。Counters, Gauges, Histograms and Meters.

2）、计数器

计数器是用来统计数量的。当前值可以是in-或使用 inc()/inc(long n)或dec()/dec(long n)增减。您可以通过调用MetricGroup上的 counter(String name)来创建和注册计数器。

本示例提供了多种实现方式，供参考。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsDemo {

//	public class LineMapper extends RichMapFunction<String, String> {
//		private transient Counter counter;
//
//		@Override
//		public void open(Configuration config) {
//			this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter");
//		}
//
//		@Override
//		public String map(String value) throws Exception {
//			this.counter.inc();
//			return value;
//		}
//	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		DataStream<Tuple2<String, Integer>> result = lines.flatMap(new FlatMapFunction<String, String>() {
			@Override
			public void flatMap(String value, Collector<String> out) throws Exception {
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(word);
				}
			}
		}).map(new MapFunction<String, Tuple2<String, Integer>>() {

			@Override
			public Tuple2<String, Integer> map(String value) throws Exception {
				return Tuple2.of(value, 1);
			}
		}).keyBy(t -> t.f0).sum(1);

//		SingleOutputStreamOperator<Tuple2<Integer, Integer>> result1 = lines.map(new RichMapFunction<String, Tuple2<Integer, Integer>>() {
//
//			@Override
//			public Tuple2<Integer, Integer> map(String value) throws Exception {
//				int subTaskId = getRuntimeContext().getIndexOfThisSubtask();// 子任务id/分区编号
//				return new Tuple2(subTaskId, 1);
//			}
//			// 按照子任务id/分区编号分组,并统计每个子任务/分区中有几个元素
//		}).keyBy(t -> t.f0).sum(1);

		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, Long, Integer> 输入的字符串，行数，统计单词的总数
		DataStream<Tuple3<String, Long, Integer>> result2 = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, Long>>() {
//			private transient Counter counter;
			private long result2LineCounter = 0;

			@Override
			public void open(Configuration config) {
//				this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter:");
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("result2LineCounter:").getCount();
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, Long>> out) throws Exception {
//				this.counter.inc();
				result2LineCounter++;
				System.out.println("计数器行数：" + result2LineCounter);
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, result2LineCounter));
				}
			}

		}).map(new MapFunction<Tuple2<String, Long>, Tuple3<String, Long, Integer>>() {

			@Override
			public Tuple3<String, Long, Integer> map(Tuple2<String, Long> value) throws Exception {
//				Tuple3<String, Long, Integer> t = Tuple3.of(value.f0, value.f1, 1);
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");
		result2.print("result2:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();
//		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//		env.setParallelism(1);
//		DataStream<String> input = env.fromElements("a", "b", "c", "a", "b", "c");
//
//		input.keyBy(value -> value).map(new RichMapFunction<String, String>() {
//			private long count = 0;
//
//			@Override
//			public void open(Configuration parameters) throws Exception {
                super.open(parameters);
//				count = getRuntimeContext().getMetricGroup().counter("myCounter").getCount();
//			}
//
//			@Override
//			public String map(String value) throws Exception {
//				count++;
//				return value + ": " + count;
//			}
//		}).print();
//
//		env.execute("Flink Count Counter Example");
	}

}
///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1
result:> (hello,1)
result2:> (hello,1,1)
result:> (123,1)
result2:> (123,1,1)
计数器行数：2
result2:> (alan,2,1)
result:> (alan,1)
result2:> (flink,2,1)
result:> (flink,1)
result2:> (good,2,1)
result:> (good,1)
计数器行数：3
result:> (alan_chan,1)
result2:> (alan_chan,3,1)
result:> (hi,1)
result2:> (hi,3,1)
result:> (flink,2)
result2:> (flink,2,2)

或者，您也可以使用自己的Counter实现：

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsDemo {

	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);

		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// Tuple3<String, Long, Integer> 输入的字符串，行数，统计单词的总数
		DataStream<Tuple3<String, Long, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, Long>>() {
			private transient Counter counter;
			@Override
			public void open(Configuration config) {
						this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter", new AlanCustomCounter());

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, Long>> out) throws Exception {
						this.counter.inc();
//				result2LineCounter++;
				System.out.println("计数器行数：" + this.counter.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, this.counter.getCount()));
				}
			}

		}).map(new MapFunction<Tuple2<String, Long>, Tuple3<String, Long, Integer>>() {

			@Override
			public Tuple3<String, Long, Integer> map(Tuple2<String, Long> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static class AlanCustomCounter implements Counter {
		private long count;

		@Override
		public void inc() {
			count += 2;
		}

		@Override
		public void inc(long n) {
			count += n;
		}

		@Override
		public void dec() {
			count -= 2;
		}

		@Override
		public void dec(long n) {
			count -= n;
		}

		@Override
		public long getCount() {
			return count;
		}

	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：2
result:> (hello,2,1)
result:> (123,2,1)
计数器行数：4
result:> (alan,4,1)
result:> (flink,4,1)
result:> (good,4,1)
计数器行数：6
result:> (alan_chan,6,1)
result:> (hi,6,1)
result:> (flink,4,2)

3）、Gauge

仪表可根据需要提供任何类型的值。为了使用Gauge，您必须首先创建一个实现org.apache.flink.metrics.Guge接口的类。返回值的类型没有限制。您可以通过调用MetricGroup上的gauge(String name, Gauge gauge) 来注册gauge。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsGaugeDemo {
//	public class MyMapper extends RichMapFunction<String, String> {
//		private transient int valueToExpose = 0;
//
//		@Override
//		public void open(Configuration config) {
//			getRuntimeContext().getMetricGroup().gauge("MyGauge", new Gauge<Integer>() {
//				@Override
//				public Integer getValue() {
//					return valueToExpose;
//				}
//			});
//		}
//
//		@Override
//		public String map(String value) throws Exception {
//			valueToExpose++;
//			return value;
//		}
//	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				System.out.println("计数器行数：" + result2LineCounter);
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();

	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

报告器会将暴露的对象转换为String，这意味着需要一个有意义的toString()实现。

4）、Histogram

直方图测量长值的分布。您可以通过调用MetricGroup上的histogram(String name, Histogram histogram) 来注册一个对象。

下面的示例是自己实现的Histogram接口，仅仅用于演示实现过程。

java 复制代码 
import java.io.Serializable;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Gauge;
//import com.codahale.metrics.Histogram;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.HistogramStatistics;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsHistogramDemo {

//	public class MyMapper extends RichMapFunction<Long, Long> {
//		private transient Histogram histogram;
//
//		@Override
//		public void open(Configuration config) {
//			this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());
//		}
//
//		@Override
//		public Long map(Long value) throws Exception {
//			this.histogram.update(value);
//			return value;
//		}
//	}

	public static class AlanHistogram implements Histogram {
		private CircularDoubleArray descriptiveStatistics = new CircularDoubleArray(10);;

		public AlanHistogram() {
		}

		public AlanHistogram(int windowSize) {
			this.descriptiveStatistics = new CircularDoubleArray(windowSize);
		}

		@Override
		public void update(long value) {
			this.descriptiveStatistics.addValue(value);
		}

		@Override
		public long getCount() {
			return this.descriptiveStatistics.getElementsSeen();
		}

		@Override
		public HistogramStatistics getStatistics() {
//			return new DescriptiveStatisticsHistogramStatistics(this.descriptiveStatistics);
			return null;
		}

		class CircularDoubleArray implements Serializable {
			private static final long serialVersionUID = 1L;
			private final double[] backingArray;
			private int nextPos = 0;
			private boolean fullSize = false;
			private long elementsSeen = 0;

			CircularDoubleArray(int windowSize) {
				this.backingArray = new double[windowSize];
			}

			synchronized void addValue(double value) {
				backingArray[nextPos] = value;
				++elementsSeen;
				++nextPos;
				if (nextPos == backingArray.length) {
					nextPos = 0;
					fullSize = true;
				}
			}

			synchronized double[] toUnsortedArray() {
				final int size = getSize();
				double[] result = new double[size];
				System.arraycopy(backingArray, 0, result, 0, result.length);
				return result;
			}

			private synchronized int getSize() {
				return fullSize ? backingArray.length : nextPos;
			}

			private synchronized long getElementsSeen() {
				return elementsSeen;
			}
		}

	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});

				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;
				this.histogram.update(result2LineCounter * 3);

				// 此处仅仅示例this.histogram.getCount()的值，没有实际的意义
				System.out.println("计数器行数：" + result2LineCounter + "  histogram:" + this.histogram.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();

	}

}


///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1  histogram:1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2  histogram:2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3  histogram:3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

Flink没有提供直方图的默认实现，但提供了一个允许使用Codahale/DropWizard直方图的包装器。要使用此包装器，

在pom.xml中添加以下依赖项：

xml 复制代码 
<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-metrics-dropwizard</artifactId>
      <version>1.17.1</version>
</dependency>

下面的示例是使用 Codahale/DropWizard直方图，如下所示：

java 复制代码 
import java.io.Serializable;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.metrics.Gauge;
//import com.codahale.metrics.Histogram;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.HistogramStatistics;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsHistogramDemo {

	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
//				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;
				
				this.histogram.update(result2LineCounter * 3);

				// 此处仅仅示例this.histogram.getCount()的值，没有实际的意义
				System.out.println("计数器行数：" + result2LineCounter + "  histogram:" + this.histogram.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：

//控制台输出：
计数器行数：1  histogram:1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2  histogram:2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3  histogram:3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

5）、Meter

仪表测量平均吞吐量。可以使用markEvent()方法注册事件的发生。可以使用markEvent(long n)方法注册同时发生多个事件。您可以通过在MetricGroup上调用meter(String name, Meter meter)来注册meter。

下面的示例展示了自定义的Meter实现，可能很不严谨，实际上应用更多的是本部分的第二个示例。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.Meter;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

//import com.codahale.metrics.Meter;
import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsMeterDemo {

	public class MyMapper extends RichMapFunction<Long, Long> {
		private transient Meter meter;

		@Override
		public void open(Configuration config) {
			this.meter = getRuntimeContext().getMetricGroup().meter("myMeter", new AlanMeter());
		}

		@Override
		public Long map(Long value) throws Exception {
			this.meter.markEvent();
			return value;
		}
	}

	public static class AlanMeter implements Meter {
		/** The underlying counter maintaining the count. */
		private final Counter counter = new SimpleCounter();;
		/** The time-span over which the average is calculated. */
		private final int timeSpanInSeconds = 0;
		/** Circular array containing the history of values. */
		private final long[] values = null;;
		/** The index in the array for the current time. */
		private int time = 0;
		/** The last rate we computed. */
		private double currentRate = 0;

		@Override
		public void markEvent() {
			this.counter.inc();
		}

		@Override
		public void markEvent(long n) {
			this.counter.inc(n);
		}

		@Override
		public long getCount() {
			return counter.getCount();
		}

		@Override
		public double getRate() {
			return currentRate;
		}

		public void update() {
			time = (time + 1) % values.length;
			values[time] = counter.getCount();
			currentRate = ((double) (values[time] - values[(time + 1) % values.length]) / timeSpanInSeconds);
		}

	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;
			private Meter meter;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new AlanMeter());
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				this.histogram.update(result2LineCounter * 3);
				this.meter.markEvent();
				// 此处仅仅示例this.histogram.getCount()、this.meter.getRate()的值，没有实际的意义，具体使用以实际使用场景为准
				System.out.println("计数器行数：" + result2LineCounter + ",  histogram:" + this.histogram.getCount() + ",   meter.getRate:" + this.meter.getRate());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1,  histogram:1,   meter.getRate:0.0
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2,  histogram:2,   meter.getRate:0.0
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3,  histogram:3,   meter.getRate:0.0
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

Flink提供了一个允许使用Codahale/DropWizard仪表的包装器。要使用此包装器，

在pom.xml中添加以下依赖项：

xml 复制代码 
<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-metrics-dropwizard</artifactId>
      <version>1.17.1</version>
</dependency>

下面使用Codahale/DropWizard注册的示例，如下所示：

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.Meter;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

//import com.codahale.metrics.Meter;
import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsMeterDemo {
	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;
			private Meter meter;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

//				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new AlanMeter());
				com.codahale.metrics.Meter dropwizardMeter = new com.codahale.metrics.Meter();
				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new DropwizardMeterWrapper(dropwizardMeter));
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				this.histogram.update(result2LineCounter * 3);
				this.meter.markEvent();
				// 此处仅仅示例this.histogram.getCount()、this.meter.getRate()的值，没有实际的意义，具体使用以实际使用场景为准
				System.out.println("计数器行数：" + result2LineCounter + ",  histogram:" + this.histogram.getCount() + ",   meter.getRate:" + this.meter.getRate());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}


//控制台输出：
计数器行数：1,  histogram:1,   meter.getRate:0.0
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2,  histogram:2,   meter.getRate:0.0
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3,  histogram:3,   meter.getRate:0.0
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

以上，本文简单的介绍了Flink 的指标体系的第一部分，即指标类型以及四种类型的代码实现示例。

本专题分为三部分，即：
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版

相关推荐
智慧化智能化数字化方案13 分钟前
华为IPD流程管理体系L1至L5最佳实践-解读
大数据·华为
PersistJiao1 小时前
在 Spark RDD 中，sortBy 和 top 算子的各自适用场景
大数据·spark·top·sortby
2301_811274312 小时前
大数据基于Spring Boot的化妆品推荐系统的设计与实现
大数据·spring boot·后端
Yz98762 小时前
hive的存储格式
大数据·数据库·数据仓库·hive·hadoop·数据库开发
青云交2 小时前
大数据新视界 -- 大数据大厂之 Hive 数据导入：多源数据集成的策略与实战（上）（3/ 30）
大数据·数据清洗·电商数据·数据整合·hive 数据导入·多源数据·影视娱乐数据
武子康2 小时前
大数据-230 离线数仓 - ODS层的构建 Hive处理 UDF 与 SerDe 处理 与 当前总结
java·大数据·数据仓库·hive·hadoop·sql·hdfs
武子康2 小时前
大数据-231 离线数仓 - DWS 层、ADS 层的创建 Hive 执行脚本
java·大数据·数据仓库·hive·hadoop·mysql
时差9532 小时前
Flink Standalone集群模式安装部署
大数据·分布式·flink·部署
锵锵锵锵~蒋2 小时前
实时数据开发 | 怎么通俗理解Flink容错机制，提到的checkpoint、barrier、Savepoint、sink都是什么
大数据·数据仓库·flink·实时数据开发
二进制_博客2 小时前
Flink学习连载文章4-flink中的各种转换操作
大数据·学习·flink
热门推荐
01(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04玄机平台应急响应—webshell查杀05Coze扣子平台完整体验和实践（附国内和国际版对比）06Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO07【目标跟踪】相机运动补偿08Unity中PICO实现 隔空取物 和 接触抓取物体09RAG 实践- Ollama+RagFlow 部署本地知识库10量化方法怎么选？如何评估量化后的大模型LLM？