45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例

一瓢一瓢的饮 alanchan2023-11-30 18:10

1、Flink 部署、概念介绍、source、transformation、sink使用示例、四大基石介绍和示例等系列综合文章链接

13、Flink 的table api与sql的基本概念、通用api介绍及入门示例
14、Flink 的table api与sql之数据类型: 内置数据类型以及它们的属性
15、Flink 的table api与sql之流式概念-详解的介绍了动态表、时间属性配置（如何处理更新结果）、时态表、流上的join、流上的确定性以及查询配置
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及FileSystem示例（1）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Elasticsearch示例（2）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Apache Kafka示例（3）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及JDBC示例（4）
16、Flink 的table api与sql之连接外部系统: 读写外部系统的连接器和格式以及Apache Hive示例（6）
17、Flink 之Table API: Table API 支持的操作（1）
17、Flink 之Table API: Table API 支持的操作（2）
18、Flink的SQL 支持的操作和语法
19、Flink 的Table API 和 SQL 中的内置函数及示例（1）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（2）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（3）
19、Flink 的Table API 和 SQL 中的自定义函数及示例（4）
20、Flink SQL之SQL Client: 不用编写代码就可以尝试 Flink SQL，可以直接提交 SQL 任务到集群上
21、Flink 的table API与DataStream API 集成（1）- 介绍及入门示例、集成说明
21、Flink 的table API与DataStream API 集成（2）- 批处理模式和inser-only流处理
21、Flink 的table API与DataStream API 集成（3）- changelog流处理、管道示例、类型转换和老版本转换示例
21、Flink 的table API与DataStream API 集成（完整版）
22、Flink 的table api与sql之创建表的DDL
24、Flink 的table api与sql之Catalogs（介绍、类型、java api和sql实现ddl、java api和sql操作catalog）-1
24、Flink 的table api与sql之Catalogs（java api操作数据库、表）-2
24、Flink 的table api与sql之Catalogs（java api操作视图）-3
24、Flink 的table api与sql之Catalogs（java api操作分区与函数）-4
25、Flink 的table api与sql之函数(自定义函数示例)
26、Flink 的SQL之概览与入门示例
27、Flink 的SQL之SELECT (select、where、distinct、order by、limit、集合操作和去重)介绍及详细示例（1）
27、Flink 的SQL之SELECT (SQL Hints 和 Joins)介绍及详细示例（2）
27、Flink 的SQL之SELECT (窗口函数)介绍及详细示例（3）
27、Flink 的SQL之SELECT (窗口聚合)介绍及详细示例（4）
27、Flink 的SQL之SELECT (Group Aggregation分组聚合、Over Aggregation Over聚合 和 Window Join 窗口关联)介绍及详细示例（5）
27、Flink 的SQL之SELECT (Top-N、Window Top-N 窗口 Top-N 和 Window Deduplication 窗口去重)介绍及详细示例（6）
27、Flink 的SQL之SELECT (Pattern Recognition 模式检测)介绍及详细示例（7）
28、Flink 的SQL之DROP 、ALTER 、INSERT 、ANALYZE 语句
29、Flink SQL之DESCRIBE、EXPLAIN、USE、SHOW、LOAD、UNLOAD、SET、RESET、JAR、JOB Statements、UPDATE、DELETE（1）
29、Flink SQL之DESCRIBE、EXPLAIN、USE、SHOW、LOAD、UNLOAD、SET、RESET、JAR、JOB Statements、UPDATE、DELETE（2）
30、Flink SQL之SQL 客户端（通过kafka和filesystem的例子介绍了配置文件使用-表、视图等）
31、Flink的SQL Gateway介绍及示例
32、Flink table api和SQL 之用户自定义 Sources & Sinks实现及详细示例
33、Flink 的Table API 和 SQL 中的时区
35、Flink 的 Formats 之CSV 和 JSON Format
36、Flink 的 Formats 之Parquet 和 Orc Format
41、Flink之Hive 方言介绍及详细示例
40、Flink 的Apache Kafka connector（kafka source的介绍及使用示例）-1
40、Flink 的Apache Kafka connector（kafka sink的介绍及使用示例）-2
40、Flink 的Apache Kafka connector（kafka source 和sink 说明及使用示例） 完整版
42、Flink 的table api与sql之Hive Catalog
43、Flink之Hive 读写及详细验证示例
44、Flink之module模块介绍及使用示例和Flink SQL使用hive内置函数及自定义函数详细示例--网上有些说法好像是错误的
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版
46、Flink 的table api与sql之配项列表及示例

文章目录

本文简单的介绍了Flink 的指标体系的第一部分，即指标类型以及四种类型的代码实现示例。

本专题分为三部分，即：
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版

本文依赖nc能正常使用。

本文分为5个部分，即指标分类、计数器、gauge、histogram和meter四个指标的代码实现。

本文的示例是在Flink 1.17版本中运行。

Flink暴露了一个度量系统，允许收集度量并将其公开给外部系统。

本文涉及的maven依赖

xml 复制代码 
	<properties>
		<encoding>UTF-8</encoding>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
		<java.version>1.8</java.version>
		<scala.version>2.12</scala.version>
		<flink.version>1.17.0</flink.version>
	</properties>

	<dependencies>
		<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-clients -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-clients</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-streaming-java</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-csv</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-json</artifactId>
			<version>${flink.version}</version>
			<scope>provided</scope>
		</dependency>

		<!-- flink连接器 -->
		<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka -->
		<dependency>
			<groupId>org.apache.flink</groupId>
			<artifactId>flink-connector-kafka</artifactId>
			<version>${flink.version}</version>
		</dependency>
	</dependencies>

1、Registering metrics 注册指标

通过调用getRuntimeContext().getMetricGroup()，您可以从任何扩展RichFunction的用户函数访问度量系统。此方法返回一个MetricGroup对象，您可以在该对象上创建和注册新度量。

1）、指标类型

Flink支持计数器、仪表盘、柱状图和计量表。Counters, Gauges, Histograms and Meters.

2）、计数器

计数器是用来统计数量的。当前值可以是in-或使用 inc()/inc(long n)或dec()/dec(long n)增减。您可以通过调用MetricGroup上的 counter(String name)来创建和注册计数器。

本示例提供了多种实现方式，供参考。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsDemo {

//	public class LineMapper extends RichMapFunction<String, String> {
//		private transient Counter counter;
//
//		@Override
//		public void open(Configuration config) {
//			this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter");
//		}
//
//		@Override
//		public String map(String value) throws Exception {
//			this.counter.inc();
//			return value;
//		}
//	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		DataStream<Tuple2<String, Integer>> result = lines.flatMap(new FlatMapFunction<String, String>() {
			@Override
			public void flatMap(String value, Collector<String> out) throws Exception {
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(word);
				}
			}
		}).map(new MapFunction<String, Tuple2<String, Integer>>() {

			@Override
			public Tuple2<String, Integer> map(String value) throws Exception {
				return Tuple2.of(value, 1);
			}
		}).keyBy(t -> t.f0).sum(1);

//		SingleOutputStreamOperator<Tuple2<Integer, Integer>> result1 = lines.map(new RichMapFunction<String, Tuple2<Integer, Integer>>() {
//
//			@Override
//			public Tuple2<Integer, Integer> map(String value) throws Exception {
//				int subTaskId = getRuntimeContext().getIndexOfThisSubtask();// 子任务id/分区编号
//				return new Tuple2(subTaskId, 1);
//			}
//			// 按照子任务id/分区编号分组,并统计每个子任务/分区中有几个元素
//		}).keyBy(t -> t.f0).sum(1);

		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, Long, Integer> 输入的字符串，行数，统计单词的总数
		DataStream<Tuple3<String, Long, Integer>> result2 = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, Long>>() {
//			private transient Counter counter;
			private long result2LineCounter = 0;

			@Override
			public void open(Configuration config) {
//				this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter:");
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("result2LineCounter:").getCount();
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, Long>> out) throws Exception {
//				this.counter.inc();
				result2LineCounter++;
				System.out.println("计数器行数：" + result2LineCounter);
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, result2LineCounter));
				}
			}

		}).map(new MapFunction<Tuple2<String, Long>, Tuple3<String, Long, Integer>>() {

			@Override
			public Tuple3<String, Long, Integer> map(Tuple2<String, Long> value) throws Exception {
//				Tuple3<String, Long, Integer> t = Tuple3.of(value.f0, value.f1, 1);
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");
		result2.print("result2:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();
//		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//		env.setParallelism(1);
//		DataStream<String> input = env.fromElements("a", "b", "c", "a", "b", "c");
//
//		input.keyBy(value -> value).map(new RichMapFunction<String, String>() {
//			private long count = 0;
//
//			@Override
//			public void open(Configuration parameters) throws Exception {
                super.open(parameters);
//				count = getRuntimeContext().getMetricGroup().counter("myCounter").getCount();
//			}
//
//			@Override
//			public String map(String value) throws Exception {
//				count++;
//				return value + ": " + count;
//			}
//		}).print();
//
//		env.execute("Flink Count Counter Example");
	}

}
///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1
result:> (hello,1)
result2:> (hello,1,1)
result:> (123,1)
result2:> (123,1,1)
计数器行数：2
result2:> (alan,2,1)
result:> (alan,1)
result2:> (flink,2,1)
result:> (flink,1)
result2:> (good,2,1)
result:> (good,1)
计数器行数：3
result:> (alan_chan,1)
result2:> (alan_chan,3,1)
result:> (hi,1)
result2:> (hi,3,1)
result:> (flink,2)
result2:> (flink,2,2)

或者，您也可以使用自己的Counter实现：

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsDemo {

	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);

		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// Tuple3<String, Long, Integer> 输入的字符串，行数，统计单词的总数
		DataStream<Tuple3<String, Long, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, Long>>() {
			private transient Counter counter;
			@Override
			public void open(Configuration config) {
						this.counter = getRuntimeContext().getMetricGroup().counter("result2LineCounter", new AlanCustomCounter());

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, Long>> out) throws Exception {
						this.counter.inc();
//				result2LineCounter++;
				System.out.println("计数器行数：" + this.counter.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, this.counter.getCount()));
				}
			}

		}).map(new MapFunction<Tuple2<String, Long>, Tuple3<String, Long, Integer>>() {

			@Override
			public Tuple3<String, Long, Integer> map(Tuple2<String, Long> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static class AlanCustomCounter implements Counter {
		private long count;

		@Override
		public void inc() {
			count += 2;
		}

		@Override
		public void inc(long n) {
			count += n;
		}

		@Override
		public void dec() {
			count -= 2;
		}

		@Override
		public void dec(long n) {
			count -= n;
		}

		@Override
		public long getCount() {
			return count;
		}

	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：2
result:> (hello,2,1)
result:> (123,2,1)
计数器行数：4
result:> (alan,4,1)
result:> (flink,4,1)
result:> (good,4,1)
计数器行数：6
result:> (alan_chan,6,1)
result:> (hi,6,1)
result:> (flink,4,2)

3）、Gauge

仪表可根据需要提供任何类型的值。为了使用Gauge，您必须首先创建一个实现org.apache.flink.metrics.Guge接口的类。返回值的类型没有限制。您可以通过调用MetricGroup上的gauge(String name, Gauge gauge) 来注册gauge。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsGaugeDemo {
//	public class MyMapper extends RichMapFunction<String, String> {
//		private transient int valueToExpose = 0;
//
//		@Override
//		public void open(Configuration config) {
//			getRuntimeContext().getMetricGroup().gauge("MyGauge", new Gauge<Integer>() {
//				@Override
//				public Integer getValue() {
//					return valueToExpose;
//				}
//			});
//		}
//
//		@Override
//		public String map(String value) throws Exception {
//			valueToExpose++;
//			return value;
//		}
//	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				System.out.println("计数器行数：" + result2LineCounter);
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();

	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

报告器会将暴露的对象转换为String，这意味着需要一个有意义的toString()实现。

4）、Histogram

直方图测量长值的分布。您可以通过调用MetricGroup上的histogram(String name, Histogram histogram) 来注册一个对象。

下面的示例是自己实现的Histogram接口，仅仅用于演示实现过程。

java 复制代码 
import java.io.Serializable;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.metrics.Gauge;
//import com.codahale.metrics.Histogram;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.HistogramStatistics;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * @author alanchan
 *
 */
public class TestMetricsHistogramDemo {

//	public class MyMapper extends RichMapFunction<Long, Long> {
//		private transient Histogram histogram;
//
//		@Override
//		public void open(Configuration config) {
//			this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());
//		}
//
//		@Override
//		public Long map(Long value) throws Exception {
//			this.histogram.update(value);
//			return value;
//		}
//	}

	public static class AlanHistogram implements Histogram {
		private CircularDoubleArray descriptiveStatistics = new CircularDoubleArray(10);;

		public AlanHistogram() {
		}

		public AlanHistogram(int windowSize) {
			this.descriptiveStatistics = new CircularDoubleArray(windowSize);
		}

		@Override
		public void update(long value) {
			this.descriptiveStatistics.addValue(value);
		}

		@Override
		public long getCount() {
			return this.descriptiveStatistics.getElementsSeen();
		}

		@Override
		public HistogramStatistics getStatistics() {
//			return new DescriptiveStatisticsHistogramStatistics(this.descriptiveStatistics);
			return null;
		}

		class CircularDoubleArray implements Serializable {
			private static final long serialVersionUID = 1L;
			private final double[] backingArray;
			private int nextPos = 0;
			private boolean fullSize = false;
			private long elementsSeen = 0;

			CircularDoubleArray(int windowSize) {
				this.backingArray = new double[windowSize];
			}

			synchronized void addValue(double value) {
				backingArray[nextPos] = value;
				++elementsSeen;
				++nextPos;
				if (nextPos == backingArray.length) {
					nextPos = 0;
					fullSize = true;
				}
			}

			synchronized double[] toUnsortedArray() {
				final int size = getSize();
				double[] result = new double[size];
				System.arraycopy(backingArray, 0, result, 0, result.length);
				return result;
			}

			private synchronized int getSize() {
				return fullSize ? backingArray.length : nextPos;
			}

			private synchronized long getElementsSeen() {
				return elementsSeen;
			}
		}

	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});

				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;
				this.histogram.update(result2LineCounter * 3);

				// 此处仅仅示例this.histogram.getCount()的值，没有实际的意义
				System.out.println("计数器行数：" + result2LineCounter + "  histogram:" + this.histogram.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();

	}

}


///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1  histogram:1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2  histogram:2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3  histogram:3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

Flink没有提供直方图的默认实现，但提供了一个允许使用Codahale/DropWizard直方图的包装器。要使用此包装器，

在pom.xml中添加以下依赖项：

xml 复制代码 
<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-metrics-dropwizard</artifactId>
      <version>1.17.1</version>
</dependency>

下面的示例是使用 Codahale/DropWizard直方图，如下所示：

java 复制代码 
import java.io.Serializable;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.metrics.Gauge;
//import com.codahale.metrics.Histogram;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.HistogramStatistics;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsHistogramDemo {

	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
//				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new AlanHistogram());
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;
				
				this.histogram.update(result2LineCounter * 3);

				// 此处仅仅示例this.histogram.getCount()的值，没有实际的意义
				System.out.println("计数器行数：" + result2LineCounter + "  histogram:" + this.histogram.getCount());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：

//控制台输出：
计数器行数：1  histogram:1
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2  histogram:2
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3  histogram:3
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

5）、Meter

仪表测量平均吞吐量。可以使用markEvent()方法注册事件的发生。可以使用markEvent(long n)方法注册同时发生多个事件。您可以通过在MetricGroup上调用meter(String name, Meter meter)来注册meter。

下面的示例展示了自定义的Meter实现，可能很不严谨，实际上应用更多的是本部分的第二个示例。

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.Meter;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

//import com.codahale.metrics.Meter;
import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsMeterDemo {

	public class MyMapper extends RichMapFunction<Long, Long> {
		private transient Meter meter;

		@Override
		public void open(Configuration config) {
			this.meter = getRuntimeContext().getMetricGroup().meter("myMeter", new AlanMeter());
		}

		@Override
		public Long map(Long value) throws Exception {
			this.meter.markEvent();
			return value;
		}
	}

	public static class AlanMeter implements Meter {
		/** The underlying counter maintaining the count. */
		private final Counter counter = new SimpleCounter();;
		/** The time-span over which the average is calculated. */
		private final int timeSpanInSeconds = 0;
		/** Circular array containing the history of values. */
		private final long[] values = null;;
		/** The index in the array for the current time. */
		private int time = 0;
		/** The last rate we computed. */
		private double currentRate = 0;

		@Override
		public void markEvent() {
			this.counter.inc();
		}

		@Override
		public void markEvent(long n) {
			this.counter.inc(n);
		}

		@Override
		public long getCount() {
			return counter.getCount();
		}

		@Override
		public double getRate() {
			return currentRate;
		}

		public void update() {
			time = (time + 1) % values.length;
			values[time] = counter.getCount();
			currentRate = ((double) (values[time] - values[(time + 1) % values.length]) / timeSpanInSeconds);
		}

	}

	public static void test1() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;
			private Meter meter;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new AlanMeter());
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				this.histogram.update(result2LineCounter * 3);
				this.meter.markEvent();
				// 此处仅仅示例this.histogram.getCount()、this.meter.getRate()的值，没有实际的意义，具体使用以实际使用场景为准
				System.out.println("计数器行数：" + result2LineCounter + ",  histogram:" + this.histogram.getCount() + ",   meter.getRate:" + this.meter.getRate());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test1();
	}

}

///验证数据///
// 输入数据
[alanchan@server2 bin]$ nc -lk 9999
hello,123
alan,flink,good
alan_chan,hi,flink

//控制台输出：
计数器行数：1,  histogram:1,   meter.getRate:0.0
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2,  histogram:2,   meter.getRate:0.0
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3,  histogram:3,   meter.getRate:0.0
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

Flink提供了一个允许使用Codahale/DropWizard仪表的包装器。要使用此包装器，

在pom.xml中添加以下依赖项：

xml 复制代码 
<dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-metrics-dropwizard</artifactId>
      <version>1.17.1</version>
</dependency>

下面使用Codahale/DropWizard注册的示例，如下所示：

java 复制代码 
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper;
import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Histogram;
import org.apache.flink.metrics.Meter;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

//import com.codahale.metrics.Meter;
import com.codahale.metrics.SlidingWindowReservoir;

/**
 * @author alanchan
 *
 */
public class TestMetricsMeterDemo {
	public static void test2() throws Exception {
		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
		env.setParallelism(1);
		// source
		DataStream<String> lines = env.socketTextStream("192.168.10.42", 9999);

		// transformation
		// RichFlatMapFunction<IN, OUT>
		// Tuple3<String, String, Integer> 输入的字符串，alan lines[行数]，统计单词的总数
		DataStream<Tuple3<String, String, Integer>> result = lines.flatMap(new RichFlatMapFunction<String, Tuple2<String, String>>() {
			private long result2LineCounter = 0;
			private Gauge<String> gauge = null;
			private Histogram histogram = null;
			private Meter meter;

			@Override
			public void open(Configuration config) {
				result2LineCounter = getRuntimeContext().getMetricGroup().counter("resultLineCounter:").getCount();

				gauge = getRuntimeContext().getMetricGroup().gauge("alanGauge", new Gauge<String>() {
					@Override
					public String getValue() {
						return "alan lines[" + result2LineCounter + "]";
					}
				});
				com.codahale.metrics.Histogram dropwizardHistogram = new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
				this.histogram = getRuntimeContext().getMetricGroup().histogram("alanHistogram", new DropwizardHistogramWrapper(dropwizardHistogram));

//				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new AlanMeter());
				com.codahale.metrics.Meter dropwizardMeter = new com.codahale.metrics.Meter();
				this.meter = getRuntimeContext().getMetricGroup().meter("alanMeter", new DropwizardMeterWrapper(dropwizardMeter));
			}

			@Override
			public void flatMap(String value, Collector<Tuple2<String, String>> out) throws Exception {
				result2LineCounter++;

				this.histogram.update(result2LineCounter * 3);
				this.meter.markEvent();
				// 此处仅仅示例this.histogram.getCount()、this.meter.getRate()的值，没有实际的意义，具体使用以实际使用场景为准
				System.out.println("计数器行数：" + result2LineCounter + ",  histogram:" + this.histogram.getCount() + ",   meter.getRate:" + this.meter.getRate());
				String[] arr = value.split(",");
				for (String word : arr) {
					out.collect(Tuple2.of(word, gauge.getValue()));
				}
			}

		}).map(new MapFunction<Tuple2<String, String>, Tuple3<String, String, Integer>>() {

			@Override
			public Tuple3<String, String, Integer> map(Tuple2<String, String> value) throws Exception {
				return Tuple3.of(value.f0, value.f1, 1);
			}
		}).keyBy(t -> t.f0).sum(2);

		// sink
		result.print("result:");

		env.execute();
	}

	public static void main(String[] args) throws Exception {
		test2();
	}

}


//控制台输出：
计数器行数：1,  histogram:1,   meter.getRate:0.0
result:> (hello,alan lines[1],1)
result:> (123,alan lines[1],1)
计数器行数：2,  histogram:2,   meter.getRate:0.0
result:> (alan,alan lines[2],1)
result:> (flink,alan lines[2],1)
result:> (good,alan lines[2],1)
计数器行数：3,  histogram:3,   meter.getRate:0.0
result:> (alan_chan,alan lines[3],1)
result:> (hi,alan lines[3],1)
result:> (flink,alan lines[2],2)

以上，本文简单的介绍了Flink 的指标体系的第一部分，即指标类型以及四种类型的代码实现示例。

本专题分为三部分，即：
45、Flink 的指标体系介绍及验证（1）-指标类型及指标实现示例
45、Flink 的指标体系介绍及验证（2）-指标的scope、报告、系统指标以及追踪、api集成示例和dashboard集成
45、Flink 的指标体系介绍及验证（3）- 完整版

相关推荐
StarRocks_labs3 小时前
从InfluxDB到StarRocks：Grab实现Spark监控平台10倍性能提升
大数据·数据库·starrocks·分布式·spark·iris·物化视图
董可伦3 小时前
Dinky 安装部署并配置提交 Flink Yarn 任务
android·adb·flink
若兰幽竹4 小时前
【Spark分析HBase数据】Spark读取并分析HBase数据
大数据·spark·hbase
R²AIN SUITE4 小时前
金融合规革命：R²AIN SUITE 如何重塑银行业务智能
大数据·人工智能
绿算技术5 小时前
“强强联手，智启未来”凯创未来与绿算技术共筑高端智能家居及智能照明领域新生态
大数据·人工智能·智能家居
只因只因爆6 小时前
spark的缓存
大数据·缓存·spark
Leo.yuan7 小时前
3D 数据可视化系统是什么？具体应用在哪方面？
大数据·数据库·3d·信息可视化·数据分析
只因只因爆7 小时前
spark小任务
大数据·分布式·spark
cainiao0806057 小时前
Java 大视界——Java 大数据在智慧交通智能停车诱导系统中的数据融合与实时更新
java·大数据·开发语言
End92810 小时前
Spark之搭建Yarn模式
大数据·分布式·spark
热门推荐
01KGG转MP3工具|非KGM文件|解密音频02YOLOv8入门 | 重要性能衡量指标、训练结果评价及分析及影响mAP的因素【发论文关注的指标】03从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑04【SpeedAI科研小助手】2分钟极速解决知网维普重复率、AIGC率过高，一键全文降！文件格式不变，公式都保留的！05Coze扣子平台完整体验和实践（附国内和国际版对比）06DeepSeek各版本说明与优缺点分析07组基轨迹建模 GBTM的介绍与实现（Stata 或 R）08YOLOv5改进 | 添加CA注意力机制 + 增加预测层 + 更换损失函数之GIoU09苍穹外卖面试总结10【解决】Android Gradle Sync 报错 Could not read workspace metadata