在网站上查找隐藏的API - 以flightaware.com为例(+使用该数据创建Plotly动画图表)

The idea is that you don't always need Selenium or BeautifulSoup to scrape data from websites. You can find references to data sources in plain HTML.

The problem with scraping is that HTML selectors are hard to pinpoint. They are also prone to changing if the interface changes and if identifiers such as IDs and classes are dynamically generated.

Today I will show you an alternative approach. Of course, this approach doesn't work all the time, it depends on the site in question. Many sites protect their data sources and offer paid access through official APIs. Some however do not, either by negligence or because they thought nobody would ever look.

Let's explore one of the examples - Flightaware.com. Normally, the flight information data that the site provides is hidden behind a paywall, but is accessible in the processed form on the main page.

If you scroll down the main page, you will see an interactive map of flights in your area.

If you click on one of the planes, you will be taken to the flight information page. The data it offers although informative but is very limited. Some of it very much scrapable with the aforementioned tools, some of it is abstracted away in charts and cannot be accessed programatically.

What if we explore the HTML source code of the page?

Scrolling through the DOM, we can see some TOKEN references, references to the template engine...

And unexpectedly, a long string of what looks like JSON data stored in a variable trackpollBootstrap. Interesting!

Let's look at it in Python.

python 复制代码
import requests
import re
import json

# FLIGHT_NUMBER = "CBJ606"
# FLIGHT_NUMBER = "TYA3422"
FLIGHT_NUMBER = "ETD826"


def get_flight_data(flight_id: str) -> dict:
    try:
        url = f"https://flightaware.com/live/flight/{flight_id}"

        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
        }
        page_source = requests.get(url, headers=headers).text

        scripts = re.findall(r"<script>(.*)</script>", page_source)

        script = scripts[-1].strip(";").removeprefix("var trackpollBootstrap = ")

        data = json.loads(script)

        return data

    except Exception as e:
        print(e)


data = get_flight_data(FLIGHT_NUMBER)

We will use ETD826 flight ID in this tutorial. You can try other IDs as well.

First, we prepare the target URL with the following structure: url = f"https://flightaware.com/live/flight/{flight_id}".

Second, we prepare some headers so as to pretend we are a real browser and get the source HTML of the page.

Because, the data lives in one of the <script> tags, we search for all the script tags with RegEx. This gives us the contents of those tags:
scripts = re.findall(r"<script>(.*)</script>", page_source)

The target script is actually the last one in the list. We process it as text, stripping all of the JavaScript bits:
script = scripts[-1].strip(";").removeprefix("var trackpollBootstrap = ")

And finally load it as JSON: data = json.loads(script).

What we are looking for is the flights key. It contains all flight information, aircraft and airline information, origin and destination information, even the amount of fuel burned in flight.

It also contains the track information - the coordinates of all the checkpoints on the map.

Let's use this data to recreate the flight path.

In order to get to the track data, we need to traverse the data dictionary, which requires 3 steps.

python 复制代码
track_data = data["flights"]["ETD826-1736484856-schedule-194p:0"]["track"]

I discovered that this the key ETD826-1736484856-schedule-194p:0 is dynamic and changes after each new flight, but we need to know the value of this key in order to get to the track data.

We can use the following expression in place of the key, which will get us to the track data, even if the value changes. It simply returns the next child value in the dictionary tree.

python 复制代码
track_data = data["flights"][next(iter(data["flights"]))]["track"]

Here we have the coordinates of the flight path on the route, the speed and altitude (I assume in the last flight) and the time of passing of each waypoint.

Let's load this data into Pandas.

python 复制代码
import pandas as pd

df = pd.DataFrame(track_data)

df

Let's separate the coordinates into latitude and longitude. First, we represent the coords column as string, because for some reason it was loaded as float datatype. Second, we strip the brackets and, finally, split the column into two. We also drop the coords column, as it is no longer needed.

python 复制代码
df["coord"] = df["coord"].map(repr)

df["coord"] = df["coord"].str.replace("[", "").str.replace("]", "")

df[["lon", "lat"]] = df["coord"].str.split(", ", expand=True)
df = df.drop("coord", axis=1)

df

Let's plot the data.

python 复制代码
import plotly.graph_objects as go
import pandas as pd

fig = go.Figure()

fig.add_trace(
    go.Scattergeo(
        locationmode="country names",
        lon=df["lon"],
        lat=df["lat"],
        hoverinfo="text",
        text=df["alt"],
        mode="lines",
        line=dict(width=3, color="red"),
    )
)

fig.update_layout(
    title_text=f"Flight path for {FLIGHT_NUMBER}",
    showlegend=False,
)

fig.show()

Let's make it better, add animation and some information!

python 复制代码
origin = data["flights"][next(iter(data["flights"]))]["activityLog"]["flights"][0][
    "origin"
]["friendlyLocation"].upper()
destination = data["flights"][next(iter(data["flights"]))]["activityLog"]["flights"][0][
    "destination"
]["friendlyLocation"].upper()

print(origin, destination)

Here, we get origin and destination cities.

python 复制代码
>>> ABU DHABI, AE SEOUL, SOUTH KOREA
python 复制代码
import plotly.graph_objects as go


def modify_t(i):
    t2 = t.copy()
    if 0 < i < df.shape[0]:
        t2[i - 1] = f"<br>FL: {df.at[i, 'alt']}<br>Speed: {df.at[i, 'gs']}"
    return t2


t = ["<br>" + origin] + ["" for j in range(df.shape[0] - 2)] + ["<br>" + destination]
c = ["red"] + ["white" for j in range(df.shape[0] - 2)] + ["red"]

fig = go.Figure(
    data=[
        go.Scattergeo(
            locationmode="country names",
            line=dict(width=3, color="rgba(255, 0, 0, 0.25)"),
            textfont=dict(size=12, color=c),
            textposition="bottom center",
            mode="lines+text",
        ),
    ],
    layout=go.Layout(
        updatemenus=[
            dict(
                type="buttons",
                buttons=[
                    dict(
                        label="▶",
                        method="animate",
                        args=[
                            None,
                            {
                                "frame": {"duration": 1},
                                "transition": {
                                    "duration": 1,
                                },
                            },
                        ],
                    )
                ],
                direction="left",
                pad={"r": 0, "t": 0},
                x=0.1,
                xanchor="right",
                y=1.05,
                yanchor="top",
            )
        ],
    ),
    frames=[
        go.Frame(
            data=[
                go.Scattergeo(
                    lon=df["lon"][:i],
                    lat=df["lat"][:i],
                    text=modify_t(i),
                ),
            ]
        )
        for i in range(df.shape[0] - 1)
    ]
    + [
        go.Frame(
            data=[
                go.Scattergeo(
                    lon=df["lon"],
                    lat=df["lat"],
                    text=t,
                ),
            ]
        )
    ],
)

fig.add_annotation(
    text="* FL is Flight Level in hundreds of Feet<br>   Speed is Ground Speed in Knots",
    xref="paper",
    yref="paper",
    x=0.94,
    y=0.00,  # 0.89, 0.1
    font=dict(color="LightGrey"),
    showarrow=False,
    align="left",
)

fig.update_layout(
    width=800,
    height=400,
    margin={"r": 0, "t": 50, "l": 0, "b": 50},
)
fig.update_layout(
    paper_bgcolor="#171717",
    title_text=f"Flight path for {FLIGHT_NUMBER}",
    title_font=dict(color="LightGrey"),
    title_x=0.12,
    title_y=0.89,
    showlegend=False,
)
center = int(df.shape[0] / 2)
fig.update_geos(
    projection_type="orthographic",
    bgcolor="#171717",
    projection_rotation={
        "lat": float(df.at[center, "lat"]),
        "lon": float(df.at[center, "lon"]),
        "roll": 0,
    },
    resolution=110,  # 50或110
    showcoastlines=False,
    showland=True,
    landcolor="#242424",
    showocean=True,
    oceancolor="#171717",
)
config = {"displayModeBar": False, "responsive": False, "staticPlot": True}
fig.show(config=config)

Plotly动画图表

Plotly动画图表TYA3422-1

python 复制代码
import plotly.graph_objects as go


def modify_t(i):
    t2 = t.copy()
    if 0 < i < df.shape[0]:
        t2[i - 1] = f"<br>FL: {df.at[i, 'alt']}<br>Speed: {df.at[i, 'gs']}"
    return t2


t = ["<br>" + origin] + ["" for j in range(df.shape[0] - 2)] + ["<br>" + destination]
c = ["red"] + ["white" for j in range(df.shape[0] - 2)] + ["red"]

fig = go.Figure(
    data=[
        go.Scattergeo(
            locationmode="country names",
            line=dict(width=3, color="rgba(255, 0, 0, 0.25)"),
            textfont=dict(size=12, color=c),
            textposition="bottom center",
            mode="lines+text",
        ),
    ],
    layout=go.Layout(
        updatemenus=[
            dict(
                type="buttons",
                buttons=[
                    dict(
                        label="▶",
                        method="animate",
                        args=[
                            None,
                            {
                                "frame": {"duration": 1},
                                "transition": {
                                    "duration": 1,
                                },
                            },
                        ],
                    )
                ],
                direction="left",
                pad={"r": 0, "t": 0},
                x=0.1,
                xanchor="right",
                y=0.91,
                yanchor="top",
            )
        ],
    ),
    frames=[
        go.Frame(
            data=[
                go.Scattergeo(
                    lon=df["lon"][:i],
                    lat=df["lat"][:i],
                    text=modify_t(i),
                ),
            ]
        )
        for i in range(df.shape[0] - 1)
    ]
    + [
        go.Frame(
            data=[
                go.Scattergeo(
                    lon=df["lon"],
                    lat=df["lat"],
                    text=t,
                ),
            ]
        )
    ],
)

fig.add_annotation(
    text="* FL is Flight Level in hundreds of Feet<br>   Speed is Ground Speed in Knots",
    xref="paper",
    yref="paper",
    x=0.94,
    y=0.1,
    font=dict(color="LightGrey"),
    showarrow=False,
    align="left",
)

fig.update_layout(width=800, height=400, margin={"r": 0, "t": 0, "l": 0, "b": 0})
fig.update_layout(
    title_text=f"Flight path for {FLIGHT_NUMBER}",
    title_font=dict(color="LightGrey"),
    title_x=0.12,
    title_y=0.89,
    showlegend=False,
)
fig.update_geos(
    resolution=110,  # 50或110
    showcoastlines=False,
    showland=True,
    landcolor="#242424",
    showocean=True,
    oceancolor="#171717",
)
config = {"displayModeBar": False, "responsive": False, "staticPlot": True}
fig.show(config=config)

Plotly动画图表

Plotly动画图表TYA3422-2

You can go a step further and store this data in a database easily. Please refer to my other tutorial on the matter: 在Python中从API数据自动创建数据库模式

谢谢阅读!

相关推荐
屿华AI18 分钟前
AUTOSAR 实战-诊断工具 | 基于 Python 与 USB-CANFD 的 UDS 诊断测试工具设计与实现
开发语言·python·测试工具
贝壳69022 分钟前
三家 AI 把我的实测数据判成造假,我在同一台机器上重跑了五组
python
用户019027581611 小时前
Telegram Bot 股票到价提醒:AlphaFeed 轮询 + sendMessage
python
黑客笔记2 小时前
Python 进程池和线程池:用等待任务理解并发
开发语言·python
卷无止境2 小时前
Python 开发中的资源管理:统一生命周期,保留资源差异
后端·python
新*2 小时前
520 万数据迁移实战:Elasticsearch 索引重建与"改名"的完整踩坑记录
python·mysql·elasticsearch·数据迁移
吴名氏.3 小时前
Java转AI Agent教程(第十五周)政策时效:过期政策不该被召回,也不能被删掉
java·人工智能·python·ai·ai agent
黑客笔记13 小时前
python小工具win7翻车事故现场
python
岁月宁静15 小时前
从零拆解 AI 语音模拟面试系统
人工智能·python·agent
benchmark_cc15 小时前
个人开发者接入五档盘口:用 QuantDash 读取单只股票的验收清单与易错点
python·数据分析·pandas·量化交易·股票数据·quantdash