作者:小小明

Pandas数据处理专家,帮助一万用户解决数据处理难题。

需求分析

寒潮的定义:

image-20210101211230798

数据的输入和输出格式:

image-20210101231950703

统计口径确认:

image-20210101230118474

我一开始不理解,24小时内降温幅度大于8度如何计算,与需求方确认后,可以通过2日温度之差来计算。同样48小时内降温幅度可以用3日温度之差来代表,72小时内降温幅度可以用4日温度之差来代表,需求方的解释:

image-20210101212126002

好了,理解清楚了需求,咱们就可以开始干活了:

读取数据

首先读取数据:

import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")
df

结果:

datenumbertemperature
02004/12/20d289-30.38
12004/12/21d289-32.67
22004/12/22d289-33.12
32004/12/23d289-31.71
42004/12/24d289-32.76
122562005/12/16e332-8.37
122572005/12/17e332-12.73
122582005/12/18e332-13.57
122592005/12/19e332-13.36
122602005/12/20e332-15.99

拿一个分组进行测试

取出某个分组,用于测试:

tmp = df.groupby('number').get_group('e332')
tmp

结果:

datenumbertemperature
119262005/1/20e332-18.06
119272005/1/21e332-8.76
119282005/1/22e332-4.77
119292005/1/23e332-10.81
119302005/1/24e332-19.91
122562005/12/16e332-8.37
122572005/12/17e332-12.73
122582005/12/18e332-13.57
122592005/12/19e332-13.36
122602005/12/20e332-15.99

获取满足寒潮定义条件的对应数据id

image-20210101211656200

上图的极端情况显示,三大满足条件的id可能出现重复的情况,所以我使用了set这个无序不重复集合来保存id:

cold_wave_idxs = set()
# 获取2天内降温幅度超过8对应的数据id
ids = tmp.index[tmp.temperature.diff(-1) >= 8].values
cold_wave_idxs.update(ids)
cold_wave_idxs.update(ids+1)
# 获取3天内降温幅度超过10对应的数据id
ids = tmp.index[tmp.temperature.diff(-2) >= 10].values
cold_wave_idxs.update(ids)
cold_wave_idxs.update(ids+1)
cold_wave_idxs.update(ids+2)
# 获取4天内降温幅度超过12对应的数据id
ids = tmp.index[tmp.temperature.diff(-3) >= 12].values
cold_wave_idxs.update(ids)
cold_wave_idxs.update(ids+1)
cold_wave_idxs.update(ids+2)
cold_wave_idxs.update(ids+3)
# 排序并转换成列表
cold_wave_idxs = sorted(cold_wave_idxs)
print(cold_wave_idxs)

结果:

[11928, 11929, 11930, 11931, 11939, 11940, 11949, 11950, 11951, 11952, 11955, 11956, 11957, 11958, 12007, 12008, 12154, 12155, 12192, 12193, 12201, 12202, 12203, 12223, 12224, 12225, 12228, 12229, 12230]

上述代码中cold_wave_idxs.update(ids+1)表示,把ids列表里每个id的后一个id也添加到最终列表里,利用了numpy数组广播变量的特性,+2和+3也是同理。

上述结果就是从站码为’e332’的分组中计算出满足寒潮定义的对应数据id。

从结果可以看出,凡是连续的id都可以看作一个寒潮的过程,所以现在我们需要将每个寒潮过程都分为一组,为了作这样的分组,我发明了一种分组编号生成器的写法,下面已经封装成了一个方法:

分组编号生成器

def generate_group_num(values, diff=1):
    group_ids = []
    group_id = 0
    last_v = 0
    for value in values:
        if value-last_v > diff:
            group_id += 1
        group_ids.append(group_id)
        last_v = value
    return group_ids

上面的方法实现了一个分组编号生成器,对于一段序列凡是连续的数字都会给一个相同的分组编号。

测试一下分组效果:

for i, cold_wave_idx_serial in pd.Series(cold_wave_idxs).groupby(generate_group_num(cold_wave_idxs)):
    cold_wave_idx_serial = cold_wave_idx_serial.values
    print(cold_wave_idx_serial)

结果:

[11928 11929 11930 11931]
[11939 11940]
[11949 11950 11951 11952]
[11955 11956 11957 11958]
[12007 12008]
[12154 12155]
[12192 12193]
[12201 12202 12203]
[12223 12224 12225]
[12228 12229 12230]

从结果可以看到,凡是连续的序列都分到了一组,不是连续的序列就没有分到一组。

测试对所有站计算寒潮

首先将前面的测试好的用于获取满足寒潮定义的id的过程封装成方法:

def get_cold_wave_idxs(df, cold_wave_level=(8, 10, 12)):
    cold_wave_idxs = set()
    ids = df.index[df.temperature.diff(-1) >= cold_wave_level[0]].values
    cold_wave_idxs.update(ids)
    cold_wave_idxs.update(ids+1)
    ids = df.index[df.temperature.diff(-2) >= cold_wave_level[1]].values
    cold_wave_idxs.update(ids)
    cold_wave_idxs.update(ids+1)
    cold_wave_idxs.update(ids+2)
    ids = df.index[df.temperature.diff(-3) >= cold_wave_level[2]].values
    cold_wave_idxs.update(ids)
    cold_wave_idxs.update(ids+1)
    cold_wave_idxs.update(ids+2)
    cold_wave_idxs.update(ids+3)
    return sorted(cold_wave_idxs)

然后运行:

cold_wave_result = []

for number, tmp in df.groupby('number'):
    cold_wave_idxs = get_cold_wave_idxs(tmp, (8, 10, 12))
    for i, cold_wave_idx_serial in pd.Series(cold_wave_idxs).groupby(generate_group_num(cold_wave_idxs)):
        cold_wave_idx_serial = cold_wave_idx_serial.values
        start_id, end_id = cold_wave_idx_serial[0], cold_wave_idx_serial[-1]
        #  假如最低温度小于4度,则说明满足全部条件
        if tmp.loc[end_id, 'temperature'] <= 4:
            cold_wave_result.append(
                (number, tmp.loc[start_id, 'date'], tmp.loc[end_id, 'date'],
                 tmp.loc[start_id, 'temperature'], tmp.loc[end_id, 'temperature'],
                 end_id-start_id+1,
                 tmp.loc[start_id, 'temperature'] -tmp.loc[end_id, 'temperature'],
                 '寒潮'
                 )
            )
cold_wave_result = pd.DataFrame(cold_wave_result, columns=[
    '站号', '开始日期', '结束日期', '开始温度', '结束温度',  '寒潮天数', '温度差', '寒潮类型'])
cold_wave_result

结果:

站号开始日期结束日期开始温度结束温度寒潮天数温度差寒潮类型
0d2892005/1/22005/1/4-19.39-30.79311.40寒潮
1d2892005/1/102005/1/13-21.66-36.06414.40寒潮
2d2892005/1/212005/1/24-12.28-27.40415.12寒潮
3d2892005/2/72005/2/8-15.31-26.47211.16寒潮
4d2892005/2/152005/2/17-12.12-23.52311.40寒潮
395e3322005/9/52005/9/62.68-6.0028.68寒潮
396e3322005/10/132005/10/14-2.52-12.76210.24寒潮
397e3322005/10/222005/10/24-0.69-12.90312.21寒潮
398e3322005/11/132005/11/15-5.96-16.56310.60寒潮
399e3322005/11/182005/11/20-7.31-18.74311.43寒潮

感觉没啥问题。

所有寒潮级别都测试一下:

测试所有寒潮级别

cold_wave_all = [
    {
        'cold_wave_temperature_diffs': (8, 10, 12),
        'min_temperature_limit': 4,
        'cold_wave_type': '寒潮'
    },
    {
        'cold_wave_temperature_diffs': (10, 12, 14),
        'min_temperature_limit': 2,
        'cold_wave_type': '强寒潮'
    },
    {
        'cold_wave_temperature_diffs': (12, 14, 16),
        'min_temperature_limit': 0,
        'cold_wave_type': '超强寒潮'
    }
]
cold_wave_result = []

for number, tmp in df.groupby('number'):
    for cold_wave_dict in cold_wave_all:
        cold_wave_idxs = get_cold_wave_idxs(tmp, cold_wave_dict['cold_wave_temperature_diffs'])
        if len(cold_wave_idxs) < 2:
            continue
        for i, cold_wave_idx_serial in pd.Series(cold_wave_idxs).groupby(generate_group_num(cold_wave_idxs)):
            cold_wave_idx_serial = cold_wave_idx_serial.values
            start_id, end_id = cold_wave_idx_serial[0], cold_wave_idx_serial[-1]
            #  假如最低温度小于指定度数,则说明满足全部条件
            if tmp.loc[end_id, 'temperature'] <= cold_wave_dict['min_temperature_limit']:
                cold_wave_result.append(
                    (number, tmp.loc[start_id, 'date'], tmp.loc[end_id, 'date'],
                     tmp.loc[start_id, 'temperature'], tmp.loc[end_id, 'temperature'],
                     end_id-start_id+1,
                     tmp.loc[start_id, 'temperature'] - tmp.loc[end_id, 'temperature'],
                     cold_wave_dict['cold_wave_type']
                     )
                )
cold_wave_result = pd.DataFrame(cold_wave_result, columns=[
    '站号', '开始日期', '结束日期', '开始温度', '结束温度',  '寒潮天数', '温度差', '寒潮类型'])
cold_wave_result

结果:

站号开始日期结束日期开始温度结束温度寒潮天数温度差寒潮类型
0d2892005/1/22005/1/4-19.39-30.79311.40寒潮
1d2892005/1/102005/1/13-21.66-36.06414.40寒潮
2d2892005/1/212005/1/24-12.28-27.40415.12寒潮
3d2892005/2/72005/2/8-15.31-26.47211.16寒潮
4d2892005/2/152005/2/17-12.12-23.52311.40寒潮
636e3322005/2/22005/2/3-7.36-20.59213.23强寒潮
637e3322005/10/132005/10/14-2.52-12.76210.24强寒潮
638e3322005/10/222005/10/24-0.69-12.90312.21强寒潮
639e3322005/1/222005/1/24-4.77-19.91315.14超强寒潮
640e3322005/2/22005/2/3-7.36-20.59213.23超强寒潮

暂时也未发现错误。那么整理一下最终代码吧:

完整代码

import pandas as pd
import numpy as np


def generate_group_num(values, diff=1):
    group_ids = []
    group_id = 0
    last_v = 0
    for value in values:
        if value-last_v > diff:
            group_id += 1
        group_ids.append(group_id)
        last_v = value
    return group_ids


def get_cold_wave_idxs(df, cold_wave_level=(8, 10, 12)):
    cold_wave_idxs = set()
    ids = df.index[df.temperature.diff(-1) >= cold_wave_level[0]].values
    cold_wave_idxs.update(ids)
    cold_wave_idxs.update(ids+1)
    ids = df.index[df.temperature.diff(-2) >= cold_wave_level[1]].values
    cold_wave_idxs.update(ids)
    cold_wave_idxs.update(ids+1)
    cold_wave_idxs.update(ids+2)
    ids = df.index[df.temperature.diff(-3) >= cold_wave_level[2]].values
    cold_wave_idxs.update(ids)
    cold_wave_idxs.update(ids+1)
    cold_wave_idxs.update(ids+2)
    cold_wave_idxs.update(ids+3)
    return sorted(cold_wave_idxs)


df = pd.read_csv("data.csv")
cold_wave_all = [
    {
        'cold_wave_temperature_diffs': (8, 10, 12),
        'min_temperature_limit': 4,
        'cold_wave_type': '寒潮'
    },
    {
        'cold_wave_temperature_diffs': (10, 12, 14),
        'min_temperature_limit': 2,
        'cold_wave_type': '强寒潮'
    },
    {
        'cold_wave_temperature_diffs': (12, 14, 16),
        'min_temperature_limit': 0,
        'cold_wave_type': '超强寒潮'
    }
]
cold_wave_result = []

for number, tmp in df.groupby('number'):
    for cold_wave_dict in cold_wave_all:
        cold_wave_idxs = get_cold_wave_idxs(tmp, cold_wave_dict['cold_wave_temperature_diffs'])
        if len(cold_wave_idxs) < 2:
            continue
        for i, cold_wave_idx_serial in pd.Series(cold_wave_idxs).groupby(generate_group_num(cold_wave_idxs)):
            cold_wave_idx_serial = cold_wave_idx_serial.values
            start_id, end_id = cold_wave_idx_serial[0], cold_wave_idx_serial[-1]
            #  假如最低温度小于指定度数,则说明满足全部条件
            if tmp.loc[end_id, 'temperature'] <= cold_wave_dict['min_temperature_limit']:
                cold_wave_result.append(
                    (number, tmp.loc[start_id, 'date'], tmp.loc[end_id, 'date'],
                     tmp.loc[start_id, 'temperature'], tmp.loc[end_id, 'temperature'],
                     end_id-start_id+1,
                     tmp.loc[start_id, 'temperature'] - tmp.loc[end_id, 'temperature'],
                     cold_wave_dict['cold_wave_type']
                     )
                )
cold_wave_result = pd.DataFrame(cold_wave_result, columns=[
    '站号', '开始日期', '结束日期', '开始温度', '结束温度',  '寒潮天数', '温度差', '寒潮类型'])
cold_wave_result.to_excel("cold_wave.xlsx", index=False)

最终得到的结果:

image-20210101232323036

(前60行数据)

本文完结!


本文转载:CSDN博客