从数据映射到文件生成：一个R语言实践案例

背景

在处理科学计算数据时，我们经常需要将不同来源的数据整合并转换成特定格式。本文将分享一个实际案例，展示如何通过理解复杂的数据映射需求，实现正确的文件生成逻辑。

问题描述

我们需要根据一个映射配置文件生成多个数据文件。具体要求如下：

1. 输入文件：

   • 映射配置文件（CSV格式），包含Row、Col和CombiNo三列
   • 多个数据源文件，存放在不同目录下，包含日期和对应的数据值

2. 输出要求：

   • 生成特定格式的数据文件
   • 每个时间序列生成固定行数的数据
   • 每行包含66个数值
   • 数值的填充根据映射配置决定

理解数据映射逻辑

起初，这个需求看起来并不复杂。但当我们深入分析时，发现了几个关键点：

1. 映射关系：

   Row Col CombiNo
   1   1   -999
   1   2   240
   

   • Row表示输出文件中的行号
   • Col表示该行中的列号（1-66）
   • CombiNo表示数据来源，-999表示填0，其他值需要从对应文件获取数据

2. 数据源文件：

   • 文件路径格式：Weekly_Stats/[场景]/[模型]/*_[模型]_[后缀]_[CombiNo].csv
   • 文件内容：包含Date和NetDrain等列
   • 需要根据日期取对应的NetDrain值

代码实现

让我们看看关键部分的实现：

1. 文件查找逻辑：

get_combo_data <- function(model, scenario, suffix, combo_no) {
  base_dir <- file.path("Weekly_Stats", scenario, model)
  
  # 构建文件匹配模式
  if(scenario == "Hindcast") {
    pattern <- paste0("\\d+_\\d+-.*result_\\d+_", model, "_00_", combo_no, "\\.csv$")
  } else {
    pattern <- paste0("\\d+_\\d+CC-.*result_\\d+_", model, "_", suffix, "_", combo_no, "\\.csv$")
  }
  
  files <- list.files(base_dir, pattern=pattern, full.names=TRUE)
  ...
}

2. 数据生成逻辑：

# 对每个时间序列
for(week_idx in 1:length(dates)) {
  current_date <- dates[week_idx]
  
  # 处理每一行
  for(row in 1:max_row) {
    row_values <- numeric(66)
    row_data <- rc_combi[rc_combi$Row == row,]
    
    # 根据映射填充数据
    for(j in 1:nrow(row_data)) {
      col <- row_data$Col[j]
      combo_no <- row_data$ComnbiNo[j]
      
      if(combo_no != -999) {
        combo_data <- combo_data_list[[as.character(combo_no)]]
        if(!is.null(combo_data)) {
          week_data <- combo_data[combo_data$Date == current_date,]
          if(nrow(week_data) > 0) {
            row_values[col] <- week_data$NetDrain
          }
        }
      }
    }
  }
}

关键优化点

在实现过程中，我们注意到几个需要优化的点：

1. 动态行数：

   • 不应硬编码输出文件的行数
   • 应该从映射配置文件中获取最大Row值

2. 输出格式：

   • 确保数值格式符合要求（科学计数法）
   • 控制空格数量满足规范

经验总结

1. 需求理解很重要：

   • 透彻理解映射规则
   • 理清数据来源和格式
   • 确认特殊值的处理方式

2. 代码实现要注意：

   • 避免硬编码关键参数
   • 保持代码的可维护性
   • 增加必要的错误处理

3. 验证很关键：

   • 确认文件查找逻辑正确
   • 验证数据映射准确性
   • 检查输出格式是否符合要求

---

From Data Mapping to File Generation: A Case Study in R

Background

In scientific computing, we often need to integrate data from different sources and convert them into specific formats. This article shares a practical case study demonstrating how to understand complex data mapping requirements and implement correct file generation logic.

Problem Description

We need to generate multiple data files based on a mapping configuration file. The specific requirements are:

1. Input Files:

   • Mapping configuration file (CSV format) containing Row, Col, and CombiNo columns
   • Multiple source data files in different directories containing dates and corresponding data values

2. Output Requirements:

   • Generate data files in a specific format
   • Generate fixed number of rows for each time series
   • Each row contains 66 values
   • Values are filled according to the mapping configuration

Understanding Data Mapping Logic

Initially, this requirement seemed straightforward. However, when we analyzed it deeply, we discovered several key points:

1. Mapping Relationship:

   Row Col CombiNo
   1   1   -999
   1   2   240
   

   • Row indicates the row number in the output file
   • Col indicates the column number in that row (1-66)
   • CombiNo indicates the data source, -999 means fill with 0, other values need to get data from corresponding files

2. Source Data Files:

   • File path format: Weekly_Stats/[scenario]/[model]/*_[model]_[suffix]_[CombiNo].csv
   • File content: includes Date and NetDrain columns
   • Need to get NetDrain value based on date

Code Implementation

Let's look at the key parts of the implementation:

1. File Finding Logic:

get_combo_data <- function(model, scenario, suffix, combo_no) {
  base_dir <- file.path("Weekly_Stats", scenario, model)
  
  # Build file matching pattern
  if(scenario == "Hindcast") {
    pattern <- paste0("\\d+_\\d+-.*result_\\d+_", model, "_00_", combo_no, "\\.csv$")
  } else {
    pattern <- paste0("\\d+_\\d+CC-.*result_\\d+_", model, "_", suffix, "_", combo_no, "\\.csv$")
  }
  
  files <- list.files(base_dir, pattern=pattern, full.names=TRUE)
  ...
}

2. Data Generation Logic:

# For each time series
for(week_idx in 1:length(dates)) {
  current_date <- dates[week_idx]
  
  # Process each row
  for(row in 1:max_row) {
    row_values <- numeric(66)
    row_data <- rc_combi[rc_combi$Row == row,]
    
    # Fill data according to mapping
    for(j in 1:nrow(row_data)) {
      col <- row_data$Col[j]
      combo_no <- row_data$ComnbiNo[j]
      
      if(combo_no != -999) {
        combo_data <- combo_data_list[[as.character(combo_no)]]
        if(!is.null(combo_data)) {
          week_data <- combo_data[combo_data$Date == current_date,]
          if(nrow(week_data) > 0) {
            row_values[col] <- week_data$NetDrain
          }
        }
      }
    }
  }
}

Key Optimization Points

During implementation, we noticed several points that needed optimization:

1. Dynamic Row Count:

   • Should not hardcode the number of rows in output file
   • Should get maximum Row value from mapping configuration file

2. Output Format:

   • Ensure numeric format meets requirements (scientific notation)
   • Control number of spaces to meet specifications

Lessons Learned

1. Requirement Understanding is Crucial:

   • Thoroughly understand mapping rules
   • Clarify data sources and formats
   • Confirm special value handling

2. Code Implementation Considerations:

   • Avoid hardcoding key parameters
   • Maintain code maintainability
   • Add necessary error handling

3. Verification is Key:

   • Confirm file finding logic is correct
   • Verify data mapping accuracy
   • Check output format compliance