15个实用的数据处理脚本，Python与Shell
15个实用的数据处理脚本，Python与Shell
# 15个实用的数据处理脚本，Python与Shell高效结合
## 1. 批量文件重命名脚本（Python）
```python
import os
def batch_rename(path, old_str, new_str):
for file in os.listdir(path):
if old_str in file:
new_file = file.replace(old_str, new_str)
os.rename(
os.path.join(path, file),
os.path.join(path, new_file)
)
print(f"Renamed: {file} -> {new_file}")
# 示例：将目录下所有包含"old_“的文件改为"new_”
batch_rename(“/path/to/files”, “old_”, “new_”)
```
## 2. 文本文件编码转换（Shell）
```bash
#!/bin/bash
# 将GB2312编码转换为UTF-8
for file in *.txt; do
iconv -f GB2312 -t UTF-8 “f i l e " > " file" > "file">"{file%.txt}_utf8.txt”
echo “Converted $file to UTF-8”
done
```
## 3. 数据去重脚本（Python）
```python
def remove_duplicates(input_file, output_file):
seen = set()
with open(input_file, ‘r’) as fin, open(output_file, ‘w’) as fout:
for line in fin:
if line not in seen:
seen.add(line)
fout.write(line)
# 示例使用
remove_duplicates(‘data.txt’, ‘dedup_data.txt’)
```
## 4. 文件批量下载（Shell）
```bash
#!/bin/bash
# 从URL列表下载文件
url_list=“urls.txt”
download_dir=“downloads”
mkdir -p “KaTeX parse error: Undefined control sequence: \- at position 34: …hile IFS= read \̲-̲r url; do …download_dir” “u r l " d o n e < " url" done < "url"done<"url_list”
```
## 5. JSON转CSV工具（Python）
```python
import json
import csv
def json_to_csv(json_file, csv_file):
with open(json_file) as f:
data = json.load(f)
with open(csv_file, ‘w’, newline=‘’) as f:
writer = csv.DictWriter(f, fieldnames=data[0].keys())
writer.writeheader()
writer.writerows(data)
# 示例
json_to_csv(‘data.json’, ‘data.csv’)
```
（限于篇幅，后续脚本只展示核心代码片段）
## 6. 日志文件时间筛选（Shell）
```bash
grep -E “2023-07-[0-9]{2}” access.log > july_logs.txt
```
## 7. 图片批量压缩（Python）
```python
from PIL import Image
import os
def compress_images(input_dir, output_dir, quality=85):
for file in os.listdir(input_dir):
if file.endswith((‘.jpg’, ‘.jpeg’, ‘.png’)):
img = Image.open(os.path.join(input_dir, file))
img.save(os.path.join(output_dir, file), quality=quality)
```
## 8. 系统进程监控（Shell）
```bash
#!/bin/bash
# 监控CPU使用率超过50%的进程
top -b -n1 | awk ‘NR>7 && $9 > 50 {print $1 “\t” $9 “\t” $12}’
```
## 9. Excel数据合并（Python）
```python
import pandas as pd
def merge_excels(file_pattern, output_file):
all_data = pd.DataFrame()
for file in glob.glob(file_pattern):
df = pd.read_excel(file)
all_data = pd.concat([all_data, df])
all_data.to_excel(output_file, index=False)
```
## 10. 批量SSH命令执行（Shell）
```bash
#!/bin/bash
# 在多台服务器上执行相同命令
hosts=(“server1” “server2” “server3”)
command=“df -h”
for host in “${hosts[@]}”; do
echo “===h o s t = = = " s s h " host ===" ssh "host==="ssh"host” “$command”
done
```
## 11. 数据库备份脚本（Shell）
```bash
#!/bin/bash
# MySQL数据库备份
DB_USER=“user”
DB_PASS=“password”
DB_NAME=“database”
BACKUP_DIR=“/backups”
mysqldump -uKaTeX parse error: Undefined control sequence: \- at position 10: DB\_USER \̲-̲pDB_PASSD B _ N A M E ∣ g z i p > " DB\_NAME | gzip > "DB_NAME∣gzip>"BACKUP_DIR/db_$(date +%Y%m%d).sql.gz"
```
## 12. 网页数据抓取（Python）
```python
import requests
from bs4 import BeautifulSoup
def scrape_links(url, selector):
r = requests.get(url)
soup = BeautifulSoup(r.text, ‘html.parser’)
return [a[‘href’] for a in soup.select(selector)]
```
## 13. 磁盘空间告警（Shell）
```bash
#!/bin/bash
threshold=80
df -h | awk -v t=$threshold ‘NR>1 {gsub(/%/,“”); if ($5 > t) print $1 " is " $5 “% full”}’
```
## 14. CSV数据清洗（Python）
```python
import pandas as pd
def clean_csv(input_file, output_file):
df = pd.read_csv(input_file)
# 填充空值
df.fillna(method=‘ffill’, inplace=True)
# 去除重复行
df.drop_duplicates(inplace=True)
df.to_csv(output_file, index=False)
```
## 15. 自动打包部署（Shell）
```bash
#!/bin/bash
# 自动构建并部署项目
project_dir=“/home/user/project”
cd “$project_dir” && \
git pull && \
npm install && \
npm run build && \
rsync -avz dist/ deploy@server:/var/www/html/
```
## 结语
这些脚本覆盖了日常数据处理中的常见需求，可以根据实际情况进行调整。Python适合复杂的数据处理任务，而Shell则在系统管理方面更胜一筹。两种语言结合使用，能大幅提升工作效率。