NanmiCoder/MediaCrawler
View on GitHub ↗
Internal82
Unique within the repo — the other 18% repeats across its own files.
Public65
Unique across all public code — a further 17% is its own but appears in other public repos too.
Original and tidy — it rarely repeats itself, and little of it turns up in other public code. About as close to genuinely its own work as this metric can measure.
Worth consolidating — in the code
Real, actionable duplication in this repo's source, most concrete first.
Top duplication hotspots
| Similarity | Overlap | File A | File B | Cross-repo |
|---|---|---|---|---|
| 33% | 478 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py |
NanmiCoder/MediaCrawler/media_platform/kuaishou/core.py |
same repo |
| 35% | 440 | NanmiCoder/MediaCrawler/media_platform/weibo/core.py |
NanmiCoder/MediaCrawler/media_platform/xhs/core.py |
same repo |
| 38% | 428 | NanmiCoder/MediaCrawler/media_platform/xhs/core.py |
NanmiCoder/MediaCrawler/media_platform/zhihu/core.py |
same repo |
| 26% | 419 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py |
NanmiCoder/MediaCrawler/media_platform/weibo/core.py |
same repo |
| 37% | 414 | NanmiCoder/MediaCrawler/media_platform/kuaishou/core.py |
NanmiCoder/MediaCrawler/media_platform/xhs/core.py |
same repo |
| 26% | 413 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py |
NanmiCoder/MediaCrawler/media_platform/douyin/core.py |
same repo |
| 33% | 410 | NanmiCoder/MediaCrawler/media_platform/douyin/core.py |
NanmiCoder/MediaCrawler/media_platform/xhs/core.py |
same repo |
| 25% | 409 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py |
NanmiCoder/MediaCrawler/media_platform/xhs/core.py |
same repo |
| 35% | 400 | NanmiCoder/MediaCrawler/media_platform/kuaishou/core.py |
NanmiCoder/MediaCrawler/media_platform/weibo/core.py |
same repo |
| 35% | 398 | NanmiCoder/MediaCrawler/media_platform/douyin/core.py |
NanmiCoder/MediaCrawler/media_platform/kuaishou/core.py |
same repo |
| 28% | 394 | NanmiCoder/MediaCrawler/media_platform/tieba/core.py |
NanmiCoder/MediaCrawler/media_platform/xhs/core.py |
same repo |
| 33% | 392 | NanmiCoder/MediaCrawler/media_platform/weibo/core.py |
NanmiCoder/MediaCrawler/media_platform/zhihu/core.py |
same repo |
| 36% | 384 | NanmiCoder/MediaCrawler/media_platform/kuaishou/core.py |
NanmiCoder/MediaCrawler/media_platform/zhihu/core.py |
same repo |
| 25% | 380 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py |
NanmiCoder/MediaCrawler/media_platform/zhihu/core.py |
same repo |
| 24% | 377 | NanmiCoder/MediaCrawler/media_platform/weibo/client.py |
NanmiCoder/MediaCrawler/media_platform/xhs/client.py |
same repo |
| 28% | 372 | NanmiCoder/MediaCrawler/media_platform/douyin/core.py |
NanmiCoder/MediaCrawler/media_platform/weibo/core.py |
same repo |
| 25% | 368 | NanmiCoder/MediaCrawler/media_platform/tieba/core.py |
NanmiCoder/MediaCrawler/media_platform/weibo/core.py |
same repo |
| 27% | 360 | NanmiCoder/MediaCrawler/media_platform/tieba/core.py |
NanmiCoder/MediaCrawler/media_platform/zhihu/core.py |
same repo |
| 19% | 339 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py |
NanmiCoder/MediaCrawler/media_platform/tieba/core.py |
same repo |
| 25% | 336 | NanmiCoder/MediaCrawler/media_platform/kuaishou/core.py |
NanmiCoder/MediaCrawler/media_platform/tieba/core.py |
same repo |
Docs & prose
Near-identical documentation — often translations (e.g. README.md ⇄ README_zh-CN.md) or mirrored guides. Usually intentional, not cleanup — listed separately so it doesn't inflate the code signal above.
Top duplication hotspots
| Similarity | Overlap | File A | File B | Cross-repo |
|---|---|---|---|---|
| 15% | 247 | NanmiCoder/MediaCrawler/README.md |
NanmiCoder/MediaCrawler/README_en.md |
same repo |
| 10% | 241 | NanmiCoder/MediaCrawler/README_en.md |
NanmiCoder/MediaCrawler/README_es.md |
same repo |
| 13% | 212 | NanmiCoder/MediaCrawler/README.md |
NanmiCoder/MediaCrawler/README_es.md |
same repo |
| 61% | 79 | NanmiCoder/MediaCrawler/docs/index.md |
NanmiCoder/MediaCrawler/docs/原生环境管理文档.md |
same repo |
| 5% | 50 | NanmiCoder/MediaCrawler/docs/项目代码结构.md |
NanmiCoder/MediaCrawler/docs/项目架构文档.md |
same repo |
| 7% | 46 | NanmiCoder/MediaCrawler/README.md |
NanmiCoder/MediaCrawler/docs/index.md |
same repo |
| 7% | 43 | NanmiCoder/MediaCrawler/README.md |
NanmiCoder/MediaCrawler/docs/原生环境管理文档.md |
same repo |
| 3% | 43 | NanmiCoder/MediaCrawler/README_es.md |
NanmiCoder/MediaCrawler/docs/index.md |
same repo |
| 3% | 41 | NanmiCoder/MediaCrawler/README_en.md |
NanmiCoder/MediaCrawler/docs/index.md |
same repo |
| 3% | 40 | NanmiCoder/MediaCrawler/README_es.md |
NanmiCoder/MediaCrawler/docs/原生环境管理文档.md |
same repo |
| 3% | 38 | NanmiCoder/MediaCrawler/README_en.md |
NanmiCoder/MediaCrawler/docs/原生环境管理文档.md |
same repo |
| 2% | 31 | NanmiCoder/MediaCrawler/README_en.md |
NanmiCoder/MediaCrawler/docs/excel_export_guide.md |
same repo |
| 48% | 30 | NanmiCoder/MediaCrawler/.github/ISSUE_TEMPLATE/bug_report.md |
NanmiCoder/MediaCrawler/.github/ISSUE_TEMPLATE/quesiton.md |
same repo |
| 4% | 29 | NanmiCoder/MediaCrawler/README.md |
NanmiCoder/MediaCrawler/docs/CDP模式使用指南.md |
same repo |
| 10% | 20 | NanmiCoder/MediaCrawler/docs/data_storage_guide.md |
NanmiCoder/MediaCrawler/docs/index.md |
same repo |
| 5% | 17 | NanmiCoder/MediaCrawler/docs/CDP模式使用指南.md |
NanmiCoder/MediaCrawler/docs/常见问题.md |
same repo |
| 3% | 17 | NanmiCoder/MediaCrawler/README.md |
NanmiCoder/MediaCrawler/docs/常见问题.md |
same repo |
| 1% | 17 | NanmiCoder/MediaCrawler/README_en.md |
NanmiCoder/MediaCrawler/docs/CDP模式使用指南.md |
same repo |
| 20% | 16 | NanmiCoder/MediaCrawler/docs/快代理使用文档.md |
NanmiCoder/MediaCrawler/docs/豌豆HTTP使用文档.md |
same repo |
| 1% | 16 | NanmiCoder/MediaCrawler/README_es.md |
NanmiCoder/MediaCrawler/docs/CDP模式使用指南.md |
same repo |
Largest files
Largest files
| LOC | File | Status |
|---|---|---|
| 894 | NanmiCoder/MediaCrawler/media_platform/tieba/help.py | indexed |
| 884 | NanmiCoder/MediaCrawler/docs/项目架构文档.md | indexed |
| 865 | NanmiCoder/MediaCrawler/media_platform/tieba/client.py | indexed |
| 768 | NanmiCoder/MediaCrawler/docs/hit_stopwords.txt | indexed |
| 726 | NanmiCoder/MediaCrawler/media_platform/bilibili/core.py | indexed |
| 704 | NanmiCoder/MediaCrawler/media_platform/xhs/client.py | indexed |
| 686 | NanmiCoder/MediaCrawler/media_platform/tieba/core.py | indexed |
| 602 | NanmiCoder/MediaCrawler/media_platform/zhihu/client.py | indexed |
| 580 | NanmiCoder/MediaCrawler/media_platform/bilibili/client.py | indexed |
| 521 | NanmiCoder/MediaCrawler/media_platform/xhs/core.py | indexed |