solution-erp/.claude/skills/iis-deploy-runbook/SKILL.md

name, description, when-to-use

name	description	when-to-use
iis-deploy-runbook	Ops runbook cho SOLUTION_ERP deploy trên Windows Server IIS — 3 site (api/admin/eoffice.solutions.com.vn), win-acme Let's Encrypt, NSSM gitea-runner shared với VIETREPORT, LibreOffice soffice headless. Dùng khi debug 500/502 prod, restart site, rotate cert, fix CI/CD runner, troubleshoot WebSocket, thêm site mới.	prod 500 error IIS site fail cert hết hạn win-acme gitea runner deploy IIS restart app pool webSocket 500 reverse proxy FE LibreOffice prod

IIS Deploy Runbook — SOLUTION_ERP

Context: VPS Windows Server shared với VIETREPORT project. IIS + URL Rewrite + ARR + WebSockets module + win-acme. Deploy qua Gitea Actions self-hosted runner.

Production topology

Internet
   │ 443 (HTTPS)
   ▼
┌─────────────────────────────────────────────────────┐
│ IIS (Windows Server VPS)                            │
│                                                      │
│ ┌─ api.solutions.com.vn  ─┐  ┌─ admin.solutions.com.vn ─┐  ┌─ eoffice.solutions.com.vn ─┐
│ │ SolutionErp-Api    │  │ SolutionErp-Admin  │  │ SolutionErp-User  │
│ │ → out-of-process   │  │ (static SPA, URL   │  │ (static SPA, URL  │
│ │   Kestrel :5443    │  │  Rewrite /api → 5443)│ │  Rewrite...)      │
│ │ ASP.NET Core 10   │  │ React build/        │  │ React build/      │
│ │                    │  │                    │  │                    │
│ └────────────────────┘  └────────────────────┘  └────────────────────┘
│                                                      │
│ Let's Encrypt (win-acme) — 3 cert auto-renew 60d    │
│ Shared gitea-runner NSSM service (with VIETREPORT)  │
│ LibreOffice 25.8.6 headless                         │
│ SQL Server 2019 Express (\\.\SQLEXPRESS)            │
└─────────────────────────────────────────────────────┘

3 IIS sites

Site	Binding	Physical path	Apool	Purpose
SolutionErp-Api	*:443:api.solutions.com.vn (+api.huypham.vn) HTTPS	C:\inetpub\solution-erp\api\	out-of-process Kestrel	ASP.NET Core 10 API (port 5443 internal)
SolutionErp-Admin	*:443:admin.solutions.com.vn (+admin.huypham.vn) HTTPS + *:80 redirect	C:\inetpub\solution-erp\fe-admin\	static (no app pool .NET)	React build fe-admin
SolutionErp-User	*:443:eoffice.solutions.com.vn HTTPS + *:80 redirect	C:\inetpub\solution-erp\fe-user\	static	React build fe-user

⚠️ Physical paths verified via appcmd list site S88 — runbook trước ghi C:\inetpub\apps\SolutionErp\{Api,Admin,User} (STALE, dir rỗng). Đường THẬT = C:\inetpub\solution-erp\{api,fe-admin,fe-user}.

SPA web.config ({fe-admin,fe-user}/public/web.config, version-controlled S88 — Vite copy public/*→dist/→deploy Copy-Item -Force; deploy.yml cũ Remove-Item …\* -Exclude web.config = trước S88 file manage-tay trên server). Rule THẬT (đọc live S88):

1. HTTP → HTTPS redirect ({HTTPS} ^OFF$, Permanent; CORS whitelist chỉ https). KHÔNG có .well-known exclude (win-acme validation khác).
2. SPA Routes fallback: non-file/non-dir/non-^/api → rewrite / (index.html). (KHÔNG có /api hay /hubs ARR proxy — FE gọi thẳng https://api.solutions.com.vn qua CORS.)
3. .webmanifest mime + 3 security header (X-Content-Type-Options / X-Frame-Options DENY / Referrer-Policy).
4. [S88 hardening] Cache: root <clientCache cacheControlMode="DisableCache"> → index.html (+ SPA-rewrite shell /dashboard…) = Cache-Control: no-cache (luôn revalidate → chống stale-shell-after-deploy = white-screen S88) · <location path="assets"><clientCache UseMaxAge 365d + cacheControlCustom="public, immutable"> → hashed asset immutable, max-age=1y. Verify: curl -I https://eoffice.solutions.com.vn/ → no-cache; curl -I .../assets/index-*.js → immutable; /dashboard → 200 (routing intact).

Quick commands

Restart 1 site

# PowerShell as Admin
Import-Module WebAdministration
Stop-WebSite -Name "SolutionErp-Api"
Start-WebSite -Name "SolutionErp-Api"

# Hoặc recycle app pool (API out-of-process):
Restart-WebAppPool -Name "SolutionErp-Api"

# Check site status:
Get-Website -Name "SolutionErp-*" | Format-Table Name, State, Bindings

Xem log API

# Serilog file rolling daily (path ĐO THẬT @S189 2026-08-11 — bản cũ ghi `Logs\log-*.txt` SAI;
# config sống: appsettings.Production.json:42 `logs/solution-erp-.log`, retained 30, roll 100MB)
Get-Content "C:\inetpub\solution-erp\api\logs\solution-erp-$(Get-Date -Format 'yyyyMMdd').log" -Tail 50

# IIS log
Get-Content "C:\inetpub\logs\LogFiles\W3SVC<ID>\u_ex$(Get-Date -Format 'yyMMdd').log" -Tail 30

# Stdout log khi crash startup
Get-Content "C:\inetpub\solution-erp\api\Logs\stdout_*.log" -Tail 30

Health check

# Từ server
curl http://127.0.0.1:5443/health/live
curl http://127.0.0.1:5443/health/ready

# Từ ngoài
curl https://api.solutions.com.vn/health/ready

🔴 Disk-free + orphan guard (S189 2026-08-11 — sự cố YC-028, PHẢI check khi prod chậm/500)

# 1. Disk C: — sự cố thật: 0 byte (repo-archive Gitea 16,5GB + npm-cache SYSTEM 12,8GB)
#    → SQL Error -2 (không mở nổi connection) → FE hiện "Không tìm thấy phiếu" (#44 class)
fsutil volume diskfree C:          # free < 5GB = ĐỎ; cache tự phình: C:\gitea\data\repo-archive
                                   # (xóa an toàn — Gitea tự tái tạo) + npm-cache SYSTEM (CI)

# 2. CPU orphan — sự cố thật: 3 sshd mồ côi (ssh bị cắt output giữa chừng) ăn 3/3 core
#    → mọi API 5-47s, SQL wait ASYNC_NETWORK_IO (client đói CPU không nhận nổi data)
$a=@{}; Get-Process | ForEach-Object { $a[$_.Id]=$_.CPU }; Start-Sleep 5
Get-Process | ForEach-Object { if($a.ContainsKey($_.Id)){ $d=$_.CPU-$a[$_.Id]; if($d -ge 1){ "$($_.Name) $($_.Id) $([math]::Round($d,1))" } } }
# sshd/powershell lạ ăn ≥1 core-s/5s → kill (CHỪA chuỗi tổ tiên session mình)

# 3. NTFS compression trên .mdf/.ldf (gotcha #88 — "giải phóng dung lượng" nén khi ổ cạn)
Get-ChildItem 'C:\Program Files\Microsoft SQL Server\MSSQL16.SQLEXPRESS\MSSQL\DATA' -File |
  Where-Object { $_.Attributes -match 'Compressed' }   # có kết quả = compact /u (DB offline nếu đang mở)

# 4. AUTO_CLOSE phải OFF (đã set @S189 — ON làm mỗi connection mở lại DB ~5s)
sqlcmd -S .\SQLEXPRESS -E -W -Q "SELECT name,is_auto_close_on FROM sys.databases WHERE database_id>4"

🔴 Bài ssh … | head (S189): cắt output sớm làm ssh client thoát nhưng process phía Windows-OpenSSH chạy tiếp → orphan spin ăn core. Lệnh ssh dài: bọc Select-Object -First N phía REMOTE, đừng | head phía local.

Let's Encrypt cert — win-acme

Check trạng thái

# Mở win-acme interactive
& "C:\tools\win-acme\wacs.exe"
# Menu > Manage renewals > list — xem 3 cert + next renew date

# Hoặc file:
Get-Content "C:\ProgramData\win-acme\Production\$(hostname)\Renewals\*.renewal.json"

Cert hết hạn emergency

# Force renew 1 cert
& "C:\tools\win-acme\wacs.exe" --renew --force --id {renewal-id}

# Full re-issue nếu renewal fail:
& "C:\tools\win-acme\wacs.exe"  # interactive → 'N' create new
# Chọn: HTTP validation, web root = site physical path, auto install IIS

Gotcha: Shared runner với VIETREPORT → win-acme HTTP challenge cần .well-known/acme-challenge/ accessible qua HTTP (port 80). Rule HTTP→HTTPS redirect trong web.config PHẢI exclude path này:

<rule name="Redirect to HTTPS" stopProcessing="true">
  <match url="(.*)" />
  <conditions>
    <add input="{HTTPS}" pattern="off" />
    <add input="{REQUEST_URI}" pattern="^/\.well-known/" negate="true" />
  </conditions>
  <action type="Redirect" url="https://{HTTP_HOST}/{R:1}" />
</rule>

Gitea Actions runner (NSSM service)

Status

# NSSM service name: gitea-runner (shared với VIETREPORT)
Get-Service gitea-runner
nssm status gitea-runner

# Restart
Restart-Service gitea-runner

# Log
Get-Content "C:\tools\gitea-runner\logs\act_runner.log" -Tail 50

Token rotate (nếu runner disconnected)

# Stop service
Stop-Service gitea-runner

# Re-register qua Gitea admin UI → Actions → Runners → get new registration token
& "C:\tools\gitea-runner\act_runner.exe" register `
    --instance https://git.baocaogiaoduc.vn `
    --token <new-token> `
    --no-interactive

# Start lại
Start-Service gitea-runner

LibreOffice headless (PDF / docx converter)

Check install

& "C:\Program Files\LibreOffice\program\soffice.exe" --version
# → LibreOffice 25.8.6.x

Test convert manual

# Tạo temp dir isolated (mô phỏng per-request pattern của LibreOfficeDocumentConverter)
$work = New-Item -ItemType Directory -Path "$env:TEMP\lo-test-$(Get-Random)"
$userInst = "$work\userinst"

& "C:\Program Files\LibreOffice\program\soffice.exe" `
    --headless `
    "-env:UserInstallation=file:///$($userInst.Replace('\', '/'))" `
    --convert-to pdf `
    --outdir $work `
    "C:\path\to\test.docx"

# Output: $work\test.pdf
ls $work
Remove-Item -Recurse -Force $work

Prod fail patterns

• 60s timeout → PDF lớn (>100 page) có thể quá. Xem LibreOfficeDocumentConverter — tăng timeout nếu cần
• Locked font fallback → Be Vietnam Pro missing → text render hỏng. Install font trên server
• Concurrent request lock → mỗi request 1 UserInstallation dir riêng → tránh lock

Debug playbook — prod error

HTTP 500 all site

Xem gotcha #25 (docs/gotchas.md):

# Likely config lock:
& "$env:SystemRoot\system32\inetsrv\appcmd.exe" list config -section:system.webServer/webSocket
# → overrideMode="Deny" → fix:
& "$env:SystemRoot\system32\inetsrv\appcmd.exe" unlock config -section:system.webServer/webSocket

HTTP 502 Bad Gateway (Admin/User → API)

1. Check API up: curl http://127.0.0.1:5443/health/live
   - Down → restart API site + check stdout log
2. Check ARR enabled: IIS Manager > server level > Application Request Routing
   - "Enable proxy" phải tick
3. Check URL Rewrite rule fe web.config
   - action type="Rewrite" url="http://127.0.0.1:5443/{R:0}"

SignalR 401 (WebSocket connect fail)

Xem gotcha #26:

1. FE console: check ?access_token= query có trong negotiate URL không
2. BE log: JwtBearer OnMessageReceived có fire cho /hubs/* không
3. IIS WebSocket module: Install-WindowsFeature Web-WebSockets (đã có)
4. Section unlock: appcmd unlock config -section:system.webServer/webSocket

Login "Network Error"

Xem docs/gotchas.md CORS + HTTPS redirect:

1. User gõ http://admin.solutions.com.vn → không redirect → CORS block
2. Fix: SPA web.config PHẢI có HTTP→HTTPS rule (đã có)
3. Test: curl -I http://admin.solutions.com.vn → expect 301 Location: https://...

DB connection fail

# 1. SQL service up?
Get-Service MSSQL* 

# 2. TCP enabled?
Import-Module SqlServer
# Hoặc check registry:
Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL*.SQLEXPRESS\MSSQLServer\SuperSocketNetLib\Tcp"

# 3. vrapp login OK?
sqlcmd -S .\SQLEXPRESS -U vrapp -P <pw> -Q "SELECT DB_NAME()"
# Expect: SolutionErp

# 4. appsettings connection string (qua Gitea secrets)
# Check C:\inetpub\solution-erp\api\appsettings.Production.json có ConnectionStrings:DefaultConnection

Deploy steps (CI/CD xanh)

Gitea Actions workflow: .gitea/workflows/deploy.yml. Flow:

Push to main
  → Runner pick up job
  → checkout repo
  → setup .NET 10 + Node 20
  → npm ci (fe-admin + fe-user, rolldown native binding OK nếu fresh node_modules)
  → dotnet restore + publish
  → npm run build (fe-admin + fe-user)
  → render appsettings.Production.json từ secrets (JWT_SECRET, DB_CONNECTION)
  → stop app pool SolutionErp-Api
  → Copy-Item publish/dist → C:\inetpub\solution-erp\{api,fe-admin,fe-user} (Remove-Item -Exclude web.config first)
  → start app pool
  → curl /health/ready → must be 200 trong 30s
  → report status

Manual deploy (emergency)

# Local build
dotnet publish src/Backend/SolutionErp.Api -c Release -o .\publish\api
cd fe-admin; npm ci; npm run build; cd ..
cd fe-user;  npm ci; npm run build; cd ..

# Scp sang server (cần plink/pscp hoặc rsync)
scp -r .\publish\api\*           user@server:C:/inetpub/solution-erp/api/
scp -r .\fe-admin\dist\*         user@server:C:/inetpub/solution-erp/fe-admin/
scp -r .\fe-user\dist\*          user@server:C:/inetpub/solution-erp/fe-user/

# Trên server:
Restart-WebAppPool -Name "SolutionErp-Api"
curl http://127.0.0.1:5443/health/ready

Backup + recovery

# DB backup (script sẵn, chưa schedule):
& "C:\inetpub\solution-erp\scripts\backup-sql.ps1"
# Output: backup/SolutionErp_<ts>.bak (compressed + retention 30d)

# Schedule daily 03:00:
schtasks /create /tn "SolutionErp Backup" `
    /tr "powershell -ExecutionPolicy Bypass -File C:\inetpub\solution-erp\scripts\backup-sql.ps1" `
    /sc DAILY /st 03:00 /ru SYSTEM

Restore: xem docs/guides/runbook.md.

Rotate credentials (Phase 5.1 backlog)

• SQL sa password (rotate)
• SQL vrapp password (update Gitea secret DB_CONNECTION + appsettings.Production.json)
• JWT secret (update Gitea secret JWT_SECRET, next deploy sẽ lan tỏa. Tất cả token cũ invalid)
• Gitea runner registration token (re-register service)
• Admin default Admin@123456 (đổi qua /system/users admin UI ngay sau deploy)

Hardening — IPv4/IPv6 port hijack (G-084 VietReport incident)

Bài học từ VPS shared với VIETREPORT (2026-04-23): VietReport team deploy Next.js app chiếm port 3000 (0.0.0.0 bind) khiến Gitea bị đẩy sang IPv6-only [::]:3000 → IIS ARR localhost:3000 resolve IPv4 first → hit Next.js thay vì Gitea → git.baocaogiaoduc.vn trả homepage VietReport.

3 rules áp dụng cho mọi service trên VPS shared:

1. Reverse-proxy luôn dùng IP literal 127.0.0.1, không dùng localhost

   • IIS ARR rewrite rule: http://127.0.0.1:5443/{R:0} ✓
   • Health check curl: curl http://127.0.0.1:5443/health/live ✓
   • Windows DNS resolver có thể cache IPv6 first → fail nếu service bind IPv4-only

2. Backend services bind loopback IPv4 explicit, không 0.0.0.0

   • ASP.NET Core Kestrel (standalone): UseUrls("http://127.0.0.1:5443") hoặc env ASPNETCORE_URLS=http://127.0.0.1:5443
   • IIS ASP.NET Core Module out-of-process: ANCM tự inject port ephemeral → KHÔNG cần manual (OK)
   • Nếu deploy Kestrel standalone qua NSSM (tương lai): hardcode 127.0.0.1 trong appsettings.Production.json

3. Service dependency cho boot order khi nhiều services cùng port family

   • NSSM: nssm set <svc> DependOnService <other>
   • Không cần cho SOLUTION_ERP hiện tại (API in IIS app pool, không NSSM service)

Hiện trạng SOLUTION_ERP — risk THẤP:

• API host trong IIS app pool out-of-process → ANCM quản lý port Kestrel ephemeral
• FE gọi trực tiếp https://api.solutions.com.vn qua CORS (không ARR proxy)
• Không có standalone Kestrel service trên port cố định
• Nhưng tương lai nếu thêm reverse proxy (fe-admin/user → /api → api.solutions.com.vn, hoặc /hubs for SignalR) → PHẢI dùng 127.0.0.1 không localhost

Related

• docs/guides/deployment-iis.md — first-time setup
• docs/guides/runbook.md — operations guide chi tiết
• docs/guides/cicd.md — CI/CD pipeline
• docs/gotchas.md — #25 webSocket lock, #26 SignalR, #28 LibreOffice 404, #29 PS 5.1 UTF-16, #33 IPv4/IPv6 port hijack (G-084)
• scripts/deploy-iis.ps1 · scripts/backup-sql.ps1 · scripts/install-libreoffice.ps1
• .gitea/workflows/deploy.yml — CI/CD definition