d085436e49
iis-deploy-runbook SKILL corrected (discovered during the index.html-hardening task):
- physical paths apps\SolutionErp\{Api,Admin,User} were STALE (empty dir) -> real
solution-erp\{api,fe-admin,fe-user} (verified via appcmd list site).
- web.config rule list fixed to the ACTUAL live config (HTTP->HTTPS + SPA-Routes excl
/api + .webmanifest + 3 security headers; NO /api|/hubs ARR proxy -- FE calls api via CORS)
+ documented the S88 cache hardening (index.html no-cache, assets immutable).
- log/appsettings/scp/backup paths swept to the real base.
Session log: index.html no-cache hardening DONE (ca136d8) + lesson reference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
366 lines
14 KiB
Markdown
366 lines
14 KiB
Markdown
---
|
|
name: iis-deploy-runbook
|
|
description: Ops runbook cho SOLUTION_ERP deploy trên Windows Server IIS — 3 site (api/admin/eoffice.solutions.com.vn), win-acme Let's Encrypt, NSSM gitea-runner shared với VIETREPORT, LibreOffice soffice headless. Dùng khi debug 500/502 prod, restart site, rotate cert, fix CI/CD runner, troubleshoot WebSocket, thêm site mới.
|
|
when-to-use:
|
|
- "prod 500 error"
|
|
- "IIS site fail"
|
|
- "cert hết hạn"
|
|
- "win-acme"
|
|
- "gitea runner"
|
|
- "deploy IIS"
|
|
- "restart app pool"
|
|
- "webSocket 500"
|
|
- "reverse proxy FE"
|
|
- "LibreOffice prod"
|
|
---
|
|
|
|
# IIS Deploy Runbook — SOLUTION_ERP
|
|
|
|
> **Context:** VPS Windows Server shared với VIETREPORT project. IIS + URL Rewrite + ARR + WebSockets module + win-acme. Deploy qua Gitea Actions self-hosted runner.
|
|
|
|
## Production topology
|
|
|
|
```
|
|
Internet
|
|
│ 443 (HTTPS)
|
|
▼
|
|
┌─────────────────────────────────────────────────────┐
|
|
│ IIS (Windows Server VPS) │
|
|
│ │
|
|
│ ┌─ api.solutions.com.vn ─┐ ┌─ admin.solutions.com.vn ─┐ ┌─ eoffice.solutions.com.vn ─┐
|
|
│ │ SolutionErp-Api │ │ SolutionErp-Admin │ │ SolutionErp-User │
|
|
│ │ → out-of-process │ │ (static SPA, URL │ │ (static SPA, URL │
|
|
│ │ Kestrel :5443 │ │ Rewrite /api → 5443)│ │ Rewrite...) │
|
|
│ │ ASP.NET Core 10 │ │ React build/ │ │ React build/ │
|
|
│ │ │ │ │ │ │
|
|
│ └────────────────────┘ └────────────────────┘ └────────────────────┘
|
|
│ │
|
|
│ Let's Encrypt (win-acme) — 3 cert auto-renew 60d │
|
|
│ Shared gitea-runner NSSM service (with VIETREPORT) │
|
|
│ LibreOffice 25.8.6 headless │
|
|
│ SQL Server 2019 Express (\\.\SQLEXPRESS) │
|
|
└─────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## 3 IIS sites
|
|
|
|
| Site | Binding | Physical path | Apool | Purpose |
|
|
|---|---|---|---|---|
|
|
| `SolutionErp-Api` | `*:443:api.solutions.com.vn` (+`api.huypham.vn`) HTTPS | `C:\inetpub\solution-erp\api\` | out-of-process Kestrel | ASP.NET Core 10 API (port 5443 internal) |
|
|
| `SolutionErp-Admin` | `*:443:admin.solutions.com.vn` (+`admin.huypham.vn`) HTTPS + `*:80` redirect | `C:\inetpub\solution-erp\fe-admin\` | static (no app pool .NET) | React build fe-admin |
|
|
| `SolutionErp-User` | `*:443:eoffice.solutions.com.vn` HTTPS + `*:80` redirect | `C:\inetpub\solution-erp\fe-user\` | static | React build fe-user |
|
|
|
|
> ⚠️ **Physical paths verified via `appcmd list site` S88** — runbook trước ghi `C:\inetpub\apps\SolutionErp\{Api,Admin,User}` (STALE, dir rỗng). Đường THẬT = `C:\inetpub\solution-erp\{api,fe-admin,fe-user}`.
|
|
|
|
**SPA web.config** (`{fe-admin,fe-user}/public/web.config`, **version-controlled S88** — Vite copy `public/*`→`dist/`→deploy `Copy-Item -Force`; `deploy.yml` cũ `Remove-Item …\* -Exclude web.config` = trước S88 file manage-tay trên server). Rule THẬT (đọc live S88):
|
|
1. **HTTP → HTTPS** redirect (`{HTTPS} ^OFF$`, Permanent; CORS whitelist chỉ https). KHÔNG có `.well-known` exclude (win-acme validation khác).
|
|
2. **SPA Routes** fallback: non-file/non-dir/non-`^/api` → rewrite `/` (index.html). *(KHÔNG có `/api` hay `/hubs` ARR proxy — FE gọi thẳng `https://api.solutions.com.vn` qua CORS.)*
|
|
3. `.webmanifest` mime + 3 security header (X-Content-Type-Options / X-Frame-Options DENY / Referrer-Policy).
|
|
4. **[S88 hardening] Cache:** root `<clientCache cacheControlMode="DisableCache">` → `index.html` (+ SPA-rewrite shell `/dashboard`…) = `Cache-Control: no-cache` (luôn revalidate → chống stale-shell-after-deploy = white-screen S88) · `<location path="assets"><clientCache UseMaxAge 365d + cacheControlCustom="public, immutable">` → hashed asset `immutable, max-age=1y`. **Verify:** `curl -I https://eoffice.solutions.com.vn/` → `no-cache`; `curl -I .../assets/index-*.js` → `immutable`; `/dashboard` → 200 (routing intact).
|
|
|
|
## Quick commands
|
|
|
|
### Restart 1 site
|
|
|
|
```powershell
|
|
# PowerShell as Admin
|
|
Import-Module WebAdministration
|
|
Stop-WebSite -Name "SolutionErp-Api"
|
|
Start-WebSite -Name "SolutionErp-Api"
|
|
|
|
# Hoặc recycle app pool (API out-of-process):
|
|
Restart-WebAppPool -Name "SolutionErp-Api"
|
|
|
|
# Check site status:
|
|
Get-Website -Name "SolutionErp-*" | Format-Table Name, State, Bindings
|
|
```
|
|
|
|
### Xem log API
|
|
|
|
```powershell
|
|
# Serilog file rolling daily
|
|
Get-Content "C:\inetpub\solution-erp\api\Logs\log-$(Get-Date -Format 'yyyyMMdd').txt" -Tail 50
|
|
|
|
# IIS log
|
|
Get-Content "C:\inetpub\logs\LogFiles\W3SVC<ID>\u_ex$(Get-Date -Format 'yyMMdd').log" -Tail 30
|
|
|
|
# Stdout log khi crash startup
|
|
Get-Content "C:\inetpub\solution-erp\api\Logs\stdout_*.log" -Tail 30
|
|
```
|
|
|
|
### Health check
|
|
|
|
```powershell
|
|
# Từ server
|
|
curl http://127.0.0.1:5443/health/live
|
|
curl http://127.0.0.1:5443/health/ready
|
|
|
|
# Từ ngoài
|
|
curl https://api.solutions.com.vn/health/ready
|
|
```
|
|
|
|
## Let's Encrypt cert — win-acme
|
|
|
|
### Check trạng thái
|
|
|
|
```powershell
|
|
# Mở win-acme interactive
|
|
& "C:\tools\win-acme\wacs.exe"
|
|
# Menu > Manage renewals > list — xem 3 cert + next renew date
|
|
|
|
# Hoặc file:
|
|
Get-Content "C:\ProgramData\win-acme\Production\$(hostname)\Renewals\*.renewal.json"
|
|
```
|
|
|
|
### Cert hết hạn emergency
|
|
|
|
```powershell
|
|
# Force renew 1 cert
|
|
& "C:\tools\win-acme\wacs.exe" --renew --force --id {renewal-id}
|
|
|
|
# Full re-issue nếu renewal fail:
|
|
& "C:\tools\win-acme\wacs.exe" # interactive → 'N' create new
|
|
# Chọn: HTTP validation, web root = site physical path, auto install IIS
|
|
```
|
|
|
|
**Gotcha:** Shared runner với VIETREPORT → win-acme HTTP challenge cần `.well-known/acme-challenge/` accessible qua HTTP (port 80). Rule HTTP→HTTPS redirect trong web.config PHẢI **exclude** path này:
|
|
|
|
```xml
|
|
<rule name="Redirect to HTTPS" stopProcessing="true">
|
|
<match url="(.*)" />
|
|
<conditions>
|
|
<add input="{HTTPS}" pattern="off" />
|
|
<add input="{REQUEST_URI}" pattern="^/\.well-known/" negate="true" />
|
|
</conditions>
|
|
<action type="Redirect" url="https://{HTTP_HOST}/{R:1}" />
|
|
</rule>
|
|
```
|
|
|
|
## Gitea Actions runner (NSSM service)
|
|
|
|
### Status
|
|
|
|
```powershell
|
|
# NSSM service name: gitea-runner (shared với VIETREPORT)
|
|
Get-Service gitea-runner
|
|
nssm status gitea-runner
|
|
|
|
# Restart
|
|
Restart-Service gitea-runner
|
|
|
|
# Log
|
|
Get-Content "C:\tools\gitea-runner\logs\act_runner.log" -Tail 50
|
|
```
|
|
|
|
### Token rotate (nếu runner disconnected)
|
|
|
|
```powershell
|
|
# Stop service
|
|
Stop-Service gitea-runner
|
|
|
|
# Re-register qua Gitea admin UI → Actions → Runners → get new registration token
|
|
& "C:\tools\gitea-runner\act_runner.exe" register `
|
|
--instance https://git.baocaogiaoduc.vn `
|
|
--token <new-token> `
|
|
--no-interactive
|
|
|
|
# Start lại
|
|
Start-Service gitea-runner
|
|
```
|
|
|
|
## LibreOffice headless (PDF / docx converter)
|
|
|
|
### Check install
|
|
|
|
```powershell
|
|
& "C:\Program Files\LibreOffice\program\soffice.exe" --version
|
|
# → LibreOffice 25.8.6.x
|
|
```
|
|
|
|
### Test convert manual
|
|
|
|
```powershell
|
|
# Tạo temp dir isolated (mô phỏng per-request pattern của LibreOfficeDocumentConverter)
|
|
$work = New-Item -ItemType Directory -Path "$env:TEMP\lo-test-$(Get-Random)"
|
|
$userInst = "$work\userinst"
|
|
|
|
& "C:\Program Files\LibreOffice\program\soffice.exe" `
|
|
--headless `
|
|
"-env:UserInstallation=file:///$($userInst.Replace('\', '/'))" `
|
|
--convert-to pdf `
|
|
--outdir $work `
|
|
"C:\path\to\test.docx"
|
|
|
|
# Output: $work\test.pdf
|
|
ls $work
|
|
Remove-Item -Recurse -Force $work
|
|
```
|
|
|
|
### Prod fail patterns
|
|
|
|
- **60s timeout** → PDF lớn (>100 page) có thể quá. Xem `LibreOfficeDocumentConverter` — tăng timeout nếu cần
|
|
- **Locked font fallback** → Be Vietnam Pro missing → text render hỏng. Install font trên server
|
|
- **Concurrent request lock** → mỗi request 1 `UserInstallation` dir riêng → tránh lock
|
|
|
|
## Debug playbook — prod error
|
|
|
|
### HTTP 500 all site
|
|
|
|
Xem gotcha #25 (docs/gotchas.md):
|
|
```powershell
|
|
# Likely config lock:
|
|
& "$env:SystemRoot\system32\inetsrv\appcmd.exe" list config -section:system.webServer/webSocket
|
|
# → overrideMode="Deny" → fix:
|
|
& "$env:SystemRoot\system32\inetsrv\appcmd.exe" unlock config -section:system.webServer/webSocket
|
|
```
|
|
|
|
### HTTP 502 Bad Gateway (Admin/User → API)
|
|
|
|
```
|
|
1. Check API up: curl http://127.0.0.1:5443/health/live
|
|
- Down → restart API site + check stdout log
|
|
2. Check ARR enabled: IIS Manager > server level > Application Request Routing
|
|
- "Enable proxy" phải tick
|
|
3. Check URL Rewrite rule fe web.config
|
|
- action type="Rewrite" url="http://127.0.0.1:5443/{R:0}"
|
|
```
|
|
|
|
### SignalR 401 (WebSocket connect fail)
|
|
|
|
Xem gotcha #26:
|
|
```
|
|
1. FE console: check ?access_token= query có trong negotiate URL không
|
|
2. BE log: JwtBearer OnMessageReceived có fire cho /hubs/* không
|
|
3. IIS WebSocket module: Install-WindowsFeature Web-WebSockets (đã có)
|
|
4. Section unlock: appcmd unlock config -section:system.webServer/webSocket
|
|
```
|
|
|
|
### Login "Network Error"
|
|
|
|
Xem `docs/gotchas.md` CORS + HTTPS redirect:
|
|
```
|
|
1. User gõ http://admin.solutions.com.vn → không redirect → CORS block
|
|
2. Fix: SPA web.config PHẢI có HTTP→HTTPS rule (đã có)
|
|
3. Test: curl -I http://admin.solutions.com.vn → expect 301 Location: https://...
|
|
```
|
|
|
|
### DB connection fail
|
|
|
|
```powershell
|
|
# 1. SQL service up?
|
|
Get-Service MSSQL*
|
|
|
|
# 2. TCP enabled?
|
|
Import-Module SqlServer
|
|
# Hoặc check registry:
|
|
Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL*.SQLEXPRESS\MSSQLServer\SuperSocketNetLib\Tcp"
|
|
|
|
# 3. vrapp login OK?
|
|
sqlcmd -S .\SQLEXPRESS -U vrapp -P <pw> -Q "SELECT DB_NAME()"
|
|
# Expect: SolutionErp
|
|
|
|
# 4. appsettings connection string (qua Gitea secrets)
|
|
# Check C:\inetpub\solution-erp\api\appsettings.Production.json có ConnectionStrings:DefaultConnection
|
|
```
|
|
|
|
## Deploy steps (CI/CD xanh)
|
|
|
|
Gitea Actions workflow: `.gitea/workflows/deploy.yml`. Flow:
|
|
|
|
```
|
|
Push to main
|
|
→ Runner pick up job
|
|
→ checkout repo
|
|
→ setup .NET 10 + Node 20
|
|
→ npm ci (fe-admin + fe-user, rolldown native binding OK nếu fresh node_modules)
|
|
→ dotnet restore + publish
|
|
→ npm run build (fe-admin + fe-user)
|
|
→ render appsettings.Production.json từ secrets (JWT_SECRET, DB_CONNECTION)
|
|
→ stop app pool SolutionErp-Api
|
|
→ Copy-Item publish/dist → C:\inetpub\solution-erp\{api,fe-admin,fe-user} (Remove-Item -Exclude web.config first)
|
|
→ start app pool
|
|
→ curl /health/ready → must be 200 trong 30s
|
|
→ report status
|
|
```
|
|
|
|
### Manual deploy (emergency)
|
|
|
|
```powershell
|
|
# Local build
|
|
dotnet publish src/Backend/SolutionErp.Api -c Release -o .\publish\api
|
|
cd fe-admin; npm ci; npm run build; cd ..
|
|
cd fe-user; npm ci; npm run build; cd ..
|
|
|
|
# Scp sang server (cần plink/pscp hoặc rsync)
|
|
scp -r .\publish\api\* user@server:C:/inetpub/solution-erp/api/
|
|
scp -r .\fe-admin\dist\* user@server:C:/inetpub/solution-erp/fe-admin/
|
|
scp -r .\fe-user\dist\* user@server:C:/inetpub/solution-erp/fe-user/
|
|
|
|
# Trên server:
|
|
Restart-WebAppPool -Name "SolutionErp-Api"
|
|
curl http://127.0.0.1:5443/health/ready
|
|
```
|
|
|
|
## Backup + recovery
|
|
|
|
```powershell
|
|
# DB backup (script sẵn, chưa schedule):
|
|
& "C:\inetpub\solution-erp\scripts\backup-sql.ps1"
|
|
# Output: backup/SolutionErp_<ts>.bak (compressed + retention 30d)
|
|
|
|
# Schedule daily 03:00:
|
|
schtasks /create /tn "SolutionErp Backup" `
|
|
/tr "powershell -ExecutionPolicy Bypass -File C:\inetpub\solution-erp\scripts\backup-sql.ps1" `
|
|
/sc DAILY /st 03:00 /ru SYSTEM
|
|
```
|
|
|
|
Restore: xem `docs/guides/runbook.md`.
|
|
|
|
## Rotate credentials (Phase 5.1 backlog)
|
|
|
|
- [ ] SQL `sa` password (rotate)
|
|
- [ ] SQL `vrapp` password (update Gitea secret `DB_CONNECTION` + appsettings.Production.json)
|
|
- [ ] JWT secret (update Gitea secret `JWT_SECRET`, next deploy sẽ lan tỏa. Tất cả token cũ invalid)
|
|
- [ ] Gitea runner registration token (re-register service)
|
|
- [ ] Admin default `Admin@123456` (đổi qua `/system/users` admin UI ngay sau deploy)
|
|
|
|
## Hardening — IPv4/IPv6 port hijack (G-084 VietReport incident)
|
|
|
|
**Bài học từ VPS shared với VIETREPORT (2026-04-23):** VietReport team
|
|
deploy Next.js app chiếm port 3000 (0.0.0.0 bind) khiến Gitea bị đẩy
|
|
sang IPv6-only `[::]:3000` → IIS ARR `localhost:3000` resolve IPv4
|
|
first → hit Next.js thay vì Gitea → `git.baocaogiaoduc.vn` trả homepage
|
|
VietReport.
|
|
|
|
**3 rules áp dụng cho mọi service trên VPS shared:**
|
|
|
|
1. **Reverse-proxy luôn dùng IP literal `127.0.0.1`**, không dùng `localhost`
|
|
- IIS ARR rewrite rule: `http://127.0.0.1:5443/{R:0}` ✓
|
|
- Health check curl: `curl http://127.0.0.1:5443/health/live` ✓
|
|
- Windows DNS resolver có thể cache IPv6 first → fail nếu service bind IPv4-only
|
|
|
|
2. **Backend services bind loopback IPv4 explicit**, không `0.0.0.0`
|
|
- ASP.NET Core Kestrel (standalone): `UseUrls("http://127.0.0.1:5443")` hoặc env `ASPNETCORE_URLS=http://127.0.0.1:5443`
|
|
- IIS ASP.NET Core Module out-of-process: ANCM tự inject port ephemeral → KHÔNG cần manual (OK)
|
|
- Nếu deploy Kestrel standalone qua NSSM (tương lai): hardcode 127.0.0.1 trong appsettings.Production.json
|
|
|
|
3. **Service dependency cho boot order** khi nhiều services cùng port family
|
|
- NSSM: `nssm set <svc> DependOnService <other>`
|
|
- Không cần cho SOLUTION_ERP hiện tại (API in IIS app pool, không NSSM service)
|
|
|
|
**Hiện trạng SOLUTION_ERP — risk THẤP:**
|
|
|
|
- API host trong IIS app pool out-of-process → ANCM quản lý port Kestrel ephemeral
|
|
- FE gọi trực tiếp `https://api.solutions.com.vn` qua CORS (không ARR proxy)
|
|
- Không có standalone Kestrel service trên port cố định
|
|
- **Nhưng** tương lai nếu thêm reverse proxy (fe-admin/user → `/api` → api.solutions.com.vn, hoặc /hubs for SignalR) → PHẢI dùng 127.0.0.1 không localhost
|
|
|
|
## Related
|
|
|
|
- `docs/guides/deployment-iis.md` — first-time setup
|
|
- `docs/guides/runbook.md` — operations guide chi tiết
|
|
- `docs/guides/cicd.md` — CI/CD pipeline
|
|
- `docs/gotchas.md` — #25 webSocket lock, #26 SignalR, #28 LibreOffice 404, #29 PS 5.1 UTF-16, **#33 IPv4/IPv6 port hijack (G-084)**
|
|
- `scripts/deploy-iis.ps1` · `scripts/backup-sql.ps1` · `scripts/install-libreoffice.ps1`
|
|
- `.gitea/workflows/deploy.yml` — CI/CD definition
|