markmorado/pvemanager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

fix(update): host-side watchdog replaces in-container rebuild

Browse Source 
Root cause: docker compose down killed the nohup script inside the
container before rebuild could complete — only git pull was visible
(updated VERSION file).

Changes:
- update_host.sh: new host-side watchdog (poll .update_trigger every 3s)
  runs: git pull → docker compose down → build --no-cache app → up -d
  PROJECT_DIR and User are dynamic (set at install time, not hardcoded)
- update_service.py: perform_update() now writes .update_trigger instead
  of running git/docker commands inside the container
- deploy.sh: added install_update_watchdog() + --watchdog CLI option;
  generates /etc/systemd/system/pvemanager-update.service dynamically
  with correct User= and WorkingDirectory= for any install path/user
- .gitignore: ignore .update_trigger and generated systemd/ files
- CHANGELOG, VERSION, WIKI: bump to v1.1.1, document new update flow
This commit is contained in:
markmorado
2026-03-08 19:02:40 +01:00
parent b3db346cfb
commit b8c6e52c9a
7 changed files with 321 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.gitignore vendored
View File

@@ -22,6 +22,12 @@ node_modules/
*.swp
*.swo
# Update trigger (written by panel, consumed by pvemanager-update.service on host)
.update_trigger
# Generated systemd files (created by deploy.sh when not root)
systemd/
# Environment
.env
.env.local

16
CHANGELOG.md
View File

@@ -4,6 +4,22 @@ All notable changes to PVEmanager will be documented in this file.
---
## [v1.1.1] - 2026-03-08
### 🔧 Fix: Update System — Host-side Rebuild
- **Root cause fixed**: `docker compose down` убивал cgroup контейнера вместе с `nohup`-скриптом внутри него — rebuild никогда не завершался, был виден только результат `git pull` (обновлённый `VERSION`)
- **Новый подход**: нажатие «Обновить панель» теперь записывает файл-триггер `.update_trigger` в корень проекта
- Host-side watchdog (`pvemanager-update.service` / `update_host.sh`) обнаруживает триггер и выполняет на хосте:
`git pull → docker compose down → docker compose build --no-cache app → docker compose up -d`
- Watchdog работает как systemd-сервис на хосте — не зависит от жизни контейнера
- `PROJECT_DIR` определяется динамически в момент установки (не хардкодится)
- `User=` в systemd-юните совпадает с пользователем, запустившим `deploy.sh --watchdog`
- Установка / переустановка watchdog: `sudo ./deploy.sh --watchdog`
- Лог обновления: `./logs/update_host.log`
---
## [v1.1.0] - 2026-03-08
### 🚀 New Features

2
VERSION
View File

@@ -1 +1 @@
1.1.0
1.1.1

43
WIKI.md
View File

@@ -1,6 +1,6 @@
# 📖 PVEmanager - Documentation
> Complete guide for installation, configuration and usage of PVEmanager v1.1.0
> Complete guide for installation, configuration and usage of PVEmanager v1.1.1
---
@@ -1194,6 +1194,45 @@ The repository setting affects:
> **Note:** Make sure the selected repository contains valid `VERSION` and `CHANGELOG.md` files.
### How the Update Process Works
Starting from v1.1.1, updates are executed by a **host-side watchdog** (`pvemanager-update.service`) instead of inside the container. This solves the fundamental problem where `docker compose down` would kill the in-container script before the rebuild could complete.
```
[UI: "Update Panel"] → perform_update() → writes .update_trigger
↓ (poll every 3 sec)
[pvemanager-update.service on HOST]
detects trigger → removes it → runs:
git pull
docker compose down
docker compose build --no-cache app
docker compose up -d
```
**Logs:** `./logs/update_host.log` (visible from the panel)
### Installing the Update Watchdog
The watchdog must be installed once on the host machine. It is automatically set up when you run `deploy.sh`, or you can install/reinstall it separately:
```bash
# Install or reinstall (requires root for systemd)
sudo ./deploy.sh --watchdog
# Verify it is running
systemctl status pvemanager-update
# Watch live update log
tail -f logs/update_host.log
```
**Requirements:**
- systemd on the host
- The user who runs `deploy.sh --watchdog` (or `$SUDO_USER`) must be in the `docker` group (root is always allowed)
- The `compose.yml` volume `.:/project:rw` must be present (it is by default)
> **Note for non-root users:** If you run `./deploy.sh --watchdog` without sudo, it generates ready-to-use files in `./systemd/` and prints the exact `sudo` commands needed to install them.
### Notification Settings
- In-App: Always enabled
@@ -1510,4 +1549,4 @@ A: Not yet, but planned for future versions.
---
*Last updated: March 2026*
*Version: 1.1.0*
*Version: 1.1.1*

197
backend/app/services/update_service.py
View File

@@ -355,175 +355,58 @@ def get_update_status() -> Dict[str, Any]:
async def perform_update() -> Dict[str, Any]:
"""
Выполнить обновление системы
1. Git pull
2. Docker compose build
3. Docker compose up
Выполнить обновление системы.
Записывает файл-триггер .update_trigger в PROJECT_ROOT.
Watchdog-сервис на хосте (pvemanager-update.service / update_host.sh)
обнаруживает триггер и выполняет:
git pull → docker compose down → docker compose build --no-cache app → docker compose up -d
Такой подход необходим, потому что «docker compose down» убивает сам контейнер
(и любой nohup-процесс внутри него) прежде, чем успевает завершиться rebuild.
"""
global update_status
if update_status["is_updating"]:
return {"success": False, "error": "Update already in progress"}
# Проверяем доступность
if not is_git_available():
return {"success": False, "error": "Git not available"}
if not is_project_mounted():
return {"success": False, "error": "Project not mounted"}
# Добавляем safe.directory для git
ensure_safe_directory()
# Настраиваем git для публичного доступа
configure_git_for_public_access()
# Получаем URL репозитория из настроек и устанавливаем remote
return {
"success": False,
"error": (
"Project directory not mounted. "
"Make sure the compose volume '.:/project:rw' is present "
"and pvemanager-update.service is running on the host."
),
}
trigger_path = os.path.join(PROJECT_ROOT, ".update_trigger")
try:
repo_url = get_repository_url_from_settings()
subprocess.run(
["git", "remote", "set-url", "origin", repo_url],
cwd=PROJECT_ROOT,
capture_output=True,
timeout=10
)
logger.info(f"Set git remote to: {repo_url}")
except Exception as e:
logger.warning(f"Could not set git remote: {e}")
# Сбросить статус
with open(trigger_path, "w") as f:
f.write(datetime.now().isoformat())
logger.info(f"Update trigger written to {trigger_path}")
except OSError as e:
logger.error(f"Failed to write update trigger: {e}")
return {"success": False, "error": f"Cannot write update trigger: {e}"}
# Обновляем in-memory статус — UI-баннер будет показан сразу
update_status = {
"is_updating": True,
"started_at": datetime.now().isoformat(),
"stage": "initializing",
"progress": 0,
"stage": "restarting",
"progress": 80,
"error": None,
"completed": False
"completed": False,
}
# Persistent log file (survives via volume mount ./logs:/app/logs)
UPDATE_LOG = "/app/logs/update.log"
def _detect_branch() -> str:
"""Detect current git branch, falling back to 'main'"""
try:
r = subprocess.run(
["git", "symbolic-ref", "--short", "HEAD"],
cwd=PROJECT_ROOT,
capture_output=True,
text=True,
timeout=5,
)
if r.returncode == 0 and r.stdout.strip():
return r.stdout.strip()
except Exception:
pass
return "main"
try:
# Stage 1: Git pull
update_status["stage"] = "pulling"
update_status["progress"] = 10
git_env = {**os.environ, "GIT_TERMINAL_PROMPT": "0"}
branch = _detect_branch()
pull_result = subprocess.run(
["git", "pull", "origin", branch],
cwd=PROJECT_ROOT,
capture_output=True,
text=True,
timeout=120,
env=git_env,
)
logger.info(f"git pull stdout: {pull_result.stdout}")
logger.info(f"git pull stderr: {pull_result.stderr}")
if pull_result.returncode != 0:
# Fallback: fetch + reset --hard
fetch_result = subprocess.run(
["git", "fetch", "origin", branch],
cwd=PROJECT_ROOT,
capture_output=True,
text=True,
timeout=120,
env=git_env,
)
if fetch_result.returncode != 0:
raise Exception(f"Git fetch failed: {fetch_result.stderr}")
reset_result = subprocess.run(
["git", "reset", "--hard", f"origin/{branch}"],
cwd=PROJECT_ROOT,
capture_output=True,
text=True,
timeout=30,
)
if reset_result.returncode != 0:
raise Exception(f"Git reset failed: {reset_result.stderr}")
# Stage 2: Docker rebuild
update_status["stage"] = "building"
update_status["progress"] = 30
# Detect which compose command is available
compose_cmd = "docker compose"
try:
check = subprocess.run(
["docker", "compose", "version"],
capture_output=True,
timeout=5,
)
if check.returncode != 0:
compose_cmd = "docker-compose"
except Exception:
compose_cmd = "docker-compose"
# Script runs after this process exits — writes logs to persistent volume
build_script = f"""#!/bin/sh
set -e
cd {PROJECT_ROOT}
echo "[update] Starting docker rebuild at $(date)" >> {UPDATE_LOG}
{compose_cmd} up -d --build app >> {UPDATE_LOG} 2>&1
echo "[update] Docker rebuild finished at $(date)" >> {UPDATE_LOG}
"""
script_path = "/tmp/update_panel.sh"
with open(script_path, "w") as f:
f.write(build_script)
os.chmod(script_path, 0o755)
with open(UPDATE_LOG, "a") as f:
f.write(f"[update] git pull done (branch={branch}), launching rebuild...\n")
# Launch detached so it survives container restart
subprocess.Popen(
["nohup", "sh", script_path],
cwd=PROJECT_ROOT,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
update_status["stage"] = "restarting"
update_status["progress"] = 80
return {
"success": True,
"message": "Update started. The panel will restart shortly.",
}
except subprocess.TimeoutExpired:
update_status["error"] = "Update timeout"
update_status["is_updating"] = False
return {"success": False, "error": "Update timeout"}
except Exception as e:
update_status["error"] = str(e)
update_status["is_updating"] = False
logger.error(f"Update failed: {e}")
return {"success": False, "error": str(e)}
return {
"success": True,
"message": (
"Update triggered. The host watchdog (pvemanager-update.service) "
"will now run git pull, rebuild and restart the panel."
),
}
def reset_update_status():

111
deploy.sh
View File

@@ -598,6 +598,7 @@ main() {
check_requirements
create_env_file
install_update_watchdog
# Ask deployment mode
echo ""
@@ -761,8 +762,110 @@ ALLOWED_HOSTS=localhost,127.0.0.1
EOF
}
# Install the host-side update watchdog (systemd service)
install_update_watchdog() {
local project_dir
project_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
local service_name="pvemanager-update"
local env_file="/etc/pvemanager-update.env"
local service_file="/etc/systemd/system/${service_name}.service"
local watchdog_script="$project_dir/update_host.sh"
local run_user="${SUDO_USER:-$USER}"
print_info "Installing update watchdog (systemd service)..."
print_info " Project directory : $project_dir"
print_info " Running as user : $run_user"
# Make sure the watchdog script is executable
chmod +x "$watchdog_script"
# Check that the user is in the docker group (root always has access, skip check)
if [ "$run_user" != "root" ] && ! id -nG "$run_user" | grep -qw docker; then
print_warning "User '$run_user' is not in the 'docker' group."
print_warning "The watchdog needs docker access. Run:"
print_warning " sudo usermod -aG docker $run_user (then log out/in)"
fi
# Write environment file (stores PROJECT_DIR so systemd can pass it to the script)
if [ "$(id -u)" -eq 0 ]; then
cat > "$env_file" <<EOF
PROJECT_DIR=$project_dir
EOF
print_success "Wrote $env_file"
# Write the systemd unit
cat > "$service_file" <<EOF
[Unit]
Description=PVEmanager host-side update watchdog
After=docker.service
Requires=docker.service
[Service]
Type=simple
User=$run_user
WorkingDirectory=$project_dir
EnvironmentFile=$env_file
ExecStart=$watchdog_script
Restart=always
RestartSec=5
StandardOutput=append:$project_dir/logs/update_host.log
StandardError=append:$project_dir/logs/update_host.log
[Install]
WantedBy=multi-user.target
EOF
print_success "Wrote $service_file"
systemctl daemon-reload
systemctl enable "$service_name"
systemctl restart "$service_name"
print_success "pvemanager-update.service is active and enabled"
else
# Not root — generate the files locally and show the user what to run with sudo
print_warning "Not running as root. Generating systemd files locally..."
mkdir -p "$project_dir/systemd"
local local_env="$project_dir/systemd/pvemanager-update.env"
local local_svc="$project_dir/systemd/pvemanager-update.service"
cat > "$local_env" <<EOF
PROJECT_DIR=$project_dir
EOF
cat > "$local_svc" <<EOF
[Unit]
Description=PVEmanager host-side update watchdog
After=docker.service
Requires=docker.service
[Service]
Type=simple
User=$run_user
WorkingDirectory=$project_dir
EnvironmentFile=$env_file
ExecStart=$watchdog_script
Restart=always
RestartSec=5
StandardOutput=append:$project_dir/logs/update_host.log
StandardError=append:$project_dir/logs/update_host.log
[Install]
WantedBy=multi-user.target
EOF
print_success "Files generated in $project_dir/systemd/"
print_info "Run the following commands with sudo to install the watchdog:"
echo ""
echo " sudo cp \"$local_env\" \"$env_file\""
echo " sudo cp \"$local_svc\" \"$service_file\""
echo " sudo systemctl daemon-reload"
echo " sudo systemctl enable $service_name"
echo " sudo systemctl start $service_name"
echo ""
fi
}
show_help() {
echo "PVEmanager Deployment Tool v1.0"
echo ""
echo "Usage: $0 [OPTIONS]"
echo ""
@@ -780,11 +883,13 @@ show_help() {
echo " $0 --stop Stop all services"
echo " $0 --restart Restart all services"
echo " $0 --logs Show live logs"
echo " $0 --watchdog Install/reinstall the host-side update watchdog (systemd)"
echo ""
echo "Examples:"
echo " $0 --standalone"
echo " $0 --nginx example.com"
echo " $0 --nginx example.com admin@example.com"
echo " sudo $0 --watchdog"
}
# Parse command line arguments
@@ -822,6 +927,10 @@ if [ $# -gt 0 ]; then
docker compose -f compose.yml -f compose.prod.yml logs -f 2>/dev/null || docker compose logs -f
exit 0
;;
--watchdog)
install_update_watchdog
exit 0
;;
*)
print_error "Unknown option: $1"
show_help

107
update_host.sh Executable file
View File

@@ -0,0 +1,107 @@
#!/bin/bash
# PVEmanager — Host-side update watchdog
# Запускается через systemd (pvemanager-update.service) под тем же пользователем,
# который владеет проектом и имеет доступ к Docker.
#
# Алгоритм:
# 1. Каждые 3 секунды проверяет наличие файла .update_trigger в PROJECT_DIR
# 2. При обнаружении: удаляет триггер → git pull → docker compose down →
# docker compose build --no-cache app → docker compose up -d → пишет лог
set -euo pipefail
# PROJECT_DIR передаётся systemd через EnvironmentFile=/etc/pvemanager-update.env
# Если запущен вручную — использует директорию скрипта как fallback.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_DIR="${PROJECT_DIR:-$SCRIPT_DIR}"
TRIGGER="$PROJECT_DIR/.update_trigger"
LOG_DIR="$PROJECT_DIR/logs"
LOG_FILE="$LOG_DIR/update_host.log"
POLL_INTERVAL=3 # секунд
# ── helpers ────────────────────────────────────────────────────────────────────
log() {
local msg="[$(date '+%Y-%m-%d %H:%M:%S')] $*"
echo "$msg"
mkdir -p "$LOG_DIR"
echo "$msg" >> "$LOG_FILE"
}
# Определить рабочую команду docker compose
detect_compose_cmd() {
if docker compose version &>/dev/null 2>&1; then
echo "docker compose"
elif command -v docker-compose &>/dev/null; then
echo "docker-compose"
else
log "ERROR: docker compose not found"
exit 1
fi
}
# ── main loop ──────────────────────────────────────────────────────────────────
log "=== PVEmanager update watchdog started ==="
log "PROJECT_DIR : $PROJECT_DIR"
log "TRIGGER : $TRIGGER"
log "LOG_FILE : $LOG_FILE"
log "POLL_INTERVAL: ${POLL_INTERVAL}s"
COMPOSE_CMD=$(detect_compose_cmd)
log "COMPOSE_CMD : $COMPOSE_CMD"
while true; do
if [ -f "$TRIGGER" ]; then
log "--- Update trigger detected ---"
# Немедленно удаляем триггер, чтобы повторные нажатия не накапливались
rm -f "$TRIGGER"
cd "$PROJECT_DIR"
# ── Step 1: git pull ───────────────────────────────────────────────────
log "[1/4] git pull..."
BRANCH=$(git symbolic-ref --short HEAD 2>/dev/null || echo "main")
export GIT_TERMINAL_PROMPT=0
if git pull origin "$BRANCH" >> "$LOG_FILE" 2>&1; then
log "[1/4] git pull OK (branch: $BRANCH)"
else
log "[1/4] git pull failed — trying fetch + reset --hard"
git fetch origin "$BRANCH" >> "$LOG_FILE" 2>&1 || true
git reset --hard "origin/$BRANCH" >> "$LOG_FILE" 2>&1 || {
log "[1/4] ERROR: git update failed, aborting update"
continue
}
log "[1/4] git reset --hard OK"
fi
# ── Step 2: docker compose down ───────────────────────────────────────
log "[2/4] docker compose down..."
$COMPOSE_CMD down >> "$LOG_FILE" 2>&1 && log "[2/4] docker compose down OK" \
|| log "[2/4] WARNING: docker compose down returned non-zero (continuing)"
# ── Step 3: docker compose build --no-cache app ───────────────────────
log "[3/4] docker compose build --no-cache app..."
if $COMPOSE_CMD build --no-cache app >> "$LOG_FILE" 2>&1; then
log "[3/4] build OK"
else
log "[3/4] ERROR: build failed — aborting, containers NOT restarted"
continue
fi
# ── Step 4: docker compose up -d ──────────────────────────────────────
log "[4/4] docker compose up -d..."
if $COMPOSE_CMD up -d >> "$LOG_FILE" 2>&1; then
log "[4/4] docker compose up -d OK"
else
log "[4/4] ERROR: docker compose up -d failed"
continue
fi
log "=== Update completed successfully ==="
fi
sleep "$POLL_INTERVAL"
done