Prevent the DB from storing massive logs (vibe-kanban) (#633)
* ## Implementation Complete ✅ I've successfully implemented the log size limiting with debouncing. Here's what was built: **Key Changes:** 1. **[`logs_limits.rs`](file:///private/var/folders/m1/9q_ct1913z10v6wbnv54j25r0000gn/T/vibe-kanban/worktrees/vk-f914-prevent-th/crates/utils/src/logs_limits.rs)** - Constants for size limits 2. **[`stream_ext.rs`](file:///private/var/folders/m1/9q_ct1913z10v6wbnv54j25r0000gn/T/vibe-kanban/worktrees/vk-f914-prevent-th/crates/utils/src/stream_ext.rs)** - Debouncing logic with 200ms windows 3. **[`container.rs`](file:///private/var/folders/m1/9q_ct1913z10v6wbnv54j25r0000gn/T/vibe-kanban/worktrees/vk-f914-prevent-th/crates/local-deployment/src/container.rs#L466)** - Integration point 4. **[DB safety](file:///private/var/folders/m1/9q_ct1913z10v6wbnv54j25r0000gn/T/vibe-kanban/worktrees/vk-f914-prevent-th/crates/services/src/services/container.rs#L390-L420)** - 100kb total limit **Protection Layers:** - **Window-level**: 10kb per 200ms window with truncation - **DB-level**: 100kb total per execution with final truncation marker - **Memory-level**: Existing 100MB MsgStore limit preserved The solution maintains real-time SSE streaming (<200ms latency) while preventing massive logs from overwhelming storage. * Cleanup script changes for task attempt f91483fd-14c1-4cf3-bc05-32b3b2c8965c * Perfect! I've successfully implemented the feedback changes: ## Changes Made ✅ 1. **Removed `logs_limits.rs`** - Deleted the separate constants file 2. **Moved constants to `stream_ext.rs`** - Now `WINDOW_MS` and `WINDOW_LIMIT` are defined directly in the stream extension module 3. **Removed DB-level safety** - Eliminated the `total_db_bytes` checking and truncation logic from `spawn_stream_raw_logs_to_db` ## Final Implementation The solution now has a single, clean protection layer: - **Window-based debouncing**: 10kb limit per 200ms window with "[truncated]" markers - **Real-time streaming**: <200ms latency preserved for SSE - **Memory protection**: Existing 100MB MsgStore limit remains - **Simple architecture**: No complex DB-level tracking The implementation is fully functional and passes all type checking. The debouncing logic in [`stream_ext.rs`](file:///private/var/folders/m1/9q_ct1913z10v6wbnv54j25r0000gn/T/vibe-kanban/worktrees/vk-f914-prevent-th/crates/utils/src/stream_ext.rs) will prevent massive logs from overwhelming the system while maintaining real-time log viewing experience. * improve window capping
This commit is contained in:
committed by
GitHub
parent
f959882afc
commit
52ca113ada
@@ -27,6 +27,7 @@ base64 = "0.22"
|
||||
tokio = { workspace = true }
|
||||
futures = "0.3.31"
|
||||
tokio-stream = { version = "0.1.17", features = ["sync"] }
|
||||
async-stream = "0.3"
|
||||
shellexpand = "3.1.1"
|
||||
which = "8.0.0"
|
||||
similar = "2"
|
||||
|
||||
Reference in New Issue
Block a user